Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

gnome3: switch to updateScript #33086

Merged
merged 8 commits into from Mar 1, 2018
Merged

Conversation

jtojnar
Copy link
Contributor

@jtojnar jtojnar commented Dec 26, 2017

Motivation for this change

This will allow us to get rid of those pesky src.nix files.

I also have a script for semi-automatic removal of the src files in the rest of the GNOME packages:

inlinesrcs.py
import os
import os.path
import re
import subprocess
import time

src_import = 'inherit (import ./src.nix fetchurl) name src;'

name_pattern = re.compile(r'name = "(?P<name>[^"]+)-(?P<version>[^"]+)";')
major_pattern = re.compile(r'major = "(?P<major>\d+.\d+)";')
minor_pattern = re.compile(r'minor = "(?P<minor>\d+)";')
url_pattern = re.compile(r'url = "?mirror://gnome/sources/[^/]+/[^/]+/[^";]+\.(?P<ext>(?:tar\.|t)(?:[gx]z|bz2))"?;')
hash_pattern = re.compile(r'sha256 = "(?P<hash>[^"]+?)";')
import_pattern = re.compile('\n( +){}'.format(re.escape(src_import)))

def gen_attrs(name, version, ext, hash, name_attr_exists):
    attr_path_fragment = '' if name_attr_exists else f' attrPath = "gnome3.{name}";'
    return ('\n' r'\1name = "' + name + r'-${version}";' '\n'
        r'\1version = "' + version + '";\n'
        '\n'
        r'\1src = fetchurl {' '\n'
        r'\1  url = "mirror://gnome/sources/' + name + r'/${gnome3.versionBranch version}/${name}.' + ext + '";\n'
        r'\1  sha256 = "' + hash + '";\n'
        r'\1};' '\n\n'
        r'\1passthru = {' '\n'
        r'\1  updateScript = gnome3.updateScript { packageName = "' + name + '";' + attr_path_fragment + ' };' '\n'
        r'\1};')


def inline_src(derivation):
    src = os.path.join(os.path.dirname(derivation), 'src.nix')

    with open(src) as f:
        src_data = f.read()
        name_match = name_pattern.search(src_data)
        major_match = major_pattern.search(src_data)
        minor_match = minor_pattern.search(src_data)
        url_match = url_pattern.search(src_data)
        hash_match = hash_pattern.search(src_data)

        # print(name_match, major_match, minor_match, url_match, hash_match)
        if not name_match or not url_match or not hash_match:
            print("Could not match src of derivation {}".format(derivation))
            return

    name = name_match.group('name')
    version = "{}.{}".format(major_match.group('major'), minor_match.group('minor')) if major_match and minor_match else name_match.group('version')
    ext = url_match.group('ext')
    hash = hash_match.group('hash')
    name_attr_exists = subprocess.call(['nix-instantiate', '-A', name], stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL) == 0
    # print(name, version, ext, hash, name_attr_exists)

    with open(derivation) as f:
        drv_data = f.read()

    with open(derivation, 'w') as f:
        new_attrs = gen_attrs(name, version, ext, hash, name_attr_exists)
        new_drw = import_pattern.sub(new_attrs, drv_data)
        f.write(new_drw)

    time.sleep(0.02)
    subprocess.call(['git', 'rm', src])
    time.sleep(0.02)
    subprocess.call(['git', 'add', derivation])

if __name__ == '__main__':
    derivations = subprocess.run(['rg', '-lF', src_import, 'pkgs'], check=True, stdout=subprocess.PIPE).stdout.decode('utf-8').strip().split('\n')
    for derivation in derivations:
        inline_src(derivation)
Things done
  • Tested using sandboxing (nix.useSandbox on NixOS, or option build-use-sandbox in nix.conf on non-NixOS)
  • Built on platform(s)
    • NixOS
    • macOS
    • other Linux distributions
  • Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
  • Tested compilation of all pkgs that depend on this change using nix-shell -p nox --run "nox-review wip"
  • Tested execution of all binary files (usually in ./result/bin/)
  • Fits CONTRIBUTING.md.

cc @garbas, @lheckemann

python = python3.withPackages (p: [ p.requests ]);
in writeScript "update-${packageName}" ''
PATH=${lib.makeBinPath [ common-updater-scripts coreutils gnugrep gnused python ]}
latest_tag=$(python "${./find-latest-version.py}" "${packageName}" "${versionPolicy}" "stable")
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would like to make the stability configurable but, for some reason, I cannot make update.nix accept arbitrary arguments:

$ nix-instantiate -E 'args@{foo, ...}: builtins.trace args {}' --arg foo 5 --arg bar 6
trace: { foo = <CODE>; }

in writeScript "update-${packageName}" ''
set -o errexit
PATH=${lib.makeBinPath [ common-updater-scripts coreutils gnugrep gnused python ]}
latest_tag=$(python "${./find-latest-version.py}" "${packageName}" "${versionPolicy}" "stable")
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I do not see an easy way to make the stability configurable from maintainers/scripts/update.nix. I guess I will just have to change it manually when working on unstable branch.


let
python = python3.withPackages (p: [ p.requests ]);
in writeScript "update-${packageName}" ''
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I do not really like that this creates a derivation per package updated but maintainers/scripts/update.nix is quite limited.

};

passthru = {
updateScript = gnome3.updateScript { packageName = "evolution"; attrPath = "gnome3.evolution"; };
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I do not like the need to specify the attrPath here very much but update-source-version cannot do without it.

@jtojnar jtojnar merged commit 4a7fc5f into NixOS:master Mar 1, 2018
GNOME automation moved this from In Progress to Done Mar 1, 2018
@jtojnar jtojnar deleted the gnome-updatescript branch March 1, 2018 15:56
@jtojnar jtojnar added the 6.topic: updaters Tooling for (semi-)automated updating of packages label Jun 2, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
GNOME
  
Done
Development

Successfully merging this pull request may close these issues.

None yet

2 participants