Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

fetchFromGitHub: add goPackagePath similar to meta.homepage #49595

Closed
wants to merge 5 commits into from
Closed

fetchFromGitHub: add goPackagePath similar to meta.homepage #49595

wants to merge 5 commits into from

Conversation

Synthetica9
Copy link
Member

@Synthetica9 Synthetica9 commented Nov 1, 2018 •
edited

Motivation for this change

Also updating circleci-cli as an example of how to use this. This would potentially be useful in 284 files:

$ ag --nix -l . ./pkgs | xargs grep --files-with-match "goPackagePath = \"github.com" | xargs grep --files-with-match "fetchFromGitHub" | wc -l
284
Things done
  • Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS)
  • Built on platform(s)
    • NixOS
    • macOS
    • other Linux distributions
  • Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
  • Tested compilation of all pkgs that depend on this change using nix-shell -p nox --run "nox-review wip"
  • Tested execution of all binary files (usually in ./result/bin/)
  • Determined the impact on package closure size (by running nix path-info -S before and after)
  • Fits CONTRIBUTING.md.

@Synthetica9
Copy link
Member Author

P. S. There are only a few packages that wouldn't be able to use this:

ag --nix -l . ./pkgs | xargs grep --files-with-match "buildGoPackage" | xargs grep --files-without-match "goPackagePath = \"github.com" | xargs rg '\s*goPackagePath = "(.*)";' --replace='$1' --no-filename | sort
bosun.org
browsh
code.cloudfoundry.org/cli
code.gitea.io/gitea
gitlab.com/esr/loccount
gitlab.com/gitlab-org/gitaly
gitlab.com/gitlab-org/gitlab-runner
gitlab.com/meutraa/mm
git.tuxfamily.org/boohu/boohu.git
git.zx2c4.com/wireguard-go
golang.org/x/tools
go.mozilla.org/sops
gopkg.in/Netflix-Skunkworks/go-jira.v1
hecate
https://github.com/asciimoo/wuzz
k8s.io/heapster
k8s.io/helm
k8s.io/kops
k8s.io/minikube
k8s.io/minikube
miniflux.app
mynewt.apache.org/newt
ngrok
pkg.deepin.io/dde/api
pkg.deepin.io/dde/daemon
rsc.io/2fa
sigs.k8s.io/kind
sigs.k8s.io/kustomize

@Synthetica9 Synthetica9 changed the title fetchFromGitHub: add goPackagePath similar to meta.homeapage fetchFromGitHub: add goPackagePath similar to meta.homepage Nov 1, 2018
@GrahamcOfBorg
Copy link

Success on x86_64-linux (full log)

Attempted: circleci-cli

Partial log (click to expand)

installing
post-installation fixup
shrinking RPATHs of ELF executables and libraries in /nix/store/5ydi03743j0qw0rrxxkib87bi741fl5h-circleci-cli-0.1.3887-bin
shrinking /nix/store/5ydi03743j0qw0rrxxkib87bi741fl5h-circleci-cli-0.1.3887-bin/bin/circleci-cli
strip is /nix/store/n4hb93w6j076xcjw5pm09rdmc09s075b-binutils-2.30/bin/strip
stripping (with command strip and flags -S) in /nix/store/5ydi03743j0qw0rrxxkib87bi741fl5h-circleci-cli-0.1.3887-bin/bin
patching script interpreter paths in /nix/store/5ydi03743j0qw0rrxxkib87bi741fl5h-circleci-cli-0.1.3887-bin
checking for references to /build in /nix/store/5ydi03743j0qw0rrxxkib87bi741fl5h-circleci-cli-0.1.3887-bin...
strip is /nix/store/n4hb93w6j076xcjw5pm09rdmc09s075b-binutils-2.30/bin/strip
/nix/store/5ydi03743j0qw0rrxxkib87bi741fl5h-circleci-cli-0.1.3887-bin

@GrahamcOfBorg
Copy link

Success on x86_64-linux (full log)

Attempted: circleci-cli, wuzz

Partial log (click to expand)

post-installation fixup
shrinking RPATHs of ELF executables and libraries in /nix/store/5ydi03743j0qw0rrxxkib87bi741fl5h-circleci-cli-0.1.3887-bin
shrinking /nix/store/5ydi03743j0qw0rrxxkib87bi741fl5h-circleci-cli-0.1.3887-bin/bin/circleci-cli
strip is /nix/store/n4hb93w6j076xcjw5pm09rdmc09s075b-binutils-2.30/bin/strip
stripping (with command strip and flags -S) in /nix/store/5ydi03743j0qw0rrxxkib87bi741fl5h-circleci-cli-0.1.3887-bin/bin
patching script interpreter paths in /nix/store/5ydi03743j0qw0rrxxkib87bi741fl5h-circleci-cli-0.1.3887-bin
checking for references to /build in /nix/store/5ydi03743j0qw0rrxxkib87bi741fl5h-circleci-cli-0.1.3887-bin...
strip is /nix/store/n4hb93w6j076xcjw5pm09rdmc09s075b-binutils-2.30/bin/strip
/nix/store/5ydi03743j0qw0rrxxkib87bi741fl5h-circleci-cli-0.1.3887-bin
/nix/store/5lg3x2dwvli1x291gwzz137qjhpnpz0y-wuzz-0.2.0-bin

@GrahamcOfBorg
Copy link

Success on aarch64-linux (full log)

Attempted: circleci-cli, wuzz

Partial log (click to expand)

post-installation fixup
shrinking RPATHs of ELF executables and libraries in /nix/store/vy7gm4zkbcjwxgll9pk67g3wrp0i43pb-circleci-cli-0.1.3887-bin
shrinking /nix/store/vy7gm4zkbcjwxgll9pk67g3wrp0i43pb-circleci-cli-0.1.3887-bin/bin/circleci-cli
strip is /nix/store/qjrnv0qw44bw1hc23zhfh26xd1c25dfs-binutils-2.30/bin/strip
stripping (with command strip and flags -S) in /nix/store/vy7gm4zkbcjwxgll9pk67g3wrp0i43pb-circleci-cli-0.1.3887-bin/bin
patching script interpreter paths in /nix/store/vy7gm4zkbcjwxgll9pk67g3wrp0i43pb-circleci-cli-0.1.3887-bin
checking for references to /build in /nix/store/vy7gm4zkbcjwxgll9pk67g3wrp0i43pb-circleci-cli-0.1.3887-bin...
strip is /nix/store/qjrnv0qw44bw1hc23zhfh26xd1c25dfs-binutils-2.30/bin/strip
/nix/store/vy7gm4zkbcjwxgll9pk67g3wrp0i43pb-circleci-cli-0.1.3887-bin
/nix/store/i69lyz91diwrr6aim3fb8idd8jv59crq-wuzz-0.2.0-bin

@GrahamcOfBorg
Copy link

Success on aarch64-linux (full log)

Attempted: circleci-cli

Partial log (click to expand)

shrinking /nix/store/6a1h5an17vpa5cjapcv6rh237yhf9yg7-kbd-2.0.4/bin/mapscrn
shrinking /nix/store/6a1h5an17vpa5cjapcv6rh237yhf9yg7-kbd-2.0.4/bin/vlock
gzipping man pages under /nix/store/6a1h5an17vpa5cjapcv6rh237yhf9yg7-kbd-2.0.4/share/man/
strip is /nix/store/qjrnv0qw44bw1hc23zhfh26xd1c25dfs-binutils-2.30/bin/strip
stripping (with command strip and flags -S) in /nix/store/6a1h5an17vpa5cjapcv6rh237yhf9yg7-kbd-2.0.4/lib  /nix/store/6a1h5an17vpa5cjapcv6rh237yhf9yg7-kbd-2.0.4/bin
patching script interpreter paths in /nix/store/6a1h5an17vpa5cjapcv6rh237yhf9yg7-kbd-2.0.4
/nix/store/6a1h5an17vpa5cjapcv6rh237yhf9yg7-kbd-2.0.4/bin/unicode_stop: interpreter directive changed from "/bin/sh" to "/nix/store/bpspdmsl0yys24gs70flsvcw3wcnl7lx-bash-4.4-p23/bin/sh"
/nix/store/6a1h5an17vpa5cjapcv6rh237yhf9yg7-kbd-2.0.4/bin/unicode_start: interpreter directive changed from "/bin/sh" to "/nix/store/bpspdmsl0yys24gs70flsvcw3wcnl7lx-bash-4.4-p23/bin/sh"
checking for references to /build in /nix/store/6a1h5an17vpa5cjapcv6rh237yhf9yg7-kbd-2.0.4...
/nix/store/vy7gm4zkbcjwxgll9pk67g3wrp0i43pb-circleci-cli-0.1.3887-bin

@GrahamcOfBorg
Copy link

Failure on aarch64-linux (full log)

Attempted: circleci-cli, gh-ost, saml2aws, wuzz

Partial log (click to expand)

installing
post-installation fixup
shrinking RPATHs of ELF executables and libraries in /nix/store/lprawkx7am2bfy1pianipcrbysdk5vh7-saml2aws-2.6.1-bin
shrinking /nix/store/lprawkx7am2bfy1pianipcrbysdk5vh7-saml2aws-2.6.1-bin/bin/saml2aws
strip is /nix/store/qjrnv0qw44bw1hc23zhfh26xd1c25dfs-binutils-2.30/bin/strip
stripping (with command strip and flags -S) in /nix/store/lprawkx7am2bfy1pianipcrbysdk5vh7-saml2aws-2.6.1-bin/bin
patching script interpreter paths in /nix/store/lprawkx7am2bfy1pianipcrbysdk5vh7-saml2aws-2.6.1-bin
checking for references to /build in /nix/store/lprawkx7am2bfy1pianipcrbysdk5vh7-saml2aws-2.6.1-bin...
strip is /nix/store/qjrnv0qw44bw1hc23zhfh26xd1c25dfs-binutils-2.30/bin/strip
error: build of '/nix/store/y7yjj5sk57m28i29iyy49lcbj6h5qjhl-gh-ost-1.0.36.drv' failed

@GrahamcOfBorg
Copy link

Success on x86_64-linux (full log)

Attempted: circleci-cli, gh-ost, saml2aws, wuzz

Partial log (click to expand)

shrinking /nix/store/dmdwx6lnvb8v34xwfglg8az8ap53nn2n-gh-ost-1.0.36-bin/bin/gh-ost
strip is /nix/store/n4hb93w6j076xcjw5pm09rdmc09s075b-binutils-2.30/bin/strip
stripping (with command strip and flags -S) in /nix/store/dmdwx6lnvb8v34xwfglg8az8ap53nn2n-gh-ost-1.0.36-bin/bin
patching script interpreter paths in /nix/store/dmdwx6lnvb8v34xwfglg8az8ap53nn2n-gh-ost-1.0.36-bin
checking for references to /build in /nix/store/dmdwx6lnvb8v34xwfglg8az8ap53nn2n-gh-ost-1.0.36-bin...
strip is /nix/store/n4hb93w6j076xcjw5pm09rdmc09s075b-binutils-2.30/bin/strip
/nix/store/5ydi03743j0qw0rrxxkib87bi741fl5h-circleci-cli-0.1.3887-bin
/nix/store/dmdwx6lnvb8v34xwfglg8az8ap53nn2n-gh-ost-1.0.36-bin
/nix/store/slkjz35iia23ln5p9q3c806dwhbzn7m4-saml2aws-2.6.1-bin
/nix/store/5lg3x2dwvli1x291gwzz137qjhpnpz0y-wuzz-0.2.0-bin

@Synthetica9 Synthetica9 force-pushed the fetchFromGithubGoPackagePath branch 4 times, most recently from 1a5bd81 to 28daa43 Compare November 1, 2018 19:03
@GrahamcOfBorg
Copy link

Failure on aarch64-linux (full log)

Attempted: circleci-cli, gh-ost, saml2aws, wuzz

Partial log (click to expand)

go/src/github.com/github/gh-ost/vendor/github.com/ngaut/log/crash_unix.go:16:3: undefined: syscall.Dup2
github.com/github/gh-ost/vendor/gopkg.in/gcfg.v1/scanner
github.com/github/gh-ost/vendor/gopkg.in/gcfg.v1
github.com/github/gh-ost/vendor/github.com/siddontang/go-mysql/mysql
github.com/github/gh-ost/vendor/github.com/siddontang/go-mysql/client
github.com/github/gh-ost/go/sql
github.com/github/gh-ost/go/mysql
github.com/github/gh-ost/go/base
builder for '/nix/store/y7yjj5sk57m28i29iyy49lcbj6h5qjhl-gh-ost-1.0.36.drv' failed with exit code 3
error: build of '/nix/store/y7yjj5sk57m28i29iyy49lcbj6h5qjhl-gh-ost-1.0.36.drv' failed

@GrahamcOfBorg
Copy link

Failure on aarch64-linux (full log)

Attempted: circleci-cli, gh-ost, saml2aws, wuzz

Partial log (click to expand)

github.com/github/gh-ost/vendor/github.com/siddontang/go-mysql/mysql
github.com/github/gh-ost/vendor/gopkg.in/gcfg.v1
github.com/github/gh-ost/vendor/github.com/outbrain/golib/sqlutils
github.com/github/gh-ost/vendor/github.com/siddontang/go-mysql/packet
github.com/github/gh-ost/vendor/github.com/siddontang/go-mysql/client
github.com/github/gh-ost/go/sql
github.com/github/gh-ost/go/mysql
github.com/github/gh-ost/go/base
builder for '/nix/store/y7yjj5sk57m28i29iyy49lcbj6h5qjhl-gh-ost-1.0.36.drv' failed with exit code 3
error: build of '/nix/store/y7yjj5sk57m28i29iyy49lcbj6h5qjhl-gh-ost-1.0.36.drv' failed

@GrahamcOfBorg
Copy link

Success on x86_64-linux (full log)

Attempted: circleci-cli, gh-ost, saml2aws, wuzz

Partial log (click to expand)

/nix/store/5ydi03743j0qw0rrxxkib87bi741fl5h-circleci-cli-0.1.3887-bin
/nix/store/dmdwx6lnvb8v34xwfglg8az8ap53nn2n-gh-ost-1.0.36-bin
/nix/store/slkjz35iia23ln5p9q3c806dwhbzn7m4-saml2aws-2.6.1-bin
/nix/store/5lg3x2dwvli1x291gwzz137qjhpnpz0y-wuzz-0.2.0-bin

@GrahamcOfBorg
Copy link

Success on x86_64-linux (full log)

Attempted: circleci-cli, gh-ost, saml2aws, wuzz

Partial log (click to expand)

/nix/store/5ydi03743j0qw0rrxxkib87bi741fl5h-circleci-cli-0.1.3887-bin
/nix/store/dmdwx6lnvb8v34xwfglg8az8ap53nn2n-gh-ost-1.0.36-bin
/nix/store/slkjz35iia23ln5p9q3c806dwhbzn7m4-saml2aws-2.6.1-bin
/nix/store/5lg3x2dwvli1x291gwzz137qjhpnpz0y-wuzz-0.2.0-bin

@matthewbauer
Copy link
Member

This seems like a good idea but it seems a little weird putting something so go-specific in fetchFromGitHub. Could we instead call it something like "shortUrl" (or better yet just "url" if go supports the https:// prefix).

@roberth
Copy link
Member

roberth commented Nov 4, 2018

I think "url" is a bit too general, considering that something fetched by fetchFromGitHub could end up in pretty much any context. If I have a thing with thing.url I'd kind of expect that that's where I can fetch the thing with a simple http request, which is not exactly the case. So even if go supports an https:// prefix I think the name shouldn't be "url".
What about "abbreviation", because it's not really a URL?

@Synthetica9
Copy link
Member Author

This seems like a good idea but it seems a little weird putting something so go-specific in fetchFromGitHub. Could we instead call it something like "shortUrl" (or better yet just "url" if go supports the https:// prefix).

I kinda agree? Maybe do both, so that you can still do inherit (goPackagePath)

@yurrriq
Copy link
Member

yurrriq commented Jan 18, 2019 •
edited

This seems to me like an inappropriate addition to fetchFromGitHub. Some casual (rip)grepping suggests nearly 10k uses, so the convenience added to around 300 packages seems hardly worth adding cruft to many more. With that said, it's not a big deal to just ignore goPackagePath in the remaining 9700-ish packages.

Could we instead update buildGoPackage to choose a sensible default (à la nakedUrl) for goPackagePath, allowing overrides when necessary, e.g. the "few packages" above?

@Synthetica9
Copy link
Member Author

@yurrriq Actually, I really like that. I'm implementing it

@Synthetica9 Synthetica9 changed the base branch from staging to master January 18, 2019 18:32
@Synthetica9
Copy link
Member Author

Whoops, wrong base branch. However, this should be good now

@yurrriq
Copy link
Member

yurrriq commented Jan 19, 2019

👍 from me once the Borg are happy.

yurrriq
yurrriq approved these changes Jan 19, 2019
View reviewed changes
Copy link
Member

@yurrriq yurrriq left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nakedUrl is generic enough and might be useful later. I prefer this approach to the previous one.

yurrriq
yurrriq requested changes Jan 19, 2019
View reviewed changes
Copy link
Member

@yurrriq yurrriq left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

On second thought... this PR should probably include an update to the documentation, describing the default goPackagePath.

FRidh
FRidh requested changes Jan 19, 2019
View reviewed changes
pkgs/top-level/all-packages.nix
@@ -290,7 +291,7 @@ in
then { inherit rev fetchSubmodules; url = "${baseUrl}.git"; }
else ({ url = "${baseUrl}/archive/${rev}.tar.gz"; } // privateAttrs)
) // passthruAttrs // { inherit name; };
in fetcher fetcherArgs // { meta.homepage = baseUrl; inherit rev; };
in fetcher fetcherArgs // { meta.homepage = baseUrl; inherit nakedUrl rev; };
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I disagree with adding nakedUrl. It's a composition of some input values, which just happens to be useful for Go packages. Instead, make the individual values available and do the composition elsewhere (pkgs/development/go-modules/generic/default.nix).

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed. go get has some heuristic to translate goPackagePath to URLs, I don't know how uniform that is. It would be better to detect the src.url in buildGoPackage.

@FRidh
Copy link
Member

FRidh commented Jan 19, 2019

Maybe we should have every fetcher have an URI attribute set that contains the separate components according to https://en.wikipedia.org/wiki/Uniform_Resource_Identifier#Generic_syntax. That way, anyone create whatever composition they like.

@zimbatm
Copy link
Member

zimbatm commented Apr 13, 2019

The principle of removing duplication is good, but the go-specific heuristic should be moved inside of buildGoPackage so that it doesn't complicate all of nixpkgs.

Is that something you would agree to pursue @Synthetica9 ? Otherwise this PR should be closed.

@worldofpeace
Copy link
Contributor

Closing due to lack of response and conflicts.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@yurrriq yurrriq yurrriq requested changes

@FRidh FRidh FRidh requested changes

@alyssais alyssais Awaiting requested review from alyssais

@basvandijk basvandijk Awaiting requested review from basvandijk

@edolstra edolstra Awaiting requested review from edolstra

@infinisil infinisil Awaiting requested review from infinisil

@nbp nbp Awaiting requested review from nbp

@peti peti Awaiting requested review from peti

@Profpatsch Profpatsch Awaiting requested review from Profpatsch Profpatsch is a code owner

@ryantm ryantm Awaiting requested review from ryantm

@ttuegel ttuegel Awaiting requested review from ttuegel

@zimbatm zimbatm Awaiting requested review from zimbatm

Assignees
No one assigned
Labels
6.topic: golang 8.has: documentation 8.has: package (new) 10.rebuild-darwin: 1-10 10.rebuild-linux: 11-100
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
8 participants
@Synthetica9 @GrahamcOfBorg @matthewbauer @roberth @yurrriq @FRidh @zimbatm @worldofpeace