buildGoModule: disable default go module proxy #71049

marsam · 2019-10-12T18:55:55Z

Since GO 1.13, the go command uses by default [1]
GOPROXY="https://proxy.golang.org,direct" as module proxy, which caches
the latest signed tree size and tree hash in $GOPATH[2], hence producing
a different hash every time proxy.golang.org/latest changes.

[1] https://golang.org/cmd/go/#hdr-Module_downloading_and_verification
[2] https://go.googlesource.com/proposal/+/master/design/25530-sumdb.md#command-client

Motivation for this change

Things done

Notify maintainers

cc: @kalbasit, @c00w, @rvolosatovs

Since GO 1.13, the go command caches the lookup results and tiles in $GOPATH[1], hence making the module directory non-deterministic. Use the `-f` flag when removing /sumdb, for compatibility with Go 1.12 because in that version does not exists that directory. [1] https://go.googlesource.com/proposal/+/master/design/25530-sumdb.md#command-client

kalbasit · 2019-10-12T19:09:28Z

pkgs/development/go-modules/generic/default.nix

@@ -48,6 +48,13 @@ let

    GO111MODULE = "on";

+    # Disable proxying to checksum database. Since Golang 1.13, go uses
+    # proxy.golang.org as default GOPROXY, and it will cache the latest signed
+    # tree size and tree hash in $GOPATH/pkg/sumdb/<sumdb-name>/latest, hence


Have you tried simply removing the entire $GOPATH/pkg/sumdb directory instead? If we turn off GOPROXY, we might end up with a diverging result from Go 1.13 used outside of nixpkgs. Context: kubernetes/test-infra#13918

sure, I'm going to try it; but IIUC the test-infra issue was because the jsonpatch retagged the repository

Yes, that's correct. This will eventually happen again, but for those who are using Go 1.13 to develop their project will not notice the retag because of the proxy. If we then try to package their software, it will fail the build leading to confusion.

I think is now done.

c00w · 2019-10-12T19:52:29Z

Which repo are you seeing this with? I need to go file a bug with the go team, but I'm only aware of a problem with contest. Another example could help figure this out :D

…

On Sat, Oct 12, 2019, 15:09 Wael Nasreddine ***@***.***> wrote: ***@***.**** commented on this pull request. ------------------------------ In pkgs/development/go-modules/generic/default.nix <#71049 (comment)>: > @@ -48,6 +48,13 @@ let GO111MODULE = "on"; + # Disable proxying to checksum database. Since Golang 1.13, go uses + # proxy.golang.org as default GOPROXY, and it will cache the latest signed + # tree size and tree hash in $GOPATH/pkg/sumdb/<sumdb-name>/latest, hence Have you tried simply removing the entire $GOPATH/pkg/sumdb directory instead? If we turn off caching, we might end up with a diverging result from Go 1.13 used outside of nixpkgs. Context: kubernetes/test-infra#13918 <kubernetes/test-infra#13918> — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#71049?email_source=notifications&email_token=AADWWNYGCCXOYDEW67ZXFKDQOIOHTA5CNFSM4JAD7HA2YY3PNVWWK3TUL52HS4DFWFIHK3DMKJSXC5LFON2FEZLWNFSXPKTDN5WW2ZLOORPWSZGOCHYS5WY#pullrequestreview-301018843>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AADWWN4RFHYUGCIUUFD3C3DQOIOHTANCNFSM4JAD7HAQ> .

marsam · 2019-10-12T21:01:58Z

@c00w I'm getting different hashes on #71037, and #71037

tflint, for instance:

platform	sha256
locally darwin/NixOS (no sandbox)	0dx0yh6y2hrqzz8bbcsk9jb0wwiraq12labzd42hybxkwayp96sb
ofborg aarch64	13yqa076mq0nhavh3lyq4p51lygxlfmlfs18lzapr9zpakgxkzs2
ofborg x86_64	16dm6g7maclrb8hmn7bk4l5g0w7xzylx4fss550xsd5x2ql1kc3k

c00w · 2019-10-13T00:09:41Z

Can we try deleting pkg/mod/cache/download/sumdb during the install phase? It looks like the proof chain may be non deterministic, so deleting it may fix the output tree.

marsam · 2019-10-14T12:18:17Z

@kalbasit, @c00w I've updated it to remove the /sumdb directory.
I guess this change will change the modSha256 for all buildGoModule packages.

kalbasit · 2019-10-15T20:25:12Z

@GrahamcOfBorg build jx pet gcsfuse

kalbasit · 2019-10-15T20:37:08Z

The modSha256sum has not changed for gcsfuse, jx or pet. Are you sure this is needed?

c00w · 2019-10-15T21:04:28Z

There is perhaps a subtle bug here, since the modules are a fixed output point, nix may reuse the cached module despite a change to how it is generated, since the fixed output hash has not changed. If you rename the derivation - it may show failures.

…

On Tue, Oct 15, 2019, 16:37 Wael Nasreddine ***@***.***> wrote: The modSha256sum has not changed for gcsfuse, jx or pet. Are you sure this is needed? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#71049?email_source=notifications&email_token=AADWWN53VBELYJN23HDFEGDQOYSXZA5CNFSM4JAD7HA2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEBKELHQ#issuecomment-542393758>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AADWWN5EH572HZAPJQM7NO3QOYSXZANCNFSM4JAD7HAQ> .

marsam · 2019-10-16T02:33:03Z

The modSha256sum has not changed for gcsfuse, jx or pet. Are you sure this is needed?

@c00w is right, modSha256 changed but the old hash is available locally so it doesn't build it.

If you agree with this change, I can update the hash of all the derivations affected.

c00w · 2019-10-16T22:25:08Z

Deleting the sum paths sounds good to me. Thanks for cleaning this up :D

…

On Tue, Oct 15, 2019, 22:33 Mario Rodas ***@***.***> wrote: The modSha256sum has not changed for gcsfuse, jx or pet. Are you sure this is needed? @c00w <https://github.com/c00w> is right, modSha256 changed but the old hash is available locally so it doesn't build it. If you agree with this change, I can update the hash of all the derivations affected. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#71049?email_source=notifications&email_token=AADWWN25JKJDL3KPOGXJCSTQOZ4ORA5CNFSM4JAD7HA2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEBK2H5A#issuecomment-542483444>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AADWWNYQV6IWMCGI2PGPZ63QOZ4ORANCNFSM4JAD7HAQ> .

nixos-discourse · 2019-10-18T17:59:21Z

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/make-buildgomodule-failsafe/4425/2

c00w · 2019-10-19T21:17:55Z

I think we should merge this as is without updating the underlying hashes. Currently if a developer is trying to update a module, without this fix they're going to have to do something relatively complicated or get frustrated and give up.

Once this fix is merged we can update the go hashes in a separate PR, or allow it to naturally happen as versions + dependencies get updated. Additionally, since most hashes haven't been changed from go1.12 (when this wasn't a problem), the number of outstanding hashes should be relatively small.

marsam · 2019-10-20T01:39:01Z

yeah, it's blocking various PRs. I'm going ahead and merging this PR, and address the mismatching hashes in another PR.

ref: NixOS/nixpkgs#71049

ofborg bot added the 6.topic: golang label Oct 12, 2019

kalbasit reviewed Oct 12, 2019

View reviewed changes

ofborg bot added 10.rebuild-darwin: 101-500 10.rebuild-linux: 101-500 labels Oct 12, 2019

marsam mentioned this pull request Oct 12, 2019

tflint: 0.12.0 -> 0.12.1 #71035

Merged

10 tasks

c00w approved these changes Oct 12, 2019

View reviewed changes

c00w mentioned this pull request Oct 12, 2019

Inconsistent go1.13 pkg/mod/cache/download folder. golang/go#34866

Closed

marsam force-pushed the buildGoModule-disable-goproxy branch from a748903 to 69ae8a9 Compare October 14, 2019 12:12

ofborg bot added 10.rebuild-darwin: 0 10.rebuild-linux: 0 and removed 10.rebuild-darwin: 101-500 10.rebuild-linux: 101-500 labels Oct 14, 2019

marsam mentioned this pull request Oct 16, 2019

prototool: 1.8.0 -> 1.9.0 #71224

Merged

10 tasks

NobbZ mentioned this pull request Oct 19, 2019

Exercism module no proxy #71391

Merged

10 tasks

andrew-d mentioned this pull request Oct 19, 2019

syncthing: 1.2.2 -> 1.3.0 #71037

Merged

10 tasks

c00w approved these changes Oct 19, 2019

View reviewed changes

marsam force-pushed the buildGoModule-disable-goproxy branch from 69ae8a9 to 589d4ff Compare October 20, 2019 01:21

marsam merged commit e18646b into NixOS:master Oct 20, 2019

marsam deleted the buildGoModule-disable-goproxy branch October 20, 2019 01:39

xrelkd mentioned this pull request Oct 26, 2019

eksctl: 0.6.0 -> 0.7.0 #71014

Merged

10 tasks

ento added a commit to ento/nur-packages that referenced this pull request Oct 29, 2019

abs: update sha of modules

4eee896

ref: NixOS/nixpkgs#71049

thefloweringash mentioned this pull request Oct 29, 2019

package.nix support for Go 1.13 thefloweringash/hass_ir_adapter#4

Merged

minijackson added a commit to minijackson/nixpkgs that referenced this pull request Nov 8, 2019

shiori: update modSha256, see NixOS#71049

4b496a3

minijackson mentioned this pull request Nov 8, 2019

shiori: update modSha256, update test to python, do more tests #73063

Merged

10 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

buildGoModule: disable default go module proxy #71049

buildGoModule: disable default go module proxy #71049

marsam commented Oct 12, 2019

kalbasit Oct 12, 2019 •

edited

marsam Oct 12, 2019

kalbasit Oct 12, 2019

c00w Oct 19, 2019

c00w commented Oct 12, 2019 via email

marsam commented Oct 12, 2019 •

edited

c00w commented Oct 13, 2019

marsam commented Oct 14, 2019

kalbasit commented Oct 15, 2019

kalbasit commented Oct 15, 2019

c00w commented Oct 15, 2019 via email

marsam commented Oct 16, 2019

c00w commented Oct 16, 2019 via email

nixos-discourse commented Oct 18, 2019

c00w commented Oct 19, 2019

marsam commented Oct 20, 2019

buildGoModule: disable default go module proxy #71049

buildGoModule: disable default go module proxy #71049

Conversation

marsam commented Oct 12, 2019

Motivation for this change

Things done

Notify maintainers

kalbasit Oct 12, 2019 • edited

Choose a reason for hiding this comment

marsam Oct 12, 2019

Choose a reason for hiding this comment

kalbasit Oct 12, 2019

Choose a reason for hiding this comment

c00w Oct 19, 2019

Choose a reason for hiding this comment

c00w commented Oct 12, 2019 via email

marsam commented Oct 12, 2019 • edited

c00w commented Oct 13, 2019

marsam commented Oct 14, 2019

kalbasit commented Oct 15, 2019

kalbasit commented Oct 15, 2019

c00w commented Oct 15, 2019 via email

marsam commented Oct 16, 2019

c00w commented Oct 16, 2019 via email

nixos-discourse commented Oct 18, 2019

c00w commented Oct 19, 2019

marsam commented Oct 20, 2019

kalbasit Oct 12, 2019 •

edited

marsam commented Oct 12, 2019 •

edited