Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

go_1_11: init at 1.11 and set as default #45630

Merged
merged 3 commits into from Aug 27, 2018
Merged

Conversation

adisbladis
Copy link
Member

@adisbladis adisbladis commented Aug 25, 2018

Motivation for this change
Things done
  • Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS)
  • Built on platform(s)
    • NixOS
    • macOS
    • other Linux distributions
  • Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
  • Tested compilation of all pkgs that depend on this change using nix-shell -p nox --run "nox-review wip"
  • Tested execution of all binary files (usually in ./result/bin/)
  • Determined the impact on package closure size (by running nix path-info -S before and after)
  • Fits CONTRIBUTING.md.

note: I want to set this as default but need to run nox-review first.

cc @cstrahan @orivej @velovix @Mic92

I recently started https://github.com/adisbladis/vgo2nix to use the new modules support with Nix.

@GrahamcOfBorg
Copy link

Failure on x86_64-darwin (full log)

Attempted: go_1_11

Partial log (click to expand)

ok      cmd/vendor/golang.org/x/arch/arm64/arm64asm     0.072s
ok      cmd/vendor/golang.org/x/arch/ppc64/ppc64asm     0.006s
ok      cmd/vendor/golang.org/x/arch/x86/x86asm 0.059s
ok      cmd/vendor/golang.org/x/crypto/ssh/terminal     0.008s
ok      cmd/vendor/golang.org/x/sys/unix        0.368s
ok      cmd/vet 3.320s
ok      cmd/vet/internal/cfg    0.007s
2018/08/25 17:19:53 Failed: exit status 1
builder for '/nix/store/qqwwvjzbp20lv7k3i1nq3gyiv129kfyr-go-1.11.drv' failed with exit code 1
error: build of '/nix/store/qqwwvjzbp20lv7k3i1nq3gyiv129kfyr-go-1.11.drv' failed

@GrahamcOfBorg
Copy link

Success on aarch64-linux (full log)

Attempted: go_1_11

Partial log (click to expand)

wrong ELF type
wrong ELF type
wrong ELF type
wrong ELF type
wrong ELF type
wrong ELF type
cannot find section .dynamic
cannot find section .dynamic
cannot find section .dynamic
/nix/store/yad18r72l0f6jjjh4bk732jiya3grv99-go-1.11

@GrahamcOfBorg
Copy link

Success on x86_64-linux (full log)

Attempted: go_1_11

Partial log (click to expand)

cannot find section .dynamic
cannot find section .dynamic
cannot find section .dynamic
cannot find section .dynamic
cannot find section .dynamic
cannot find section .dynamic
cannot find section .dynamic
cannot find section .dynamic
cannot find section .dynamic
/nix/store/wmhb04kda7jqzh4jgn94fva8v8iksci6-go-1.11

@CMCDragonkai
Copy link
Member

Does buildGoPackage need to change to support the new go modules because it doesn't use GOPATH?

@CMCDragonkai
Copy link
Member

CMCDragonkai commented Aug 26, 2018

I noticed that buildGoPackage with go overridden to this go version and when running it in a nix-shell to try and build something, it replies with:

go: cannot use modules with build cache disabled

This is because the $GOCACHE is off inside the nix-shell.

Furthermore this is the $GOPATH:

/nix/store/wmhb04kda7jqzh4jgn94fva8v8iksci6-go-1.11/share/go:/nix/store/wmhb04kda7jqzh4jgn94fva8v8iksci6-go-1.11/share/go:/nix/store/wmhb04kda7jqzh4jgn94fva8v8iksci6-go-1.11/share/go:/nix/store/wmhb04kda7jqzh4jgn94fva8v8iksci6-go-1.11/share/go:/nix/store/wmhb04kda7jqzh4jgn94fva8v8iksci6-go-1.11/share/go:/nix/store/wmhb04kda7jqzh4jgn94fva8v8iksci6-go-1.11/share/go

I'm not sure why it's repeating the same path over and over again.

I think this means buildGoPackage for Go modules might have work differently.

To test:

{
  pkgs ? import (fetchTarball https://github.com/NixOS/nixpkgs/archive/86c4c0699aad5331da6950dfa0727d19ddc3ce09.tar.gz) {},
  goPath ? "go_1_11"
}:
  with pkgs;
  let
    go = lib.getAttrFromPath (lib.splitString "." goPath) pkgs;
    buildGoPackage = pkgs.buildGoPackage.override { go = go; };
  in
    buildGoPackage { ... };

@adisbladis
Copy link
Member Author

adisbladis commented Aug 26, 2018

Does buildGoPackage need to change to support the new go modules because it doesn't use GOPATH?

@CMCDragonkai buildGoPackage can be oblivious to go modules. The way vgo2nix generate nix expressions is exactly the same as go2nix and dep2nix.

@CMCDragonkai
Copy link
Member

@adisbladis I discovered this after experimenting with it. The problem is that I wish to shared dependencies between what is loaded by Nix and what can be used by the Go 1.11 compiler when in module mode. That way I'm not downloading the same dependency twice, once during nix-build, and another time during nix-shell and then using go build... etc.

FYI: This sharing of deps between building and development is achieved in the Python Nix.

@adisbladis adisbladis changed the title go_1_11: init at 1.11 go_1_11: init at 1.11 and set as default Aug 26, 2018
@adisbladis
Copy link
Member Author

adisbladis commented Aug 26, 2018

I have set go 1.11 as default, both in the go and buildGoPackage attributes.
nox-review passes.

@adisbladis
Copy link
Member Author

@GrahamcOfBorg build go_1_11

@GrahamcOfBorg
Copy link

Failure on x86_64-darwin (full log)

Attempted: go_1_11

Partial log (click to expand)

applying patch /nix/store/b3ra2sifmzbybchyxrf81fh7bx5m612g-skip-external-network-tests.patch
patching file src/cmd/go/go_test.go
Hunk #1 succeeded at 4948 (offset 2 lines).
patching file src/net/dial_test.go
applying patch /nix/store/hm2jzy93m5sl48alp2dhklz2b592dn9b-skip-nohup-tests.patch
patching file src/os/signal/signal_test.go
Hunk #2 FAILED at 308.
1 out of 2 hunks FAILED -- saving rejects to file src/os/signal/signal_test.go.rej
builder for '/nix/store/k2rh67qmi14cznayf3ay8n1l4ln54wq4-go-1.11.drv' failed with exit code 1
error: build of '/nix/store/k2rh67qmi14cznayf3ay8n1l4ln54wq4-go-1.11.drv' failed

@adisbladis
Copy link
Member Author

Hopefully the last darwin issues are now sorted.

@GrahamcOfBorg build go_1_11

@GrahamcOfBorg
Copy link

Success on x86_64-linux (full log)

Attempted: go_1_11

Partial log (click to expand)

cannot find section .dynamic
cannot find section .dynamic
cannot find section .dynamic
cannot find section .dynamic
cannot find section .dynamic
cannot find section .dynamic
cannot find section .dynamic
cannot find section .dynamic
cannot find section .dynamic
/nix/store/x6xaljnw9ldhzvhy0j35ai4r2n090p3k-go-1.11

@GrahamcOfBorg
Copy link

Success on x86_64-linux (full log)

Attempted: go_1_11

The following builds were skipped because they don't evaluate on x86_64-linux: goa, gopherclient

Partial log (click to expand)

a) For `nixos-rebuild` you can set
  { nixpkgs.config.allowBroken = true; }
in configuration.nix to override this.

b) For `nix-env`, `nix-build`, `nix-shell` or any other Nix command you can add
  { allowBroken = true; }
to ~/.config/nixpkgs/config.nix.


/nix/store/x6xaljnw9ldhzvhy0j35ai4r2n090p3k-go-1.11

@GrahamcOfBorg
Copy link

Success on aarch64-linux (full log)

Attempted: go_1_11

Partial log (click to expand)

wrong ELF type
wrong ELF type
wrong ELF type
wrong ELF type
wrong ELF type
wrong ELF type
cannot find section .dynamic
cannot find section .dynamic
cannot find section .dynamic
/nix/store/d8cxb1x1igcx5j16d75gdp9q04xfhl4b-go-1.11

@GrahamcOfBorg
Copy link

Failure on x86_64-darwin (full log)

Attempted: go_1_11

The following builds were skipped because they don't evaluate on x86_64-darwin: goa, gopherclient

Partial log (click to expand)

applying patch /nix/store/b3ra2sifmzbybchyxrf81fh7bx5m612g-skip-external-network-tests.patch
patching file src/cmd/go/go_test.go
Hunk #1 succeeded at 4948 (offset 2 lines).
patching file src/net/dial_test.go
applying patch /nix/store/hm2jzy93m5sl48alp2dhklz2b592dn9b-skip-nohup-tests.patch
patching file src/os/signal/signal_test.go
Hunk #2 FAILED at 308.
1 out of 2 hunks FAILED -- saving rejects to file src/os/signal/signal_test.go.rej
builder for '/nix/store/k2rh67qmi14cznayf3ay8n1l4ln54wq4-go-1.11.drv' failed with exit code 1
error: build of '/nix/store/k2rh67qmi14cznayf3ay8n1l4ln54wq4-go-1.11.drv' failed

@GrahamcOfBorg
Copy link

Success on aarch64-linux (full log)

Attempted: go_1_11

Partial log (click to expand)

these derivations will be built:
  /nix/store/fklwp8dbfcqfwllp2x4lzifpf5a3wlam-go-1.11.drv
waiting for locks or build slots...
/nix/store/d8cxb1x1igcx5j16d75gdp9q04xfhl4b-go-1.11

@GrahamcOfBorg
Copy link

Failure on x86_64-darwin (full log)

Attempted: go_1_11

Partial log (click to expand)

ok      cmd/vendor/golang.org/x/arch/arm64/arm64asm     0.074s
ok      cmd/vendor/golang.org/x/arch/ppc64/ppc64asm     0.008s
ok      cmd/vendor/golang.org/x/arch/x86/x86asm 0.072s
ok      cmd/vendor/golang.org/x/crypto/ssh/terminal     0.008s
ok      cmd/vendor/golang.org/x/sys/unix        0.366s
ok      cmd/vet 3.077s
ok      cmd/vet/internal/cfg    0.015s
2018/08/26 15:58:14 Failed: exit status 1
builder for '/nix/store/1fs12aapjp8fpzb8znjq5kzna1qb00ld-go-1.11.drv' failed with exit code 1
error: build of '/nix/store/1fs12aapjp8fpzb8znjq5kzna1qb00ld-go-1.11.drv' failed

@GrahamcOfBorg
Copy link

Success on x86_64-linux (full log)

Attempted: go_1_11

Partial log (click to expand)

cannot find section .dynamic
cannot find section .dynamic
cannot find section .dynamic
cannot find section .dynamic
cannot find section .dynamic
cannot find section .dynamic
cannot find section .dynamic
cannot find section .dynamic
cannot find section .dynamic
/nix/store/x6xaljnw9ldhzvhy0j35ai4r2n090p3k-go-1.11

@GrahamcOfBorg
Copy link

Failure on x86_64-darwin (full log)

Attempted: go_1_11

The following builds were skipped because they don't evaluate on x86_64-darwin: goa, gopherclient

Partial log (click to expand)

ok      cmd/vendor/golang.org/x/arch/arm64/arm64asm     0.070s
ok      cmd/vendor/golang.org/x/arch/ppc64/ppc64asm     0.012s
ok      cmd/vendor/golang.org/x/arch/x86/x86asm 0.061s
ok      cmd/vendor/golang.org/x/crypto/ssh/terminal     0.037s
ok      cmd/vendor/golang.org/x/sys/unix        0.428s
ok      cmd/vet 3.562s
ok      cmd/vet/internal/cfg    0.011s
2018/08/26 16:11:20 Failed: exit status 1
builder for '/nix/store/1fs12aapjp8fpzb8znjq5kzna1qb00ld-go-1.11.drv' failed with exit code 1
error: build of '/nix/store/1fs12aapjp8fpzb8znjq5kzna1qb00ld-go-1.11.drv' failed

@GrahamcOfBorg
Copy link

Success on x86_64-linux (full log)

Attempted: go_1_11

The following builds were skipped because they don't evaluate on x86_64-linux: goa, gopherclient

Partial log (click to expand)

a) For `nixos-rebuild` you can set
  { nixpkgs.config.allowBroken = true; }
in configuration.nix to override this.

b) For `nix-env`, `nix-build`, `nix-shell` or any other Nix command you can add
  { allowBroken = true; }
to ~/.config/nixpkgs/config.nix.


/nix/store/x6xaljnw9ldhzvhy0j35ai4r2n090p3k-go-1.11

@GrahamcOfBorg
Copy link

Success on aarch64-linux (full log)

Attempted: go_1_11

The following builds were skipped because they don't evaluate on aarch64-linux: goa, gopherclient

Partial log (click to expand)

a) For `nixos-rebuild` you can set
  { nixpkgs.config.allowBroken = true; }
in configuration.nix to override this.

b) For `nix-env`, `nix-build`, `nix-shell` or any other Nix command you can add
  { allowBroken = true; }
to ~/.config/nixpkgs/config.nix.


/nix/store/d8cxb1x1igcx5j16d75gdp9q04xfhl4b-go-1.11

@GrahamcOfBorg
Copy link

Success on x86_64-darwin (full log)

Attempted: go_1_11

The following builds were skipped because they don't evaluate on x86_64-darwin: goa, gopherclient

Partial log (click to expand)


ALL TESTS PASSED
---
Installed Go for darwin/amd64 in /nix/store/5ldnryj7nf9ai54rz9als5srp0zz9c6j-go-1.11/share/go
Installed commands in /nix/store/5ldnryj7nf9ai54rz9als5srp0zz9c6j-go-1.11/share/go/bin
post-installation fixup
strip is /nix/store/vvbmqsh5ikrxkiwj71dwfbzcwf4bfav8-cctools-binutils-darwin/bin/strip
stripping (with command strip and flags -S) in /nix/store/5ldnryj7nf9ai54rz9als5srp0zz9c6j-go-1.11/bin
patching script interpreter paths in /nix/store/5ldnryj7nf9ai54rz9als5srp0zz9c6j-go-1.11
/nix/store/5ldnryj7nf9ai54rz9als5srp0zz9c6j-go-1.11

@adisbladis
Copy link
Member Author

Darwin issues are resolved. I think this PR should be good to go now.

cc @cstrahan @orivej @velovix @Mic92

Copy link
Member

@Mic92 Mic92 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ping maintainers of broken packages.

@@ -21,5 +21,6 @@ buildGoPackage rec {
description = "A framework for building microservices in Go using a unique design-first approach";
license = licenses.mit;
maintainers = [ maintainers.rushmorem ];
broken = true;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Member

@Mic92 Mic92 Aug 26, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Goa seemed to build right now, is go110 just required here? https://hydra.nixos.org/build/80201530

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

goa does indeed build with buildGo110Package. I recently had build issues with goa (#43328 (comment)) so I mistakenly assumed it was still broken.

Fixed and squashed.

@@ -40,5 +40,6 @@ buildGoPackage rec {
license = licenses.mit;
maintainers = with maintainers; [ orivej ];
platforms = platforms.linux;
broken = true;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

cc @orivej

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'll fix it.

@Mic92
Copy link
Member

Mic92 commented Aug 26, 2018

@GrahamcOfBorg build docker kubernetes

@GrahamcOfBorg
Copy link

Success on x86_64-linux (full log)

Attempted: docker, kubernetes

Partial log (click to expand)

shrinking /nix/store/2p9fxs80k431vgfb98461snlf9nlxh2y-docker-18.06.1-ce/libexec/docker/dockerd
shrinking /nix/store/2p9fxs80k431vgfb98461snlf9nlxh2y-docker-18.06.1-ce/libexec/docker/docker
patching script interpreter paths in /nix/store/2p9fxs80k431vgfb98461snlf9nlxh2y-docker-18.06.1-ce
checking for references to /build in /nix/store/2p9fxs80k431vgfb98461snlf9nlxh2y-docker-18.06.1-ce...
shrinking RPATHs of ELF executables and libraries in /nix/store/cvpcb43waakdb7w79mkngraagsybdsfb-docker-18.06.1-ce-man
gzipping man pages under /nix/store/cvpcb43waakdb7w79mkngraagsybdsfb-docker-18.06.1-ce-man/share/man/
patching script interpreter paths in /nix/store/cvpcb43waakdb7w79mkngraagsybdsfb-docker-18.06.1-ce-man
checking for references to /build in /nix/store/cvpcb43waakdb7w79mkngraagsybdsfb-docker-18.06.1-ce-man...
/nix/store/2p9fxs80k431vgfb98461snlf9nlxh2y-docker-18.06.1-ce
/nix/store/aji0748l754ippz1nms1f2k4mlnsp5r5-kubernetes-1.10.5

@GrahamcOfBorg
Copy link

Success on aarch64-linux (full log)

Attempted: docker, kubernetes

Partial log (click to expand)

patching script interpreter paths in /nix/store/2i4hik7ipaswmj20w8k7zak1dr27hf0x-kubernetes-1.10.5-man
checking for references to /build in /nix/store/2i4hik7ipaswmj20w8k7zak1dr27hf0x-kubernetes-1.10.5-man...
shrinking RPATHs of ELF executables and libraries in /nix/store/vy3s0zrk8cqvqvh343snzij0p51dmnq1-kubernetes-1.10.5-pause
shrinking /nix/store/vy3s0zrk8cqvqvh343snzij0p51dmnq1-kubernetes-1.10.5-pause/bin/pause
strip is /nix/store/y4ymnvgxygpq05h03kyzbj572zmh6zla-binutils-2.30/bin/strip
stripping (with command strip and flags -S) in /nix/store/vy3s0zrk8cqvqvh343snzij0p51dmnq1-kubernetes-1.10.5-pause/bin
patching script interpreter paths in /nix/store/vy3s0zrk8cqvqvh343snzij0p51dmnq1-kubernetes-1.10.5-pause
checking for references to /build in /nix/store/vy3s0zrk8cqvqvh343snzij0p51dmnq1-kubernetes-1.10.5-pause...
/nix/store/mipfcs3kfnj9vw8zwwb46qsc7nlklb41-docker-18.06.1-ce
/nix/store/2hc0gxwlp0qsf6qjar95l8v7f5jd7s7k-kubernetes-1.10.5

@GrahamcOfBorg
Copy link

Success on x86_64-darwin (full log)

Attempted: docker, kubernetes

Partial log (click to expand)

2018/08/26 17:45:33 WARN: man/src/trust.md does not exist, skipping
2018/08/26 17:45:33 WARN: man/src/volume/prune.md does not exist, skipping
2018/08/26 17:45:33 WARN: man/src/volume/rm.md does not exist, skipping
Generate legacy manpages
post-installation fixup
patching script interpreter paths in /nix/store/39wkfwyj4j3iyg627qnnzj2hn73wcxnf-docker-18.06.1-ce
gzipping man pages under /nix/store/vgc39hdz1bzflmz2zqgfhnzc0spqcrjj-docker-18.06.1-ce-man/share/man/
patching script interpreter paths in /nix/store/vgc39hdz1bzflmz2zqgfhnzc0spqcrjj-docker-18.06.1-ce-man
/nix/store/39wkfwyj4j3iyg627qnnzj2hn73wcxnf-docker-18.06.1-ce
/nix/store/g8kax50im9k9fz8c37xx1w7n5l3vqvci-kubernetes-1.10.5

@adisbladis
Copy link
Member Author

@GrahamcOfBorg test docker

@GrahamcOfBorg
Copy link

Success on aarch64-linux

Attempted: tests.docker

No partial log is available.

@GrahamcOfBorg
Copy link

Success on aarch64-linux (full log)

Attempted: go_1_11

The following builds were skipped because they don't evaluate on aarch64-linux: gopherclient

Partial log (click to expand)

a) For `nixos-rebuild` you can set
  { nixpkgs.config.allowBroken = true; }
in configuration.nix to override this.

b) For `nix-env`, `nix-build`, `nix-shell` or any other Nix command you can add
  { allowBroken = true; }
to ~/.config/nixpkgs/config.nix.


/nix/store/d8cxb1x1igcx5j16d75gdp9q04xfhl4b-go-1.11

@GrahamcOfBorg
Copy link

Success on x86_64-darwin (full log)

Attempted: go_1_11

The following builds were skipped because they don't evaluate on x86_64-darwin: gopherclient

Partial log (click to expand)

a) For `nixos-rebuild` you can set
  { nixpkgs.config.allowBroken = true; }
in configuration.nix to override this.

b) For `nix-env`, `nix-build`, `nix-shell` or any other Nix command you can add
  { allowBroken = true; }
to ~/.config/nixpkgs/config.nix.


/nix/store/5ldnryj7nf9ai54rz9als5srp0zz9c6j-go-1.11

@GrahamcOfBorg
Copy link

Success on x86_64-linux (full log)

Attempted: go_1_11

The following builds were skipped because they don't evaluate on x86_64-linux: gopherclient

Partial log (click to expand)

cannot find section .dynamic
cannot find section .dynamic
cannot find section .dynamic
cannot find section .dynamic
cannot find section .dynamic
cannot find section .dynamic
cannot find section .dynamic
cannot find section .dynamic
cannot find section .dynamic
/nix/store/x6xaljnw9ldhzvhy0j35ai4r2n090p3k-go-1.11

@GrahamcOfBorg
Copy link

Success on x86_64-linux (full log)

Attempted: tests.docker

Partial log (click to expand)

docker: exit status 1
syncing
docker: running command: sync
docker: exit status 0
test script finished in 40.61s
cleaning up
killing docker (pid 597)
vde_switch: EOF on stdin, cleaning up and exiting
vde_switch: Could not remove ctl dir '/build/vde1.ctl': Directory not empty
/nix/store/k22vlvkw0hyacnzin0gbqyimadwj8dah-vm-test-run-docker

@adisbladis adisbladis merged commit faf2f0b into NixOS:master Aug 27, 2018
@adisbladis adisbladis deleted the go-1_11 branch August 28, 2018 03:30
sha256 = "1k18d6rkijlgzn1zw4wphzcv6a6w9hb1msgrsh1102jb18644f2q";
};

GOCACHE = "off";
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure this is working. The derivation is 1.1GB on my machine, and on closer inspection it seems to be mostly this directory:

$ du -hcs /nix/store/vk289i3s3ki84681q8pallmgcmsjg708-go-1.11/share/go/pkg/obj/go-build
673M	/nix/store/vk289i3s3ki84681q8pallmgcmsjg708-go-1.11/share/go/pkg/obj/go-build
673M	total

and there is a README:

$ cat /nix/store/vk289i3s3ki84681q8pallmgcmsjg708-go-1.11/share/go/pkg/obj/go-build/README
This directory holds cached build artifacts from the Go build system.
Run "go clean -cache" if the directory is getting too large.
See golang.org to learn more about Go.

Do we want that directory as part of our go derivation?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this was solved here: #47260

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

6 participants