minikube: switch to build from source & mostly fix on Darwin #23776

copumpkin · 2017-03-12T01:15:03Z

Linux behavior should be largely unchanged but Darwin now mostly works. I was unable to spin it up against the VMWare Fusion driver but I think it'll work with the xhyve one. I'll probably make another couple of changes after I get the xhyve driver working on Darwin. Would appreciate someone giving it a go on Linux.

cc @ebzzry @peterhoeg

Motivation for this change

Source builds are cooler than binaries 😄 also, it's nice when things work on Darwin.

Things done

Tested using sandboxing
(nix.useSandbox on NixOS,
or option build-use-sandbox in nix.conf
on non-NixOS)
Built on platform(s)
- NixOS
- macOS
- Linux
Tested compilation of all pkgs that depend on this change using nix-shell -p nox --run "nox-review wip"
Tested execution of all binary files (usually in ./result/bin/)
Fits CONTRIBUTING.md.

mention-bot · 2017-03-12T01:15:04Z

@copumpkin, thanks for your PR! By analyzing the history of the files in this pull request, we identified @peterhoeg and @ebzzry to be potential reviewers.

peterhoeg · 2017-03-12T01:49:17Z

pkgs/applications/networking/cluster/minikube/default.nix

-             then "0cdcabsx5l4jbpyj3zzyz5bnzks6wl64bmzdsnk41x92ar5y5yal"
-             else "12f3b7s5lwpvzx4wj6i6h62n4zjshqf206fxxwpwx9kpsdaw6xdi";
+  binPath = [ kubernetes ]
+    ++ stdenv.lib.optionals stdenv.isLinux [ kubernetes libvirt qemu docker-machine-kvm ]


kubernetes is duplicated here

peterhoeg · 2017-03-12T01:50:02Z

pkgs/applications/networking/cluster/minikube/default.nix

+  buildFlagsArray = ''
+    -ldflags=
+      -X k8s.io/minikube/pkg/version.version=${version}
+      -X k8s.io/minikube/pkg/version.isoVersion=v1.0.7


Shouldn't we move the isoversion into a variable? It changes with almost every release.

Yup, I forgot to do that, thanks

peterhoeg · 2017-03-12T01:50:45Z

pkgs/applications/networking/cluster/minikube/default.nix

+    # mkdir -p $bin/share/bash-completion/completions/
+    # HOME=$PWD $bin/bin/minikube completion bash > $bin/share/bash-completion/completions/minikube
+  '' + stdenv.lib.optionalString stdenv.isDarwin ''
+    install_name_tool -delete_rpath $out/lib $bin/bin/localkube
  '';


We end up with all the sources under /share. Are those needed? If not, we should blow them away.

That seems like a more general buildGoPackage thing, right? Also, the bin output doesn't have the source.

peterhoeg · 2017-03-12T01:59:19Z

I've added some comments to the code but on linux minikube builds, but doesn't run with these changes:

peter@dolores:~ $ minikube start --vm-driver=kvm --container-runtime=rkt --network-plugin=cni -v4
Starting local Kubernetes cluster...
Starting VM...
Running pre-create checks...
Creating machine...
(minikube) Downloading /home/peter/.minikube/cache/boot2docker.iso from file:///home/peter/.minikube/cache/iso/minikube-v1.0.7.iso...
(minikube) Creating SSH key...
Waiting for machine to be running, this may take a few minutes...
Detecting operating system of created instance...
Waiting for SSH to be available...
Detecting the provisioner...
Provisioning with buildroot...
Copying certs to the local machine directory...
Copying certs to the remote machine...
Setting Docker configuration on the remote daemon...
Checking connection to Docker...
Docker is up and running!
SSH-ing files into VM...
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x20 pc=0x97b58a]

goroutine 70 [running]:
panic(0x199a500, 0xc420018160)
/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-go-1.7.4/share/go/src/runtime/panic.go:500 +0x1a1
k8s.io/minikube/pkg/minikube/assets.(*MemoryAsset).Read(0xc4202a89a0, 0xc4207ca000, 0x8000, 0x8000, 0x8000, 0x8000, 0xc42076c370)
/tmp/nix-build-minikube-0.17.1.drv-0/go/src/k8s.io/minikube/pkg/minikube/assets/vm_assets.go:136 +0x2a
io.copyBuffer(0x7f6e685776b0, 0xc4201c2480, 0x7f6e684fadf8, 0xc4202a89a0, 0xc4207ca000, 0x8000, 0x8000, 0x199f640, 0x0, 0x7f6e685776b0)
/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-go-1.7.4/share/go/src/io/io.go:390 +0x147
io.Copy(0x7f6e685776b0, 0xc4201c2480, 0x7f6e684fadf8, 0xc4202a89a0, 0xc4201c2480, 0x12, 0x0)
/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-go-1.7.4/share/go/src/io/io.go:360 +0x68
k8s.io/minikube/pkg/minikube/sshutil.Transfer.func1(0xc4206d9590, 0x2877d40, 0xc4201c2480, 0x1b6d2b8, 0x4, 0x0, 0x1b73e52, 0x9, 0x7f6e684fadf8, 0xc4202a89a0)
/tmp/nix-build-minikube-0.17.1.drv-0/go/src/k8s.io/minikube/pkg/minikube/sshutil/sshutil.go:120 +0x310
created by k8s.io/minikube/pkg/minikube/sshutil.Transfer
/tmp/nix-build-minikube-0.17.1.drv-0/go/src/k8s.io/minikube/pkg/minikube/sshutil/sshutil.go:122 +0x4f9

copumpkin · 2017-03-12T02:34:30Z

Thanks for the quick review! I'll check on Linux and fix the things you mention 😄

copumpkin · 2017-03-12T04:14:48Z

Ah, the nil pointer thing is actually tricky. I'm poking at it now.

copumpkin · 2017-03-12T06:57:39Z

I pushed an update but I still don't think it's quite working. The trickiness arises because localkube needs to be a static linux cgo binary and we can't build those nicely right now, especially not on Darwin. So I'm telling it to get localkube automatically from upstream instead. It's not ideal but it should behave the same way. Unfortunately when I run minikube start, it starts the VM just fine, copies localkube to it, but then status shows me that localkube is stopped. I need to look into what's wrong there, but I'm going to bed now 😄

LnL7 · 2017-03-12T23:40:24Z

I'm seeing this, there was a similar issue with fzf before.

github.com/dhiltgen/docker-machine-kvm/cmd/docker-machine-driver-kvm
installing
/private/var/folders/qv/5k4lhv3n1hg2jk29cj3smtp00000gn/T/nix-build-docker-machine-kvm-0.8.2.drv-0/go /private/var/folders/qv/5k4lhv3n1hg2jk29cj3smtp00000gn/T/nix-build-docker-machine-kvm-0.8.2.drv-0
/private/var/folders/qv/5k4lhv3n1hg2jk29cj3smtp00000gn/T/nix-build-docker-machine-kvm-0.8.2.drv-0
post-installation fixup
stripping (with flags -S) in /nix/store/csg7gahmydn6vmcaprsq5x01hgac8igv-docker-machine-kvm-0.8.2-bin/bin 
patching script interpreter paths in /nix/store/csg7gahmydn6vmcaprsq5x01hgac8igv-docker-machine-kvm-0.8.2-bin
patching script interpreter paths in /nix/store/sgdjl8ch9akjsz1wmdj7iz25hg9qa469-docker-machine-kvm-0.8.2
cycle detected in the references of ‘/nix/store/csg7gahmydn6vmcaprsq5x01hgac8igv-docker-machine-kvm-0.8.2-bin’

copumpkin · 2017-03-12T23:41:27Z

Oh, I can fix that separately. I think I almost have it working on both darwin and linux now, but localkube is proving to be quite tricky 😄

copumpkin · 2017-03-13T02:13:26Z

@peterhoeg okay I think this is ready. It works fine on Darwin, but I don't have a linux host I can virtualize on to test on. Can you (or anyone else) try it out?

Linux behavior should be largely unchanged but we now build minikube ourselves. Unfortunately localkube is still tricky to build so I pull in a binary version from upstream.

copumpkin · 2017-03-13T02:30:11Z

Just enabled the completions (tested with sandbox and disabled the online check), too

peterhoeg · 2017-03-13T04:18:51Z

Yep, works now, thanks @copumpkin! 👍

copumpkin · 2017-03-13T04:20:20Z

Thanks for checking! Someday we'll get a source build of localkube too :)

…

On Mon, Mar 13, 2017 at 00:18 Peter Hoeg ***@***.***> wrote: Yep, works now, thanks @copumpkin <https://github.com/copumpkin>! 👍 — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#23776 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAAKP0SyRa2yExxOp2F4MBWIOdYBGwb_ks5rlMOtgaJpZM4MaZWR> .

copumpkin requested a review from peterhoeg March 12, 2017 01:15

peterhoeg reviewed Mar 12, 2017

View reviewed changes

copumpkin force-pushed the minikube-source branch from ca88d41 to 296d542 Compare March 12, 2017 06:49

LnL7 added the 6.topic: darwin Running or building packages on Darwin label Mar 12, 2017

copumpkin force-pushed the minikube-source branch from 296d542 to 0702714 Compare March 13, 2017 00:01

minikube: switch to build from source & fix on Darwin

fe339d2

Linux behavior should be largely unchanged but we now build minikube ourselves. Unfortunately localkube is still tricky to build so I pull in a binary version from upstream.

copumpkin force-pushed the minikube-source branch from 0702714 to fe339d2 Compare March 13, 2017 02:29

copumpkin merged commit 26fc26a into NixOS:master Mar 13, 2017

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

minikube: switch to build from source & mostly fix on Darwin #23776

minikube: switch to build from source & mostly fix on Darwin #23776

copumpkin commented Mar 12, 2017

mention-bot commented Mar 12, 2017

peterhoeg Mar 12, 2017

peterhoeg Mar 12, 2017

copumpkin Mar 12, 2017

peterhoeg Mar 12, 2017

copumpkin Mar 12, 2017

peterhoeg commented Mar 12, 2017

copumpkin commented Mar 12, 2017

copumpkin commented Mar 12, 2017

copumpkin commented Mar 12, 2017

LnL7 commented Mar 12, 2017

copumpkin commented Mar 12, 2017

copumpkin commented Mar 13, 2017

copumpkin commented Mar 13, 2017

peterhoeg commented Mar 13, 2017

copumpkin commented Mar 13, 2017 via email

minikube: switch to build from source & mostly fix on Darwin #23776

minikube: switch to build from source & mostly fix on Darwin #23776

Conversation

copumpkin commented Mar 12, 2017

Motivation for this change

Things done

mention-bot commented Mar 12, 2017

peterhoeg Mar 12, 2017

Choose a reason for hiding this comment

peterhoeg Mar 12, 2017

Choose a reason for hiding this comment

copumpkin Mar 12, 2017

Choose a reason for hiding this comment

peterhoeg Mar 12, 2017

Choose a reason for hiding this comment

copumpkin Mar 12, 2017

Choose a reason for hiding this comment

peterhoeg commented Mar 12, 2017

copumpkin commented Mar 12, 2017

copumpkin commented Mar 12, 2017

copumpkin commented Mar 12, 2017

LnL7 commented Mar 12, 2017

copumpkin commented Mar 12, 2017

copumpkin commented Mar 13, 2017

copumpkin commented Mar 13, 2017

peterhoeg commented Mar 13, 2017

copumpkin commented Mar 13, 2017 via email