WIP: Add frakti to run kubernetes pods in KVM #21966

moretea · 2017-01-18T10:51:33Z

Motivation for this change

Frakti uses hyperd to start the different containers that live in one pod in a KVM vm.
This offers better isolation.

Things done

Tested using sandboxing
(nix.useSandbox on NixOS,
or option build-use-sandbox in nix.conf
on non-NixOS)
Built on platform(s)
- NixOS
- macOS
- Linux
Tested compilation of all pkgs that depend on this change using nix-shell -p nox --run "nox-review wip"
Tested execution of all binary files (usually in ./result/bin/)
Fits CONTRIBUTING.md.

mention-bot · 2017-01-18T10:51:34Z

@moretea, thanks for your PR! By analyzing the history of the files in this pull request, we identified @offlinehacker, @nckx and @domenkozar to be potential reviewers.

moretea · 2017-01-18T10:54:12Z

At this moment, I'm not able to build hyperd, and have no clue how to fix this.
It tries to build (part of?) docker and fails with the following error message:

...
github.com/hyperhq/hyperd/vendor/github.com/docker/docker/daemon/graphdriver/btrfs
# github.com/hyperhq/hyperd/vendor/github.com/docker/docker/daemon/graphdriver/btrfs
go/src/github.com/hyperhq/hyperd/vendor/github.com/docker/docker/daemon/graphdriver/btrfs/btrfs.go:164: args.name undefined (type C.struct_btrfs_ioctl_vol_args_v2 has no field or method name)
github.com/hyperhq/hyperd/vendor/github.com/docker/docker/daemon/graphdriver/btrfs
# github.com/hyperhq/hyperd/vendor/github.com/docker/docker/daemon/graphdriver/btrfs
go/src/github.com/hyperhq/hyperd/vendor/github.com/docker/docker/daemon/graphdriver/btrfs/btrfs.go:164: args.name undefined (type C.struct_btrfs_ioctl_vol_args_v2 has no field or method name)
github.com/hyperhq/hyperd/vendor/github.com/docker/docker/daemon/graphdriver/btrfs
# github.com/hyperhq/hyperd/vendor/github.com/docker/docker/daemon/graphdriver/btrfs
go/src/github.com/hyperhq/hyperd/vendor/github.com/docker/docker/daemon/graphdriver/btrfs/btrfs.go:164: args.name undefined (type C.struct_btrfs_ioctl_vol_args_v2 has no field or method name)
github.com/hyperhq/hyperd/vendor/github.com/docker/docker/daemon/graphdriver/btrfs
# github.com/hyperhq/hyperd/vendor/github.com/docker/docker/daemon/graphdriver/btrfs
go/src/github.com/hyperhq/hyperd/vendor/github.com/docker/docker/daemon/graphdriver/btrfs/btrfs.go:164: args.name undefined (type C.struct_btrfs_ioctl_vol_args_v2 has no field or method name)
github.com/hyperhq/hyperd/vendor/github.com/docker/docker/daemon/graphdriver/btrfs
# github.com/hyperhq/hyperd/vendor/github.com/docker/docker/daemon/graphdriver/btrfs
go/src/github.com/hyperhq/hyperd/vendor/github.com/docker/docker/daemon/graphdriver/btrfs/btrfs.go:164: args.name undefined (type C.struct_btrfs_ioctl_vol_args_v2 has no field or method name)
github.com/hyperhq/hyperd/vendor/github.com/docker/docker/daemon/graphdriver/btrfs
# github.com/hyperhq/hyperd/vendor/github.com/docker/docker/daemon/graphdriver/btrfs
go/src/github.com/hyperhq/hyperd/vendor/github.com/docker/docker/daemon/graphdriver/btrfs/btrfs.go:164: args.name undefined (type C.struct_btrfs_ioctl_vol_args_v2 has no field or method name)
github.com/hyperhq/hyperd/vendor/github.com/docker/docker/daemon/graphdriver/btrfs
# github.com/hyperhq/hyperd/vendor/github.com/docker/docker/daemon/graphdriver/btrfs
go/src/github.com/hyperhq/hyperd/vendor/github.com/docker/docker/daemon/graphdriver/btrfs/btrfs.go:164: args.name undefined (type C.struct_btrfs_ioctl_vol_args_v2 has no field or method name)
github.com/hyperhq/hyperd/vendor/github.com/docker/docker/daemon/graphdriver/btrfs
# github.com/hyperhq/hyperd/vendor/github.com/docker/docker/daemon/graphdriver/btrfs
go/src/github.com/hyperhq/hyperd/vendor/github.com/docker/docker/daemon/graphdriver/btrfs/btrfs.go:164: args.name undefined (type C.struct_btrfs_ioctl_vol_args_v2 has no field or method name)
builder for ‘/nix/store/z7sw93xk8azwxaazrq3b8pjjnp31nypm-hyperd-20170117-0195fad.drv’ failed with exit code 11
error: build of ‘/nix/store/z7sw93xk8azwxaazrq3b8pjjnp31nypm-hyperd-20170117-0195fad.drv’ failed

/cc @offlinehacker @tailhook (since you're the maintainers of Docker). Do you know what's going wrong here?

offlinehacker · 2017-01-18T11:09:37Z

Try deleting all the vendor folders of the vendor folders, this is usually an issue.

edude03 · 2017-07-08T02:21:24Z

pkgs/applications/virtualization/hyperd/default.nix

+    leaveDotGit = true;
+    deepClone = true;
+    url = "https://github.com/hyperhq/hyperd.git";
+    sha256 = "0d5fkc3vniyh5nkfrna2bfzaxhni6njq0h03rp2x13vm2ha78dy4";


I got 0sb6h6c7n0q0qskvylp94f4vkp9h6yjrmc2lqm78ri8p033raj5s

edude03 · 2017-07-08T02:22:55Z

nixos/modules/virtualisation/hyperd.nix

+      listenStreams = [ "/run/libvirt/virtlockd-sock" ];
+    };
+
+    systemd.services.virtlockd = {


had to comment this out because it conflicts with libvirtd, maybe libvirtd just needs to be a dependency for this?

The exact error is error: The unique option systemd.services.virtlockd.description' is defined multiple times, in /etc/nixos/nixpkgs/nixos/modules/virtualisation/libvirtd.nix' and /etc/nixos/nixpkgs/nixos/modules/virtualisation/hyperd.nix'.` for reference

This should be /var/run/hyperd.sock

I got the same error.

I guess this redefines a service that already exists

edude03 · 2017-07-08T15:30:28Z

nixos/modules/virtualisation/hyperd.nix

+    Kernel = ${cfg.kernel}
+    Initrd = ${cfg.initrd}
+    Bridge = ${cfg.bridge}
+    BridgeIP =${cfg.bridge_ip} 


I added Hypervisor = libvirt here, but I guess this should be a config option

I got it working with kvm actually.

edude03 · 2017-07-08T15:31:49Z

nixos/modules/virtualisation/hyperd.nix

+      wantedBy = [ "multi-user.target" ];
+      after = [ "systemd-udev-settle.service" ];
+
+      path = [


I added

pkgs.e2fsprogs pkgs.utillinux pkgs.kmod pkgs.iptables

To get this to start

indeed, added those myself as well 👍

edude03 · 2017-07-08T15:33:01Z

nixos/modules/virtualisation/hyperd.nix

+
+      serviceConfig = {
+        ExecStart = ''
+          @${pkgs.hyperd}/bin/hyperd \


I had to remove the @ symbol here, not sure what it does but with it in place, I'd get an error saying logtostrerr is not in $PATH, removing that option would cause -v=0 not in path etc

edude03 · 2017-07-08T15:34:22Z

In the end I've yet to get it to work, hyperctl is unable to connect to the daemon, I'm still trying to figure that out

0xABAB · 2017-07-08T19:42:31Z

pkgs/applications/virtualization/hyperd/default.nix

+    mkdir -p src/${goPackagePath}
+    cp -r ${src}/* src/${goPackagePath}
+    cp -r ${src}/.git src/${goPackagePath}
+    chmod -R +w *


0xABAB · 2017-07-08T19:44:11Z

pkgs/applications/virtualization/hyperd/default.nix

+    mv src/${goPackagePath}/vendor/github.com/* src/github.com/
+    rmdir src/${goPackagePath}/vendor/github.com
+    mv src/${goPackagePath}/vendor/* src/.
+    cd src/${goPackagePath}


Please don't call cd outside a subshell.

0xABAB · 2017-07-08T19:45:26Z

pkgs/applications/virtualization/hyperd/default.nix

+  '';
+
+  preFixup = ''
+    # remove references to go compiler, gcc and glibc


I guess we have a remove-references feature for this (should be easy to find). If it is not applicable, please elaborate in the code.

0xABAB · 2017-07-08T19:46:11Z

pkgs/applications/virtualization/hyperd/default.nix

+
+  installPhase = ''
+    mkdir -p $out/bin
+    cp ./hyperd $out/bin


Use {} syntax to combine both lines into one.

0xABAB

Going in the right direction, and I like that there are tests from the beginning.

moretea · 2017-07-10T09:30:57Z

I have made some more progress locally as well. Will merge these charges and push an update today.

edude03 · 2017-07-11T15:14:04Z

@moretea I'm super excited to see the updated PR :D

edude03 · 2017-08-15T17:14:12Z

@moretea are you still cranking along on this?

offlinehacker · 2017-09-23T18:52:03Z

Have you tried this with #25426

bachp · 2017-09-23T19:07:24Z

@moretea I have a nix derivation for runv and hyperstart that I'm trying to get workingas an alterantive docker runtime. I think especially hyperstart might be useful to provide a default kernel and initrd. If you are interested I will push my WIP branch next time I'm at my dev machine.

moretea · 2017-09-23T19:11:03Z

Cool. I haven't touched this in a while. Looking forward to what you'd come up with. Op 23 sep. 2017 9:07 p.m. schreef "Pascal Bach" <notifications@github.com>:

…

@moretea <https://github.com/moretea> I have a nix derivation for runv <https://github.com/hyperhq/runv> and hyperstart <https://github.com/hyperhq/hyperstart> that I'm trying to get workingas an alterantive docker runtime. I think especially hyperstart might be useful to provide a default kernel and initrd. If you are interested I will push my WIP branch next time I'm at my dev machine. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#21966 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAGGlB_VCaIyA6n-ngMtaaXGinDSFaHCks5slVbwgaJpZM4Lmu2_> .

bachp · 2017-09-23T19:35:25Z

@moretea The current WIP branch is here: https://github.com/bachp/nixpkgs/tree/docker-runv

mmahut · 2019-08-01T17:50:45Z

Any update on this pull request?

mmahut · 2019-08-19T11:38:07Z

Closing due to lack of activity, feel free to reopen this if needed.

moretea added 3 commits January 18, 2017 11:46

hyperd: init (TODO: determine which version)

3484f09

frakti: init (TODO: determine version)

c418572

WIP: Add frakti to k8s

8da6a71

moretea added 2 commits January 18, 2017 16:08

build hyper correctly

962ad88

some work on a hyperd service module

d0a552b

pSub added the 2.status: work-in-progress label May 5, 2017

edude03 reviewed Jul 8, 2017

View reviewed changes

0xABAB reviewed Jul 8, 2017

View reviewed changes

0xABAB suggested changes Jul 8, 2017

View reviewed changes

mmahut closed this Aug 19, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

WIP: Add frakti to run kubernetes pods in KVM #21966

WIP: Add frakti to run kubernetes pods in KVM #21966

moretea commented Jan 18, 2017 •

edited

mention-bot commented Jan 18, 2017

moretea commented Jan 18, 2017

offlinehacker commented Jan 18, 2017

edude03 Jul 8, 2017

edude03 Jul 8, 2017

moretea Jul 10, 2017

edude03 Jul 11, 2017

edude03 Jul 11, 2017

edude03 Jul 8, 2017

moretea Jul 10, 2017

edude03 Jul 8, 2017

moretea Jul 10, 2017

edude03 Jul 8, 2017

edude03 commented Jul 8, 2017

0xABAB Jul 8, 2017

0xABAB Jul 8, 2017

0xABAB Jul 8, 2017

0xABAB Jul 8, 2017

0xABAB left a comment

moretea commented Jul 10, 2017 via email •

edited

edude03 commented Jul 11, 2017

edude03 commented Aug 15, 2017

offlinehacker commented Sep 23, 2017

bachp commented Sep 23, 2017

moretea commented Sep 23, 2017 via email

bachp commented Sep 23, 2017

mmahut commented Aug 1, 2019

mmahut commented Aug 19, 2019

WIP: Add frakti to run kubernetes pods in KVM #21966

WIP: Add frakti to run kubernetes pods in KVM #21966

Conversation

moretea commented Jan 18, 2017 • edited

Motivation for this change

Things done

mention-bot commented Jan 18, 2017

moretea commented Jan 18, 2017

offlinehacker commented Jan 18, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

edude03 commented Jul 8, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

0xABAB left a comment

Choose a reason for hiding this comment

moretea commented Jul 10, 2017 via email • edited

edude03 commented Jul 11, 2017

edude03 commented Aug 15, 2017

offlinehacker commented Sep 23, 2017

bachp commented Sep 23, 2017

moretea commented Sep 23, 2017 via email

bachp commented Sep 23, 2017

mmahut commented Aug 1, 2019

mmahut commented Aug 19, 2019

moretea commented Jan 18, 2017 •

edited

moretea commented Jul 10, 2017 via email •

edited