@GrahamcOfBorg build kubernetes kubernetes_1_13_5

@GrahamcOfBorg eval

Should I add some documentation re: mkKubernetes?

Should I add some documentation re: mkKubernetes?

Yes, but I wouldn't recommend writing it until you've got feedback and you're sure it's not going to change. Mostly to save you rewrites if you change anything.

Specifics I'd like feedback on:

1. Which versions do we want to hardcode? https://github.com/kubernetes/kops/blob/master/channels/stable seems a good source to me
2. Do we like mkKubernetes? (I certainly do)
3. How should I document this? Should I document mkKops while I'm at it?

Thanks!

Thanks, looks awesome, exactly what I need for refactored nixos kibernetes module I'm working on 😀 I will try it, and look into it, most likely during the weekend.

I did some simplifications:

• changed from callPackage to callPackages, which allows overriding attributes per package
• expose only minor versions, not patch versions in all-packages
• remove mkKubernetes, as i do not see major benefit having that method exposed (you can use overrideDerivation to change source for particular version).
• updated version.json with new kubernetes 1.15 and change default version to the latest one
• simplified mkversion.sh script and replace /usr/bin/env shebang with nix-shell

This is how you can build different versions of kubernetes and override different things, like go version:

• Override go version

nix-build -E 'with import ./. { }; kubernetes_1_15.override { go = go_1_12; }'

• Override source

# What i'm doing here is stupid, but you get an idea
nix-build -E 'with import ./. { }; kubernetes.overrideDerivation (p: { src = kubernetes_1_14.src;
})'

• Change components

nix-build -E 'with import ./. { }; kubernetes_1_4.override { components = ["cmd/kubectl" "cmd/kubeadm"]; }'

Looks like kubernetes 1.15 has problems building, will have to take a look what is wrong.

Ok, fixed kubernetes 1.15 builds, looks like i had to set $HOME, as it creates some cache there.

I will make more explicit which binaries gets installed, as now we get a few of build time binaries in $out/bin

So old versions did not build, due to go-1_12 being used, but i think it's more than ok to have last 5 versions. @yurrriq @worldofpeace are you ok with changes? for explanation take a look here: #66496 (comment) I will do a rebase then and join some commits.

@GrahamcOfBorg build kubernetes kubernetes_1_11 kubernetes_1_12 kubernetes_1_13 kubernetes_1_14 kubernetes_1_15 kubectl

All good on not having patch specific versions in all-packages.nix, also I think the removal of mkKubernetes needs feedback from @yurrriq.

I did removed mkKubernetes as you can change source with overrideDerivations. We do not have explicit build functions for versions exposed in other packages that expose multiple versions like openssl.

Changes look good to me. I think we should add some documentation with an override example.

Hmm some builds failed on x86_64-linux, which is odd based on fact, that aarch64-linux builds succeed.

@GrahamcOfBorg build kubernetes kubernetes_1_11 kubernetes_1_12 kubernetes_1_13 kubernetes_1_14 kubernetes_1_15 kubectl

Thanks, @offlinehacker and @johanot. I'm on a work trip now too, and head home Saturday fwiw. Should be able to help once I'm back.

I am working on refactoring of nixos kubernetes module, that will also take into account this package.

@offlinehacker Sounds good!

To be honest, my thoughts on the Kubernetes module of the future are... a bit more radical, perhaps. :-) I'm working on a proposal along the lines of:

• Pull the existing module out of NixOS/nixpkgs and into a separate dedicated repository.
• A new minimal module will be available in nixpkgs allowing for simple enable/disable and custom config of main kubernetes components (apiserver, controller-manager, scheduler, kubelet)
• The smart kubernetes module we have today will be available as an out-of-tree module much like Simple NixOS Mailserver - perhaps as a flake.
• The future in-tree Kubernetes module will be "Kubernetes the hard way". No magic, No auto-provisioning of secrets or dynamic config.

Besides reducing the number of kubernetes NixOS options from ~145 to ~20, it will also become much easier to support many different versions of Kubernetes with the same module. In fact, I looked into 1.15 a bit further yesterday, and it looks like there's at least one option that has to be removed on the kubelet, since the --allow-privileged flag has been removed. If we get rid of all these very specialized options, module maintenance and backward compatibility becomes less of a pain.

Anyway.. This will definitely not happen for 19.09. So let's discuss that later.

Before that happens, if you can help fixing current tests, when you get back, it would be much appreciated

Sure thing.

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/kubernetes-the-nixos-module-of-the-future/3922/1

@johanot @yurrriq what is status with this pull request? Can we merge it after resolving conflicts and rebase, now when #67563 has been merged?

@offlinehacker I think we should discuss it further over at Discourse and see more fleshed out proposed solutions before doing anything to nixpkgs proper.

We are using discourse regarding discussions regarding tickets? I mean what is the reason not discussing here?

No strong reason either way, the discussion was just opened over there and is not tied to this pr specifically. I just don't want to merge this before we reach some sort of agreement. If you don't want to participate I'm happy to bring your opinions for you, if you tell me what they are. 😃

The discussion on Discourse, to me, seems only indirectly related to this, so I don't think we need to wait for consensus about the module there.

I rebased and squashed.

@srhb agreed, will join the discussion, but it is more related to nixos module as i can see.

I still think this pull request is unrelated to other developments as it is just improvement to package multiple versions of kubernetes. I'm just unaware if latest version from this pull requests works with fixes that @johanot implemented? I will have time during the weekend to test it out.

I agree that this PR is at least tangential to the module discussion, but I don't want to accidentally imply that the module supports the versions of kubernetes in nixpkgs proper if we decide to keep on only supporting a single version. It might just require a bit more documentation.

That said, now that I've been sort of forced to think about it, I am opposed to this PR in its entirety and think we should only keep the latest version in nixpkgs. If users have specific needs for older software, they can go ahead and override as needed.

@srhb, can you explain the reasoning behind your opposition some more?

To me, including the most recent patch versions of a few recent minor version (as we have it now) seems ideal. I think it's quite common that users will need different versions, and rather than have multiple ad hoc implementations of a generic builder, I think it would be better to have an "official" one. I also don't see a problem with include some known version/hash attrsets, especially those supported by kops, EKS, AKS, etc.

If the community at large is opposed to this, I suppose I'll just keep it in my nur-packages.

We would also need a support for multiple versions at @GateHubNet and @xtruder to have kubectl support for older versions of kubernetes, as we are not upgrading with same pace as "official" kubernetes. If there would LTS kubernetes it would be nice..

Also in gke you can select which version of kubernetes you want and the same is with some other tools like kubeadm and kops, so I cannot see why shouldn't we support multiple versions?

I've also given this an extra thought. I think I agree on the argument that; if supporting older versions involves nothing else than overriding version and hash, I don't see why users cannot overlay as needed?

I have no principled arguments against having multiple versions in nixpkgs, as long as we can guarantee that they're all stable and someone actively maintain them. Unfortunately, I don't have the time myself to guarantee the working order of all versions from 1.9 -> 1.15, nor can I say whether the NixOS module will be properly backward compatible. If possible, I'd prefer the NixOS module to be compatible with all nixpkgs-available package versions, and thus I think the module discussion is relevant.

And I might have answer for that, I was working hard to make improvements to kubernetes nixos module, the current progress can be seen here: https://github.com/xtruder/nixpkgs/tree/nixos/kubernetes/refactoring that should also take into account multiple versions. Unfortunately I did not have time to make proper pull request 😕 I can see what I can do in following weekends, but I want to have tests for all versions of kubernetes that we have in nixpkgs, this would be ideal.

Re: maintenance of versions. We can use (or enhance) this script to stay current with e.g. kops supported versions. https://github.com/NixOS/nixpkgs/blob/d89a311678b1f1cc7abd728c06d062f1465c97b9/pkgs/applications/networking/cluster/kubernetes/mkversions.sh

@yurrriq

@srhb, can you explain the reasoning behind your opposition some more?

I'm mostly worried about adding complexity that we can't easily bail out on currently. I would prefer that we crystallize a plan that takes into account the module as well as the packages. Right now, it seems several people are working on various proposals and PRs for kubernetes, and I would like to try and ensure that everything fits together very well, not just for a few releases, but hopefully for longer. I'm also not very fond of promoting kops to some sort of special truth for the nixpkgs ecosystem. It's not a big deal, but mostly just trying to gate everything with disclaimers that things might be changing soon.

That said, I'm not strongly opposed to having multiple versions in nixpkgs -- at least not as strongly opposed as trying to make the module support several versions. I will not attempt to block your efforts here, but I do care that we take special care to ensure the module in whatever shape or form we decide upon in the coming months, is very well supported by the kubernetes package(s) in nixpkgs proper. :-)

I chose kops arbitrarily, and agree it should not be given any special status. We should probably just include the latest patch versions of the last three minor versions, which are supported as per the k8s documentation: https://kubernetes.io/docs/setup/release/version-skew-policy/#supported-versions

It would seem to me those three versions ought to be supported by the module as well.

I still very much want an easy way to build specific versions locally, whether via generic builder or overrides. Perhaps this is already addressed by #67563

Thoughts?

I'm undecided, i mean multiple version support might be helpful for us, but we will just use overlay if this will not get merged and there is decision to only support latest version. My thought is that nixpkgs monorepo has too much stuff anyway, and should be split and in that case having single version is reasonable decision.

So, I think now the decision is either to include the three supported versions, and the generic builder, or close this PR. I'm pretty sure #67563 addressed my needs.

Closing this as it seems like, collectively, we don't want to go this direction. Thanks to everyone for their feedback.