Openstack modules #19529

nlewo · 2016-10-13T20:29:30Z

I continued the works started by @domenkozar on Openstack NixOS modules.
I'm now able to deploy an Openstack compute node, create a network, a floating IP, boot a VM and ssh to it. So all basic components seem to work well;)
Note the Neutron driver is currently linuxbridge but I plan to add OVS.

There is still some (a lot of?) work to have clean modules.
I would like to know if some people are interested in these modules and if we could define what is required to go to a merge in master.

For those who are not familiar with Openstack but want to try it, I wrote a short guide to easily play with these modules, thanks to nixops.

ebtables

mention-bot · 2016-10-13T20:29:32Z

@nlewo, thanks for your PR! By analyzing the history of the files in this pull request, we identified @edolstra, @bjornfor and @peti to be potential reviewers.

domenkozar · 2016-10-14T08:57:56Z

Cool :) I've used this work for https://github.com/domenkozar/snabb-openstack-testing and never got to finishing the cleanup phase, which needs quite some work as you said :)

Sometimes, neutron tries to create its service but keystone admin user has not been created yet.

We added an interface that is used by the public network. This allows us to ping a VM from the hypervisor.

nlewo · 2016-11-22T07:54:05Z

The test has been updated to ping a VMs from the hypervisor through the public network thanks to the neutron linuxbridge driver.
All credentials and token are variabilized.
Admin and service tenants are created in the systemd post start section.

@domenkozar, do you know someone who could be interested to comment or review this PR ?

badmutex · 2016-11-30T15:58:27Z

@nlewo Thank you for this. I saw your post on the mailing list and was eager to try it.
I had to modify openstack.nix from your guide. Namely:

virtualisation.keystone.enableSingleNode

does not exist in 59daf08

I had to instead set

virtualisation.keystone.enable

I'm also deploying on to virtualbox rather than libvirtd.

Below is the full openstack.nix I'm using.
Deploying with

$ git rev-parse HEAD
59daf08b6fa388828484e4f01f62c15aa73660a0
$ nixops deploy -d openstack -I nixpkgs=`pwd`

{

  openstack =
    {config, pkgs, nixpkgs, ...}:
    let

      credentials = {
        keystoneAdminUsername = "admin";
        keystoneAdminTenant = "admin";
        keystoneAdminPassword = "admin";
      };
      osConfig = {
        endpointPublic = config.networking.privateIPv4;
      } // credentials;
      keystoneConfig = {
        enable = true;
      } // osConfig;
      otherConfig = {
        enableSingleNode = true;
      } // osConfig;

    in

    {

      deployment.targetEnv = "virtualbox";
      deployment.virtualbox.memorySize = 4096;
      deployment.virtualbox.headless = true;

      nixpkgs.config.allowBroken = true;

      networking.firewall.enable = false;

      environment.systemPackages = with pkgs.pythonPackages;
        [ neutronclient
          keystoneclient
          glanceclient
          novaclient
        ];

      virtualisation.keystone = keystoneConfig;
      virtualisation.glance = otherConfig;
      virtualisation.neutron = otherConfig;
      virtualisation.nova = otherConfig;

    };

}

spinus

Hey, I did small review. Biggest thing I noticed is that there is a lot of passwords in the configs and scripts which probably could be extracted to files (so for example files with secrets can be deployed by "nixops" or other secret managment, instead of putting them in nix store.

I'm new to open stack but I'm interested in that work and will be happy to review test and support with commit power if needed. Please ping me about that whenever you want.

spinus · 2016-12-01T01:39:10Z

nixos/lib/make-disk-image.nix

@@ -42,6 +42,8 @@ pkgs.vmTools.runInLinuxVM (
          mv closure xchg/
        '';
      buildInputs = [ pkgs.utillinux pkgs.perl pkgs.e2fsprogs pkgs.parted ];
+      allowSubstitutes = false;


Are you sure you want to change system level parameters in package pull request?
If you need those in package tests maybe worth to override those variables in package specific tests rather than globally?

I don't know why this was set... so I remove it and tests are still ok.

spinus · 2016-12-01T01:39:39Z

nixos/modules/virtualisation/glance.nix

+  cfg = config.virtualisation.glance;
+  commonConf = ''
+    [database]
+    connection = mysql://glance:${cfg.dbPassword}@${cfg.dbHost}/glance


passwords in nix store? I would prefer not.

@spinus Could you provide a pointer related to secret managment? I don't know what is secret managment good pratice.

@nlewo I don't think we have something like that. But there is known thing that all derivations end up in /nix/store so all the passwords go there.

What I do to avoid it is:

I follow nixops pattern to put ALL credentials in the key store (https://nixos.org/nixops/manual/#opt-deployment.keys) - those keys are not going to be written in /nix/store, they will end up in /run/keys/.

in the scripts I use:

run-something --secret "$(</run/keys/<my-secret>)"

Config files are little harder. If it's possible to use that by the process, use mix of config files and environment variables. If it's not possible, I keep template of config file in nix store, and than, on preStart script I copy this file to /var/lib// and fill the template (for example with sed) with keys from /run/keys.

I would prefer to not do this now since this PR is already really big. This will add more complexity. Moreover, current modules target a dev environment, where password are not really important.
What do you think to address this problem in another PR (once/if this one is merged)?

Oh, I have no merge power here, I'm only talking :-) You don't even have to listen to me :-) probably better to ask people who have merge power.

Personally I woudn't merge thing in that state. I agree it's really big. If I would be doing that, I would create separate PR for each service and than another PR for "top-level" service which configures smaller services.

yep, I also think I have to create a PR per components.

It's the best when mysql client uses mysql client config files only. They allow !include, and thus you could have non-sensitive parameters in a "public" file and secret file included in it.

@ip1981 Sorry, I don't understand your point. Could you point me a doc or an example related to the pattern you are suggesting?

Like this https://github.com/ip1981/mywatch#configuration

spinus · 2016-12-01T01:39:56Z

nixos/modules/virtualisation/glance.nix

+    project_domain_id = default
+    user_domain_id = default
+    username = ${cfg.serviceUsername}
+    password = ${cfg.servicePassword}


password in nix store

spinus · 2016-12-01T01:40:25Z

nixos/modules/virtualisation/glance.nix

+      example = "localhost";
+    };
+
+    dbPassword = mkOption {


password in nix store

spinus · 2016-12-01T01:41:14Z

nixos/modules/virtualisation/glance.nix

+      '';
+    };
+
+    keystoneAdminPassword = mkOption {


password in nix store

spinus · 2016-12-01T01:48:52Z

nixos/modules/virtualisation/keystone.nix

+
+    systemd.services.keystone-all = {
+        description = "OpenStack Keystone Daemon";
+        after = [ "mysql.service" "network.target"];


if you have after, maybe you want "requires" or "wants" as well?

I added requires statement.

Remember that if you use requires and mysql fail or restart this component will be failed as well, make sure that's what you expect :-)

This suggests that you use local mysql only. If so, you don't need any passwords: use UNIX-socket authentication. But I think local mysql is not a strategy, and then the service better not to depends on mysql, or poll it.

spinus · 2016-12-01T01:54:17Z

nixos/modules/virtualisation/neutron.nix

+
+    publicInterface = mkOption {
+      default = "eth0";
+      description = "The pysical interface name used by the linuxbridge agent";


I don't know openstack very well so probably I'm missing a bit. Usually I think the setup is that public interface is the separate one that bridge interface for vms, no? what's the story here?

This interface is added to the bridge created for an network which is 'bound' to an external one. I update the documentation to provide some help about this.

spinus · 2016-12-01T01:56:52Z

nixos/modules/virtualisation/neutron.nix

+
+    systemd.services.neutron-server = {
+      description = "OpenStack Neutron Daemon";
+      after = [ "rabbitmq.service" "mysql.service" "network.target"];


maybe "requires" or "wants" needed as well?

Added requires.

spinus · 2016-12-01T01:59:22Z

nixos/modules/virtualisation/neutron.nix

+        chown neutron:neutron /var/lock/neutron/ /var/lib/neutron
+
+        # TODO: move out of here
+        mysql -u root -N -e "create database neutron;" || true


probably some dbHost value not propagated here

spinus · 2016-12-01T02:03:16Z

nixos/modules/virtualisation/neutron.nix

+
+        # Wait until the keystone is available for use
+        count=0
+        while ! keystone user-get ${cfg.keystoneAdminUsername} > /dev/null


Here you could do a nice trick. Becasue keystone can hang for whatever reason, you could prefix it with timeout i.e: "timeout 30 keystone...."

Next thing (just to consider) is to use $SECONDS variable instead of count, so you know for how long you will be trying. ($SECONDS gives you seconds since bash started, so you could check condition like :

max=$(($SECONDS+30)) if [ $SECONDS < $max ];then break fi

)

I prefer to let it as it is since this is not really a timeout. We try 30 times but each try can be longer than 1 seconds.

nlewo · 2016-12-01T08:06:12Z

@badi thx for your feedback. I upgraded the blog post according to your remarks. There is now a nixos test that builds a VM ready to use. I think this is the most convenient way to do some tries.

nlewo · 2016-12-01T22:06:50Z

@spinus thanks for all of your comments.

spinus · 2016-12-01T22:23:57Z

@nlewo no worries. I'm interested in helping with this, I was just too lazy to start it myself but if you need a tester, code reviews or some other help let me know.

nlewo · 2016-12-04T21:59:52Z

I start to split this PR into several sub PR. The first one is keystone #20905.

domenkozar · 2016-12-06T09:47:09Z

Sorry as I don't have time to review this, but others have jumped on the ship. Thanks for getting this upstream!

Just a reference, here is WIP backend for nixops: NixOS/nixops#408

nlewo · 2016-12-06T10:40:27Z

@domenkozar, this PR is too big and really not attractive for reviewers... It's why I submitted a PR that only addresses Keystone #20905. For sake of clarity, I squashed our commits into one and I mentioned your works in the commit message (I hope you're ok with that).

spinus · 2016-12-06T10:44:35Z

@domenkozar what do you think about passwords and secrets, should this be passed as file (or env variable, or other) or still worth to merge this with passwords in nix store as a first draft?

ip1981 · 2016-12-06T10:50:03Z

worth to merge this with passwords in nix store as a first draft

why rush?

spinus · 2016-12-06T11:32:00Z

@ip1981 no rush, I'm just exposing some comments
check :
#19529 (comment)
#19529 (comment)

domenkozar · 2016-12-06T12:15:28Z

@spinus I don't know how to handle that best. It's the reason I haven't submitted this upstream. They shouldn't be in store, but we do have some services that do it today, but eventually someone will be screwed.

spinus · 2016-12-06T12:18:33Z

@domenkozar that is my impression too hence I suggested to refactor this to keep secrets out of nix store but wanted some confirmation from the core team. Cheers.

grahamc · 2016-12-22T00:34:36Z

Hi @nlewo, I am thrilled that you're putting this effort in. I hope the (long) delay to getting this merged hasn't discouraged you too much.

One thing I've noticed is basically all our current openstack modules need upgrades. Is that a project you could undertake as well?

nlewo · 2016-12-22T08:41:11Z

Hi @grahamc, I understand it takes time to upstream all these modules and since this is the first time I'm doing this... it takes even more times!

I already planned to upgrade our openstack version, but I would prefer to first have modules for basic openstack modules in order to be able to easily do functional tests.
Keystone is merged and I'll submit glance in the next few days.

Do you have an idea (order of magnitude) on how many time we could continue to have the liberty release? (I know several production lines where Juno is still in use...)
This would allow me to know if I can continue to upstream nova and neutron modules or if I have to first upgrade openstack packages.

Are there known security issues on liberty components?

Thanks for your interest.

Mic92 · 2017-03-12T21:45:37Z

Glance and Keystone are now already part of nixos.

nlewo · 2017-03-18T18:00:12Z

I will create new PRs regarding neutron and nova (as I did for keystone and glance). So I close this one mainly to clean the backlog;)

domenkozar and others added 14 commits September 29, 2016 21:11

nixos: add keystone, nova, neutron, glance modules

6c4a967

neutron: be able to define plugin packages

a59b4af

neutron: enable snabb

5b017f1

neutron: add extraML2Config option

5521ec0

nova: use qemu instead of kvm

fd14833

make-disk-image.nix: build locally as image may be big

dfb6107

make-disk-image.nix: preferLocalBuild=true

acf0ffc

neutron: write test temp files to $PWD instead of /tmp/

06ba7bb

neutron: set default value to extraPackages

d7034b6

neutron: enable l3 agent and set various options

bd3418b

openstack: default provisionning for a ready to use mode

2afbb46

nova: temporary run it as root to handle iptables

744f337

neutron: Fix a conflict on /etc/ethertype file between iptables and

111855e

ebtables

openstack: add endpoint_type option

d7677d5

domenkozar added 6.topic: nixos 2.status: work-in-progress labels Oct 14, 2016

nlewo added 7 commits October 19, 2016 20:50

openstack: use env variables in bootstraping

14feba9

keystone: adminToken as cfg variable

82ea46f

keystone: typo

f0a8ece

openstack: cfg variables for mysql credentials

09b2b69

openstack: service credentials variabilized

35d1bbd

openstack: update descriptions

f072149

keystone: cleaning

9e1cff1

nlewo force-pushed the openstack branch from 06084ab to 9e1cff1 Compare November 5, 2016 07:52

nlewo added 4 commits November 6, 2016 23:31

neutron: variabilize public interface

8908da3

openstack: ensure keystone admin has been created

a121113

Sometimes, neutron tries to create its service but keystone admin user has not been created yet.

nixos/tests/openstack: ping the booted VM

67de1a8

We added an interface that is used by the public network. This allows us to ping a VM from the hypervisor.

neutron: variabilize metadata secret

59daf08

nlewo changed the title ~~wip: Openstack modules~~ Openstack modules Nov 22, 2016

spinus suggested changes Dec 1, 2016

View reviewed changes

nlewo force-pushed the openstack branch from 6cd04b7 to 86cfd88 Compare December 1, 2016 21:43

openstack: changes according to the spinus's review

ac110a1

nlewo force-pushed the openstack branch from 86cfd88 to ac110a1 Compare December 2, 2016 09:16

nlewo closed this Mar 18, 2017

Openstack modules #19529

Openstack modules #19529

Conversation

nlewo commented Oct 13, 2016

mention-bot commented Oct 13, 2016

domenkozar commented Oct 14, 2016

nlewo commented Nov 22, 2016

badmutex commented Nov 30, 2016

spinus left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nlewo commented Dec 1, 2016

nlewo commented Dec 1, 2016

spinus commented Dec 1, 2016

nlewo commented Dec 4, 2016

domenkozar commented Dec 6, 2016

nlewo commented Dec 6, 2016

spinus commented Dec 6, 2016

ip1981 commented Dec 6, 2016

spinus commented Dec 6, 2016

domenkozar commented Dec 6, 2016

spinus commented Dec 6, 2016

grahamc commented Dec 22, 2016

nlewo commented Dec 22, 2016

Mic92 commented Mar 12, 2017

nlewo commented Mar 18, 2017