Nixops Core, Plugin Restructuring PR #1179

johnalotoski · 2019-07-09T02:39:03Z

This PR offers a restructuring of nixops to utilize a plugin architecture. It follows on the initial work
of @grahamc found at PR #981.

This restructuring maintains the core functionality of nixops, but only retains internal support for the
"None" backend. The idea is that all other backends can be moved into their own repositories and act as plugins which can be pulled into nixops core as desired by the end user. There are a number of reasons to consider this approach, such as those mentioned at issue #1158.

The Pluggy plugin system, utilized by pytest, has continued to be utilized in this PR.

The code that was removed from nixops for AWS and Hetzner backend functionality has been placed into their own plugin repos: nixops-aws and nixops-hetzner. The code for each of these new plugin repos is largely the same as it was while a part of nixops with the exception of some pluggy hook implementation code and necessary namespace modifications to accommodate backend plugin code utilizing core nixops code.

In this PR, nixops is implementing three pluggy hooks:

load: returns a list of plugin modules to be loaded
nixexprs: returns a list of plugin Nix expressions to import
parser: used to pass parser and subparser objects to plugins to extend the CLI as needed.

To build nixops with plugins of interest, a similar approach to what Terraform in nixpkgs does was implemented. In the nixops repo an 'all-plugins.txt' file describes repos where nixops plugins exist. A script, 'update-all', can be run to check those repos for any plugin updates, detected as the
latest github plugin repo releases, and it records the detected plugins to a 'data.nix' file in a format consumable by fetchFromGitHub.

Nixops can be built with the desired plugins by issuing variants of the following commands where "plugin1" and "plugin2" would be example names of available plugins found in the data.nix file and the command containing callPackage points to a local plugin repo:

nix-build release.nix -A build.x86_64-linux --arg p "(p: [ p.plugin1 ])"
./dev-shell --arg p "(p: [ p.plugin1 p.plugin2 ])"
./dev-shell --arg p "(p: [ (p.callPackage ../myplugin/release.nix {}) ])"

The python test library has been run against the AWS plugin and the tests TestEc2RdsDbinstanceTest and TestEc2WithNvmeDeviceMapping do fail. However, these same tests also fail on the current nixops master branch without the plugin architecture, so this appears unrelated to the plugin restructuring. These failed tests also appear to be addressed and resolved within unmerged PR #969. We have used
the AWS plugin to successfully spin up and tear down clusters of several EC2 servers with Route53 records for both on-demand and spot instances without noticing any functional issues.

The Hetzner plugin is available for testing, but has not yet been tested.

A new nixops command has been added to print the plugins which are utilized in the nixops build:

nixops list-plugins [-v]

The AWS and Hetzner plugins utilize the load and nixexprs pluggy hooks mentioned above, but are not yet using the parser hook which could be used to migrate AWS or Hetzner specific nixops CLI commands to their own nixops CLI plugin namespace, such as the nixops backup/restore command currently exclusive to the AWS backend.

A working example of a nixops plugin using the pluggy parser hook is our packet.net plugin. Building nixops with this plugin extends the nixops CLI with a nixops packet [CMD] subparser where the sos-console sub-command is currently available:

nixops packet sos-console -d $CLUSTER $MACHINE

In order to accomplish plugin CLI extensibility, the main nixops script which lived outside the module structure has had all its function definitions moved into the nixops module folder structure in script_defs.py. The 'scripts/nixops' file still exists, but now contains just parser configuration and execution. To facilitate the script restructuring, explicit argument passing is now done where global variables were initially utilized in the script. This way, script functions are now available to
plugins for extensibility purposes.

This PR does not address documentation generation or test library changes needed to follow this plugin architecture. Such changes are likely implementable in a similar fashion to what has been described above.

This commit makes modifications to move backend code to pluggy plugins which will exist in other repos. The exception to this is the None backend type which remains present. Modifications in this initial restructuring include: - Deletion of files to be moved to other repos - Addition of files related to pluggy hooks - Ex: nixops/plugins/* files - Modification of files to utilize pluggy hooks and pluggy resources - Ex: eval-machine-info.nix - release.nix changes to enable using nixops with plugins using CLI patterns of: nix-build release.nix -A build.x86_64-linux --arg p "(p: [ p.plugin1 ])" ./dev-shell --arg p "(p: [ p.plugin1 p.plugin2 ])" ./dev-shell --arg p "(p: [ (p.callPackage ../myplugin/release.nix {}) ])" - Modification of scripts/nixops to include a new command: nixops list-plugins [-v]

To enable plugin CLI extensibility, this commit extracts the function definitions out of scripts/nixops and places them in nixops/script_defs.py. Explicit argument passing is now done where global variables were initially utilized in the script. This way, script functions are now available to plugins for extensibility purposes.

asymmetric · 2019-07-09T11:25:40Z

These failed tests also appear to be addressed and resolved within unmerged PR #969

Speaking of, has anyone looked into what we would do with the unmerged PRs targeting the now-moved backends?

Are there enough for this to be a concern? Or is it feasible to reopen them on new repos?

This very crude search returns 12 out of 65 open PRs.

AmineChikhaoui · 2019-07-09T12:49:55Z

Very nice ! Thanks @johnalotoski and everybody involved in this, this is very exciting.
I'll try to play with this in the next days :-)

One comment I mentioned already to @grahamc is the git commit history, it would be really nice to find a way to keep the history in the new backend repos (git blame has been very helpful to understand past decisions/implementation details). Graham mentioned that it shouldn't be very hard to do so would be nice to share the steps to do that for other backends. This is not urgent for now, but would be nice to have once we do the split.

johnalotoski · 2019-07-09T16:26:16Z

Hi @asymmetric,

Speaking of, has anyone looked into what we would do with the unmerged PRs targeting the now-moved backends? Are there enough for this to be a concern? Or is it feasible to reopen them on new repos?

Without looking at the various PRs' code in detail, I'm guessing a lot of them could be moved to their respective new plugin repos and be adapted, if necessary, with minimal effort. Good question for community discussion on how best to handle that.

Hi @AmineChikhaoui, have fun playing! :) Keeping the git commit history, as you suggest, sounds like a good idea. I'm assuming that if the community would like to move forward with this approach, that the nixops-aws and nixops-hetzner plugins will get moved under NixOS repo control. We could then make sure the git history gets pulled in when that happens, as well as do some cleanup (recent plugin commits getting squashed down, updating the plugin README, etc).

A couple other thoughts:

Getting an example skeleton plugin made to show the basic plugin code requirements may be helpful to the community.
If this plugin approach is adopted, then until the other backends are migrated, they will be broken. It may be worth considering maintaining the existing version of nixops in nixpkgs separately from a nixops plugin architecture version until it can mature to a reasonable level.

grahamc · 2019-07-22T17:30:02Z

I'd like to put together a to-do list on merging this PR. Could you help with that @johnalotoski?

doc/manual/overview.xml

This commit makes modifications to move backend code to pluggy plugins which will exist in other repos. The exception to this is the None backend type which remains present. Modifications in this initial restructuring include: - Deletion of files to be moved to other repos - Addition of files related to pluggy hooks - Ex: nixops/plugins/* files - Modification of files to utilize pluggy hooks and pluggy resources - Ex: eval-machine-info.nix - release.nix changes to enable using nixops with plugins using CLI patterns of: nix-build release.nix -A build.x86_64-linux --arg p "(p: [ p.plugin1 ])" ./dev-shell --arg p "(p: [ p.plugin1 p.plugin2 ])" ./dev-shell --arg p "(p: [ (p.callPackage ../myplugin/release.nix {}) ])" - Modification of scripts/nixops to include a new command: nixops list-plugins [-v]

To enable plugin CLI extensibility, this commit extracts the function definitions out of scripts/nixops and places them in nixops/script_defs.py. Explicit argument passing is now done where global variables were initially utilized in the script. This way, script functions are now available to plugins for extensibility purposes.

also make the exported nix expressions from the plugins part of the search path which useful for backends such as VirtualBox.

AmineChikhaoui · 2019-07-22T20:59:29Z

@johnalotoski can you rebase/fix conflicts and introduce this change as well, I needed that for VirtualBox but could be needed in other backends:

$ git show b5b805215b5071bfe7a1e256e660d977faf329be
commit b5b805215b5071bfe7a1e256e660d977faf329be (HEAD -> amine-nixops-core-pr, origin/amine-nixops-core-pr)
Author: AmineChikhaoui <amine.chikhaoui91@gmail.com>
Date:   Mon Jul 22 16:43:19 2019 -0400

    add pkgs to config exporters scope

    also make the exported nix expressions from the plugins part of the
    search path which useful for backends such as VirtualBox.

diff --git a/nix/eval-machine-info.nix b/nix/eval-machine-info.nix
index ddb1119..17e7787 100644
--- a/nix/eval-machine-info.nix
+++ b/nix/eval-machine-info.nix
@@ -18,7 +18,7 @@ rec {
     pluginNixExprs;
   pluginOptions = { imports = (foldl (a: e: a ++ e.options) [] importedPluginNixExprs); };
   pluginResources = map (e: e.resources) importedPluginNixExprs;
-  pluginDeploymentConfigExporters = (foldl (a: e: a ++ (e.config_exporters { inherit optionalAttrs; })) [] importedPluginNixExprs);
+  pluginDeploymentConfigExporters = (foldl (a: e: a ++ (e.config_exporters { inherit optionalAttrs pkgs; })) [] importedPluginNixExprs);

   networks =
     let
diff --git a/nixops/deployment.py b/nixops/deployment.py
index 76a56d7..09caaee 100644
--- a/nixops/deployment.py
+++ b/nixops/deployment.py
@@ -251,7 +251,11 @@ class Deployment(object):


     def _nix_path_flags(self):
-        flags = list(itertools.chain(*[["-I", x] for x in (self.extra_nix_path + self.nix_path)])) + self.extra_nix_flags
+        extraexprs = [path
+                      for paths in get_plugin_manager().hook.nixexprs()
+                      for path in paths]
+
+        flags = list(itertools.chain(*[["-I", x] for x in (self.extra_nix_path + self.nix_path + extraexprs)])) + self.extra_nix_flags
         flags.extend(["-I", "nixops=" + self.expr_path])
         return flags

johnalotoski · 2019-07-22T21:49:24Z

Hi @grahamc, @AmineChikhaoui, happy to rebase out conflicts and apply the changes mentioned above, but it may be a few days before I can return to this. In the meantime, @grahamc, apart from the requested changes already mentioned above, migration of the aws and hetzner repos from IOHK to Nixos is also on the ToDo list. I can help with a more detailed, formal list once I clear a few things from my work queue and can get back to this.

PsyanticY · 2019-07-24T11:43:13Z

Hi, Created a plugin for Hashicorp vault resources here . tested it and its working fine. one thing to note though is that we need to create the nix/lib.nix file for every plugin where resources needs to reference other resources It would be nice if we find a way to make it global but it is no big deal. Thanks @johnalotoski for working on this. cc @AmineChikhaoui

AmineChikhaoui · 2019-07-24T13:01:01Z

one thing to note though is that we need to create the nix/lib.nix file for every plugin where resources needs to reference other resources It would be nice if we find a way to make it global but it is no big deal.

I don't think that's necessary, you can pull lib.nix from the search path by doing <nixops/lib.nix> I think.

PsyanticY · 2019-07-25T17:45:49Z

AmineChikhaoui · 2019-07-25T18:17:38Z

Thanks @PsyanticY for putting that together, just want to note that GCE isn't working correctly yet, there is still a couple of issues with having the default bootstrap image. Datadog backend is still pending testing as well but should be easy to get it ready.

nixops-core-pr update

johnalotoski · 2019-07-29T15:54:01Z

@johnalotoski can you rebase/fix conflicts and introduce this change as well, I needed that for VirtualBox but could be needed in other backends:

@AmineChikhaoui: done using your PR

johnalotoski · 2019-07-29T16:08:34Z

Thanks for the list @PsyanticY. We also have a packet.net plugin available with the SOS console command integrated into the nixops CLI: https://github.com/input-output-hk/nixops-packet

nixos-discourse · 2019-08-01T17:53:51Z

This pull request has been mentioned on Nix community. There might be relevant details there:

https://discourse.nixos.org/t/nix-office-hours/2762/20

danbst · 2019-08-06T21:27:15Z

Niiice, delete all the code!

Would NixOps be merged into nixpkgs tree then?

johnalotoski · 2019-08-07T17:43:32Z

In preparation for a move of the AWS and Hetzner plugin repos to NixOS repo control, we've made new branches of the nixops aws and hetzner plugins which have the nixops commit history pulled in and then pruned down by git filter-branch and git removing any nixops files no longer relevant to the particular plugin followed by pruning any resulting empty commits. git blame info should be intact for these plugin branches, found at nixops-aws-fb2 branch and nixops-hetzner-fb2 branch.

nixos-discourse · 2019-08-13T14:08:39Z

This pull request has been mentioned on Nix community. There might be relevant details there:

https://discourse.nixos.org/t/nixos-19-09-feature-freeze/3707/1

doc/manual/default.nix

doc/manual/manual.xml

* To be migrated to the virtualbox plugin

AmineChikhaoui · 2019-09-04T15:27:27Z

@johnalotoski Let's rebase this PR one more time.
I've tagged nixops-aws/nixops-hetzner repositories, so let's update that as well to make sure we point to NixOS org repos.

johnalotoski · 2019-09-04T19:32:52Z

@AmineChikhaoui, references to the plugin repos updated to the nixos github account and merge conflict resolved.

AmineChikhaoui · 2019-09-04T20:49:11Z

Thanks a lot @johnalotoski @disassembler @grahamc .
I have created a branch pre-plugins with the previous state before the merge if anyone needs to use that still.
There is still some couple of details that needs to be addressed which we can work on until the next release:

figure out how to handle plugins documentation/release notes.
make sure functional tests are available in the nixops plugins repositories and easy to run during release preparation.

In the next weeks, I'll work on finishing the setup of the remaining plugins repositories so that we can move them -most likely- to nix-community organization for easier management.
Another thing to be done is making this available in an easy way in nixpkgs, the idea I have for now is to do something similar to vim_configurable where we expose a nixops package that can be customized with extra plugins other than the core ones.

johnalotoski added 2 commits July 8, 2019 19:49

AmineChikhaoui reviewed Jul 22, 2019

View reviewed changes

doc/manual/overview.xml Show resolved Hide resolved

johnalotoski and others added 3 commits July 22, 2019 16:53

add pkgs to config exporters scope

b5b8052

also make the exported nix expressions from the plugins part of the search path which useful for backends such as VirtualBox.

Merge branch 'nixops-core-pr' into amine-nixops-core-pr

051f431

disassembler mentioned this pull request Jul 28, 2019

[WIP]: Rewritting to use boto3 #1186

Closed

Merge pull request #1 from AmineChikhaoui/amine-nixops-core-pr

7f1bc97

nixops-core-pr update

disassembler added 2 commits August 15, 2019 18:34

use pkgs.nixosOptionsDoc if available for docbook

b33d5c0

use same nixpkgs for all plugins

8d70c69

asymmetric reviewed Aug 15, 2019

View reviewed changes

doc/manual/default.nix Show resolved Hide resolved

AmineChikhaoui reviewed Aug 30, 2019

View reviewed changes

doc/manual/manual.xml Show resolved Hide resolved

Removes virtualbox doc section from overview.xml

40eb96a

* To be migrated to the virtualbox plugin

johnalotoski added 2 commits September 4, 2019 13:39

Update the aws and hetzner repos to the nixos github account

baa765b

Resolved merge conflicts from b7cac11

66b0a7c

AmineChikhaoui merged commit 284924c into NixOS:master Sep 4, 2019

asymmetric mentioned this pull request Oct 25, 2019

RDS: support vpc-security-groups for RDS-instances/dbSubnetGroups #1020

Closed

tewfik-ghariani mentioned this pull request Nov 24, 2019

Security groups on AWS #1209

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Nixops Core, Plugin Restructuring PR #1179

Nixops Core, Plugin Restructuring PR #1179

johnalotoski commented Jul 9, 2019 •

edited

asymmetric commented Jul 9, 2019 •

edited

AmineChikhaoui commented Jul 9, 2019

johnalotoski commented Jul 9, 2019

grahamc commented Jul 22, 2019

AmineChikhaoui commented Jul 22, 2019

johnalotoski commented Jul 22, 2019

PsyanticY commented Jul 24, 2019

AmineChikhaoui commented Jul 24, 2019

PsyanticY commented Jul 25, 2019 •

edited

AmineChikhaoui commented Jul 25, 2019

johnalotoski commented Jul 29, 2019

johnalotoski commented Jul 29, 2019

nixos-discourse commented Aug 1, 2019

danbst commented Aug 6, 2019

johnalotoski commented Aug 7, 2019

nixos-discourse commented Aug 13, 2019

AmineChikhaoui commented Sep 4, 2019

johnalotoski commented Sep 4, 2019

AmineChikhaoui commented Sep 4, 2019

Navigation Menu

Nixops Core, Plugin Restructuring PR #1179

Nixops Core, Plugin Restructuring PR #1179

Conversation

johnalotoski commented Jul 9, 2019 • edited

asymmetric commented Jul 9, 2019 • edited

AmineChikhaoui commented Jul 9, 2019

johnalotoski commented Jul 9, 2019

grahamc commented Jul 22, 2019

AmineChikhaoui commented Jul 22, 2019

johnalotoski commented Jul 22, 2019

PsyanticY commented Jul 24, 2019

AmineChikhaoui commented Jul 24, 2019

PsyanticY commented Jul 25, 2019 • edited

AmineChikhaoui commented Jul 25, 2019

johnalotoski commented Jul 29, 2019

johnalotoski commented Jul 29, 2019

nixos-discourse commented Aug 1, 2019

danbst commented Aug 6, 2019

johnalotoski commented Aug 7, 2019

nixos-discourse commented Aug 13, 2019

AmineChikhaoui commented Sep 4, 2019

johnalotoski commented Sep 4, 2019

AmineChikhaoui commented Sep 4, 2019

johnalotoski commented Jul 9, 2019 •

edited

asymmetric commented Jul 9, 2019 •

edited

PsyanticY commented Jul 25, 2019 •

edited