nixos/curator: init elasticsearch curator #44340

shmish111 · 2018-08-02T08:28:40Z

Motivation for this change

I needed to use this tool on my NixOS ES cluster

Things done

Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS)
Built on platform(s)
- NixOS
- macOS
- other Linux distributions
Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
Tested compilation of all pkgs that depend on this change using nix-shell -p nox --run "nox-review wip"
Tested execution of all binary files (usually in ./result/bin/)
Determined the impact on package closure size (by running nix path-info -S before and after)
Fits CONTRIBUTING.md.

basvandijk · 2018-08-03T00:27:45Z

nixos/modules/services/search/curator.nix

+
+  options.services.curator = {
+
+    enable = mkOption {


Maybe use mkEnableOption here.

basvandijk · 2018-08-03T00:29:34Z

nixos/modules/services/search/curator.nix

+    };
+in {
+
+  options.services.curator = {


I think we should name the module elasticsearch-curator instead of just curator because it's more descriptive and consistent with the name of the Python package.

basvandijk · 2018-08-03T00:30:41Z

nixos/modules/services/search/curator.nix

+
+  config = mkIf cfg.enable {
+
+    systemd.services.curator = {


As with the module name I think we should also name the systemd unit elasticsearch-curator instead of just curator.

basvandijk · 2018-08-03T00:32:00Z

nixos/modules/services/search/curator.nix

+      enable = cfg.enable;
+      startAt = cfg.interval;
+      serviceConfig = {
+      ExecStart = ''${pkgs.python36Packages.elasticsearch-curator}/bin/curator --config ${curatorConfig} ${curatorAction}'';


<nitpicking> could you indent this line? </nitpicking>

basvandijk · 2018-08-03T00:34:00Z

nixos/modules/services/search/curator.nix

+        # not a Python "NoneType"
+        client:
+          hosts: ${builtins.toJSON cfg.hosts}
+          port: 9200


Shouldn't the port also be configurable? We should probably default it to services.elasticsearch.port.

shmish111 · 2018-08-15T07:52:50Z

@basvandijk I have updated the PR according to your PR, do I need to squash the extra commit?

basvandijk

Could you also rename the file to elasticsearch-curator.nix?

basvandijk · 2018-08-15T09:03:07Z

nixos/modules/services/search/curator.nix

+with lib;
+
+let
+    cfg = config.services.curator;


This should now be: config.services.elasticsearch-curator

basvandijk · 2018-08-15T09:03:41Z

nixos/modules/services/search/curator.nix

+    };
+    port = mkOption {
+      description = "the port that elasticsearch is listening on";
+      type = types.int


You're missing a semicolon here.

shmish111 · 2018-08-15T09:51:50Z

fixed. should I squash @basvandijk ?

basvandijk · 2018-08-15T22:36:14Z

@shmish111 thank you. A squash would be nice.

Any chance you could add a NixOS test for this? Maybe extend the existing ELK test.

shmish111 · 2018-08-16T08:13:32Z

@basvandijk do you think it is ok to add it to the ELK test? I feel like it would be unnecessary overhead to have a test that boots up another ES machine when I could just add in some test cases to the existing ELK test. I suppose I would just create an index in ES, run a curator action that should delete it and then check it has been deleted.

basvandijk · 2018-08-16T08:16:50Z

@shmish111 extending the existing ELK test like you suggest is the way to go IMHO. It would not only test curator but also elasticsearch itself.

shmish111 · 2018-08-16T08:26:40Z

@basvandijk how do I run the nixpkgs tests? I can't find documentation anywhere :-(

shmish111 · 2018-08-17T11:16:19Z

@basvandijk I have squashed. Also, I can't get the test passing on master so I can't really do anything until I do that.

uname -a
Linux ip-10-0-4-156.us-east-2.compute.internal 4.14.32 #1-NixOS SMP Sat Mar 31 16:10:43 UTC 2018 x86_64 GNU/Linux

nix-build nixos/tests/elk.nix

This fails to connect to elasticsearch. It seems that elasticsearch has failed to start but I can't work out how to debug in order to find out why.

basvandijk · 2018-08-18T18:03:29Z

See the following on how to run tests interactively:
https://nixos.org/nixos/manual/index.html#sec-running-nixos-tests
That will get you into a console on the VM with which you can debug the issue.

shmish111 · 2018-08-20T15:12:33Z

@basvandijk I've tried to play around but it's difficult to debug as I can't get into a shell in the running VM. All I can see is the output of startAll etc. I managed to see that elasticsearch does start eventually however it take a long long time for some reason. Then logstash is failing, with "TypeError('Invalid argument: family must be 4 or 6');". Also cat /tmp/logstash.out never appears.

Have you managed to run these tests successfully? I am trying to run these on a NixOS machine on AWS with the image NixOS x86_64-linux 18.03.132946.1caae7247b8 (hvm-ebs) as a t2.xlarge instance.

I think the test maintainers need to look at this, does this need to prevent my PR? I'm happy to add a test but the existing test needs to be fixed first and at the moment that is beyond me.

basvandijk

Regarding the test, I can successfully run nix-build nixos/tests/elk.nix on my Mac Book Pro mid 2015 running NixOS 18.03. Maybe a t2.xlarge can't run VMs?

Do you have some statements available that create an index on elasticsearch and an actionYAML definition that deletes that index? Then I can try running that test to see if it works.

basvandijk · 2018-08-20T23:45:54Z

nixos/modules/services/search/elasticsearch-curator.nix

+  config = mkIf cfg.enable {
+
+    systemd.services.elasticsearch-curator = {
+      enable = cfg.enable;


You don't need to set this since enable is true by default and the whole definition is already under a mkIf.

shmish111 · 2018-08-21T08:42:06Z

@basvandijk ok, I've fixed the cfg.enable thing. I've also added a test for curator, this just runs curator after the other tests and checks that the logstash index has been deleted. This index must have been created for the logstash queries to succeed. Can you run these tests because I have no access to a NixOS machine that isn't on AWS at the moment, and I suspect you are correct and that VMs aren't running correctly on AWS instances.

basvandijk · 2018-08-21T15:43:56Z

Great we're getting closer! I pushed some more fixes to your branch. In the end you might want to squash all of them.

Could you look into fixing the elasticsearch-curator python package because building it fails with:

nix-build -A python36Packages.elasticsearch-curator
...
Collecting requests-aws4auth>=0.9 (from elasticsearch-curator==5.5.4)
  Could not find a version that satisfies the requirement requests-aws4auth>=0.9 (from elasticsearch-curator==5.5.4) (from versions: )
No matching distribution found for requests-aws4auth>=0.9 (from elasticsearch-curator==5.5.4)

I see we don't have a package for requests-aws4auth so that needs to be packaged.

basvandijk · 2018-08-21T16:02:21Z

You might also want to add yourself as maintainer of the module using:

  meta.maintainers = with maintainers; [ shmish111  ];

and add shmish111 to the maintainer-list.nix.

basvandijk · 2018-08-25T14:54:51Z

@GrahamcOfBorg test elk.ELK-5 elk.ELK-6

GrahamcOfBorg · 2018-08-25T14:59:07Z

No attempt on aarch64-linux (full log)

The following builds were skipped because they don't evaluate on aarch64-linux: tests.elk.ELK-5, tests.elk.ELK-6

Partial log (click to expand)

Cannot nix-instantiate `tests.elk.ELK-5' because:
error: attribute 'ELK-5' in selection path 'tests.elk.ELK-5' not found

Cannot nix-instantiate `tests.elk.ELK-6' because:
error: attribute 'ELK-6' in selection path 'tests.elk.ELK-6' not found

GrahamcOfBorg · 2018-08-25T15:57:38Z

Timed out, unknown build status on x86_64-linux (full log)

Attempted: tests.elk.ELK-5, tests.elk.ELK-6

Partial log (click to expand)

one: running command: cat /tmp/logstash.out | grep flowers
one# cat: /tmp/logstash.out: No such file or directory
one: exit status 1
one: running command: cat /tmp/logstash.out | grep flowers
one# cat: /tmp/logstash.out: No such file or directory
one: exit status 1
one: running command: cat /tmp/logstash.out | grep flowers
one# cat: /tmp/logstash.out: No such file or directory
building of '/nix/store/q5wv95vqb8bifpzyz0qvcvb78r5vddi8-vm-test-run-ELK-5.drv' timed out after 1800 seconds
error: build of '/nix/store/mp71gw6kysf1y7nilnnvz5gq70dyh7di-vm-test-run-ELK-6.drv', '/nix/store/q5wv95vqb8bifpzyz0qvcvb78r5vddi8-vm-test-run-ELK-5.drv' failed

https://www.elastic.co/guide/en/elasticsearch/client/curator/5.5/index.html

…tor service

TODO: we're still missing the requests-aws4auth>=0.9 dependency

Needed by elasticsearch-curator.

…elasticsearch-curator

FRidh · 2018-08-25T16:54:58Z

pkgs/development/python-modules/requests-aws4auth/default.nix

+
+  propagatedBuildInputs = [ requests ];
+
+  doCheck = false;


Why are the tests disabled? Please include a comment.

The test hang. I'm not sure why. Good point about adding a comment. Will do.

GrahamcOfBorg added 6.topic: nixos 8.has: module (update) 10.rebuild-darwin: 0 10.rebuild-linux: 0 labels Aug 2, 2018

basvandijk suggested changes Aug 3, 2018

View reviewed changes

basvandijk suggested changes Aug 15, 2018

View reviewed changes

shmish111 force-pushed the es-curator branch from 40693f6 to 188d4fd Compare August 17, 2018 11:13

basvandijk suggested changes Aug 20, 2018

View reviewed changes

shmish111 requested a review from FRidh as a code owner August 21, 2018 15:42

GrahamcOfBorg added 6.topic: python 10.rebuild-darwin: 1-10 10.rebuild-linux: 1-10 and removed 10.rebuild-darwin: 0 10.rebuild-linux: 0 labels Aug 21, 2018

basvandijk force-pushed the es-curator branch 2 times, most recently from 8c8862c to 50dd177 Compare August 25, 2018 14:53

GrahamcOfBorg added the 8.has: package (new) label Aug 25, 2018

GrahamcOfBorg added 8.has: changelog 8.has: documentation 2.status: merge conflict labels Aug 25, 2018

basvandijk approved these changes Aug 25, 2018

View reviewed changes

shmish111 and others added 8 commits August 25, 2018 18:53

nixos/curator: init elasticsearch curator

3744467

https://www.elastic.co/guide/en/elasticsearch/client/curator/5.5/index.html

elasticsearch-curator: don't need to add enable to elasticsearch-cura…

2ec33f5

…tor service

elasticsearch-curator: add test

8420005

elasticsearch-curator: include the module in the module-list & fix bug

3220003

elasticsearch-curator: include missing boto3 dependency

16ce3c8

TODO: we're still missing the requests-aws4auth>=0.9 dependency

pythonPackages.requests-aws4auth: init at 0.9

61b6467

Needed by elasticsearch-curator.

elasticsearch-curator: disable test-suite because it hangs

8aa3318

nixos/tests/elk.nix: make sure the test doesn't wait for too long on …

241377e

…elasticsearch-curator

FRidh reviewed Aug 25, 2018

View reviewed changes

basvandijk added 2 commits August 25, 2018 18:59

elasticsearch-curator: add note to the NixOS release notes

228705f

requests-aws4auth: only disable tests on python >= 3

d5e7d48

basvandijk force-pushed the es-curator branch from a174a6a to d5e7d48 Compare August 25, 2018 23:13

GrahamcOfBorg removed the 2.status: merge conflict label Aug 25, 2018

elasticsearch-curator: explain why we disable the test-suite

954eb34

basvandijk merged commit a144c79 into NixOS:master Aug 25, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

nixos/curator: init elasticsearch curator #44340

nixos/curator: init elasticsearch curator #44340

shmish111 commented Aug 2, 2018 •

edited

basvandijk Aug 3, 2018

basvandijk Aug 3, 2018

basvandijk Aug 3, 2018

basvandijk Aug 3, 2018

basvandijk Aug 3, 2018

shmish111 commented Aug 15, 2018

basvandijk left a comment

basvandijk Aug 15, 2018

basvandijk Aug 15, 2018

shmish111 commented Aug 15, 2018

basvandijk commented Aug 15, 2018

shmish111 commented Aug 16, 2018

basvandijk commented Aug 16, 2018

shmish111 commented Aug 16, 2018

shmish111 commented Aug 17, 2018

basvandijk commented Aug 18, 2018

shmish111 commented Aug 20, 2018

basvandijk left a comment

basvandijk Aug 20, 2018 •

edited

shmish111 commented Aug 21, 2018

basvandijk commented Aug 21, 2018 •

edited

basvandijk commented Aug 21, 2018

basvandijk commented Aug 25, 2018

GrahamcOfBorg commented Aug 25, 2018

GrahamcOfBorg commented Aug 25, 2018

FRidh Aug 25, 2018

basvandijk Aug 25, 2018

nixos/curator: init elasticsearch curator #44340

nixos/curator: init elasticsearch curator #44340

Conversation

shmish111 commented Aug 2, 2018 • edited

Motivation for this change

Things done

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

shmish111 commented Aug 15, 2018

basvandijk left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

shmish111 commented Aug 15, 2018

basvandijk commented Aug 15, 2018

shmish111 commented Aug 16, 2018

basvandijk commented Aug 16, 2018

shmish111 commented Aug 16, 2018

shmish111 commented Aug 17, 2018

basvandijk commented Aug 18, 2018

shmish111 commented Aug 20, 2018

basvandijk left a comment

Choose a reason for hiding this comment

basvandijk Aug 20, 2018 • edited

Choose a reason for hiding this comment

shmish111 commented Aug 21, 2018

basvandijk commented Aug 21, 2018 • edited

basvandijk commented Aug 21, 2018

basvandijk commented Aug 25, 2018

GrahamcOfBorg commented Aug 25, 2018

GrahamcOfBorg commented Aug 25, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

shmish111 commented Aug 2, 2018 •

edited

basvandijk Aug 20, 2018 •

edited

basvandijk commented Aug 21, 2018 •

edited