@apeschar, thanks for your PR! By analyzing the history of the files in this pull request, we identified @offlinehacker, @bjornfor and @rickynils to be potential reviewers.

Is it reasonable to continue packaging the older ES's?

I guess so:

• ES 2.4 is still very popular.
• Due to backwards incompatibilities some packages (such as Graylog2) won't work with ES 5.x.

I don't know anything about ES 1.x. Maybe drop it, if there's no dependencies?

Does @offlinehacker have an opinion on this?

I doubt people are still using 1.x much, but keeping 2.x is reasonable.

Removing 1.x seems to break the elasticsearchPlugins build. This is a mess anyway, as some plugins in there are for 1.x, and some for 2.x, but they all use the 1.x code for building.

So I suggest doing this in a separate PR, and seeing if we can merge the ES 5 addition now.

I don't have a chance to test this right now but LGTM in principle

cc @fadenb who was working on this, too

Looks good to me. I can test this properly on Monday.

Did you test whether the current plugin logic of the module is working? I only got it to look for modules relative to the nix store path with my code.

When starting the service with package elasticsearch5 it fails after a brief moment:
max file descriptors [4096] for elasticsearch process is too low, increase to at least [65536]
Putting a LimitNOFILE = "1024000"; into the serviceConfig fixes that issue. I believe that this is also safe for elastic 2.x.

Another issues that comes up is: max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]

@apeschar With minor modifications your service is working fine for me. Take a look at https://gist.github.com/fadenb/dd01c673edf59099ed08cfaa0c2cbaf4

I'm also getting the "Too many open files" error on elasticsearch-2.4.4. It would be great to get the LimitNOFILE fix from @fadenb on release-17.03 as well.

Speaking of which: since this whole change is backwards compatible we could cherry-pick this on release-17.03. I would love to try out the latest elasticsearch on our production systems!

@fadenb, thanks! I added your changes to the pull request.

I haven't looked into the plugins-- it won't break anything for 2.x, but I'm not sure if installing plugins works for 5.x. So that's something worth testing.

I'm more or less away for a week or so, so I won't be able to test this until then...

I also upgraded kibana to 5.4 using:

{
  kibana = super.kibana.overrideAttrs (_oldAttrs: {
    src = super.fetchurl {
      url = "https://artifacts.elastic.co/downloads/kibana/kibana-5.4.0-linux-x86_64.tar.gz";
      sha256 = "1g5i81wq77fk6pyaq3rpfqs2m23xsbz2cndh3rg4b59ibg5qv0sq";
    };
  });
}

The web interface appears in my browser:

but kibana is getting timeouts on elasticsearch:

Is somebody able to replicate my kibana-5.4 issue together with elasticsearch-5.4?

I discovered a workaround for my kibana issue:

• The timeout occurs when kibana tries to add an index pattern. I see my browser (chrome) POSTing some JSON to: http://kibana.lumi.guide/es_admin/.kibana/index-pattern/mypattern-*'. For some reason that request times out.

• If I perform the same request via curl (in the chrome network inspector right click on the request and choose Copy->Copy as cURL) it succeeds instantly. After that my index pattern is created and kibana functions correctly.

Note that Elastic recommends keeping the whole ELK stack on the same version so in #26252 I upgraded logstash, kibana and the beats to 5.4.0.