Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

elasticsearch6: refactor plugins #44339

Closed
wants to merge 1 commit into from
Closed

elasticsearch6: refactor plugins #44339

wants to merge 1 commit into from

Conversation

shmish111
Copy link
Contributor

@shmish111 shmish111 commented Aug 2, 2018 •
edited

The plugins command didn't actually work with elasticsearch6 so
I have added a new derivation for elasticsearch6 plugins. This
also means there won't be problems with versions of plugins that
only work for ES5 and not ES6.

Unfortunately it is down to the user to use the elasticsearch6Plugins
packages when setting up an elasticsearch server but I couldn't
think of a way around this.

Currently there is only 1 plugin, the discovery ec2 plugin since
this is the one I am using.

Motivation for this change

elasticsearchPlugins doesn't work with ES 6

Things done
  • Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS)
  • Built on platform(s)
    • NixOS
    • macOS
    • other Linux distributions
  • Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
  • Tested compilation of all pkgs that depend on this change using nix-shell -p nox --run "nox-review wip"
  • Tested execution of all binary files (usually in ./result/bin/)
  • Determined the impact on package closure size (by running nix path-info -S before and after)
  • Fits CONTRIBUTING.md.

@shmish111
elasticsearch6: refactor plugins
80b11ae 
The plugins command didn't actually work with elasticsearch6 so
I have added a new derivation for elasticsearch6 plugins. This
also means there won't be problems with versions of plugins that
only work for ES5 and not ES6.

Unfortunately it is down to the user to use the elasticsearch6Plugins
packages when setting up an elasticsearch server but I couldn't
think of a way around this.

Currently there is only 1 plugin, the discovery ec2 plugin since
this is the one I am using.
basvandijk
basvandijk suggested changes Aug 3, 2018
View reviewed changes
pkgs/servers/search/elasticsearch/plugins-6.x.nix
discovery-ec2 = esPlugin {
name = "elasticsearch-discovery-ec2-${version}";
pluginName = "discovery-ec2";
version = "${elk6Version}";
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This can be simplified to version = elk6Version;.

@Mic92
Copy link
Member

Mic92 commented Aug 3, 2018 •
edited

I think we should drop elasticsearch 1.7 and elasticsearch2 before the next release. Both are end of life: https://www.elastic.co/support/eol

@basvandijk
Copy link
Member

@Mic92 agreed.

@Mic92 Mic92 mentioned this pull request Aug 3, 2018
Merged
9 tasks
@Mic92
Copy link
Member

Mic92 commented Aug 3, 2018

I added your plugin also in this pull request: #44389

@Mic92
Copy link
Member

Mic92 commented Aug 3, 2018

none of the elasticsearch plugins I tried work.
When I start the elasticsearch service, I get:

AccessControlException: access denied ("java.io.FilePermission" "/var/lib/elasticsearch/plugins/elasticsearch-analysis-lemmagen/meta-plugin-descriptor.properties" "read")

When I start elasticearch manually:

org.elasticsearch.bootstrap.StartupException: java.lang.IllegalStateException: Could not load plugin descriptor for plugin directory [commons-logging-1.1.3.jar]
        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:140) ~[elasticsearch-6.3.2.jar:6.3.2]
        at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:127) ~[elasticsearch-6.3.2.jar:6.3.2]
        at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) ~[elasticsearch-6.3.2.jar:6.3.2]
        at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) ~[elasticsearch-cli-6.3.2.jar:6.3.2]
        at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-cli-6.3.2.jar:6.3.2]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:93) ~[elasticsearch-6.3.2.jar:6.3.2]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:86) ~[elasticsearch-6.3.2.jar:6.3.2]
Caused by: java.lang.IllegalStateException: Could not load plugin descriptor for plugin directory [commons-logging-1.1.3.jar]
        at org.elasticsearch.plugins.PluginsService.readPluginBundle(PluginsService.java:487) ~[elasticsearch-6.3.2.jar:6.3.2]
        at org.elasticsearch.plugins.PluginsService.findBundles(PluginsService.java:469) ~[elasticsearch-6.3.2.jar:6.3.2]
        at org.elasticsearch.plugins.PluginsService.<init>(PluginsService.java:147) ~[elasticsearch-6.3.2.jar:6.3.2]
        at org.elasticsearch.node.Node.<init>(Node.java:311) ~[elasticsearch-6.3.2.jar:6.3.2]
        at org.elasticsearch.node.Node.<init>(Node.java:252) ~[elasticsearch-6.3.2.jar:6.3.2]
        at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:213) ~[elasticsearch-6.3.2.jar:6.3.2]
        at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:213) ~[elasticsearch-6.3.2.jar:6.3.2]
        at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:326) ~[elasticsearch-6.3.2.jar:6.3.2]
        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:136) ~[elasticsearch-6.3.2.jar:6.3.2]
        ... 6 more
Caused by: java.nio.file.FileSystemException: /var/lib/elasticsearch/plugins/discovery-ec2/commons-logging-1.1.3.jar/plugin-descriptor.properties: Not a directory
        at sun.nio.fs.UnixException.translateToIOException(UnixException.java:91) ~[?:?]
        at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102) ~[?:?]
        at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107) ~[?:?]
        at sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:214) ~[?:?]
        at java.nio.file.Files.newByteChannel(Files.java:361) ~[?:1.8.0_181]
        at java.nio.file.Files.newByteChannel(Files.java:407) ~[?:1.8.0_181]
        at java.nio.file.spi.FileSystemProvider.newInputStream(FileSystemProvider.java:384) ~[?:1.8.0_181]
        at java.nio.file.Files.newInputStream(Files.java:152) ~[?:1.8.0_181]
        at org.elasticsearch.plugins.PluginInfo.readFromProperties(PluginInfo.java:163) ~[elasticsearch-6.3.2.jar:6.3.2]
        at org.elasticsearch.plugins.PluginsService.readPluginBundle(PluginsService.java:484) ~[elasticsearch-6.3.2.jar:6.3.2]
        at org.elasticsearch.plugins.PluginsService.findBundles(PluginsService.java:469) ~[elasticsearch-6.3.2.jar:6.3.2]
        at org.elasticsearch.plugins.PluginsService.<init>(PluginsService.java:147) ~[elasticsearch-6.3.2.jar:6.3.2]
        at org.elasticsearch.node.Node.<init>(Node.java:311) ~[elasticsearch-6.3.2.jar:6.3.2]
        at org.elasticsearch.node.Node.<init>(Node.java:252) ~[elasticsearch-6.3.2.jar:6.3.2]
        at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:213) ~[elasticsearch-6.3.2.jar:6.3.2]
        at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:213) ~[elasticsearch-6.3.2.jar:6.3.2]
        at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:326) ~[elasticsearch-6.3.2.jar:6.3.2]
        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:136) ~[elasticsearch-6.3.2.jar:6.3.2]
        ... 6 more

Any idea what is missing?

@shmish111
Copy link
Contributor Author

@Mic92 does 44389 make my PR obsolete? If so I am happy to close and will keep an eye on 44389.

I am using this fork in production using master branch of nixpkgs for the elasticsearch package. Did you get that error using this fork?

@Mic92
Copy link
Member

Mic92 commented Aug 15, 2018 •
edited

I got the error in my pull request, which includes your changes from this pull request. Do I miss anything? https://github.com/NixOS/nixpkgs/blob/3f889f30bee92c3f2d5eaf53e56aef68f61efc4a/pkgs/servers/search/elasticsearch/plugins.nix

@shmish111
Copy link
Contributor Author

@Mic92 sorry for the delay in answering, what version of elasticsearch are you using for this? I am using the elasticsearch6-oss package in my fork.

@shmish111
Copy link
Contributor Author

@Mic92 perhaps its this https://github.com/NixOS/nixpkgs/pull/44389/files#diff-2d12e9f116784c475c19d55f07fee064L22

Previously when using es6 your if statement was not using the es5 branch so the security manager was disabled, now it is not.

@shmish111
Copy link
Contributor Author

@Mic92 can this PR be closed now?

@shmish111
Copy link
Contributor Author

To answer my own question, I have tried master branch and it works for me now so closing this PR. Thanks for the good work @Mic92 !

@shmish111 shmish111 closed this Sep 6, 2018
@flokli flokli mentioned this pull request Apr 8, 2019
Closed
8 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@basvandijk basvandijk basvandijk requested changes

Assignees
No one assigned
Labels
8.has: package (new) 10.rebuild-darwin: 1-10 10.rebuild-linux: 1-10 12. first-time contribution
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
5 participants
@shmish111 @Mic92 @basvandijk @GrahamcOfBorg @Janik-Haag