Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

rake fails to activate properly under bundle exec #3888

Closed
headius opened this issue May 12, 2016 · 17 comments
Closed

rake fails to activate properly under bundle exec #3888

headius opened this issue May 12, 2016 · 17 comments
Assignees

Comments

@headius
Copy link
Member

headius commented May 12, 2016

Forked from #3852 by @meghalidhoble.

From comment #3852 (comment)

Today I was trying to run the tests with jruby version 9.1.0.0; and here I could not reproduce the above issue; now I am getting different problem the command "rake test" is failing with weird error for rake gem.

[logstash]# bundle exec rake test
Could not find rake-11.1.2 in any of the sources
Run `bundle install` to install missing gems.
[logstash]#
[logstash]# rake test
Could not find rake-11.1.2 in any of the sources
Run `bundle install` to install missing gems.
[ logstash]#
[logstash]# bundle list |grep rake
  * rake (11.1.2)
[logstash]#

This only appears to affect JRuby 9k and probably relates to #3850 and #3867.

@headius
Copy link
Member Author

headius commented May 12, 2016

@meghalidhoble Can you provide info on how we can reproduce this? Environment (OS, JRuby version, etc) and step-by-step instructions please :-)

@meghalidhoble
Copy link

Yes sure, @headius!
OS : Rhel7.2
Jruby version : 9.1.0.0 (using RVM install also tried with direct tarball download for jruby-9.0.0.0)

Steps:

  1. Downloaded the source code (git clone) of the target application (logstash - https://github.com/elastic/logstash)
  2. Install Jruby using rvm install jruby (default it installs latest)
  3. gem install bundler - works fine
  4. bundle install inside the source code - works fine
  5. rake test:install-core (any of the rake command fails with below error)
[root@pts00433-vm3 logstash]# bundle exec rake test:install-core
Could not find rake-11.1.2 in any of the sources
Run `bundle install` to install missing gems.
[root@pts00433-vm3 logstash]# 

[root@pts00433-vm3 logstash]# gem list rake
*** LOCAL GEMS ***
rake (11.1.2, default: 10.4.2)

[root@pts00433-vm3 logstash]# gem uninstall rake -aIx
ERROR:  While executing gem ... (Gem::InstallError)
    gem "rake" cannot be uninstalled because it is a default gem
[root@pts00433-vm3 logstash]# irb

NOTE: I found that the multiple versions of rake were giving problem so tried removing the default one failed.
Bundler version seen: bundler-1.9.10

Thanks,
Meghali

@headius
Copy link
Member Author

headius commented May 13, 2016

Hmm...I was unable to reproduce either inside or outside rvm:

$ rake test:install-core
mkdir -p vendor
mkdir vendor/_
Downloading http://jruby.org.s3.amazonaws.com/downloads/1.7.25/jruby-bin-1.7.25.tar.gz
13.97%^C

There must be something different about your environment. Can you see anything in ENV that might be affecting its ability to locate gems?

@headius
Copy link
Member Author

headius commented May 13, 2016

No love with bundle exec either :-(

(This is under rvm JRuby 9.0.5.0)

$ bundle exec rake test:install-core
Downloading http://jruby.org.s3.amazonaws.com/downloads/1.7.25/jruby-bin-1.7.25.tar.gz
12.48%

@headius
Copy link
Member Author

headius commented May 13, 2016

Ah-ha...I did not let it run long enough. It did error after the download.

So it seems the problem is similar to the other cases, where a re-launch of JRuby is not able to find libraries that are obviously installed. I will investigate today.

@headius headius self-assigned this May 13, 2016
@meghalidhoble
Copy link

Alright @headius, thanks!

@headius
Copy link
Member Author

headius commented May 13, 2016

Taking notes as I go...seems that it's the bootstrap task that conks out, so basically everything that depends on that (which seems to be almost everything) also fails.

And only under bundler.

[] ~/projects/jruby/logstash $ rake bootstrap
Installing minitar >= 0 because the build process needs it.
mkdir -p build
mkdir -p build/bootstrap
[bootstrap] Fetching and installing gem: minitar (>= 0)
Fetching: minitar-0.5.4.gem (100%)
Successfully installed minitar-0.5.4
^C
[] ~/projects/jruby/logstash $ bundle exec rake bootstrap
Could not find rake-11.1.2 in any of the sources
Run `bundle install` to install missing gems.

@mkristian This isn't in an embedded scenario, but is it the same bundler issue again?

@headius
Copy link
Member Author

headius commented May 13, 2016

Ok, at least part of the problem here seems to be that bundler is adjusting GEM_HOME to point at .../logstash/vendor/jruby/2.3.0...but there's nothing in that directory for me. Little wonder if can't find any gems...

@headius
Copy link
Member Author

headius commented May 13, 2016

Here's what I've been able to figure out.

With the instructions you provide, JRuby 9k does indeed fail. It appears to be a relaunch of JRuby that can't find gems.

Investigating the error itself, I found that at that point the env var GEM_HOME points at a directory with no gems in it...and nothing else.

On a hunch I switched your build to vendoring JRuby 9.1 and did a bundle install --path vendor/bundle.

$ bundle install --path vendor/bundle
Fetching gem metadata from https://rubygems.org/
Fetching version metadata from https://rubygems.org/
Fetching dependency metadata from https://rubygems.org/
Resolving dependencies.....
Installing rake 11.1.2
Installing addressable 2.3.8
Installing cabin 0.8.1
...
Installing logstash-devutils 1.0.1
Bundle complete! 18 Gemfile dependencies, 72 gems now installed.
Bundled gems are installed into ./vendor/bundle.
Post-install message from jar-dependencies:
...

[] ~/projects/jruby/logstash $ bundle exec rake bootstrap
Installing bundler ~> 1.9.4 because the build process needs it.
[bootstrap] Fetching and installing gem: bundler (~> 1.9.4)
Fetching: bundler-1.9.10.gem (100%)
Successfully installed bundler-1.9.10
1 gem installed
Invoking bundler install...
rake aborted!
NoMethodError: undefined method `each_pair' for #<String:0x33845e3d>
/Users/headius/projects/jruby/logstash/lib/bootstrap/bundler.rb:89:in `invoke!'
/Users/headius/projects/jruby/logstash/rakelib/vendor.rake:124:in `block in (root)'
Tasks: TOP => bootstrap => vendor:all => vendor:gems
(See full trace by running task with --trace)

The rake tasks that failed before now proceed correctly! And then something different fails inside Logstash's bundler patches!

Patches, you say?

I discovered a number of patches and tweaks to Bundler in your bundler.rb that appear to be doing all sorts of "interesting" things with paths and workflows.

At this point I'm a bit out of my depth and I don't know how to proceed. I'm not sure what all logstash is patching, or why, or what impact and interaction that has with JRuby 9.1, Ruby 2.3 stdlib, and RubyGems. Is there someone at Elastic that's familiar with all this unusual build-munging you're doing? I think we need you folks to do a bit more probing into these patches to make sure they're not part of the problem.

@emmanuellyautomated
Copy link

@headius, yeah I've been battling this issue as well. Will report back when I have a proper solution.

@emmanuellyautomated
Copy link

The issue outlined by @headius, seems to be caused by the USE_RUBY=1 flag.

@headius
Copy link
Member Author

headius commented Oct 5, 2016

@emmanuellyautomated I don't know what the USE_RUBY=1 does, but it sounds like it may be scrubbing out some env we need to properly relaunch.

@emmanuellyautomated
Copy link

So when using the system version of jruby, I saw that the GEM_PATH had two entries in it. This may have confused bundler somehow. Using one path (the one at ~/.rbenv/versions/<jruby>/lib/ruby/gems/shared) seemed to make the issue go away.

NOTE: by doing this, things worked with or without the USE_RUBY flag.

@guyboertje
Copy link

@headius, @emmanuellyautomated

I am looking into this as part of the official push to move Logstash from 1.7.25 to 9.1.6.0

@guyboertje
Copy link

Found the problem in Logstash. This issue can be closed.

the fault is in logstash/lib/bootstrap/bundler.rb:89

::Gem.paths = ENV['GEM_HOME'] = ENV['GEM_PATH'] = LogStash::Environment.logstash_gem_home

This sends a string to #paths=(env) instead of a hash and a quirk in 1.7.X in jruby/lib/ruby/shared/rubygems/path_support.rb:28 masks the string not hash bug.

In 9K the #paths=(env) iterates over the env hash using #each_pair.

In Logstash this can be fixed with:

ENV['GEM_HOME'] = ENV['GEM_PATH'] = LogStash::Environment.logstash_gem_home
::Gem.paths = ENV

@guyboertje
Copy link

PS. We do a lot of bundler and rubygems patching to support a fully vendored turnkey jruby application deployable to machines without internet access.

@enebo
Copy link
Member

enebo commented Jan 5, 2017

@guyboertje thanks for the update.

@enebo enebo closed this as completed Jan 5, 2017
@enebo enebo added this to the Invalid or Duplicate milestone Jan 5, 2017
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

5 participants