Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

JRuby 1.7.25/26 uninitialized constant Aws::Client::Errors #3920

Closed
creddy opened this issue May 24, 2016 · 33 comments
Closed

JRuby 1.7.25/26 uninitialized constant Aws::Client::Errors #3920

creddy opened this issue May 24, 2016 · 33 comments

Comments

@creddy
Copy link

creddy commented May 24, 2016

This issue appears similar to #3645 except that this if for JRuby 1.7.25/26 vs 9.x

Environment

jruby -v
jruby 1.7.26-SNAPSHOT (1.9.3p551) 2016-05-24 5b83abb on Java HotSpot(TM) 64-Bit Server VM 1.8.0_74-b02 [darwin-x86

jruby 1.7.25 (1.9.3p551) 2016-04-13 867cb81 on Java HotSpot(TM) 64-Bit Server VM 1.8.0_74-b02 [darwin-x86_64]

uname -a
Darwin LAXM00618506A 15.4.0 Darwin Kernel Version 15.4.0: Fri Feb 26 22:08:05 PST 2016; root:xnu-3248.40.184~3/RELEASE_X86_64 x86_64

Expected Behavior

When a bucket object does not exist a Aws::S3::Errors::NoSuchKey error is raised

  ~ [jruby-1.7.23]  jruby -v
jruby 1.7.23 (1.9.3p551) 2015-11-24 f496dd5 on Java HotSpot(TM) 64-Bit Server VM 1.8.0_74-b02 [darwin-x86_64]
  ~ [jruby-1.7.23]  irb
irb(main):001:0> require "aws-sdk"
=> true
irb(main):002:0> require "securerandom"
=> true
irb(main):003:0>
irb(main):004:0* credentials = { region: "us-east-1" }
=> {:region=>"us-east-1"}
irb(main):005:0> bucket = "BUCKET_NAME"
=> "BUCKET_NAME"
irb(main):006:0> connection = Aws::S3::Resource.new(credentials)
=> #<Aws::S3::Resource>
irb(main):007:0> connection.bucket(bucket).object(SecureRandom.uuid).get(SecureRandom.uuid)
Aws::S3::Errors::NoSuchKey: The specified key does not exist

Actual Behavior

A NameError: uninitialized constant Aws::Client::Errors is raised

  jruby -v
jruby 1.7.26-SNAPSHOT (1.9.3p551) 2016-05-24 5b83abb on Java HotSpot(TM) 64-Bit Server VM 1.8.0_74-b02 [darwin-x86_64]
  irb
irb(main):001:0> require "aws-sdk"
=> true
irb(main):002:0> require "securerandom"
=> true
irb(main):003:0>
irb(main):004:0* credentials = { region: "us-east-1" }
=> {:region=>"us-east-1"}
irb(main):005:0> bucket = "BUCKET_NAME"
=> "BUCKET_NAME"
irb(main):006:0> connection = Aws::S3::Resource.new(credentials)
=> #<#<Module:0x2e185cd7>::Resource>
irb(main):007:0> connection.bucket(bucket).object(SecureRandom.uuid).get(SecureRandom.uuid)
NameError: uninitialized constant Aws::Client::Errors
 jruby -v
jruby 1.7.25 (1.9.3p551) 2016-04-13 867cb81 on Java HotSpot(TM) 64-Bit Server VM 1.8.0_74-b02 [darwin-x86_64]
  irb
irb(main):001:0> require "aws-sdk"
=> true
irb(main):002:0> require "securerandom"
=> true
irb(main):003:0>
irb(main):004:0* credentials = { region: "us-east-1" }
=> {:region=>"us-east-1"}
irb(main):005:0> bucket = "BUCKET_NAME"
=> "BUCKET_NAME"
irb(main):006:0> connection = Aws::S3::Resource.new(credentials)
=> #<#<Module:0x5851bd4f>::Resource>
irb(main):007:0> connection.bucket(bucket).object(SecureRandom.uuid).get(SecureRandom.uuid)
NameError: uninitialized constant Aws::Client::Errors

Thank you

@rtlong
Copy link

rtlong commented Jun 17, 2016

Only since no one else has said anything on this, I'll add that I get the same results -- jruby 1.7.25 (2.0.0p598) 2016-04-13 867cb81 on Java HotSpot(TM) 64-Bit Server VM 1.8.0_66-b17 [linux-amd64].

@eliaslevy
Copy link
Contributor

Just got bitten by this using JRuby 1.7.25 and AWS SDK gem 2.4.4.

@eliaslevy
Copy link
Contributor

The backtrace:

NameError: uninitialized constant Aws::Client::Errors
   const_missing at org/jruby/RubyModule.java:2719
       const_get at org/jruby/RubyModule.java:2638
           error at /opt/jruby-1.7.25/lib/ruby/gems/shared/gems/aws-sdk-core-2.4.4/lib/aws-sdk-core/xml/error_handler.rb:25
            call at /opt/jruby-1.7.25/lib/ruby/gems/shared/gems/aws-sdk-core-2.4.4/lib/aws-sdk-core/xml/error_handler.rb:9
            call at org/jruby/RubyProc.java:281
              on at /opt/jruby-1.7.25/lib/ruby/gems/shared/gems/aws-sdk-core-2.4.4/lib/seahorse/client/response.rb:43
      on_success at /opt/jruby-1.7.25/lib/ruby/gems/shared/gems/aws-sdk-core-2.4.4/lib/seahorse/client/http/response.rb:139
            call at org/jruby/RubyProc.java:281
        listener at /opt/jruby-1.7.25/lib/ruby/gems/shared/gems/aws-sdk-core-2.4.4/lib/seahorse/client/http/response.rb:166
            call at org/jruby/RubyProc.java:281
         on_done at /opt/jruby-1.7.25/lib/ruby/gems/shared/gems/aws-sdk-core-2.4.4/lib/seahorse/client/http/response.rb:130
      on_success at /opt/jruby-1.7.25/lib/ruby/gems/shared/gems/aws-sdk-core-2.4.4/lib/seahorse/client/http/response.rb:137
              on at /opt/jruby-1.7.25/lib/ruby/gems/shared/gems/aws-sdk-core-2.4.4/lib/seahorse/client/response.rb:42
            call at /opt/jruby-1.7.25/lib/ruby/gems/shared/gems/aws-sdk-core-2.4.4/lib/aws-sdk-core/xml/error_handler.rb:8
            call at /opt/jruby-1.7.25/lib/ruby/gems/shared/gems/aws-sdk-core-2.4.4/lib/aws-sdk-core/plugins/helpful_socket_errors.rb:10
            call at /opt/jruby-1.7.25/lib/ruby/gems/shared/gems/aws-sdk-core-2.4.4/lib/aws-sdk-core/plugins/s3_request_signer.rb:65
            call at /opt/jruby-1.7.25/lib/ruby/gems/shared/gems/aws-sdk-core-2.4.4/lib/aws-sdk-core/plugins/s3_redirects.rb:15
            call at /opt/jruby-1.7.25/lib/ruby/gems/shared/gems/aws-sdk-core-2.4.4/lib/aws-sdk-core/plugins/retry_errors.rb:87
            call at /opt/jruby-1.7.25/lib/ruby/gems/shared/gems/aws-sdk-core-2.4.4/lib/aws-sdk-core/plugins/s3_accelerate.rb:42
            call at /opt/jruby-1.7.25/lib/ruby/gems/shared/gems/aws-sdk-core-2.4.4/lib/aws-sdk-core/plugins/s3_md5s.rb:31
            call at /opt/jruby-1.7.25/lib/ruby/gems/shared/gems/aws-sdk-core-2.4.4/lib/aws-sdk-core/plugins/s3_expect_100_continue.rb:21
            call at /opt/jruby-1.7.25/lib/ruby/gems/shared/gems/aws-sdk-core-2.4.4/lib/aws-sdk-core/plugins/s3_bucket_name_restrictions.rb:12
            call at /opt/jruby-1.7.25/lib/ruby/gems/shared/gems/aws-sdk-core-2.4.4/lib/aws-sdk-core/plugins/s3_bucket_dns.rb:31
            call at /opt/jruby-1.7.25/lib/ruby/gems/shared/gems/aws-sdk-core-2.4.4/lib/aws-sdk-core/rest/handler.rb:7
            call at /opt/jruby-1.7.25/lib/ruby/gems/shared/gems/aws-sdk-core-2.4.4/lib/aws-sdk-core/plugins/user_agent.rb:12
            call at /opt/jruby-1.7.25/lib/ruby/gems/shared/gems/aws-sdk-core-2.4.4/lib/seahorse/client/plugins/endpoint.rb:41
            call at /opt/jruby-1.7.25/lib/ruby/gems/shared/gems/aws-sdk-core-2.4.4/lib/aws-sdk-core/plugins/param_validator.rb:21
            call at /opt/jruby-1.7.25/lib/ruby/gems/shared/gems/aws-sdk-core-2.4.4/lib/seahorse/client/plugins/raise_response_errors.rb:14
            call at /opt/jruby-1.7.25/lib/ruby/gems/shared/gems/aws-sdk-core-2.4.4/lib/aws-sdk-core/plugins/s3_sse_cpk.rb:19
            call at /opt/jruby-1.7.25/lib/ruby/gems/shared/gems/aws-sdk-core-2.4.4/lib/aws-sdk-core/plugins/s3_accelerate.rb:33
            call at /opt/jruby-1.7.25/lib/ruby/gems/shared/gems/aws-sdk-core-2.4.4/lib/aws-sdk-core/plugins/param_converter.rb:20
            call at /opt/jruby-1.7.25/lib/ruby/gems/shared/gems/aws-sdk-core-2.4.4/lib/seahorse/client/plugins/response_target.rb:21
    send_request at /opt/jruby-1.7.25/lib/ruby/gems/shared/gems/aws-sdk-core-2.4.4/lib/seahorse/client/request.rb:70
      get_object at /opt/jruby-1.7.25/lib/ruby/gems/shared/gems/aws-sdk-core-2.4.4/lib/seahorse/client/base.rb:207

@headius
Copy link
Member

headius commented Aug 19, 2016

Hmm, I had not seen this one come by.

@eliaslevy Thanks for the updated stack trace!

Poking around for a few minutes...

@headius
Copy link
Member

headius commented Aug 19, 2016

I suspect this is a threading issue with autoloads. The Errors module referenced is accessed in error_handler.rb as Aws.const_get(svc).const_get(:Errors), and defined in aws-sdk-core.rb as autoload :Errors, 'aws-sdk-core/waiters/errors'.

So there's two possibilities I see:

  1. The const_get is failing to trigger autoload when it should.
  2. Multiple threads are triggering the autoload at the same time, and one of them sees a partially-complete result.

The latter should be fixed, but it's possible thread-unsafeties have crept back into our loading.

@headius
Copy link
Member

headius commented Aug 19, 2016

Well, I'm surprised. I came up with a test that I expected to work in MRI, but it fails. Perhaps I'm doing something wrong here:

loop do
  class Foo
    if defined? Bar
      remove_const :Bar
    end
    autoload :Bar, 'bar.rb'
  end

  go = false
  threads = (1..100).map {Thread.new { 1 until go; Foo::Bar }}
  go = true
  threads.each(&:join)
end

Both JRuby and MRI eventually raise NameError for missing Foo::Bar. This is unexpected to me.

@headius
Copy link
Member

headius commented Aug 19, 2016

I filed https://bugs.ruby-lang.org/issues/12688 to clarify thread-safety here.

@eliaslevy
Copy link
Contributor

FWIW this occurs in a single threaded script.

@headius
Copy link
Member

headius commented Aug 24, 2016

@eliaslevy I will try your exact example again.

@headius
Copy link
Member

headius commented Aug 24, 2016

Well I can confirm this works fine in JRuby 9.1.3.0. I'll try 1.7 now.

irb(main):013:0> connection = Aws::S3::Resource.new(region: 'us-east-1', access_key_id: 'XXX', secret_access_key: 'XXX')
=> #<Aws::S3::Resource>
irb(main):014:0> connection.bucket(bucket).object(SecureRandom.uuid).get(SecureRandom.uuid)
Aws::S3::Errors::NoSuchKey: The specified key does not exist.

@headius
Copy link
Member

headius commented Aug 24, 2016

Confirmed broken still on 1.7.25. I guess it wasn't a fluke!

@headius headius added this to the JRuby 1.7.26 milestone Aug 24, 2016
@headius
Copy link
Member

headius commented Aug 24, 2016

After a little digging, I found that the code in question was updated by @kares for 9k in b216c5d. The changes made look like the right sort of changes for autoloading to fire properly, so I'm going to see about backporting it.

@headius
Copy link
Member

headius commented Aug 24, 2016

Yay! It works!

irb(main):001:0> require 'aws-sdk'
r=> true
irb(main):002:0> require 'securerandom'
=> true
irb(main):003:0> connection = Aws::S3::Resource.new(region: 'us-east-1', access_key_id: 'XXX', secret_access_key: 'XXX')
=> #<Aws::S3::Resource>
irb(main):004:0> connection.bucket('jrubylogs').object(SecureRandom.uuid).get(SecureRandom.uuid)
Aws::S3::Errors::NoSuchKey: The specified key does not exist.
    from /Users/headius/projects/jruby-1.7/lib/ruby/gems/shared/gems/aws-sdk-core-2.5.6/lib/seahorse/client/plugins/raise_response_errors.rb:15:in `call'

I will verify the patch and try to find tests for it. @kares do you remember if you added or enabled any tests for b216c5d?

@headius
Copy link
Member

headius commented Aug 24, 2016

@kares I started looking in the log neighborhood around b216c5d and realized there's a series of commits all related to autoload. Maybe we should backport all of them?

@kares
Copy link
Member

kares commented Aug 24, 2016

@headius I am not sure - it still isn't 100% good on 9K ... would put in the bare minimal that resolves this.
or maybe all but there might be a few conflicts to deal with - which increases the risk of doing smt wrong.

@headius
Copy link
Member

headius commented Aug 24, 2016

@kares I went ahead with it since it seemed to pass tests AND it fixes this issue.

@creddy Obviously our best recommendation would be to move to 9k :-)

@headius
Copy link
Member

headius commented Aug 24, 2016

Damn...it breaks another spec. https://travis-ci.org/jruby/jruby/builds/154864874

We will have to take a longer look at this. I will revert for now.

@JasonLunn
Copy link
Contributor

Bump

@headius
Copy link
Member

headius commented Sep 22, 2016

@JasonLunn Bumping because you need this fixed in JRuby 1.7?

@JasonLunn
Copy link
Contributor

This issue definitely impacts our existing 1.7.x installs in production. While we have a 9K transition in the works, it is not complete. If this issue isn't going to get resolved in 1.7.x and I won't get a fix until I can upgrade to 9K, that is useful information too. In the meantime, with the issue open and slated for 1.7.27 release, I wanted to highlight the weeks of inactivity because of the popularity of the AWS-SDK.

@headius
Copy link
Member

headius commented Sep 22, 2016

@JasonLunn Ok, thanks for clarifying that. We would like to fix it. The process involves reviewing and backporting a series of autoload improvements done by @kares and others. Just haven't gotten around to it :-\

@JasonLunn
Copy link
Contributor

Understood

@JasonLunn
Copy link
Contributor

JasonLunn commented Oct 14, 2016

Bump - any closer to clarifying if the 1.7.x series will see a backport of this fix?

@headius
Copy link
Member

headius commented Oct 18, 2016

@JasonLunn We do still want to backport this fix, but with 1.7 EOL at end of year our priorities are leaning toward 9k work.

@grddev
Copy link
Contributor

grddev commented Nov 24, 2016

Ran into seemingly the same issue today, with a NameError: uninitialized constant Aws::Client::Errors, but in my experiments, it seems this is not the original error, but an error caused by the fact that the Aws::S3 module has the wrong name (#<Module:0x...> instead of Aws::S3). After bisecting it seems like 15d5c20 was the cause of that regression.

In my testing it doesn't really seem like a race condition, as I was able to reproduce the problem (seemingly) deterministically with the following snippet:

autoload(:C, 'whatever')
p Object.const_set(:C, Module.new)

Which prints #<Module:0x...> since the above commit, and printed C before. If I remove the autoload statement, which in my case doesn't even autoload, the code prints C in all versions of JRuby I have tested.

A workaround is to require aws-sdk like so:

module Aws; module S3; end; end
require 'aws-sdk'
require 'aws-sdk-core/s3'

But this obviously gets a bit tedious if you use several AWS services.

EDIT: Added second require line, as it doesn't actually load S3 otherwise.

@kares
Copy link
Member

kares commented Nov 24, 2016

@grddev are you running multiple threads by any chance? if not so, can we easily reproduce the isssue?

@grddev
Copy link
Contributor

grddev commented Nov 24, 2016

@kares if it was unclear, I only need the two lines above to reproduce the issue.

$ RBENV_VERSION=jruby-1.7.26 ruby -e 'autoload(:C, "whatever"); p Object.const_set(:C, Module.new)'
#<Module:0x1324409e>
$ RBENV_VERSION=jruby-1.7.23 ruby -e 'autoload(:C, "whatever"); p Object.const_set(:C, Module.new)'
C

@enebo
Copy link
Member

enebo commented Apr 24, 2017

Ok I reverted the revert 02c988a with the only change as fixing const_defined? to be its original 1.9 impl. I believe the issue was that when we back-ported (or tried to) we left in the 2.x semantics which allows :: in const_defined?.

@headius can you review this.

I committed this because I was hoping someone @grddev @joshuaspence @JasonLunn @creddy @eliaslevy or @rtlong can try the original AWS use case and say whether this is fixed and perhaps the logstash guys can try things out with the broader commit I just landed: 45047a7.

I have not actually see this go green yet but I am at the end of the work day and extremely optimistic :)

@enebo
Copy link
Member

enebo commented Apr 24, 2017

DOH...bizarrely this is not working...False alarm. At least @grddev reduced case is not working. I am confused now why this was working before the revert now...

elasticsearch-bot pushed a commit to logstash-plugins/logstash-input-sqs that referenced this issue May 4, 2017
I have occassionally seen the folling error in our Logstash error logs:

```
{"level":"ERROR","loggerName":"logstash.pipeline","timeMillis":1483972688181,"thread":"[main]<sqs","logEvent":{"message":"A plugin had an unrecoverable error. Will restart this plugin.\n  Plugin: <LogStash::Inputs::SQS queue=>\"logstash\", region=>\"us-east-1\", threads=>4, id=>\"6ea8b94e902199fe214a76edafcbc1c24eee708e-3\", enable_metric=>true, codec=><LogStash::Codecs::JSON id=>\"json_d46ee53c-b664-49c2-a64f-c86a0d5b9599\", enable_metric=>true, charset=>\"UTF-8\">, polling_frequency=>20>\n  Error: uninitialized constant Aws::Client::Errors"}}
```

This pull request applies to same monkey-patch as was done in logstash-plugins/logstash-output-sqs#20 in order to workaround jruby/jruby#3920.

Fixes #35
@headius
Copy link
Member

headius commented May 5, 2017

@enebo So are we still targeting this for 1.7.27? As far as I know we don't have a working patch.

@enebo
Copy link
Member

enebo commented May 5, 2017

@headius well it was a hope but I don't understand what is wrong so we might have to just accept we will never get this one on 1.7.x.

@headius headius removed this from the JRuby 1.7.27 milestone May 5, 2017
@enebo
Copy link
Member

enebo commented May 16, 2017

1.7.x is EOL and this will not get fixed. Marking WONTFIX

@enebo enebo closed this as completed May 16, 2017
@enebo enebo added this to the Won't Fix milestone May 16, 2017
@seanguo
Copy link

seanguo commented Aug 4, 2017

@headius We had a similar issue after updated Jruby from 1.7.21 to 1.7.27. It happened when we do
require 'json'
The stacktrace is as following:
org.jruby.embed.EvalFailedException: (NameError) uninitialized constant JSON::Ext::Parser with cause: org.jruby.embed.EvalFailedException: (NameError) uninitialized constant JSON::Ext::Parser@T: javax.script.ScriptException: org.jruby.embed.EvalFailedException: (NameError) uninitialized constant JSON::Ext::Parser at org.jruby.embed.jsr223.JRubyEngine.wrapException(JRubyEngine.java:104) at org.jruby.embed.jsr223.JRubyEngine.eval(JRubyEngine.java:93) at org.jruby.embed.jsr223.JRubyEngine.eval(JRubyEngine.java:142) ... Caused by: org.jruby.embed.EvalFailedException: (NameError) uninitialized constant JSON::Ext::Parser at org.jruby.embed.internal.EmbedEvalUnitImpl.run(EmbedEvalUnitImpl.java:132) at org.jruby.embed.jsr223.JRubyEngine.eval(JRubyEngine.java:90) ... 7 more Caused by: org.jruby.exceptions.RaiseException: (NameError) uninitialized constant JSON::Ext::Parser at org.jruby.RubyModule.const_missing(org/jruby/RubyModule.java:2746) at RUBY.Ext(/jruby/lib/ruby/shared/json/ext.rb:16) at RUBY.JSON(.../jruby/lib/ruby/shared/json/ext.rb:12) at RUBY.(root)(.../jruby/lib/ruby/shared/json/ext.rb:9) at org.jruby.RubyKernel.require(org/jruby/RubyKernel.java:1040) at RUBY.(root)(.../jruby/lib/ruby/shared/rubygems/core_ext/kernel_require.rb:1) at RUBY.require(.../jruby/lib/ruby/shared/rubygems/core_ext/kernel_require.rb:54) at RUBY.JSON(.../jruby/lib/ruby/shared/json.rb:58) at org.jruby.RubyKernel.require(org/jruby/RubyKernel.java:1040) at RUBY.(root)(.../jruby/lib/ruby/shared/json.rb:54) at RUBY.(root)(.../jruby/lib/ruby/shared/rubygems/core_ext/kernel_require.rb:1)

But it's not always reproducible. Is this related to this issue?

prateekbaheti pushed a commit to mingle/elasticsearch-opsworks that referenced this issue Aug 9, 2017
…rmenn>

- Updating jruby to latest, multithreading issue in the old version would cause sdk failures
  jruby/jruby#3920
- Updating aws-sdk gem
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

9 participants