Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Lazy load Digest::SHA256 is not thread safe #1279

Closed
vinayshastry opened this issue Nov 27, 2013 · 6 comments
Closed

Lazy load Digest::SHA256 is not thread safe #1279

vinayshastry opened this issue Nov 27, 2013 · 6 comments

Comments

@vinayshastry
Copy link

vinayshastry commented Nov 27, 2013

Hi,

Recently we observed that our production log has load error for Digest::SHA256 resulting in,

NotImplementedError: the Digest::SHA256() function is unimplemented on this machine

This is reproducible on master with the following change,

  1. Add sleep into RubyDigest so that it takes a while to assign metadata,
    on 413f5dd, at line 228 of RubyDigest, insert snippet,
try {
            System.out.println("sleep 10 seonds before instance v set");
            Thread.sleep(10000);
        } catch (InterruptedException e) {
            e.printStackTrace();  //To change body of catch statement use File | Settings | File Templates.
        }
  1. Run the following ruby script,
require 'digest'
threads = []
100.times do
  threads << (Thread.new do
    begin
      Digest::SHA256.new
    rescue Exception => e
      puts "[DEBUG]e => #{e.inspect}"
    end
  end)
end
threads.each(&:join)

I have attached an image of the RubyDigest with sleep code.

screen shot 2013-11-27 at 2 15 59 pm

Thanks,
Vinay.

@tmornini
Copy link

I'm seeing this too on my project...

NotImplementedError: the Digest::SHA256() function is unimplemented on this machine
allocate at org/jruby/RubyClass.java:224
hexdigest at /Users/tmornini/.rbenv/versions/jruby-1.7.10/lib/ruby/gems/shared/gems/aws-sdk-1.32.0/lib/aws/core/signers/version_4.rb:195
body_digest at /Users/tmornini/.rbenv/versions/jruby-1.7.10/lib/ruby/gems/shared/gems/aws-sdk-1.32.0/lib/aws/core/signers/version_4.rb:188
sign_request at /Users/tmornini/.rbenv/versions/jruby-1.7.10/lib/ruby/gems/shared/gems/aws-sdk-1.32.0/lib/aws/core/signers/version_4.rb:61
sign_request at /Users/tmornini/.rbenv/versions/jruby-1.7.10/lib/ruby/gems/shared/gems/aws-sdk-1.32.0/lib/aws/core/client.rb:699
client_request at /Users/tmornini/.rbenv/versions/jruby-1.7.10/lib/ruby/gems/shared/gems/aws-sdk-1.32.0/lib/aws/core/client.rb:485
call at org/jruby/RubyProc.java:271
build_request at /Users/tmornini/.rbenv/versions/jruby-1.7.10/lib/ruby/gems/shared/gems/aws-sdk-1.32.0/lib/aws/core/response.rb:171
initialize at /Users/tmornini/.rbenv/versions/jruby-1.7.10/lib/ruby/gems/shared/gems/aws-sdk-1.32.0/lib/aws/core/response.rb:111
new_response at /Users/tmornini/.rbenv/versions/jruby-1.7.10/lib/ruby/gems/shared/gems/aws-sdk-1.32.0/lib/aws/core/client.rb:197
client_request at /Users/tmornini/.rbenv/versions/jruby-1.7.10/lib/ruby/gems/shared/gems/aws-sdk-1.32.0/lib/aws/core/client.rb:483
log_client_request at /Users/tmornini/.rbenv/versions/jruby-1.7.10/lib/ruby/gems/shared/gems/aws-sdk-1.32.0/lib/aws/core/client.rb:384
client_request at /Users/tmornini/.rbenv/versions/jruby-1.7.10/lib/ruby/gems/shared/gems/aws-sdk-1.32.0/lib/aws/core/client.rb:470
return_or_raise at /Users/tmornini/.rbenv/versions/jruby-1.7.10/lib/ruby/gems/shared/gems/aws-sdk-1.32.0/lib/aws/core/client.rb:366
client_request at /Users/tmornini/.rbenv/versions/jruby-1.7.10/lib/ruby/gems/shared/gems/aws-sdk-1.32.0/lib/aws/core/client.rb:469
list_queues at (eval):3
_each_item at /Users/tmornini/.rbenv/versions/jruby-1.7.10/lib/ruby/gems/shared/gems/aws-sdk-1.32.0/lib/aws/sqs/queue_collection.rb:171
_each_batch at /Users/tmornini/.rbenv/versions/jruby-1.7.10/lib/ruby/gems/shared/gems/aws-sdk-1.32.0/lib/aws/core/collection/simple.rb:51
each_batch at /Users/tmornini/.rbenv/versions/jruby-1.7.10/lib/ruby/gems/shared/gems/aws-sdk-1.32.0/lib/aws/core/collection.rb:80
each at /Users/tmornini/.rbenv/versions/jruby-1.7.10/lib/ruby/gems/shared/gems/aws-sdk-1.32.0/lib/aws/core/collection.rb:47
delete at dev/scrub_queues:57
public_send at org/jruby/RubyKernel.java:1963
dispatch at /Users/tmornini/.rbenv/versions/jruby-1.7.10/lib/ruby/gems/shared/gems/celluloid-0.15.2/lib/celluloid/calls.rb:25
dispatch at /Users/tmornini/.rbenv/versions/jruby-1.7.10/lib/ruby/gems/shared/gems/celluloid-0.15.2/lib/celluloid/calls.rb:67
handle_message at /Users/tmornini/.rbenv/versions/jruby-1.7.10/lib/ruby/gems/shared/gems/celluloid-0.15.2/lib/celluloid/actor.rb:322
task at /Users/tmornini/.rbenv/versions/jruby-1.7.10/lib/ruby/gems/shared/gems/celluloid-0.15.2/lib/celluloid/actor.rb:416
initialize at /Users/tmornini/.rbenv/versions/jruby-1.7.10/lib/ruby/gems/shared/gems/celluloid-0.15.2/lib/celluloid/tasks.rb:55
initialize at /Users/tmornini/.rbenv/versions/jruby-1.7.10/lib/ruby/gems/shared/gems/celluloid-0.15.2/lib/celluloid/tasks.rb:47
create at /Users/tmornini/.rbenv/versions/jruby-1.7.10/lib/ruby/gems/shared/gems/celluloid-0.15.2/lib/celluloid/tasks/task_fiber.rb:13

@tmornini
Copy link

Solved via:

require 'digest'
require 'digest/sha2'

:-)

@headius
Copy link
Member

headius commented May 11, 2016

Still an issue, but appears to be differently broken:

$ jruby blah.rb
sleep 10 seonds before instance v set
[DEBUG]e => #<NameError: uninitialized constant Digest::SHA256>[DEBUG]e => #<NameError: uninitialized constant Digest::SHA256>[DEBUG]e => #<NameError: uninitialized constant Digest::SHA256>[DEBUG]e => #<NameError: uninitialized constant Digest::SHA256>
[DEBUG]e => #<NameError: uninitialized constant Digest::SHA256>
[DEBUG]e => #<NameError: uninitialized constant Digest::SHA256>
[DEBUG]e => #<NameError: uninitialized constant Digest::SHA256>[DEBUG]e => #<NameError: uninitialized constant Digest::SHA256>




[DEBUG]e => #<NameError: uninitialized constant Digest::SHA256>
[DEBUG]e => #<NameError: uninitialized constant Digest::SHA256>
[DEBUG]e => #<NameError: uninitialized constant Digest::SHA256>
[DEBUG]e => #<NameError: uninitialized constant Digest::SHA256>
[DEBUG]e => #<NameError: uninitialized constant Digest::SHA256>
[DEBUG]e => #<NameError: uninitialized constant Digest::SHA256>[DEBUG]e => #<NameError: uninitialized constant Digest::SHA256>

[DEBUG]e => #<NameError: uninitialized constant Digest::SHA256>
[DEBUG]e => #<NameError: uninitialized constant Digest::SHA256>
[DEBUG]e => #<NameError: uninitialized constant Digest::SHA256>

The following patch fixes it but I have no investigated whether there's a bug (e.g. in our locking multi-thread require) we should fix.

diff --git a/lib/ruby/stdlib/digest.rb b/lib/ruby/stdlib/digest.rb
index d6daf36..2625f31 100644
--- a/lib/ruby/stdlib/digest.rb
+++ b/lib/ruby/stdlib/digest.rb
@@ -6,22 +6,24 @@ module Digest
   REQUIRE_MUTEX = Mutex.new

   def self.const_missing(name) # :nodoc:
-    case name
-    when :SHA256, :SHA384, :SHA512
-      lib = 'digest/sha2.so'
-    else
-      lib = File.join('digest', name.to_s.downcase)
-    end
+    Digest::REQUIRE_MUTEX.synchronize do
+      case name
+      when :SHA256, :SHA384, :SHA512
+        lib = 'digest/sha2.so'
+      else
+        lib = File.join('digest', name.to_s.downcase)
+      end

-    begin
-      require lib
-    rescue LoadError
-      raise LoadError, "library not found for class Digest::#{name} -- #{lib}", caller(1)
-    end
-    unless Digest.const_defined?(name)
-      raise NameError, "uninitialized constant Digest::#{name}", caller(1)
+      begin
+        require lib
+      rescue LoadError
+        raise LoadError, "library not found for class Digest::#{name} -- #{lib}", caller(1)
+      end
+      unless Digest.const_defined?(name)
+        raise NameError, "uninitialized constant Digest::#{name}", caller(1)
+      end
+      Digest.const_get(name)
     end
-    Digest.const_get(name)
   end

   class ::Digest::Class

@headius
Copy link
Member

headius commented Jun 9, 2016

Fixed for 9k and 1.7.

headius added a commit to jruby/ruby that referenced this issue Jun 9, 2016
headius added a commit to jruby/ruby that referenced this issue Jun 9, 2016
kares added a commit that referenced this issue Jun 30, 2016
needed since d5f0387

... additional part for the #1279 fix (on JRuby 1.7)
maxjacobson added a commit to codeclimate/codeclimate-duplication that referenced this issue Nov 2, 2016
This is motivated by a rare error which was fixed in 9.1.3. Some more
context in this issue:

jruby/jruby#1279
maxjacobson added a commit to codeclimate/codeclimate-duplication that referenced this issue Nov 2, 2016
This is motivated by a rare error which was fixed in 9.1.3. Some more
context in this issue:

jruby/jruby#1279
@perlun
Copy link
Contributor

perlun commented Sep 27, 2017

Incredibly weird - I just saw this when building a Docker image. Here is the backtrace:

Backtrace

NotImplementedError: the Digest::SHA256() function is unimplemented on this machine
  org/jruby/RubyClass.java:232:in `allocate'
  /opt/jruby/lib/ruby/gems/shared/gems/bundler-1.15.4/lib/bundler/rubygems_gem_installer.rb:28:in `block in validate_bundler_checksum'
  org/jruby/RubyIO.java:1156:in `open'
  org/jruby/RubyKernel.java:306:in `open'
  /opt/jruby/lib/ruby/stdlib/rubygems/package/file_source.rb:30:in `with_read_io'
  /opt/jruby/lib/ruby/gems/shared/gems/bundler-1.15.4/lib/bundler/rubygems_gem_installer.rb:27:in `validate_bundler_checksum'
  /opt/jruby/lib/ruby/gems/shared/gems/bundler-1.15.4/lib/bundler/rubygems_gem_installer.rb:17:in `pre_install_checks'
  /opt/jruby/lib/ruby/stdlib/rubygems/installer.rb:281:in `install'
  /opt/jruby/lib/ruby/gems/shared/gems/bundler-1.15.4/lib/bundler/source/rubygems.rb:136:in `block in install'
  /opt/jruby/lib/ruby/gems/shared/gems/bundler-1.15.4/lib/bundler/rubygems_integration.rb:195:in `preserve_paths'
  /opt/jruby/lib/ruby/gems/shared/gems/bundler-1.15.4/lib/bundler/source/rubygems.rb:135:in `install'
  /opt/jruby/lib/ruby/gems/shared/gems/bundler-1.15.4/lib/bundler/installer/gem_installer.rb:55:in `install'
  /opt/jruby/lib/ruby/gems/shared/gems/bundler-1.15.4/lib/bundler/installer/gem_installer.rb:15:in `install_from_spec'
  /opt/jruby/lib/ruby/gems/shared/gems/bundler-1.15.4/lib/bundler/installer/parallel_installer.rb:115:in `block in worker_pool'
  /opt/jruby/lib/ruby/gems/shared/gems/bundler-1.15.4/lib/bundler/worker.rb:63:in `apply_func'
  /opt/jruby/lib/ruby/gems/shared/gems/bundler-1.15.4/lib/bundler/worker.rb:58:in `block in process_queue'
  org/jruby/RubyKernel.java:1292:in `loop'
  /opt/jruby/lib/ruby/gems/shared/gems/bundler-1.15.4/lib/bundler/worker.rb:55:in `process_queue'
  /opt/jruby/lib/ruby/gems/shared/gems/bundler-1.15.4/lib/bundler/worker.rb:89:in `block in create_threads'

Environment

Bundler   1.15.4
Rubygems  2.6.13
Ruby      2.3.3p0 (2017-09-06 revision 56859) [java]
GEM_HOME  /build/ruby/vendor/bundle/jruby/2.3.0
GEM_PATH
Git       not installed
Platform  universal-java-1.8
OpenSSL   JRuby-OpenSSL 0.9.21

The Dockerfile has this FROM line:

FROM jruby:9.1.13-jre as jruby-builder

I can very well create a new GitHub issue for this if you like, but it feels like it's the original problem coming back somehow... so we could as well reopen this one.

@perlun
Copy link
Contributor

perlun commented Sep 27, 2017

(I only saw this one single time. Subsequent times it has worked flawlessly.)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

4 participants