Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Kernel#require cannot find compiled ruby classes in jruby 9.0.0.0 #3270

Closed
CeesZ opened this issue Aug 20, 2015 · 28 comments
Closed

Kernel#require cannot find compiled ruby classes in jruby 9.0.0.0 #3270

CeesZ opened this issue Aug 20, 2015 · 28 comments

Comments

@CeesZ
Copy link

CeesZ commented Aug 20, 2015

I successfully transferred an application running a large number of
Ruby programs in a JAR from jruby-complete-1.7.19 to jruby-complete-9.0.0.0.
However, when I use JRUBYC to compile one or more of the ruby programs to
Java classes (e.g replacing test2.rb by test2.class in the same position in
the JAR filestructure) I get "LoadError: no such file to load -- test2" as a result of 'require "tes2"'
This worked fine in all previous versions of JRuby but maybe the procedure has changed?

@CeesZ
Copy link
Author

CeesZ commented Aug 20, 2015

Sorry, see I misspelled , should be 'require "test2".

@MSNexploder
Copy link
Contributor

You'll probably need to pass -Xaot.loadClasses=true to JRuby or -Djruby.aot.loadClasses=true to the JVM. (see @headius comment - #3018 (comment))

HTH

@CeesZ
Copy link
Author

CeesZ commented Aug 21, 2015

Thank you very much for pointing this out.
However this option seems not to be supported in JRUBY-COMPLETE:

JRUBY lists aot.loadClasses as a valid option:

jruby -v
jruby 9.0.0.0 (2.2.2) 2015-07-21 e10ec96 Java HotSpot(TM) Client VM 24.75-b04 on 1.7.0_75-b13 +jit [Windows 7-x86]
jruby -Xproperties

Look for .class before .rb to load AOT-compiled code

Options: [true, false], Default: false.

#aot.loadClasses=false

JRUBY-COMPLETE ( based on the same version of JRUBY) does not seem to support this option:

java -jar jruby-complete-9.0.0.0.jar -v
jruby 9.0.0.0 (2.2.2) 2015-07-21 e10ec96 Java HotSpot(TM) Client VM 24.75-b04 on 1.7.0_75-b13 +jit [Windows 7-x86]

java -jar jruby-complete-9.0.0.0.jar -Xaot.loadClasses=true
jruby: invalid extended option aot.loadClasses=true (-X will list valid options)

java -jar lib\java\jruby-complete-9.0.0.0.jar -X
jruby: missing argument
Extended options:
-X-O run with ObjectSpace disabled (default; improves performance)
-X+O run with ObjectSpace enabled (reduces performance)
-X-C disable all compilation
-X-CIR disable all compilation and use IR runtime
-X+C force compilation of all scripts before they are run (except eval)
-X+CIR force compilation and use IR runtime
-X+JIR JIT compilation and use IR runtime
-X+T use Truffle
-Xsubstring? list options that contain substring in their name
-Xprefix... list options that are prefixed wtih prefix

@CeesZ
Copy link
Author

CeesZ commented Aug 21, 2015

Further progress:
The second option you mentioned, -Djruby.aot.loadClasses=true to the JVM does work with JRUBY-COMPLETE. It exposed however a new problem that require_relative does not work with compiled classes. After replacing
require_relative "filename"
with
require File.expand_path('filename', File.dirname(__FILE__))
all the rest seems to work fine.
There was an earlier issue concerning require_relative, but that was cured before this release. This current problem seems to only occur for compiled programs.
CZ

@mkristian
Copy link
Member

@CeesZ yes jruby-complete can not use those -X options since they are meant for the launcher which does convert them to -Djruby. ones for the java invocation.

@CeesZ
Copy link
Author

CeesZ commented Aug 21, 2015

Yes, thank you for explaining; I understand that now.

@headius
Copy link
Member

headius commented Sep 4, 2015

@CeesZ We'll narrow this to just the require_relative issue. Can you provide a small sample repository that shows this problem? With the jruby.aot.loadClasss property set, I don't see why require_relative wouldn't work just fine.

@headius headius added this to the JRuby 9.0.2.0 milestone Sep 4, 2015
@CeesZ
Copy link
Author

CeesZ commented Sep 8, 2015

Sorry for late reply. At the moment on holiday in Scotland with only occasional access to internet. Will be back home at the end of this week and work on a small sample repository.
=Cees

On 4 sep. 2015, at 01:21, Charles Oliver Nutter notifications@github.com wrote:

@CeesZ We'll narrow this to just the require_relative issue. Can you provide a small sample repository that shows this problem? With the jruby.aot.loadClasss property set, I don't see why require_relative wouldn't work just fine.


Reply to this email directly or view it on GitHub.

@CeesZ
Copy link
Author

CeesZ commented Sep 11, 2015

I constructed the following archive (test1.jar) for testing require_relative:

/app
  |
  |__ test1.rb                          puts "test1: __FILE__ = " + __FILE__
  |                                     require_relative 'test2'
  |
  |__ test2.rb                          puts "test2: __FILE__ = " + __FILE__
  |                                     require_relative 'sub/test3'
  |
  |__/sub
       |__ test3.rb                     puts "test3: __FILE__ = " + __FILE__
       |                                require_relative 'test4'
       |
       |__ test4.rb                     puts "test4: __FILE__ = " + __FILE__

run with jruby-complete:

java -Djruby.aot.loadClasses=true -cp test1.jar;jruby-complete-9.0.0.0.jar org.jruby.Main -e "require 'uri:classloader:/app/test1'"

output (as expected)

test1: __FILE__ = uri:classloader:/app/test1.rb
test2: __FILE__ = uri:classloader:/app/test2.rb
test3: __FILE__ = uri:classloader:/app/sub/test3.rb
test4: __FILE__ = uri:classloader:/app/sub/test4.rb

test2.jar is identical to test1.jar, with .rb files replaced by their .class files; using same java command:

java -Djruby.aot.loadClasses=true -cp test2.jar;jruby-complete-9.0.0.0.jar org.jruby.Main -e "require 'uri:classloader:/app/test1'"

instead of loading test2.class from uri:classloader:/app/test2.class it attempts
to find test1.rb outside the jar probaly using the working directory:

test1: __FILE__ = uri:classloader:/app/test1.class
Errno::ENOENT: No such file or directory - C:\Projects\test_project\app\test1.rb
          realpath at org/jruby/RubyFile.java:805
  require_relative at uri:classloader:/jruby/kernel/kernel.rb:21
             <top> at app/test1.rb:2
           require at org/jruby/RubyKernel.java:940
            (root) at uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/rubygems/core_ext/kernel_require.rb:1
             <top> at -e:1

I also noticed some odd behavior of __FILE__ when using .class files,
maybe this gives a clue as to what is going wrong:
for the same instance of __FILE__

(supposedly a String object)

puts __FILE__          # => uri:classloader:/app/test1.class
puts "#{__FILE__}"     # => app/test1.rb
puts __FILE__.class    # => String

when using .rb files:

puts __FILE__          # => uri:classloader:/app/test1.rb
puts "#{__FILE__}"     # => uri:classloader:/app/test1.rb

Hope this will be of some help to solve the problem.

==Cees

@CeesZ
Copy link
Author

CeesZ commented Sep 11, 2015

Sorry tried to attach the test jars, but does not seem to work

@mkristian
Copy link
Member

I can reproduce it on my machine (not windows) and get

Errno::ENOENT: No such file or directory - /Users/cmeier/projects/active/jruby/app/test1.rb
          realpath at org/jruby/RubyFile.java:805
  require_relative at uri:classloader:/jruby/kernel/kernel.rb:11
             <top> at app//test1.rb:2
           require at org/jruby/RubyKernel.java:939
            (root) at uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/rubygems/core_ext/kernel_require.rb:1
             <top> at -e:1

it does not complain not finding test2 but not finding test1 on the filesystem - indeed this is looking odd.

@CeesZ thanks for the testcase - my own trial never failed ;)

@CeesZ
Copy link
Author

CeesZ commented Sep 11, 2015

OK, here are the test jars:

https://www.dropbox.com/s/fe1n7p8juuj532b/test1.jar?dl=0
https://www.dropbox.com/s/d5kscle3z9lzxju/test2.jar?dl=0

@mkristian maybe it is caused by a File.dirname request on "#{__FILE__}" instead of __FILE__ ?

@headius
Copy link
Member

headius commented Oct 7, 2015

@mkristian Do you have any thoughts on this? We'd like to put 9.0.2 out soon.

@mkristian
Copy link
Member

looking now . . .

@mkristian
Copy link
Member

the caller in this line
https://github.com/jruby/jruby/blob/master/core/src/main/ruby/jruby/kernel/kernel.rb#L7
is running the jar with the ruby scripts

uri:classloader:/app/test1.rb:2:in `<top>'
/Users/cmeier/projects/active/jruby/lib/ruby/stdlib/rubygems/core_ext/kernel_require.rb:1:in `require'
/Users/cmeier/projects/active/jruby/lib/ruby/stdlib/rubygems/core_ext/kernel_require.rb:1:in `(root)'
-e:1:in `<top>'

with the compiled files it is

app//test1.rb:2:in `<top>'
/Users/cmeier/projects/active/jruby/lib/ruby/stdlib/rubygems/core_ext/kernel_require.rb:1:in `require'
/Users/cmeier/projects/active/jruby/lib/ruby/stdlib/rubygems/core_ext/kernel_require.rb:1:in `(root)'
-e:1:in `<top>'

adjusting the file from the kernel require_relative method with file = "uri:classloader://#{file.sub(/.rb/, '.class')}" fixes things.

@headius stepping into the caller is new territory for nothing I am able to do today

@mkristian
Copy link
Member

the class has __FILE__ compiled into the class, in the above example it is app//test1.rb which is different from what the FileResource from the LoadService delivers: uri:classloader://app/test1.class

this is probably also the reason why there are two representation of FILE around, one from the class-file and one from the LoadService.

@enebo
Copy link
Member

enebo commented Oct 15, 2015

@mkristian @headius This is the last blocker for 9.0.2.0. There multiple things I do not understand at play here :)

FILE in a compiled version of a .rb file SHOULD be a .rb name shouldn't it? I mean compilation is just a different way of storing a .rb. In that sense I think it is weird to ever display this as .class. So I do not think the solution is to do rename FILE. My stance is compilation is just a different save format for a .rb file and it should not change any user-visible behavior from the act of compilation.

If I require_relative a AOT compiled class should I even care what the extension is of the file I am in? I do care about the base parent path but I would think we would do that correctly? Can't we just strip the extension off the caller data and this will just work? We don't actually know if we will be loading a .rb or a .class from this file.

@mkristian
Copy link
Member

@enebo I thought absolute path does need the file extension. but just tried it on both jruby + mri and it works. yes, this is one thing which can be improved easily.

@mkristian
Copy link
Member

not sure if this is related or not, but there are more inconsistencies:
test.rb

puts __FILE__
puts "#{__FILE__}"

then compile and put it into a jar

jrubyc test.rb
jar -cf some.jar test.class
rm test.rb

now

$ jruby -e 'require "uri:classloader:/test.class"'
uri:classloader:/test.class
uri:classloader:/test.class

some with "." on the classpath

$ java -cp .:jruby-complete-9.0.2.0-SNAPSHOT.jar org.jruby.Main -e 'require "uri:classloader:/test.class"'
uri:classloader:/test.class
uri:classloader:/test.class

but

$ java -cp some.jar:jruby-complete-9.0.2.0-SNAPSHOT.jar org.jruby.Main -e 'require "uri:classloader:/test.class"'
LoadError: no such file to load -- uri:classloader:/test.class
  require at org/jruby/RubyKernel.java:939
  require at uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/rubygems/core_ext/kernel_require.rb:54
    <top> at -e:1

and whatever jruby or jruby-complete they all can not find the file without ".class"

$ jruby -e 'require "uri:classloader:/test"'
LoadError: no such file to load -- uri:classloader:/test
  require at org/jruby/RubyKernel.java:939
  require at /Users/cmeier/projects/active/jruby/lib/ruby/stdlib/rubygems/core_ext/kernel_require.rb:54
    <top> at -e:1

@enebo
Copy link
Member

enebo commented Oct 15, 2015

@mkristian could this be because of special logic for uri:classloader somehow?

@mkristian
Copy link
Member

@enebo I really would like to take this uri:classloader: path out of the equation and once the following example works, it should basically fix the original problem here:

test.rb

p __FILE__
Dir.chdir('/tmp')
require_relative 'a'

and a.rb with

puts 'hello'

then execute it with MRI

$ ruby -v test.rb
ruby 2.2.3p173 (2015-08-18 revision 51636) [x86_64-darwin14]
"test.rb"
hello

and with jruby

$ ../bin/jruby -v test.rb
jruby 9.0.2.0-SNAPSHOT (2.2.2) 2015-10-14 d0bcfee Java HotSpot(TM) 64-Bit Server VM 25.51-b03 on 1.8.0_51-b16 +jit [darwin-x86_64]
"test.rb"
Errno::ENOENT: No such file or directory - /private/tmp/t.rb
          realpath at org/jruby/RubyFile.java:814
  require_relative at uri:classloader:/jruby/kernel/kernel.rb:11
             <top> at t.rb:3

slightly different setup but same error.

@mkristian
Copy link
Member

diff --git a/core/src/main/ruby/jruby/kernel/kernel.rb b/core/src/main/ruby/jruby/kernel/kernel.rb
index b4dd5e13..6ec4164 100644
--- a/core/src/main/ruby/jruby/kernel/kernel.rb
+++ b/core/src/main/ruby/jruby/kernel/kernel.rb
@@ -8,7 +8,12 @@ module Kernel
     file = $` # just the filename
     raise LoadError, "cannot infer basepath" if /\A\((.*)\)/ =~ file # eval etc.

-    absolute_feature = File.expand_path(relative_arg, File.dirname(File.realpath(file)))
+    dir = File.dirname(file)
+    if File.exists?(dir)
+      absolute_feature = File.expand_path(relative_arg, File.dirname(File.realpath(file)))
+    else
+      absolute_feature = File.join(dir, relative_arg.sub(/\\.[^.\/]+$/, ''))
+    end

     require absolute_feature
   end

will "fix" this original example but it will fail in case there is an empty directory app where you execute the application. then it will fail in the same manner as before.

searching some files or directories on the filesystem based on the PWD is not the right thing to do.

headius added a commit that referenced this issue Oct 18, 2015
I opted to make the Filename operand load the filename set into
the containing scope, which makes it simple to set it in a single
place when loading IR. Unsure whether this is the best approach or
not.
@headius
Copy link
Member

headius commented Oct 18, 2015

I pushed a change that makes AOT-compiled IR set a proper FILE based on how it was loaded. My approach was to make the Filename operand retrieve the filename from its containing scope, which delegates up to whatever the top scope is in that context. I'm not sure if this is the best way but it was simple to implement.

@headius
Copy link
Member

headius commented Oct 18, 2015

[] ~/projects/jruby $ cat blah.rb
puts __FILE__

[] ~/projects/jruby $ jruby blah.rb
blah.rb

[] ~/projects/jruby $ cd ..

[] ~/projects $ jruby jruby/blah.rb
jruby/blah.rb

[] ~/projects $ cd jruby

[] ~/projects/jruby $ jrubyc blah.rb

[] ~/projects/jruby $ jruby -Xaot.loadClasses=true blah.class
blah.class

[] ~/projects/jruby $ cd ..

[] ~/projects $ jruby -Xaot.loadClasses=true jruby/blah.class
jruby/blah.class

@enebo
Copy link
Member

enebo commented Oct 18, 2015

I know this is changing behavior but I find it strange that FILE is transformed with AOT'd to be .class. It is the file it is loaded from but I think it means an AOT-version of a ruby file now has different outward appearange. It also means it is impossible to run debugging tools on AOT'd code. Can anyone comment on what would happen if we make FILE report original .rb instead of its ultmate .class?

@enebo
Copy link
Member

enebo commented Oct 18, 2015

After talking to @headius I think my desire for this change is unimportant for this particular fix. If we are running debugging tools and AOT those instructions into the compiled class it will emit the actual name so FILE is less clear how it is important (or not). Something about it still bugs me but I will open another issue when/if I can think of a better reason for disliking this .rb/.class mismatch on AOT'd file.

@mkristian
Copy link
Member

so my suspicion on File.realpath bit me here: #3401

anyways can we close this one now ?

@headius
Copy link
Member

headius commented Oct 20, 2015

@mkristian If there's no other changes needed for require_relative and friends, we can close this. FILE behavior should match 1.7 now.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

5 participants