Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fiddle fixes #4518

Closed
wants to merge 13 commits into from
Closed

Fiddle fixes #4518

wants to merge 13 commits into from

Conversation

headius
Copy link
Member

@headius headius commented Mar 1, 2017

In #4511, @jellymann provided some nice improvements to our "badly broken" implementation of Fiddle. Initially these changes looked ok, but then it turned out they broke logic on Windows that in turn caused basic things like gem list to fail.

@enebo will comment with his findings, and we will work with @jellymann to get things in order.

headius added 2 commits March 1, 2017 13:42
This is a re-merge on a separate branch due to Wnidows breakage
caused by the original fixes.
This is a re-commit on a new branch because of Windows breakage
caused by jruby#4511.
@enebo
Copy link
Member

enebo commented Mar 1, 2017

Before these fixes if I run gem list (with -d) I see some goofiness:

Exception `LoadError' at C:/opt/jruby/lib/ruby/stdlib/ffi/library.rb:228 - LoadError
Exception `FFI::NotFoundError' at C:/opt/jruby/lib/ruby/stdlib/ffi/library.rb:241 - Function 'CreateSymbolicLinkW' not found in [shlwapi]

So something is not loading properly somewhere and this may or may not end up being related since it is hard to tell where we are really calling FFI vs calling Fiddle which is calling FFI. Here is the output with the patches applied:

/c/opt/jruby master * 693% ./bin/jruby -d -S gem list
Exception `LoadError' at C:/opt/jruby/lib/ruby/stdlib/ffi/library.rb:228 - LoadError
Exception `FFI::NotFoundError' at C:/opt/jruby/lib/ruby/stdlib/ffi/library.rb:241 - Function 'CreateSymbolicLinkW' not found in [shlwapi]
io/console not supported; tty will not be manipulated
Exception `FFI::NotFoundError' at C:/opt/jruby/lib/ruby/stdlib/ffi/library.rb:308 - Function 'sys_errlist' not found in [#<FFI::DynamicLibrary:0x3c836c44>]
Exception `FFI::NotFoundError' at C:/opt/jruby/lib/ruby/stdlib/ffi/library.rb:308 - Function 'sys_errlist' not found in [#<FFI::DynamicLibrary:0x3c836c44>]
Exception `FFI::NotFoundError' at C:/opt/jruby/lib/ruby/stdlib/ffi/library.rb:308 - Function 'sys_errlist' not found in [#<FFI::DynamicLibrary:0x3c836c44>]
Exception `FFI::NotFoundError' at C:/opt/jruby/lib/ruby/stdlib/ffi/library.rb:308 - Function 'sys_errlist' not found in [#<FFI::DynamicLibrary:0x3c836c44>]
Exception `FFI::NotFoundError' at C:/opt/jruby/lib/ruby/stdlib/ffi/library.rb:308 - Function 'sys_errlist' not found in [#<FFI::DynamicLibrary:0x3c836c44>]
Exception `FFI::NotFoundError' at C:/opt/jruby/lib/ruby/stdlib/ffi/library.rb:308 - Function 'sys_errlist' not found in [#<FFI::DynamicLibrary:0x3c836c44>]
Exception `FFI::NotFoundError' at C:/opt/jruby/lib/ruby/stdlib/ffi/library.rb:308 - Function 'sys_errlist' not found in [#<FFI::DynamicLibrary:0x3c836c44>]
Exception `FFI::NotFoundError' at C:/opt/jruby/lib/ruby/stdlib/ffi/library.rb:308 - Function 'sys_errlist' not found in [#<FFI::DynamicLibrary:0x3c836c44>]
Exception `FFI::NotFoundError' at C:/opt/jruby/lib/ruby/stdlib/ffi/library.rb:308 - Function 'sys_errlist' not found in [#<FFI::DynamicLibrary:0x3c836c44>]
Exception `FFI::NotFoundError' at C:/opt/jruby/lib/ruby/stdlib/ffi/library.rb:308 - Function 'sys_errlist' not found in [#<FFI::DynamicLibrary:0x3c836c44>]
Exception `FFI::NotFoundError' at C:/opt/jruby/lib/ruby/stdlib/ffi/library.rb:308 - Function 'sys_errlist' not found in [#<FFI::DynamicLibrary:0x3c836c44>]
Exception `FFI::NotFoundError' at C:/opt/jruby/lib/ruby/stdlib/ffi/library.rb:308 - Function 'sys_errlist' not found in [#<FFI::DynamicLibrary:0x3c836c44>]
Exception `FFI::NotFoundError' at C:/opt/jruby/lib/ruby/stdlib/ffi/library.rb:308 - Function 'sys_errlist' not found in [#<FFI::DynamicLibrary:0x3c836c44>]
ERROR:  Loading command: list (FFI::NotFoundError)
        Function 'sys_errlist' not found in [#<FFI::DynamicLibrary:0x3c836c44>]
        C:/opt/jruby/lib/ruby/stdlib/ffi/library.rb:308:in `attach_variable'
        C:/opt/jruby/lib/ruby/stdlib/ffi/libc.rb:20:in `<module:LibC>'
        C:/opt/jruby/lib/ruby/stdlib/ffi/libc.rb:4:in `<module:FFI>'
        C:/opt/jruby/lib/ruby/stdlib/ffi/libc.rb:3:in `<main>'
        org/jruby/RubyKernel.java:961:in `require'
        C:/opt/jruby/lib/ruby/stdlib/rubygems/core_ext/kernel_require.rb:55:in `require'
        C:/opt/jruby/lib/ruby/stdlib/fiddle/jruby.rb:1:in `<main>'
        org/jruby/RubyKernel.java:961:in `require'
        C:/opt/jruby/lib/ruby/stdlib/rubygems/core_ext/kernel_require.rb:55:in `require'
        C:/opt/jruby/lib/ruby/stdlib/fiddle/jruby.rb:2:in `<main>'
        org/jruby/RubyKernel.java:961:in `require'
        C:/opt/jruby/lib/ruby/stdlib/rubygems/core_ext/kernel_require.rb:55:in `require'
        C:/opt/jruby/lib/ruby/stdlib/fiddle.rb:1:in `<main>'
        org/jruby/RubyKernel.java:961:in `require'
        C:/opt/jruby/lib/ruby/stdlib/rubygems/core_ext/kernel_require.rb:55:in `require'
        C:/opt/jruby/lib/ruby/stdlib/fiddle.rb:3:in `<main>'
        org/jruby/RubyKernel.java:961:in `require'
        C:/opt/jruby/lib/ruby/stdlib/rubygems/core_ext/kernel_require.rb:55:in `require'
        C:/opt/jruby/lib/ruby/stdlib/fiddle/import.rb:1:in `<main>'
        org/jruby/RubyKernel.java:961:in `require'
        C:/opt/jruby/lib/ruby/stdlib/rubygems/core_ext/kernel_require.rb:55:in `require'
        C:/opt/jruby/lib/ruby/stdlib/fiddle/import.rb:2:in `<main>'
        org/jruby/RubyKernel.java:961:in `require'
        C:/opt/jruby/lib/ruby/stdlib/rubygems/core_ext/kernel_require.rb:55:in `require'
        C:/opt/jruby/lib/ruby/stdlib/win32/importer.rb:1:in `<class:(root)>'
        C:/opt/jruby/lib/ruby/stdlib/win32/importer.rb:2:in `<class:(root)>'
        C:/opt/jruby/lib/ruby/stdlib/win32/registry.rb:1:in `<main>'
        org/jruby/RubyKernel.java:961:in `require'
        C:/opt/jruby/lib/ruby/stdlib/rubygems/core_ext/kernel_require.rb:55:in `require'
        C:/opt/jruby/lib/ruby/stdlib/win32/registry.rb:2:in `<main>'
        org/jruby/RubyKernel.java:961:in `require'
        C:/opt/jruby/lib/ruby/stdlib/rubygems/core_ext/kernel_require.rb:55:in `require'
        C:/opt/jruby/lib/ruby/stdlib/win32/resolv.rb:1:in `<main>'
        org/jruby/RubyKernel.java:961:in `require'
        C:/opt/jruby/lib/ruby/stdlib/rubygems/core_ext/kernel_require.rb:55:in `require'
        C:/opt/jruby/lib/ruby/stdlib/win32/resolv.rb:7:in `<main>'
        org/jruby/RubyKernel.java:961:in `require'
        C:/opt/jruby/lib/ruby/stdlib/rubygems/core_ext/kernel_require.rb:55:in `require'
        C:/opt/jruby/lib/ruby/stdlib/resolv.rb:1:in `<main>'
        org/jruby/RubyKernel.java:961:in `require'
        C:/opt/jruby/lib/ruby/stdlib/rubygems/core_ext/kernel_require.rb:55:in `require'
        C:/opt/jruby/lib/ruby/stdlib/resolv.rb:176:in `Hosts'
        C:/opt/jruby/lib/ruby/stdlib/resolv.rb:174:in `Resolv'
        C:/opt/jruby/lib/ruby/stdlib/resolv.rb:39:in `(root)'
        C:/opt/jruby/lib/ruby/stdlib/rubygems/remote_fetcher.rb:1:in `(root)'
        C:/opt/jruby/lib/ruby/stdlib/rubygems/remote_fetcher.rb:7:in `(root)'
        C:/opt/jruby/lib/ruby/stdlib/rubygems/spec_fetcher.rb:1:in `(root)'
        C:/opt/jruby/lib/ruby/stdlib/rubygems/spec_fetcher.rb:2:in `<main>'
        org/jruby/RubyKernel.java:979:in `load'
        C:/opt/jruby/bin/gem:4:in `<main>'
ERROR:  While executing gem ... (NoMethodError)
    undefined method `invoke_with_build_args' for nil:NilClass
        C:/opt/jruby/lib/ruby/stdlib/rubygems/command_manager.rb:169:in `process_args'
        C:/opt/jruby/lib/ruby/stdlib/rubygems/command_manager.rb:139:in `run'
        C:/opt/jruby/lib/ruby/stdlib/rubygems/gem_runner.rb:55:in `run'
        C:/opt/jruby/bin/jgem:21:in `<main>'
        org/jruby/RubyKernel.java:979:in `load'
        C:/opt/jruby/bin/gem:4:in `<main>'
Exception `Gem::SystemExitException' at C:/opt/jruby/lib/ruby/stdlib/rubygems/user_interaction.rb:392 - Exiting RubyGems with exit_code 1

Mostly this looks like a single repeated error.

@danini-the-panini
Copy link
Contributor

danini-the-panini commented Mar 11, 2017

Hi @headius and @enebo. Thank you for taking the time to review my changes. I will begin immediately to resolve the issues on Windows. Following that, I will look into fixing the FFI/Fiddle function issue from #4510, along with as much else I can.

I assume gem list was working before my changes to Fiddle?

@headius
Copy link
Member Author

headius commented Mar 11, 2017

@jellymann Yes, it works without your changes, so something must have gone awry. @enebo above pointed out that there are some errors happening internally related to FFI, but those may not be related to the new failures that happen after the patch.

I now also have a Windows VM I can use. Let us know if you need some assistance, and we will look forward to some additional patches!

Logistically, you can either create a new PR to headius/jruby@fiddle_fixes, or just make a new PR against jruby/jruby@master containing the whole shebang.

Thank you for your help!

Daniel Smith added 4 commits March 12, 2017 12:10
Previous commit broke this functionality by only allowing Fiddle::Pointer types.
This should be supported in future. Currently, attempting to convert IO to pointer causes a Segfault. Raising to avoid this for now.
@danini-the-panini
Copy link
Contributor

@headius @enebo I have made a couple of PRs in to this branch to resolve a couple of things:

  1. Fixed LibC on Windows, turns out a lot of LibC is missing in the Windows DLL. fix: FFI LibC not loading on Windows headius/jruby#1
  2. Fixed the interface to Fiddle::Function#call to accept similar pointer-like arguments as MRI. Fiddle::Function fixes headius/jruby#2

Daniel Smith added 2 commits March 12, 2017 13:59
Accidentally left them in in last commit
Previously, `Fiddle::Pointer`s passed to `Fiddle::Function#call` were actually being silently converted to `FFI::Pointer`s because `FFI::Function#call` calls `#to_ptr`, which coincidentally (or not) returns the underlying `FFI::Pointer`. I now convert these pointers explicitely and remove the `#to_ptr` method.
@headius
Copy link
Member Author

headius commented Mar 13, 2017

@jellymann Looks great! I've merged your PRs in.

@enebo You have a dev env set up for Windows, want to give the branch a quick check?

@headius
Copy link
Member Author

headius commented Mar 13, 2017

I just got a build going myself and I'm seeing a new error when I attempt to gem install anything:

C:\Users\headius\jruby (fiddle-fixes)
λ gem install rails
ERROR:  Loading command: install (RuntimeError)
        can't parse the function prototype: int WsControl(int, int, void *, void *, void *, void *
ERROR:  While executing gem ... (NoMethodError)
    undefined method `invoke_with_build_args' for nil:NilClass

I'll see if I can find the problem.

@headius
Copy link
Member Author

headius commented Mar 13, 2017

It looks like that error was just our Fiddle being a bit more strict about a missing closing parenthesis. However, fixing that (patch below) it still fails to bind this function from the wsock32 library.

diff --git a/lib/ruby/stdlib/win32/resolv.rb b/lib/ruby/stdlib/win32/resolv.rb
index 183d5d38da..72a685a722 100644
--- a/lib/ruby/stdlib/win32/resolv.rb
+++ b/lib/ruby/stdlib/win32/resolv.rb
@@ -264,7 +264,7 @@ else
         extend Importer
         dlload "wsock32.dll"
       end
-      WsControl = WSock32.extern "int WsControl(int, int, void *, void *, void *, void *", :stdcall
+      WsControl = WSock32.extern "int WsControl(int, int, void *, void *, void *, void *)", :stdcall
       WSAGetLastError = WSock32.extern "int WSAGetLastError(void)", :stdcall

       MAX_TDI_ENTITIES = 512

Here's the errors with debugging turned on in RubyGems. Looks like there's still several other problems loading the needed functions on Windows.

λ gem install --debug rails
NOTE:  Debugging mode prints all exceptions even when rescued
Exception `LoadError' at C:/Users/headius/jruby/lib/ruby/stdlib/ffi/library.rb:228 - LoadError
Exception `FFI::NotFoundError' at C:/Users/headius/jruby/lib/ruby/stdlib/ffi/library.rb:241 - Function 'clearenv' not found in [msvcrt.dll]
Exception `LoadError' at C:/Users/headius/jruby/lib/ruby/stdlib/ffi/library.rb:228 - LoadError
Exception `FFI::NotFoundError' at C:/Users/headius/jruby/lib/ruby/stdlib/ffi/library.rb:241 - Function 'memrchr' not found in [msvcrt.dll]
Exception `FFI::NotFoundError' at C:/Users/headius/jruby/lib/ruby/stdlib/ffi/library.rb:308 - Function 'stdin' not found in [#<FFI::DynamicLibrary:0x7ad1caa2>]
Exception `Fiddle::DLError' at C:/Users/headius/jruby/lib/ruby/stdlib/fiddle/jruby.rb:357 - unknown symbol WsControl
Exception `Fiddle::DLError' at C:/Users/headius/jruby/lib/ruby/stdlib/fiddle/import.rb:292 - cannot find the function: WsControl()
Exception `Fiddle::DLError' at C:/Users/headius/jruby/lib/ruby/stdlib/fiddle/import.rb:292 - cannot find the function: WsControl()
Exception `Fiddle::DLError' at C:/Users/headius/jruby/lib/ruby/stdlib/fiddle/import.rb:292 - cannot find the function: WsControl()
Exception `Fiddle::DLError' at C:/Users/headius/jruby/lib/ruby/stdlib/fiddle/import.rb:292 - cannot find the function: WsControl()
Exception `Fiddle::DLError' at C:/Users/headius/jruby/lib/ruby/stdlib/fiddle/import.rb:292 - cannot find the function: WsControl()
Exception `Fiddle::DLError' at C:/Users/headius/jruby/lib/ruby/stdlib/fiddle/import.rb:292 - cannot find the function: WsControl()
Exception `Fiddle::DLError' at C:/Users/headius/jruby/lib/ruby/stdlib/fiddle/import.rb:292 - cannot find the function: WsControl()
Exception `Fiddle::DLError' at C:/Users/headius/jruby/lib/ruby/stdlib/fiddle/import.rb:292 - cannot find the function: WsControl()
ERROR:  Loading command: install (Fiddle::DLError)
        cannot find the function: WsControl()
        C:/Users/headius/jruby/lib/ruby/stdlib/fiddle/import.rb:292:in `import_function'
        C:/Users/headius/jruby/lib/ruby/stdlib/fiddle/import.rb:165:in `extern'
        C:/Users/headius/jruby/lib/ruby/stdlib/win32/resolv.rb:267:in `<module:WsControl>'
        C:/Users/headius/jruby/lib/ruby/stdlib/win32/resolv.rb:262:in `<eval>'
        org/jruby/RubyModule.java:2853:in `module_eval'
        C:/Users/headius/jruby/lib/ruby/stdlib/win32/resolv.rb:128:in `<module:Resolv>'
        C:/Users/headius/jruby/lib/ruby/stdlib/win32/resolv.rb:10:in `<module:Win32>'
        C:/Users/headius/jruby/lib/ruby/stdlib/win32/resolv.rb:9:in `<main>'
        org/jruby/RubyKernel.java:961:in `require'
        C:/Users/headius/jruby/lib/ruby/stdlib/rubygems/core_ext/kernel_require.rb:1:in `(root)'
        C:/Users/headius/jruby/lib/ruby/stdlib/rubygems/core_ext/kernel_require.rb:55:in `<class:require>'
        C:/Users/headius/jruby/lib/ruby/stdlib/resolv.rb:177:in `<class:Hosts>'
        C:/Users/headius/jruby/lib/ruby/stdlib/resolv.rb:175:in `<main>'
...

headius added a commit to headius/jruby that referenced this pull request Mar 13, 2017
@enebo
Copy link
Member

enebo commented Mar 13, 2017

Something else I believe is wrong here....WsControl is for windows 98 support using winsock. I suspect we should not be loading this at all...

@enebo
Copy link
Member

enebo commented Mar 13, 2017

Ok I got this to work again but I know I reverted one of your (@jellyman) fixes:

diff --git a/lib/ruby/stdlib/fiddle/jruby.rb b/lib/ruby/stdlib/fiddle/jruby.rb
index f69a6fc..d3d9ef6 100644
--- a/lib/ruby/stdlib/fiddle/jruby.rb
+++ b/lib/ruby/stdlib/fiddle/jruby.rb
@@ -97,12 +97,12 @@ module Fiddle
     private

     def make_pointer(arg, type)
-      return arg if type != TYPE_VOIDP
+      return arg #if type != TYPE_VOIDP
       Pointer[arg]
     end

     def make_native(ptr, type)
-      return ptr if type != TYPE_VOIDP
+      return ptr #if type != TYPE_VOIDP
       ptr.ffi_ptr
     end
   end

So the issue is that if you run this:

require 'win32/importer'

module Kernel32
  extend Importer
  dlload "kernel32"
end
getv = Kernel32.extern "int GetVersionExA(void *)", :stdcall
info = [ 148, 0, 0, 0, 0 ].pack('V5') + "\0" * 128
require 'pp'
pp info
getv.call(info)
pp info
pp info.unpack('V5')
#info.unpack('V5')[4] == 2  # VER_PLATFORM_WIN32_NT

You will see that the param to GetVersionExA is an out parameter of void * but you make a new pointer so the original string loses the out data.

@danini-the-panini
Copy link
Contributor

@headius @enebo Thanks for checking it out. I'll take a look at it on my Windows box as soon as I can.

@danini-the-panini
Copy link
Contributor

@enebo I can't seem to see what the issue is with the above code. I ran it on my Windows machine and it runs exactly the same as it does on MRI.

C:\Users\Daniel\Projects\jruby>bin\jruby C:\Users\Daniel\Desktop\foo.rb
"\x94\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x0
0\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x0
0\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x0
0\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x0
0\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x0
0\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x0
0\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x0
0\x00\x00\x00\x00\x00\x00\x00\x00"
"\x94\x00\x00\x00\x06\x00\x00\x00\x01\x00\x00\x00\xB1\x1D\x00\x00\x02\x00\x00\x0
0Service Pack 1\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\
x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\
x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\
x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\
x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\
x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00"
[148, 6, 1, 7601, 2]

C:\Users\Daniel\Projects\jruby>bin\jruby C:\Users\Daniel\Desktop\foo.rb
"\x94\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x0
0\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x0
0\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x0
0\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x0
0\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x0
0\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x0
0\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x0
0\x00\x00\x00\x00\x00\x00\x00\x00"
"\x94\x00\x00\x00\x06\x00\x00\x00\x01\x00\x00\x00\xB1\x1D\x00\x00\x02\x00\x00\x0
0Service Pack 1\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\
x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\
x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\
x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\
x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\
x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00"
[148, 6, 1, 7601, 2]

@danini-the-panini
Copy link
Contributor

@headius I noticed you've merged my PRs on your fork, but how come the changes aren't being reflected in this PR? Did I do something wrong?

@danini-the-panini
Copy link
Contributor

@headius Also, I cannot reproduce the failing gem install --debug rails either. 😕
There are no mentions of WsControl in the debug output.

I'm running Windows 7 SP1 64-bit. Perhaps if you are running a newer/different version of Windows they maybe removed the WsControl function?

@enebo
Copy link
Member

enebo commented Mar 25, 2017

@jellymann This is confusing both @headius and I got the same error on his branch with win7 and win10 (respectively). It was going down the wrong branch because of the snippet I highlighted. The out param data was identical to the before puts (pp) so the comparison with 2 in that 5 element was seeing 0. My guess based on what I commented out is that ptr.ffi_ptr actually allocates another pointer or something like that?

@danini-the-panini
Copy link
Contributor

danini-the-panini commented Mar 25, 2017

@enebo @headius Looks like FFI might be doing some magic when Strings are passed as out arguments, so I bypassed my previous "fix" just for strings: headius#3

I've tested it (and made sure I wasn't on the wrong branch this time) and it seems to be functioning as expected.

@headius
Copy link
Member Author

headius commented Apr 19, 2017

@enebo Can you re-test this on Windows?

Fix: using strings as out arguments in Fiddle
@headius
Copy link
Member Author

headius commented Apr 19, 2017

I've merged headius#3 into my branch there.

@enebo
Copy link
Member

enebo commented Apr 19, 2017

With the head of this branch I see:

> bin\jruby -d -S gem list                                                            
Exception `LoadError' at C:/opt/jruby/lib/ruby/stdlib/ffi/library.rb:228 - LoadError  
Exception `FFI::NotFoundError' at C:/opt/jruby/lib/ruby/stdlib/ffi/library.rb:241 - Fu
nction 'CreateSymbolicLinkW' not found in [shlwapi]                                   
io/console not supported; tty will not be manipulated                                 
Exception `FFI::NotFoundError' at C:/opt/jruby/lib/ruby/stdlib/ffi/library.rb:308 - Fu
nction 'sys_nerr' not found in [#<FFI::DynamicLibrary:0x4145bad8>]                    
Exception `FFI::NotFoundError' at C:/opt/jruby/lib/ruby/stdlib/ffi/library.rb:308 - Fu
nction 'sys_nerr' not found in [#<FFI::DynamicLibrary:0x4145bad8>]                    
Exception `FFI::NotFoundError' at C:/opt/jruby/lib/ruby/stdlib/ffi/library.rb:308 - Fu
nction 'sys_nerr' not found in [#<FFI::DynamicLibrary:0x4145bad8>]                    
Exception `FFI::NotFoundError' at C:/opt/jruby/lib/ruby/stdlib/ffi/library.rb:308 - Fu
nction 'sys_nerr' not found in [#<FFI::DynamicLibrary:0x4145bad8>]                    
Exception `FFI::NotFoundError' at C:/opt/jruby/lib/ruby/stdlib/ffi/library.rb:308 - Fu
nction 'sys_nerr' not found in [#<FFI::DynamicLibrary:0x4145bad8>]                    
Exception `FFI::NotFoundError' at C:/opt/jruby/lib/ruby/stdlib/ffi/library.rb:308 - Fu
nction 'sys_nerr' not found in [#<FFI::DynamicLibrary:0x4145bad8>]                    
Exception `FFI::NotFoundError' at C:/opt/jruby/lib/ruby/stdlib/ffi/library.rb:308 - Fu
nction 'sys_nerr' not found in [#<FFI::DynamicLibrary:0x4145bad8>]                    
Exception `FFI::NotFoundError' at C:/opt/jruby/lib/ruby/stdlib/ffi/library.rb:308 - Fu
nction 'sys_nerr' not found in [#<FFI::DynamicLibrary:0x4145bad8>]                    
Exception `FFI::NotFoundError' at C:/opt/jruby/lib/ruby/stdlib/ffi/library.rb:308 - Fu
nction 'sys_nerr' not found in [#<FFI::DynamicLibrary:0x4145bad8>]                    
Exception `FFI::NotFoundError' at C:/opt/jruby/lib/ruby/stdlib/ffi/library.rb:308 - Fu
nction 'sys_nerr' not found in [#<FFI::DynamicLibrary:0x4145bad8>]                    
Exception `FFI::NotFoundError' at C:/opt/jruby/lib/ruby/stdlib/ffi/library.rb:308 - Fu
nction 'sys_nerr' not found in [#<FFI::DynamicLibrary:0x4145bad8>]                    
Exception `FFI::NotFoundError' at C:/opt/jruby/lib/ruby/stdlib/ffi/library.rb:308 - Fu
nction 'sys_nerr' not found in [#<FFI::DynamicLibrary:0x4145bad8>]                    
Exception `FFI::NotFoundError' at C:/opt/jruby/lib/ruby/stdlib/ffi/library.rb:308 - Fu
nction 'sys_nerr' not found in [#<FFI::DynamicLibrary:0x4145bad8>]                    
ERROR:  Loading command: list (FFI::NotFoundError)                                    
        Function 'sys_nerr' not found in [#<FFI::DynamicLibrary:0x4145bad8>]          
        C:/opt/jruby/lib/ruby/stdlib/ffi/library.rb:308:in `attach_variable'          
        C:/opt/jruby/lib/ruby/stdlib/ffi/libc.rb:21:in `<module:LibC>'                
        C:/opt/jruby/lib/ruby/stdlib/ffi/libc.rb:4:in `<module:FFI>'                  
        C:/opt/jruby/lib/ruby/stdlib/ffi/libc.rb:3:in `<main>'                        
        org/jruby/RubyKernel.java:961:in `require'                                    
        C:/opt/jruby/lib/ruby/stdlib/rubygems/core_ext/kernel_require.rb:55:in `requir
e'                                                                                    
        C:/opt/jruby/lib/ruby/stdlib/fiddle/jruby.rb:1:in `<main>'                    
        org/jruby/RubyKernel.java:961:in `require'                                    
        C:/opt/jruby/lib/ruby/stdlib/rubygems/core_ext/kernel_require.rb:55:in `requir
e'                                                                                    
        C:/opt/jruby/lib/ruby/stdlib/fiddle/jruby.rb:2:in `<main>'                    
        org/jruby/RubyKernel.java:961:in `require'                                    
        C:/opt/jruby/lib/ruby/stdlib/rubygems/core_ext/kernel_require.rb:55:in `requir
e'                                                                                    
        C:/opt/jruby/lib/ruby/stdlib/fiddle.rb:1:in `<main>'                          
        org/jruby/RubyKernel.java:961:in `require'                                    
        C:/opt/jruby/lib/ruby/stdlib/rubygems/core_ext/kernel_require.rb:55:in `requir
e'                                                                                    
        C:/opt/jruby/lib/ruby/stdlib/fiddle.rb:3:in `<main>'                          
        org/jruby/RubyKernel.java:961:in `require'                                    
        C:/opt/jruby/lib/ruby/stdlib/rubygems/core_ext/kernel_require.rb:55:in `requir
e'                                                                                    
        C:/opt/jruby/lib/ruby/stdlib/fiddle/import.rb:1:in `<main>'                   
        org/jruby/RubyKernel.java:961:in `require'                                    
        C:/opt/jruby/lib/ruby/stdlib/rubygems/core_ext/kernel_require.rb:55:in `requir
e'                                                                                    
        C:/opt/jruby/lib/ruby/stdlib/fiddle/import.rb:2:in `<main>'                   
        org/jruby/RubyKernel.java:961:in `require'                                    
        C:/opt/jruby/lib/ruby/stdlib/rubygems/core_ext/kernel_require.rb:55:in `requir
e'                                                                                    
        C:/opt/jruby/lib/ruby/stdlib/win32/importer.rb:1:in `<class:(root)>'          
        C:/opt/jruby/lib/ruby/stdlib/win32/importer.rb:2:in `<class:(root)>'          
        C:/opt/jruby/lib/ruby/stdlib/win32/registry.rb:1:in `<main>'                  
        org/jruby/RubyKernel.java:961:in `require'                                    
        C:/opt/jruby/lib/ruby/stdlib/rubygems/core_ext/kernel_require.rb:55:in `requir
e'                                                                                    
        C:/opt/jruby/lib/ruby/stdlib/win32/registry.rb:2:in `<main>'                  
        org/jruby/RubyKernel.java:961:in `require'                                    
        C:/opt/jruby/lib/ruby/stdlib/rubygems/core_ext/kernel_require.rb:55:in `requir
e'                                                                                    
        C:/opt/jruby/lib/ruby/stdlib/win32/resolv.rb:1:in `<main>'                    
        org/jruby/RubyKernel.java:961:in `require'                                    
        C:/opt/jruby/lib/ruby/stdlib/rubygems/core_ext/kernel_require.rb:55:in `requir
e'                                                                                    
        C:/opt/jruby/lib/ruby/stdlib/win32/resolv.rb:7:in `<main>'                    
        org/jruby/RubyKernel.java:961:in `require'                                    
        C:/opt/jruby/lib/ruby/stdlib/rubygems/core_ext/kernel_require.rb:55:in `requir
e'                                                                                    
        C:/opt/jruby/lib/ruby/stdlib/resolv.rb:1:in `<main>'                          
        org/jruby/RubyKernel.java:961:in `require'                                    
        C:/opt/jruby/lib/ruby/stdlib/rubygems/core_ext/kernel_require.rb:55:in `requir
e'                                                                                    
        C:/opt/jruby/lib/ruby/stdlib/resolv.rb:177:in `Hosts'                         
        C:/opt/jruby/lib/ruby/stdlib/resolv.rb:175:in `Resolv'                        
        C:/opt/jruby/lib/ruby/stdlib/resolv.rb:40:in `(root)'                         
        C:/opt/jruby/lib/ruby/stdlib/rubygems/remote_fetcher.rb:1:in `(root)'         
        C:/opt/jruby/lib/ruby/stdlib/rubygems/remote_fetcher.rb:7:in `(root)'         
        C:/opt/jruby/lib/ruby/stdlib/rubygems/spec_fetcher.rb:1:in `(root)'           
        C:/opt/jruby/lib/ruby/stdlib/rubygems/spec_fetcher.rb:2:in `<main>'           
        org/jruby/RubyKernel.java:979:in `load'                                       
        C:/opt/jruby/bin/gem:4:in `<main>'                                            
ERROR:  While executing gem ... (NoMethodError)                                       
    undefined method `invoke_with_build_args' for nil:NilClass                        
        C:/opt/jruby/lib/ruby/stdlib/rubygems/command_manager.rb:169:in `process_args'
                                                                                      
        C:/opt/jruby/lib/ruby/stdlib/rubygems/command_manager.rb:139:in `run'         
        C:/opt/jruby/lib/ruby/stdlib/rubygems/gem_runner.rb:55:in `run'               
        C:/opt/jruby/bin/jgem:21:in `<main>'                                          
        org/jruby/RubyKernel.java:979:in `load'                                       
        C:/opt/jruby/bin/gem:4:in `<main>'                                            
Exception `Gem::SystemExitException' at C:/opt/jruby/lib/ruby/stdlib/rubygems/user_int
eraction.rb:392 - Exiting RubyGems with exit_code 1                                   

@enebo
Copy link
Member

enebo commented Apr 19, 2017

I am not sure what is up. Should ffi/libc actually be loading? I do not think some of this stuff exists on windows. Can someone else try checking out head of headius fiddle-fixes and see what they see. This still sort of feels like it is going down the wrong path.

@enebo
Copy link
Member

enebo commented Apr 20, 2017

@headius @jellymann ah I should point out in my output above I commented out sys_errlist in libc because it could not find it...once it was commented out it just could not find the next field 'sys_nerr'. So your output will be slightly different.

@danini-the-panini
Copy link
Contributor

@enebo Perhaps they were removed in a later version of windows. I only have Windows 7 available to me, and those functions exist.

@enebo
Copy link
Member

enebo commented Apr 24, 2017

@jellymann wow I did just build a new Win 10 box so it is possible but I sure hope not. Also I am thinking MRI would not work either then (at least FFI). I will see if I can figure more out I guess.

@danini-the-panini
Copy link
Contributor

@enebo The reason I pulled in LibC is to get a handle on the malloc and free function. We could probably then just take out all the functions except those two, and then not worry about what functions exist on what platforms.

I think MRI works because it doesn't load LibC like this at startup, since it uses it's own malloc/free functions that are linked at compile time. JRuby obviously can't do that since it's not C 😅
(https://github.com/ruby/ruby/blob/5b3c9fc9623feb3a94291048f64e4a8d061081b1/gc.c#L8022)

@enebo
Copy link
Member

enebo commented Apr 25, 2017

@jellymann do you think you could roll something which only binds those two functions? Also I don't grok why this works on win7 but not win10. Something seems strange there. Windows does occasionally remove stuff but I am surprised in that it all seems removed.

@larskanis
Copy link

While working on an FFI issue on MRI I stumbled across several of the issues that are addressed in this PR. I was just about to prepare a similar PR with a subset of these changes, but then noted this one. It would be great, if this PR could be finished!

To the LibC issue: many of the functions are not available on Windows (unless on cygwin). However isn't it better to implement Fiddle::Pointer.malloc on top of FFI::MemoryPointer than on LibC functions?

larskanis added a commit to larskanis/ffi that referenced this pull request Jul 24, 2017
This is particular interesting because FFI stores thread local call info
on the stack, which is retrieved when a callback is received.
This is something that Fiddle doesn't do, but a callback should be handled
gracefully nevertheless.

The test "from fiddle to ffi" fails without the previous commit.

These tests succeed with Ruby-2.3+ and on JRuby (provided
jruby/jruby#4518 will be merged).
larskanis added a commit to larskanis/ffi that referenced this pull request Jul 24, 2017
This is particular interesting because FFI stores thread local call info
on the stack, which is retrieved when a callback is received.
This is something that Fiddle doesn't do, but a callback should be handled
gracefully nevertheless.

The test "from fiddle to ffi" fails without the previous commit.

These tests succeed on MRI and on JRuby (provided
jruby/jruby#4518 will be merged).
larskanis added a commit to larskanis/ffi that referenced this pull request Jul 26, 2017
This is particular interesting because FFI stores thread local call info
on the stack, which is retrieved when a callback is received.
This is something that Fiddle doesn't do, but a callback should be handled
gracefully nevertheless.

The test "from fiddle to ffi" fails without the previous commit.

These tests succeed on MRI and on JRuby (provided
jruby/jruby#4518 will be merged).
@danini-the-panini
Copy link
Contributor

@headius @enebo Hi All, sorry I haven't been active on this lately. I had to sell my PC a few months ago, but I have recently aquired a new one which is running Windows 10 this time.

I pulled down my ds-fiddle-windows branch and it seems to work when running bin\jruby -d -S gem list. Perhaps I should make a PR for it into this branch?

I'm also experiencing some new issues on my side, for example I cannot run the MRI specs unless I run them in Windows 7 compatibility mode, otherwise I get a exit code error -1073741819. Do any of you know what could be causing this?

In other news, I was able to successfully run some of the examples from Mittsu on JRuby without too much tweaking. There still seems to be a discrepancy between JRuby and MRI with regard to what can and cannot be implicitly converted to a pointer. JRuby/JFFI seem to only support Nil, String and MemoryIO, whereas MRI can convert integers and Fiddle::Closures as well. Not sure what else there is, but I will start investigating. Maybe we need to define more PointerParameterStrategy implementations for the various types?

@headius
Copy link
Member Author

headius commented Dec 6, 2017

@jellymann PR sure! Exit code error: I have not seen that but we can work with you to look into it. Running mittsu: excellent! We have had few if any changes to fiddle since your earlier PRs but we definitely want to align things with MRI. Let's do it!

cc @tenderlove since he wrote fiddle and may be able to offer some advice.

Daniel Smith added 2 commits December 7, 2017 11:14
This was causing `gem list` to fail. Apparently, there are quite a number of libc functions and variables that are not available on Windows.
Even though they're wrapped in begin/recue blocks (since they're also known to not be available in OSX), I'm avoiding them if Windows so that the extra debug noise doesn't show up. If not, why not?
@danini-the-panini
Copy link
Contributor

@headius Here is the PR: headius#4

larskanis added a commit to larskanis/ffi that referenced this pull request Jan 24, 2018
This is particular interesting because FFI stores thread local call info
on the stack, which is retrieved when a callback is received.
This is something that Fiddle doesn't do, but a callback should be handled
gracefully nevertheless.

The test "from fiddle to ffi" fails without the previous commit.

These tests succeed on MRI and on JRuby (provided
jruby/jruby#4518 will be merged).
larskanis added a commit to larskanis/ffi that referenced this pull request Feb 13, 2018
This is particular interesting because FFI stores thread local call info
on the stack, which is retrieved when a callback is received.
This is something that Fiddle doesn't do, but a callback should be handled
gracefully nevertheless.

The test "from fiddle to ffi" fails without the previous commit.

These tests succeed on MRI and on JRuby (provided
jruby/jruby#4518 will be merged).
larskanis added a commit to larskanis/ffi that referenced this pull request Feb 13, 2018
This is particular interesting because FFI stores thread local call info
on the stack, which is retrieved when a callback is received.
This is something that Fiddle doesn't do, but a callback should be handled
gracefully nevertheless.

The test "from fiddle to ffi" fails without the previous commit.

These tests succeed on MRI and on JRuby (provided
jruby/jruby#4518 will be merged).
larskanis added a commit to larskanis/ffi that referenced this pull request Jan 25, 2019
This is particular interesting because FFI stores thread local call info
on the stack, which is retrieved when a callback is received.
This is something that Fiddle doesn't do, but a callback should be handled
gracefully nevertheless.

The test "from fiddle to ffi" fails without the previous commit.

These tests succeed on MRI and on JRuby (provided
jruby/jruby#4518 will be merged).
@headius
Copy link
Member Author

headius commented Oct 2, 2024

Pinging this one after 7 years of neglect. 😭

I merged in headius#4 just now... sorry we lost track of this so many years ago.

If you are still out there @danini-the-panini we'd love to revisit this changes in the fiddle gem, where we are working to import the JRuby logic: ruby/fiddle#147

@danini-the-panini
Copy link
Contributor

danini-the-panini commented Oct 3, 2024

@headius I'm still out here, and would love to help! I'm a little over-capacity at the moment both professionally and personally, so I can't promise anything. At the very least I can try get Mittsu running in a dev environment on Windows again when I find a gap to breathe.

@headius
Copy link
Member Author

headius commented Oct 10, 2024

@danini-the-panini That's great to hear! The FFI-based fiddle has been incorporated into the fiddle gem officially, and will provide fiddle functionality for both JRuby and TruffleRuby going forward! If you get some cycles to return to this work, definitely go to the source: https://github.com/ruby/fiddle.

For now I will close this PR, since it should be rehomed on the fiddle repository anyway. Any fixes you can provide will be greatly appreciated!

I will be standing by to help!

@headius headius closed this Oct 10, 2024
@enebo enebo added this to the Non-Release milestone Nov 4, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants