(I looked at the other problem, with the stale .jar files left in a temp folder. It was very similar in nature: require 'openssl' and you get it. Same behavior there; it appears only with jruby-complete, not with the "normal" jruby.)

the original problem you are referring to was when you kill the jruby
process hard, then the JVM can not clean up files marked as
deleteOnExit() since the process is gone. this is true only for
jruby-complete or any scenario where you need to have native libraries or
jars loaded from within a jar-file (in your case the jruby-complete.jar)
which are copied to an temp file and used from there. you are saying that
on windows those file stay around, i.e. the JVM does not delete them on
exit.

Correct. But this time, it's not only when you kill the jruby process hard. It happens all the time, every time. It's totally deterministic. So, maybe this is an entirely different source of the problem than in the original issue.

Any ideas on where we can start digging in this? I just downloaded jruby and set up the local build environment. Verified that it's still a problem with latest HEAD.

If someone else wants to take over the debugging, feel free. 😄

@perlun that are the two places I know where temp files are created:
https://github.com/jruby/jruby/blob/master/core/src/main/java/org/jruby/util/JRubyClassLoader.java#L87
https://github.com/jnr/jffi/blob/master/src/main/java/com/kenai/jffi/internal/StubLoader.java#L344

I see the same issue on JRuby 1.7.24 that after exiting JRuby it leaves jffi*.dll file in temp directory (which is specified in java.io.tmpdir Java property).

In addition, I see jruby-* directories created with jruby*openssl.jar, jruby*bcpkix*.jar, and jruby*bcprov*.jar files created. These jruby-* directories in temp directory remain as well after exiting JRuby.

win won't delete any files that are "still" open ... which might just fit this case esp. the StubLoader's.
the native library gets loaded and it won't get unloaded until its finalizer is run (which is likely only around the class loader gets released) ... might just happen after the JVM cleans up the deleteOnExist temp files.

I think we should modify the code to clean up the tempfiles using a JVM shutdown hook instead: https://dzone.com/articles/know-jvm-series-2-shutdown

@perlun could you test it actually helps the windows case esp. when a temp file is a loaded .dll library ?
... cause deleteOnExit already does File#delete from a shutdown hook but maybe the order will matter.

we have/had a similar problem with a I think it is redstorm gem which spawns java/jruby processes and when they are not needed anymore they are kill hard, i.e. the JVM has no chance to cleanup anything. this resulted in filling up the /tmp directory (under linux) and eventually running out of disk space within a few days :(

the solution or workaround this problem is cleanup the /tmp directory on each new startup of jruby. the lookout-jruby is doing this: https://github.com/lookout/lookout-jruby/blob/master/lib/lookout/jruby.rb#L10

reading all this here, I think that could be at least something JRuby itself could do: cleanup stale temp files on startup. of course there is always a set of native dll and/or jars laying around but pollution of the temp space is limited.

If the processes are killed hard (with kill -9) then it is understandable that the temp files are not deleted. If JRuby process stops normally on Linux then it does cleanup of all temp files. But on Windows, unfortunately, no cleanup is done if the JRuby process stops normally. It would be very nice if it would be possible to fix this on Windows.

I do not think that it would be a good idea to try the cleanup of old files on each new startup of JRuby as temp directory might be different for each execution of JRuby. As well it might cause some side effects when JRuby will try to delete temp files that it didn't create (e.g. when several JRuby processes start at the same time and try to delete each others files). I think it would be better to leave such cleanup to be project / application specific.

For application specific cleanup on startup, it would be good to have some documentation where all created temp files are listed. Currently, I see a lot of them on Windows:

• jffi*.dll
• jar_cache*.tmp (in the content of this binary file I see references to org/jruby/ext/ classes)
• jffi*.tmp (it seems that earlier JRuby versions created that, not present for 1.7.24)
• jruby-* directories (with ssl jar files - these are now the largest and create several megabytes of new temp files for each JRuby run)

I'm wondering about jruby-* directories with jar files - why is it necessary to create temp jar files and load them? Why aren't the classes in these jar files included in the main jruby-complete.jar file?

@mkristian are these files held open by the process after startup? I guess for a dll maybe it has to still be there (although at some level that seems odd to me). Another solution would be to have loading delete them after the loading has occurred.

@enebo all these jffi*.dll, jar_cache*.tmp, jruby-*/* files are kept open by JVM process and therefore Windows does not allow to delete them. These files cannot be deleted also e.g. from Windows Explorer.

We currently have implemented a workaround which deletes these files during the application startup. We have our JRuby based JIRA plugin (which is OSGi component) and what we see that even when a plugin is disabled and JRuby instance is terminated (but the JVM process still continues to work) then still these temp files cannot be deleted. Only after JVM process restart we can delete previous temp files.

@rsim yeah I guess that was the answered I assumed based on us not deleting until shutdown hooks.

Another thought is versioning the dlls on windows and installing them in one location consistently with some checksum/signature check to know if it has been corrupted. Then it would only grow as large as jffi development grew which would not be extreme...

@enebo if technically it is impossible to delete these files on Windows during JRuby shutdown then it would be better to create them with a file name that uses some checksum/signature and not a random string. But that probably still needs to be in java.io.tmpdir directory as that's probably the only place where you can assume that write access will be provided.

using random strings inside the temp directory is security feature

@enebo the jffi is actually loading the native library and deletes it immediately, which works OK on unix.

Just clarifying what I've heard here... the jffi libraries can't be deleted while the JVM is running (or during deleteOnExit at JVM level) because it is still being accessed and Windows doesn't allow the delete. Right?

What if we had jffi on Windows unpack itself to the same filename every time, but base that filename on a SHA of the library? At least then there would only ever be one copy of a given library, even if it doesn't get deleted.

Oh, there's another option that I think Wayne used to fix some of these issues a long time ago: for a full JRuby dist, ship an unpacked version of all platforms jffi libraries. That way we can always load the one we want without unpacking anything to a temp location.

This would not help embedded use cases where there's no JRuby home on the filesystem, but it should help command-line JRuby users on Windows.

@headius I am pretty sure we ship unpack jffi libraries with the binary distributions of jruby

fixed filenames on tem directories gives any intruder the chance to replace the dll with his/her own dll and gaining privileges of the jruby process. but I am not a security guy - so maybe this is not a big deal

Also I believe people run into this via jruby-complete.jar as well. So I also think we should unpack to a consistent location and the security aspect of a predictable path??? Does not seem like a huge problem to me. I guess we could add a fingerprint checker to make sure it is not corrupt or replace?

@enebo just ask google on how to 'Safely Create Temporary Files' and their security implications of not doing so.

maybe it is more a *nix problem and not so much on windows those predictable temp filenames

not a security expert either ... but sounds like a valid security issue to have predictable tmp files on win.

although, it might just be that @enebo was suggesting using a predictable path within the jruby.home ...
did not look into how those .dll files are generated and used but there might be some options to avoid 'em, assuming they can be "pre-generated" in some way:

• for the .exe installer its easy - pre-generate during installation
• for .zip do as it goes now but also provide with a way to pre-generate (within the jruby.home)

@mkristian we could possibly SHA fingerprint the DLL and give a distinct name for the dll per version and verify that it has not been tampered with. This seems somewhat extreme to me but if we:
a) used same name for same version
b) verified file has contents we expect
We could be consistent and safe.

I guess on Windows I do not consider it to have the same level of mutl-userness that unix and /tmp have. I can see that someone can modify files in users temp directory as a non-Admin but so many windows apps run as admin I do not think our risks are really adding to the problem.

If we're already shipping the JFFI binaries unpacked in the dist, then why are we unpacking it at all on boot?

I can also confirm that the temp jar files are still being left in place, though I think we knew that much.

The jar files will be simpler to solve, since we already have modified the nested classloading logic to close them. They just need to be set up as deleteOnExit, I believe.

The jffi dll will be trickier. I'm not sure how we can force it to unload.

It looks like the best we can do would be to load the jffi DLL inside a separate classloader. When that classloader gets cleaned up, all native libraries it loaded will be unloaded (which is somewhat unfortunate given issues like #4506).

The simpler answer might be that we provide a way to unpack or install the DLL in a known location and always look for it in that location, comparing it with our built-in library so we know we're loading the right one. A single file will stick around, but multiple runs will not create new tempfiles.

Ok, I keep forgetting that we only do the classloader teardown when run from a ScriptingContainer. Modifying the JRuby runtime teardown to also do the classloader teardown appears to delete the temp jars just fine:

> java -jar .\maven\jruby-complete\target\jruby-complete-9.3.0.0-SNAPSHOT.jar -ryaml -e "YAML.parse('1')"
deleting C:\Users\headius\AppData\Local\Temp\jruby-13844\jruby7153466497775121249psych.jar

However I'm thinking we might want to modify this logic to instead delete those files using the JVM deleteOnExit logic, which I believe we can also use to eliminate the temporary directory (the directory lingers with the current logic). This will also help avoid deleting them if there's some straggling code still trying to use the file (though I'd expect that to be a bug since we are also explicitly closing the jar file).

I have tried a number of things and still we are unable to delete the leftover jar files. The jffi library files also remain, though I have not found any good way to eliminate them.

The jar files opened by our URLClassLoader refuse to be deleted, even if we explicitly close the classloader that references them. I tried modifying the current JRubyClassLoader structure to inject an additional child URLClassLoader, with logic to close it and with the temp files marked for deleteOnExit. It did not appear to have any impact and the jars continue to accumulate.

I think at this point, our best option will be to unpack to a common location and name, possibly via some configuration, and with file locking and/or other verification to confirm we are reusing the proper file.

The logic I am considering would use the same filename for the same jar file, only adding process-specific identifiers if the original file is present but unusable for some reason.

So the jffi dll would be unpacked to something like jffi-SHA.dll and loaded from that name. In order to confirm the library is the one we expect, the file would be locked for read (to prevent any TOC2TOU attacks), read fully and SHA hashed, and that hash confirmed against our in-jar dll. If the SHA hashes match, then we proceed to load the dll into the process.

In the same way, any nested jar files would be unpacked with a SHA hash and locked and confirmed before being loaded into our classloaders.

We would make a best effort to unpack these files only once, with a fallback to a filename like jffi-SHA-PID.dll with a warning to indicate the existing file is unusable.

This will limit the number of left-behind files to only those with unique SHA hashes. Since many of these files do not change for minor releases, that number should also be fewer than the number of JRuby versions run on a given system.

When the files are not in use, they can be safely deleted as before.

I have made a discovery that may solve our issues.

NIO.2, introduced in Java 7, opens up more low-level file operations than just going through FileInput/OuputStream or FileChannel. Among these is the option DELETE_ON_CLOSE option, which on Windows will actually use the native FILE_FLAG_DELETE_ON_CLOSE option to trigger Windows itself to delete the file when all open handles have been closed. This is being used by NIO.2's Files.createTempFile, and may make it possible for us to delete these files.

I pushed a possible workaround for this in jnr/jffi#99, which adds a jffi.extract.name property users can set to always unpack the dll to the same filename (and reuse it if a previous run already extracted it).

Marking this for 9.2.17.0 so we can decide if we want to ship this small enhancement.

After some discussion we decided not to do this in 9.2.17.0. We may backport into a future 9.2.x release if someone is able to help us verify that the additional property sufficiently resolves (or works around) this issue for the use cases it affects (primarily when deploying JRuby using the complete jar or inside a war/ear/app without a full distribution).

Hi @headius - thanks for this. I see that it made 9.2.18.0! I went back to update my post on compiling JARs with JRuby and ran into another observation.

I just tried it out:

• Windows 10 Professional
• JRuby 9.2.18.0
• openjdk version "1.8.0_265"
  ** OpenJDK Runtime Environment (AdoptOpenJDK)(build 1.8.0_265-b01)
  ** OpenJDK 64-Bit Server VM (AdoptOpenJDK)(build 25.265-b01, mixed mode)

The command I am using is:

java -Djffi.extract.name="ffi-single.jar" -Djffi.extract.dir="c:/Users/mohit/AppData/Local/Temp" -jar project1.jar input.txt

My observation is this:

• the ffi-single.jar works as expected. It creates it with that name, recreates it if it does not exist, and uses it if it does.
• However, I notice that we still get randomly named jruby6345690224849002223extract folders with jruby-core and jruby-stdlib JARS

Is there a way to not have new directories created for the core and stdlib JARS?

Thanks, as always.

@mohits Ok firstly I believe you want to name that file ffi-single.dll on Windows. The file that is being extracted is the native stub, which is a dynamic library.

Second... I thought jffi.extract.dir should have stopped the temp dirs from bring created. I will review those properties quick.

A quick look over the logic seems to confirm... if jffi.extract.dir is set, then jffiExtractDir will be set and it should use that directory rather than a generated name. Perhaps you could try to confirm this on your end?

Also, I see the logic will add an appropriate dynamic-library extension if you do not specify one in your jffi.extract.name, so I would guess your files are being unpacked as ffi-single.jar.dll.

@mohits Ok firstly I believe you want to name that file ffi-single.dll on Windows. The file that is being extracted is the native stub, which is a dynamic library.

@headius - yes, that was a mistake and it adds DLL by itself and it works. I changed that to DLL and that works correctly also. So, this works as expected.

A quick look over the logic seems to confirm... if jffi.extract.dir is set, then jffiExtractDir will be set and it should use that directory rather than a generated name. Perhaps you could try to confirm this on your end?

I am doing this and still seeing new directories...

java -Djffi.extract.dir="c:/Users/mohit/AppData/Local/Temp" -Djffi.extract.name="ffi-single.dll" -jar project1.jar

I have tried these variations:

• "c:/users" - forward slash
• "c:\users" - single backslash
• "c:\users" - double backslash
• "single" - not absolute path
• "c:/users/temp/" - ending with a slash

None of them work. The DLL is always created at %TEMP% for me, and the additional randomly named directory is created there each time.

Let me know if I can check anything else. It's the weekend soon, so I will be able to quickly try it out :)

OK, I looked at the code - https://github.com/jnr/jffi/pull/81/files
and I noticed that it says:
// try user-specified location only if present

So, I created the 'single' directory in my TEMP folder and I ran it again

java -Djffi.extract.dir="c:/Users/mohit/AppData/Local/Temp/single" -Djffi.extract.name="ffi-single.dll" -jar project1.jar

There is some progress: ffi-single.jar.dll is extracted to c:/users/mohit/AppData/Local/Temp/single/ffi-single.jar.dll
However, there is still a randomly generated folder (jruby______extract) in TEMP with these files:

jruby-core-9.2.18.0-complete.jar
jruby-stdlib-9.2.18.0.jar

Hope this helps.

@mohits Those other files are unpacked from the nested jar logic in JRuby, so that is a separate issue (though I can't find an issue for it at the moment). The problem is the same... Windows does not allow deleting open files, but we can't tell the JDK to close the jar before attempting to delete it.

The logic is here... maybe you can come up with a solution!

https://github.com/jruby/jruby/blob/master/core/src/main/java/org/jruby/util/JRubyClassLoader.java#L180-L229

See also #6218.

Thanks @headius - let me take a look. Basically, I was hoping that those files would also be extracted to the jffi.extract.dir path (though yes, I realise that the config name is different) since then we could use the same logic to know where to keep/ delete the files. I can imagine the couple of challenges there but I thought that those were handled with the jffi changes that you had made.

@mohits Those files are unpacked by JRuby so it is unrelated to jffi. The property to set is jruby.ji.nested.jar.tmpdir=blah or -Xji.nested.jar.tmpdir=blah passed to JRuby. The logic is there in https://github.com/jruby/jruby/blob/master/core/src/main/java/org/jruby/util/JRubyClassLoader.java#L157-L175.

I had forgotten this property even existed.

Oh it is also possible this will just pick a specific tmp dir, but still unpack the jars with a unique path or name every time. We may want to mimic the logic that jnr-ffi does for unpacking the jffi dll.

Thank you for digging into this. It's time for bed here, but I will try these within the next day or so, and create a proper example that people can refer to!

I'm going to have to untarget this, because we have no known fix and no plans to investigate further. The native libraries loaded by the JVM are notoriously difficult to clean up while that JVM is running, since they are loaded by the classloader and only unloaded (hopefully) once the classloader is dereferenced and can be garbage collected. This might inform a path forward, where we load the library into its own child classloader, but how that integrates with jffi across apps and JRuby instances is very much unclear.