It seems like the complete log was truncated, or else the bits I need didn't get in there. Can you do another attempt without -X and gist that too?

We do build with Java 7 periodically, so that shouldn't be the issue. Unfortunately I can't see the actual error.

@RochesterinNYC are you sure it works with 7u111-2.6.7-0ubuntu0.14.04.2 as it is only two days older then 7u111-2.6.7-0ubuntu0.14.04.3 but has some mentioning of 'unsafe' with it - these two unsafe methods could be the cause of the failure. to the see the compilation error it shold help to add -Dmaven.compiler.fork=false to commandline. with the forked compiler most output is hidden.

I think the full log output is available here: https://gist.githubusercontent.com/RochesterinNYC/fac1f36bb98ded1f5a135f10b2c8741c/raw/ec88989ef61a62fe4aca23030b482113720d2b73/compile.log

We feel that it works with 7u111-2.6.7-0ubuntu0.14.04.2 as our last successful build of jruby was completed August 10th 2016 at ~12 AM. The next build occurred on August 17th 2016 at 12:30 AM and it failed to build jruby. As we use locked versions of Ant and Maven in the build process, but use apt-get install for openjdk-7-jdk, this is the only build input that could have changed. So for the August 10th build, 7u111-2.6.7-0ubuntu0.14.04.2 was used, and for the failed August 17th build, 7u111-2.6.7-0ubuntu0.14.04.3 was used.

Hm, additionally, we're having some trouble trying to build with -Dmaven.compiler.fork=false. We specify this flag/argument but the output logs show that fork is ultimately set to true: https://gist.githubusercontent.com/RochesterinNYC/fac1f36bb98ded1f5a135f10b2c8741c/raw/eb703754f25d7df987c7530ece60e59ab24c7a88/compile-with-attempted-fork-false.log

The specified "fork=false" shows up in the logs if you search for them, but fork is ultimately set as "fork = true".

@RochesterinNYC ok I can reproduce the error with your Docker setup. can you give me some hints on how I can find the output and how to maybe run maven manually ? bit lost in the filestructure in /binary-builder

It shouldn't be too difficult to take that last javac command line and run it. It's the line starting with [DEBUG] -d /tmp/x86_64-linux-gnu (the second one in your output) and ending with -XDignore.symbol.file=true. Everything from -d and on should be passed to javac, and the whole command probably needs to go in a shell script to be runnable (way too long for command line).

Ideally, that will show us why the javac step is blowing up. I sure hate Maven for hiding the actual error.

@headius the error I see is:

checking org.jruby.util.unsafe.UnsafeHolder]
/tmp/x86_64-linux-gnu/ports/jruby/9.1.2.0_ruby-2.3.0/jruby-9.1.2.0/core/src/main/java/org/jruby/util/unsafe/UnsafeHolder.java:96: error: cannot find symbol
        U.fullFence();
         ^
  symbol:   method fullFence()
  location: variable U of type Unsafe
/tmp/x86_64-linux-gnu/ports/jruby/9.1.2.0_ruby-2.3.0/jruby-9.1.2.0/core/src/main/java/org/jruby/util/unsafe/UnsafeHolder.java:100: error: cannot find symbol
        U.loadFence();
         ^
  symbol:   method loadFence()
  location: variable U of type Unsafe
/tmp/x86_64-linux-gnu/ports/jruby/9.1.2.0_ruby-2.3.0/jruby-9.1.2.0/core/src/main/java/org/jruby/util/unsafe/UnsafeHolder.java:104: error: cannot find symbol
        U.storeFence();
         ^
  symbol:   method storeFence()
  location: variable U of type Unsafe

so I missed the bootclasspath and this line could be the problem:

    <compilerArg>-J-Xbootclasspath/p:${settings.localRepository}/com/headius/unsafe-mock/8.92.1/unsafe-mock-8.92.1.jar</compilerArg>

i.e. maven can not resolve ${settings.localRepository} - I do remember having problems with this before but I am guessing here.

why is thie -J-Xbootclasspath and not -Xbootclasspath ?

@mkristian So it's not picking up our bootstrap unsafe mock then I guess?

@mkristian The -J is because it's being passed to javac.

@headius so I am confused as if I change the core/pom.rb to use -Xbootclasspath/p:${unsafe.jar} instead of -J-Xbootclasspath/p:${unsafe.jar} then the compile phase passes.

@mkristian I feel like we've had this discussion before...now I'm confused too.

@headius both seems to valid if I look at javac -help and javac -X so I am OK to add both variants. locally on jdk8 on macOS it works with both.

Is anything needed from us in terms of reproducibility or debug logs?

@RochesterinNYC I can reproduce the error and I have a workaround. not sure why things changed with jdk version. will produce a PR and let the jruby core team decide if they want to go with it.

Hello,

We're not sure that this fixes our problem. We tried to build JRuby 9.1.3.0 in the manner described above, however the build still fails. The command we ran in our container:

./bin/binary-builder --name=jruby --version=9.1.3.0_ruby-2.3.1 --sha256=2acde32e0069a2d9a5593c45386545bf66eb08bdb3d514c49e8258980db5fcab

In the logs, we saw that -Xbootclasspath was passed to the java compiler:

-Xbootclasspath/p:/root/.m2/repository/com/headius/unsafe-mock/8.92.1/unsafe-mock-8.92.1.jar

@sesmith177 Well that's frustrating :-( I'm not sure what else would be causing a problem.

Can you try removing all bootclasspath lines from the compile plugins (use @mkristian's patch as a guide) and see if we at least get a reasonable error?

For background: the unsafe-mock library was added so we could compile code that uses Unsafe.fullFence -- a Java 8 feature -- even when building using Java 7. fullFence is used mostly to guarantee the integrity of array-based instance variable tables when mutating across threads. Java 7 does this through other means.

There must be a way to get more information out of that blasted compiler plugin!

Reopened and targeted to 9.1.5.0.

Hello @headius,

We removed the bootclasspath lines from core/pom.rb and core/pom.xml. The build still failed; logs are here.

Thanks

Ok, so this appears to be something other than the bootclasspath failing. One more thing to try: just remove the -J version of that flag. I'm wondering if forcing our Unsafe into the underlying JVM is breaking javac, where the plain -X version would not have that effect.

@headius Removing the '-J-Xbootclasspath/p:${unsafe.jar}' line caused it to compile correctly with no errors.

I was sure I got it to work before I suggested a patch for our build. well, did not test the patch on the docker container :(

still think we can use maven profiles to the bootclasspath for the different jdks the right way. have to play around with this.

For context, the way I've been making these jruby source edits and playing around is once you boot up the container and execute a failed build of jruby, the jruby source tarball will be at path_to/binary-builder/ports/archives/jruby-src-9.1.5.0.tar.gz. I untar that and make whatever edits necessary, tar it back up, and ensure the modified tarball is at that path with that filename. Then, when you execute binary-builder again, it won't download jruby source again but rather just use that tarball for building.

@headius Removing the '-J-Xbootclasspath/p:${unsafe.jar}' line caused it to compile correctly with no errors.

Of course it did! 😞 I wonder if we could fudge this with method handles and eliminate the mocks. What a pain.

One other thing did occur to me...we are backing off from forcing instance variables to be volatile in favor of an explicit API. So that would reduce our need for the Java 8+ fence methods on Unsafe.

I also did a quick benchmark of a MethodHandle-based fullFence that doesn't require a mock, and it ends up with basically the same performance as calling Unsafe directly. Pretty impressive.

Are there any other testing or logs or things you'd want us to try to provide feedback/info?

I think this requires heavier lifting on our side, like moving both the unsafe mock usage AND the utility library wrapped around it to a separate library, so we can build it however we please without affecting the JRuby codebase. That's not going to happen for 9.1.6.0 but it could happen for 9.1.7.0.

Then we could just drop the pesky bootclasspath flag entirely.

@headius thanks for the update. Please keep us involved and let us know if there's anything the CF Buildpacks Team can do.