First, this ignores the entire file that relies on the MRI file structure, as it seems unlikely that any of those tests will ever be applicable.

Second, it adds the test_sexp.rb file which was absent from the index for some unknown reason. It also introduces a couple of excludes for failing tests in that file.

Third, it clears the excludes for a couple of tests that are currently passing, and updates the description for one of the excludes.

Otherwise, an invalid byte sequence inside an identifier causes Ripper to continuously parse the same invalid byte sequence over and over.

They are identical (after fixing tokadd_mbchar)

Otherwise, it doesn't recognize variables introduced in the block (parameters), resulting in vcall nodes instead of var_ref

The previous implementation called assignable on a bunch of things that were unlikely to be strings, while only supporting string arguments. This changes the logic so that it relies on getting the identifier value from the lexer instead.

In addition, this inlines the logic from assignable in all cases where the implementation is known, and splits the cases for identifiers and constants, as they have very little to do with each other. Given that the parser has inlined the definition of MRI's user_varaible and keyword variable, it seems to make sense to also inline the assignable actions.

While inlining the assignable implementation, it seamed to make sense to also inline is_id_var (and remove its unused argument).

The corresponding construct was in the main parser, and allowed it correctly parse "m x do; end", where Ripper would fail and assign the block to x rather than m.

It seems like MRI is doing this.

It might not be important, but MRI seems to do this.

Instead of ending the heredoc before emitting the final string content part, this instead just emits the final string content part and positions the cursor right before the end marker, such that the next call to the heredoc term should end the heredoc using the early return.

The motivation for doing this was that with ripper the final string content of a heredoc wouldn't be dispatched to any token handler, but instead just propagated.

The old behavior was clearly wrong for this corner-case, but this approach might have other problems.

As an added bonus, the parsing of magic comment key values no longer stops at the first unrecognized key, and thus writing

# -*- warn_past_scope: true; coding: ISO-8859-1 -*-

will now apply the encoding even if JRuby doesn't support the magic comment attribute warn_past_scope.

Improve compatibility with MRI's Ripper for JRuby 9.1