Skip to content
Permalink

Comparing changes

Choose two branches to see what’s changed or to start a new pull request. If you need to, you can also or learn more about diff comparisons.

Open a pull request

Create a new pull request by comparing changes across two branches. If you need to, you can also . Learn more about diff comparisons here.
base repository: jruby/jruby
Failed to load repositories. Confirm that selected base ref is valid, then try again.
Loading
base: f7f31b427bdc
Choose a base ref
...
head repository: jruby/jruby
Failed to load repositories. Confirm that selected head ref is valid, then try again.
Loading
compare: 0a4be2d255e8
Choose a head ref
  • 14 commits
  • 17 files changed
  • 2 contributors

Commits on Dec 16, 2017

  1. Update the MRI ripper spec excludes 

    First, this ignores the entire file that relies on the MRI file structure, as it seems unlikely that any of those tests will ever be applicable.

Second, it adds the test_sexp.rb file which was absent from the index for some unknown reason. It also introduces a couple of excludes for failing tests in that file.

Third, it clears the excludes for a couple of tests that are currently passing, and updates the description for one of the excludes.
    @grddev
    grddev committed Dec 16, 2017
    Copy the full SHA
    7ecf6ee View commit details

  2. Return false on tokadd_mbchar failure 

    Otherwise, an invalid byte sequence inside an identifier causes Ripper to continuously parse the same invalid byte sequence over and over.
    @grddev
    grddev committed Dec 16, 2017
    Copy the full SHA
    62462ec View commit details

  3. Call tokadd_mbchar from tokenAddMBC 

    They are identical (after fixing tokadd_mbchar)
    @grddev
    grddev committed Dec 16, 2017
    Copy the full SHA
    32d2c20 View commit details

  4. Make opt_bv_decl consistent with MRI

    @grddev
    grddev committed Dec 16, 2017
    Copy the full SHA
    bdc6d11 View commit details

  5. Use current scope in is_id_var 

    Otherwise, it doesn't recognize variables introduced in the block (parameters), resulting in vcall nodes instead of var_ref
    @grddev
    grddev committed Dec 16, 2017
    Copy the full SHA
    4a1b16d View commit details

  6. Rework Ripper scope variable analysis 

    The previous implementation called assignable on a bunch of things that were unlikely to be strings, while only supporting string arguments. This changes the logic so that it relies on getting the identifier value from the lexer instead.

In addition, this inlines the logic from assignable in all cases where the implementation is known, and splits the cases for identifiers and constants, as they have very little to do with each other. Given that the parser has inlined the definition of MRI's user_varaible and keyword variable, it seems to make sense to also inline the assignable actions.

While inlining the assignable implementation, it seamed to make sense to also inline is_id_var (and remove its unused argument).
    @grddev
    grddev committed Dec 16, 2017
    Copy the full SHA
    5cbaac8 View commit details

  7. Align Ripper parser with main Ruby parser 

    Not 100% sure about the impact of this, but it seems unlikely that having two separate Ruby grammars for Ripper and the main parser would be a good thing.
    @grddev
    grddev committed Dec 16, 2017
    Copy the full SHA
    1763e4e View commit details

  8. Add command-args-state begin to Ripper 

    The corresponding construct was in the main parser, and allowed it correctly parse "m x do; end", where Ripper would fail and assign the block to x rather than m.
    @grddev
    grddev committed Dec 16, 2017
    Copy the full SHA
    c0cfb5f View commit details

  9. Reset the cmd-args-state from Ripper push-local-scope 

    It seems like MRI is doing this.
    @grddev
    grddev committed Dec 16, 2017
    Copy the full SHA
    5bd4137 View commit details

  10. Clear command-args-state in brace blocks 

    It might not be important, but MRI seems to do this.
    @grddev
    grddev committed Dec 16, 2017
    Copy the full SHA
    56e54a7 View commit details

  11. Rework handling of heredoc end 

    Instead of ending the heredoc before emitting the final string content part, this instead just emits the final string content part and positions the cursor right before the end marker, such that the next call to the heredoc term should end the heredoc using the early return.

The motivation for doing this was that with ripper the final string content of a heredoc wouldn't be dispatched to any token handler, but instead just propagated.

The old behavior was clearly wrong for this corner-case, but this approach might have other problems.
    @grddev
    grddev committed Dec 16, 2017
    Copy the full SHA
    f1234e8 View commit details

  12. Dispatch on_operator_ambiguous for Ripper

    @grddev
    grddev committed Dec 16, 2017
    Copy the full SHA
    b702e52 View commit details

  13. Dispatch on_magic_comment for Ripper 

    As an added bonus, the parsing of magic comment key values no longer stops at the first unrecognized key, and thus writing

# -*- warn_past_scope: true; coding: ISO-8859-1 -*-

will now apply the encoding even if JRuby doesn't support the magic comment attribute warn_past_scope.
    @grddev
    grddev committed Dec 16, 2017
    Copy the full SHA
    d0f5e04 View commit details

  14. Merge pull request #4898 from grddev/ripper-fixes 

    Improve compatibility with MRI's Ripper
    @enebo
    enebo authored Dec 16, 2017
    Copy the full SHA
    0a4be2d View commit details
5 changes: 2 additions & 3 deletions core/src/main/java/org/jruby/ext/ripper/HeredocTerm.java
View file
Original file line number Diff line number Diff line change
@@ -201,10 +201,9 @@ public int parseString(RipperLexer lexer, LexerSource src) throws java.io.IOExce
str = tok;
}

lexer.dispatchHeredocEnd();
lexer.heredoc_restore(this);
lexer.setStrTerm(new StringTerm(-1, '\0', '\0'));
lexer.pushback(c);
lexer.setValue(lexer.createStr(str, 0));
lexer.flush_string_content(lexer.getEncoding());
return RubyParser.tSTRING_CONTENT;
}
}
21 changes: 7 additions & 14 deletions core/src/main/java/org/jruby/ext/ripper/RipperLexer.java
View file
Original file line number Diff line number Diff line change
@@ -98,8 +98,12 @@ public class RipperLexer extends LexingCommon {
public boolean ignoreNextScanEvent = false;

protected void ambiguousOperator(String op, String syn) {
warn("`" + op + "' after local variable or literal is interpreted as binary operator");
warn("even though it seems like " + syn);
parser.dispatch("on_operator_ambiguous", getRuntime().newSymbol(op), getRuntime().newString(syn));
}

protected void onMagicComment(String name, ByteList value) {
super.onMagicComment(name, value);
parser.dispatch("on_magic_comment", getRuntime().newString(name), getRuntime().newString(value));
}

private int getFloatToken(String number, int suffix) {
@@ -2346,18 +2350,7 @@ public void readUTFEscapeRegexpLiteral(ByteList buffer) throws IOException {
// This is different than MRI in that we return a boolean since we only care whether it was added
// or not. The MRI version returns the byte supplied which is never used as a value.
public boolean tokenAddMBC(int first_byte, ByteList buffer) {
int length = precise_mbclen();

if (length <= 0) {
compile_error("invalid multibyte char (" + getEncoding() + ")");
return false;
}

tokAdd(first_byte, buffer); // add first byte since we have it.
lex_p += length - 1; // we already read first byte so advance pointer for remainder
if (length > 1) tokCopy(length - 1, buffer); // copy next n bytes over.

return true;
return tokadd_mbchar(first_byte, buffer);
}

// MRI: parser_tokadd_utf8 sans regexp literal parsing
Loading