Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Ripper fails to lexical analyse around escaped newline on JRuby #4787

Closed
aycabta opened this issue Sep 10, 2017 · 2 comments
Closed

Ripper fails to lexical analyse around escaped newline on JRuby #4787

aycabta opened this issue Sep 10, 2017 · 2 comments
Labels
parser
Milestone
JRuby 9.1.14.0

Comments

@aycabta
Copy link

aycabta commented Sep 10, 2017 •
edited
Loading

The code below is valid on JRuby 9.1.13.0, CRuby 2.4.1 and more old days:

1 == 2 ？ \
  puts(true) : \
  puts(false)

But Ripper on JRuby fails to lexical analyse source code around escaped newline. It returns strange tokens.

I'm in need when using Ripper for RDoc.

Environment

$ uname -a
Linux x250 4.11.6-3-ARCH #1 SMP PREEMPT Thu Jun 22 12:21:46 CEST 2017 x86_64 GNU/Linux
$ ruby -v
jruby 9.1.13.0 (2.3.3) 2017-09-06 8e1c115 OpenJDK 64-Bit Server VM 25.131-b11 on 1.8.0_131-b11 +jit [linux-x86_64]

Source Code

require 'ripper'
require 'pp'

pp Ripper.lex("1 == 2 ? \\\n  puts(true) : \\\n  puts(false)")

Expected Behavior

This is on CRuby 2.4.1.

[[[1, 0], :on_int, "1"],
 [[1, 1], :on_sp, " "],
 [[1, 2], :on_op, "=="],
 [[1, 4], :on_sp, " "],
 [[1, 5], :on_int, "2"],
 [[1, 6], :on_sp, " "],
 [[1, 7], :on_op, "?"],
 [[1, 8], :on_sp, " "],
 [[1, 9], :on_sp, "\\\n"],
 [[2, 0], :on_sp, "  "],
 [[2, 2], :on_ident, "puts"],
 [[2, 6], :on_lparen, "("],
 [[2, 7], :on_kw, "true"],
 [[2, 11], :on_rparen, ")"],
 [[2, 12], :on_sp, " "],
 [[2, 13], :on_op, ":"],
 [[2, 14], :on_sp, " "],
 [[2, 15], :on_sp, "\\\n"],
 [[3, 0], :on_sp, "  "],
 [[3, 2], :on_ident, "puts"],
 [[3, 6], :on_lparen, "("],
 [[3, 7], :on_kw, "false"],
 [[3, 12], :on_rparen, ")"]]

Actual Behavior

This is on JRuby 9.1.3.0 as described previously.

[[[1, 0], :on_int, "1"],
 [[1, 1], :on_sp, " "],
 [[1, 2], :on_op, "=="],
 [[1, 4], :on_sp, " "],
 [[1, 5], :on_int, "2"],
 [[1, 6], :on_sp, " "],
 [[1, 7], :on_op, "?"],
 [[1, 8], :on_sp, " "],
 [[1, 9], :on_ident, "\\\n"],
 [[2, 0], :on_sp, "  "],
 [[2, 2], :on_lparen, "puts("],
 [[2, 7], :on_kw, "true"],
 [[2, 11], :on_rparen, ")"],
 [[2, 12], :on_sp, " "],
 [[2, 13], :on_op, ":"],
 [[2, 14], :on_ident, "\\\n"],
 [[3, 0], :on_sp, "  "],
 [[3, 2], :on_lparen, "puts("],
 [[3, 7], :on_kw, "false"],
 [[3, 12], :on_rparen, ")"]]
aycabta added a commit to aycabta/rdoc that referenced this issue Sep 10, 2017
@aycabta
Skip a test on JRuby
a67b8bc 
The test_accept_verbatim_nl_after_backslash is failed on JRuby
because Ripper on JRuby fails to lexical analyse around escaped newline.
I filed a issue for it: jruby/jruby#4787
The "skip" should be removed after the issue is resolved.
@aycabta aycabta mentioned this issue Sep 10, 2017
Merged
@enebo
Copy link
Member

enebo commented Oct 30, 2017

If I put original heredoc into string like:

contents = <<EOS
1 == 2 ？ \
  puts(true) : \
  puts(false)
EOS

then all is happy. when put into a interpolated string with \ then the lexer will actually see the \ character in the stream and it all gets weird. Makes me wonder how well we actually execute that code too...

enebo added a commit that referenced this issue Oct 30, 2017
@enebo
Fixes #4787 Ripper fails to lexical analyse around escaped newline on…
2d078e3 
… JRuby.

Missing a dispatch token (tSP) when we see \\ and then \n.
@enebo enebo added this to the JRuby 9.1.14.0 milestone Oct 30, 2017
@enebo enebo closed this as completed in 3b6f308 Oct 30, 2017
@enebo enebo added the parser label Oct 30, 2017
@aycabta
Copy link
Author

aycabta commented Oct 30, 2017

Thank you!

@aycabta aycabta mentioned this issue Oct 30, 2017
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
parser
Projects
None yet
Milestone
JRuby 9.1.14.0
Development

No branches or pull requests
2 participants
@enebo @aycabta