RE: #4070, #3719

…in readonly transaction

Duplicate keys where just ignored before and inserted again.
Fix #4958.

Extra tests for symbol encoding Re: #4070 + #3719 and possibly #3880

specs for Unicode symbols

…eadonly-transaction

an example test case for #4770 that shows how unicode readings fails …

Fix Hash#rehash method for duplicate keys

… day

would come but I was hoping I could cheat with 8859_1 strings enough to merge
for 9.2.  Unfortunately, lack of encoding through method methods means we cannot
accurately throw properly encoded error messages since he have long lost the
encoding of the method.

So with that said, most people will not be able to notice this change.  All the
String methods for methods still exist.  These methods will assume all data is
UTF-8.  This could be a dicey gamble but since it is the default for Ruby it
likely will not impact anyone.  Also since UTF-8 is 7bit clean for ASCII we
should totally not have an issue there.

Some API signatures have changed.  This means 9.2 might end up causing some
issues if you are doing something very low-level with our code base (although
changes up to this point have already broken some APIs within parsing and IR
and that is unavoidable).  The method reader/writer maps needed their return
type changed from <String, DynamicMethod> to <ByteList, DynamicMethod>.  It
was an option to make a new name for methods which returned this but this is
so far down in our APIs I think we should run with it.  Also ProfiledMethod
has changed from String to ByteList (e.g. getName() -> ByteList from String).
Ultimately ProfiledMethod needs to be properly encoded to be able to report
the method in a properly encoded way...so there you have it.

Next phase is to change interpreters and the JIT to stop from going:
ByteList -> String -> ByteList.  I did not do that part here since this is
pretty large already.

And use the same function from both Ripper and main parser.

While not really related to Lexing, this is a component that is shared between Ripper and the main parser, and that seemed like the lesser evil.

It seems to have been `!ENCODING_IS_ASCII8BIT(str)` from the beginning, so I'm not sure why it was the opposite here.

The code was copied from Parser support, so it was clearly broken before, but it had to be fixed now as parts of the Ripper test suite relies on $! rather than explicitly catching the exception.

This uses the fact that Regexp tokenization is handled by a single StringTerm, and thus all tSTRING_CONTENT fragments are easily collectable until the tREGEXP_END comes with the options that we need for validation.

The validation itself is a copied/simplified version of what is performed by the main parser, as large parts the validation depended on the AST structure, which we do not have here.

Technically, this doesn't perform the validation at the same point in time as the main parser, as it performs the validation when encountering the tREGEXP_END token rather than when processing the regexp rule.

I speculate that the difference doesn't really matter given that the only thing we could do with the tREGEXP_END token is to apply the regexp rule.

Use a separate variable to track whether things are dynamic or not, and use a List to avoid tracking the last element explicitly.

The methods were only retained to provide the old interface, but by directly calling the new methods in the Lexer, we can remove the old methods, given that we don't really need to be backwards compatible here.

Add support for validating Regexp in Ripper

Fix a typo

A happy new year! 🎆🎍

`findEncodingNoError` does not raise any error.

Fix documents

... as a side effect op_pow(long) got resurrected for Fixnum/Bignum

closes GH-4978

…I need

to make changes to generated code I am trying to comment/document some portions
and reduce some duplication when possible.