I would like to help on this. I did already some initial investigation and coding.
This also look like a duplicate of issue #4495, isn't it?

I prepared initial fix but I have some observations and questions.

1. Dir.open, Dir.new, Dir.foreach support encoding both as string and as Encoding in CRuby:

p Dir.open('.', encoding: 'UTF-8').entries
p Dir.open('.', encoding: Encoding.find('UTF-8')).entries
p Dir.new('.', encoding: 'UTF-8').entries
p Dir.new('.', encoding: Encoding.find('UTF-8')).entries
Dir.foreach('.', encoding: 'UTF-8'){|f| p f}
Dir.foreach('.', encoding: Encoding.find('UTF-8')){|f| p f}

2. I found MRI test test_dir_enc and implemented changes to make the test pass (i.e. encoding as String).
   I didn't find any MRI test with Encoding type.
   I would like to add them, but I'm not sure where to add. Should this be test_dir.rb under test/mri or under test/jruby?
   If test.mri then shouldn't this be contributed to MRI repo? (I believe I've seen such recommendation somewhere, but can't find it again)

What's the recommended approach in this case?

3. Dir.foreach supports second parameter already in JRuby, but I believe it's not correctly implemented (bug). Second parameter should be "keyword argument", not regular parameter, i.e.

Dir.foreach('.', Encoding.find('UTF-8')){|f| p f}

• CRuby (2.6.4): fails with ArgumentError (wrong number of arguments (given 2, expected 1))
• JRuby: runs fine (i.e. bug)

correct syntax is Dir.foreach('.', encoding: 'UTF-8')

I would like to continue with:

• fix foreach to use keyword arguments
• support both string and Encoding in all above methods
• implement remaining tests, but I need some guidance (until then I will prepare tests under test/jruby)

@MariuszCwikla Thanks for your help with this and #4495 (now fixed by #5915)! Your plan going forward looks fine to me, and I am standing by to offer any assistance I can.

Just ran back into this with Dir.each_child - guess that one has been missed given entries, foreach, and children are all fine.

I have pushed #6551 to try to unify this encoding logic and bring these methods into harmony. The available tests are no greener than they were before, but test/mri/ruby/test_dir_m17n runs now (was failing to load) and could be iterated on.

@MariuszCwikla Dunno if you are still interested in helping here but going through the test_dir_m17n failures one by one might be fun! 😀

This is not ready and .15 will be coming out very soon.

All original cases now work properly after #6551 and subsequent commits 04f0c82 and 937792b. Specs were added for the kwarg use cases but not for the error case that we used to allow.