-
-
Notifications
You must be signed in to change notification settings - Fork 925
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Truffle] Unable to unzip large gzip file #4133
Comments
I can run this example with a gzipped file of 5 MB (so about the same size) in 90 seconds. The file attached to this issue doesn't seem to ever finish and gradually consumes more memory. A heap dump shows it's all ASCII leaf ropes. A majority of the ropes seem to be 8,216 or 25 bytes long. Are these some buffer sizes somewhere? Presumably we're creating pathologically deep ropes here and need a new flattening heuristic. I don't understand why a real gzipped file behaves differently to a random one, but the random one is very slow anyway. |
@chrisseaton My current understanding is that append to rope is worst case because every append will add two nodes which consumes a lot of memory. @nirvdrum said he has some ideas on how to fix this. The shape looks something like this:
|
Are you saying that's definitely how the deflate code works or are you just making a good guess? Can you point me at it the right place in the source? |
@chrisseaton I'm pretty sure it uses a
https://github.com/jruby/jruby/blob/master/lib/ruby/truffle/pr-zlib/lib/pr/zlib.rb#L138 Also, it looks like |
I'm going to close this since it appears to give the expected results now. I took some timings that shows there some room for improvement:
jruby+truffle
|
Environment
gzipped_versions.gz
Found while trying to do
bundle install
Expected Behavior
The script prints a bunch of text.
Actual Behavior
Fails with Out of Memory Error
The text was updated successfully, but these errors were encountered: