Issue 7513: stack overflow due to recursive GC marking and release #9040
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Motivation / Problem
Fixes #7513 which is a stack overflow due to recursive marking for garbage collection. Whacking that mole yields a stack overflow due to recursive releasing, so that's tackled here as well.
Description
The bug is solved by rewriting bits of code to go from a recursive marking and releasing to an approach that is more iterative, which is done by having a queue of objects that need to be processed and going through that queue iteratively. Whenever the iterative step on an object requires the same step to be taken on another object (release or GC marking), then that object is added to the queue so it will be processed at a later stage.
Limitations
Depending on the code, now you might hit out of memory issues instead of the game crashing due to a stack overflow. I intentionally tried to make a few changes to the rest of the squirrel code, so it's probably not all according to coding style. I rather leave the coding style fixes for Squirrel for another time, as not doing those makes it easier to see the differences between stock and our version of squirrel.
Freeing and GC marking can cause lags with such large amounts of elements, but that's inherent to such massive amounts of elements; millions in my test cases in debug builds. This makes both freeing and GC marking slightly slower, but the benefit is the game not crashing.
Checklist for review
Some things are not automated, and forgotten often. This list is a reminder for the reviewers.