bpo-37788: Fix reference leak when Thread is never joined #26103
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
pitrou
added
needs backport to 3.8
|
Note the unit test is originally from PR #25226. |
|
I started a buildbot run here: https://buildbot.python.org/all/#/changes/4460 (edit: previous run was on a bogus changeset) |
|
I also ran @vstinner 's regression test and it seems to work fine here: |
When a Thread is not joined after it has stopped, its lock may remain in the _shutdown_locks set until interpreter shutdown. If many threads are created this way, the _shutdown_locks set could therefore grow endlessly. To avoid such a situation, purge expired locks each time a new one is added or removed.
pitrou
force-pushed
the
bpo-37788-thread-join-leak
branch
from
dc79edc
to
16edc6a
Compare
shihai1991
approved these changes
May 14, 2021
| # does not leak. Test written for reference leak checks. | ||
| def noop(): pass | ||
| with threading_helper.wait_threads_exit(): | ||
| threading.Thread(target=noop).start() |
There was a problem hiding this comment.
Would you mind to add a test case that the target function isn't just only the pass
There was a problem hiding this comment.
Hmm, what do you mean exactly? I'm not sure I understand your suggestion.
There was a problem hiding this comment.
Oh, sorry. I didn't explain it clearly. I suggest that the target of Thread could be more complex ;)
Something like:
def noop():
# we don't what code will run in the thread in fact.
time.sleep(random.random())
with threading_helper.wait_threads_exit():
threading.Thread(target=noop).start()There was a problem hiding this comment.
But what would that change? What matters is what happens after the thread ends.
There was a problem hiding this comment.
Hm, give me a second.
No matter how complex of the target of thread, the thread will finish in time in python level. So we don't adding a complex testcase in here is fine. right?
Thanks for checking with the buildbots! Seems that the failures are unrelated to this, so I think we are good to go |
|
Ok, merging then. |
|
Thanks @pitrou for the PR 🌮🎉.. I'm working now to backport this PR to: 3.10, 3.8, 3.9. |
|
Sorry, @pitrou, I could not cleanly backport this to |
|
GH-26138 is a backport of this pull request to the 3.10 branch. |
|
Sorry @pitrou, I had trouble checking out the |
miss-islington
pushed a commit
to miss-islington/cpython
that referenced
this pull request
May 14, 2021
…6103) When a Thread is not joined after it has stopped, its lock may remain in the _shutdown_locks set until interpreter shutdown. If many threads are created this way, the _shutdown_locks set could therefore grow endlessly. To avoid such a situation, purge expired locks each time a new one is added or removed. (cherry picked from commit c10c2ec) Co-authored-by: Antoine Pitrou <antoine@python.org>
|
Thanks for the fix @pitrou, it looks simple :-) |
pitrou
removed
needs backport to 3.8
pitrou
added a commit
to pitrou/cpython
that referenced
this pull request
May 15, 2021
…onGH-26103) When a Thread is not joined after it has stopped, its lock may remain in the _shutdown_locks set until interpreter shutdown. If many threads are created this way, the _shutdown_locks set could therefore grow endlessly. To avoid such a situation, purge expired locks each time a new one is added or removed.. (cherry picked from commit c10c2ec) Co-authored-by: Antoine Pitrou <antoine@python.org>
miss-islington
added a commit
that referenced
this pull request
May 15, 2021
…26103) (GH-26138) When a Thread is not joined after it has stopped, its lock may remain in the _shutdown_locks set until interpreter shutdown. If many threads are created this way, the _shutdown_locks set could therefore grow endlessly. To avoid such a situation, purge expired locks each time a new one is added or removed. (cherry picked from commit c10c2ec) Co-authored-by: Antoine Pitrou <antoine@python.org>
miss-islington
pushed a commit
that referenced
this pull request
May 15, 2021
…6103) (GH-26142) When a Thread is not joined after it has stopped, its lock may remain in the _shutdown_locks set until interpreter shutdown. If many threads are created this way, the _shutdown_locks set could therefore grow endlessly. To avoid such a situation, purge expired locks each time a new one is added or removed.. (cherry picked from commit c10c2ec) Co-authored-by: Antoine Pitrou <antoine@python.org> Automerge-Triggered-By: GH:pitrou
|
|
Hey there! |
jpmckinney
added a commit
to open-contracting/kingfisher-process
that referenced
this pull request
Jan 27, 2022
jpmckinney
added a commit
to open-contracting/yapw
that referenced
this pull request
Jan 28, 2022
Note that the previous code was not thread-safe. A thread-safe version would have been:
for t in list(threads):
if not t.is_alive():
# A set is used for `threads`, because `set.discard()` is O(1) but `list.remove()` is O(n).
threads.discard(t)
# Python 3.8 and below has a memory leak. python/cpython#26103
t.join()
jpmckinney
added a commit
to open-contracting/kingfisher-process
that referenced
this pull request
Jan 28, 2022
…thon/cpython#26103" This reverts commit cd4a74a.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
When a Thread is not joined after it has stopped, its lock may remain in the _shutdown_locks set until interpreter shutdown. If many threads are created this way, the _shutdown_locks set could therefore grow endlessly. To avoid such a situation, purge expired locks each time a new one is added or removed.
https://bugs.python.org/issue37788
Automerge-Triggered-By: GH:pitrou