I should also mention #64067 where the first part was introduced, but contrary to it this one isn't backported by any upstream distributions, reason unknown.

I'd appreciate review of the patch given that I basically rewrote it.

I can't give a comprehensive review due to a lack of time and I'm far from a C-developer myself anyway. But out of interest, why didn't you use my patch at https://github.com/NixOS/nixpkgs/pull/63616/files#diff-7af205fc79201d40af30ad4983bcc777?

Huh, I guess I... missed it? I'll compare them, thanks!

Well I did tell you that I pushed it, but its easy to miss 🤷‍♀️

Anyway, we definitely have to be a little more careful here than with #64067, since its's C, malloc/free is involved and there is no other-distro precedent.

Could we apply this patch only when building tensorflow?

@FRidh We could do that but it's a bug nevertheless, I feel it could show up in other massively parallel builds too. If there are any we'll have until the EOL that is ;)

I minimized the patch, also bringing it as close to the spirit of original patch as possible. Hopefully this helps with the review.

@FRidh Thinking more about it, could be problematic because Tensorflow will then link to its custom Python 2 library and then we'll have a conflicting Python library loaded into our usual interpreter.

EDIT: Fixable via patchelf though.

Reading more into the original patch: unless I'm mistaken there is actually a race condition in it too! Namely, in C (but not in Python) file is opened exclusively to prevent concurrent writes but then fclose()d before rename(), so another Python process can open .tmp file in this short period of time and begin writing while the file has already been renamed. This code isn't in final Python 3.4 release AFAICS so I suppose it has been replaced with something else.

Did I overlook something there? The fix is quite easy (move fclose() a bit further down) so I propose to do just that.

Could very well be a bug, probably wouldn't happen much in practice and the exclusive open was added as an afterthought.

Hehe, actually the bug that I described is present in Python right now but in a very light form - because id() is used the chance of this actually happening is pretty slim. Because rename() works differently on POSIX and Windows simply moving rename into with block wouldn't work for Windows here I think, so it's not that simple to fix. I'll report it but don't think it's nearly serious enough to fix it by ourselves given id() is 64-bit address of an object so the chances of a clash actually happening in such conditions are pretty slim.

Patch updated, tested this one through the night running Tensorflow compilations.

could be problematic because Tensorflow will then link to its custom Python 2 library and then we'll have a conflicting Python library loaded into our usual interpreter.

EDIT: Fixable via patchelf though.

Yes, did the same with tkinter. Well, let's just see how it turns out. If we get trouble that may be the way forward.

Checked out tkinter - nice hack! I wonder if something similar could be done for things like Java (headless + GUI libraries as modules of some kind)...