For clarification, this cannot fundamentally fix the problem inherent in GCC's linker I describe in #41340 (comment) (which is that the amount of flags you can pass to GCC must not exceed 128 KB; which means that if you have 1000 Haskell dependencies ala /nix/store/abj3z3bpiipa8h21dmlc5hjf0jbgpn89-utf8-string-1.0.1.1/lib/ghc-8.2.2/utf8-string-1.0.1.1/libHSutf8-string-1.0.1.1-7sQ7eR5H5ED7poickls5aO.a, you exceed it), can it?

If that is correct, reducing flags is still very useful for project that are somewhere below this upper bound; could you put a rough number estimate on how many flags we save with this PR?

Yeah there is definitely still a limit, but I don't think we ever hit it before 18.03. My understanding is this was a regression that only came up by 18.03. I think we can fix it by just eliminating all duplication, and I believe this haskell stuff cut it down by ~30%.

But, I'm also very skeptical of your claim that it has to be under 128KB. If true, that would mean that the Linux + GCC actually has a lower maximum argument count than macOS. In practice, I think I've only seen Linux hit it on Linux when we were still getting the 6x duplication. Almost every time this breaks macOS before it breaks Linux.

But, I'm also very skeptical of your claim that it has to be under 128KB.

@matthewbauer I've made a repro in Python on my Ubuntu 16.04:

import os
import sys

# num_chars = 128000 # works
# num_chars = (2**16) # works
# num_chars = (2**17) # fails with E2BIG (Argument list too long)
# num_chars = (2**17 - 3) # works
num_chars = (2**17 - 2) # fails with E2BIG (Argument list too long)

sys.stdout.write("trying with " + str(num_chars) + " chars\n")

os.execve('/bin/true', [], {
  'x': 'a' * num_chars
})

This shows

trying with 131069 chars

and

trying with 131070 chars
OSError: [Errno 7] Argument list too long

confirming the 128KB limit for me.

Can you try this too?

Yeah, I'm seeing the same limit. Interesting that we aren't hitting it more often. On macOS you don't hit this at all because clang isn't setting an environment variable like gcc. When we did hit this on macOS, it was always at linking, but I suspect that binutils doesn't pass those through an env var avoiding this limit.

Do you have an example package where you hit this on Linux?

Do you have an example package where you hit this on Linux?

Not any more. I have a project with 360 recursive Haskell dependencies total, which had hit the limit originally, but some of the previous flag deduplication improvements (see also #49206 (comment)) brought some integer factor improvements that brought it back under the limit. I expect to hit it again as the project grows.