Well, if 7.60.0 breaks Nix, maybe it breaks other things as well? Perhaps we should just hold off on upgrading to 7.60.0, or is there a particular reason to update?

The master is already at 7.60.0. This would fix Nix while we investigate. The curl program works (#41312 (comment)), and after a few tries I was not able to reproduce this with curl with varying command line flags.

👍 for 7.59 for now, bit of info-dump on the subject follows:

I've been using 7.59 for a while now for this reason. The problem is that the threaded resolver (on by default) doesn't have a fd to listen on when resolving (unless using c-ares) so things end up waiting the full timeout.

I believe our usage of curl is unusual-- but AFAIK not incorrect -- which is how we get impacted by this but others don't.

Look at how curl itself leverages the multi API when doing a single "easy" request -- as well as in the various examples-- and among other differences you'll notice they claim one should just sleep 100ms and call again, instead of trusting curl_multi_wait to actually work as we expect it to. Might be worth asking curl folks to take a look at what we're doing.

Upstream curl has a commit or two that might help (curl/curl@3e0dee0 is relevant and is a similar problem for the "protocol" state) but curl master doesn't fix this for me.

And I haven't debugged this further because Nix+7.60 also seems especially unhappy with our removing of handles from the multi stack when they're possibly already part of an http2 pipeline (I guess?) which has been causing crashing behavior and been what I was focusing on investigating.

(crashes might be due to openssl global context badness when forking, but unsure)

I was kinda hoping someone somewhere would produce a patch/bug report (I've been checking other distributions and the curl mailing list) but maybe Nix is just that special and other software isn't having problems :(.

Thanks for the explanation!
I've bisected curl to find that the first bad commit is curl/curl@6763622, but I don't understand the issue yet.
I'd like to merge this PR now, and probably downgrade curl in staging later.

@orivej that was my finding as well when bisecting! (dtzWill/nix@ba509d8)

Sorry for not better communicating findings earlier, but wasn't sure problem wasn't in my setup/environment Thanks for investigating and getting a workaround in place!

Related upstream issue: curl/curl#2636 (comment)

Which at least partially matches what I discovered/reported above regarding our use being considered the issue. (we tell curl_mult_wait to sleep up to 10sec).

Not sure what the best alternative is, one approach I was exploring is to use a timeout of 100ms whenever we have active requests, and increase the amount (to 10s or whatever) if literally all we're doing is polling on our wakeup pipe. I'm tackling other issues which made this less obviously a "win" but it might do the trick for most folks?

(relevant code in Nix, FWIW: https://github.com/NixOS/nix/blob/4ac4f675df3da01b6d814cd328dd3219dd472ac9/src/libstore/download.cc#L482)

My understanding is that upstream recognizes that the threaded resolver is a bit more difficult to use than the blocking and the asynchronous resolvers, and can not change that. (curl/curl#2636 (comment)) I propose that we build Nix with curl with the asynchronous resolver in master (#41609), and then build the default curl with the asynchronous resolver in staging.

Since curl with c-ares can't support nsswitch plugins (#41609 (comment)), we can not enable c-ares by default. I propose that we just build Nix with curl with blocking resolver, as long as it does not support threaded resolver (#41619).