This change appears to be effectively forkbombing my Mac. My network isn't very large, something like 10 machines, but I do have probably 5-10 keys per host.

When I run a nixops deploy, it brings the Mac's networking stack to a standstill. :(

edit This is on macOS 10.15.4, btw.

Nice. I'll revert for now, we can / should add a global limit on the number of keys being sent at a time.

Huh, 10 connections with 100 sessions kills networking stack? What kind of Mac is this?

iMac Pro, 64GiB, 10 cores, so no slouch.

I'm as surprised as you are. But there were so many SSH processes running, my nixops deploy hung and I couldn't even ps aux|wc -l to count them all from another terminal.

I've seen similar problems, actually. As far as I could tell it had to do with the thundering herd of connections. I wonder if we could pre-calculate how many keys need to be sent, and allocate a total max or something? I dunno, that gets complicated.

Just tested head, that's much better. Thanks for the remarkably fast turnaround!

Another possibility is this: I'm using gpg-agent to manage my SSH keys, and my SSH private key is stored on a YubiKey. I was under the impression that NixOps always uses the SSH key it generates for root@host at nixops create time for each host, but every once in awhile, NixOps will prompt me for my card's PIN anyway, for reasons I don't understand. So I wonder if some of the SSH sessions are hitting the YubiKey and it, or gpg-agent, just can't keep up?

After speaking with @grahamc on IRC about this, it sounds like which SSH key NixOps uses is not necessarily deterministic, and that it may use a personal SSH key? If so, anyone who's using a hardware device to manage their SSH private keys is going to have problems with this implementation, assuming I'm right about what was going on here.