It looks like when the buffers are full (they are pretty small of course), smoltcp can't find anywhere to queue the ARP request and a kind of a deadlock is the result: no packets can go out anymore.

Wait. Which buffers get full? The ARP request is not queued anywhere; it is generated on the fly instead of the outgoing packet (the UDP packet in this case) when there is no cache entry. Do you have a log at TRACE verbosity?

Oh, that's a good point! I'll see if I can extract a trace log, but probably after Christmas. Thanks for the heads up, I hadn't looked deep enough in the code.

Ok, I have some more details now. First, I reduced the ENTRY_LIFETIME in the neighbor cache to 5s, to speed up reproduction.

I disabled the trace messages for UDP "buffer to send ... octets" and "sending ... octets", to be able to keep the data rate at the full 100 MBit/s.

When switching the log level to trace, the problem went away - the log looks like this

[TRACE] [0]: adding
[DEBUG] address 192.168.1.1 not in neighbor cache, sending ARP request
[TRACE] #0: neighbor 192.168.1.1 missing, silencing until t+3.000s
[TRACE] filled 192.168.1.1 => 68-05-ca-3a-c7-43 (was empty)
[TRACE] #0:192.168.1.222:54321:192.168.1.1:54321: receiving 3 octets
[TRACE] #0: neighbor 192.168.1.1 discovered, unsilencing
[DEBUG] address 192.168.1.1 not in neighbor cache, sending ARP request
[TRACE] #0: neighbor 192.168.1.1 missing, silencing until t+3.000s
[TRACE] #0: neighbor 192.168.1.1 discovered, unsilencing

with the last 3 lines repeating in 5s intervals.

After some trial and error, I removed the net_debug call here, and suddenly could reproduce the problem again!

I then replaced the debug call by a simple (not-optimized-out) delay loop, and by tuning the length of the loop I could make the problem appear and disappear.

That leads me to believe that the problem must be on the STM32 side, and there's some race condition that prevents sending that ARP packet or swallows it.

Sounds about right. Closing as not an issue in smoltcp.

Which STM32 driver are you using?

https://github.com/stm32-rs/stm32-eth - is there another one I could try?

@birkenfeld: There is a basic version for STM32H7… at https://github.com/quartiq/stabilizer/blob/master/src/eth.rs, which might be similar. We've had some fun with memory coherency, but off the top of my head, that shouldn't actually be a problem for the Cortex-M4 core in the STM32F4 series.

Reopening since this was narrowed down to an issue in the ARP code in stm32-rs/stm32-eth#10.

I think I found another problem that might be the cause of the missing ARP.

During socket egress, there is a call to lookup_hardware_addr, which does the following:

On the first call to poll just after the address expired, the call in self.neighbor_cache.lookup will take the last branch on the method, increasing the silent_until timestamp and returning NotFound:

This will cause lookup_hardware_addr to craft an ARP packet and try to send it via the device, but this sending can fail with Error::Exhausted (which I think it's happening in the case of this issue), the error will bubble up to socket_egress:

which will just hit a break, however, if the next call to poll happens before one second has passed (very likely), self.neighbor_cache.lookup will now return RateLimited, which will cause lookup_hardware_addr to return Error::Unaddressable which will cause socket_egress to mark the NeighborState as Waiting (for 3 seconds) for an ARP response without ever having sent a single ARP request.

The underlying problem is that we set the silent_until timestamp in self.neighbor_cache.lookup before we know if we really managed to send the ARP request.

@whitequark is my understanding of this correct ? If so, do you have a preferred way to address this ?

Thanks everyone for figuring this out!