I was doing some stress testing of TCP connections and saw an oddity go by that has me scratching my head.
Here's the order of events as best as I can figure from correlating the server-side PCAP and the modem console log:
- Modem is online and has a working TCP connection to the MQTT broker
- Modem loses usable connection for many seconds (stuck in CEREG: 5,"FFFE","FFFFFFFF",7,,,"11100000","11100000")
- Our application calls mqtt_abort while in this state
- Modem gets back to a usable connection
- Modem reconnects to MQTT broker on new socket
- Broker recognizes new connection from existing client ID and accepts new connection, but also sends FIN to old connection/port
- The modem acknowledges the FIN from the broker but doesn't immediately send a FIN in response
- Many minutes later, a FIN shows up at the broker from the modem on a random source port, but with the sequence number that would've been used to close the first connection
- Step 8 keeps repeating at slower back-offs and random source ports
TL;DR: the initial FIN packet from the modem was lost, but retries intermittently and repeatedly show up at the broker for hours. I've seen it so far 5 times over 3 hours.
I suspect the initial FIN from the modem to the server would've been triggered in step 3 when our application aborted the connection. I suppose there is a tiny chance it would've been sent circa step 7, but I would've expected it to get through easily in that case.
The changing source ports makes sense, since the NAT device on the cellular provider's network likely forgot the port mapping soon after the initial FIN packets went by.
How long is the modem going to hang on to the old socket before it gives up? Our application forgot the socket handle just after aborting the connection.
