Bad BSD socket behavior - possibly triggered by unexpected mobile-terminated TCP FIN?

[jbrzozoski]

I'm still trying more TCP stress testing similar to this inquiry. However, now I'm getting highly repeatable lockups on the node I am doing testing with.

Application code that has been working fine and reliable for months is now locking up within seconds after boot.  The only change was that I was doing "bad things" on the network before rebooting.  (Lost traffic, slow traffic, keeping sockets quiet for an abusively long time w/o traffic, abandoning sockets w/o warning the host or network, etc)

It appears that the thread that handles the BSD sockets works fine after reboot while starting up a new connection to the server, but then locks up soon after.  The failure seems to happen the next time the application tries to send/receive TCP traffic.

Here's the sequence of what I think is happening:

1. Modem is online and has a good TCP connection to server X port 1883
2. Without warning the cellular network or server X, I reboot the modem [NAT mapping and server both remember connection]
3. Modem goes back online and opens a new TCP connection to server X port 1883
4. Server X recognizes a new connection from the same client, and sends a FIN to the old port
5. The FIN packet is routed by the cellular network back to the modem
6. The BSD socket locks up on the next TCP action by the application

I can't prove step 5 is happening, but I've got strong evidence of all the other steps.  And this lockup is HIGHLY repeatable, while it never used to happen, ever.  The only difference I can think of is the FIN packets are actually reaching the modem.  Our normal reboot/reset procedures do their best to close the TCP connection before rebooting, which would cause the NAT mapping to disappear soon after.

On a reboot, if the application doesn't request a specific source port number for an outbound TCP connection, will the nRF9160 use the same port number every time?  If so, I highly suspect that is part of the issue, since the FIN received on step 5 would be showing up on the same port number as our valid connection we just opened on step 3, with the correct port and IP address for the server side of the connection as well.

I'm going to go look if I can make the MQTT socket use a changing port number when I open the socket and see if that stops this lockup...

[jbrzozoski]

As a quick hack, I tried binding the socket to a changing port number after opening it in mqtt_transport_xxx.c.  The lockup seems to be gone now. 

I'd still like some confirmation from someone with visibility into the modem firmware whether my theory is plausible.

If this does look like the issue, I'd argue that a quick fix would be for the modem firmware to use a changing source port if the application doesn't specifically bind one.  If the current system uses an incrementing port number, maybe just seed it from the RTC on bootup?  It doesn't need to be random, just not the same on every reboot.

Additionally, some more effort should be made to not fail if an invalid FIN for an ip/port we actually have open shows up, but I understand that might be harder to distinguish.  Maybe if the ACK sequence from the server is way out of line you could ignore it?

[Didrik Rokhaug]

Hi, and thanks for the detailed report.

I have forwarded it to our modem team and will keep you updated about any progress.

What modem firmware and bsdlib versions do you use?

Best regards,

Didrik

[jbrzozoski]

MFW 1.1.0 ... BSDLIB 0.5.1

[Didrik Rokhaug]

According to our modem team, the modem uses a random TCP source port for when a socket bind is not used.

Other than that, we will need a modem trace in order to debug further.

[jbrzozoski]

Unable to recreate this to get a modem trace yet.

The modem is now sending a RST packet instead of locking up at step 6, like I would expect.

Something was definitely different the other day.  I'm now trying to remember what other testing I had been doing leading up to the failures, in case one of those tests was somehow a prerequisite for the failure.

I'm still trying...