RE: MQTT Connection Resets - Keepalive

JONATHAN LL — Thu, 19 Oct 2023 09:06:19 GMT

Hi,

So from the logs it seems that the issue is likely the keepalive.

In both cases a RST, ACK sent from server terminating the TCP connection, this is likely due to the MQTT connection timing out. Even though the client request a certain MQTT keepalive it doesn’t mean that the server supports that value.

An to add from our expert on the matter:

My current conclusion (strong guess without seeing server side traces and/or not knowing how the server side architecture) is that the NAT binding times out on the carrier network side (AT&T) and thus the load balancer / Firewall on the Hive could side (FQDN c59829c0af344d2aa83346efbde5e10c.s1.eu.hivemq.cloud) does not recognize the TCP session anymore (or the session timed out) or the PINREQ packet gets forwarded to a wrong MQTT server as the source IP/Port has changed thus managing existing session mappings. In all cases the cloud side server / load balancer / Firewall responds with RST. Business as usual.

Why I refer to load balancer / firewall? That is an educated guess based on the IP header identification and TTL fields.

During the 3-way handshake the server side uses identifications from 0 onwards and TTL is 234.
Once the connection moves to TLS the identification field jumps to e.g. 0x9ad7 onwards → high chances this is a different node behind the load balancer as the “IP session” is completely different. TTL is still the same 234..
When the server side responds with an RTS the IP identification is copied from the source packet and but TTL is 208 i.e. not the expected 234 → high chances the RTS comes from a different server or load balancer.

There are few obvious ways to tackle the issue:

better choice of a protocol.
Use of Private/enterprise APNs → no issues with NATs or they are under the control of the customer.
Use more aggressive keepalive, which is closer what the NAT timeouts are on the network side.
Use of IPv6 instead of IPv4, since there is not NAT issues with IPv6.

Regards,
Jonathan

RE: MQTT Connection Resets - Keepalive

ERob — Wed, 18 Oct 2023 18:48:02 GMT

Thanks Jonathan. Network is AT&T, and using public cloud for the HiveMQ brokers.

RE: MQTT Connection Resets - Keepalive

JONATHAN LL — Wed, 18 Oct 2023 08:50:30 GMT

Hi,

Will look at the trace and provide some feedback, update will follow.

What network provider are you using?

And are you using public cloud for these brokers or do you host your own servers in HiveMQ?

Regards,
Jonathan