Socket send function hangs

RE: Socket send function hangs

Håkon Alseth — Mon, 07 Sep 2020 14:02:40 GMT

Hi Nikil,

Sorry for the late response.

The second trace here is also a bit strange, as there seem to be packets missing from the trace. The first one looks more complete than the second one.

Based on the link that you sent, the behavior is consistent with the first trace (or the valid part of it atleast), where it takes approx. 375 seconds before the AWS server sends a TCP RST. It may indicate that network load balancer is active, as the trace shows different IPs to the same DNS when you try to initiate a connection the second time. If it is, the restriction here should apply: https://docs.aws.amazon.com/elasticloadbalancing/latest/network/network-load-balancers.html#connection-idle-timeout

Kind regards,

Håkon

RE: Socket send function hangs

Nikil Rao — Fri, 04 Sep 2020 11:27:00 GMT

Hi Håkon,

Do you have any update on this? Thanks!

RE: Socket send function hangs

Nikil Rao — Tue, 01 Sep 2020 15:01:22 GMT

Hi,

I've attached a trace I took today. Yes, I'm using mfw 1.2.0.

Another interesting thing that I found while scouring the internet to find a solution for this problem was this - https://stackoverflow.com/questions/58322484/how-to-keep-tcp-connections-alive-on-aws-network-loadbalancer. I'm not sure if this is even related or not but maybe there's some additional TCP timeout apart from the MQTT timeout which I need to set somewhere. The TCP timeout mentioned in the link above is 350 seconds which seems to be exactly the duration above which I have problems.

devzone.nordicsemi.com/.../trace_2D00_2020_2D00_09_2D00_01T14_2D00_40_2D00_23.284Z.bin

RE: Socket send function hangs

Håkon Alseth — Tue, 01 Sep 2020 13:57:13 GMT

Hi,

The trace show a TCP reset coming from the AWS server after 375 seconds, but it might be that the timestamping has been corrupted in the modem trace (its highspeed and high throughput, no UART flow control). If you provide another trace showing the issue, I can try to see if this one also shows the same behavior.

You're testing with mfw v1.2.0, right?

Kind regards,

Håkon

RE: Socket send function hangs

Nikil Rao — Mon, 31 Aug 2020 10:56:23 GMT

Hi Håkon,

I have CONFIG_MQTT_KEEPALIVE=1200 in my code. Moreover this is what I get when I enable MQTT logs -

[00:00:21.134,094]<dbg> net_mqtt_sock_tls.mqtt_client_tls_connect: (0x20020ff8): Created socket 1
[00:00:31.134,979]<dbg> net_mqtt_sock_tls.mqtt_client_tls_connect: (0x20020ff8): Connect completed
[00:00:31.135,101]<dbg> net_mqtt_enc.connect_request_encode: (0x20020ff8): Encoding Protocol Description. Str:MQTT Size:00000004.
[00:00:31.135,101]<dbg> net_mqtt_enc.pack_utf8_str: (0x20020ff8): >> str_size:00000006 cur:0x2002919a, end:0x2002957d
[00:00:31.135,131]<dbg> net_mqtt_enc.pack_uint16: (0x20020ff8): >> val:0004 cur:0x2002919a, end:0x2002957d
[00:00:31.135,131]<dbg> net_mqtt_enc.connect_request_encode: (0x20020ff8): Encoding Protocol Version 04.
[00:00:31.135,131]<dbg> net_mqtt_enc.pack_uint8: (0x20020ff8): >> val:04 cur:0x200291a0, end:0x2002957d
[00:00:31.135,162]<dbg> net_mqtt_enc.pack_uint8: (0x20020ff8): >> val:00 cur:0x200291a1, end:0x2002957d
[00:00:31.135,162]<dbg> net_mqtt_enc.connect_request_encode: (0x20020ff8): Encoding Keep Alive Time 04b0.
[00:00:31.135,162]<dbg> net_mqtt_enc.pack_uint16: (0x20020ff8): >> val:04b0 cur:0x200291a2, end:0x2002957d
[00:00:31.135,162]<dbg> net_mqtt_enc.connect_request_encode: (0x20020ff8): Encoding Client Id. Str:nrf-352656100388050 Size:00000013.
[00:00:31.135,192]<dbg> net_mqtt_enc.pack_utf8_str: (0x20020ff8): >> str_size:00000015 cur:0x200291a4, end:0x2002957d
[00:00:31.135,192]<dbg> net_mqtt_enc.pack_uint16: (0x20020ff8): >> val:0013 cur:0x200291a4, end:0x2002957d
[00:00:31.135,223]<dbg> net_mqtt_enc.mqtt_encode_fixed_header: (0x20020ff8): << msg type:0x10 length:0x0000001f
[00:00:31.135,223]<dbg> net_mqtt_enc.packet_length_encode: (0x20020ff8): >> length:0x0000001f cur:0x00000000, end:0x00000000
[00:00:31.135,223]<dbg> net_mqtt_enc.mqtt_encode_fixed_header: (0x20020ff8): Fixed header length = 02
[00:00:31.135,223]<dbg> net_mqtt_enc.pack_uint8: (0x20020ff8): >> val:10 cur:0x20029198, end:0x2002957d
[00:00:31.135,253]<dbg> net_mqtt_enc.packet_length_encode: (0x20020ff8): >> length:0x0000001f cur:0x20029199, end:0x2002957d
[00:00:31.135,986]<dbg> net_mqtt.client_connect: (0x20020ff8): Connect completed

Encoding Keep Alive Time 04b0. shows that it is indeed 1200 seconds

RE: Socket send function hangs

Håkon Alseth — Mon, 31 Aug 2020 10:39:55 GMT

Hi Nikil,

Judging by the trace, it seems that the server (aws) sends a TCP RST (reset) after approx. 375 seconds, which indicates a keep-alive of 250 (1.5 * 250 = 375).

Is your local MQTT keep-alive set to 250 in the trace? If yes, can you please set it to 1200 and do a new trace?

Kind regards,

Håkon

RE: Socket send function hangs

Nikil Rao — Mon, 31 Aug 2020 09:00:57 GMT

Hi,

Thanks a lot for the reply. I reproduced the AWS disconnect on the nrf9160dk and I've attached a modem trace. Could you please analyze it and let me know if it is indeed an NAT timeout?

Yes, right now we have kept the message send interval to 300 seconds while we wait for the new modem firmware

devzone.nordicsemi.com/.../trace_2D00_2020_2D00_08_2D00_31T08_2D00_41_2D00_21.471Z.bin

RE: Socket send function hangs

Håkon Alseth — Fri, 28 Aug 2020 13:27:02 GMT

Hi,

[quote user="Nikil Rao"]Thanks a lot for the response. Do you have an estimate on when the new modem firmware will be out?[/quote]

Unfortunately, I do not have the exact timeline. If you contact Thomas Page (thomas.page@nordicsemi.no) - he might have more details on this matter.

[quote user="Nikil Rao"]Doing this however, the device needs a reset after every 20 connect cycles (every 200 minutes). Due to the high current consumption when the device resets and connects to the LTE network every 200 minutes, the lifetime of our device reduces drastically. Could you please advice on possible solutions we can try until the new modem firmware is out? Thanks![/quote]

My apologies for the inconvenience this bug is causing. I understand that a reset is costly.

I unfortunately do not have a good workaround for your case. The problem is that in the network chain, you're NAT'ed through a LTE network, which might have their own timeouts (a "NAT timeout") on TCP connections, which would then override any keepalive the server has on a strict IP level. If you provide a modem trace, we can verify if its the network restricting the keep-alive.

Is it a possibility to use PSM in your application, and keep the "keep alive" to under 327 sec? Or does this add up to a higher current consumption over-time compared to your current applications connect/disconnect algorithm?

Kind regards,

Håkon

RE: Socket send function hangs

Nikil Rao — Wed, 26 Aug 2020 08:39:22 GMT

Hi Håkon,

Thanks a lot for the response. Do you have an estimate on when the new modem firmware will be out?

We have a use case where we connect to AWS and send data every 10 minutes. Since I'm not able to keep a connection to AWS alive for more than 5 minutes (https://devzone.nordicsemi.com/support/252089) I'm trying a workaround where I send a message and then disconnect from AWS MQTT. I then reconnect after 10 minutes and do the whole cycle again. Doing this however, the device needs a reset after every 20 connect cycles (every 200 minutes). Due to the high current consumption when the device resets and connects to the LTE network every 200 minutes, the lifetime of our device reduces drastically. Could you please advice on possible solutions we can try until the new modem firmware is out? Thanks!

Nikil

RE: Socket send function hangs

Håkon Alseth — Thu, 13 Aug 2020 12:50:54 GMT

Hi,

We recently found an issue that corresponds to your description when connect/disconnect looping with a TLS connection:

[quote user=""]On the 21st connection/disconnection cycle the function gets stuck[/quote]

We are currently testing and implementing this fix into a newer modem fw version (v1.2.2). At this time, with the currently available modem fw v1.2.0 or older), we unfortunately do not have a good workaround (a reset shall help, but that is not a good fix).

I must apologize for the inconvenience this has caused.

Kind regards,

Håkon