Socket error: POLLHUP, Connection was closed by the AWS IoT broker

[Nikil Rao]

Hi, 

I'm trying to periodically send sensor data to the AWS server and followed the documentation here to connect and send data to the server. I use these configurations - 

CONFIG_LTE_PSM_REQ_RPTAU="11000001" (320 hours)

CONFIG_LTE_PSM_REQ_RAT="00000001" (2 seconds)

CONFIG_MQTT_KEEPALIVE=1200 (20 min)

I have programmed the device to send data (QOS 0), poll the socket for 2 seconds and then sleeps for a specified interval of time (sleep_time). I see that if I have the sleep time anywhere between 1 to 328 seconds, the device works without any problem. If I increase the sleep time to say 358 seconds, the first message is sent properly but the second message receives a Socket error: POLLHUP, Connection was closed by the AWS IoT broker message. I've played with the CONFIG_MQTT_KEEPALIVE value but I cannot increase the data send interval above 328 seconds. Do you have any ideas how I could debug this? Thanks in advance!

Nikil

[Heidi]

Hi!

 

I've played with the CONFIG_MQTT_KEEPALIVE valu

 In what sense? Have you tried decreasing it? I believe a lot of brokers have the default keepalive time as 60 seconds.  

I think taking a modem trace is the best course of action here. That way we can see the actual TCP packets going back and forth. 

Best regards,

Heidi

[Nikil Rao]

Hi Heidi,

Thanks for the quick reply! I have tried decreasing it and increasing it but I have to send messages with an interval < 5 min in all cases to not get that error. According to the AWS documentation here (Under protocols->Connection inactivity (keep-alive interval)) it is mentioned that I can request a keep alive interval of 30-1200 seconds. I will try to take a modem trace and send it to you for analysis, thanks!

In case I cannot increase the data send interval (we would like to send data every 30 minutes) is the only solution to ping the AWS server frequently (every 5 minutes) and send data every 30 minutes? That would mean unnecessary power consumption every time we send a ping message. Is there any way I could avoid that?

Nikil

[Nikil Rao]

I also found this post which seems to be describing the same problem

[Nikil Rao]

Hi,

We have a custom board and I'll try to collect the modem trace. These are the logs I get from MQTT - 

[00:18:31.920,135] <dbg> aws_iot.aws_iot_send: Publishing to topic: test/things/protobuf/publish/testing/
[00:18:31.920,135] <dbg> net_mqtt.mqtt_publish: (0x20020ff8): [CID 0x20021df8]:[State 0x06]: >> Topic size 0x00000026, Data size 0x00000039
[00:18:31.920,257] <dbg> net_mqtt_enc.pack_utf8_str: (0x20020ff8): >> str_size:00000028 cur:0x2002919a, end:0x2002957d
[00:18:31.920,288] <dbg> net_mqtt_enc.pack_uint16: (0x20020ff8): >> val:0026 cur:0x2002919a, end:0x2002957d
[00:18:31.920,288] <dbg> net_mqtt_enc.mqtt_encode_fixed_header: (0x20020ff8): << msg type:0x30 length:0x00000061
[00:18:31.920,288] <dbg> net_mqtt_enc.packet_length_encode: (0x20020ff8): >> length:0x00000061 cur:0x00000000, end:0x00000000
[00:18:31.920,318] <dbg> net_mqtt_enc.mqtt_encode_fixed_header: (0x20020ff8): Fixed header length = 02
[00:18:31.920,318] <dbg> net_mqtt_enc.pack_uint8: (0x20020ff8): >> val:30 cur:0x20029198, end:0x2002957d
[00:18:31.920,318] <dbg> net_mqtt_enc.packet_length_encode: (0x20020ff8): >> length:0x00000061 cur:0x20029199, end:0x2002957d
[00:18:31.920,349] <dbg> net_mqtt.client_write_msg: (0x20020ff8): [0x20021df8]: Transport writing message.
[00:18:31.920,410] <dbg> net_mqtt.client_write_msg: (0x20020ff8): Transport write failed, err_code = -104, closing connection
[00:18:31.920,410] <dbg> net_mqtt_sock_tls.mqtt_client_tls_disconnect: (0x20020ff8): Closing socket 1
[00:18:31.920,471] <dbg> aws_iot.mqtt_evt_handler: MQTT_EVT_DISCONNECT: result = -104
[00:18:31.932,220] <dbg> net_mqtt.mqtt_publish: (0x20020ff8): [CID 0x20021df8]:[State 0x00]: << result 0xffffff98

I see that MQTT disconnects with an error code of -104 ( ECONNRESET 104 /* Connection reset by peer */). Can we figure out why the connection is being reset? Thanks!

Nikil

[Nikil Rao]

I debugged this further and it actually seems to be caused by a POLLHUP error. When I poll the socket after sending a message, I get a revents value of 0x11 on the file descriptor. This means that the POLLIN and POLLHUP flags were set. Since the POLLIN flag was set, I call the aws_iot_input function which finally leads to the recv function in mqtt_client_tls_read to return zero and I get disconnected from AWS.