Calls to MQTT library hang indefinitely when client is not connected.

[Ethan Barry]

Hello!

I'm writing an application that uses MQTT to post some sensor data to a server. My networking code is basically a duplicate of this sample, except I set up the LTE connection before starting the network loop, have a larger volume of data to transfer, and read sensor values from serial in a different thread.

Occasionally the MQTT event handler function receives MQTT_EVT_DISCONNECT with reason code 113 (ECONNABORTED) right before making a network call (mqtt_ping(), mqtt_connect(), &c.). In that case, the network call seems to hang indefinitely. I can't tell how the example application copes with this—do I have to re-run try_to_connect() immediately before every network operation? I'd be happy if the library calls just timed out, instead of blocking indefinitely. Is there a config option for that?

As far as I can see, the documentation doesn't specify any invariants I need to uphold beside "mqtt_client struct's memory shall be resident throughout the program's lifetime." Is there some condition I'm missing? I never access any part of the MQTT data structures or library from any thread besides my networking thread, and I've verified there are no deadlocks, &c. I'm certain it is the library functions that are hanging.

Thanks in advance!

—EMB

[dejans]

Hi,

Do you get any errors when this "hanging" takes place? 

Occasionally the MQTT event handler function receives MQTT_EVT_DISCONNECT with reason code 113 (ECONNABORTED) right before making a network call (mqtt_ping(), mqtt_connect(), &c.).

Can you provide a complete log which shows ECONNABORTED?

How does your application recover from indefinite hanging?

Best regards,
Dejan

[Ethan Barry]

00> [00:04:28.296,356] <inf> tracing: ### CALLING MQTT_CONNECT().
00> [00:04:34.280,670] <inf> tracing: ### CALLED MQTT_CONNECT().
00> [00:04:34.281,036] <inf> tracing: ###### PASSED MQTT_CONNECT()!
00> [00:04:34.281,402] <inf> tracing: ###### EXITED PREPARE_FDS().
00> [00:04:34.281,738] <inf> tracing: ### CALLING WAIT().
00> [00:04:34.282,073] <inf> tracing: ###### NFDS > 0.
00> [00:04:36.282,806] <inf> tracing: ###### POLLED.
00> [00:04:38.289,764] <inf> tracing: ### * MQTT DISCONNECTED WITH ERROR -113! *
00> [00:04:38.290,588] <inf> tracing: ### CALLING MQTT_CONNECT().

This is a sample of the logs. The last line you see here was the last line logged—the call to mqtt_connect() blocked for hours, until I reset the board. As far as other errors, no, I don't get any. I've examined modem traces, and the network calls always seem to block after a TCP retransmit. However, the application may make many successful retransmissions before any blocking or hanging happens. The MQTT logging hasn't told me anything yet.

My application doesn't recover from this—that's the issue. The library calls sometimes block indefinitely, and I can't "recover" in any way, unless I preempt that operation and kill it. I'd rather not have to kill my networking thread, though.

[Ethan Barry]

This is the last line when the MQTT log level is set to debug.

[00:10:29.617,401] <dbg> net_mqtt_sock_tcp: mqtt_client_tcp_connect: (net_thread): Created socket 4

[dejans]

Hi,

Can you show your build command?
Do you use any of the mqtt_publisher sample's overlays?

Ethan Barry said:

This is a sample of the logs.

It might be useful to get full log. You can upload it using Insert->Image/video/file->Upload.

Ethan Barry said:

I've examined modem traces

You could upload modem trace as well.

Best regards,
Dejan

[Ethan Barry]

My build command is this (I'm using CMake directly):
cmake -DBOARD=nrf9160dk/nrf9160/ns -Bbuild -DDTC_OVERLAY_FILE=boards/nrf9160dk_nrf9160.overlay

Then I call make from the build directory. I have incorporated the necessary overlays into my config file, as this is only ever going to be built for one board. I'll attach that as well.

The full modem trace is in Wireshark pcapng format.

Thanks again,

—Ethan B.

05543.prj.conf

modemtrace.pcapng