Thingy91 requires reset running mqtt_simple

RE: Thingy91 requires reset running mqtt_simple

cachro2 — Tue, 24 Nov 2020 16:28:22 GMT

Hi Didrik,

Thanks for the update! I diff'd my current application source, which I am testing, against your code and did not notice much of a difference. I will try running a test with the firmware you provided as well. Currently, however, I am on day 11 of my test and experienced 11 MQTT connection resets so far. The resets do not occur at any specific time interval and sometimes are only a few hours apart. However, the code I provided above cycles the connection just fine.

Please let me know what you find.

Thanks.

RE: Thingy91 requires reset running mqtt_simple

Didrik Rokhaug — Tue, 24 Nov 2020 10:26:35 GMT

Hi, and sorry for the very late answer.

It's good to hear that you have found a workaround.

I have also set up a test myself, but it has not gotten disconnected yet.

Here is the application I am using:

devzone.nordicsemi.com/.../8360.mqtt_5F00_simple_5F00_repeating.zip

It will also enable modem tracing automatically, so you should not have to do anything else than just starting capturing the trace in the Trace Collector application.

Another possible reason for why you do not get a trace is that you have opened the wrong port. Is the "Auto device/port filter" enabled?

RE: Thingy91 requires reset running mqtt_simple

cachro2 — Wed, 18 Nov 2020 17:22:11 GMT

Still unable to take a modem trace to pinpoint the exact cause of disconnect.. the modem trace instructions do not work for the DK and software package versions specified above. Would still be interested in resolving this issue. Based on the time of the disconnect's occurrence, the issue might have something to do with certificates expiring or some timeout.

Nevertheless, for those interested, I found a simple workaround to cycle the MQTT connection upon disconnect. This way we don't lose critical information on the device with a hard reset.

void main(void)
{
	int err;

	printk("The MQTT simple sample started\n");

#if defined(CONFIG_MQTT_LIB_TLS)
	err = certificates_provision();
	if (err != 0) {
		printk("Failed to provision certificates\n");
		return;
	}
#endif /* defined(CONFIG_MQTT_LIB_TLS) */

        /* reboot MQTT modem when disconnected */
        while(1) {

                modem_configure();

                err = client_init(&client);
                if (err != 0) {
                        printk("ERROR: client_init %d\n", err);
                        break;
                }

                err = mqtt_connect(&client);
                if (err != 0) {
                        printk("ERROR: mqtt_connect %d\n", err);
                        break;
                }

                err = fds_init(&client);
                if (err != 0) {
                        printk("ERROR: fds_init %d\n", err);
                        break;
                }

                while (1) {
                        err = poll(&fds, 1, mqtt_keepalive_time_left(&client));
                        if (err < 0) {
                                printk("ERROR: poll %d\n", errno);
                                break;
                        }

                        err = mqtt_live(&client);
                        if ((err != 0) && (err != -EAGAIN)) {
                                printk("ERROR: mqtt_live %d\n", err);
                                break;
                        }

                        if ((fds.revents & POLLIN) == POLLIN) {
                                err = mqtt_input(&client);
                                if (err != 0) {
                                        printk("ERROR: mqtt_input %d\n", err);
                                        break;
                                }
                        }

                        if ((fds.revents & POLLERR) == POLLERR) {
                                printk("POLLERR\n");
                                break;
                        }

                        if ((fds.revents & POLLNVAL) == POLLNVAL) {
                                printk("POLLNVAL\n");
                                break;
                        }
                }
                
                printk("Disconnecting MQTT client...\n");
                
                err = mqtt_disconnect(&client);
                if (err) {
                        printk("Could not disconnect MQTT client. Error: %d\n", err);
                }
                
                printk("Aborting MQTT client...\n");
                
                mqtt_abort(&client);
                if (err) {
                        printk("Could not abort MQTT client. Error: %d\n", err);
                }

                nrf_delay_ms(120000);
        }

        printk("Resetting device...\n");

        NVIC_SystemReset();
}

I added a 120 second delay which was tested to be a sufficient amount of time for our AWS instance to boot back up after an accidental reboot. I also replaced the 'returns' with 'breaks' (line 23, 29, 35) because otherwise the application hung up. Unfortunately, in those scenarios, a hard reset is required.

A final note: I took the device for a 50 mile drive yesterday and was very impressed by its performance. Kudos to the Nordic team for a solid solution.

RE: Thingy91 requires reset running mqtt_simple

cachro2 — Mon, 16 Nov 2020 16:35:07 GMT

I am testing the DK at the moment.

First flipped SW5 to program the nRF52 with: nrfjprog --program nRF9160_DK_board_controller_FW.hex --sectorerase -f nrf52 -r --verify
Then flipped SW5 back to program the DK.

Since I could not find proj.conf in Segger, I modified the proj.conf directly within the folder C:\ncs\nrf\samples\nrf9160\mqtt_simple and added CONFIG_BSD_LIBRARY_TRACE_ENABLED=y as shown in the instructions.

I then closed the Segger project and re-opened/re-configured all MQTT settings. As always, I use the board name nrf9160dk_nrf9160ns to open the mqtt_simple project.

After programming, I opened LTE Link Monitor and ran AT%XMODEMTRACE=1,2 which responded with OK.

I then opened Trace Collector and it remains at 0 bytes without incrementing.

RE: Thingy91 requires reset running mqtt_simple

Didrik Rokhaug — Mon, 16 Nov 2020 16:21:51 GMT

Yes, the instructions should work for all the modem firmware versions.

Are you using the Thingy or the DK?

If you are using the DK, the board name should be nrf9160dk_nrf9160ns (when building an application that will run on the nRF9160 and not the nRF52840 board controller).

For the Thingy, you should use thingy91_nrf9160ns. Also, if you are using the Thingy, you must run the connectivity_bridge application on the nRF52840, otherwise you will not be able to get the modem trace from the Thingy.

Did you remember to add CONFIG_BSD_LIBRARY_TRACE_ENABLED=y to the prj.conf? If you are using SES, you must re-open or re-configure the project for changes to prj.conf to take effect.

Does it help to send AT%XMODEMTRACE=1,2 to the modem?

RE: Thingy91 requires reset running mqtt_simple

cachro2 — Mon, 16 Nov 2020 16:01:50 GMT

Thanks Didrik, I will generate the logs and the modem trace shortly.

Have you confirmed the modem trace instructions you sent work for modem v1.2.2? The trace collector file does not increment in size after following the instructions. I am testing with the DK. Is the board name when you open the project supposed to be nrf9160dk_nrf9160ns?

2020-11-16T16:01:35.544Z INFO Application data folder: C:\Users\KOPTER\AppData\Roaming\nrfconnect\pc-nrfconnect-tracecollector
2020-11-16T16:01:35.615Z DEBUG App pc-nrfconnect-tracecollector v1.0.5 official
2020-11-16T16:01:35.615Z DEBUG App path: C:\Users\KOPTER\.nrfconnect-apps\node_modules\pc-nrfconnect-tracecollector
2020-11-16T16:01:35.616Z DEBUG nRFConnect 3.6.0 is supported by the app (^3.0.0)
2020-11-16T16:01:35.616Z DEBUG nRFConnect path: C:\Users\KOPTER\AppData\Local\Programs\nrfconnect\resources\app.asar
2020-11-16T16:01:35.616Z DEBUG HomeDir: C:\Users\KOPTER
2020-11-16T16:01:35.616Z DEBUG TmpDir: C:\Users\KOPTER\AppData\Local\Temp
2020-11-16T16:01:36.552Z VERBOSE Could not fetch serial number for serial port at COM1
2020-11-16T16:01:36.552Z VERBOSE Could not fetch serial number for serial port at COM2
2020-11-16T16:01:39.179Z INFO Device opened
2020-11-16T16:01:41.006Z INFO Tracefile created: C:\Users\KOPTER\AppData\Roaming\nrfconnect\pc-nrfconnect-tracecollector\trace-2020-11-16T16-01-41.005Z.bin

RE: Thingy91 requires reset running mqtt_simple

Didrik Rokhaug — Mon, 16 Nov 2020 15:12:23 GMT

Yes, that error certainly explains why the device stops publishing.

The mqtt_simple sample will not try to re-connect if it looses connection, so it will not be able to (or even try) to send or receive anything after it gets disconnected.

The question then becomes why you get disconnected. Error code 57 is ENOTCONN, which means that you lost the TLS/TCP connection to the MQTT broker.

The connection could be terminated by both the broker and the client, but I can't tell which or why from the log.

Do you have any logs from the broker?

A modem trace will also let us see the IP traffic between the client and the broker. That will at least tell us who terminated the connection. It might also help us figuring out why the connection is closed.

RE: Thingy91 requires reset running mqtt_simple

cachro2 — Mon, 16 Nov 2020 14:35:14 GMT

UPDATE:

Another over night test with LTE Link Monitor running yielded a MQTT client disconnected error.

Assuming this is the issue, what is the best way to counteract disconnects automatically?

RE: Thingy91 requires reset running mqtt_simple

cachro2 — Mon, 16 Nov 2020 01:49:12 GMT

Hi Didrik,

I have conducted a number of tests over the past few days to confirm what I am seeing. I've confirmed that it is an issue that occurs with the default mqtt_simple firmware (nRF Connect Toolchain Version v1.4.0) provided in the SDK. I have removed all firmware modifications and still see the issue.

I have tested both the Thingy91 and the nRF9160 DK which yielded the same results.

My setup is as follows.

Project settings for Thingy91, as an example:

I configured publish/subscribe topic,mqtt client id and broker to point at our AWS instance.

The AWS instance is running Mosquitto MQTT broker. I launched a mosquitto_sub client which is subscribed to the unit's topic and prints the messages that it receives to a log file.

mosquitto_sub -h localhost -t "pub/test" -v | while IFS= read -r line; do printf '%s %s\n' "$(date)" "$line"; done >> pub_mqtt.log

Additionally, mosquitto_pub is set to publish a single message to the unit every hour.

#!/bin/bash
exec 3>&1 4>&2
trap 'exec 2>&4 1>&3' 0 1 2 3
exec 1>log_mqtt.out 2>&1

while true;
do
        NOW=$(date +"%m-%d-%Y %H:%M:%S")
        echo "Current date: $NOW"
        mosquitto_pub -h localhost -t "sub/test" -m "abc"
        sleep 3600
done

In my most recent test, the pub_mqtt.log shows that the unit stops posting after 23 hours.

Sat Nov 14 22:36:43 UTC 2020 pub/test abc
Sat Nov 14 23:36:44 UTC 2020 pub/test abc
Sun Nov 15 00:36:44 UTC 2020 pub/test abc
Sun Nov 15 01:36:44 UTC 2020 pub/test abc
Sun Nov 15 02:36:43 UTC 2020 pub/test abc
Sun Nov 15 03:36:44 UTC 2020 pub/test abc
Sun Nov 15 04:36:43 UTC 2020 pub/test abc
Sun Nov 15 05:36:45 UTC 2020 pub/test abc
Sun Nov 15 06:36:43 UTC 2020 pub/test abc
Sun Nov 15 07:36:44 UTC 2020 pub/test abc
Sun Nov 15 08:36:43 UTC 2020 pub/test abc
Sun Nov 15 09:36:44 UTC 2020 pub/test abc
Sun Nov 15 10:36:43 UTC 2020 pub/test abc
Sun Nov 15 11:36:44 UTC 2020 pub/test abc
Sun Nov 15 12:36:43 UTC 2020 pub/test abc
Sun Nov 15 13:36:44 UTC 2020 pub/test abc
Sun Nov 15 14:36:43 UTC 2020 pub/test abc
Sun Nov 15 15:36:44 UTC 2020 pub/test abc
Sun Nov 15 16:36:43 UTC 2020 pub/test abc
Sun Nov 15 17:36:44 UTC 2020 pub/test abc
Sun Nov 15 18:36:46 UTC 2020 pub/test abc
Sun Nov 15 19:36:45 UTC 2020 pub/test abc
Sun Nov 15 20:36:45 UTC 2020 pub/test abc
Sun Nov 15 21:36:45 UTC 2020 pub/test abc

All of my other tests looked very similar. When the unit froze, I attempted to manually publish a single message in the terminal via mosquitto_pub with no response. However, after I switched the Thingy91 on/off it resumed posting.

So the setup above is fairly simple and easy to recreate.

Do you have any logs you can share? Can you produce some?

Could you take a modem trace, so we can see what happens on the modem side?

I have not yet taken a modem trace but will do so if need be. For the most part, everything is set to default as far as testing goes. I am connected to the Thingy91 via an external debugger (nrf9160DK) which does not allow me to run LTE Link Monitor to monitor the status of the Thingy. I will therefore monitor the status of the nrf9160DK running the firmware as opposed to the Thingy overnight.

Is all the communication going from the device to the cloud? Or do you send data both ways?

Default mqtt_simple firmware which echos message sent.

What QoS level are you using?

The default which is QoS 0.

What SIM card are you using, and what network are you connected to?

iBasis SIM cards that came with the Thingy91 and DK. The unit currently is connected to Verizon Wireless but also will connect to AT&T Mobility.

Are you using eDRX and/or PSM? Are you doing anything else to save power?

Whatever the default settings are in mqtt_simple firmware.

How is your logic for sending the data? Could you share some code of the modifications you have done to the sample?

No changes, logic for sending/receiving data listed above.

UPDATE:

Running mqtt_simple on the nRF9160 DK with LTE Link Monitor running in the background yielded this error:

2020-11-16T06:03:59.412Z DEBUG modem << Received: abc
2020-11-16T06:03:59.415Z DEBUG modem << Publishing: abc
2020-11-16T06:03:59.418Z DEBUG modem << to topic: pubs/ehubo_1 len: 12
2020-11-16T06:03:59.466Z DEBUG modem << [mqtt_evt_handler:238] MQTT client disconnected -57
2020-11-16T06:03:59.468Z DEBUG modem << ERROR: mqtt_live -57
2020-11-16T06:03:59.469Z DEBUG modem << Disconnecting MQTT client...
2020-11-16T06:03:59.470Z DEBUG modem << Could not disconnect MQTT client. Error: -57
2020-11-16T06:04:06.106Z DEBUG modem << +CSCON: 0

Could this be the problem? The unit is still connected to LTE and the AT commands returned good results as seen in the logs.

devzone.nordicsemi.com/.../2020_2D00_11_2D00_16T04_5F00_58_5F00_17.025Z_2D00_log.txt

I am using modem firmware mfwnrf9160122.zip for the DK.

Thank you for your support.

Robert

RE: Thingy91 requires reset running mqtt_simple

Didrik Rokhaug — Wed, 11 Nov 2020 14:36:11 GMT

Hi,

[quote user=""]Perhaps, we need to reset the radio using a specific AT command?[/quote]

No, that is not necessary.

[quote user=""]The mqtt_simple firmware does not interact with the ADP5360 PMIC via I2C like the asset_tracker firmware does. Could the PMIC be causing the issue?[/quote]

I don't think so. If you look in the board files, you will see that it initializes the PMIC. So as long as the application is compiled for the Thingy:91, that should not be a problem.

[quote user=""]I ran a power consumption test over the weekend which published data to our instance every 10 seconds and noticed that approx. every 22 hours the Thingy91 stopped publishing data to our instance. After a power cycle (on/off switch) it resumed posting. Do you know why this could be?[/quote]

Not without more information.

Do you have any logs you can share? Can you produce some?

Could you take a modem trace, so we can see what happens on the modem side?

As you are using a Thingy:91, you must make sure that you are running the connectivity_bridge application on the nRF52840. Otherwise you will not be able to get a modem trace from the Thingy.

Is all the communication going from the device to the cloud? Or do you send data both ways?

What QoS level are you using?

What SIM card are you using, and what network are you connected to?

Are you using eDRX and/or PSM? Are you doing anything else to save power?

How is your logic for sending the data? Could you share some code of the modifications you have done to the sample?

[quote user=""]One additional question: According to this documentation, the EEPROM onboard the Thingy apparently is critical for correct operation and should not be changed. Can you please elaborate what information is stored there and if this will affect our units running the mqtt_simple firmware in the field?[/quote]

I have asked our developers for more information, but I do not belive that it is the problem. My suspicion is that the data is used by the asset_tracker application and nRF Cloud. But I will come back to you on this when I know for certain.

Best regards,

Didrik