Thingy91 requires reset running mqtt_simple

[cachro2]

Hi,

We are using the Thingy91, which is running the mqtt_simple (nRF SDK 1.4 and modem firmware 1.2.2) example, to push data direct to our AWS instance. Everything is running smoothly aside from the following issue:

I ran a power consumption test over the weekend which published data to our instance every 10 seconds and noticed that approx. every 22 hours the Thingy91 stopped publishing data to our instance. After a power cycle (on/off switch) it resumed posting. Do you know why this could be?

The mqtt_simple firmware does not interact with the ADP5360 PMIC via I2C like the asset_tracker firmware does. Could the PMIC be causing the issue?

Perhaps, we need to reset the radio using a specific AT command? I have successfully implemented AT command functionality within mqtt_simple which we can use if need be.

One additional question: According to this documentation, the EEPROM onboard the Thingy apparently is critical for correct operation and should not be changed. Can you please elaborate what information is stored there and if this will affect our units running the mqtt_simple firmware in the field?

Thanks,

Robert

[Didrik Rokhaug]

Yes, the instructions should work for all the modem firmware versions.

Are you using the Thingy or the DK?

If you are using the DK, the board name should be nrf9160dk_nrf9160ns (when building an application that will run on the nRF9160 and not the nRF52840 board controller).

For the Thingy, you should use thingy91_nrf9160ns. Also, if you are using the Thingy, you must run the connectivity_bridge application on the nRF52840, otherwise you will not be able to get the modem trace from the Thingy.

Did you remember to add CONFIG_BSD_LIBRARY_TRACE_ENABLED=y to the prj.conf? If you are using SES, you must re-open or re-configure the project for changes to prj.conf to take effect.

Does it help to send AT%XMODEMTRACE=1,2 to the modem?

[cachro2]

I am testing the DK at the moment. 

First flipped SW5 to program the nRF52 with: nrfjprog --program nRF9160_DK_board_controller_FW.hex --sectorerase -f nrf52 -r --verify
Then flipped SW5 back to program the DK.

Since I could not find proj.conf in Segger, I modified the proj.conf directly within the folder C:\ncs\nrf\samples\nrf9160\mqtt_simple and added CONFIG_BSD_LIBRARY_TRACE_ENABLED=y as shown in the instructions.

I then closed the Segger project and re-opened/re-configured all MQTT settings. As always, I use the board name nrf9160dk_nrf9160ns to open the mqtt_simple project.

After programming, I opened LTE Link Monitor and ran AT%XMODEMTRACE=1,2 which responded with OK.

I then opened Trace Collector and it remains at 0 bytes without incrementing. 

[cachro2]

Still unable to take a modem trace to pinpoint the exact cause of disconnect.. the modem trace instructions do not work for the DK and software package versions specified above. Would still be interested in resolving this issue. Based on the time of the disconnect's occurrence, the issue might have something to do with certificates expiring or some timeout.

Nevertheless, for those interested, I found a simple workaround to cycle the MQTT connection upon disconnect. This way we don't lose critical information on the device with a hard reset. 

 

void main(void)
{
	int err;

	printk("The MQTT simple sample started\n");

#if defined(CONFIG_MQTT_LIB_TLS)
	err = certificates_provision();
	if (err != 0) {
		printk("Failed to provision certificates\n");
		return;
	}
#endif /* defined(CONFIG_MQTT_LIB_TLS) */

        /* reboot MQTT modem when disconnected */
        while(1) {

                modem_configure();

                err = client_init(&client);
                if (err != 0) {
                        printk("ERROR: client_init %d\n", err);
                        break;
                }

                err = mqtt_connect(&client);
                if (err != 0) {
                        printk("ERROR: mqtt_connect %d\n", err);
                        break;
                }

                err = fds_init(&client);
                if (err != 0) {
                        printk("ERROR: fds_init %d\n", err);
                        break;
                }

                while (1) {
                        err = poll(&fds, 1, mqtt_keepalive_time_left(&client));
                        if (err < 0) {
                                printk("ERROR: poll %d\n", errno);
                                break;
                        }

                        err = mqtt_live(&client);
                        if ((err != 0) && (err != -EAGAIN)) {
                                printk("ERROR: mqtt_live %d\n", err);
                                break;
                        }

                        if ((fds.revents & POLLIN) == POLLIN) {
                                err = mqtt_input(&client);
                                if (err != 0) {
                                        printk("ERROR: mqtt_input %d\n", err);
                                        break;
                                }
                        }

                        if ((fds.revents & POLLERR) == POLLERR) {
                                printk("POLLERR\n");
                                break;
                        }

                        if ((fds.revents & POLLNVAL) == POLLNVAL) {
                                printk("POLLNVAL\n");
                                break;
                        }
                }
                
                printk("Disconnecting MQTT client...\n");
                
                err = mqtt_disconnect(&client);
                if (err) {
                        printk("Could not disconnect MQTT client. Error: %d\n", err);
                }
                
                printk("Aborting MQTT client...\n");
                
                mqtt_abort(&client);
                if (err) {
                        printk("Could not abort MQTT client. Error: %d\n", err);
                }

                nrf_delay_ms(120000);
        }

        printk("Resetting device...\n");

        NVIC_SystemReset();
}

I added a 120 second delay which was tested to be a sufficient amount of time for our AWS instance to boot back up after an accidental reboot. I also replaced the 'returns' with 'breaks' (line 23, 29, 35) because otherwise the application hung up. Unfortunately, in those scenarios, a hard reset is required.

A final note: I took the device for a 50 mile drive yesterday and was very impressed by its performance. Kudos to the Nordic team for a solid solution.

[Didrik Rokhaug]

Hi, and sorry for the very late answer.

It's good to hear that you have found a workaround.

I have also set up a test myself, but it has not gotten disconnected yet.

Here is the application I am using:

8360.mqtt_simple_repeating.zip

It will also enable modem tracing automatically, so you should not have to do anything else than just starting capturing the trace in the Trace Collector application.

Another possible reason for why you do not get a trace is that you have opened the wrong port. Is the "Auto device/port filter" enabled?

[cachro2]

Hi Didrik,

Thanks for the update! I diff'd my current application source, which I am testing, against your code and did not notice much of a difference. I will try running a test with the firmware you provided as well. Currently, however, I am on day 11 of my test and experienced 11 MQTT connection resets so far. The resets do not occur at any specific time interval and sometimes are only a few hours apart. However, the code I provided above cycles the connection just fine.

Please let me know what you find.

Thanks.