MQTT return ERROR after 24 hours

RE: MQTT return ERROR after 24 hours

Carl Richard — Mon, 14 Jun 2021 15:07:19 GMT

Hi, Robert!

Thank you for the update! It's great to hear that things seemingly are running better and more reliable now. However, I don't like the fact that it misbehaves to the degree we have seen here with bad reception. Hopefully a modem trace could give some information as well!

Best regards,
Carl Richard

RE: MQTT return ERROR after 24 hours

RobertT — Mon, 14 Jun 2021 11:57:17 GMT

An update:

- last week I got the new O2 sim card ( for IoT ) and took the set up I had on site and brought it on the bench ( nrf52840 dk communicating over uart to nrf9160 dk ) . Did not change anything on it except for the sim card.

- I am now into the 5th day of the system running continuously without issues.

A next step would be to get the same set up with O2 sim card on site and see how it behaves in that area or put the O2 sim card into another set up like this ( just different devices ) on the bench. I will do one of them this week and let you know again what happened.

All the best,

Robert.

RE: MQTT return ERROR after 24 hours

Carl Richard — Fri, 28 May 2021 08:54:08 GMT

Hi again!

Understood. This means that the nRF9160 itself likely has crashed. If it happens to all modems it's not a faulty device at least, and more likely a bug in the code/modem. We just released a new modem version (v1.3.0), so you could look into upgrading one of the devices to see if that makes a difference as well. Looking forward to new results with a different SIM!

In order to monitor the on site device you must either save relevant information to the flash, so that it can be read out afterward or have it connected to a computer for UART logging. Given that the same issues arise from the device connected to your computer I think the best approach is to get the logs and trace from that device first.

I've spoken with the engineer that have taken your other case and we've agreed that he will follow you up further, if that's okay for you. However, I'll leave this case open and I will monitor the other case so that I can assist when necessary.

Best regards,
Carl Richard

RE: MQTT return ERROR after 24 hours

RobertT — Thu, 27 May 2021 15:26:16 GMT

Hello, Carl!

I know that the modem does not reply to AT commands once offline , the answer I got was " ERROR" to all commands sent once it went offline. The interesting part is that it happened to all modems that I used so far on site in Cambridge; only once I got a modem working for a couple of days continuously on my bench and I am about 20 miles outside Cambridge. Taking this network coverage related path, sometime this week or early next week I should receive an O2 IoT sim card which I want to try out and see if the same behaviour happens.

Is it possible to monitor the dk that I have on site, since that will have valuable information? I am asking this because so far I had the monitored dk connected to my PC and the one on site is on battery.

I have opened a couple of day back a topic related to modem re-connection. Should I close it and continue here? ( devzone.nordicsemi.com/.../nrf9160-dk-fails-to-reconnect-to-network )

All the best,

Robert.

RE: MQTT return ERROR after 24 hours

Carl Richard — Thu, 27 May 2021 15:08:04 GMT

Hello again, Robert!

Apologies for the delayed answer and thanks for the elaborate description and testing. I may have focused to much on the application side of things, while it seems like your problems may be related to the modem. The best approach now will be to get a modem trace, so that we can get some insight into the modem behavior. Please follow this guide when doing the trace.

The device should be able to reconnect without a hard reset, but it may be that the disconnection renders it in an erroneous state. Does the device still respond to AT commands when offline?

Thank you for your patience.

Best regards,
Carl Richard

RE: MQTT return ERROR after 24 hours

RobertT — Tue, 18 May 2021 23:13:33 GMT

After some while getting back with some updates on the nrf9160 behaviour and also some extra info which hopefully might help.

On site testing I placed 2 nrf9160 dk's 100m apart from one another, communicating to the same broker, same data, same setup etc.

Test were not comprehensive and only been on them for like 2 days. I noticed that both modems disconnected in the same time. I saw this on the database timestamp I put on the data. This happened twice and also a third time I was in the virtual google cloud machine console looking at the mosquitto topic and got nothing from neither of them in the same time, so they offline in the same time.

On the broker side, mosquitto hosted in google cloud machine, I do get a timeout after 60s . I think there is no ping for 60s so the broker closes the connection. Mosquitto does log same message when I just turn off the modems, just wanted to see if i might get a different log error for when they are off and when they stop communicating, had to test out this also...

This does not explain why the modems do not reconnect and start sending data again, since they do have voltage and they do get the AT commands over uart , which are the right commands since I put an oscilloscope probe on the traces .

Related to the missing "A" in the "AT" , yes, more than certain there are sometimes, some packets sent incompletely or with strange characters but the modems do not disconnect because of that. I am saying this because I also saw live such a packet with erroneous characters and the modem staid connected afterwards.

When the modem goes into an off the network state if I turn it off and on again everything works fine, therefore I can also exclude that the network coverage goes away completely.

Is there a possibility for the modem to loose network signal at some point for a short period and then just can't connect again without the hard reset? I am asking this because I went on the Ibasis website and saw that there is limited coverage for UK and also I could not get hold of a Vodafone iot sim card so far to test them out also; Vodafone present as offering nbiot in uk.

Yes, it sounds stupid, but I'm expecting for the hard reset logic, network discovery and connection/reconnection wise, to be the same as when the modem losses network coverage and tries to reconnect again.

Please advise of anything I might try, logs I might give back, to reach a stable state with the modems.

The area I am testing is Cambridge but did had the same behaviour also somewhere like 20 miles outside Cambridge.

At the moment I have a timer which does a hard reset of the voltage every 3 hours and it does the trick.

All the best,

Robert.

RE: MQTT return ERROR after 24 hours

Carl Richard — Tue, 27 Apr 2021 12:44:56 GMT

Hi again, Robert!

Thanks, I can still see the same errors as earlier, with the the MQTTPUB command missing the "A" in AT. In addition, it seems like the extra logging wasn't enabled properly. How are you configuring the project?

The certificates should not affect the behavior, other than affecting which cloud services that can be connected to. How is the hardware different in the second setup?

Best regards,
Carl Richard

RE: MQTT return ERROR after 24 hours

RobertT — Sun, 25 Apr 2021 14:51:34 GMT

The old set-up but with the new mqtt broker: " AT#XMQTTCON=1,"nRF9160_apr_test_21","","","test.mosquitto.org",1883 " stoped again after 24 hours.

The new set-up but with old mqtt broker still works after 3 days and counting.

Logs attached from the old set-up with the extra config added : " CONFIG_AT_CMD_LOG_LEVEL_DBG=y " .

The only thing different between them 2 set-ups are :

- hardware...which is obvious;

- mqtt broker;

- and possibly, I'm not sure if or if it is relevant, the certificates that I added...the one that does not work I added standard the nrf cloud certificates but on the new one, which still works, I plaid around a couple of months back and I think it has the google cloud or aws certificates;

Other than the ones above I can't think of any differences between the 2. The modem firmware is the same on both : " mfw_nrf9160_1.2.3 " .

All the best,

Robert.

devzone.nordicsemi.com/.../logs6.log

devzone.nordicsemi.com/.../logs_5F00_terminal6.log

RE: MQTT return ERROR after 24 hours

RobertT — Thu, 22 Apr 2021 18:18:04 GMT

Did a second set-up, mirroring the first one, but with the mqtt broker from above and also added the setting in .proj file. I am live here : sudo mosquitto_sub -t nRF9160_apr_test_21 -h test.mosquitto.org -p 1883 , when the sensors transmit.

When the modem goes offline / off again will get back to you.

They both run in parallel to see if they go down in the same time, etc.

Will get back once they fail.

All the best,

Robert.

RE: MQTT return ERROR after 24 hours

RobertT — Wed, 21 Apr 2021 15:57:48 GMT

Will do that and get back to you.

Thank you!

RE: MQTT return ERROR after 24 hours

Carl Richard — Wed, 21 Apr 2021 11:55:12 GMT

Hi again!

Based on the logs the nRF9160 still isn't receiving the complete MQTT AT commands. I'm not sure what's happening, but could you try to set CONFIG_AT_CMD_LOG_LEVEL_DBG=y for the next run as well. This will give us more verbose logs.

In addition, could you try to connect to a more simple MQTT broker, for example:

AT#XMQTTCON=1,"nRF9160_apr_test_21","","","test.mosquitto.org",1883

So that we can rule out any issues with the particular broker connection.

Best regards,
Carl Richard

RE: MQTT return ERROR after 24 hours

RobertT — Mon, 19 Apr 2021 17:43:53 GMT

Hello again!

I attach the 2 log files, terminal and data, with the added CONFIG_LOG_STRDUP_MAX_STRING=256 .

This are the commands that I send out :

1 - > "AT\r\n";
2 - > "AT+CFUN=1\r\n";
3 - > "AT+CFUN?\r\n";
4 - >"AT#XMQTTCON=1,\"test\",\"user\",\"12345678\",\"34.105.xxx.xxx\",10803\r\n";
5 - > * here I add the information received from a sensor* - > AT#XMQTTPUB=\"gw-event/received_data/\",1,\"$P11,2128,211,1111,52.23445786825217,0.1434466448266854,Car,CR\",1,0\r\n";

All the best,

Robert.

devzone.nordicsemi.com/.../logs5.log devzone.nordicsemi.com/.../logs55.log

RE: MQTT return ERROR after 24 hours

Carl Richard — Mon, 12 Apr 2021 10:56:45 GMT

Hi again, Robert!

Thanks for the logs. The MQTT AT-Commands seems to be malformed somehow, resulting in the nRF9160 returning an error. See below:

10> T#XMQTTPUB="gw-event/received_data/",1,"$P1~
10> [05:20:21.881,256] <dbg> at_cmd.at_cmd_write: Awaiting response for T#XMQTTPUB="gw-event/received_data/",1,"$P1~

Here the "A" in "AT" is missing, and the message is cut short as well. Could check if the correct AT command is sent from the host device? This could also be related to buffer sizes, so if possible please share a full length AT command with me so I could check that.

For later logs please add CONFIG_LOG_STRDUP_MAX_STRING=256 in your prj.conf, since the messages currently are too long to be displayed.

Best regards,
Carl Richard

RE: MQTT return ERROR after 24 hours

RobertT — Sun, 11 Apr 2021 18:35:11 GMT

Hello, Richard!

I attached 2 log files, data and terminal logging.

The NRF9160DK stoped again without any reply on the UART channel, while the NRF52840 was sending data every 20 seconds as it should. I pushed " RESET " on the NRF9160DK and started working again.

The NRF9160 DK USB is connected to the PC and the NRF52840 DK is connected to an USB mains adapter. The time when it worked for 3 days they were both connected to the mains through an USB adapter. The PC is set to never go to sleep so I do have power all the time. Doesn't make sense to mention this setup but I though to add it in also...

I started LOGGING after I connected to the NRF9160DK. If the logs are not the proper ones please let me know.

All the best,

Robert.


=====added by me manually because I started loging after I turned ON the RTT Viewer===

=====START HERE====
LOG: J-Link RTT Viewer V6.98b: Logging started.
LOG: Terminal 0 added.
LOG: Terminal 10 added.
LOG: Connecting to J-Link via USB...
LOG: Device "NRF9160_XXAA" selected.
LOG: ConfigTargetSettings() start
LOG: ---Setting ROM table---
LOG: ConfigTargetSettings() end
LOG: Found SW-DP with ID 0x6BA02477
LOG: DPIDR: 0x6BA02477
LOG: Scanning AP map to find all available APs
LOG: AP[7]: Stopped AP scan as end of AP map has been reached
LOG: AP[0]: AHB-AP (IDR: 0x84770001)
LOG: AP[1]: AHB-AP (IDR: 0x24770011)
LOG: AP[2]: JTAG-AP (IDR: 0x12880000)
LOG: AP[3]: APB-AP (IDR: 0x54770002)
LOG: AP[4]: JTAG-AP (IDR: 0x12880000)
LOG: AP[5]: JTAG-AP (IDR: 0x12880000)
LOG: AP[6]: MEM-AP (IDR: 0x128800A1)
LOG: Iterating through AP map to find AHB-AP to use
LOG: AP[0]: Core found
LOG: AP[0]: AHB-AP ROM base: 0xE00FF000
LOG: CPUID register: 0x410FD212. Implementer code: 0x41 (ARM)
LOG: Found Cortex-M33 r0p2, Little endian.
LOG: FPUnit: 8 code (BP) slots and 0 literal slots
LOG: Security extension: implemented
LOG: Secure debug: enabled
LOG: CoreSight components:
LOG: ROMTbl[0] @ E00FF000
LOG: ROMTbl[0][0]: E000E000, CID: B105900D, PID: 000BBD21 Cortex-M33
LOG: ROMTbl[0][1]: E0001000, CID: B105900D, PID: 000BBD21 DWT
LOG: ROMTbl[0][2]: E0002000, CID: B105900D, PID: 000BBD21 FPB
LOG: ROMTbl[0][3]: E0000000, CID: B105900D, PID: 000BBD21 ITM
LOG: ROMTbl[0][5]: E0041000, CID: B105900D, PID: 002BBD21 ETM
LOG: ROMTbl[0][6]: E0042000, CID: B105900D, PID: 000BBD21 CSS600-CTI
LOG: RTT Viewer connected.
LOG: Terminal logging started.
LOG: Data logging started.
=====END HERE======

=====WHAT IS BELLLOW IS ALL THAT THE SYSTEM SAID AFTERWARDS====

# SEGGER J-Link RTT Viewer V6.98b Data Log File
# Compiled: 15:05:00 on Mar 12 2021
# Logging started @ 10 Apr 2021 22:18:33

devzone.nordicsemi.com/.../logsRTT3.log

RE: MQTT return ERROR after 24 hours

Carl Richard — Fri, 09 Apr 2021 07:02:47 GMT

Thanks for the update. The nRF9160 must have crashed somehow, which is unfortunate. Hopefully the logs will provide some insight!

Best regards,
Carl Richard

RE: MQTT return ERROR after 24 hours

RobertT — Tue, 06 Apr 2021 17:11:19 GMT

An update + unknown:

- after 3 days of working the modem stoped again transmitting over MQTT; connected the oscilloscope probe to the UART and got nothing back this time, no "ERROR" message or anything else. The NRF52840 was transmitting data to the NRF9160 continuously though.

I pressed " RESET" on the NRF52840 DK and it had no effect on the NRF9160 DK, still no reply on UART ...like an " OK " back from the at commands or an " ERROR " ; I pressed " RESET" on the NRF9160 DK and I got the " READY" reply message and it started working again as expected with all the reply messages on UART.

- I started the RTT Viewer V6.98b on Windows and will get back with the logs when the modem stops next time.

All the best,

Robert.

RE: MQTT return ERROR after 24 hours

RobertT — Tue, 06 Apr 2021 14:47:04 GMT

Yes, I did confuse the 2 of them. Will get on it and get back to you. All the best!

RE: MQTT return ERROR after 24 hours

Carl Richard — Tue, 06 Apr 2021 13:18:23 GMT

Hi again!

This sample writes the logs over the RTT protocol. So if you connect to the device with the JLink RTT Viewer (or using the built in terminal in SES), the logs should be printed there.

Logs are not to be confused with a modem trace, which only can be read by us here. I don't need a modem trace from you yet, as I don't believe that this is a modem issue.

Best regards,
Carl Richard

RE: MQTT return ERROR after 24 hours

RobertT — Tue, 06 Apr 2021 11:27:54 GMT

I was saying I do not know how to access the logs. Thank you!

RE: MQTT return ERROR after 24 hours

Carl Richard — Tue, 06 Apr 2021 11:22:53 GMT

You can post the logs here either by uploading the file directly or by pressing "Insert"->"Code" and copying in the log contents. If the information is sensitive I can make this case private.

Best regards,
Carl Richard

RE: MQTT return ERROR after 24 hours

RobertT — Tue, 06 Apr 2021 10:37:36 GMT

I did enable logging "CONFIG_SLM_LOG_LEVEL_DBG=y" ...a web-link with instructions on how to pass over the logs to you? ... in the sense that I've read that only development can read the logs...

RE: MQTT return ERROR after 24 hours

Carl Richard — Tue, 06 Apr 2021 09:39:19 GMT

Hi again!

Understood. It seems like a part of your answer was lost, but I take it as the application works now? The solution is arguably not clean, as you have to send the full series of commands every time, so we could look further into this if you would like that. Again, any logs are appreciated!

It seems like your device disconnects from the broker at some point, since it works when you do the connection procedure every time.

Best regards,
Carl Richard

RE: MQTT return ERROR after 24 hours

RobertT — Tue, 06 Apr 2021 09:02:57 GMT

Hello, Richard!

In the first instance I was giving the series of commands only once in the function " void App_init(const app_global_functions_t * functions) " ...now I also Included those function with each callback when I have a message on UART, in the functions " lib_data->setDataReceivedCb(uart_write);
lib_data->setBcastDataReceivedCb(uart_write); " .

The called function "uart_write " sends the series of commands " AT\r\n ; AT+CFUN=1\r\n ; AT+CFUN?\r\n ; AT#XMQTTCON=1,"test","USER_HERE","PASSWORD_HERE","34.105.208.xxx",10803\r\n ; AT#XMQTTPUB="gw-event/received_data/",1,"PAYLOAD_HERE",1,0\r\n ; " each time.

For the last 3 days the modem worked continuously. I

RE: MQTT return ERROR after 24 hours

Carl Richard — Tue, 06 Apr 2021 08:06:52 GMT

Hi again!
Have you had any progress?

Best regards,
Carl Richard

RE: MQTT return ERROR after 24 hours

RobertT — Wed, 31 Mar 2021 12:17:43 GMT

Yes, Serial LTE Modem(SLM) . When I hit " Reset " on the nrf52840 it sends the succesion of commands again and everything start working. Will try and separate the commands to see which one makes everything working again.