MQTT return ERROR after 24 hours

[RobertT]

Hello!

 I am sending MQTT commands over UART from an NRF52840 DK to the NRF9160 DK which has the ibasis sim card inside. I noticed 3 times so far that after 24 hours , 1 hour tolerance, I get only  ERROR messages returned after sending the MQTT commands with the payload.  Basically I need to reset the NRF52840 DK and  re-send all the MQTT commands for connecting to the MQTT broker, autenthificating, opening connection and initialising the modem ( AT+CFUN = 1 etc) .

 I was wondering if there were previous experiences like this.  I did use an osciloscope and the NRF52840 DK is sending the right commands with the good payload over UART even after 24 hours, it's just like the modem closes the connection or something...

All the best,

Robert.

[RobertT]

 The old set-up but with the new mqtt broker: "  AT#XMQTTCON=1,"nRF9160_apr_test_21","","","test.mosquitto.org",1883 "  stoped again after 24 hours.

 The new set-up but with old mqtt broker still works after 3 days and counting.

Logs attached from the old set-up with the extra config added : " CONFIG_AT_CMD_LOG_LEVEL_DBG=y "  .

 The only thing different between them 2 set-ups are :

- hardware...which is obvious;

- mqtt broker;

- and possibly, I'm not sure if or if it is relevant, the certificates that I added...the one that does not work I added standard the nrf cloud certificates but on the new one, which still works, I plaid around a couple of months back and I think it has the google cloud or aws certificates;

Other than the ones above I can't think of any differences between the 2. The modem firmware is the same on both : "  mfw_nrf9160_1.2.3 " .

All the best,

Robert.

Fullscreen logs6.log Download

# SEGGER J-Link RTT Viewer V6.98b Data Log File
# Compiled: 15:05:00 on Mar 12 2021
# Logging started @ 22 Apr 2021 19:09:37

# Logging stopped @ 25 Apr 2021 15:41:50

logs_terminal6.log

[Carl Richard]

Hi again, Robert!

Thanks, I can still see the same errors as earlier, with the the MQTTPUB command missing the "A" in AT. In addition, it seems like the extra logging wasn't enabled properly. How are you configuring the project? 

The certificates should not affect the behavior, other than affecting which cloud services that can be connected to. How is the hardware different in the second setup?

Best regards,
Carl Richard

[RobertT]

After some while getting back with some updates on the nrf9160 behaviour and also some extra info which hopefully might help.

 On site testing I placed 2 nrf9160 dk's 100m apart from one another, communicating to the same broker, same data, same setup etc.

 Test were not comprehensive and only been on them for like 2 days. I noticed that both modems disconnected in the same time. I saw this on the database timestamp I put on the data.  This happened twice and also a third time I was in the virtual google cloud machine console looking at the mosquitto topic and got nothing from neither of them in the same time, so they offline in the same time.

 On the broker side, mosquitto hosted in google cloud machine, I do get a timeout after 60s . I think there is no ping for 60s so the broker closes the connection.  Mosquitto does log same message  when I just turn off the modems, just wanted to see if i might get a different log error for when they are off and when they stop communicating, had to test out this also...

 This does not explain why the modems do not reconnect and start sending data again, since they do have voltage and they do get the AT commands over uart , which are the right commands since I put an oscilloscope probe on the traces .

 Related to the missing "A" in the "AT" , yes,  more than certain there are sometimes, some packets sent incompletely or with strange characters but the modems do not disconnect because of that. I am saying this because I also saw live such a packet with erroneous characters and the modem staid connected afterwards.

  When the modem goes into an off the network state if I turn it off and on again everything works fine, therefore I can also exclude that the network coverage goes away completely.

 Is there a possibility for the modem to loose network signal at some point  for a short period and then just can't connect again without the hard reset? I am asking this because I went on the Ibasis website and saw that there is limited coverage for UK and also I could not get hold of a Vodafone iot sim card so far to test them out also; Vodafone present as offering nbiot in uk.

Yes, it sounds stupid, but I'm expecting for the hard reset logic, network discovery and connection/reconnection wise, to be the same as when the modem losses network coverage and tries to reconnect again.

Please advise of anything I might try, logs I might give back, to reach a stable state  with the modems.

The area I am testing is Cambridge but did had the same behaviour also somewhere like 20 miles outside Cambridge.

 At the moment I have a timer which does a hard reset of the voltage every 3 hours and it does the trick.

All the best,

Robert.

[Carl Richard]

Hello again, Robert!

Apologies for the delayed answer and thanks for the elaborate description and testing. I may have focused to much on the application side of things, while it seems like your problems may be related to the modem. The best approach now will be to get a modem trace, so that we can get some insight into the modem behavior. Please follow this guide when doing the trace.

The device should be able to reconnect without a hard reset, but it may be that the disconnection renders it in an erroneous state. Does the device still respond to AT commands when offline?

Thank you for your patience.

Best regards,
Carl Richard

[RobertT]

Hello, Carl!

 I know that the modem does not reply to AT commands once offline , the answer I got was " ERROR" to all commands sent once it went offline. The interesting part is that it happened to all modems that I used so far on site in Cambridge; only once I got a modem working for a couple  of days continuously on my bench and I am about 20 miles outside Cambridge. Taking this network coverage related path, sometime this week or early next week I should receive an O2 IoT sim card which I want to try out and see if the same behaviour happens.

 Is it possible to monitor the dk that I have on site, since that will have valuable information? I am asking this because so far I had the monitored dk connected to my PC and the one on site is on battery.

I have opened a couple of day back a topic related to modem re-connection. Should I close it and continue here? ( devzone.nordicsemi.com/.../nrf9160-dk-fails-to-reconnect-to-network )

All the best,

Robert.