NRF9160 SLM self reboot cause

RE: NRF9160 SLM self reboot cause

Hakon — Wed, 26 Mar 2025 12:01:46 GMT

[quote user="9160-DEV"]P0.26, P0.30-31[/quote]

I believe these could work, if you can use them as uart pins. Otherwise I assume you can use the SWD interface with RTT as modem trace backend. You can read this comment.

RE: NRF9160 SLM self reboot cause

9160-DEV — Mon, 17 Mar 2025 08:28:27 GMT

I can but the issue is I don't know how.

I have some pins broken out to headers that I can collect data from (P0.26, P0.30-31, COEX0-2, VIO/SCLK/SDATA, MAGPIO0-2). Could it be configured to use on any of those? I assume then a TTL/USB converter & collect the data via a serial port and the trace app?

RE: NRF9160 SLM self reboot cause

Hakon — Mon, 17 Mar 2025 08:13:32 GMT

Would it be possible to capture a modem trace (PCAP)? That could reveal why you get the MQTT disconnect.

RE: NRF9160 SLM self reboot cause

9160-DEV — Thu, 13 Feb 2025 08:38:27 GMT

Just to follow up on this, I don't think it's a network issue. I've added code to detect the stuck command (no OK/ERROR within 2 minutes) and reboot the NRF. The rebooted SLM application connects to the network and starts working again straight away. If I go back to waiting for the OK/ERROR response from the MQTTPUB, it can take anywhere between 40mins and a couple of hours to resume working,

RE: NRF9160 SLM self reboot cause

9160-DEV — Mon, 10 Feb 2025 14:31:53 GMT

Hi Hakon,

I think I'm understanding where the issue comes from. I have now seen 24h running, and it looks like the original problem was caused by my misunderstanding that the SLM application can only receive one command at a time, and when there was an issue I was overloading it with repeated commands. I changed my code to wait for OK/ERROR responses and it is a lot more stable.

However, you recall from my last message that it sometimes takes a long time for the MQTT PUB to respond, this is still an issue.

When the MQTT delay occurs, the rest of the SLM appears to run in the background but the messages aren't output until the command is complete (eg after 1h). I have therefore disabled some debugging (hex output) and turned off %CESQ messages. This has removed corrupted messages that were being received by the Arduino after some time.

Please see the below log:

(01:56:55.953) AT#XMQTTPUB="sensor/255/battery/V","11.66",0,0
(01:58:55.932) Stuck cmd: AT#XMQTTPUB="sensor/255/battery/V","11.66",0,0		<----- (No response received)
(02:39:46.635) #XMQTTEVT: 1,-113
(02:39:46.635) MQTT Disconnected.ERROR
(02:39:46.635) 
(02:39:46.635) +CEREG: 5,"33EF","0031E215",9,,,"11100000","00101000"
(02:39:46.635) Connected
(02:39:46.665) +CEREG: 5,"1BFA","0036821F",9,,,"11100000","00101000"
(02:39:46.665) Connected
(02:39:46.665) +CEREG: 5,"33EF","0031E215",9,,,"11100000","00101000"
(02:39:46.665) Connected
(02:39:46.665) +CEREG: 5,"1BFA","0036821F",9,,,"11100000","00101000"
(02:39:46.678) Connected
(02:39:46.678) +CEREG: 5,"33EF","0031E215",9,,,"11100000","00101000"
(02:39:46.678) Connected
(02:39:46.678) +CEREG: 5,"1BFA","0036821F",9,,,"11100000","00101000"
(02:39:46.678) Connected
(02:39:46.678) +CEREG: 5,"33EF","0031E215",9,,,"11100000","00101000"
(02:39:46.695) Connected
(02:39:46.695) +CEREG: 5,"1BFA","0036821F",9,,,"11100000","00101000"
(02:39:46.695) Connected
(02:39:46.695) +CEREG: 5,"33EF","0031E215",9,,,"11100000","00101000"
(02:39:46.695) Connected
(02:39:46.695) +CEREG: 5,"1BFA","0036821F",9,,,"11100000","00101000"
(02:39:46.710) Connected
(02:39:46.710) +CEREG: 5,"33EF","0031E215",9,,,"11100000","00101000"
(02:39:46.710) Connected
(02:39:46.710) +CEREG: 5,"1812","0033751E",7,,,"11100000","00101000"
(02:39:46.710) Connected
(02:39:46.710) +CGEV: ME PDN DEACT 0
(02:39:46.730) +CNEC_EMM: 9
(02:39:46.730) +CEREG: 2,"1812","0033751E",7,0,9
(02:39:46.730) Not Connected: 2
(02:39:47.131) AT%XTEMP?
(02:39:47.149) CPU Temperature: 30 deg
(02:39:47.149) OK
(02:39:47.643) 
(02:39:47.643) SLM: [10:00:56.345,458] <ESC>[1;31m<err> slm_at_host: AT command failed: -113<ESC>[0m
(02:39:47.643) SLM: [10:00:56.345,733] <ESC>[1;31m<err> slm_mqtt: ERROR: mqtt_live -128<ESC>[0m
(02:39:47.643) SLM: [10:00:56.345,733] <ESC>[0m<inf> slm_mqtt: MQTT thread terminated<ESC>[0m
(02:39:48.139) AT%XTEMP?
(02:39:48.139) CPU Temperature: 30 deg
(02:39:48.139) OK

In it you can see that XMQTTPUB did not receive an OK/ERROR for >40 minutes. It was sent at 01:56:55 and the ERROR was received at 02:39.46. In that time no other commands were sent to the SLM.

Once the SLM started responding, lots of CEREG messages come in. I believe these were buffered by the SLM but couldn't be sent earlier because the MQTT was blocking.

In this case the SLM application was effectively stuck for 40minutes even though there was a valid connection (CEREG: 5). There must either be a faulty timeout in the SLM, or some kind of bug. There is no other debug code.

I'm unable to get any extra debug/monitoring because this application only has UART, no RTT. I can try to add RTT but unsure of the process. As I said before, I'm happy to buy a Segger, but unsure how to implement it. I'm happy to add the full log, but I don't think there's much info in there that will help - there are limited SLM debug messages and it basically shows it sending PUB commands for 24h (~2000 commands) before one fails, which is in the log above.

Is there a reason the SLM blocks for such a large time, or is there a delay that can be shorted?

Thanks

RE: NRF9160 SLM self reboot cause

Hakon — Thu, 06 Feb 2025 11:07:57 GMT

Can I see the code? Also, if possible can you please capture a modem trace of all relevant errors?

RE: NRF9160 SLM self reboot cause

9160-DEV — Tue, 04 Feb 2025 17:09:11 GMT

Hi Hakon,

I've not seen the reset again following the change to print out the RESETREAS register unfortunately.

I have however seen the other issue lots - I think there's a problem in the SLM, or down to the way I'm using it. It appears to get stuck, perhaps there's a timeout somewhere I could change?

The general pattern is to request the temperature, then send an MQTT publish, for example;

(02:06:45.772) AT%XTEMP?
(02:06:45.806) CPU Temperature: 23 deg
(02:06:45.806) OK
(02:06:47.772) AT#XMQTTPUB="sensor/255/cpu_temp/C","23",0,0
(02:06:47.794) OK

Then after some time, there is no response to the AT#MQTTPUB command. The reason for UART_RX_RDY failure appears to be that the buffer is full because I had continued to send AT commands - so I have changed the code to wait for OK/ERROR response instead.

Now after some time waiting for OK, there is a heap overflow where the CESQ isn't output by the SLM but is stored internally;

(02:44:17.964) SLM: [04:06:22.879,882] <ESC>[1;33m<wrn> at_monitor: No heap space for incoming notification: %CESQ: 28,1,11,1
(02:44:28.236) SLM: [04:06:33.144,195] <ESC>[1;33m<wrn> at_monitor: No heap space for incoming notification: %CESQ: 29,1,3,0
(02:44:38.860) SLM: [04:06:43.771,484] <ESC>[1;33m<wrn> at_monitor: No heap space for incoming notification: %CESQ: 29,1,13,1
...

Then, after some hours (of nothing being output rather than the heap error), the OK/ERROR finally arrives and notifications start coming again;

(05:56:02.744) #XMQTTEVT: 1,-113
(05:56:02.788) ERROR
(05:56:02.788) ERROR
(05:56:02.788) %XTEMP: 21
(05:56:02.788) CPU Temperature: 21 deg
(05:56:02.788) OK
(05:56:02.788) %CESQ: 28,1,7,1

I've looked at the SLM code and can't see an obvious reason for the pause... but then I don't really understand it. Also how can I get to see the MQTT error output via UART (eg "NET_ERR"), which might help?

Thanks

RE: NRF9160 SLM self reboot cause

Hakon — Tue, 28 Jan 2025 15:14:56 GMT

I don't think there is an AT command for that.

RE: NRF9160 SLM self reboot cause

9160-DEV — Tue, 28 Jan 2025 12:21:25 GMT

Hello Hakon,

I haven’t. Is it possible to check that via the SLM AT commands, or should I make changes to the SLM code to output the register on start?

Many thanks

RE: NRF9160 SLM self reboot cause

Hakon — Tue, 28 Jan 2025 11:43:47 GMT

Hello,

[quote user="9160-DEV"][EDIT] Maybe not - I have seen the reboot again too - perhaps there are two issues!? [/quote]

It resets even after setting

CONFIG_RESET_ON_FATAL_ERROR=n

Have you checked the RESETREAS register?

RE: NRF9160 SLM self reboot cause

9160-DEV — Mon, 27 Jan 2025 19:14:38 GMT

Unsure if this is progress, but I have made a change to the prj options and have a different error. I saw in another answer to try

CONFIG_RESET_ON_FATAL_ERROR=n

CONFIG_THREAD_NAME=y

I have started to see the below error (last 2 runs, after a couple of hours...)

slm_uart_handler: UART_RX_RDY failure: -35

(18:59:47.348) AT%XTEMP?
(18:59:47.348) AT#XMQTTPUB="sensor/255/battery/V","11.59",0,0
(18:59:47.348) CPU Temperature: 30 deg
(18:59:47.348) AT#XMQTTPUB="sensor/255/cpu_temp/C","30",0,0
(18:59:48.355) SLM: [01:50:07.503,997] <ESC>[0m<dbg> slm_at_host: cmd_send: RX
(18:59:48.391) SLM:                                       41 54 25 58 54 45 4d 50  3f                      |AT%XTEMP ?       <ESC>[0m
(18:59:48.391) SLM: [01:50:07.504,791] <ESC>[0m<dbg> slm_at_host: slm_at_send_indicate: TX
(18:59:48.391) SLM:                                       0d 0a 25 58 54 45 4d 50  3a 20 33 30 0d 0a 0d 0a |..%XTEMP : 30....
(18:59:48.391) SLM:                                       4f 4b 0d 0a                                      |OK..             <ESC>[0m
(18:59:48.424) SLM: [01:50:07.504,852] <ESC>[0m<dbg> slm_at_host: cmd_send: RX
(18:59:48.424) SLM:                                      41 54 23 58 4d 51 54 54  0 42 3d 22 73 65 6e |AT#XMQTT PUB="sen
(18:59:48.424) SLM:                                       73 6f 72 2f 32 35 35 2f   61 74 74 65 72 79 2f |sor/255/ battery/
(18:59:48.424) SLM:                                       56 22 2c 22 31 31 2e 35  22 2c 30 2c 30       |V","11.5 9",0,0  <ESC>[0m
(19:00:17.219) AT%XTEMP?
(19:00:17.219) AT#XMQTTPUB="sensor/255/battery/V","11.63",0,0
(19:00:47.106) AT%XTEMP?
(19:00:47.106) AT#XMQTTPUB="sensor/255/battery/V","11.63",0,0
(19:01:16.993) AT%XTEMP?
(19:01:16.993) AT#XMQTTPUB="sensor/255/battery/V","11.63",0,0
(19:01:46.863) AT%XTEMP?
(19:01:46.863) AT#XMQTTPUB="sensor/255/battery/V","11.63",0,0
(19:02:16.750) AT%XTEMP?
(19:02:16.750) AT#XMQTTPUB="sensor/255/battery/V","11.63",0,0
(19:02:17.774) SLM: [01:52:36.916,259] <ESC>[1;31m<err> slm_uart_handler: UART_RX_RDY failure: -35, dropped: 59<ESC>[0m
(19:02:46.638) AT%XTEMP?
(19:02:46.638) AT#XMQTTPUB="sensor/255/battery/V","11.63",0,0
(19:02:47.645) SLM: [01:53:06.796,569] <ESC>[1;31m<err> slm_uart_handler: UART_RX_RDY failure: -35, dropped: 59<ESC>[0m
(19:03:16.508) AT%XTEMP?
(19:03:16.508) AT#XMQTTPUB="sensor/255/battery/V","11.63",0,0
(19:03:17.532) SLM: [01:53:36.674,346] <ESC>[1;31m<err> slm_uart_handler: UART_RX_RDY failure: -35, dropped: 21<ESC>[0m
(19:03:17.548) SLM: [01:53:36.674,407] <ESC>[1;33m<wrn> slm_uart_handler: Disabling UART RX: No event space.<ESC>[0m
(19:03:17.564) SLM: [01:53:36.679,992] <ESC>[1;31m<err> slm_uart_handler: UART_RX_RDY failure: -35, dropped: 38<ESC>[0m
(19:03:46.395) AT%XTEMP?
(19:03:46.395) AT#XMQTTPUB="sensor/255/battery/V","11.63",0,0
(19:03:47.419) SLM: [01:54:06.564,178] <ESC>[1;31m<err> slm_uart_handler: UART_RX_RDY failure: -35, dropped: 59<ESC>[0m
(19:04:16.282) AT%XTEMP?
(19:04:16.282) AT#XMQTTPUB="sensor/255/battery/V","11.63",0,0
(19:04:17.306) SLM: [01:54:36.447,021] <ESC>[1;31m<err> slm_uart_handler: UART_RX_RDY failure: -35, dropped: 59<ESC>[0m
(19:04:46.169) AT%XTEMP?
(19:04:46.169) AT#XMQTTPUB="sensor/255/battery/V","11.56",0,0
(19:04:47.193) SLM: [01:55:06.335,906] <ESC>[1;31m<err> slm_uart_handler: UART_RX_RDY failure: -35, dropped: 59<ESC>[0m
(19:05:16.056) AT%XTEMP?
(19:05:16.056) AT#XMQTTPUB="sensor/255/battery/V","11.59",0,0
(19:05:17.064) SLM: [01:55:36.217,376] <ESC>[1;31m<err> slm_uart_handler: UART_RX_RDY failure: -35, dropped: 41<ESC>[0m
(19:05:45.943) AT%XTEMP?
(19:05:45.943) AT#XMQTTPUB="sensor/255/battery/V","11.66",0,0
(19:06:15.814) AT%XTEMP?
(19:06:15.830) AT#XMQTTPUB="sensor/255/battery/V","11.63",0,0
(19:06:45.701) AT%XTEMP?
(19:06:45.701) AT#XMQTTPUB="sensor/255/battery/V","11.63",0,0
(19:07:15.604) AT%XTEMP?
(19:07:15.604) AT#XMQTTPUB="sensor/255/battery/V","11.56",0,0

Is that progress... ?! Any ideas? Thanks!

[EDIT] Maybe not - I have seen the reboot again too - perhaps there are two issues!? 😞