NRF9160: Serial LTE Modem app eventually crashes after switching to DATA and AT modes numerous times

RE: NRF9160: Serial LTE Modem app eventually crashes after switching to DATA and AT modes numerous times

ronald-blackline — Mon, 28 Nov 2022 14:27:58 GMT

Hi,

Sorry took me a while to respond back. And to quickly answer your questions, we are suspecting the issue is related to UART RX and that fix you reference might be solution we are looking for. Also, our SLM is based on NCS v1.8.0 SDK. We are planning to move to the latest SDK soon but not now and I will keep this ticket updated with our new findings.

thanks,

Ronald

RE: NRF9160: Serial LTE Modem app eventually crashes after switching to DATA and AT modes numerous times

Håkon Alseth — Mon, 14 Nov 2022 16:29:06 GMT

Hi,

My apologies.

The only place the uart_tx function should return -16 (EBUSY) is if the transmitter is already active:

https://github.com/nrfconnect/sdk-zephyr/blob/v2.7.0-ncs1/drivers/serial/uart_nrfx_uarte.c#L710

Q1: However, since there's is a larger time gap between the transfers, could the issue you're seeing be related to the UART RX?

More specifically, this fix in newer NCS versions (present in ncs v2.0.0): https://github.com/nrfconnect/sdk-zephyr/commit/ba6d9139bbbda3211e0c1f65f4ccca0719f17fa5#diff-ad7eb7605ae0d455f16d9a4183fcbaca5249e8e6f8cc7338c5dea4f1d48bd453L869-R868

Q2: Could you share more information about your setup?

What is the baudrate used, and do you have "CONFIG_SLM_EXTERNAL_XTAL=y" set locally?

Q3: Have you tried to compile SLM on a newer version of NCS to see if you see the same behavior there?

Q4: Does the issue always show itself after some minutes (8 to 10 mins) and is this issue consistent on other boards as well?

Kind regards,

Håkon

RE: NRF9160: Serial LTE Modem app eventually crashes after switching to DATA and AT modes numerous times

ronald-blackline — Mon, 14 Nov 2022 14:26:22 GMT

Håkon Alseth , any updates on this?

RE: NRF9160: Serial LTE Modem app eventually crashes after switching to DATA and AT modes numerous times

ronald-blackline — Fri, 28 Oct 2022 13:12:54 GMT

Hi,

CONFIG_HEAP_MEM_POOL_SIZE is set to 16384.

As for your other question, I lost the map file with the crash dump I initially shared. Here is a new crash dump:

00> [00:10:34.480,590] <inf> slm_at_host: datamode off pending
00> [00:10:34.490,783] <wrn> slm_at_host: uart_tx failed: -16
00> [00:10:34.490,814] <inf> slm_at_host: Exit datamode
00> [00:10:34.538,360] <err> os: ***** HARD FAULT *****
00> [00:10:34.538,360] <err> os:   Fault escalation (see below)
00> [00:10:34.538,360] <err> os: ***** BUS FAULT *****
00> [00:10:34.538,360] <err> os:   Precise data bus error
00> [00:10:34.538,391] <err> os:   BFAR Address: 0xeb00eb
00> [00:10:34.538,391] <err> os: r0/a1:  0x00eb00eb  r1/a2:  0x200248b4  r2/a3:  0x00000000
00> [00:10:34.538,391] <err> os: r3/a4:  0x00000020 r12/ip:  0x00000000 r14/lr:  0x0003c385
00> [00:10:34.538,421] <err> os:  xpsr:  0x2100001a
00> [00:10:34.538,421] <err> os: Faulting instruction address (r15/pc): 0x000367a2
00> [00:10:34.538,421] <err> os: >>> ZEPHYR FATAL ERROR 0: CPU exception on CPU 0
00> [00:10:34.538,421] <err> os: Fault during interrupt handling
00>
00> [00:10:34.538,421] <err> os: Current thread: 0x20016e88 (unknown)
00> [00:10:34.931,762] <err> fatal_error: Resetting system

Based on this new dump, the fault happened at 0x000367a2:

0003679e <sys_heap_free>:
{
   3679e:    b510          push    {r4, lr}
    if (mem == NULL) {
   367a0:    b191          cbz    r1, 367c8 <sys_heap_free+0x2a>
   367a2:    6800          ldr    r0, [r0, #0]
    set_chunk_used(h, c, false);
   367a4:    2200          movs    r2, #0
    return big_heap(h) ? 8 : 4;
   367a6:    6883          ldr    r3, [r0, #8]
   367a8:    f5b3 4f00     cmp.w    r3, #32768    ; 0x8000
   367ac:    bf2c          ite    cs
   367ae:    2408          movcs    r4, #8
   367b0:    2404          movcc    r4, #4
    return (mem - chunk_header_bytes(h) - base) / CHUNK_UNIT;
   367b2:    1b0c          subs    r4, r1, r4
   367b4:    1a24          subs    r4, r4, r0
   367b6:    08e4          lsrs    r4, r4, #3
    set_chunk_used(h, c, false);
   367b8:    4621          mov    r1, r4
   367ba:    f7ff fe7c     bl    364b6 <set_chunk_used>
    free_chunk(h, c);
   367be:    4621          mov    r1, r4
}

As for `Current thread: 0x20016e88 (unknown)`, the map file shows this:

 .bss.z_idle_threads
                0x20016e88       0x80 zephyr/kernel/libkernel.a(init.c.obj)
                0x20016e88                z_idle_threads

Please let me know if you need for information.

regards,

Ronald

RE: NRF9160: Serial LTE Modem app eventually crashes after switching to DATA and AT modes numerous times

Håkon Alseth — Wed, 26 Oct 2022 10:09:39 GMT

Hi,

Based on the PC content, the dereferencing of heap->heap lead to a fault. What is your configured heap size?

ie. CONFIG_HEAP_MEM_POOL_SIZE.

The log shows that it takes approx. 45 ms from when the uart failed to when the fault occurs. Could the content of "r14/lr: 0x0003d749" and the "Current thread: 0x20016e48 (unknown)" give any indication what happened?

Since the current thread is a ram mapped variable, you will have to check the build/zephyr/zephyr.map to see which thread this belongs to.

Kind regards,

Håkon