We have a product with the nRF5340, which are connected in mesh networks of around 20 devices to a border router via OpenThread. We are on the latest SDK version (v.2.5.0).
We are having problems that our devices are randomly rebooting, sometimes after days, sometimes after weeks, sometimes a device is stable for days and then it gets into a endless crashing loop, with only hours or minutes of uptime in between. This causes quite a bit of instability in the network, and increases packet loss while the network reforms.
We have devices that have weeks of uptime, and other devices running the exact same firmware are rebooting in intervals of 1-2 hours (due to the kernel panic).
Here are a couple log extracts / kernel panics:
rtt:~$ [00:24:57.743,652] <err> net_otPlat_radio: Error while appending to otMessage rtt:~$ [00:25:07.755,462] <err> net_otPlat_radio: Error while appending to otMessage rtt:~$ [00:25:17.757,110] <err> net_otPlat_radio: Error while appending to otMessage rtt:~$ [00:25:17.771,240] <err> net_otPlat_radio: Error while appending to otMessage rtt:~$ [00:25:17.772,796] <err> net_otPlat_radio: Error while appending to otMessage rtt:~$ rtt:~$ [00:25:21.150,085] <err> spinel_ipc_backend_rsp_ntf: No response within timeout 500 rtt:~$ ASSERTION FAIL [0] @ WEST_TOPDIR/zephyr/drivers/ieee802154/ieee802154_nrf5.c:1153 rtt:~$ 802.15.4 serialization error: -5 rtt:~$ [00:25:21.150,177] <err> os: r0/a1: 0x00000004 r1/a2: 0x00000481 r2/a3: 0x00000003 rtt:~$ [00:25:21.150,177] <err> os: r3/a4: 0x20004140 r12/ip: 0x0000000c r14/lr: 0x0003c8e3 rtt:~$ [00:25:21.150,207] <err> os: xpsr: 0x41000000 rtt:~$ [00:25:21.150,207] <err> os: r4/v1: 0x200233ec r5/v2: 0x000b9547 r6/v3: 0x2000d9d0 rtt:~$ [00:25:21.150,238] <err> os: r7/v4: 0x00000000 r8/v5: 0x00000000 r9/v6: 0x62b76bdb rtt:~$ [00:25:21.150,238] <err> os: r10/v7: 0x00000000 r11/v8: 0x00000000 psp: 0x200233b8 rtt:~$ [00:25:21.150,268] <err> os: EXC_RETURN: 0x0 rtt:~$ [00:25:21.150,299] <err> os: Faulting instruction address (r15/pc): 0x000886a2 rtt:~$ [00:25:21.150,329] <err> os: >>> ZEPHYR FATAL ERROR 4: Kernel panic on CPU 0 rtt:~$ [00:25:21.150,360] <err> os: Current thread: 0x20005e38 (openthread) rtt:~$ [00:25:21.207,489] <err> os: Halting system rtt:~$ rtt:~$ [00:00:53.192,596] <err> spinel_ipc_backend_rsp_ntf: No response within timeout 500 rtt:~$ ASSERTION FAIL [0] @ WEST_TOPDIR/zephyr/drivers/ieee802154/ieee802154_nrf5.c:1153 rtt:~$ 802.15.4 serialization error: -5 rtt:~$ [00:00:53.192,687] <err> os: r0/a1: 0x00000004 r1/a2: 0x00000481 r2/a3: 0x00000003 rtt:~$ [00:00:53.192,687] <err> os: r3/a4: 0x20004140 r12/ip: 0x0000000c r14/lr: 0x0003c8e3 rtt:~$ [00:00:53.192,718] <err> os: xpsr: 0x41000000 rtt:~$ [00:00:53.192,718] <err> os: r4/v1: 0x20024454 r5/v2: 0x000b9547 r6/v3: 0x200157ec rtt:~$ [00:00:53.192,749] <err> os: r7/v4: 0x20006790 r8/v5: 0x00000000 r9/v6: 0x0001c379 rtt:~$ [00:00:53.192,749] <err> os: r10/v7: 0x00000000 r11/v8: 0x00000000 psp: 0x20024420 rtt:~$ [00:00:53.192,779] <err> os: EXC_RETURN: 0x0 rtt:~$ [00:00:53.192,810] <err> os: Faulting instruction address (r15/pc): 0x000886a2 rtt:~$ [00:00:53.192,840] <err> os: >>> ZEPHYR FATAL ERROR 4: Kernel panic on CPU 0 rtt:~$ [00:00:53.192,871] <err> os: Current thread: 0x20006698 (ot_radio_workq) rtt:~$ [00:00:53.248,443] <err> os: Halting system
I found pretty much nothing regarding this issue anywhere, and since this is coming from the OpenThread stack, I'm also not quite sure where to get this issue addressed. Any idea?
Thanks