ncs-esb-ble-mpsl-demo hard faults on nRF5340

[jbreal]

I have compiled and run Torbjørn Øvrebekk's ESB MPSL sample (github.com/.../ncs-esb-ble-mpsl-demo) for the nRF5340, which works great, but the cpunet on the PTX side hard faults randomly after some hours. I made the requisite change to 'esb_peripherals.h' as indicated by the warning message printed on the console. No other changes were made. I have tried nRF Connect SDK 2.6.0 and 2.6.1.

[03:00:30.626,678] <inf> rpc_net: Send TX packet, data 0 0x31, len 8
[03:00:30.627,166] <inf> rpc_net: ESB TX success
[03:00:30.627,319] <dbg> rpc_net: rpc_esb_event_send: evt send ok
[00:00:00.000,488] <dbg> rpc_net: serialization_init: nRF RPC init begin
[00:00:00.101,074] <err> nrf_rpc_ipc: IPC endpoint bond timeout
[00:00:00.101,104] <err> NRF_RPC: Failed to initialize transport, err: -32
[00:00:01.101,226] <err> NRF_RPC: Not all groups are ready to use.
*** Booting nRF Connect SDK v3.5.99-ncs1 ***
[00:00:01.101,348] <inf> main: ESB BLE Multiprotocol Example - Netcore project
[00:00:01.101,409] <inf> bt_sdc_hci_driver: SoftDevice Controller build revision:
36 f0 e5 0e 87 68 48 fb 02 fd 9f 82 cc 32 e5 7b |6....hH. .....2.{
91 b1 5c ed |..\.
[00:00:01.101,806] <inf> bt_hci_raw: Bluetooth enabled in RAW mode
[00:00:01.102,050] <wrn> main: Change ESB_EVT_IRQ and ESB_EVT_IRQHandler in esb_peripherals.h to use SWI3 instead of SWI0!

ncs-esb-ble-mpsl-demo hard faults on nRF5340

[ovrebekk]

Hi John

Are you just running the default example, without making any changes to the appcore or netcore projects? 

Are you running the code on a standard nRF5340DK? 

Do you not get any more logging output from the netcore when the crash occurs? 

Best regards
Torbjørn

[jbreal]

Default example, no changes except as noted regarding SWI3. I tried both a standard build and with debug optimizations, in an attempt to set a breakpoint, but I haven't been able to figure out how to make that happen yet. I will run it again this evening to get more output for you. Each time I've run it before, I failed to enable logging in the serial terminal, and then accidentally resized the terminal and lost my output before taking a screenshot.

[jbreal]

[00:41:01.386,718] <dbg> rpc_net: rpc_esb_event_send: evt send ok
[00:41:02.883,148] <inf> rpc_net: Send TX packet, data 0 0x69, len 8
[00:41:02.883,636] <inf> rpc_net: ESB TX success
[00:41:02.883,789] <dbg> rpc_net: rpc_esb_event_send: evt send ok
[00:41:04.383,819] <inf> rpc_net: Send TX packet, data 0 0x6a, len 8
[00:41:04.384,307] <inf> rpc_net: ESB TX success
[00:41:04.384,460] <dbg> rpc_net: rpc_esb_event_send: evt send ok
[00:41:05.884,490] <inf> rpc_net: Send TX packet, data 0 0x6b, len 8
ASSERTION FAIL @ WEST_TOPDIR/external/zephyr/include/zephyr/spinlock.h:136
[00:41:05.884,674] <err> os: ***** HARD FAULT *****
[00:41:05.884,704] <err> os: Fault escalation (see below)
[00:41:05.884,704] <err> os: ARCH_EXCEPT with reason 4

[00:41:05.884,735] <err> os: r0/a1: 0x00000004 r1/a2: 0x00000088 r2/a3: 0x00000002
[00:41:05.884,765] <err> os: r3/a4: 0x21002358 r12/ip: 0x00000010 r14/lr: 0x0102c7ad
[00:41:05.884,765] <err> os: xpsr: 0x4100001c
[00:41:05.884,765] <err> os: Faulting instruction address (r15/pc): 0x0103038c
[00:41:05.884,826] <err> os: >>> ZEPHYR FATAL ERROR 4: Kernel panic on CPU 0
[00:41:05.884,857] <err> os: Fault during interrupt handling

[00:41:05.884,887] <err> os: Current thread: 0x21003fc0 (unknown)
ÿ[00:00:00.000,488] <dbg> rpc_net: serialization_init: nRF RPC init begin
[00:00:00.101,074] <err> nrf_rpc_ipc: IPC endpoint bond timeout
[00:00:00.101,104] <err> NRF_RPC: Failed to initialize transport, err: -32
[00:00:01.101,226] <err> NRF_RPC: Not all groups are ready to use.
*** Booting nRF Connect SDK v3.5.99-ncs1-1 ***
[00:00:01.101,348] <inf> main: ESB BLE Multiprotocol Example - Netcore project
[00:00:01.101,409] <inf> bt_sdc_hci_driver: SoftDevice Controller build revision:
36 f0 e5 0e 87 68 48 fb 02 fd 9f 82 cc 32 e5 7b |6....hH. .....2.{
91 b1 5c ed |..\.
[00:00:01.101,806] <inf> bt_hci_raw: Bluetooth enabled in RAW mode
[00:00:01.102,050] <wrn> main: Change ESB_EVT_IRQ and ESB_EVT_IRQHandler in esb_peripherals.h to use SWI3 instead of SWI0!

[ovrebekk]

Hi John

What about the hardware? Are you using a normal devkit, or some other board? 

Best regards
Torbjørn

[jbreal]

both sides are just a stock nRF5340DK, with a small whip antenna attached

[ovrebekk]

Hi John

Thanks for the info. I will set aside some time tomorrow morning to set up a test, and try to reproduce the error during the day. If you have made any more discoveries on your side that could be of interest please let me know. 

Best regards
Torbjørn