ncs-esb-ble-mpsl-demo hard faults on nRF5340

[jbreal]

I have compiled and run Torbjørn Øvrebekk's ESB MPSL sample (github.com/.../ncs-esb-ble-mpsl-demo) for the nRF5340, which works great, but the cpunet on the PTX side hard faults randomly after some hours. I made the requisite change to 'esb_peripherals.h' as indicated by the warning message printed on the console. No other changes were made. I have tried nRF Connect SDK 2.6.0 and 2.6.1.

[03:00:30.626,678] <inf> rpc_net: Send TX packet, data 0 0x31, len 8
[03:00:30.627,166] <inf> rpc_net: ESB TX success
[03:00:30.627,319] <dbg> rpc_net: rpc_esb_event_send: evt send ok
[00:00:00.000,488] <dbg> rpc_net: serialization_init: nRF RPC init begin
[00:00:00.101,074] <err> nrf_rpc_ipc: IPC endpoint bond timeout
[00:00:00.101,104] <err> NRF_RPC: Failed to initialize transport, err: -32
[00:00:01.101,226] <err> NRF_RPC: Not all groups are ready to use.
*** Booting nRF Connect SDK v3.5.99-ncs1 ***
[00:00:01.101,348] <inf> main: ESB BLE Multiprotocol Example - Netcore project
[00:00:01.101,409] <inf> bt_sdc_hci_driver: SoftDevice Controller build revision:
36 f0 e5 0e 87 68 48 fb 02 fd 9f 82 cc 32 e5 7b |6....hH. .....2.{
91 b1 5c ed |..\.
[00:00:01.101,806] <inf> bt_hci_raw: Bluetooth enabled in RAW mode
[00:00:01.102,050] <wrn> main: Change ESB_EVT_IRQ and ESB_EVT_IRQHandler in esb_peripherals.h to use SWI3 instead of SWI0!

ncs-esb-ble-mpsl-demo hard faults on nRF5340

[ovrebekk]

Hi John

Are you just running the default example, without making any changes to the appcore or netcore projects? 

Are you running the code on a standard nRF5340DK? 

Do you not get any more logging output from the netcore when the crash occurs? 

Best regards
Torbjørn

[jbreal]

[00:41:01.386,718] <dbg> rpc_net: rpc_esb_event_send: evt send ok
[00:41:02.883,148] <inf> rpc_net: Send TX packet, data 0 0x69, len 8
[00:41:02.883,636] <inf> rpc_net: ESB TX success
[00:41:02.883,789] <dbg> rpc_net: rpc_esb_event_send: evt send ok
[00:41:04.383,819] <inf> rpc_net: Send TX packet, data 0 0x6a, len 8
[00:41:04.384,307] <inf> rpc_net: ESB TX success
[00:41:04.384,460] <dbg> rpc_net: rpc_esb_event_send: evt send ok
[00:41:05.884,490] <inf> rpc_net: Send TX packet, data 0 0x6b, len 8
ASSERTION FAIL @ WEST_TOPDIR/external/zephyr/include/zephyr/spinlock.h:136
[00:41:05.884,674] <err> os: ***** HARD FAULT *****
[00:41:05.884,704] <err> os: Fault escalation (see below)
[00:41:05.884,704] <err> os: ARCH_EXCEPT with reason 4

[00:41:05.884,735] <err> os: r0/a1: 0x00000004 r1/a2: 0x00000088 r2/a3: 0x00000002
[00:41:05.884,765] <err> os: r3/a4: 0x21002358 r12/ip: 0x00000010 r14/lr: 0x0102c7ad
[00:41:05.884,765] <err> os: xpsr: 0x4100001c
[00:41:05.884,765] <err> os: Faulting instruction address (r15/pc): 0x0103038c
[00:41:05.884,826] <err> os: >>> ZEPHYR FATAL ERROR 4: Kernel panic on CPU 0
[00:41:05.884,857] <err> os: Fault during interrupt handling

[00:41:05.884,887] <err> os: Current thread: 0x21003fc0 (unknown)
ÿ[00:00:00.000,488] <dbg> rpc_net: serialization_init: nRF RPC init begin
[00:00:00.101,074] <err> nrf_rpc_ipc: IPC endpoint bond timeout
[00:00:00.101,104] <err> NRF_RPC: Failed to initialize transport, err: -32
[00:00:01.101,226] <err> NRF_RPC: Not all groups are ready to use.
*** Booting nRF Connect SDK v3.5.99-ncs1-1 ***
[00:00:01.101,348] <inf> main: ESB BLE Multiprotocol Example - Netcore project
[00:00:01.101,409] <inf> bt_sdc_hci_driver: SoftDevice Controller build revision:
36 f0 e5 0e 87 68 48 fb 02 fd 9f 82 cc 32 e5 7b |6....hH. .....2.{
91 b1 5c ed |..\.
[00:00:01.101,806] <inf> bt_hci_raw: Bluetooth enabled in RAW mode
[00:00:01.102,050] <wrn> main: Change ESB_EVT_IRQ and ESB_EVT_IRQHandler in esb_peripherals.h to use SWI3 instead of SWI0!

[ovrebekk]

Hi John

What about the hardware? Are you using a normal devkit, or some other board? 

Best regards
Torbjørn

[jbreal]

both sides are just a stock nRF5340DK, with a small whip antenna attached

[ovrebekk]

Hi John

Thanks for the info. I will set aside some time tomorrow morning to set up a test, and try to reproduce the error during the day. If you have made any more discoveries on your side that could be of interest please let me know. 

Best regards
Torbjørn

[ovrebekk]

Hi John

I set up a test and quickly ran into the same assert. The assert is caused by a check enabled when SPIN_VALIDATE is set. I will run another test with this test disabled to see what happens. 

If you want to run a similar test simply add the following config to the netcore project:

CONFIG_SPIN_VALIDATE=n

Best regards
Torbjørn

[ovrebekk]

Hi John

Quick update: Running with SPIN_VALIDATE disabled I am not seeing any issues or instabilities. Did you run some testing on your end? 

It would be good to know why the validation fails in the first place, but for the time being I would recommend simply disabling this feature. 

Best regards
Torbjørn