Usage fault in NCS v2.7.0

RE: Usage fault in NCS v2.7.0

Narendra — Fri, 15 Nov 2024 12:28:44 GMT

Updating NCS to v2.8.0 fixed the above issue. Not generating reported error or fault at the same point.

Thanks,

Narendra

RE: Usage fault in NCS v2.7.0

Narendra — Thu, 14 Nov 2024 08:39:11 GMT

Increased the stack size up to 8192, but didn't help. (CONFIG_BT_HCI_TX_STACK_SIZE=8192).

CONFIG_BT_RX_STACK_SIZE=8192 already set.

Also when setting CONFIG_BT_HCI_TX_PRIO=7 in pjr.conf file, receiving an error in the build process.

Please check below is the application log, which may help to debug the issue. As mentioned above our device is working as an extender, having one peripheral and multiple central roles. Transmitting BLE packets on peripheral and central both side after connection.

[00:00:34.580,718] <inf> SL: Initiating connection
[00:00:34.580,780] <inf> PSH: NUS server send. Length 20
[00:00:34.580,810] <inf> PSH: ServerTx
[00:00:34.581,085] <dbg> BCS: on_sent: Data send, conn 0x20004430
[00:00:34.581,085] <inf> PSH: Server Tx Complete
[00:00:34.679,901] <inf> SL: Central connected: 1
[00:00:34.680,114] <inf> SL: Security changed: level 0
[00:00:34.680,175] <inf> SL: MTU exchange pending
[00:00:34.704,040] <inf> SL: MTU exchange successful
[00:00:34.704,071] <inf> SL: Start scanning: interval 181 ms, window 43 ms
[00:00:34.705,230] <inf> CSH: NUS Client module initialized
[00:00:34.705,261] <inf> CSH: Service discovery completed
[00:00:34.705,413] <inf> CSH: NUS client send: connIndx 1, pkt length 20
[00:00:34.705,688] <inf> CSH: Client Tx Complete 1
[00:00:34.705,780] <inf> PSH: NUS server send. Length 20
[00:00:34.705,810] <inf> PSH: ServerTx
ASSERTION FAIL [err == 0] @ WEST_TOPDIR/zephyr/subsys/bluetooth/host/hci_core.c:333
  command opcode 0x0406 timeout with err -11
[00:00:44.706,146] <err> os: r0/a1:  0x00000004  r1/a2:  0x0000014d  r2/a3:  0x00000002
[00:00:44.706,176] <err> os: r3/a4:  0x00000004 r12/ip:  0x00000010 r14/lr:  0x0003b5fb[0m
[00:00:44.706,176] <err> os:  xpsr:  0x01000000
[00:00:44.706,207] <err> os: Faulting instruction address (r15/pc): 0x000642fe
[00:00:44.706,237] <err> os: >>> ZEPHYR FATAL ERROR 4: Kernel panic on CPU 0
[00:00:44.706,268] <err> os: Current thread: 0x200042d0 (BT TX)
[00:00:44.940,032] <err> os: Halting system

RE: Usage fault in NCS v2.7.0

Susheel Nuguru — Wed, 13 Nov 2024 12:58:44 GMT

Now we have the context, There seems to be stack overflow in the BT transmit thread.

please try below in your prj.conf, try to increase the stack sizes if you already have that size.

CONFIG_BT_RX_STACK_SIZE=2048
CONFIG_BT_HCI_TX_STACK_SIZE=1024

you can experiment with recalibrating the rx and tx thread prio for the host as below

CONFIG_BT_HCI_TX_PRIO=7
CONFIG_BT_RX_PRIO=8

pristine build and try again. I hope this might help

RE: Usage fault in NCS v2.7.0

Narendra — Wed, 13 Nov 2024 10:21:56 GMT

Some additional logs, set

CONFIG_ASSERT=y

ASSERTION FAIL [err == 0] @ WEST_TOPDIR/zephyr/subsys/bluetooth/host/hci_core.c:333
  command opcode 0x0406 timeout with err -11
[00:00:23.780,364] <err> os: r0/a1:  0x00000004  r1/a2:  0x0000014d  r2/a3:  0x00000002
[00:00:23.780,395] <err> os: r3/a4:  0x00000004 r12/ip:  0x00000010 r14/lr:  0x0003d0b7
[00:00:23.780,395] <err> os:  xpsr:  0x01000000
[00:00:23.780,426] <err> os: Faulting instruction address (r15/pc): 0x000694be
[00:00:23.780,456] <err> os: >>> ZEPHYR FATAL ERROR 4: Kernel panic on CPU 0
[00:00:23.780,487] <err> os: Current thread: 0x20004320 (BT TX)
[00:00:24.014,434] <err> os: Halting system

RE: Usage fault in NCS v2.7.0

Narendra — Wed, 13 Nov 2024 09:23:20 GMT

Yes, right. But I was busy with some other task. Just resume this.

And yes, I have added the config you suggested in prj.conf file.

The above logs were generated with the new configuration.

[00:01:18.508,911] <err> os: Current thread: 0x20012d58 (MPSL Work)

is this not a thread name?

RE: Usage fault in NCS v2.7.0

Susheel Nuguru — Wed, 13 Nov 2024 08:20:08 GMT

That bad that it has been almost 2 months and you still have this issue. I still cannot see the thread name where the fault instruction happens.

Are you sure you added the configs I suggested to add in prj.conf. Without thread name it is hard for anyone to get the context of this hardfault and hence no one will be able to give you a proper debugging direction.

RE: Usage fault in NCS v2.7.0

Narendra — Tue, 12 Nov 2024 11:00:38 GMT

Please find below the updated debug log.

*** Booting My Application v1.3.0-beta-41ee9b2d1389 ***
*** Using nRF Connect SDK v2.7.0-5cb85570ca43 ***
*** Using Zephyr OS v3.6.99-100befc70c74 ***
[00:00:00.008,239] <inf> fs_nvs: 2 Sectors of 4096 bytes
[00:00:00.008,239] <inf> fs_nvs: alloc wra: 0, fd0
[00:00:00.008,270] <inf> fs_nvs: data wra: 0, 1c
[00:00:00.008,422] <inf> bt_sdc_hci_driver: SoftDevice Controller build revision: 
                                            d6 da c7 ae 08 db 72 6f  2a a3 26 49 2a 4d a8 b3 |......ro *.&I*M..
                                            98 0e 07 7f                                      |....             
[00:00:00.010,711] <inf> bt_hci_core: HW Platform: Nordic Semiconductor (0x0002)
[00:00:00.010,742] <inf> bt_hci_core: HW Variant: nRF52x (0x0002)
[00:00:00.010,772] <inf> bt_hci_core: Firmware: Standard Bluetooth controller (0x00) Version 214.51162 Build 1926957230
[00:00:00.011,230] <inf> bt_hci_core: No ID address. App mus0m
[0m
[00:00:06.044,769] <wrn> bt_hci_core: opcode 0x200a status 0x0d
[00:00:07.264,739] <wrn> bt_hci_core: opcode 0x200a status 0x0d
[00:01:02.571,838] <wrn> bt_conn: conn 0x20004688 failed to establish. RF noise?
[00:01:02.572,296] <wrn> bt_hci_core: opcode 0x2022 status 0x02
[00:01:18.508,789] <err> os: ***** USAGE FAULT *****
[00:01:18.508,819] <err> os:   Unaligned memory accessm
[00:01:18.508,819] <err> os: r0/a1:  0x2000beb5  r1/a2:  0x0000002c  r2/a3:  0x2002d36c
[00:01:18.508,850] <err> os: r3/a4:  0x00000040 r12/ip:  0x00000000 r14/lr:  0x0006a4a5
[00:01:18.508,850] <err> os:  xpsr:  0x21000000
[00:01:18.508,850] <err> os: Faulting instruction address (r15/pc): 0x0006a48e
[00:01:18.508,880] <err> os: >>> ZEPHYR FATAL ERROR 31: Unknown error on CPU 0
[00:01:18.508,911] <err> os: Current thread: 0x20012d58 (MPSL Work)
[00:01:18.738,616] <err> os: Halting system

RE: Usage fault in NCS v2.7.0

Susheel Nuguru — Tue, 24 Sep 2024 05:03:51 GMT

It would still be nice to know the context of the thread that is triggering this Fault exception. Right now it shows

Current thread: 0x2000bcb8 (unknown)

Can you add these in your prj.conf, prestine build and run your applicaiton to provide some more detailed fault into here? That was, we can know what is the context of the fault, that will have impact on our debugging direction.

CONFIG_ASSERT_VERBOSE=y
CONFIG_ASSERT_NO_COND_INFO=n
CONFIG_ASSERT_NO_MSG_INFO=n
CONFIG_RESET_ON_FATAL_ERROR=n
CONFIG_THREAD_NAME=y
CONFIG_STACK_SENTINEL=y

RE: Usage fault in NCS v2.7.0

Narendra — Thu, 19 Sep 2024 10:04:41 GMT

I think main stack size should not be the issue. I have run the thread analyzer and checked. The main thread occupies only 42% of the stack. Following is the log of the thread analyzer.

[00:03:16.586,669] [00:03:16.587,188] <inf> thread_analyzer:  BT CTLR ECDH [00:03:16.587,829] <inf> thread_analyzer: [00:03:16.589,233] <inf> thread_analyzer:  BT RX WQ [00:03:16.589,874] <inf> thread_analyzer: [00:03:16.590,393] <inf> thread_analyzer:  BT TX [00:03:16.591,033] <inf> thread_analyzer: [00:03:16.591,522] <inf> thread_analyzer:  thread_analyzer [00:03:16.592,163] <inf> thread_analyzer: [00:03:16.592,926] <inf> thread_analyzer:  mcumgr smp [00:03:16.593,566] <inf> thread_analyzer: [00:03:16.599,060] <inf> thread_analyzer:  usbd_workq [00:03:16.599,700] <inf> thread_analyzer: [00:03:16.600,341] <inf> thread_analyzer:  BT LW WQ [00:03:16.605,987] <inf> thread_analyzer: [00:03:16.607,452] <inf> thread_analyzer:  sysworkq [00:03:16.608,062] <inf> thread_analyzer: [00:03:16.608,581] <inf> thread_analyzer:  MPSL Work [00:03:16.609,222] <inf> thread_analyzer: [00:03:16.610,107] <inf> thread_analyzer:  usbworkq [00:03:16.610,748] <inf> thread_analyzer: [00:03:16.616,271] <inf> thread_analyzer:  idle [00:03:16.616,882] <inf> thread_analyzer: [00:03:16.618,103] <inf> thread_analyzer:  main [00:03:16.618,743] <inf> thread_analyzer: [00:03:16.624,450] <inf> thread_analyzer:  ISR0

<inf> thread_analyzer: Thread analyze: : STACK: unused 792 usage 168 / 960 (17 %); CPU: 0 % : Total CPU cycles used: 0 : STACK: unused 6088 usage 2104 / 8192 (25 %); CPU: 0 % : Total CPU cycles used: 4859 : STACK: unused 224 usage 1824 / 2048 (89 %); CPU: 0 % : Total CPU cycles used: 2516 : STACK: unused 136 usage 888 / 1024 (86 %); CPU: 0 % : Total CPU cycles used: 49158 : STACK: unused 1888 usage 160 / 2048 (7 %); CPU: 0 % : Total CPU cycles used: 0 : STACK: unused 200 usage 824 / 1024 (80 %); CPU: 0 % : Total CPU cycles used: 433 : STACK: unused 1184 usage 160 / 1344 (11 %); CPU: 0 % : Total CPU cycles used: 1 : STACK: unused 6312 usage 1880 / 8192 (22 %); CPU: 0 % : Total CPU cycles used: 1590 : STACK: unused 184 usage 840 / 1024 (82 %); CPU: 0 % : Total CPU cycles used: 6916 : STACK: unused 2620 usage 5572 / 8192 (68 %); CPU: 0 % : Total CPU cycles used: 7486 : STACK: unused 272 usage 48 / 320 (15 %); CPU: 98 % : Total CPU cycles used: 6319381 : STACK: unused 4684 usage 3508 / 8192 (42 %); CPU: 0 % : Total CPU cycles used: 44629 : STACK: unused 1120 usage 928 / 2048 (45 %)

I have looked at the migration guide. I have not changed board version (hwv1 to hwv2) and multi-image parent and child to system build. Could it be the issue?

Thanks,

Narendra.

RE: Usage fault in NCS v2.7.0

Simonr — Thu, 19 Sep 2024 09:45:50 GMT

The first thing you could try is to increase the CONFIG_MAIN_STACK_SIZE for example to twice the size, as that has been the issue in cases like this before. If that doesn't work out, make sure you followed the migration guide from NCS 2.6.1 to 2.7.0 here as some changes are required when moving from the NCS 2.6.x base to 2.7.x.

https://docs.nordicsemi.com/bundle/ncs-latest/page/nrf/releases_and_maturity/migration/migration_guide_2.7.html

Best regards,

Simon