Unable to use NanoPB with Sidewalk

[vawkes]

Hello, I am trying to use the subghz template in ncs 2.5.2 but I am running into an issue when enabling NanoPB. When enabling NanoPB with CONFIG_NANOPB=y the application faults when connecting to BLE to register on the network. If I am already registered on the network the application runs fine but it seems there is something in the BLE registration that is causing a fault with NanoPB.

00> [00:00:02.054,931] <inf> sid_ble_conn: BT Connected
00> [00:00:03.348,114] <err> os: ***** BUS FAULT *****
00> [00:00:03.348,114] <err> os:   Precise data bus error
00> [00:00:03.348,144] <err> os:   BFAR Address: 0x110001
00> [00:00:03.348,175] <err> os: r0/a1:  0x20010150  r1/a2:  0x00110001  r2/a3:  0x00000000
00> [00:00:03.348,175] <err> os: r3/a4:  0x00000000 r12/ip:  0x00000003 r14/lr:  0x0003b44b
00> [00:00:03.348,205] <err> os:  xpsr:  0x41000200
00> [00:00:03.348,236] <err> os: s[ 0]:  0x200087d0  s[ 1]:  0x20016bc6  s[ 2]:  0x00000000  s[ 3]:  0x0002fefb
00> [00:00:03.348,236] <err> os: s[ 4]:  0x2001acf4  s[ 5]:  0x0003cacd  s[ 6]:  0x2001ad60  s[ 7]:  0x2001ad60
00> [00:00:03.348,266] <err> os: s[ 8]:  0x20016ec0  s[ 9]:  0x2001acf4  s[10]:  0x2001ad60  s[11]:  0x20010168
00> [00:00:03.348,266] <err> os: s[12]:  0x00000014  s[13]:  0x20016bc6  s[14]:  0x00000000  s[15]:  0x00067ef7
00> [00:00:03.348,327] <err> os: fpscr:  0x00000000
00> [00:00:03.348,327] <err> os: r4/v1:  0x200101c8  r5/v2:  0x2000a838  r6/v3:  0x0008129e[0m
00> [00:00:03.348,358] <err> os: r7/v4:  0x20016af4  r8/v5:  0x00000004  r9/v6:  0x00000000
00> [00:00:03.348,358] <err> os: r10/v7: 0x00000000  r11/v8: 0x00000000    psp:  0x200100b0
00> [00:00:03.348,388] <err> os: EXC_RETURN: 0xffffffed
00> [00:00:03.348,388] <err> os: Faulting instruction address (r15/pc): 0x00069d7a
00> [00:00:03.348,419] <err> os: >>> ZEPHYR FATAL ERROR 25: Unknown error on CPU 0
00> [00:00:03.348,480] <err> os: Current thread: 0x200033a0 (unknown)
00> [00:00:04.049,621] <err> os: Halting system

Call Stack:

[Elfving]

Hello Timothy,

Sorry about the wait. The Easter period is a public holiday in Norway.

I don't think the combination of NanoPB and Sidewalk has been tested. Have you tried debugging to find where the bus fault was raised?

Regards,

Elfving

[vawkes]

No worries. At the moment the call stack and fault info above are all I have. It is pretty easier to reproduce by just adding CONFIG_NANOPB=y if you need more info immediately. I will see if I can get some more debug information later.

[samtday]

Are there changes I should be aware of transitioning from 2.9.0 or am I good to just adjust the build configuration and run with it?

[samtday]

It seems the transition to v3.0.1 would require me to begin from an entirely different project (after downloading the 'Amazon Sidewalk Add-on') to be able to include the sidewalk SDK. Is this what you expect given I am currently working on v2.9.0. Is there a way to point my current project to the appropriate version instead of having to start from scratch in the new SDK?

[samtday]

I was able to migrate to the newest versions of the SDK and build a fresh example of sid_end_device with the only change being the addition of 'CONFIG_NANOPB=y' to prj.conf and it is still not running appropriately.

[Elfving]

samtday said:

It seems the transition to v3.0.1 would require me to begin from an entirely different project (after downloading the 'Amazon Sidewalk Add-on') to be able to include the sidewalk SDK. Is this what you expect given I am currently working on v2.9.0.

Not at all, I am just trying to figure out what is wrong here by giving us the same baseline (given that I allready tested this on NCS 3 etc.). If you already have a large project set-up on NCS 2.9 we can continue with NCS 2.9  Though I would suggest to start off by looking at the default sample instead of the project you have, as there could be other things that you've added that makes it run into this issue. Please try the default sample and let me know how it goes.

samtday said:

I was able to migrate to the newest versions of the SDK and build a fresh example of sid_end_device with the only change being the addition of 'CONFIG_NANOPB=y' to prj.conf and it is still not running appropriately.

What do you mean by it not running appropriately? Do you get a build error?

Regards,

Elfving

[samtday]

I was able to migrate my v2.9.0 based project up to v3.0.0 and build successfully. I also tried running the Amazon Sidewalk Add-on example and enabled the CONFIG_NANOPB. In both cases the build was successful but when I flash the project to my nrf5340 DK, it crashes.

When I say not running appropriately I mean it runs and crashes with a similar error print out to the one I shared before:

*** Booting Sidewalk v0.1.99-addon-12c29cd0964e ***
*** Using nRF Connect SDK v3.0.0-3bfc46578e42 ***
*** Using Zephyr OS v4.0.99-a0e545cb437a ***
----------------------------------------------------------------
Sidewalk SDK        = 1.18.0.18
APP_BUILD_VERSION   = v1.0.0-add-on
APP_NAME            = sidewalk
build time          = May  6 2025 15:30:17
board               = nrf5340dk/nrf5340/cpuapp
----------------------------------------------------------------
[00:00:00.001,647] <inf> application_state: working = true
[00:00:00.072,631] <inf> bt_hci_core: HW Platform: Nordic Semiconductor (0x0002)
[00:00:00.072,662] <inf> bt_hci_core: HW Variant: nRF53x (0x0003)
[00:00:00.072,692] <inf> bt_hci_core: Firmware: Standard Bluetooth controller (0x00) Version 199.32083 Build 4145153724
[00:00:00.075,225] <inf> bt_hci_core: Identity: ED:92:4B:94:92:3C (random)
[00:00:00.075,225] <inf> bt_hci_core: HCI: version 6.0 (0x0e) revision 0x20e8, manufacturer 0x0059
[00:00:00.075,256] <inf> bt_hci_core: LMP: version 6.0 (0x0e) subver 0x20e8
metal: warning:   tx_vq: freeing non-empty virtqueue
metal: warning:   rx_vq: freeing non-empty virtqueue
[00:00:00.081,756] <err> sid_mfg: Flash protect failed -22
[00:00:00.146,514] <err> sidewalk_events: radio init err -6
[00:00:00.170,562] <err> sidewalk_events: radio sleep err -6
[00:00:00.170,593] <inf> sidewalk_events: Sidewalk link switch to BLE
[00:00:00.174,560] <inf> sid_ble: Enable BT
[00:00:00.273,345] <inf> bt_hci_core: HW Platform: Nordic Semiconductor (0x0002)
[00:00:00.273,376] <inf> bt_hci_core: HW Variant: nRF53x (0x0003)
[00:00:00.273,406] <inf> bt_hci_core: Firmware: Standard Bluetooth controller (0x00) Version 199.32083 Build 4145153724
[00:00:00.275,909] <inf> bt_hci_core: Identity: ED:92:4B:94:92:3C (random)
[00:00:00.275,939] <inf> bt_hci_core: HCI: version 6.0 (0x0e) revision 0x20e8, manufacturer 0x0059
[00:00:00.275,970] <inf> bt_hci_core: LMP: version 6.0 (0x0e) subver 0x20e8
[00:00:00.275,970] <inf> sid_ble: BT initialized
[00:00:00.282,257] <inf> app: Status changed: not ready
[00:00:00.282,379] <inf> app: Device Unregistered, Time Sync Fail, Link status: {BLE: Down, FSK: Down, LoRa: Down}
[00:00:00.946,136] <inf> sid_ble_conn: BT Connected
[00:00:01.522,735] <err> os: ***** BUS FAULT *****
[00:00:01.522,766] <err> os:   Precise data bus error
[00:00:01.522,766] <err> os:   BFAR Address: 0x110001
[00:00:01.522,796] <err> os: r0/a1:  0x2000f1b8  r1/a2:  0x00110001  r2/a3:  0x00000000
[00:00:01.522,827] <err> os: r3/a4:  0x00000000 r12/ip:  0x20004e88 r14/lr:  0x000234e3
[00:00:01.522,827] <err> os:  xpsr:  0x41000200
[00:00:01.522,827] <err> os: s[ 0]:  0x2001c858  s[ 1]:  0x2000f230  s[ 2]:  0x2000a098  s[ 3]:  0x0000000b
[00:00:01.522,857] <err> os: s[ 4]:  0x00000008  s[ 5]:  0x000234e3  s[ 6]:  0x2001c8dc  s[ 7]:  0x200077a2
[00:00:01.522,857] <err> os: s[ 8]:  0x00000000  s[ 9]:  0x00000000  s[10]:  0x00000000  s[11]:  0x0004e7a5
[00:00:01.522,888] <err> os: s[12]:  0x00000000  s[13]:  0x00000000  s[14]:  0x000689cb  s[15]:  0x200077d0
[00:00:01.522,888] <err> os: fpscr:  0x00000000
[00:00:01.522,918] <err> os: r4/v1:  0x2000f230  r5/v2:  0x2000a098  r6/v3:  0x000689cb
[00:00:01.522,918] <err> os: r7/v4:  0x00000008  r8/v5:  0x00000000  r9/v6:  0x20007640
[00:00:01.522,949] <err> os: r10/v7: 0x00000000  r11/v8: 0x00000000    psp:  0x2000f160
[00:00:01.522,949] <err> os: EXC_RETURN: 0xfffffffd
[00:00:01.522,949] <err> os: Faulting instruction address (r15/pc): 0x0004fb3c
[00:00:01.522,979] <err> os: >>> ZEPHYR FATAL ERROR 25: Unknown error on CPU 0
[00:00:01.523,010] <err> os: Current thread: 0x20003a78 (unknown)
[00:00:01.662,963] <err> os: Halting system

[Elfving]

Hi again Sam, and thank you for your patience,

You are not seeing this issue with NANOPB not enabled right? If so I would assume that the mfg.hex is fine. Could you try running addr2line on that faulting instruction set?

Here are some steps on how to use addr2line:

1. Check if you have arm-none-eabi-addr2line set up by running "where arm-none-eabi-addr2line" in a cmd window.
   1. This should return something like "C:\Program Files (x86)\GNU Tools ARM Embedded\4.9 2015q3\bin\arm-none-eabi-addr2line.exe"
   2. If it does not return this, then there might be some issues with the gnuarmemb path configuration
2. Next step is to investigate what is in the R15/pc and R14/lr addresses
   1. arm-none-eabi-addr2line -e <path to _build\zephyr\zephyr.elf> <0x000211a8> for r15/pc
   2. arm-none-eabi-addr2line -e <path to _build\zephyr\zephyr.elf> <0x00018b3d> for r14/lr

2. Should give you what function call that causes the kernel panic deep down in the stack. Remember to replace "<path to _build\zephyr\zephyr.elf>" with your actual path

Regards,

Elfving

[samtday]

Elfving, thanks for the guidance.

I ran into issues with the "arm-none-eabi-addr2line" command.

1. There is no zephyr.elf at that location.
2. I did find a few zephyr.elf files at other locations, but when I attempt to run the addr2line command, the output is: "The system cannot find the file specified"

All this to say, I did go look a the zephyr.map file and found where the address should be:

.text.load_descriptor_values
0x0000000000050af4 0xf4 modules/nanopb/libmodules__nanopb.a(pb_common.c.obj)

[samtday]

Since the error seems to be related to stack size and the crash I received is coming from the Sidewalk Thread, I tried increasing the following config variables within prj.conf

CONFIG_SIDEWALK_THREAD_STACK_SIZE=24576 (from 16384 & 8192)

CONFIG_SYSTEM_WORKQUEUE_STACK_SIZE=8192 (from 4096)

CONFIG_SIDEWALK_HEAP_SIZE=10240 (from 5120)

But none of these changes affected the outcome.

[Elfving]

We are seeing this on our side as well now, and the relevant R&D team is looking into it. There might be some sort of conflict since nanobp is already in use by the protocol library for sidewalk.

Could you expand a bit on your use-case for nanopb? 

Regards,

Elfving

[samtday]

Elfving: we are building a device with the goal of collecting various sensor values and publishing this data to AWS via Sidewalk. Due to the bandwidth limitations of sidewalk we wanted to utilize protobuf to balance the throughput of data.

Sam