bt_enable fails when net core has Multiprotocol RPMSG or IPC Radio configured for Bluetooth and IEEE802.15.4

[timl2415]

Hi, I recently upgraded our ncs SDK version from 2.6.0 to 2.9.0 for a custom matter application development using an nRF5340. We also support a smaller debug app that is BLE only. I noticed that on 2.6.0 we could flash the multiprotocol rpmsg child image to the net core for both the BLE builds that use bt_enable to start the stack, and the matter application. In 2.9.0 it seems like if a multiprotocol net core image is loaded, either ipc radio or multiprotocol rpmsg, bt_enable always fails with: 

<err> bt_hci_core: HCI driver open failed (-11)

Digging through the source code it looks like we time out waiting on a response from the net core. Is there a reason bt_enable wouldn't be expected to work if the net core is configured for multiprotocol operation?

[Hung Bui]

Hi Timothy, 
There is a fundamental change between NCS v2.6.0 and NCS v2.9.0 is the use of sysbuild. Could you double check you used that in your build ? 
Do you have the same issue with any Matter sample ? I assume they also use IPC radio on the netcore ? 

[timl2415]

I am currently porting some applications for custom boards from 2.6.0 to 2.9.0 so I did integrate sysbuild and hardware model v2. The application I'm having problems with isn't a matter application, its just a bluetooth application with ble/802154 ipc_radio running on the netcore. Perhaps I'll start by programming a ble shell sample on the nrf5340dk app core and multiprotocol ipc radio on the nrf5340dk net core. Is that configuration expected to work?

[Hung Bui]

Hi Timothy, 
Could you be more specific on the BLE Shell sample ? 

Any BLE sample in the SDK will use ipc_radio for the netcore. 
Please try testing with a simple BLE sample, peripheral_lbs for example ? 

If you still see the error, try the same firmware on a DK.
I assume you are testing on a custom board ? 
Please pay extra attention to the device tree of the custom board. Make sure the netcore get flashed. 

[timl2415]

I was able to reproduce the issue pretty easily in the shell_bt_nus sample on the nrf5340dk. First I built and flashed the app core and the net core then pin reset. I was able to connect sucessfully to the bt nus service and send some shell commands. I then added the following to the Kconfig.sysbuild: 

config NETCORE_IPC_RADIO_IEEE802154
    default y

After doing a clean build I reflashed the net core and app core with

nrfjprog -f nrf53 --program build/merged_CPUNET.hex --sectorerase

nrfjprog -f nrf53 --program build/merged.hex --sectorerase

and hit the reset button. The UART logs indicate a failure to bind to the net core endpoint:

*** Booting nRF Connect SDK v2.9.0-7787b2649840 ***
*** Using Zephyr OS v3.7.99-1f8f3dc29142 ***
Starting Bluetooth NUS shell transport example

uart:~$ E: Endpoint binding failed with -11
E: HCI driver open failed (-11)
E: BLE enable failed (err: -11)

So there seems to be some issue with the ipc_radio where if both ieee802154 and ble protocols are enabled, the app core won't be able to bind with just the ble service alone.

[Hung Bui]

Hi Timothy, 
I have reproduced the issue here. As far as I can see the matter sample, for example light_blub has the same set up where NRF_DEFAULT_IPC_RADIO, NETCORE_IPC_RADIO_IEEE802154, NETCORE_IPC_RADIO_BT_HCI_IPC is enabled in Kconfig.sysbuild. 

I can see that there are some configuration needed in the \sysbuild\ipc_radio\ folder. I tried to copy that to shell_bt_nus but still the issue still remain. The netcore was crashed and this explain why you have error -11: 

[00:00:01.001,159] <err> spinel_ipc_backend: IPC endpoint bind timed out
                   
ASSERTION FAIL [0] @ WEST_TOPDIR/nrf/applications/ipc_radio/src/802154.c:14
                           
	       802.15.4 serialization error
                                                                  
[00:00:01.001,281] <err> os: r0/a1:  0x00000004  r1/a2:  0x0000000e  r2/a3:  0x00000003
    
[00:00:01.001,281] <err> os: r3/a4:  0x00000004 r12/ip:  0x00000014 r14/lr:  0x01013247
    
[00:00:01.001,312] <err> os:  xpsr:  0x09000000
                                            
[00:00:01.001,312] <err> os: Faulting instruction address (r15/pc): 0x010283ee
             
[00:00:01.001,373] <err> os: >>> ZEPHYR FATAL ERROR 4: Kernel panic on CPU 0
               
[00:00:01.001,403] <err> os: Current thread: 0x210034b8 (unknown)
                          
[00:00:01.064,086] <err> os: Halting system
                                                
                                                          

I'm checking internally and will get back to you. 

[timl2415]

Any update on this? On my end it seems like there is a crash happening in the ipc backend when there is more than one IPC endpoint on the same ipc instance.

[Hung Bui]

Hi Timothy, 
I have reproduced the issue and the issue is now forwarded to R&D. I haven't heard from them yet. 
The thing is that Matter sample use the same thing but still works fine, so there must be something wrong when we try to add IEEE802154 into the shell_bt_nus sample.