bt_enable fails when net core has Multiprotocol RPMSG or IPC Radio configured for Bluetooth and IEEE802.15.4

[timl2415]

Hi, I recently upgraded our ncs SDK version from 2.6.0 to 2.9.0 for a custom matter application development using an nRF5340. We also support a smaller debug app that is BLE only. I noticed that on 2.6.0 we could flash the multiprotocol rpmsg child image to the net core for both the BLE builds that use bt_enable to start the stack, and the matter application. In 2.9.0 it seems like if a multiprotocol net core image is loaded, either ipc radio or multiprotocol rpmsg, bt_enable always fails with: 

<err> bt_hci_core: HCI driver open failed (-11)

Digging through the source code it looks like we time out waiting on a response from the net core. Is there a reason bt_enable wouldn't be expected to work if the net core is configured for multiprotocol operation?

[Hung Bui]

Hi Timothy, 
There is a fundamental change between NCS v2.6.0 and NCS v2.9.0 is the use of sysbuild. Could you double check you used that in your build ? 
Do you have the same issue with any Matter sample ? I assume they also use IPC radio on the netcore ? 

[timl2415]

I am currently porting some applications for custom boards from 2.6.0 to 2.9.0 so I did integrate sysbuild and hardware model v2. The application I'm having problems with isn't a matter application, its just a bluetooth application with ble/802154 ipc_radio running on the netcore. Perhaps I'll start by programming a ble shell sample on the nrf5340dk app core and multiprotocol ipc radio on the nrf5340dk net core. Is that configuration expected to work?

[Hung Bui]

Hi Timothy, 
Could you be more specific on the BLE Shell sample ? 

Any BLE sample in the SDK will use ipc_radio for the netcore. 
Please try testing with a simple BLE sample, peripheral_lbs for example ? 

If you still see the error, try the same firmware on a DK.
I assume you are testing on a custom board ? 
Please pay extra attention to the device tree of the custom board. Make sure the netcore get flashed. 

[timl2415]

I was able to reproduce the issue pretty easily in the shell_bt_nus sample on the nrf5340dk. First I built and flashed the app core and the net core then pin reset. I was able to connect sucessfully to the bt nus service and send some shell commands. I then added the following to the Kconfig.sysbuild: 

config NETCORE_IPC_RADIO_IEEE802154
    default y

After doing a clean build I reflashed the net core and app core with

nrfjprog -f nrf53 --program build/merged_CPUNET.hex --sectorerase

nrfjprog -f nrf53 --program build/merged.hex --sectorerase

and hit the reset button. The UART logs indicate a failure to bind to the net core endpoint:

*** Booting nRF Connect SDK v2.9.0-7787b2649840 ***
*** Using Zephyr OS v3.7.99-1f8f3dc29142 ***
Starting Bluetooth NUS shell transport example

uart:~$ E: Endpoint binding failed with -11
E: HCI driver open failed (-11)
E: BLE enable failed (err: -11)

So there seems to be some issue with the ipc_radio where if both ieee802154 and ble protocols are enabled, the app core won't be able to bind with just the ble service alone.

[Hung Bui]

Hi Timothy, 
I have reproduced the issue here. As far as I can see the matter sample, for example light_blub has the same set up where NRF_DEFAULT_IPC_RADIO, NETCORE_IPC_RADIO_IEEE802154, NETCORE_IPC_RADIO_BT_HCI_IPC is enabled in Kconfig.sysbuild. 

I can see that there are some configuration needed in the \sysbuild\ipc_radio\ folder. I tried to copy that to shell_bt_nus but still the issue still remain. The netcore was crashed and this explain why you have error -11: 

[00:00:01.001,159] <err> spinel_ipc_backend: IPC endpoint bind timed out
                   
ASSERTION FAIL [0] @ WEST_TOPDIR/nrf/applications/ipc_radio/src/802154.c:14
                           
	       802.15.4 serialization error
                                                                  
[00:00:01.001,281] <err> os: r0/a1:  0x00000004  r1/a2:  0x0000000e  r2/a3:  0x00000003
    
[00:00:01.001,281] <err> os: r3/a4:  0x00000004 r12/ip:  0x00000014 r14/lr:  0x01013247
    
[00:00:01.001,312] <err> os:  xpsr:  0x09000000
                                            
[00:00:01.001,312] <err> os: Faulting instruction address (r15/pc): 0x010283ee
             
[00:00:01.001,373] <err> os: >>> ZEPHYR FATAL ERROR 4: Kernel panic on CPU 0
               
[00:00:01.001,403] <err> os: Current thread: 0x210034b8 (unknown)
                          
[00:00:01.064,086] <err> os: Halting system
                                                
                                                          

I'm checking internally and will get back to you.