MPSL not working correctly with custom ble hci_rpmsg image

[duralabs]

We have been trying to get MPSL up and running in a BLE and ESB coexistence project but haven't had much luck.

We use an app core image that enables the BLE interface and listens for RPC events from the net core. The net core is a modified version of the BLE hci_rpmsg image with support added for RPC and ESB. We use RPC to notify the app core when data is received over ESB from other Nordic devices. BLE only needs to run for a short period of time and that period of time is user initiated. When implementing the MPSL piece, the BLE hci image is able to bind to "zephyr_bt_hci_rpmsg_ipc" but the "net_buf_get" while loop that delivers data to the app core is blocked from sending anything. We haven't been able to figure out how to allow "net_buf_get" to send BLE data to the app core once MPSL is implemented. 

When we init MPSL, we schedule a request to open a session and create a time slot. This process completes without any issues and we are notified that the time slot has begun. At this time, we'll enable ESB and send some data to a nearby nrf5340dk and that process is also successful. After this happens, we would assume that we would notify MPSL that the process has completed and that the bluetooth communication channel should be allowed to continue. 

We have reviewed the following MPSL sample projects but continue to run into errors with the bluetooth hci communication piece.

1. https://github.com/too1/ncs-esb-ble-mpsl-demo/tree/master
2. https://github.com/nrfconnect/sdk-nrf/tree/main/samples/mpsl/timeslot

Oddly enough, the bluetooth portion worked without any issues until we enabled the following kconfig values for MPSL.

CONFIG_MPSL=y
CONFIG_MPSL_TIMESLOT_SESSION_COUNT=2
CONFIG_RING_BUFFER=y
CONFIG_DYNAMIC_INTERRUPTS=y
CONFIG_DYNAMIC_DIRECT_INTERRUPTS=y

Are there any examples where MPSL is used for BLE and ESB coexistence on a board with an app core and a net core? Unfortunately the first link above looks like it runs on an older infrastructure that doesn't apply to our use-case. Any help would be appreciated as we've spent a lot of time trying to get MPSL working correctly.

[ovrebekk]

Hi Mike

The only sample showing how to run ESB and BLE in timeslot on the nRF53 is this one.

I have an ambition to update the ncs-esb-ble-mpsl-demo sample to run on the nRF53 as well, but I don't know how much time I will need to get this implemented. When I start work on this I will use the other sample for reference, no need to reinvent the wheel. 

Best regards
Torbjørn 

[duralabs]

Thanks for the response.

We tried building the project that you suggested but there are quite a few build errors. For the app core, the build complains about unknown type name 'NRF_RADIO_Type'. For the net core, the build complains about undefined reference to `mpsl_work_q'. 

For the NRF_RADIO_Type error, I see that nrf5340_network.h contains that definition but I'm unsure why the application core is trying to reference it.

Do you have any recommendations for getting the project to build correctly?

[ovrebekk]

Sounds good. 

duralabs said:

How's everything coming along?

Slowly so far, the week ended up busier than I anticipated... But I will dive more into it today, and expect to have more time available next week. I leave for vacation in December, so the hope is to have this wrapped up by the end of next week. 

Best regards
Torbjørn

[duralabs]

Not a problem. Hoping you can get it up and running before your vacation. Let me know if there's anything I can to do help.

[ovrebekk]

Hi 

I was able to get communication between the cores working by adding the hci_rpmsg sample code into my project like shown in the jori-nordic example, but the issue I have now is that as soon as I try to connect to the Bluetooth device the code just hangs. It seems the calls to app_esb_resume() and app_esb_suspend() in the on_timeslot_start_stop(..) handler is interfering with the Bluetooth controller when it is trying to establish a connection. 

I will debug further tomorrow and let you know if I find the cause of the issue. 

Best regards
Torbjørn

[duralabs]

Thanks for the update. We appreciate the time you are investing with us.

[ovrebekk]

Hi Mike

I finally had a bit of a breakthrough today. For a reason I don't yet understand it seems the default interrupt configuration of the ESB library was not playing well with the nRF5340 MPSL configuration. 

I realized in order to get ESB to work properly I had to change the definition of the ESB_EVT_IRQ and ESB_EVT_IRQHandler here to use SWI3 instead of SWI0 (I expect SWI1 and SWI2 might also work). 

Could you try to pull the latest update from the repo and make this change to the esb_peripherals.h file, and see if you can also get it to work? 

Best regards
Torbjørn

[duralabs]

That's great to hear! I'll take a look today and report back with our findings. Thanks for all the work you've put into this.

[duralabs]

We got the sample up and running and had some pretty good luck initially. We've started to run into an issue though when the MPSL callback triggers MPSL_TIMESLOT_SIGNAL_RADIO. Calling RADIO_IRQHandler in the callback appears to be producing a bus fault. Do you have any ideas around why this might be the case? Attached is the trace. 

[00:00:33.305,023] <err> os: ***** BUS FAULT *****
[00:00:33.305,023] <err> os:   Precise data bus error
[00:00:33.305,084] <err> os:   BFAR Address: 0x40
[00:00:33.305,084] <err> os: r0/a1:  0x00000040  r1/a2:  0x21005136  r2/a3:  0x00000042
[00:00:33.305,114] <err> os: r3/a4:  0x00000040 r12/ip:  0x0101f08d r14/lr:  0x01024b57
[00:00:33.305,145] <err> os:  xpsr:  0x2100002d
[00:00:33.305,145] <err> os: Faulting instruction address (r15/pc): 0x010344a0
[00:00:33.305,206] <err> os: >>> ZEPHYR FATAL ERROR 25: Unknown error on CPU 0
[00:00:33.305,206] <err> os: Fault during interrupt handling

 

[Hung Bui]

Hi Mike, 
Torbjørn is away for 2 weeks. I will try help to in the mean time. 

My suggestion is to try testing with a simple BLE sample on the app core and Torbjørn's sample on the netcore before testing with your application. 

Could you try follow his suggestion to clone this branch: https://github.com/too1/ncs-esb-ble-mpsl-demo/tree/feature_5340_support_test1

Then edit these 2 lines to use SWI3 instead of SWI0: https://github.com/nrfconnect/sdk-nrf/blob/main/subsys/esb/esb_peripherals.h#L20-L22

Then flash Torbjørn sample's to the netcore.

Next on the app core, use a BLE sample, for example the peripheral_lbs that I attached here. 
The main change in the sample is this config: 

CONFIG_NCS_INCLUDE_RPMSG_CHILD_IMAGE=n

LED and button has been removed as well. 

6318.peripheral_lbs_app.7z

Regarding your error, you can use addr2line.exe to find which exact line of the file that cause the error at the instruction address 0x010344a0. 
Most likely it's from the netcore. So you will need to input the zephyr.elf from Torbjørn sample in build\zephr\ folder into addr2line.exe to find the file and the line. 

[duralabs]

Thanks for getting back with me while Torbjørn is on vacation. 

We are using the recommended cloned branch with the SWI3 change to esb_peripherals.h. I'll try the sample with the peripheral_lbs app core image and see if that helps.  

For the addr2line.exe recommendation, do you have a solution that we can try that is Mac compatible? We run exclusively on Macs and don't have access to a Windows machine. 

[Hung Bui]

Hi Mike, 
I believe on MAC you can use atos for the same purpose: https://stackoverflow.com/questions/54320054/how-to-use-atos-or-addr2line-on-mac-os-x

But you can also send me the .elf file that I can try to use addr2line here. You may need to convert the ticket to a private one if you don't want to publicly share the .elf file.