IPC breaks when enabling bluetooth mesh on nrf54l15

[Fridtjof]

Hello,

We are developing a product based on the nRF54L15, using BT Mesh. We also use the FLPR. For communication between the cores, we use ICMSG.

Separately, ICMSG and BT Mesh work well, but for some reason, when we include BT Mesh, cpuapp cannot set up ICMSG properly, even if the ICMSG setup is done before `bt_enable()`. The callback `ipc_ept_cfg.cb.bound` is never called. We are getting serial logging from the FLPR, so we know that is running and its `ipc_ept_cfg.cb.bound` is called.

This question (The IPC and bt_enable are turned on at the same time, and the chip is not working) about the nRF53 is similar, but there is no IPC Radio Firmware on the nRF54 as far as I can tell.

Here is the relevant part of our prj.conf for cpuapp:

CONFIG_SYSTEM_WORKQUEUE_STACK_SIZE=8192
CONFIG_MAIN_STACK_SIZE=8192
CONFIG_LOG_BUFFER_SIZE=8192
CONFIG_BT_RX_STACK_SIZE=5120

CONFIG_LOG=y
CONFIG_REQUIRES_FLOAT_PRINTF=y

CONFIG_BT_SETTINGS=y
CONFIG_FLASH=y
CONFIG_FLASH_MAP=y
CONFIG_SETTINGS=y
CONFIG_SOC_FLASH_NRF_PARTIAL_ERASE=n
CONFIG_SPI_NOR=n
CONFIG_NVS=n
CONFIG_NVS_LOOKUP_CACHE=n
CONFIG_SETTINGS_NVS_NAME_CACHE=n
CONFIG_ZMS=y
CONFIG_SETTINGS_ZMS_CUSTOM_SECTOR_COUNT=y
CONFIG_SETTINGS_ZMS_SECTOR_COUNT=8
CONFIG_ZMS_LOOKUP_CACHE=y
CONFIG_ZMS_LOOKUP_CACHE_SIZE=512
CONFIG_ZMS_LOOKUP_CACHE_FOR_SETTINGS=y

CONFIG_HWINFO=y
CONFIG_GPIO=y

CONFIG_BT=y
CONFIG_BT_L2CAP_TX_BUF_COUNT=5
CONFIG_BT_PERIPHERAL=y
CONFIG_BT_OBSERVER=y
CONFIG_BT_BROADCASTER=y

CONFIG_BT_CTLR_DUP_FILTER_LEN=0
CONFIG_BT_CTLR_LE_ENC=n
CONFIG_BT_CTLR_LE_PING=n
CONFIG_BT_DATA_LEN_UPDATE=n
CONFIG_BT_PHY_UPDATE=n
CONFIG_BT_CTLR_MIN_USED_CHAN=n
CONFIG_BT_CTLR_PRIVACY=n
CONFIG_BT_CTLR_CHAN_SEL_2=n

CONFIG_BT_MESH=y
CONFIG_BT_MESH_MODEL_EXTENSIONS=y
CONFIG_BT_MESH_RELAY=y
CONFIG_BT_MESH_FRIEND=y
CONFIG_BT_MESH_PB_GATT=y
CONFIG_BT_MESH_PB_ADV=y
CONFIG_BT_MESH_GATT_PROXY=y

CONFIG_BT_MESH_SUBNET_COUNT=2
CONFIG_BT_MESH_APP_KEY_COUNT=2
CONFIG_BT_MESH_MODEL_GROUP_COUNT=2
CONFIG_BT_MESH_LABEL_COUNT=3

CONFIG_BT_MESH_ADV_BUF_COUNT=10
CONFIG_BT_MESH_TX_SEG_MSG_COUNT=3

CONFIG_BT_MESH_RX_SEG_MAX=32
CONFIG_BT_MESH_TX_SEG_MAX=32

CONFIG_MBOX=y
CONFIG_IPC_SERVICE=y
CONFIG_IPC_SERVICE_BACKEND_ICMSG=y
CONFIG_IPC_SERVICE_LOG_LEVEL_DBG=y

Any help is appreciated

- Fridtjof

[Fridtjof]

Hello again, Håkon.

I apologize, but I now believe I messed up the test which made me conclude that the Bluetooth was breaking the IPC. After removing different modules from my project, it seems like it is GNSS which is breaking IPC. IPC works fine with all the other modules, including bluetooth.

IPC breaks even with the minimal GNSS config shown below. Even if I remove all code which uses gnss.

CONFIG_GNSS=y
CONFIG_UART_ASYNC_API=y
CONFIG_GNSS_QUECTEL_LCX6G=y

Do the GNSS and IPC systems interact somehow?

[Håkon Alseth]

Good afternoon Fridtjof!

 

I am unable to get this behavior on my end, and I also cannot see how those configurations should break the ipc communication in general.

Where do you add the gnss dt node? I suspect this is to one of the uart instances?

 

If yes, which one? And does this happen if you disable uart in the remote (ie. flpr) project?

 

Kind regards,

Håkon

[Fridtjof]

Good afternoon,

I am still trying to produce a minimal sample, but it is taking longer than expected because I ran into some other strange issues. I will come back to you when I have more of a clue what I'm dealing with.

Thanks,
-Fridtjof

[Håkon Alseth]

Hi Fridtjof,

 

No worries, feel free to update when you have the time.

  

Kind regards,

Håkon

[Fridtjof]

Hi again Håkon,

I think I found a fix for the issue. In our code, cpuflpr's task is to just wait for commands from cpuapp. This means that unless it gets a command from cpuapp, it doesn't ever send any IPC messages to cpuapp. The only exception to this is the handshake message at IPC setup.

It is my understanding that on IPC setup, cpuflpr places a magic number in its send buffer, triggers an interrupt on cpuapp which then reads the buffer, and if the number is read correctly, cpuapp will call its setup callback.

It seems like GNSS is somehow hindering the handshake. I believe it is by stopping the interrupt which triggers cpuapp to read the buffer. I think this because IPC will work if flpr waits one second after IPC setup, then sends a message to cpuapp, which triggers it to read the IPC buffer, seeing the magic number, and completing setup.

I have attached a sample which shows this behavior, and has three comments with the text "FIX" where I have found ways to get around the problem:

1. Send an extra message from cpuflpr to trigger cpuapp to read the IPC buffer
2. Disable GNSS (we cannot use this since we need GNSS)
3. Enable multithreading on cpuflpr (we cannot use this since we need all the RAM we can get on cpuflpr

gnss_ipc.zip

Could you answer whether my understanding of IPC handshaking and explanation of the problem is correct?

Best regards,
Fridtjof