Issues with OpenThread and Bluetooth multiprotocol support locking up

RE: Issues with OpenThread and Bluetooth multiprotocol support locking up

ajlennon — Tue, 29 Aug 2023 09:39:22 GMT

And you couldn't get it to fail? We're connecting to an OTBR implementation running with an nRF52840 dongle with the RCP firmware. When Shreya Patel tried I think it took up to 15 minutes for her to generate a failure. We weren't sure it if was related to network activity...

https://github.com/DynamicDevices/openthread-border-router-block

github.com/.../main

RE: Issues with OpenThread and Bluetooth multiprotocol support locking up

Susheel Nuguru — Fri, 25 Aug 2023 10:38:29 GMT

I have tested this on my desk with nRF52833 and nRF52840DK.

Can you give me the details of the network you are connecting to?

RE: Issues with OpenThread and Bluetooth multiprotocol support locking up

Susheel Nuguru — Fri, 25 Aug 2023 07:57:10 GMT

Trying you code now Alex, sorry for delays.

RE: Issues with OpenThread and Bluetooth multiprotocol support locking up

ajlennon — Thu, 24 Aug 2023 13:24:18 GMT

We're a bit stuck now Susheel Nuguru Any thoughts or advice ?

RE: Issues with OpenThread and Bluetooth multiprotocol support locking up

Shreya Patel — Wed, 23 Aug 2023 17:26:52 GMT

I tried to flash the MTD code on nRF52833 DK and it is able to lock up the CLI after some time (for my OpenThread Network it takes 15 minutes).
Next, in debug mode, after it locks up (by doing ping 8.8.8.8), I get the below logs every time I try to continue debugging.

RE: Issues with OpenThread and Bluetooth multiprotocol support locking up

ajlennon — Tue, 22 Aug 2023 17:28:49 GMT

A bit more information... the MTD code that fails pinging on the nRF52840 dongle works OK on the nRF52833 DK (!?!)

RE: Issues with OpenThread and Bluetooth multiprotocol support locking up

ajlennon — Tue, 22 Aug 2023 17:25:01 GMT

I can make this lock up every time by running `ot ping 8.8.8.8` (or similar)

RE: Issues with OpenThread and Bluetooth multiprotocol support locking up

ajlennon — Tue, 22 Aug 2023 16:41:15 GMT

OK I have simplified everything down and there is a test case here - see README for details

github.com/.../mps_test

RE: Issues with OpenThread and Bluetooth multiprotocol support locking up

Susheel Nuguru — Mon, 21 Aug 2023 14:11:49 GMT

[quote user="Alex J Lennon"]Good idea on the basic test build - I will put something together - thanks![/quote]

Yes, that would be a good way to narrow down the problem.

RE: Issues with OpenThread and Bluetooth multiprotocol support locking up

ajlennon — Mon, 21 Aug 2023 14:10:38 GMT

I've tried various things to affect the build and stack size e.g.

```

# Enable OpenThread features set

#CONFIG_OPENTHREAD_MTD=y

CONFIG_OPENTHREAD_FTD=y

#CONFIG_OPENTHREAD_SLAAC=y

# Minimise size

#CONFIG_NET_SOCKETS=n

#CONFIG_OPENTHREAD_SHELL=n

#CONFIG_COAP=n

#CONFIG_COAP_UTILS=n

#CONFIG_OPENTHREAD_COAP=n

#CONFIG_NRF_802154_SER_LOG=y

#CONFIG_ASSERT=y

#CONFIG_ISR_STACK_SIZE=4096

#CONFIG_IDLE_STACK_SIZE=1024

#CONFIG_BT_RX_STACK_SIZE=4096

#CONFIG_SHELL_STACK_SIZE=8192

#CONFIG_NET_RX_STACK_SIZE=4096

#CONFIG_NET_TX_STACK_SIZE=4096

#CONFIG_MPSL_WORK_STACK_SIZE=4096

#CONFIG_PRIVILEGED_STACK_SIZE=2048

#CONFIG_SYSTEM_WORKQUEUE_STACK_SIZE=8192

#CONFIG_OPENTHREAD_RADIO_WORKQUEUE_STACK_SIZE=4096

#CONFIG_IPC_SERVICE_BACKEND_RPMSG_WQ_STACK_SIZE=2048

```

Good idea on the basic test build - I will put something together - thanks!

RE: Issues with OpenThread and Bluetooth multiprotocol support locking up

Susheel Nuguru — Mon, 21 Aug 2023 14:08:41 GMT

One of our OpenThread expert thinks that this could be memory related and also maybe due to the processing power difference (nRF53 has both more processing power and memory).

As a first step you could try to increase the stack sizes and see if any of the memory optimizations tips mentioned here works. If that does not help, please share a minimalistic project for us to reproduce this.

RE: Issues with OpenThread and Bluetooth multiprotocol support locking up

ajlennon — Mon, 21 Aug 2023 14:05:14 GMT

Thanks - please let me know if there's anything I can turn on in the dongle build that might help in terms of debug info. Happy to make any changes...

RE: Issues with OpenThread and Bluetooth multiprotocol support locking up

Susheel Nuguru — Mon, 21 Aug 2023 13:59:34 GMT

Without the debug info, there is little information for me to debug, but I will ask the experts to help me in this. Will be back to you when I find some guidance.

RE: Issues with OpenThread and Bluetooth multiprotocol support locking up

ajlennon — Fri, 18 Aug 2023 16:14:21 GMT

What you do mean by "locking up"?

Hopefully the above shows this. CLI stops responding
Do you have a call stack trace in "locking up" state?

Unfortunately not.
Can you start a app in debugger and reproduce the issue and get the context of the code that is being executed in this locked up state?

Unfortunately not on the dongle

RE: Issues with OpenThread and Bluetooth multiprotocol support locking up

ajlennon — Fri, 18 Aug 2023 16:12:13 GMT

Thanks for responding Susheel. So to try to show you what's happening a little more easily I've created two videos.

The first shows a forked version of the CLI working in multiprotocol mode on the nRF5340

https://youtu.be/OYG-1Hd4U_c

The second is the same CLI code built for the nRF52840 dongle. You'll see it locks up fairly immediately. About 1:30-2 minutes in you can see it gets further then locks up.

https://youtu.be/eps8GdLtuc8

Hopefully this gives an indication of what I am seeing.

Yes I was thinking the same in terms of the dual core versus single core maybe leading to some kinds of overflow.

I've tried a few changes to increase stack sizes but to no avail.

Any sign-posting on what I could attempt would be much appreciated!

RE: Issues with OpenThread and Bluetooth multiprotocol support locking up

Susheel Nuguru — Fri, 18 Aug 2023 09:59:49 GMT

[quote user=""]

Can anybody help me understand what I am doing wrong here.

I assume from the documentation that Bluetooth + OpenThread is supposed to be supported with multiprotocol on nrf52840?

[/quote]

It seems like a stack overflow issue, but to be able to confirm this and debug I would have to know more details from you.

What you do mean by "locking up"?
Do you have a call stack trace in "locking up" state?
Can you start a app in debugger and reproduce the issue and get the context of the code that is being executed in this locked up state?

The CPU on nRF52 is slower than nRF53 (nRF53 also is dual core which makes it even more efficient in pre and post processing of radio packets).
So a single core on nRF52 might not be just enough to process both of your OpenThread and BLE activity and is causing the buffers to overflow (my guess).

RE: Issues with OpenThread and Bluetooth multiprotocol support locking up

ajlennon — Thu, 17 Aug 2023 14:16:37 GMT

[quote userid="8080" url="~/f/nordic-q-a/102952/issues-with-openthread-and-bluetooth-multiprotocol-support-locking-up"]However when I run this code on the nRF52840 DK then it is fine.[/quote]

I meant to say when I run on the nRF5340 DK it is fine!