Communication breakage between nrf52840 chip and host (imx6ull on stress testing)

[Vignesh Ravi]

Upon stress testing of the thread layer communication over a period of time (around 30 mins to 1 hr) the communication (over UART, 115200 baud rate) between the host (imx6ull based SoC) and nrf52840 chip gets disconnected and it does not recover until otbr-agent is restarted.

Continuous commands were sent from controller to the development kit to reproduce the issue.

1. 868b8d791fac9752d154ef0f0614ca15019056a9 - otbr-agent commit id (github.com/.../ot-br-posix)
2. IEEE 802.15.4 hardware platform - nrf52840
3. Build was done using Yocto recipe.
4. Network topology - Star topology (direct communication between controller and development kit)

Expected behaviour is no communication between the nrf52840 chip and the host(imx6ull based SoC).

Should we raise the transmit buffer size and if we can how can it be done?
Or is there any way to increase the tx timeout of a packet to higher value and will it help resolve the issue.

[Vignesh Ravi]

Expected behaviour is communication breakage should not happen between the nrf52840 chip and the host(imx6ull based SoC).
Sorry for the confusion

[Edvin]

Hello,

What is programmed on the nRF52840? Is it the RCP or NCP sample? Did you do any changes to the application before flashing it?

What HW are you running on (on the nRF)? Is it a DK or a custom board?

Did you try to analyze the UART wires? Does the otbr actually send data that the nRF doesn't respond to, or is it dead? (using a logic analyzer, such as the saleae logic analyzer).

Best regards,

Edvin

[Vignesh Ravi]

Hello Edvin,

1. RCP is programmed on the nRF52840 and there is no changes in application before flashing.
2. We are using custom board (nrf integrated with the imx6ull SoC).
3. Can I hook a JTAG debugger instead of using a logic analyser to check the data transfer, will it be same?

Thanks,
Vignesh R

[Edvin]

This means that you are using custom board files when building, right? Or are you building for the nrf52840dk_nrf52840 board?

What protocol are you using between your otbr and the nRF? Is it UART or USB? If it is UART, you will not be able to hook onto the UART. I you are using USB, there is not much use of hooking on.

Another thing you can try, regardless, is to attach a debugger and see if you can see any logs from the nRF. Does it say anything when/before it stops working? And is it consistent? Does it always happen after 1h? Or only occasionally? Did you try the same with a DK instead of the imx6ull? I am not saying it wouldn't work, but if there is a difference in behavior, it may give some pointers to what the issue may be.

Best regards,

Edvin

And what HW is your otbr running on? A computer? Raspberry pi?

[Vignesh Ravi]

Dear Edvin,
Please find our response for your queries;

>> This means that you are using custom board files when building, right? Or are you building for the nrf52840dk_nrf52840 board?
We are building the application that is part of "/ncs/v2.3.0/nrf/samples/openthread/coprocessor". In the "Board" drop down text box of "Build Configuration" we select nrf52840dk_nrf52840.

>>What protocol are you using between your otbr and the nRF? Is it UART or USB? If it is UART, you will not be able to hook onto the UART. I you are using USB, there is not much use of hooking on.
Between the OTBR and the nRF, we are using UART. We have modified the nrf52840dk_nrf52840.overlay as per our UART pin configuration between the host & rcp. If required we can share the modified file.

>> Another thing you can try, regardless, is to attach a debugger and see if you can see any logs from the nRF. Does it say anything when/before it stops working?
It would be more helpful if you can share us any document to retrieve logs from the nRF through a debugger.

>> And is it consistent? Does it always happen after 1h? Or only occasionally?
No, it is not consistent it occurs occasionally. Sometimes after few mins and sometimes after 1hr or 2hrs.

>> And what HW is your otbr running on? A computer? Raspberry pi?
Our otbr runs on the NXP i.MX 6ULL. The nrf52840dk is connected to "NXP i.MX 6ULL" through UART.

[Edvin]

Vignesh Ravi said:

We are building the application that is part of "/ncs/v2.3.0/nrf/samples/openthread/coprocessor". In the "Board" drop down text box of "Build Configuration" we select nrf52840dk_nrf52840.

Does this mean you are actually using a DK, or are you using a custom board?

Vignesh Ravi said:

It would be more helpful if you can share us any document to retrieve logs from the nRF through a debugger.

If you add these two lines in your prj.conf:

CONFIG_LOG=y
CONFIG_LOG_BACKEND_RTT=y

you should see something like "ieee802154_nrf5: nRF5 802154 radio initialized" when you start the application. To monitor this log, attach a JLink debugger, and open JLink RTT Viewer.

Best regards,

Edvin

[Vignesh Ravi]

Hi Edvin,

As suggested, the above two configurations were enabled in the prj.conf. The host is able to properly communicate with ncp module, however only few debug messages were seen in the RTT terminal as captured in the below excerpts;

SEGGER J-Link V7.86d - Real time terminal output
SEGGER J-Link V9.7, SN=59701277
Process: JLinkExe
[00:00:00.002,349] <inf> ieee802154_nrf5: nRF5 802154 radio initialized
[00:00:00.002,532] <err> qspi_nor: JEDEC id [c2 28 16] expect [c2 28 17] 
*** Booting Zephyr OS build v3.2.99-ncs2 ***
[00:00:00.004,211] <inf> coprocessor_sample:

=========================================================
OpenThread Coprocessor application is now running on NCS
========================================================= 


Are we missing any other other configurations ?

In addition, we would also like to notify that PTA configurations were also enabled as part of the RCP firmware. Is this HandleRcpTimeout() issue is due to enabling PTA ? any inputs would be much helpful.