jLink USB-UART chip on nRF9160dk malfunction

[Georgi Valkov]

We are currently experiencing malfunction with the big chip on nRF9160dk board, which provides USB-UART and jLink functionality.
After 30 minutes, to a few hours, we either stop receiving data from one or both of the USB-UARTs, or the data is garbled due to lost bytes.

Currently nRF9160 is using a modified version of the GPS sample, which sends a copy of the NMEA strings over UART1, to UART0.RX on nRF52840, while UART0.TX connects to the USB-UART on the computer. All samples and boards are attached to the case, except for the app running on nRF52840, which is provided as a binary image. The app is very simple: read from RX and write to TX. The source code we are using is shown on the video, but relays on a proprietary UART driver, which we cannot provide. The driver is very well tested and uses DMA to insure optimal performance and no drops of packets.

Following all tests from the videos linked below, we were able to exclude both CPUs: 9160 and 52840, as well as the software running on them. This leaves the chip providing USB-UART functionality as the only possible point of failure, confirmed after restarting power to the board.

https://youtu.be/04RJwDOnUE8
https://youtu.be/pwJoxr1ODTk
https://youtu.be/Dc6Hxk1oKuQ

8780.2021-03-17.7z

6303.nRF52840 UART-loopback.hex

3806.nRF52840 UART-loopback.7z

[Georgi Valkov]

nRF9160dk j-link/UART chip drops UART data

status: reproduced multiple times on two different nRF9160dk boards: the old one has a GPS antenna connected using a cable, the new board has the GPS antenna mounted on the board.

occurs: a few minutes after we connect the USB cable, the UART starts dropping random bytes or stops delivering data completely. This could affect either the UART connected to nRF9160 or nRF52840, or sometimes both at the same time:

https://youtu.be/6wDAXyt7Bvg

[Simon]

I've been on easter vacation the last week, hence the delay.

I took a brief look at the videos and saw that everything worked when you reset the nrf9160. I think this is because the UART communication between the 9160 and the 52840 needs to be in sync, as mentioned in this ticket. However, after that reply was given, a solution is implemented to fix this, check it out here: zephyr/boards/arm/nrf9160dk_nrf9160/nrf52840_reset.c.

If I have understood this correctly, the function in nrf52840_reset.c will run when the nrf9160 is reset, which will reset the 52840 as well and make it so that the UART communication are in sync.

Does this make sense? Have I understood your problem correctly?

Best regards,

Simon

[Simon]

I've reported your issue internally, I'll keep you updated.

What firmware do you have on the Interface MCU. Download the latest J-Link Software and Documentation Pack, then you can check (and update) the programmer firmware using J-Link Configurator, which is part of J-Link Software and Documentation Pack. 

Best regards,

Simon

[Georgi Valkov]

Hello Simon!

The last time I did that, it was the same firmware version as yours.

[Georgi Valkov]

Hi Simon!

Any progress from your team? I just received a new nRF9160dk board directly from Nordic. Revision 1.0.0 from 2021.6. I assume it was manufactured in the middle of February this year? It has the big GPS antenna on-board. I used the j-link configurator to upgrade the on-board firmware from 2nd of February to 27th of April. All it took to reproduce the issue was programming the GPS sample compiled with SDK v1.5.1. It takes usually around 20-30 minutes to reproduce, sometimes less.

Is it possible to debug or upgrade the board controller? I see version 1.0.0 is available to download, but no description, so I assume all boards come with this by default?

https://www.nordicsemi.com/Software-and-tools/Development-Kits/nRF9160-DK/Download#infotabs

[Simon]

To update the firmware, click on "update firmware of selected..", or follow these steps. However, it seems like you have the latest firmware.

I've pinged internally about this.

[Simon]

Got some updates:

"There are some known issues with Segger UART.
One thing to try is disable mass storage device (https://docs.zephyrproject.org/latest/guides/tools/nordic_segger.html?highlight=msddisable#disabling-the-mass-storage-device-functionality)."

[Simon]

I talked to a colleague internally, who's been working on a bug with the Segger UART. A jlink OB fw that includes a fix for this bug will get released soon. I'll keep you updated. Hopefully this is related to the issue you're exeperiencing

[Simon]

J-Link v7.20a should include fix for issues related to UART HWFC on nRF91-DK, nRF53-DK, and other K22-based DKs. Could you test that out? https://www.segger.com/downloads/jlink/#J-LinkSoftwareAndDocumentationPack 

Download J-Link v7.20a, open the Configurator tool, connect the kit and click on "update firmware", and the fix will be applied?

Update: I see now that flow control is not enabled by default on UART0 on the nrf9160dk_nrf9160ns, which is used for logging (prinkt(), CONFIG_LOG() etc..) maybe enabling that will give a more reliable connection and prevent the issue from happening. Try adding hw-flow-control in uart0, like done here (use an overlay file instead of directly modifying the board DTS file), and also enable flow control in the terminal, e.g. if you're using Termite:

If the issue is happening on the nRF52840 as well, you could try to enable it in UART0 on that chip as well, which is used for logging.

The rts/cts pins should be set correctly by default, such that it will be connected to the Interface MCU: https://infocenter.nordicsemi.com/index.jsp?topic=%2Fug_nrf91_dk%2FUG%2Fnrf91_DK%2Fmcu_virtual_com_port.html&cp=2_0_5_3_0_1. So you don't need to modify them in dts/overlay

My apologies for not thinking about this initially, as it should be the first go-to.

[Georgi Valkov]

Not fixed in 7.20a.

By default KiTTY was using Flow control: XON/XOFF, I also tried RTS/CTS, and OFF. Either way the nRF9160 and nRF52840 terminals start dropping bytes or stops receiving any data.

Meanwhile the BLE client still receives data even though the USB UART does not: nRF9160 UART1 → nRF52840 UART0 → BLE, and USB UART. Resetting nRF9160 and nRF52840 makes no difference, hence I would look for issues with either of them. We and Segger should focus on fixing the board controller.

How do I use overlay files for dts? I tried editing nrf9160dk_nrf9160_common.dts directly. We'll see if it makes any difference EDIT: with hardware flow control enabled for UART0, the USB UART stopped receiving any data after some time. Restarting the CPU makes no difference. I can tell it takes effect, because if I enable it for UART1, I get errors that UART1 is not able to send data. Probably because my link between nRF9160 and nRF52840 is not configured properly to use flow control, and my nRF52840 UART driver does not use it at the moment. All CPUs should be fast enough to handle a couple of UARTs without any drops.

I also got a Blue screen related to the Segger drivers. This usually happens if I use the computer for several days and put it to hibernate or Sleep multiple times. The Segger j-link driver could be leaking data  or corrupting memory when connecting and disconnecting the j-link hardware or UARTS. At some point it causes a blue screen.

[Simon]

Hello Georgi,

I will look into this on Friday or next Tuesday (There are national holidays on Thursday and Monday in Norway).

Best regards,

Simon

[Simon]

You can make sure that flow control is enabled in DTS by checking the file <application>/<build folder>/zephyr/zephyr.dts

Would you be able to upload the new hex files (connectivity_bridge_dk.uart + gps_dk.hex) where flow control is enabled and I will forward them internally.

Best regards,

Simon