jLink USB-UART chip on nRF9160dk malfunction

[Georgi Valkov]

We are currently experiencing malfunction with the big chip on nRF9160dk board, which provides USB-UART and jLink functionality.
After 30 minutes, to a few hours, we either stop receiving data from one or both of the USB-UARTs, or the data is garbled due to lost bytes.

Currently nRF9160 is using a modified version of the GPS sample, which sends a copy of the NMEA strings over UART1, to UART0.RX on nRF52840, while UART0.TX connects to the USB-UART on the computer. All samples and boards are attached to the case, except for the app running on nRF52840, which is provided as a binary image. The app is very simple: read from RX and write to TX. The source code we are using is shown on the video, but relays on a proprietary UART driver, which we cannot provide. The driver is very well tested and uses DMA to insure optimal performance and no drops of packets.

Following all tests from the videos linked below, we were able to exclude both CPUs: 9160 and 52840, as well as the software running on them. This leaves the chip providing USB-UART functionality as the only possible point of failure, confirmed after restarting power to the board.

https://youtu.be/04RJwDOnUE8
https://youtu.be/pwJoxr1ODTk
https://youtu.be/Dc6Hxk1oKuQ

8780.2021-03-17.7z

6303.nRF52840 UART-loopback.hex

3806.nRF52840 UART-loopback.7z

[Georgi Valkov]

nRF9160dk j-link/UART chip drops UART data

status: reproduced multiple times on two different nRF9160dk boards: the old one has a GPS antenna connected using a cable, the new board has the GPS antenna mounted on the board.

occurs: a few minutes after we connect the USB cable, the UART starts dropping random bytes or stops delivering data completely. This could affect either the UART connected to nRF9160 or nRF52840, or sometimes both at the same time:

https://youtu.be/6wDAXyt7Bvg

[Simon]

I've been on easter vacation the last week, hence the delay.

I took a brief look at the videos and saw that everything worked when you reset the nrf9160. I think this is because the UART communication between the 9160 and the 52840 needs to be in sync, as mentioned in this ticket. However, after that reply was given, a solution is implemented to fix this, check it out here: zephyr/boards/arm/nrf9160dk_nrf9160/nrf52840_reset.c.

If I have understood this correctly, the function in nrf52840_reset.c will run when the nrf9160 is reset, which will reset the 52840 as well and make it so that the UART communication are in sync.

Does this make sense? Have I understood your problem correctly?

Best regards,

Simon

[Georgi Valkov]

Hello Simon!

Simon said:

Does this make sense? Have I understood your problem correctly?

No you did not. According to my observations:

1. USB UART0 is connected directly to nRF9160 UART0.
2. USB UART1 is connected to nRF52840, I can make it print a counter in a loop, e.g. "*** UART test *** xxx xxx\r\n"; or I can read NMEA strings from nRF9160 UART1 and send them to USB UART1.
3. When the issue happens, either one of both of the USB UARTs could be affected.
4. Meanwhile NMEA data is also sent to a BLE client and is valid, which means that the communication between nRF9160 and nRF52840 is connect, hence no need to perform a reset.
5. I used the reset button and PROG/DEBUG switch to reset both CPUs. Despite of this, the data I see on the USB UARTs still contains many dropped bytes.
6. Eventually the USB UARTs stop sending any data, and the only think that helps is powering off the board.

Please use the attached boards and samples to test. You should be able to reproduce the issue within a few minutes. I already reproduced it on two boards. Use the following samples:

• nRF9160: nrf/samples/nrf9160/gps_dk.hex (source available)
• nRF52840: nrf/applications/connectivity_bridge_dk.uart-miniport.hex (preferred, custom UART, loopback always active) or
• nRF52840: nrf/applications/connectivity_bridge_dk.hex (source available, loopback activates when a BLE client is connected)

2021-04-07.7z

[Georgi Valkov]

Reproduced with nRF9160dk connected to the USB port of my OpenWRT router. This should rule out the two Windows 10 computers we used.

[Simon]

I was able to reproduce it now with the provided hex files.

I will start debugging/investigating it now, using the complete samples

[Georgi Valkov]

Well done and good luck!

[Simon]

I've reported your issue internally, I'll keep you updated.

What firmware do you have on the Interface MCU. Download the latest J-Link Software and Documentation Pack, then you can check (and update) the programmer firmware using J-Link Configurator, which is part of J-Link Software and Documentation Pack. 

Best regards,

Simon

[Georgi Valkov]

Hello Simon!

The last time I did that, it was the same firmware version as yours.

[Georgi Valkov]

Hi Simon!

Any progress from your team? I just received a new nRF9160dk board directly from Nordic. Revision 1.0.0 from 2021.6. I assume it was manufactured in the middle of February this year? It has the big GPS antenna on-board. I used the j-link configurator to upgrade the on-board firmware from 2nd of February to 27th of April. All it took to reproduce the issue was programming the GPS sample compiled with SDK v1.5.1. It takes usually around 20-30 minutes to reproduce, sometimes less.

Is it possible to debug or upgrade the board controller? I see version 1.0.0 is available to download, but no description, so I assume all boards come with this by default?

https://www.nordicsemi.com/Software-and-tools/Development-Kits/nRF9160-DK/Download#infotabs

[Simon]

To update the firmware, click on "update firmware of selected..", or follow these steps. However, it seems like you have the latest firmware.

I've pinged internally about this.

[Simon]

Got some updates:

"There are some known issues with Segger UART.
One thing to try is disable mass storage device (https://docs.zephyrproject.org/latest/guides/tools/nordic_segger.html?highlight=msddisable#disabling-the-mass-storage-device-functionality)."

[Simon]

I talked to a colleague internally, who's been working on a bug with the Segger UART. A jlink OB fw that includes a fix for this bug will get released soon. I'll keep you updated. Hopefully this is related to the issue you're exeperiencing