jLink USB-UART chip on nRF9160dk malfunction

[Georgi Valkov]

We are currently experiencing malfunction with the big chip on nRF9160dk board, which provides USB-UART and jLink functionality.
After 30 minutes, to a few hours, we either stop receiving data from one or both of the USB-UARTs, or the data is garbled due to lost bytes.

Currently nRF9160 is using a modified version of the GPS sample, which sends a copy of the NMEA strings over UART1, to UART0.RX on nRF52840, while UART0.TX connects to the USB-UART on the computer. All samples and boards are attached to the case, except for the app running on nRF52840, which is provided as a binary image. The app is very simple: read from RX and write to TX. The source code we are using is shown on the video, but relays on a proprietary UART driver, which we cannot provide. The driver is very well tested and uses DMA to insure optimal performance and no drops of packets.

Following all tests from the videos linked below, we were able to exclude both CPUs: 9160 and 52840, as well as the software running on them. This leaves the chip providing USB-UART functionality as the only possible point of failure, confirmed after restarting power to the board.

https://youtu.be/04RJwDOnUE8
https://youtu.be/pwJoxr1ODTk
https://youtu.be/Dc6Hxk1oKuQ

8780.2021-03-17.7z

6303.nRF52840 UART-loopback.hex

3806.nRF52840 UART-loopback.7z

[Georgi Valkov]

nRF9160dk j-link/UART chip drops UART data

status: reproduced multiple times on two different nRF9160dk boards: the old one has a GPS antenna connected using a cable, the new board has the GPS antenna mounted on the board.

occurs: a few minutes after we connect the USB cable, the UART starts dropping random bytes or stops delivering data completely. This could affect either the UART connected to nRF9160 or nRF52840, or sometimes both at the same time:

https://youtu.be/6wDAXyt7Bvg

[Simon]

I've been on easter vacation the last week, hence the delay.

I took a brief look at the videos and saw that everything worked when you reset the nrf9160. I think this is because the UART communication between the 9160 and the 52840 needs to be in sync, as mentioned in this ticket. However, after that reply was given, a solution is implemented to fix this, check it out here: zephyr/boards/arm/nrf9160dk_nrf9160/nrf52840_reset.c.

If I have understood this correctly, the function in nrf52840_reset.c will run when the nrf9160 is reset, which will reset the 52840 as well and make it so that the UART communication are in sync.

Does this make sense? Have I understood your problem correctly?

Best regards,

Simon

[Simon]

Got some updates:

"There are some known issues with Segger UART.
One thing to try is disable mass storage device (https://docs.zephyrproject.org/latest/guides/tools/nordic_segger.html?highlight=msddisable#disabling-the-mass-storage-device-functionality)."

[Simon]

I talked to a colleague internally, who's been working on a bug with the Segger UART. A jlink OB fw that includes a fix for this bug will get released soon. I'll keep you updated. Hopefully this is related to the issue you're exeperiencing

[Simon]

J-Link v7.20a should include fix for issues related to UART HWFC on nRF91-DK, nRF53-DK, and other K22-based DKs. Could you test that out? https://www.segger.com/downloads/jlink/#J-LinkSoftwareAndDocumentationPack 

Download J-Link v7.20a, open the Configurator tool, connect the kit and click on "update firmware", and the fix will be applied?

Update: I see now that flow control is not enabled by default on UART0 on the nrf9160dk_nrf9160ns, which is used for logging (prinkt(), CONFIG_LOG() etc..) maybe enabling that will give a more reliable connection and prevent the issue from happening. Try adding hw-flow-control in uart0, like done here (use an overlay file instead of directly modifying the board DTS file), and also enable flow control in the terminal, e.g. if you're using Termite:

If the issue is happening on the nRF52840 as well, you could try to enable it in UART0 on that chip as well, which is used for logging.

The rts/cts pins should be set correctly by default, such that it will be connected to the Interface MCU: https://infocenter.nordicsemi.com/index.jsp?topic=%2Fug_nrf91_dk%2FUG%2Fnrf91_DK%2Fmcu_virtual_com_port.html&cp=2_0_5_3_0_1. So you don't need to modify them in dts/overlay

My apologies for not thinking about this initially, as it should be the first go-to.

[Georgi Valkov]

Not fixed in 7.20a.

By default KiTTY was using Flow control: XON/XOFF, I also tried RTS/CTS, and OFF. Either way the nRF9160 and nRF52840 terminals start dropping bytes or stops receiving any data.

Meanwhile the BLE client still receives data even though the USB UART does not: nRF9160 UART1 → nRF52840 UART0 → BLE, and USB UART. Resetting nRF9160 and nRF52840 makes no difference, hence I would look for issues with either of them. We and Segger should focus on fixing the board controller.

How do I use overlay files for dts? I tried editing nrf9160dk_nrf9160_common.dts directly. We'll see if it makes any difference EDIT: with hardware flow control enabled for UART0, the USB UART stopped receiving any data after some time. Restarting the CPU makes no difference. I can tell it takes effect, because if I enable it for UART1, I get errors that UART1 is not able to send data. Probably because my link between nRF9160 and nRF52840 is not configured properly to use flow control, and my nRF52840 UART driver does not use it at the moment. All CPUs should be fast enough to handle a couple of UARTs without any drops.

I also got a Blue screen related to the Segger drivers. This usually happens if I use the computer for several days and put it to hibernate or Sleep multiple times. The Segger j-link driver could be leaking data  or corrupting memory when connecting and disconnecting the j-link hardware or UARTS. At some point it causes a blue screen.

[Simon]

Hello Georgi,

I will look into this on Friday or next Tuesday (There are national holidays on Thursday and Monday in Norway).

Best regards,

Simon

[Simon]

You can make sure that flow control is enabled in DTS by checking the file <application>/<build folder>/zephyr/zephyr.dts

Would you be able to upload the new hex files (connectivity_bridge_dk.uart + gps_dk.hex) where flow control is enabled and I will forward them internally.

Best regards,

Simon

[Georgi Valkov]

Hello Simon, and please excuse me for replaying late.

I was currently working to restore some features like power saving when no devices are connected, and UART pins from DTS. I will look into implementing flow control and connecting the proper pins today and I will likely be able to release the apps you requested tomorrow. I also plan to provide you with the complete source code, so you can test it internally.

[Simon]

Sounds good. Hopefully enabling flow control will resolve the issue.

[Georgi Valkov]

Hello Simon!

I spent the last few days carefully examining the datasheets, then implemented and tested flow control. I enabled it for: nRF9160 UART0-1, as well as nRF52840 UART0-1. Unfortunately my expectations were right: it does not help. As a reminder, the first path is reliable, the second is not:

nRF9160  UART1 TX  18 → 20 nRF52840 UART0 RX → BLE
nRF9160  UART1 TX  18 → 20 nRF52840 UART0 RX → nRF52840 UART0 TX 05 → VCOM1 RX

Restarting nRF9160 or nRF52840 does not help. Using a different host does not help, I tried Windows, macOS, OpenWRT. I used three different boards, all are affected. This should isolate the board controller which provides j-link and virtual COM ports as the only possible cause.

I also improved the UART miniport driver integration with Zephyr: power saving, baud-rate configuration and the entire connectivity bridge functionality is now available with my driver. You can switch between UART miniport (prj.conf) or NRFX UART (prj.nrfx.conf) by changing the project file, and everything functions exactly the same, except my driver is reliable while with Nordic's driver connectivity bridge will eventually malfunction, as you know from the other support case. The UART pin configuration, as well as status, current-speed, and hw-flow-control are now obtained from the board .dts configuration. I will send you the source code in a private message.

[Simon]

Thanks, I'll take a look at it tomorrow.