nrf5340DK debugger restarts, port disconnected

[Niklas Parnesten]

nrfjprog restarts nrf5340DK debugger

We are running a container environment in Ubuntu 20.04 on WSL2 and the USB ports are frwarded using usbipd

When trying to flash (or just reset) the nrf5340DK after connecting and disconnecting a terminal emulator program
the debugger chip seems to be restarting and the port is lost to the container.

This means we have to connect the ports again and restart the container, making it impossible to run automatic tests for instance

Steps to recreate:
1. Build the zephyr/samples/hello_world sample for nrf5340dk_nrf5340_cpuapp
2. flash the board with nrfjprog
3. Open a terminal emulator program (putty and picocom tested)
4. Reset the board to se the printout
5. Exit the terminal emulator program
6. Try to flash: the following error occurs:

ERROR: Unable to connect to a debugger.
ERROR: [SeggerBackend] - JLinkARM.dll Open returned error 'Cannot connect to J-Link.'
ERROR: [SeggerBackend] - JLinkARM.dll Open returned error 'Failed to open DLL'
ERROR: JLinkARM DLL reported an error. Try again. If error condition
ERROR: persists, run the same command again with argument --log, contact Nordic
ERROR: Semiconductor and provide the generated log.log file to them.

The we need to connect the port again in WSL and the restart the container to get it working again.

The question is, why do we end up in this state after using the emulator and is it possible to flash without resetting the debugger?

See attached log files

Environment:
nrfjprog: 10.15.2
jLink: DLL version V7.62a, compiled Feb 23 2022 17:02:35
Firmware: J-Link OB-nRF5340-NordicSemi compiled Dec 3 2021 15:46:49

Hardware: PCA10095 2.0.0

Log from when it fails:

Fullscreen 5824.nrfjprog_nrf5340_NOK.txt Download

[2022-Feb-25 16:04:53] [ info] --------------------------------------------------------------------------------
[2022-Feb-25 16:04:53] [ info] nrfjprog --program build/zephyr/hello_zephyr.hex --recover --verify -f NRF53 --log 
[2022-Feb-25 16:04:53] [ info] nrfjprog version 10.15.2 external
[2022-Feb-25 16:04:53] [ info] --------------------------------------------------------------------------------
[2022-Feb-25 16:04:53] [ info] Load library at /opt/nrf-command-line-tools/lib/libnrfjprogdll.so.
[2022-Feb-25 16:04:53] [ info] Library loaded, loading member functions.
[2022-Feb-25 16:04:53] [ info] Member functions succesfully loaded.
[2022-Feb-25 16:04:53] [debug] [ Client] - open
[2022-Feb-25 16:04:53] [debug] [ Client] - start
[2022-Feb-25 16:04:53] [ info] [ Client] - stdout: Jlinkarm nRF Worker ready. Handling sequence 0fb58d33-9d91-434f-81c7-7d68de21c7a9.
[2022-Feb-25 16:04:53] [trace] [ Client] - Command open executed for 2 milliseconds with result 0
[2022-Feb-25 16:04:53] [debug] [ Client] - enum_emu_snr
[2022-Feb-25 16:04:53] [debug] [SeggerBackend] - Logger sink registered in Segger backend logger
[2022-Feb-25 16:04:53] [debug] [  JLink] - Logger sink registered in JLink logger
[2022-Feb-25 16:04:53] [debug] [  nRF53] - open
[2022-Feb-25 16:04:53] [debug] [  nRF53] - just_check_family
[2022-Feb-25 16:04:53] [debug] [SeggerBackend] - open_dll
[2022-Feb-25 16:04:53] [debug] [SeggerBackend] - No J-Link DLL path was provided. Attempting to auto detect.
[2022-Feb-25 16:04:53] [ info] [SeggerBackend] - Load library at /opt/SEGGER/JLink/libjlinkarm.so.7.
[2022-Feb-25 16:04:53] [ info] [SeggerBackend] - Library loaded, loading member functions.
[2022-Feb-25 16:04:53] [ info] [SeggerBackend] - Member functions succesfully loaded.
[2022-Feb-25 16:04:53] [debug] [SeggerBackend] - dll_version
[2022-Feb-25 16:04:53] [ info] [SeggerBackend] - Segger dll version 7.62.a loaded.
[2022-Feb-25 16:04:53] [trace] [ Worker] - Command open executed for 1 milliseconds with result 0
[2022-Feb-25 16:04:53] [debug] [  nRF53] - enum_emu_con_info
[2022-Feb-25 16:04:53] [debug] [SeggerBackend] - is_connected_to_emu
[2022-Feb-25 16:04:53] [debug] [SeggerBackend] - enum_emu_con_info
[2022-Feb-25 16:04:53] [trace] [ Client] - Command enum_emu_con_info executed for 32 milliseconds with result 0
[2022-Feb-25 16:04:53] [debug] [ Client] - connect_to_emu_with_snr
[2022-Feb-25 16:04:53] [debug] [SeggerBackend] - is_connected_to_emu
[2022-Feb-25 16:04:53] [trace] [ Worker] - Command enum_emu_con_info executed for 32 milliseconds with result 0
[2022-Feb-25 16:04:53] [debug] [  nRF53] - connect_to_emu_with_snr
[2022-Feb-25 16:04:53] [debug] [SeggerBackend] - is_connected_to_emu
[2022-Feb-25 16:04:53] [debug] [SeggerBackend] - connect_to_emu_with_snr
[2022-Feb-25 16:04:53] [debug] [SeggerBackend] - is_connected_to_emu
[2022-Feb-25 16:04:53] [debug] [SeggerBackend] - ---just_enum_emu_snr
[2022-Feb-25 16:04:53] [debug] [SeggerBackend] - ---just_get_num_emus
[2022-Feb-25 16:04:53] [debug] [SeggerBackend] - ---just_connect_to_emu_with_snr
[2022-Feb-25 16:04:53] [debug] [SeggerBackend] - ---just_connect_to_emu_without_snr
[2022-Feb-25 16:04:53] [debug] [SeggerBackend] - Segger logging enabled.
[2022-Feb-25 16:05:01] [trace] [ Client] - Command connect_to_emu_with_snr executed for 8177 milliseconds with result -102
[2022-Feb-25 16:05:01] [trace] [  JLink] -    ***** Error: 
[2022-Feb-25 16:05:01] [trace] [  JLink] - Cannot connect to J-Link.
[2022-Feb-25 16:05:01] [trace] [  JLink] - - 8061.153ms returns "Cannot connect to J-Link."  
[2022-Feb-25 16:05:01] [trace] [  JLink] - JLINK_HasError()  
[2022-Feb-25 16:05:01] [error] [SeggerBackend] - JLinkARM.dll Open returned error 'Cannot connect to J-Link.'
[2022-Feb-25 16:05:01] [trace] [  JLink] - JLINK_OpenEx(...)  
[2022-Feb-25 16:05:01] [trace] [  JLink] - - 0.053ms returns "Failed to open DLL"  
[2022-Feb-25 16:05:01] [trace] [  JLink] - JLINK_HasError()  
[2022-Feb-25 16:05:01] [error] [SeggerBackend] - JLinkARM.dll Open returned error 'Failed to open DLL'
[2022-Feb-25 16:05:01] [debug] [SeggerBackend] - is_connected_to_emu
[2022-Feb-25 16:05:01] [trace] [  JLink] - JLINK_IsOpen()  
[2022-Feb-25 16:05:01] [trace] [  JLink] - - 0.003ms returns 0x00  
[2022-Feb-25 16:05:01] [trace] [  JLink] - JLINK_HasError()  
[2022-Feb-25 16:05:01] [trace] [ Worker] - Command connect_to_emu_with_snr executed for 8177 milliseconds with result -102
[2022-Feb-25 16:05:01] [debug] [  nRF53] - close
[2022-Feb-25 16:05:01] [debug] [SeggerBackend] - is_connected_to_emu
[2022-Feb-25 16:05:01] [trace] [  JLink] - JLINK_IsOpen()  
[2022-Feb-25 16:05:01] [trace] [  JLink] - - 0.006ms returns 0x00  
[2022-Feb-25 16:05:01] [trace] [  JLink] - JLINK_HasError()  
[2022-Feb-25 16:05:01] [debug] [SeggerBackend] - close
[2022-Feb-25 16:05:01] [debug] [SeggerBackend] - disconnect_from_emu
[2022-Feb-25 16:05:01] [debug] [SeggerBackend] - is_connected_to_emu
[2022-Feb-25 16:05:01] [trace] [  JLink] - JLINK_IsOpen()  
[2022-Feb-25 16:05:01] [trace] [  JLink] - - 0.003ms returns 0x00  
[2022-Feb-25 16:05:01] [trace] [  JLink] - JLINK_HasError()  
[2022-Feb-25 16:05:01] [debug] [SeggerBackend] - Segger Backend closed.
[2022-Feb-25 16:05:01] [debug] [  nRF53] - nRF family DLL closed
[2022-Feb-25 16:05:01] [trace] [ Client] - Command close executed for 10 milliseconds with result 0
[2022-Feb-25 16:05:01] [debug] [ Client] - terminate
[2022-Feb-25 16:05:01] [trace] [ Client] - Command terminate executed for 0 milliseconds with result 0
[2022-Feb-25 16:05:01] [trace] [ Worker] - Command close executed for 10 milliseconds with result 0
[2022-Feb-25 16:05:01] [trace] [ Worker] - Command terminate executed for 0 milliseconds with result 0
[2022-Feb-25 16:05:01] [trace] [ Worker] - Executed 5 commands for 8220 milliseconds
[2022-Feb-25 16:05:01] [debug] [ Client] - Child process terminated with result 0
[2022-Feb-25 16:05:01] [debug] [ Client] - Worker process exited with code: 0
[2022-Feb-25 16:05:01] [debug] [ Client] - Worker process exited with code: 0
[2022-Feb-25 16:05:01] [trace] [ Client] - Executed 5 commands for 8221 milliseconds
[2022-Feb-25 16:05:01] [debug] [ Client] - terminate

Log from when it works:

8156.nrfjprog_nrf5340_OK.txt

investigation.zip

[Niklas Parnesten]

Hi Håkon,

Håkon Alseth said:

Does the nRF5340-DK in question behave like this on the host OS?

A few time I have seen the board restarting (all leds stops lighting) but it is not a problem on the host OS, nrfjprog will flash anyway just with a little delay - thats on windows

Håkon Alseth said:

Is the issue similar on other PCs, and with different USB cable(s)? Note: please check with device directly connected to PC (ie. no hubs)

We have seen this on at least two different setups.

My PC only have usb-c ports, I will try to dig up a converter so I can test without my hub.

Håkon Alseth said:

Do you have more than one nRF5340-DK v2.0.0 that also behaves similar?

We have seen this on the four DK's we have tested so far (all nrf5340-DK v2.0.0)

[Niklas Parnesten]

I can confirm now that I get the same behavior without a USB hub also (only using an adapter)

[Håkon Alseth]

I am sorry, but I'm not able to recreate the scenario where the debugger drops out over USB. I have talked to several colleagues, our DK test team, and we unfortunately have not encountered such an issue before.

This seems to be very tightly coupled with the wsl+docker+usbipd setup that you are currently running.

 

One other thing that came to mind is the hardware solution that you're running on. Are you by any chance running AMD in both your tested setups? If you google "usb issues amd" - there has been several issues lately, which should be improved if you update your BIOS + drivers in the host OS.

 

Kind regards,

Håkon

[Niklas Parnesten]

OK,

Unfortunately I am running on Intel HW.

br

/Niklas

[Håkon Alseth]

Hi Niklas,

 

I'll keep an eye out for this behavior in the future, and so will the team that works with the development kits.

One last thing I can think of is the mass storage part of the debugger, which can be problematic in certain scenarios. If you open J-Link commander (JLinkExe on mac/linux) and write "msddisable" - this mass storage (ie. the "JLink" drive) should be disabled on the next power cycle.

 

Kind regards,

Håkon

[Niklas Parnesten]

Hi Håkon,

Thanks for your support!

Unfortunately I get the same behavior when the mass storage are disabled.

One of my colleagues have made a more extensive analysis of the problem we see. I will attach their findings here, maybe it can give you some insight 

B/R

Niklas