nrfjprog hangs in multiply parallel environment

[RobMeades]

I use nrfjprog in a CI system (on Windows 10) which has multiple NRF52 boards attached to it.  The boards are all run in parallel with the -s parameter to nrfjprog and the -USB parameter to jlink (which is used to obtain RTT logging from the boards) indicating which board is being addressed by each command-line operation.  I also apply mutex locks so that nrfjprog is never called at the same time to program two different boards.

About 1 in 10 times my call to nrfjprog just hangs, does nothing.  When I terminate it the return value is 15, "The arguments passed do not perform a valid operation", but the parameters are always the same:

nrfjprog -f nrf52 --program nrf52840_xxaa.hex --chiperase --verify -r -s <serialnumber>

Retrying the command doesn't help, nrfjprog just continues to hang.  I believe that on the occasions this occurs jlink has just completed a logging session and the board is being programmed for the next thing.  Like I say, it doesn't happen all the time, roughly 1 in 10 times, which is enough to cause frequent CI failures.

Any suggestions as to what I might do to work around this?

Version numbers are:

nrfjprog version: 10.6.0
JLinkARM.dll version: 6.80a

[Amanda Hsieh]

Hi Rob, 

Could you try the latest version of nRF-Command-Line-Tools v10.10? Please let me know it can help or not. Thanks. 

-Amanda H.

[RobMeades]

Thanks: after updating to v10.10 I've not yet seen a recurrence of the problem (20 runs so far).  I will continue pounding it, but initial results are good.

[RobMeades]

And here's the same log file again, after a few more runs, with another fail case tacked on the end of it:

8272.log.log

[Amanda Hsieh]

Hi Rob, 

Thanks for the logs. 

Looks like it stops at the same point both places (in read_device_version, at the first read_access_port_register)

Not sure why it's halting while printing out the date and need more time to investigate the issue. Sorry for the inconvenience. 

-Amanda H.

[RobMeades]

FYI, it may not be your problem: we've tried various things including bypassing nrfjprog entirely and calling J-Link directly to do the download: the same things happens, J-Link just freezes some of the time.  Hence I've raised this query with Segger:

forum.segger.com/.../

[Amanda Hsieh]

Thanks for sharing. Please also kindly keep updating the status. We want to know if anything we can help.

-Amanda H.

[RobMeades]

Unfortunately Segger say they are unable to help and recommend I spend EURO800 on another of their tools instead on the off-chance that might work, which I'm afraid I'm not very motivated to do.

"It seems like one of the processes is not terminating correctly.  This might leave the probe and the software in an undefined state.  It is user responsibility that a running session is closed properly. Such undefined states are not supported by us."

forum.segger.com/.../7610-J-Link-freeze-in-multiply-parallel-environment

[RobMeades]

Update: I think one of the Segger engineers got a bit annoyed at me for not wanting to buy their SDK but he did then give me the information I was after. In case anyone else has this problem:

If you are using J-Link for RTT logging (reading from its RTT-socket), driven from an [e.g. Python] script, there is only ONE way to reliably shut J-Link down again and that is to keep a stdin pipe open to it and send it the string "exit\n" before terminating it. This is the official advice from SEGGER (see the thread linked above for his full answer) and experiment shows it to be true.

If you send J-Link SIGTERM/CTRL-C/CTRL-BREAK signals, or if you just terminate it, some percentage of the time (with us it was anything from 10% to 40% of the time) J-Link will hang the next time you try to start it on that board and only power-cycling the board will fix the problem; we tried everything, including programmatically resetting the PC's USB port, and nothing helped; the Segger engineer thought that the J-Link DLL had probably sent a partial command to the debug chip and the debug chip was stuck waiting for more. When this happens, if you take a J-Link log (command-line parameter -Log <Path>) the log will end with "Out of sync, resynchronizing..." and then nothing more.

Curiously this seems to particularly affect the debug chip on NRF52840-DK boards: we also use a Segger Base debug box in the same test system and, although that also gets out of sync, rather than halting J-Link it prints an error message, exits and a retry (without power-cycling the debugger or the board) worked, which never ever happened when using J-Link with the debug chip on the NRF52840-DK board. It was just dead.

Anyway, having implemented this send-exit-over-stdin-and-then-terminate scheme we can now use J-Link for RTT logging and then for something else afterwards reliably.