Thingy91: pynrfjprog deadlocks at 100% cpu usage

[Triffid_Hunter]

I'm using the script from https://devzone.nordicsemi.com/f/nordic-q-a/53208/updating-nrf9160-modem-firmware-through-the-command-line/215357#215357

Under strace, it never forks any other process and doesn't create the file it's looking for, so I'm really not sure what it's expecting to happen.

Full strace at http://triffid-hunter.no-ip.info/pynrf-flash.strace since this forum thing keeps throwing errors when I try to attach it.

https://devzone.nordicsemi.com/f/nordic-q-a/62397/nrfjprog-bug-gets-stuck-due-to-temporary-directory-permissions seems related however my permissions are fine, and the directory tree in /tmp is recreated identically if I wipe it first.

Since this is apparently the only way to update the NRF91 modem firmware from a terminal, and all the examples seem to be complaining about out-of-date modem firmware, I'm once again stuck.

[Triffid_Hunter]

Relevant section of strace, in case it helps:

stat("/home/triffid/Projects/NRF/NRFCommandLineTools-10.9.0/JLink_Linux_V680a_x86_64/", {st_mode=S_IFDIR|0755, st_size=1550, ...}) = 0
openat(AT_FDCWD, "/home/triffid/Projects/NRF/NRFCommandLineTools-10.9.0/JLink_Linux_V680a_x86_64/", O_RDONLY|O_NONBLOCK|O_CLOEXEC|O_DIRECTORY) = 3
fstat(3, {st_mode=S_IFDIR|0755, st_size=1550, ...}) = 0
getdents64(3, /* 57 entries */, 32768)  = 2112
getdents64(3, /* 0 entries */, 32768)   = 0
close(3)                                = 0
stat("/", {st_mode=S_IFDIR|0755, st_size=178, ...}) = 0
openat(AT_FDCWD, "/tmp/nrfjprogdll/highlevel", O_RDONLY|O_NONBLOCK|O_CLOEXEC|O_DIRECTORY) = 3
fstat(3, {st_mode=S_IFDIR|0755, st_size=180, ...}) = 0
getdents64(3, /* 9 entries */, 32768)   = 328
getdents64(3, /* 0 entries */, 32768)   = 0
close(3)                                = 0
lstat("/tmp/nrfjprogdll/highlevel", {st_mode=S_IFDIR|0755, st_size=180, ...}) = 0
stat("/tmp", {st_mode=S_IFDIR|S_ISVTX|0777, st_size=420, ...}) = 0
getpid()                                = 18780
lstat("/tmp/file12YHdh", 0x7ffd8bf8a1d0) = -1 ENOENT (No such file or directory)
stat("/tmp/nrfjprogdll/highlevel/file12YHdh", 0x7ffd8bf8a230) = -1 ENOENT (No such file or directory)
stat("/tmp/nrfjprogdll/highlevel/file12YHdh.lock", 0x7ffd8bf8a120) = -1 ENOENT (No such file or directory)
stat("/tmp/nrfjprogdll/highlevel/file12YHdh.lock", 0x7ffd8bf8a070) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/tmp/nrfjprogdll/highlevel/file12YHdh.lock", O_WRONLY|O_CREAT|O_TRUNC, 0666) = 3
stat("/tmp/nrfjprogdll/highlevel/file12YHdh", 0x7ffd8bf89e90) = -1 ENOENT (No such file or directory)
stat("/", {st_mode=S_IFDIR|0755, st_size=178, ...}) = 0
stat("/tmp/nrfjprogdll/highlevel/file12YHdh", 0x7ffd8bf8a230) = -1 ENOENT (No such file or directory)
stat("/tmp/nrfjprogdll/highlevel/file12YHdh", 0x7ffd8bf8a230) = -1 ENOENT (No such file or directory)
stat("/tmp/nrfjprogdll/highlevel/file12YHdh", 0x7ffd8bf8a230) = -1 ENOENT (No such file or directory)
stat("/tmp/nrfjprogdll/highlevel/file12YHdh", 0x7ffd8bf8a230) = -1 ENOENT (No such file or directory)
stat("/tmp/nrfjprogdll/highlevel/file12YHdh", 0x7ffd8bf8a230) = -1 ENOENT (No such file or directory)
stat("/tmp/nrfjprogdll/highlevel/file12YHdh", 0x7ffd8bf8a230) = -1 ENOENT (No such file or directory)
stat("/tmp/nrfjprogdll/highlevel/file12YHdh", 0x7ffd8bf8a230) = -1 ENOENT (No such file or directory)
stat("/tmp/nrfjprogdll/highlevel/file12YHdh", 0x7ffd8bf8a230) = -1 ENOENT (No such file or directory)
stat("/tmp/nrfjprogdll/highlevel/file12YHdh", 0x7ffd8bf8a230) = -1 ENOENT (No such file or directory)
stat("/tmp/nrfjprogdll/highlevel/file12YHdh", 0x7ffd8bf8a230) = -1 ENOENT (No such file or directory)
stat("/tmp/nrfjprogdll/highlevel/file12YHdh", 0x7ffd8bf8a230) = -1 ENOENT (No such file or directory)
stat("/tmp/nrfjprogdll/highlevel/file12YHdh", 0x7ffd8bf8a230) = -1 ENOENT (No such file or directory)
stat("/tmp/nrfjprogdll/highlevel/file12YHdh", 0x7ffd8bf8a230) = -1 ENOENT (No such file or directory)
stat("/tmp/nrfjprogdll/highlevel/file12YHdh", 0x7ffd8bf8a230) = -1 ENOENT (No such file or directory)
stat("/tmp/nrfjprogdll/highlevel/file12YHdh", 0x7ffd8bf8a230) = -1 ENOENT (No such file or directory)
stat("/tmp/nrfjprogdll/highlevel/file12YHdh", 0x7ffd8bf8a230) = -1 ENOENT (No such file or directory)
stat("/tmp/nrfjprogdll/highlevel/file12YHdh", 0x7ffd8bf8a230) = -1 ENOENT (No such file or directory)

And repeats indefinitely until killed.

Where is this "/tmp/nrfjprogdll/highlevel/file12YHdh" file supposed to come from, and why does the python library busy loop on checking if it exists?

[Triffid_Hunter]

With logging.basicConfig(level=logging.DEBUG) I get this:

INFO:modem_update:Modem firmware upgrade
DEBUG:pynrfjprog.HighLevel:HighLevel_logger_open
DEBUG:pynrfjprog.HighLevel:HighLevel_logger_open: nRFJProg logger opened to callback at address 0X7F49C2B28010
DEBUG:pynrfjprog.HighLevel:HighLevel_dll_open
DEBUG:pynrfjprog.HighLevel:HighLevel_dll_open: Found J-Link library at "/".
DEBUG:pynrfjprog.HighLevel:HighLevel_dll_open: Copying Segger dll to temporary folder to allow a new probe connection.
DEBUG:pynrfjprog.HighLevel:HighLevel_dll_open: Copy "/"->"/tmp/nrfjprogdll/highlevel/filew8XONI"
*** (deadlocks here until killed)
Terminated

But despite saying it's copying, it never attempts to copy anything:

17292 lstat("/tmp/filew8XONI", 0x7ffc94e44b90) = -1 ENOENT (No such file or directory)
17292 stat("/tmp/nrfjprogdll/highlevel/filew8XONI", 0x7ffc94e44bf0) = -1 ENOENT (No such file or directory)
17292 stat("/tmp/nrfjprogdll/highlevel/filew8XONI.lock", 0x7ffc94e44ae0) = -1 ENOENT (No such file or directory)
17292 stat("/tmp/nrfjprogdll/highlevel/filew8XONI.lock", 0x7ffc94e44a30) = -1 ENOENT (No such file or directory)
17292 openat(AT_FDCWD, "/tmp/nrfjprogdll/highlevel/filew8XONI.lock", O_WRONLY|O_CREAT|O_TRUNC, 0666) = 3
17292 getpid() = 17292
17292 write(2, "DEBUG:pynrfjprog.HighLevel:HighLevel_dll_open:\tCopy \"/\"->\"/tmp/nrfjprogdll/highlevel/filew8XONI\"\n", 97) = 97
17292 stat("/tmp/nrfjprogdll/highlevel/filew8XONI", 0x7ffc94e44850) = -1 ENOENT (No such file or directory)
17292 stat("/", {st_mode=S_IFDIR|0755, st_size=178, ...}) = 0
17292 stat("/tmp/nrfjprogdll/highlevel/filew8XONI", 0x7ffc94e44bf0) = -1 ENOENT (No such file or directory)
17292 stat("/tmp/nrfjprogdll/highlevel/filew8XONI", 0x7ffc94e44bf0) = -1 ENOENT (No such file or directory)
17292 stat("/tmp/nrfjprogdll/highlevel/filew8XONI", 0x7ffc94e44bf0) = -1 ENOENT (No such file or directory)
17292 stat("/tmp/nrfjprogdll/highlevel/filew8XONI", 0x7ffc94e44bf0) = -1 ENOENT (No such file or directory)
17292 stat("/tmp/nrfjprogdll/highlevel/filew8XONI", 0x7ffc94e44bf0) = -1 ENOENT (No such file or directory)
17292 stat("/tmp/nrfjprogdll/highlevel/filew8XONI", 0x7ffc94e44bf0) = -1 ENOENT (No such file or directory)
17292 stat("/tmp/nrfjprogdll/highlevel/filew8XONI", 0x7ffc94e44bf0) = -1 ENOENT (No such file or directory)
17292 stat("/tmp/nrfjprogdll/highlevel/filew8XONI", 0x7ffc94e44bf0) = -1 ENOENT (No such file or directory)
17292 stat("/tmp/nrfjprogdll/highlevel/filew8XONI", 0x7ffc94e44bf0) = -1 ENOENT (No such file or directory)
17292 stat("/tmp/nrfjprogdll/highlevel/filew8XONI", 0x7ffc94e44bf0) = -1 ENOENT (No such file or directory)
17292 stat("/tmp/nrfjprogdll/highlevel/filew8XONI", 0x7ffc94e44bf0) = -1 ENOENT (No such file or directory)
17292 stat("/tmp/nrfjprogdll/highlevel/filew8XONI", 0x7ffc94e44bf0) = -1 ENOENT (No such file or directory)
17292 stat("/tmp/nrfjprogdll/highlevel/filew8XONI", 0x7ffc94e44bf0) = -1 ENOENT (No such file or directory)
17292 stat("/tmp/nrfjprogdll/highlevel/filew8XONI", 0x7ffc94e44bf0) = -1 ENOENT (No such file or directory)
17292 stat("/tmp/nrfjprogdll/highlevel/filew8XONI", 0x7ffc94e44bf0) = -1 ENOENT (No such file or directory)
17292 stat("/tmp/nrfjprogdll/highlevel/filew8XONI", 0x7ffc94e44bf0) = -1 ENOENT (No such file or directory)
17292 stat("/tmp/nrfjprogdll/highlevel/filew8XONI", 0x7ffc94e44bf0) = -1 ENOENT (No such file or directory)
17292 stat("/tmp/nrfjprogdll/highlevel/filew8XONI", 0x7ffc94e44bf0) = -1 ENOENT (No such file or directory)
17292 stat("/tmp/nrfjprogdll/highlevel/filew8XONI", 0x7ffc94e44bf0) = -1 ENOENT (No such file or directory)

I can't find any instance of the word "Copy" in ~/.local/lib/python3.7/site-packages/pynrfjprog or github.com/.../pynrfjprog other than unrelated copyright notices, so I'm not sure where the message "Copy / -> /tmp/..." and associated code is.

[Håkon Alseth]

Hi,

 

Unfortunately, this is a bug that we recently found in pynrfjprog. We are working on fixing this, but its not yet ready for release.

My apologies for the inconvenience this has caused.

 

Is it possible that you can use nRF connect for desktop (programmer app, works on *nix + win) to upgrade the modem on this device?

 

Kind regards,

Håkon

[Triffid_Hunter]

Håkon Alseth said:

Is it possible that you can use nRF connect for desktop (programmer app, works on *nix + win) to upgrade the modem on this device?

Not until https://devzone.nordicsemi.com/f/nordic-q-a/64132/nordic-thingy91-error-updating-modem-firmware is fixed for Linux, and since I have a very strong penchant for Makefiles and command-line tools (since I enjoy hands-off automation and CI) that would only ever be an unsustainable kludge for me.

How usable are the tools mentioned in https://devzone.nordicsemi.com/f/nordic-q-a/14671/nrfjprog-for-arm-cpu-raspberry-pi-3 these days?

[Håkon Alseth]

Hi,

 

Triffid_Hunter said:

How usable are the tools mentioned in https://devzone.nordicsemi.com/f/nordic-q-a/14671/nrfjprog-for-arm-cpu-raspberry-pi-3 these days?

that is a shell script on top of JLinkExe, which does not have support for programming a nRF9160 modem, unfortunately. You can use it to program the application (JLinkExe -> loadfile <hex>), but not the modem.

 

Triffid_Hunter said:

Not until https://devzone.nordicsemi.com/f/nordic-q-a/64132/nordic-thingy91-error-updating-modem-firmware is fixed for Linux, and since I have a very strong penchant for Makefiles and command-line tools (since I enjoy hands-off automation and CI) that would only ever be an unsustainable kludge for me.

That case is related to using mcuboot for upgrading the modem (ie: a USB CDC device) on the thingy:91, and most of the errors in that case is related to either non-installed JLink binaries / nrfjprog binaries, or path problems towards one or both of these. I'm not excluding issues with other distros, but its more likely that its deps or path related.

I tried the sequence for mcuboot-upgrade-of-modem + programming it standalone using the nrf connect v3.4.2 - programmer app, and both are working in ubuntu 20.04 amd64. If you get an error message when trying this procedure either via thingy:91's mcuboot or via an external debugger, please share it with me and I'll try to help out.

 

Kind regards,

Håkon