Thingy91: pynrfjprog deadlocks at 100% cpu usage

[Triffid_Hunter]

I'm using the script from https://devzone.nordicsemi.com/f/nordic-q-a/53208/updating-nrf9160-modem-firmware-through-the-command-line/215357#215357

Under strace, it never forks any other process and doesn't create the file it's looking for, so I'm really not sure what it's expecting to happen.

Full strace at http://triffid-hunter.no-ip.info/pynrf-flash.strace since this forum thing keeps throwing errors when I try to attach it.

https://devzone.nordicsemi.com/f/nordic-q-a/62397/nrfjprog-bug-gets-stuck-due-to-temporary-directory-permissions seems related however my permissions are fine, and the directory tree in /tmp is recreated identically if I wipe it first.

Since this is apparently the only way to update the NRF91 modem firmware from a terminal, and all the examples seem to be complaining about out-of-date modem firmware, I'm once again stuck.

[Triffid_Hunter]

Håkon Alseth said:

Is this the JLink version you're using?

JLink_Linux_V630h_x86_64

nrfjprog -v says nrfjprog version: 10.9.0, JLinkARM.dll version: 6.80a

Håkon Alseth said:

Did you update over ttyACMx in this trace log? Could you attach the error code when trying to connect using a standard swd debugger? a screenshot would suffice.

[Triffid_Hunter]

Hi Håkon,

We had one of Nordic's FAEs (George Xiong) come over today to have a poke at things here.

He brought a couple of NRF9160-DKs with him, and the NRF Connect app says the same thing (Unsupported device. The detected device could not be recognized as neither JLink device nor Nordic USB device) even with the official DK plugged in to my machine.

nrfjprog recognised it (and the nrf9160) just fine, but apparently nrfjprog can't flash modem firmware.

He was unfamiliar with the pynrfjprog library and thus unable to help make any progress there.

His computer was also unable to flash the modem firmware, but we scavenged a colleagues' laptop which was able to do it - so that gets us over this roadblock for now but isn't a sustainable solution.

[Håkon Alseth]

Hi,

 

I have been discussing your issues with a couple of colleagues.

To sum up the behavior (there might be more, but these are the two we have been able to spot):

* Cannot access /tmp/nrfjprogdll fully

* nrf connect seems to find an older JLink driver (v6.30h) - this driver supports Cortex M33, but not the "nrf9160_xxaa" device profile.

 

We do have an open issue related to /tmp/nrfjprogdll path, which is related to if two users try to use nrfjprog. This will then be problematic for the second user (unless its root, or the users share rights), due to permissions to the folder and files within the folder.

A typical issue of this matter is that nrfjprog (and pynrfjprog) shows lines like these in the strace:

stat("/tmp/nrfjprogdll/highlevel/file12YHdh", 0x7ffd8bf8a230) = -1 ENOENT (No such file or directory)

You can try to run "sudo chown $MY_USER:$MY_USER /tmp/nrfjprogdll -R" in that case, and see if the issue disappears. We are addressing this specific issue in the upcoming nrfjprog v10.10.0.

 

As for the JLink driver installation, from the trace it looks like you have an older installation in your home folder. Is this correct? If yes, could you try to remove that?

The default location it looks at is "/opt/SEGGER/JLink" (which is a symlink to /opt/SEGGER/JLink_$(VERSION), on my end: /opt/SEGGER/JLink_V680a/).

Could you try to install as described above, ie. /opt/SEGGER/JLink ?

 

Could you share which specific debugger you are using?

 

Kind regards,

Håkon

[Triffid_Hunter]

Håkon Alseth said:

We do have an open issue related to /tmp/nrfjprogdll path, which is related to if two users try to use nrfjprog. This will then be problematic for the second user (unless its root, or the users share rights), due to permissions to the folder and files within the folder.

A typical issue of this matter is that nrfjprog (and pynrfjprog) shows lines like these in the strace:

stat("/tmp/nrfjprogdll/highlevel/file12YHdh", 0x7ffd8bf8a230) = -1 ENOENT (No such file or directory)

I only have one user on this machine, and changing the owner had no effect since I already own everything in that path.

According to strace, pynrfjprog never attempts to create that file/dir, which is why it doesn't exist:

13529 openat(AT_FDCWD, "/tmp/nrfjprogdll/highlevel", O_RDONLY|O_NONBLOCK|O_CLOEXEC|O_DIRECTORY) = -1 ENOENT (No such file or directory)
13529 lstat("/tmp/nrfjprogdll/highlevel", 0x7ffd6c987050) = -1 ENOENT (No such file or directory)
13529 mkdir("/tmp/nrfjprogdll/highlevel", 0777) = 0
13529 stat("/tmp/nrfjprogdll/highlevel/file4bYtba", 0x7ffd6c987230) = -1 ENOENT (No such file or directory)
13529 stat("/tmp/nrfjprogdll/highlevel/file4bYtba.lock", 0x7ffd6c987120) = -1 ENOENT (No such file or directory)
13529 stat("/tmp/nrfjprogdll/highlevel/file4bYtba.lock", 0x7ffd6c987070) = -1 ENOENT (No such file or directory)
13529 openat(AT_FDCWD, "/tmp/nrfjprogdll/highlevel/file4bYtba.lock", O_WRONLY|O_CREAT|O_TRUNC, 0666) = 3
13529 write(2, "DEBUG:pynrfjprog.HighLevel:HighLevel_dll_open:\tCopy \"/\"->\"/tmp/nrfjprogdll/highlevel/file4bYtba\"\n", 97) = 97
13529 stat("/tmp/nrfjprogdll/highlevel/file4bYtba", 0x7ffd6c986e90) = -1 ENOENT (No such file or directory)
13529 stat("/tmp/nrfjprogdll/highlevel/file4bYtba", 0x7ffd6c987230) = -1 ENOENT (No such file or directory)
13529 stat("/tmp/nrfjprogdll/highlevel/file4bYtba", 0x7ffd6c987230) = -1 ENOENT (No such file or directory)
13529 stat("/tmp/nrfjprogdll/highlevel/file4bYtba", 0x7ffd6c987230) = -1 ENOENT (No such file or directory)

This is after I deleted /tmp/nrfjprogdll, as you can see it recreates /tmp/nrfjprogdll/highlevel, creates file4bYtba.lock, but never tries to create file4bYtba itself - then spins in an endless loop waiting for it to exist.

Note the lack of any -1 EACCESS (Permission denied) responses, which would show up if there was a permission issue.

What exactly is supposed to create that file? It seems that that thing is failing, but pynrfjprog fails to capture any error.
It says it's trying to copy something ("/" apparently..?), but strace can't see it actually try to achieve this.

For your multi-user permissions issue, you could use mktemp -d or use /tmp/nrfjprogdll-$username$ - but that won't help with my  case since I don't (seem to) have a permissions problem.

[Håkon Alseth]

Hi,

 

This is the strace (here for reference: pynrfjprog.7z) from my side when doing a modem fw upgrade via pynrfjprog:

stat("/opt/SEGGER/JLink/libjlinkarm.so.6.80.1", {st_mode=S_IFREG|0755, st_size=18584272, ...}) = 0
openat(AT_FDCWD, "/tmp/nrfjprogdll/highlevel", O_RDONLY|O_NONBLOCK|O_CLOEXEC|O_DIRECTORY) = 3

 

For comparison, this is the same procedure in your strace:

stat("/", {st_mode=S_IFDIR|0755, st_size=178, ...}) = 0
openat(AT_FDCWD, "/tmp/nrfjprogdll/highlevel", O_RDONLY|O_NONBLOCK|O_CLOEXEC|O_DIRECTORY) = 3

 

This looks like a weakness in our libraries (or the search algorithm).

I believe that pynrfjprog isn't able to successfully find your segger driver path. Could you please place it in /opt/SEGGER/JLink to see if it is able to detect it properly then?

 

What distro are you running?

Which debugger are you currently using?

 

Kind regards,

Håkon