Thingy91: pynrfjprog deadlocks at 100% cpu usage

RE: Thingy91: pynrfjprog deadlocks at 100% cpu usage

Håkon Alseth — Mon, 17 Aug 2020 08:55:58 GMT

Hi,

I have to apologize for the bumpy ride wrt. running our tools in Linux. We are continuously trying to improve, and you have very good feedback for us. It is highly appreciated!

[quote user="Triffid_Hunter"]It certainly does seem like your software needs improvements in its error handling, a simple "The JLink library at <path> is too old, please update" or "The JLink library can't be found in <searchpath>" would have saved a ton of drama - with the specific path and the issue with them presented in this manner, it would take but a moment to drop a symlink there.[/quote]

Yes, it is unfortunately restrictions in the current built binary, which I will bring up with the team as improvements in the future.

[quote user="Triffid_Hunter"]PS: why aren't you using environ[LD_LIBRARY_PATH] for this? It exists specifically for this exact purpose, and I already have my JLink library path listed in there so nrfjprog can find it.[/quote]

I'll also discuss this with them.

[quote user="Triffid_Hunter"]but pynrfjprog doesn't seem to have improved.[/quote]

Is the strace equal as before? ie. loads of "stat("/tmp/nrfjprogdll/highlevel/file12YHdh", 0x7ffd8bf8a230) = -1 ENOENT (No such file or directory)" ?

The file<rand> is a copy of the local /path/to/JLink/libjlinkarm.so, as shown here:

hkn /tmp/nrfjprogdll/highlevel 
$ file /opt/SEGGER/JLink/libjlinkarm.so.6.80.1 
/opt/SEGGER/JLink/libjlinkarm.so.6.80.1: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, BuildID[sha1]=329721bc307868771dcf0d4ee24af5790fc4b16d, stripped

hkn /tmp/nrfjprogdll/highlevel 
$ file filezT91qO
filezT91qO: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, BuildID[sha1]=329721bc307868771dcf0d4ee24af5790fc4b16d, stripped

If that is still not created, I suspect that pynrfjprog still isn't able to successfully find the segger installation.

As a quick workaround, can you try to copy the libjlinkarm.so to the tmp/nrfjprogdll/highlevel/file<rand-prefx> and see how it behaves?

I see we have a race-condition in updating the thread, as re-installing pynrfjprog seems to have helped on this scenario.

Kind regards,

Håkon

RE: Thingy91: pynrfjprog deadlocks at 100% cpu usage

Triffid_Hunter — Mon, 17 Aug 2020 08:53:40 GMT

I reinstalled pynrfjprog and it seems to be working now.

It seems like my problems were caused by your software totally ignoring LD_LIBRARY_PATH and instead trying to silently use a hard-coded path outside my home dir, coupled with a complete lack of usable error checking and reporting.

Now I'm finding that the modem won't talk to anything but I'll make a separate post about that.

RE: Thingy91: pynrfjprog deadlocks at 100% cpu usage

Triffid_Hunter — Fri, 14 Aug 2020 12:03:10 GMT

I made /opt/SEGGER/JLink a symlink to my JLink_V680a folder, and NRF Connect programmer is rather happier about it (I'm not even sure how it was finding the older version before, it never asked me where to look), but pynrfjprog doesn't seem to have improved.

I've tried with both a regular JLink (possibly a knockoff) as well as an official Nordic NRF9160-DK delivered by your FAE today.

I'm using Gentoo because it's the only distribution I've tried that doesn't fight me when I tell it what I want.

It certainly does seem like your software needs improvements in its error handling, a simple "The JLink library at <path> is too old, please update" or "The JLink library can't be found in <searchpath>" would have saved a ton of drama - with the specific path and the issue with them presented in this manner, it would take but a moment to drop a symlink there.

PS: why aren't you using environ[LD_LIBRARY_PATH] for this? It exists specifically for this exact purpose, and I already have my JLink library path listed in there so nrfjprog can find it.

RE: Thingy91: pynrfjprog deadlocks at 100% cpu usage

Håkon Alseth — Fri, 14 Aug 2020 11:54:55 GMT

Hi,

This is the strace (here for reference: devzone.nordicsemi.com/.../pynrfjprog.7z) from my side when doing a modem fw upgrade via pynrfjprog:

stat("/opt/SEGGER/JLink/libjlinkarm.so.6.80.1", {st_mode=S_IFREG|0755, st_size=18584272, ...}) = 0
openat(AT_FDCWD, "/tmp/nrfjprogdll/highlevel", O_RDONLY|O_NONBLOCK|O_CLOEXEC|O_DIRECTORY) = 3

For comparison, this is the same procedure in your strace:

stat("/", {st_mode=S_IFDIR|0755, st_size=178, ...}) = 0
openat(AT_FDCWD, "/tmp/nrfjprogdll/highlevel", O_RDONLY|O_NONBLOCK|O_CLOEXEC|O_DIRECTORY) = 3

This looks like a weakness in our libraries (or the search algorithm).

I believe that pynrfjprog isn't able to successfully find your segger driver path. Could you please place it in /opt/SEGGER/JLink to see if it is able to detect it properly then?

What distro are you running?

Which debugger are you currently using?

Kind regards,

Håkon

RE: Thingy91: pynrfjprog deadlocks at 100% cpu usage

Triffid_Hunter — Fri, 14 Aug 2020 09:48:08 GMT

[quote userid="2115" url="~/f/nordic-q-a/64656/thingy91-pynrfjprog-deadlocks-at-100-cpu-usage/264582"][/quote]

We do have an open issue related to /tmp/nrfjprogdll path, which is related to if two users try to use nrfjprog. This will then be problematic for the second user (unless its root, or the users share rights), due to permissions to the folder and files within the folder.

A typical issue of this matter is that nrfjprog (and pynrfjprog) shows lines like these in the strace:

stat("/tmp/nrfjprogdll/highlevel/file12YHdh", 0x7ffd8bf8a230) = -1 ENOENT (No such file or directory)

I only have one user on this machine, and changing the owner had no effect since I already own everything in that path.

According to strace, pynrfjprog never attempts to create that file/dir, which is why it doesn't exist:

13529 openat(AT_FDCWD, "/tmp/nrfjprogdll/highlevel", O_RDONLY|O_NONBLOCK|O_CLOEXEC|O_DIRECTORY) = -1 ENOENT (No such file or directory)
13529 lstat("/tmp/nrfjprogdll/highlevel", 0x7ffd6c987050) = -1 ENOENT (No such file or directory)
13529 mkdir("/tmp/nrfjprogdll/highlevel", 0777) = 0
13529 stat("/tmp/nrfjprogdll/highlevel/file4bYtba", 0x7ffd6c987230) = -1 ENOENT (No such file or directory)
13529 stat("/tmp/nrfjprogdll/highlevel/file4bYtba.lock", 0x7ffd6c987120) = -1 ENOENT (No such file or directory)
13529 stat("/tmp/nrfjprogdll/highlevel/file4bYtba.lock", 0x7ffd6c987070) = -1 ENOENT (No such file or directory)
13529 openat(AT_FDCWD, "/tmp/nrfjprogdll/highlevel/file4bYtba.lock", O_WRONLY|O_CREAT|O_TRUNC, 0666) = 3
13529 write(2, "DEBUG:pynrfjprog.HighLevel:HighLevel_dll_open:\tCopy \"/\"->\"/tmp/nrfjprogdll/highlevel/file4bYtba\"\n", 97) = 97
13529 stat("/tmp/nrfjprogdll/highlevel/file4bYtba", 0x7ffd6c986e90) = -1 ENOENT (No such file or directory)
13529 stat("/tmp/nrfjprogdll/highlevel/file4bYtba", 0x7ffd6c987230) = -1 ENOENT (No such file or directory)
13529 stat("/tmp/nrfjprogdll/highlevel/file4bYtba", 0x7ffd6c987230) = -1 ENOENT (No such file or directory)
13529 stat("/tmp/nrfjprogdll/highlevel/file4bYtba", 0x7ffd6c987230) = -1 ENOENT (No such file or directory)

This is after I deleted /tmp/nrfjprogdll, as you can see it recreates /tmp/nrfjprogdll/highlevel, creates file4bYtba.lock, but never tries to create file4bYtba itself - then spins in an endless loop waiting for it to exist.

Note the lack of any -1 EACCESS (Permission denied) responses, which would show up if there was a permission issue.

What exactly is supposed to create that file? It seems that that thing is failing, but pynrfjprog fails to capture any error.
It says it's trying to copy something ("/" apparently..?), but strace can't see it actually try to achieve this.

For your multi-user permissions issue, you could use mktemp -d or use /tmp/nrfjprogdll-$username$ - but that won't help with my case since I don't (seem to) have a permissions problem.

RE: Thingy91: pynrfjprog deadlocks at 100% cpu usage

Håkon Alseth — Fri, 14 Aug 2020 09:19:54 GMT

Hi,

I have been discussing your issues with a couple of colleagues.

To sum up the behavior (there might be more, but these are the two we have been able to spot):

* Cannot access /tmp/nrfjprogdll fully

* nrf connect seems to find an older JLink driver (v6.30h) - this driver supports Cortex M33, but not the "nrf9160_xxaa" device profile.

A typical issue of this matter is that nrfjprog (and pynrfjprog) shows lines like these in the strace:

stat("/tmp/nrfjprogdll/highlevel/file12YHdh", 0x7ffd8bf8a230) = -1 ENOENT (No such file or directory)

You can try to run "sudo chown $MY_USER:$MY_USER /tmp/nrfjprogdll -R" in that case, and see if the issue disappears. We are addressing this specific issue in the upcoming nrfjprog v10.10.0.

As for the JLink driver installation, from the trace it looks like you have an older installation in your home folder. Is this correct? If yes, could you try to remove that?

The default location it looks at is "/opt/SEGGER/JLink" (which is a symlink to /opt/SEGGER/JLink_$(VERSION), on my end: /opt/SEGGER/JLink_V680a/).

Could you try to install as described above, ie. /opt/SEGGER/JLink ?

Could you share which specific debugger you are using?

Kind regards,

Håkon

RE: Thingy91: pynrfjprog deadlocks at 100% cpu usage

Triffid_Hunter — Fri, 14 Aug 2020 07:34:24 GMT

Hi Håkon,

We had one of Nordic's FAEs (George Xiong) come over today to have a poke at things here.

He brought a couple of NRF9160-DKs with him, and the NRF Connect app says the same thing (Unsupported device. The detected device could not be recognized as neither JLink device nor Nordic USB device) even with the official DK plugged in to my machine.

nrfjprog recognised it (and the nrf9160) just fine, but apparently nrfjprog can't flash modem firmware.

He was unfamiliar with the pynrfjprog library and thus unable to help make any progress there.

His computer was also unable to flash the modem firmware, but we scavenged a colleagues' laptop which was able to do it - so that gets us over this roadblock for now but isn't a sustainable solution.

RE: Thingy91: pynrfjprog deadlocks at 100% cpu usage

Triffid_Hunter — Thu, 13 Aug 2020 09:28:11 GMT

[quote userid="2115" url="~/f/nordic-q-a/64656/thingy91-pynrfjprog-deadlocks-at-100-cpu-usage/264241"]Is this the JLink version you're using?[/quote]

JLink_Linux_V630h_x86_64

nrfjprog -v says nrfjprog version: 10.9.0, JLinkARM.dll version: 6.80a

[quote userid="2115" url="~/f/nordic-q-a/64656/thingy91-pynrfjprog-deadlocks-at-100-cpu-usage/264241"]Did you update over ttyACMx in this trace log? Could you attach the error code when trying to connect using a standard swd debugger? a screenshot would suffice.[/quote]

RE: Thingy91: pynrfjprog deadlocks at 100% cpu usage

Håkon Alseth — Wed, 12 Aug 2020 11:25:10 GMT

Hi,

Is this the JLink version you're using?

JLink_Linux_V630h_x86_64

If that is the case, could you please update to a newer version? This one is quite old, and does not include support for nrf9160 (Cortex m33). You can download a bundle from here: https://www.nordicsemi.com/Software-and-tools/Development-Tools/nRF-Command-Line-Tools/Download#infotabs

Did you update over ttyACMx in this trace log? Could you attach the error code when trying to connect using a standard swd debugger? a screenshot would suffice.

Kind regards,

Håkon

RE: Thingy91: pynrfjprog deadlocks at 100% cpu usage

Triffid_Hunter — Wed, 12 Aug 2020 10:10:32 GMT

I ran nRF Connect programmer under strace to see what it's doing when I try to update modem firmware, doesn't look like it's doing much at all with the serial port - opens the port, writes a few bytes, hands it off to another process which waits a moment then quits.

strace log here: triffid-hunter.no-ip.info/nrfc.strace.gz (gzipped because it's 47MB uncompressed) - ACM0 is opened at line 424297.

RE: Thingy91: pynrfjprog deadlocks at 100% cpu usage

Triffid_Hunter — Wed, 12 Aug 2020 08:24:59 GMT

I've also tried using a JLink, but nRF Connect Programmer won't recognise it, claiming "ERROR Unsupported device. The detected device could not be recognized as neither JLink device nor Nordic USB device." even though it works fine with nrfjprog :-

$ nrfjprog --deviceversion
NRF9160_xxAA_REV2

RE: Thingy91: pynrfjprog deadlocks at 100% cpu usage

Triffid_Hunter — Wed, 12 Aug 2020 08:06:52 GMT

Same error with latest version of nRF connect and updated programmer:

2020-08-12T07:54:14.478Z INFO Application data folder: /home/triffid/.config/nrfconnect/pc-nrfconnect-programmer
2020-08-12T07:54:14.574Z DEBUG App pc-nrfconnect-programmer v1.4.4 official
2020-08-12T07:54:14.574Z DEBUG App path: /home/triffid/.nrfconnect-apps/node_modules/pc-nrfconnect-programmer
2020-08-12T07:54:14.575Z DEBUG nRFConnect 3.4.2 is supported by the app (^3.3.0)
2020-08-12T07:54:14.575Z DEBUG nRFConnect path: /tmp/.mount_nrfconlNUQmN/resources/app.asar
2020-08-12T07:54:14.575Z DEBUG HomeDir: /home/triffid
2020-08-12T07:54:14.575Z DEBUG TmpDir: /tmp
2020-08-12T07:54:14.582Z INFO Using nrfjprog library 10.9.0, pc-nrfjprog-js 1.7.3
2020-08-12T07:54:14.612Z ERROR Error while probing devices: Error occured when get serial numbers. Errorcode: CouldNotCallFunction (0x9)
Lowlevel error: INVALID_OPERATION (fffffffe)

2020-08-12T07:54:25.278Z INFO Modem DFU starts to write...
2020-08-12T07:54:25.279Z INFO Writing /home/triffid/Projects/NRF/thingy91_fw_2020-04-29_bc7ade8b/mfwnrf9160120.zip to device
2020-08-12T07:54:25.719Z ERROR MCUboot DFU failed. Please make sure that the device is in MCUboot mode and try again.

I followed the instructions wrt holding SW3 during poweron to access MCUBoot mode, although this doesn't seem to make any difference to the USB descriptors that it delivers:

$ lsusb -d 1915:520f -v

Bus 001 Device 026: ID 1915:520f Nordic Semiconductor ASA Weltrend USB Mouse
Device Descriptor:
bLength                18
bDescriptorType         1
bcdUSB               2.00
bDeviceClass          239 Miscellaneous Device
bDeviceSubClass         2
bDeviceProtocol         1 Interface Association
bMaxPacketSize0        64
idVendor           0x1915 Nordic Semiconductor ASA
idProduct          0x520f
bcdDevice            0.11
iManufacturer           1 Nordic Semiconductor
iProduct                2 Thingy:91 UART
iSerial                 3 PCA20035_e7cec1bcfe84
bNumConfigurations      1
Configuration Descriptor:
    bLength                 9
    bDescriptorType         2
    wTotalLength       0x008d
    bNumInterfaces          4
    bConfigurationValue     1
    iConfiguration          0
    bmAttributes         0xe0
      Self Powered
      Remote Wakeup
    MaxPower              100mA
    Interface Association:
      bLength                 8
      bDescriptorType        11
      bFirstInterface         0
      bInterfaceCount         2
      bFunctionClass          2 Communications
      bFunctionSubClass       2 Abstract (modem)
      bFunctionProtocol       0
      iFunction               0
    Interface Descriptor:
      bLength                 9
      bDescriptorType         4
      bInterfaceNumber        0
      bAlternateSetting       0
      bNumEndpoints           1
      bInterfaceClass         2 Communications
      bInterfaceSubClass      2 Abstract (modem)
      bInterfaceProtocol      0
      iInterface              0
      CDC Header:
        bcdCDC               1.10
      CDC Call Management:
        bmCapabilities       0x02
          use DataInterface
        bDataInterface          1
      CDC ACM:
        bmCapabilities       0x02
          line coding and serial state
      CDC Union:
        bMasterInterface        0
        bSlaveInterface         1
      Endpoint Descriptor:
        bLength                 7
        bDescriptorType         5
        bEndpointAddress     0x81 EP 1 IN
        bmAttributes            3
          Transfer Type            Interrupt
          Synch Type               None
          Usage Type               Data
        wMaxPacketSize     0x0010 1x 16 bytes
        bInterval              10
    Interface Descriptor:
      bLength                 9
      bDescriptorType         4
      bInterfaceNumber        1
      bAlternateSetting       0
      bNumEndpoints           2
      bInterfaceClass        10 CDC Data
      bInterfaceSubClass      0
      bInterfaceProtocol      0
      iInterface              0
      Endpoint Descriptor:
        bLength                 7
        bDescriptorType         5
        bEndpointAddress     0x82 EP 2 IN
        bmAttributes            2
          Transfer Type            Bulk
          Synch Type               None
          Usage Type               Data
        wMaxPacketSize     0x0040 1x 64 bytes
        bInterval               0
      Endpoint Descriptor:
        bLength                 7
        bDescriptorType         5
        bEndpointAddress     0x01 EP 1 OUT
        bmAttributes            2
          Transfer Type            Bulk
          Synch Type               None
          Usage Type               Data
        wMaxPacketSize     0x0040 1x 64 bytes
        bInterval               0
    Interface Association:
      bLength                 8
      bDescriptorType        11
      bFirstInterface         2
      bInterfaceCount         2
      bFunctionClass          2 Communications
      bFunctionSubClass       2 Abstract (modem)
      bFunctionProtocol       0
      iFunction               0
    Interface Descriptor:
      bLength                 9
      bDescriptorType         4
      bInterfaceNumber        2
      bAlternateSetting       0
      bNumEndpoints           1
      bInterfaceClass         2 Communications
      bInterfaceSubClass      2 Abstract (modem)
      bInterfaceProtocol      0
      iInterface              0
      CDC Header:
        bcdCDC               1.10
      CDC Call Management:
        bmCapabilities       0x02
          use DataInterface
        bDataInterface          1
      CDC ACM:
        bmCapabilities       0x02
          line coding and serial state
      CDC Union:
        bMasterInterface        2
        bSlaveInterface         3
      Endpoint Descriptor:
        bLength                 7
        bDescriptorType         5
        bEndpointAddress     0x83 EP 3 IN
        bmAttributes            3
          Transfer Type            Interrupt
          Synch Type               None
          Usage Type               Data
        wMaxPacketSize     0x0010 1x 16 bytes
        bInterval              10
    Interface Descriptor:
      bLength                 9
      bDescriptorType         4
      bInterfaceNumber        3
      bAlternateSetting       0
      bNumEndpoints           2
      bInterfaceClass        10 CDC Data
      bInterfaceSubClass      0
      bInterfaceProtocol      0
      iInterface              0
      Endpoint Descriptor:
        bLength                 7
        bDescriptorType         5
        bEndpointAddress     0x84 EP 4 IN
        bmAttributes            2
          Transfer Type            Bulk
          Synch Type               None
          Usage Type               Data
        wMaxPacketSize     0x0040 1x 64 bytes
        bInterval               0
      Endpoint Descriptor:
        bLength                 7
        bDescriptorType         5
        bEndpointAddress     0x02 EP 2 OUT
        bmAttributes            2
          Transfer Type            Bulk
          Synch Type               None
          Usage Type               Data
        wMaxPacketSize     0x0040 1x 64 bytes
        bInterval               0
Device Status:     0x0000
(Bus Powered)

I tried numerous times, both with and without SW3 held to no avail.

RE: Thingy91: pynrfjprog deadlocks at 100% cpu usage

Triffid_Hunter — Wed, 12 Aug 2020 06:37:06 GMT

Just FYI, amazon is throwing 403 errors at your client when it tries to do partial updates.

13:05:22.626 › Checking for update
13:05:42.052 › Found version 3.4.2 (url: nrfconnect-3.4.2-x86_64.AppImage)
13:05:47.330 › Downloading update from nrfconnect-3.4.2-x86_64.AppImage
13:05:47.349 › updater cache dir: /home/triffid/.cache/nrfconnect-updater
13:05:47.363 › No cached update info available
13:07:08.977 › dkTihOXnG2ovqMEkpxwLPI+I duplicated in blockmap (same size), it doesn't lead to broken differential downloader, just corresponding block will be skipped)
13:07:08.977 › JRb1UK4Z/x76FrzcLRE0UX6J duplicated in blockmap (same size), it doesn't lead to broken differential downloader, just corresponding block will be skipped)
13:07:08.977 › oqYY7uJtPGEu9GT1URW/ZIZc duplicated in blockmap (same size), it doesn't lead to broken differential downloader, just corresponding block will be skipped)
13:07:08.977 › gRW5vkI5NXYlu2aP5yoTNMeY duplicated in blockmap (same size), it doesn't lead to broken differential downloader, just corresponding block will be skipped)
13:07:08.977 › 9LuhCEOQzb8vn0X29vVoEb83 duplicated in blockmap (same size), it doesn't lead to broken differential downloader, just corresponding block will be skipped)
13:07:08.977 › o6Hp7Jg7kvy5lZV89L6QRBa7 duplicated in blockmap (same size), it doesn't lead to broken differential downloader, just corresponding block will be skipped)
13:07:08.977 › /L8C1zeBidj9/kArl4fnYFM5 duplicated in blockmap (same size), it doesn't lead to broken differential downloader, just corresponding block will be skipped)
13:07:08.978 › tSaId5LMeXrZEioF0oMGS+Z7 duplicated in blockmap (same size), it doesn't lead to broken differential downloader, just corresponding block will be skipped)
13:07:08.978 › br2FvDMSa6Bk8w6LNGa/5RFf duplicated in blockmap (same size), it doesn't lead to broken differential downloader, just corresponding block will be skipped)
13:07:08.979 › File has 1302 changed blocks
13:07:08.979 › [
{
"kind": 0,
"start": 0,
"end": 159006
},
{
"kind": 1,
"start": 159006,
"end": 191774
},
{
"kind": 0,
"start": 191774,
"end": 59020876
},
{
"kind": 1,
"start": 59020876,
"end": 59032521
},
{
"kind": 0,
"start": 59032522,
"end": 66092751
},
{
"kind": 1,
"start": 66092750,
"end": 66813059
},
{
"kind": 0,
"start": 66812143,
"end": 66829834
},
{
"kind": 1,
"start": 66830750,
"end": 67114423
},
{
"kind": 0,
"start": 67113436,
"end": 67124550
},
{
"kind": 1,
"start": 67125537,
"end": 67323400
},
{
"kind": 0,
"start": 88126999,
"end": 88137299
},
{
"kind": 1,
"start": 67333700,
"end": 67509636
},
{
"kind": 0,
"start": 67508589,
"end": 67630889
},
{
"kind": 1,
"start": 67631936,
"end": 67653730
},
{
"kind": 0,
"start": 67652683,
"end": 67661468
},
{
"kind": 1,
"start": 67662515,
"end": 74477772
},
{
"kind": 0,
"start": 66812143,
"end": 66853473
},
{
"kind": 1,
"start": 74519102,
"end": 74660620
},
{
"kind": 0,
"start": 74640545,
"end": 74650179
},
{
"kind": 1,
"start": 74670254,
"end": 86724996
},
{
"kind": 0,
"start": 66683393,
"end": 66693181
},
{
"kind": 1,
"start": 86734784,
"end": 93452268
},
{
"kind": 0,
"start": 92925548,
"end": 100429017
},
{
"kind": 1,
"start": 100955737,
"end": 100990952
}
]
13:07:08.990 › Full: 98,728.24 KB, To download: 26,570.51 KB (27%)
13:07:08.992 › Differential download: github.com/.../nrfconnect-3.4.2-x86_64.AppImage
13:07:08.995 › download range: bytes=159006-191773
13:07:09.680 › Redirect to github-production-release-asset-2e65be.s3.amazonaws.com/.../ca45de00-d7cf-11ea-856e-beed2fafbb84
13:07:17.848 › download range: bytes=59020876-59032520
13:07:19.483 › download range: bytes=66092750-66813058
13:29:14.325 › download range: bytes=66830750-67114422
13:29:14.957 › Cannot download differentially, fallback to full download: HttpError: 403 Forbidden
Headers: {
"connection": [
"close"
],
"content-type": [
"application/xml"
],
"date": [
"Wed, 12 Aug 2020 05:29:14 GMT"
],
"server": [
"AmazonS3"
],
"transfer-encoding": [
"chunked"
],
"x-amz-id-2": [
"MJwp4R8UJc4aCwKW4b828qX1sqOVxulGmzRPJIcrpu94JbavvDoGYS5o2h0Ix6Ke96eZ6nWlYTU="
],
"x-amz-request-id": [
"8C1F1693119B1923"
]
}
at createHttpError (/tmp/.mount_nrfconNHk6h0/resources/app.asar/node_modules/builder-util-runtime/src/httpExecutor.ts:30:10)
at ClientRequest.<anonymous> (/tmp/.mount_nrfconNHk6h0/resources/app.asar/node_modules/electron-updater/src/differentialDownloader/DifferentialDownloader.ts:200:20)
at ClientRequest.emit (events.js:194:13)
at URLRequest.<anonymous> (/tmp/.mount_nrfconNHk6h0/resources/electron.asar/browser/api/net.js:207:12)
at URLRequest.emit (events.js:194:13)
From previous event:
at /tmp/.mount_nrfconNHk6h0/resources/app.asar/node_modules/graceful-fs/graceful-fs.js:333:16
at FSReqCallback.oncomplete (fs.js:150:20)
From previous event:
at FileWithEmbeddedBlockMapDifferentialDownloader.doDownloadFile (/tmp/.mount_nrfconNHk6h0/resources/app.asar/node_modules/electron-updater/src/differentialDownloader/DifferentialDownloader.ts:115:82)
at FileWithEmbeddedBlockMapDifferentialDownloader.downloadFile (/tmp/.mount_nrfconNHk6h0/resources/app.asar/node_modules/electron-updater/src/differentialDownloader/DifferentialDownloader.ts:89:17)
at FileWithEmbeddedBlockMapDifferentialDownloader.doDownload (/tmp/.mount_nrfconNHk6h0/resources/app.asar/node_modules/electron-updater/src/differentialDownloader/DifferentialDownloader.ts:76:17)
at /tmp/.mount_nrfconNHk6h0/resources/app.asar/node_modules/electron-updater/src/differentialDownloader/FileWithEmbeddedBlockMapDifferentialDownloader.ts:14:16
From previous event:
at FileWithEmbeddedBlockMapDifferentialDownloader.download (/tmp/.mount_nrfconNHk6h0/resources/app.asar/node_modules/electron-updater/src/differentialDownloader/FileWithEmbeddedBlockMapDifferentialDownloader.ts:8:17)
at Object.<anonymous> (/tmp/.mount_nrfconNHk6h0/resources/app.asar/node_modules/electron-updater/src/AppImageUpdater.ts:53:14)
at Generator.next (<anonymous>)
From previous event:
at Object.task (/tmp/.mount_nrfconNHk6h0/resources/app.asar/node_modules/electron-updater/src/AppImageUpdater.ts:37:11)
at /tmp/.mount_nrfconNHk6h0/resources/app.asar/node_modules/electron-updater/src/AppUpdater.ts:609:25
at Generator.next (<anonymous>)
at processImmediate (internal/timers.js:443:21)
From previous event:
at AppImageUpdater.executeDownload (/tmp/.mount_nrfconNHk6h0/resources/app.asar/node_modules/electron-updater/src/AppUpdater.ts:544:68)
at AppImageUpdater.executeDownload (/tmp/.mount_nrfconNHk6h0/resources/app.asar/node_modules/electron-updater/src/BaseUpdater.ts:27:18)
at AppImageUpdater.doDownloadUpdate (/tmp/.mount_nrfconNHk6h0/resources/app.asar/node_modules/electron-updater/src/AppImageUpdater.ts:33:17)
at AppImageUpdater.downloadUpdate (/tmp/.mount_nrfconNHk6h0/resources/app.asar/node_modules/electron-updater/src/AppUpdater.ts:419:19)
at callFunction (/tmp/.mount_nrfconNHk6h0/resources/electron.asar/browser/rpc-server.js:252:24)
at /tmp/.mount_nrfconNHk6h0/resources/electron.asar/browser/rpc-server.js:411:10
at EventEmitter.<anonymous> (/tmp/.mount_nrfconNHk6h0/resources/electron.asar/browser/rpc-server.js:275:21)
at EventEmitter.emit (events.js:194:13)
at WebContents.<anonymous> (/tmp/.mount_nrfconNHk6h0/resources/electron.asar/browser/api/web-contents.js:418:21)
at WebContents.emit (events.js:194:13)

I've been waiting a couple hours for it to finish the full download and it's still going - perhaps you should put a progress bar and some numbers in the client?

RE: Thingy91: pynrfjprog deadlocks at 100% cpu usage

Håkon Alseth — Tue, 11 Aug 2020 13:41:42 GMT

Hi,

[quote user="Triffid_Hunter"]How usable are the tools mentioned in https://devzone.nordicsemi.com/f/nordic-q-a/14671/nrfjprog-for-arm-cpu-raspberry-pi-3 these days?[/quote]

that is a shell script on top of JLinkExe, which does not have support for programming a nRF9160 modem, unfortunately. You can use it to program the application (JLinkExe -> loadfile <hex>), but not the modem.

[quote user="Triffid_Hunter"]Not until https://devzone.nordicsemi.com/f/nordic-q-a/64132/nordic-thingy91-error-updating-modem-firmware is fixed for Linux, and since I have a very strong penchant for Makefiles and command-line tools (since I enjoy hands-off automation and CI) that would only ever be an unsustainable kludge for me.[/quote]

That case is related to using mcuboot for upgrading the modem (ie: a USB CDC device) on the thingy:91, and most of the errors in that case is related to either non-installed JLink binaries / nrfjprog binaries, or path problems towards one or both of these. I'm not excluding issues with other distros, but its more likely that its deps or path related.

I tried the sequence for mcuboot-upgrade-of-modem + programming it standalone using the nrf connect v3.4.2 - programmer app, and both are working in ubuntu 20.04 amd64. If you get an error message when trying this procedure either via thingy:91's mcuboot or via an external debugger, please share it with me and I'll try to help out.

Kind regards,

Håkon

RE: Thingy91: pynrfjprog deadlocks at 100% cpu usage

Triffid_Hunter — Tue, 11 Aug 2020 12:56:16 GMT

[quote userid="2115" url="~/f/nordic-q-a/64656/thingy91-pynrfjprog-deadlocks-at-100-cpu-usage/264057"]Is it possible that you can use nRF connect for desktop (programmer app, works on *nix + win) to upgrade the modem on this device?[/quote]

Not until https://devzone.nordicsemi.com/f/nordic-q-a/64132/nordic-thingy91-error-updating-modem-firmware is fixed for Linux, and since I have a very strong penchant for Makefiles and command-line tools (since I enjoy hands-off automation and CI) that would only ever be an unsustainable kludge for me.

How usable are the tools mentioned in https://devzone.nordicsemi.com/f/nordic-q-a/14671/nrfjprog-for-arm-cpu-raspberry-pi-3 these days?

RE: Thingy91: pynrfjprog deadlocks at 100% cpu usage

Håkon Alseth — Tue, 11 Aug 2020 12:48:43 GMT

Hi,

Unfortunately, this is a bug that we recently found in pynrfjprog. We are working on fixing this, but its not yet ready for release.

My apologies for the inconvenience this has caused.

Is it possible that you can use nRF connect for desktop (programmer app, works on *nix + win) to upgrade the modem on this device?

Kind regards,

Håkon

RE: Thingy91: pynrfjprog deadlocks at 100% cpu usage

Triffid_Hunter — Tue, 11 Aug 2020 10:43:39 GMT

With logging.basicConfig(level=logging.DEBUG) I get this:

INFO:modem_update:Modem firmware upgrade
DEBUG:pynrfjprog.HighLevel:HighLevel_logger_open
DEBUG:pynrfjprog.HighLevel:HighLevel_logger_open: nRFJProg logger opened to callback at address 0X7F49C2B28010
DEBUG:pynrfjprog.HighLevel:HighLevel_dll_open
DEBUG:pynrfjprog.HighLevel:HighLevel_dll_open: Found J-Link library at "/".
DEBUG:pynrfjprog.HighLevel:HighLevel_dll_open: Copying Segger dll to temporary folder to allow a new probe connection.
DEBUG:pynrfjprog.HighLevel:HighLevel_dll_open: Copy "/"->"/tmp/nrfjprogdll/highlevel/filew8XONI"
*** (deadlocks here until killed)
Terminated

But despite saying it's copying, it never attempts to copy anything:

17292 lstat("/tmp/filew8XONI", 0x7ffc94e44b90) = -1 ENOENT (No such file or directory)
17292 stat("/tmp/nrfjprogdll/highlevel/filew8XONI", 0x7ffc94e44bf0) = -1 ENOENT (No such file or directory)
17292 stat("/tmp/nrfjprogdll/highlevel/filew8XONI.lock", 0x7ffc94e44ae0) = -1 ENOENT (No such file or directory)
17292 stat("/tmp/nrfjprogdll/highlevel/filew8XONI.lock", 0x7ffc94e44a30) = -1 ENOENT (No such file or directory)
17292 openat(AT_FDCWD, "/tmp/nrfjprogdll/highlevel/filew8XONI.lock", O_WRONLY|O_CREAT|O_TRUNC, 0666) = 3
17292 getpid() = 17292
17292 write(2, "DEBUG:pynrfjprog.HighLevel:HighLevel_dll_open:\tCopy \"/\"->\"/tmp/nrfjprogdll/highlevel/filew8XONI\"\n", 97) = 97
17292 stat("/tmp/nrfjprogdll/highlevel/filew8XONI", 0x7ffc94e44850) = -1 ENOENT (No such file or directory)
17292 stat("/", {st_mode=S_IFDIR|0755, st_size=178, ...}) = 0
17292 stat("/tmp/nrfjprogdll/highlevel/filew8XONI", 0x7ffc94e44bf0) = -1 ENOENT (No such file or directory)
17292 stat("/tmp/nrfjprogdll/highlevel/filew8XONI", 0x7ffc94e44bf0) = -1 ENOENT (No such file or directory)
17292 stat("/tmp/nrfjprogdll/highlevel/filew8XONI", 0x7ffc94e44bf0) = -1 ENOENT (No such file or directory)
17292 stat("/tmp/nrfjprogdll/highlevel/filew8XONI", 0x7ffc94e44bf0) = -1 ENOENT (No such file or directory)
17292 stat("/tmp/nrfjprogdll/highlevel/filew8XONI", 0x7ffc94e44bf0) = -1 ENOENT (No such file or directory)
17292 stat("/tmp/nrfjprogdll/highlevel/filew8XONI", 0x7ffc94e44bf0) = -1 ENOENT (No such file or directory)
17292 stat("/tmp/nrfjprogdll/highlevel/filew8XONI", 0x7ffc94e44bf0) = -1 ENOENT (No such file or directory)
17292 stat("/tmp/nrfjprogdll/highlevel/filew8XONI", 0x7ffc94e44bf0) = -1 ENOENT (No such file or directory)
17292 stat("/tmp/nrfjprogdll/highlevel/filew8XONI", 0x7ffc94e44bf0) = -1 ENOENT (No such file or directory)
17292 stat("/tmp/nrfjprogdll/highlevel/filew8XONI", 0x7ffc94e44bf0) = -1 ENOENT (No such file or directory)
17292 stat("/tmp/nrfjprogdll/highlevel/filew8XONI", 0x7ffc94e44bf0) = -1 ENOENT (No such file or directory)
17292 stat("/tmp/nrfjprogdll/highlevel/filew8XONI", 0x7ffc94e44bf0) = -1 ENOENT (No such file or directory)
17292 stat("/tmp/nrfjprogdll/highlevel/filew8XONI", 0x7ffc94e44bf0) = -1 ENOENT (No such file or directory)
17292 stat("/tmp/nrfjprogdll/highlevel/filew8XONI", 0x7ffc94e44bf0) = -1 ENOENT (No such file or directory)
17292 stat("/tmp/nrfjprogdll/highlevel/filew8XONI", 0x7ffc94e44bf0) = -1 ENOENT (No such file or directory)
17292 stat("/tmp/nrfjprogdll/highlevel/filew8XONI", 0x7ffc94e44bf0) = -1 ENOENT (No such file or directory)
17292 stat("/tmp/nrfjprogdll/highlevel/filew8XONI", 0x7ffc94e44bf0) = -1 ENOENT (No such file or directory)
17292 stat("/tmp/nrfjprogdll/highlevel/filew8XONI", 0x7ffc94e44bf0) = -1 ENOENT (No such file or directory)
17292 stat("/tmp/nrfjprogdll/highlevel/filew8XONI", 0x7ffc94e44bf0) = -1 ENOENT (No such file or directory)

I can't find any instance of the word "Copy" in ~/.local/lib/python3.7/site-packages/pynrfjprog or github.com/.../pynrfjprog other than unrelated copyright notices, so I'm not sure where the message "Copy / -> /tmp/..." and associated code is.

RE: Thingy91: pynrfjprog deadlocks at 100% cpu usage

Triffid_Hunter — Tue, 11 Aug 2020 08:53:00 GMT

Relevant section of strace, in case it helps:

stat("/home/triffid/Projects/NRF/NRFCommandLineTools-10.9.0/JLink_Linux_V680a_x86_64/", {st_mode=S_IFDIR|0755, st_size=1550, ...}) = 0
openat(AT_FDCWD, "/home/triffid/Projects/NRF/NRFCommandLineTools-10.9.0/JLink_Linux_V680a_x86_64/", O_RDONLY|O_NONBLOCK|O_CLOEXEC|O_DIRECTORY) = 3
fstat(3, {st_mode=S_IFDIR|0755, st_size=1550, ...}) = 0
getdents64(3, /* 57 entries */, 32768)  = 2112
getdents64(3, /* 0 entries */, 32768)   = 0
close(3)                                = 0
stat("/", {st_mode=S_IFDIR|0755, st_size=178, ...}) = 0
openat(AT_FDCWD, "/tmp/nrfjprogdll/highlevel", O_RDONLY|O_NONBLOCK|O_CLOEXEC|O_DIRECTORY) = 3
fstat(3, {st_mode=S_IFDIR|0755, st_size=180, ...}) = 0
getdents64(3, /* 9 entries */, 32768)   = 328
getdents64(3, /* 0 entries */, 32768)   = 0
close(3)                                = 0
lstat("/tmp/nrfjprogdll/highlevel", {st_mode=S_IFDIR|0755, st_size=180, ...}) = 0
stat("/tmp", {st_mode=S_IFDIR|S_ISVTX|0777, st_size=420, ...}) = 0
getpid()                                = 18780
lstat("/tmp/file12YHdh", 0x7ffd8bf8a1d0) = -1 ENOENT (No such file or directory)
stat("/tmp/nrfjprogdll/highlevel/file12YHdh", 0x7ffd8bf8a230) = -1 ENOENT (No such file or directory)
stat("/tmp/nrfjprogdll/highlevel/file12YHdh.lock", 0x7ffd8bf8a120) = -1 ENOENT (No such file or directory)
stat("/tmp/nrfjprogdll/highlevel/file12YHdh.lock", 0x7ffd8bf8a070) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/tmp/nrfjprogdll/highlevel/file12YHdh.lock", O_WRONLY|O_CREAT|O_TRUNC, 0666) = 3
stat("/tmp/nrfjprogdll/highlevel/file12YHdh", 0x7ffd8bf89e90) = -1 ENOENT (No such file or directory)
stat("/", {st_mode=S_IFDIR|0755, st_size=178, ...}) = 0
stat("/tmp/nrfjprogdll/highlevel/file12YHdh", 0x7ffd8bf8a230) = -1 ENOENT (No such file or directory)
stat("/tmp/nrfjprogdll/highlevel/file12YHdh", 0x7ffd8bf8a230) = -1 ENOENT (No such file or directory)
stat("/tmp/nrfjprogdll/highlevel/file12YHdh", 0x7ffd8bf8a230) = -1 ENOENT (No such file or directory)
stat("/tmp/nrfjprogdll/highlevel/file12YHdh", 0x7ffd8bf8a230) = -1 ENOENT (No such file or directory)
stat("/tmp/nrfjprogdll/highlevel/file12YHdh", 0x7ffd8bf8a230) = -1 ENOENT (No such file or directory)
stat("/tmp/nrfjprogdll/highlevel/file12YHdh", 0x7ffd8bf8a230) = -1 ENOENT (No such file or directory)
stat("/tmp/nrfjprogdll/highlevel/file12YHdh", 0x7ffd8bf8a230) = -1 ENOENT (No such file or directory)
stat("/tmp/nrfjprogdll/highlevel/file12YHdh", 0x7ffd8bf8a230) = -1 ENOENT (No such file or directory)
stat("/tmp/nrfjprogdll/highlevel/file12YHdh", 0x7ffd8bf8a230) = -1 ENOENT (No such file or directory)
stat("/tmp/nrfjprogdll/highlevel/file12YHdh", 0x7ffd8bf8a230) = -1 ENOENT (No such file or directory)
stat("/tmp/nrfjprogdll/highlevel/file12YHdh", 0x7ffd8bf8a230) = -1 ENOENT (No such file or directory)
stat("/tmp/nrfjprogdll/highlevel/file12YHdh", 0x7ffd8bf8a230) = -1 ENOENT (No such file or directory)
stat("/tmp/nrfjprogdll/highlevel/file12YHdh", 0x7ffd8bf8a230) = -1 ENOENT (No such file or directory)
stat("/tmp/nrfjprogdll/highlevel/file12YHdh", 0x7ffd8bf8a230) = -1 ENOENT (No such file or directory)
stat("/tmp/nrfjprogdll/highlevel/file12YHdh", 0x7ffd8bf8a230) = -1 ENOENT (No such file or directory)
stat("/tmp/nrfjprogdll/highlevel/file12YHdh", 0x7ffd8bf8a230) = -1 ENOENT (No such file or directory)
stat("/tmp/nrfjprogdll/highlevel/file12YHdh", 0x7ffd8bf8a230) = -1 ENOENT (No such file or directory)

And repeats indefinitely until killed.

Where is this "/tmp/nrfjprogdll/highlevel/file12YHdh" file supposed to come from, and why does the python library busy loop on checking if it exists?