NRFUTIL: Trace utility is not stable

[vytautas]

I'm experiencing frequent crashes of the trace utility.

nrfutil trace lte --input-serialport /dev/ttyACM1 --output-pcapng wireshark-$(date +%s).pcapng
⠐ Saving trace to /home/<REMOVED>/wireshark-1727275519.pcapng (16236 bytes)
Error: Subprocess /home/<REMOVED>/.nrfutil/bin/nrfutil-trace failed with unexpected exit code None

nrfutil-trace 2.11.0 (8fa6588 2024-09-18)
commit-hash: 8fa6588c66efdcebc2676bf7c393cb5036184f7f
commit-date: 2024-09-18
host: x86_64-unknown-linux-gnu
build-timestamp: 2024-09-19T06:25:28.763301537Z
classification: nrf-external

I understand the information provided is not sufficient to root cause the issue.

Perhaps there is a combination of options that can allow me to collect more usable logs?

[Øyvind]

Hello, 

I've tested on my macbook and another linux device, and was not able reproduce. Have you updated you nrfutil?

KInd regards,
Øyvind

[vytautas]

I recently installed the trace and device utilities.

nrfutil-trace is already the newest version (2.11.0)
nrfutil-device is already the newest version (2.5.4)
[00:00:01] ###### 100% [Upgrade packages] Upgrade packages

I don't think I've updated nrfutil itself. Is it was of the Nordic command line tools package?

For some additional context, it doesn't crash immediately. Here's an example:

-rw-r--r--  1 <REDACTED> <REDACTED> 8.5K Sep 26 14:57 wireshark-1727351557.pcapng
-rw-r--r--  1 <REDACTED> <REDACTED>  36K Sep 26 15:52 wireshark-1727352383.pcapng
-rw-r--r--  1 <REDACTED> <REDACTED> 2.8K Sep 26 16:37 wireshark-1727356319.pcapng
-rw-r--r--  1 <REDACTED> <REDACTED>  14K Sep 26 17:26 wireshark-1727357939.pcapng
-rw-r--r--  1 <REDACTED> <REDACTED> 4.8K Sep 26 18:22 wireshark-1727360906.pcapng
-rw-r--r--  1 <REDACTED> <REDACTED> 1.6K Sep 26 18:50 wireshark-1727364236.pcapng
-rw-r--r--  1 <REDACTED> <REDACTED>  18K Sep 26 20:05 wireshark-1727365903.pcapng
-rw-r--r--  1 <REDACTED> <REDACTED>  480 Sep 26 20:17 wireshark-1727370429.pcapng
-rw-r--r--  1 <REDACTED> <REDACTED> 1.1K Sep 26 20:27 wireshark-1727371105.pcapng
-rw-r--r--  1 <REDACTED> <REDACTED>  20K Sep 26 21:26 wireshark-1727371694.pcapng
-rw-r--r--  1 <REDACTED> <REDACTED>  792 Sep 26 21:38 wireshark-1727375313.pcapng
-rw-r--r--  1 <REDACTED> <REDACTED>  608 Sep 26 21:59 wireshark-1727375962.pcapng
-rw-r--r--  1 <REDACTED> <REDACTED>  400 Sep 26 22:03 wireshark-1727377168.pcapng
-rw-r--r--  1 <REDACTED> <REDACTED>    0 Sep 26 22:03 wireshark-1727377413.pcapng
-rw-r--r--  1 <REDACTED> <REDACTED>  412 Sep 26 22:09 wireshark-1727377496.pcapng
-rw-r--r--  1 <REDACTED> <REDACTED>  284 Sep 26 22:13 wireshark-1727377823.pcapng
-rw-r--r--  1 <REDACTED> <REDACTED>  160 Sep 26 22:15 wireshark-1727378049.pcapng
-rw-r--r--  1 <REDACTED> <REDACTED>  416 Sep 26 22:20 wireshark-1727378151.pcapng
-rw-r--r--  1 <REDACTED> <REDACTED>  160 Sep 26 22:21 wireshark-1727378479.pcapng
-rw-r--r--  1 <REDACTED> <REDACTED> 2.4K Sep 26 22:44 wireshark-1727378638.pcapng
-rw-r--r--  1 <REDACTED> <REDACTED>  288 Sep 26 22:50 wireshark-1727379871.pcapng
-rw-r--r--  1 <REDACTED> <REDACTED>    0 Sep 26 22:51 wireshark-1727380286.pcapng
-rw-r--r--  1 <REDACTED> <REDACTED> 7.5K Sep 27 00:13 wireshark-1727380310.pcapng
-rw-r--r--  1 <REDACTED> <REDACTED>  22K Sep 27 01:42 wireshark-1727385249.pcapng
-rw-r--r--  1 <REDACTED> <REDACTED> 8.4K Sep 27 02:36 wireshark-1727390577.pcapng
-rw-r--r--  1 <REDACTED> <REDACTED> 2.9K Sep 27 02:46 wireshark-1727393853.pcapng
-rw-r--r--  1 <REDACTED> <REDACTED>  28K Sep 27 03:44 wireshark-1727394442.pcapng
-rw-r--r--  1 <REDACTED> <REDACTED> 2.8K Sep 27 04:19 wireshark-1727397999.pcapng
-rw-r--r--  1 <REDACTED> <REDACTED>    0 Sep 27 04:20 wireshark-1727400040.pcapng
-rw-r--r--  1 <REDACTED> <REDACTED> 9.0K Sep 27 05:14 wireshark-1727400065.pcapng
-rw-r--r--  1 <REDACTED> <REDACTED> 2.2K Sep 27 05:20 wireshark-1727403352.pcapng
-rw-r--r--  1 <REDACTED> <REDACTED> 1.1K Sep 27 05:25 wireshark-1727403679.pcapng
-rw-r--r--  1 <REDACTED> <REDACTED>  916 Sep 27 05:31 wireshark-1727404007.pcapng
-rw-r--r--  1 <REDACTED> <REDACTED>  924 Sep 27 05:37 wireshark-1727404335.pcapng
-rw-r--r--  1 <REDACTED> <REDACTED> 4.4K Sep 27 06:13 wireshark-1727404662.pcapng
-rw-r--r--  1 <REDACTED> <REDACTED> 1.1K Sep 27 06:23 wireshark-1727406847.pcapng
-rw-r--r--  1 <REDACTED> <REDACTED> 4.5K Sep 27 07:25 wireshark-1727407437.pcapng
-rw-r--r--  1 <REDACTED> <REDACTED> 5.6K Sep 27 08:10 wireshark-1727411249.pcapng
-rw-r--r--  1 <REDACTED> <REDACTED> 7.5K Sep 27 09:13 wireshark-1727413913.pcapng

I ran the command in a while loop. Whenever it crashed, it would immediately create a new trace. You can look up the file creation times above to understand the frequency a bit better.

Could you post how you ran the trace utility? I can try running it the same way.

[Øyvind]

I used the command you provided. i was able to run it for a while, collecting 18k of data. Have not heard of this issue before, so will need to investigate more. 

Here is my version

oysa@NordicTechSupportM2 ~ % nrfutil --version 
nrfutil 7.13.0 (8289424 2024-07-01)
commit-hash: 82894242d19ff24a1541712312b3ea3af0ca8f85
commit-date: 2024-07-01
host: aarch64-apple-darwin
build-timestamp: 2024-07-01T07:34:47.247880000Z
classification: nrf-external

oysa@NordicTechSupportM2 ~ % nrfutil trace --version
nrfutil-trace 2.11.0 (8fa6588 2024-09-18)
commit-hash: 8fa6588c66efdcebc2676bf7c393cb5036184f7f
commit-date: 2024-09-18
host: aarch64-apple-darwin
build-timestamp: 2024-09-19T06:23:01.954159000Z
classification: nrf-external

Could you try running with --log-level debug i.e. 

nrfutil trace lte --log-level debug --input-serialport /dev/ttyACM1 --output-pcapng wireshark-$(date +%s).pcapng

[vytautas]

Currently only 7.11.0 version is available for download for Linux (Running Debian)

https://docs.nordicsemi.com/bundle/nrfutil/page/guides/installing.html

https://www.nordicsemi.com/Products/Development-tools/nRF-Util

The trace utility version is the same.

Where are the debug logs stored? I'm not getting them in stdout.

[Øyvind]

Hi, I had a colleague test one more time with linux, and there does seem to be an issue with nrfutil. You should be able to collect modem traces through cellular monitor while waiting for a fix.

Kind regards,
Øyvind

[vytautas]

Understood. Thanks for the information.

Please let me know when a fix is available.