Capturing Modem Trace on ARM System

[ibiglari]

Hi Team!

I need to capture a modem trace over a long period of time while driving around, and there is no way my laptop's battery can handle that. I thought I'd use a Raspberry Pi and a big power bank, but unfortunately nRF Toolkit doesn't work on ARM based Linux systems. Is it possible to capture the raw modem trace on a system and then feed it to trace collector?

Kind Regards,
Iman Biglari

[Håkon Alseth]

Hi,

 

Unfortunately, the nRF Connect for desktop application does not run on ARM Linux.

However, the modem trace functionality only opens up a communication channel over uart.

You can open up the uart port with any terminal program baud=1M and store the content to a file, but I would recommend that you test this solution over time to ensure that there's no issues.

 

Kind regards,

Håkon

[Achim Kraus]

Let me start, that in my experience this topic is extremely important!

For the very most issues here in this forum, Nordic asks for traces.

(From my view point, that is more a general way to process issues, asking something back.)

But, if Nordic wants trace from the wild or long-term scenarios, using a PC doesn't work. Using a Raspi is a much better approach. I'm not sure, if something as OTG Card Readers will work at all with the USB port of a Thingy:91, but it would take anyway long to be available.

Or, what did Nordic use?

(Sometimes, reading all the issues here in the forum, I have to confess, that I start to belief, Nordic doesn't trace too much in the wild or for long terms ;-) .)

So, again, as sum-up:

If Nordic wants traces for issues from the wild or long-term scenarios, a PC is not that useful.

Using a Raspi with sd-card and USB powerbank is also in my experience the "shortest way to go".  

So:

Are there any other preconditions, which must be considered?

E.g. "Framing"?

[ibiglari]

I could share the small application and script I wrote for RPi, but Nordic advised me something was "wrong" with the traces I had captured (didn't specify what exactly - More information here).

[Håkon Alseth]

We are continuously trying to improve our deliveries, and tracing has been a front that we've tried to improve upon. Lately with this PR:

https://github.com/nrfconnect/sdk-nrf/pull/9364

 

I'll relay your feedback internally.

 

Kind regards,

Håkon

[Achim Kraus]

How much of your users will be able to use a external flash on their own with the provided documentation? We will see ...

Edited:

Hm, that's only for the nRF9160-DK with the "external Flash" already on board, right?

It doesn't work for a Thingy:91 "out-of-the-box", there a really external Flash would be required?

To use a nRF9160-DK, USB power bank fails (it's the power bank, which switches off), but using an "external battery" works. So the setup for the wild will be a box with the nRF9160-DK and a battery.

1 MB for traces is not too large. Is it considered to be used as "ring buffer", means it keeps the last 1 MB?

[Achim Kraus]

> Nordic advised me something was "wrong" with the traces

Yep, that information, what's wrong, would be required in order to continue with that idea.

[AlanIoT]

Sorry to revive this thread after so much time, but is there any chance you could share the script or maybe even any new methods you found? We've captured a bin file, but so far it isn't decoded / translated properly into wireshark. Any hints would be very helpful, thanks :)

[Achim Kraus]

AFAIK, Nordic "always" asks about modem traces, but I'm not aware, that this long lasting issue about consolidating "long term modem traces" has been solved. I'm not even sure, if it's possible at all to read such long traces, if it's more than a few hours and MBs.

In the meantime (I use NCS 2.9.2), the modem trace function comes with some more options. Especially the one using an external flash is pretty nice. It records the trace and to read it out, it just writes that to the UART and so it's possible to read it with the cellular monitor.

If you want to use it with other boards than the DKs, you may need to add your boards to the "nrf/snippets/nrf91-modem-trace-ext-flash". One pitfall doing so may be the "CONFIG_SPI_NOR_FLASH_LAYOUT_PAGE_SIZE" and the "CONFIG_NRF_MODEM_LIB_TRACE_FLASH_SECTORS" which both must fit to the flash size.

But it requires an external flash and is also limited by the size of the flash and the selected level.

[AlanIoT]

Hi Achim

I think one of the more plausible and flexible ways to extract valid modem traces in the field with minimal effort is the solution suggested by ibiglari. To cache / intermediately store the trace bit by bit on an external device via UART (same process you'd use to get it in the lab, but mobile) and then convert it / play it back to a nrfutil compatible System at a later time. But I agree with you, I can't find a cohesive and well documented way of achieving this. I'm sure we'll figure it out soon, but some documentation and a sample to help more easily gather modem traces from field devices would be greatly appreciated. I'll try to make a post / guide as soon as I've figured out a way to reliably get the trace collection via ARM-device working.

[AlanIoT]

For those who come after:

the process is very simple in the end:

1. On the ARM device, you need an FTDI cable capable of 1Mbaud communication.

2. Plug this cable into one of the USB ports in the ARM device

3. capture the raw input from the serial connection that opened when plugging in the cable. I recommend going as simple as possible; the following line worked for me:

configure the serial device:
stty -F {device_path} {baud_rate} raw -clocal -crtscts
save the modem trace coming through the serial connection to your log file:
cat {device_path} > {log_file}

example:
stty -F /dev/ttyUSB0 1000000 raw -clocal -crtscts
cat /dev/ttyUSB0 > filename.bin

4. then copy the .bin file to a non-ARM platform of your choice.

5. Make sure you have nrfutil installed (download, move to path, make executable)

6. make sure you have the correct plugin installed:

nrfutil install trace

7. convert the .bin file into a pcapng file using the nrfutil trace command:

nrfutil trace lte --input-file /path/to/filename.bin --output-pcapng /path/to/wireshark_capture.pcapng

8. Done. The trace can now be analysed as a wireshark capture.

hope this helps. variations and other complementary info can be found in the nrfutil documentation