Flushing the LTE modem trace buffer

RE: Flushing the LTE modem trace buffer

JONATHAN LL — Tue, 02 Sep 2025 23:56:32 GMT

Hi Peter,

[quote user="Neal Riedel"]It seems to be sending data to the host PC since I can observe the serial data line toggling on an oscilloscope. However, the nrfutil trace application reports "0 bytes".[/quote]

Where do you inspect with the oscilloscope? Between the device and board controller, or between the board controller and computer?

Do you have access to an external UART to serial bridge to try with (connected where you now have the oscilloscope)?

[quote user="Peter Murphy"]The results seem to suggest that running the nrfutil trace lte ... command at the linux command line sometime behaves differently than when using the Cellular Monitor application to capture modem traces.[/quote]

Will check with the developers of the tool here and get back to you.

Regards,
Jonathan

RE: Flushing the LTE modem trace buffer

Peter Murphy — Tue, 02 Sep 2025 23:00:48 GMT

Hi Jonathan,

We re-ran our tests using the Cellular Monitor v2.4.4 application and don't see the issue of the trace files sometimes being empty. The tests were carried out on the same linux computer used to generate the results detailed earlier. In addition, the same application code was installed on the 9151DK. The results seem to suggest that running the nrfutil trace lte ... command at the linux command line sometime behaves differently than when using the Cellular Monitor application to capture modem traces. Using the Cellular Monitor application is not an option for us because we're doing things in a scripted test automation framework.

Regards,

Peter

RE: Flushing the LTE modem trace buffer

JONATHAN LL — Thu, 21 Aug 2025 21:45:32 GMT

[quote user="Neal Riedel"]We are not using RTT, we are using the UART backend[/quote]

It was a possible partial explanation on why the data was gone sometimes, if RTT was used it could cause data being dropped. But since you are not using RTT this is not the likely case. But data would be droppede either way with HWFC on or off. Its just if a response is created or not when the data is dropped.

Will try to find an explanation for this, but do you have a logic analyzer, or can you check the data that is being transmitted?

Regards,
Jonathan

RE: Flushing the LTE modem trace buffer

Neal Riedel — Thu, 21 Aug 2025 21:09:47 GMT

I don't understand. We are not using RTT, we are using the UART backend. It is connected to uart1. THe behavior seems to be the same with and without HW flow control.

It seems to be sending data to the host PC since I can observe the serial data line toggling on an oscilloscope. However, the nrfutil trace application reports "0 bytes".

RE: Flushing the LTE modem trace buffer

JONATHAN LL — Thu, 21 Aug 2025 20:34:10 GMT

The HWFC is not the issue, enabled or not the difference would be if the trace data is silently dropped or that the dropped data is registered as a dropped event. But the trace data is gone either way.

This happens if RTT is used and since RTT holdes priority it will take over while the buffer is filled up, if the buffer is full the trace data will be dropped

Having HWFC enabled does not enable a greater risk of missing traces if RTT is on, though it would be more visible in the binary trace than if HWFC is disabled.

An alterative route is to use other pins for for UART and use UART to serial and get the data that way. But there can be issues here as baudrate used is 1 000 000.

Regards,
Jonathan

RE: Flushing the LTE modem trace buffer

Neal Riedel — Thu, 21 Aug 2025 17:50:38 GMT

Thanks, Jonathan.

I tried disabling the HW flow control on uart1, but it had no effect on this problem.

I also connected a scope to the uart1 TXD pin and monitoring the activity when the problem occurs. The nrfutil trace command displays this:

Saving trace to trace_log3.pcapng (0 bytes)

However I can clearly see activity on the TXD pin, indicating that the 9151 is sending data to the nrfutil application. So I wonder if the trace data is actually being queue in the linux PC?

RE: Flushing the LTE modem trace buffer

JONATHAN LL — Wed, 20 Aug 2025 23:52:15 GMT

Hi Neal,

This looks ok.

[quote user="Neal Riedel"]We do have HW flow control set on this port. Could that be the issue?[/quote]

Still checking with the devz and discussing here what could be the reason.

Regards,
Jonathan

RE: Flushing the LTE modem trace buffer

Neal Riedel — Fri, 15 Aug 2025 18:13:45 GMT

Hi Jonathan,

I'm the guy writing the software. ;)

We use the nrf91-modem-trace-uart snippet in the build. Double-checking the generated zephyr.dts file, it has this:

        chosen {
                zephyr,entropy = &psa_rng;
                zephyr,flash-controller = &flash_controller;
                zephyr,console = &uart0;
                zephyr,shell-uart = &uart0;
                zephyr,uart-mcumgr = &uart0;
                zephyr,flash = &flash0;
                zephyr,sram = &sram0_ns_app;
                zephyr,code-partition = &slot0_ns_partition;
                nordic,pm-ext-flash = &gd25wb256;
                nordic,modem-trace-uart = &uart1;
        };

And also this:

                        uart1: arduino_serial: uart@9000 {
                                compatible = "nordic,nrf-uarte";
                                reg = < 0x9000 0x1000 >;
                                interrupts = < 0x9 0x1 >;
                                status = "okay";
                                current-speed = < 0xf4240 >;
                                pinctrl-0 = < &uart1_default >;
                                pinctrl-1 = < &uart1_sleep >;
                                pinctrl-names = "default", "sleep";
                                hw-flow-control;
                        };

So I think it is correct. We do have HW flow control set on this port. Could that be the issue?

Thanks,

Neal Riedel

RE: Flushing the LTE modem trace buffer

JONATHAN LL — Fri, 15 Aug 2025 11:36:41 GMT

Update from devz:

Do they have the Device Tree overlay as given in the nrf91-modem-trace-uart snippet?

/* Enable uart1 for tracing. */
&uart1 {
status = "okay";
current-speed = < 1000000 >;
};

/ {
chosen {
nordic,modem-trace-uart = &uart1;
};
};

If not, modem will likely drop a lot of trace messages because of the uart baud rate. Not sure if that can explain the PCAP being completely blank though. Enabling HWFC would help on loosing traces in the board controller if RTT is enabled, as RTT has priority.

Regards,
Jonathan

RE: Flushing the LTE modem trace buffer

JONATHAN LL — Thu, 14 Aug 2025 08:38:15 GMT

Hi Peter,

Looking in to this and have forwarded the question to our devz, will update.

But the documentation says that the modem buffers the data.

The trace backend needs to handle trace data at ~1 Mbps to avoid filling up the buffer in the modem. If the modem buffer is full, the modem drops modem traces until the buffer has space available again.

Ref: https://docs.nordicsemi.com/bundle/ncs-latest/page/nrf/libraries/modem/nrf_modem_lib/nrf_modem_lib_trace.html#modem_trace_flash_backend

Regards,
Jonathan