gdb commands to list Zephyr app threads?

[tedhavelka]

Hello Devzone Community,

I am developing an nRF9160 based firmware for a battery powered device, and am attempting to examine the firmware using `gdb` at the command line.  I have a JLink debugger from Segger as my physical debugging and programming device.  I have Segger's JLink utilities installed on an Ubuntu 20.04 LTS host.  The JLinkGDBServer version is 7.60.

My firmware is based on Nordic Semi ncs v1.6.1, and also has some basis yet specifically on Nordic's sample app `aws_iot`.  While the aws_iot sample app is not designed to automatically enter nRF9160's lowest power modes for a particular time period, my understanding is that all the low power modes and features of the nRF9160 are achievable with proper amendments in firmware to this sample app.

My present goal is to see a complete listing of Zephyr RTOS and application (my specific app code) threads in a remote gdb debugging session.  I'm able to start Segger's JLinkGDBServer and the GNU cross debugger at the command line, and interact some with my nRF9160 + Zephyr firmware.  I also have Zephyr's thread analyzer module enabled in my project Kconfig settings in file prj.conf.  But I get a disagreement between the threads that Zephyr RTOS 2.6.0 reports and what threads gdb with Segger's Zephyr RTOS plugin sees.

Excerpt 1 - Zephyr thread analyzer reported threads:

   1) 'at_cmd_socket_thread'     stack size 1472 bytes, stack used 432 bytes, 29%
   2) 'thread_simple_cli'        stack size 4096 bytes, stack used 1768 bytes, 43%
   3) 'time_thread'              stack size 1024 bytes, stack used 240 bytes, 23%
   4) 'sysworkq'                 stack size 2048 bytes, stack used 168 bytes, 8%
   5) 'idle 00'                  stack size 320 bytes, stack used 56 bytes, 17%

Excerpt 2 - gdb reported threads:

(gdb) thread find [a-z0-9]
Thread 1 has target id 'Thread 57005'
Thread 2 has target id 'Remote target'

At gdb's prompt I have also tried the command which is mentioned in this Memfault dot com gdb tutorial.  There's a lot of gdb I have yet to learn, but the following command appears to be a shorthand way to request a backtrace of all threads in the remote application.

Excerpt 3 - gdb command to show backtrace of threads:

(gdb) thread apply all bt
[New Remote target]

Thread 2 (Remote target):
#0  z_arm_reset () at /home/ted/projects/zephyr-based/zephyr/arch/arm/core/aarch32/cortex_m/reset.S:69

Thread 1 (Thread 57005):
#0  z_arm_reset () at /home/ted/projects/zephyr-based/zephyr/arch/arm/core/aarch32/cortex_m/reset.S:69

This listing of backtraces however is short lived.  In a given gdb session, the second and later times I invoke this command the report gives only Thread 2 has extent and having a backtrace to report.

Question (1)  Is there a sequence of debugging commands I must issue, to put firmware into a state where the debugger can correctly report Zephyr and app threads?

Question (2)  should Segger's Zephyr RTOS plugin shared object file support gdb to list the same threads which Zephyr itself reports?

- Ted

[tedhavelka]

Hello Hakon,

Thank you for explaining the image upload dialog.  I'm pretty sure I've tried pressing that "Upload" button, and not been able to upload a file, but I will try again soon.

As for our nRF9160DK board version, here is what the Nordic specific `at%hwversion` returns:

modem reply: %HWVERSION: nRF9160 SICA B1A

It would be difficult for me to share the full firmware on my side which draws about 300uA down to 266uA, as it is work related.  I've been attempting to reach the nRF9160 lowest current draw specification by a couple of routes:  modifying my larger work firmware project, e.g. commenting out call to nRF library routine `at_configure()`, along with M33 GPIO configurations and suspending of UART devices.  I also modified a "hello world" sample from your colleague Didrik Rokhaug.  It is that modified "hello world" with CONFIG_NRF_MODEM_LIB=y, CONFIG_SERIAL=n and a few other symbols which draws ~50nA, far too little current for me to trust my physical set up.

I will attempt to zip and attach the modified, low power "hello world" which builds.  My hope with this project from your team has been that, when I get it to behave and show the ~1.4 to 1.8uA current draw, I could configure my larger work related firmware to enable those and only those Kconfig options.

After a first effort in this, I am unsure I can retain even half the functionality of Nordic's aws_iot sample app, upon which my larger work related project firmware is based.  As I disable certain Kconfig symbols related to networking and MQTT, my compilations fail.  I am taking notes of my findings, but it is not yet clear to me how much I'll need to adapt or requite some of the network and protocol nRF library routines for the AWS client code and MQTT API in Zephyr source tree to function while allowing those lowest current draw options.

hello-world-low-power-0311.zip

Ok, got the zip file to upload.  The posting software plug-in -- at least in Firefox on a Linux host -- does not accept .tar archives.

So the attached file does not quite match the hello_world samples shared by Didrik in post 85747 but the source is largely the same, and the prj.conf settings are effectively the same.  For some reason, the later two of Didrik's three Zephyr sample apps failed to build on two Ubuntu workstations of mine.  `cmake` quickly enters an endless loop, processing only the Kconfig symbols from what I can tell, then cycling back to "regenerate...".

Tomorrow I will post a couple of photos of my physical connections from PPK II to nRF9160DK.  I will ask whether my current measuring set up appears correct for an "Ampere mode" type of measurement.

Thank you again for your help.  Hope to get to the bottom of this challenging issue and bring our nRF9160 to or within a factor of two of its lowest advertised current consumption!

- Ted

[Håkon Alseth]

Hi Ted,

 

Thanks for providing test software!

I added three things to your project:

1. A blinking LED (P0.02)

2. prj.conf::CONFIG_NRF_MODEM_LIB_SYS_INIT=y

3. child_image/spm.conf (the secure image), holding "CONFIG_SERIAL=n"

 

Project for reference:

286558.zip

 

I saw that a comment from Didrik was that you should enable the nrf_modem_lib, but it is also important that its enabled on sys_init (ie. "in the background" before you enter main()).

 

Here's the current profile from PPK2 (from boot-up):

 

If you connect the JLinkRTTViewer (select nrf9160_xxaa as device), you will also see the logging sequence:

There's a bit of congestion, as the log buffer is being filled up, but other than that, it prints just fine.

 

Please note: with the RTT prints running, the debugger is attached, and the DUT current consumption will be higher! I measure around 5.7 mA at my end, which will drop to the above uA level if you power cycle the nRF (from PPK2 software).

 

Kind regards,

Håkon

[tedhavelka]

Hello Håkon,

You have answered two big questions for me, and you may close this ticket now if that's appropriate.  Your first reply dated about 2022 March 28, in which you outline how to open two terminal windows and invoke gdb server in one, then gdb in the other produces good results on my end.  I can now see expected Zephyr application threads by setting a reasonable break point, reaching it, then entering in gdb client `info threads`.

You also provided the final configuration piece I needed to demo to my team the deep sleep, low current use of the SiP in an nRF9160DK board.  After our latest exchange I found that symbol assignment `CONFIG_NRF_MODEM_LIB_SYS_INIT=y` was the only additional thing I needed in the sample app I posted here as a zip archive a few days ago.  Here is the amended sample app, based on Didrik Rokhaug's and your code examples, which I can compile and run:

   zephyr-nrf9160dk-low-power-app.zip

On my custom hardware and Zephyr based app I am still measuring higher current, in the tens of microamps.  But I now have "living proof" that we can configure the nRF9160 itself for a very low current.  I just need to adapt our project Kconfig settings, and likely some of our run time firmware behavior, to be able to enter this low low current use mode while still allowing some of the aws_iot based sample code to compile and run periodically.

Thank you again for your thorough and rapid help, and to Didrik Rokhaug too!

- Ted

[Håkon Alseth]

Hi Ted,

 

I am very glad to hear this! I'll ofcourse bring your feedback to Didrik as well!

tedhavelka said:

On my custom hardware and Zephyr based app I am still measuring higher current, in the tens of microamps.  But I now have "living proof" that we can configure the nRF9160 itself for a very low current.  I just need to adapt our project Kconfig settings, and likely some of our run time firmware behavior, to be able to enter this low low current use mode while still allowing some of the aws_iot based sample code to compile and run periodically.

If you are measuring high current, I can recommend that you switch from "ampere meter" to source meter, as this one is usually more accurate in the lower micro-amp area. Here's what I measure with your example, where the higher pulses is the startup-sequence:

 Here's a picture on how you setup source meter mode with the nRF9160-DK:

https://infocenter.nordicsemi.com/topic/ug_ppk2/UG/ppk/measure_current_source_meter.html?cp=10_8_5_1

 

Hope you have a wonderful weekend, Ted!

 

Kind regards,

Håkon