gdb commands to list Zephyr app threads?

[tedhavelka]

Hello Devzone Community,

I am developing an nRF9160 based firmware for a battery powered device, and am attempting to examine the firmware using `gdb` at the command line.  I have a JLink debugger from Segger as my physical debugging and programming device.  I have Segger's JLink utilities installed on an Ubuntu 20.04 LTS host.  The JLinkGDBServer version is 7.60.

My firmware is based on Nordic Semi ncs v1.6.1, and also has some basis yet specifically on Nordic's sample app `aws_iot`.  While the aws_iot sample app is not designed to automatically enter nRF9160's lowest power modes for a particular time period, my understanding is that all the low power modes and features of the nRF9160 are achievable with proper amendments in firmware to this sample app.

My present goal is to see a complete listing of Zephyr RTOS and application (my specific app code) threads in a remote gdb debugging session.  I'm able to start Segger's JLinkGDBServer and the GNU cross debugger at the command line, and interact some with my nRF9160 + Zephyr firmware.  I also have Zephyr's thread analyzer module enabled in my project Kconfig settings in file prj.conf.  But I get a disagreement between the threads that Zephyr RTOS 2.6.0 reports and what threads gdb with Segger's Zephyr RTOS plugin sees.

Excerpt 1 - Zephyr thread analyzer reported threads:

   1) 'at_cmd_socket_thread'     stack size 1472 bytes, stack used 432 bytes, 29%
   2) 'thread_simple_cli'        stack size 4096 bytes, stack used 1768 bytes, 43%
   3) 'time_thread'              stack size 1024 bytes, stack used 240 bytes, 23%
   4) 'sysworkq'                 stack size 2048 bytes, stack used 168 bytes, 8%
   5) 'idle 00'                  stack size 320 bytes, stack used 56 bytes, 17%

Excerpt 2 - gdb reported threads:

(gdb) thread find [a-z0-9]
Thread 1 has target id 'Thread 57005'
Thread 2 has target id 'Remote target'

At gdb's prompt I have also tried the command which is mentioned in this Memfault dot com gdb tutorial.  There's a lot of gdb I have yet to learn, but the following command appears to be a shorthand way to request a backtrace of all threads in the remote application.

Excerpt 3 - gdb command to show backtrace of threads:

(gdb) thread apply all bt
[New Remote target]

Thread 2 (Remote target):
#0  z_arm_reset () at /home/ted/projects/zephyr-based/zephyr/arch/arm/core/aarch32/cortex_m/reset.S:69

Thread 1 (Thread 57005):
#0  z_arm_reset () at /home/ted/projects/zephyr-based/zephyr/arch/arm/core/aarch32/cortex_m/reset.S:69

This listing of backtraces however is short lived.  In a given gdb session, the second and later times I invoke this command the report gives only Thread 2 has extent and having a backtrace to report.

Question (1)  Is there a sequence of debugging commands I must issue, to put firmware into a state where the debugger can correctly report Zephyr and app threads?

Question (2)  should Segger's Zephyr RTOS plugin shared object file support gdb to list the same threads which Zephyr itself reports?

- Ted

[tedhavelka]

Good evening Håkon,

Ted from the west coast writing.  I want to give you a thank you for these specific, spot on instructions to invoke gdb server, and arm-none-eabi-gdb client.  I've just gotten this to work and can finally see correct thread summary of running Nordic ncs, Zephyr based application.  Thank you!

My apologies as well, I have been feverishly ( at times sloggingly ) working through coding and Kconfig modifications in order to correctly fully enter deep sleep mode with our nRF9160 based board design.  I have a longer running post regarding nRF9160 deep sleep configuration.  I have yet to solve this problem, but your team mate Didrik Rokhaug has provided some low power sample apps which build for the nRF9160DK.  I have one of these development boards on hand.  LTE modem firmware reports `%HWVERSION: nRF9160 SICA B1A` for our version of the nRF9160.

Simple as these hello_world applications are it has taken some time to reach a build-able sample.  I needed to combine two of the samples to achieve what seems an unrealistic 46 nano-amps current draw from the nRF9160 SiP itself.  As I understand it, using PPK 2 connected to the DK board's P22 header measures current of the SiP only, not the external hardware.

The work project I am developing on the firmware side is more complex.  Unfortunately I cannot make a simple symbol for symbol comparison of Didrik's low power hello world prj.conf against the work-related prj.conf file.  Closest I can get is to compare assignments of like symbols, then note the many extra symbols in the work project.  I am now reviewing each extra symbol -- many related to Nordic ncs aws_iot sample app -- one symbol at a time.  I am attempting to set each one to 'n'.  There are times this breaks the build, but I am trying to bring the work project Kconfig to match the hello_world low power config.  If I can match them, I should see low power on my custom board, or know that I've failed to turn off custom hardware outside the 9160.

When I connect Nordic PPK2 device to nRF9160DK board in ampere meter mode, is it reasonable to see only tens of nano-amps as the SiP operates in deep sleep mode?  Or is this current measurement a sign that I have a mistake in my test configuration?

My apology for changing topic question in my post reply.  Thank you again for the significant help with gdb server and Zephyr thread debugging!

- Ted

[Håkon Alseth]

Hi Ted,

 

tedhavelka said:

Ted from the west coast writing.  I want to give you a thank you for these specific, spot on instructions to invoke gdb server, and arm-none-eabi-gdb client.  I've just gotten this to work and can finally see correct thread summary of running Nordic ncs, Zephyr based application.  Thank you!

I am glad to have helped out!

tedhavelka said:

Simple as these hello_world applications are it has taken some time to reach a build-able sample.  I needed to combine two of the samples to achieve what seems an unrealistic 46 nano-amps current draw from the nRF9160 SiP itself.  As I understand it, using PPK 2 connected to the DK board's P22 header measures current of the SiP only, not the external hardware.

The work project I am developing on the firmware side is more complex.  Unfortunately I cannot make a simple symbol for symbol comparison of Didrik's low power hello world prj.conf against the work-related prj.conf file.  Closest I can get is to compare assignments of like symbols, then note the many extra symbols in the work project.  I am now reviewing each extra symbol -- many related to Nordic ncs aws_iot sample app -- one symbol at a time.  I am attempting to set each one to 'n'.  There are times this breaks the build, but I am trying to bring the work project Kconfig to match the hello_world low power config.  If I can match them, I should see low power on my custom board, or know that I've failed to turn off custom hardware outside the 9160.

When I connect Nordic PPK2 device to nRF9160DK board in ampere meter mode, is it reasonable to see only tens of nano-amps as the SiP operates in deep sleep mode?  Or is this current measurement a sign that I have a mistake in my test configuration?

My apology for changing topic question in my post reply.  Thank you again for the significant help with gdb server and Zephyr thread debugging!

The PPK2 has a resolution of nano-meter in the lowest mode, as shown here:

https://infocenter.nordicsemi.com/topic/ug_ppk2/UG/ppk/ppk_measure_resolution.html?cp=10_8_7_2_1

However, it will not be able to measure consistently in the nano-amp range.

 

Here's the numbers (ballpark) which you should see when the nRF9160 is in sleep:

https://infocenter.nordicsemi.com/topic/ps_nrf9160/_tmp/alta.nRF9160/autodita/CURRENT/parameters.id_current_sleep.html?cp=2_0_0_4_5_0_1

 

If you see lower nA numbers, then something is most likely wrong, ie. not properly powered or connected.

Is the slider for "enable power output" highlighted?

 

Kind regards,

Håkon