Creating a stack trace of all threads from a coredump

[timl2415]

I have a multithreaded application running on NCS 2.9.0 and I want to be able to see what all the threads are doing at the time of a crash via coredump. I have the following options configured in my project:

CONFIG_DEBUG_COREDUMP=y

CONFIG_DEBUG_COREDUMP_BACKEND_LOGGING=y

CONFIG_DEBUG_COREDUMP_MEMORY_DUMP_THREADS=y

CONFIG_DEBUG_COREDUMP_SHELL=y

I then followed the instructions here to start the gdb server and connect to the instance.

https://docs.zephyrproject.org/latest/services/debugging/coredump.html

The backtrace worked fine for the crash but only dumped info relevant to the crashing thread (the shell_uart thread leading to my fault CLI in this instance), but for other faults this may not be sufficient. I ran into a few problems while trying to get relevant info from the other threads running on the system:

There seems to be a typo in gdbstub.py that seems to disable all thread awareness:

    def arch_supports_thread_operations(self):
        return False

This is checked in a few places and gates where the stub accesses the thread metadata from the coredump. After setting this to return True, it seems like all thread ID's, priorities, and statuses are output but the frame is a duplicate of the crash thread frame:

(gdb) info threads
  Id   Target Id                                                                           Frame 
* 1    Thread 1 (name: shell_uart, state: 0x80, user_options: 0x0, prio: 0xe)              _CmdFault (shell=0x7e0bc <shell_uart>, argc=1, 
    argv=0x20018994 <shell_uart_stack+3684>) at /Users/timothy.lee/project/src/Sensor/console.c:217
  2    Thread 2 (name: wdt_lowprio_thread_id, state: 0x10, user_options: 0x0, prio: 0xe)   _CmdFault (shell=0x7e0bc <shell_uart>, argc=1, 
    argv=0x20018994 <shell_uart_stack+3684>) at /Users/timothy.lee/project/src/Sensor/console.c:217
  3    Thread 3 (name: task_1_thread, state: 0x2, user_options: 0x0, prio: 0x9)            _CmdFault (shell=0x7e0bc <shell_uart>, argc=1, 
    argv=0x20018994 <shell_uart_stack+3684>) at /Users/timothy.lee/project/src/Sensor/console.c:217
  4    Thread 4 (name: task_2_thread, state: 0x10, user_options: 0x0, prio: 0xe)           _CmdFault (shell=0x7e0bc <shell_uart>, argc=1,

The debug logs from the gdbstub seem to indicate that gdb wants to read the registers from each thread but its unimplemented on the stub so the frame is just duplicated from the crash frame:

    def handle_register_group_read_packet(self):
        # the 'g' packet for reading a group of registers
        pass

I was trying to extend the stub myself by scanning through the threads metadata from the coredump but im not sure where the program counter is kept for each thread (or if it exists at all). Does the thread metadata contain any information about the program counter and stack of each thread?

[timl2415]

Figured it out, it seems like the required functionality to parse and send the thread stacks is implemented already in arm_cortex_m.py, although these won't be hooked into the gdbstub unless the methods in the parent GdbStub class are marked as abstract so that the ARM-specific methods are called instead. I'm guessing these aren't marked as abstract by default since they are TODO for all non arm cortex-m architectures.