Saving coredumps to external flash

RE: Saving coredumps to external flash

Vidar Berg — Mon, 27 May 2024 10:43:46 GMT

Hello Sam,

The team has not been able to prioritize this feature request, unfortunately.

Best regards,

Vidar

RE: Saving coredumps to external flash

motters — Fri, 24 May 2024 19:16:48 GMT

Hello,

Thought i'd pop back 8 months later and see if this is now supported?

Cheers,

Sam

RE: Saving coredumps to external flash

Vidar Berg — Fri, 29 Sep 2023 08:49:15 GMT

Hi Sam,

Unfortunately, saving the coredump to external flash is still not supported. The test I did was for internal flash. For external flash, I believe it might be necessary to create a separate backend.

https://github.com/zephyrproject-rtos/zephyr/issues/54325

Cheers,

Vidar

RE: Saving coredumps to external flash

motters — Fri, 29 Sep 2023 08:36:27 GMT

Hi,

I did explore this but i had issue with writing to external flash.

I think the zephyr method only supports writing to internal uC flash.

Cheers,

Sam

RE: Saving coredumps to external flash

Vidar Berg — Fri, 29 Sep 2023 08:19:04 GMT

Sorry, there is a flash backend; it's just that our documentation has not been updated to include it yet. It is currently covered in the upstream Zephyr documentation at https://docs.zephyrproject.org/latest/services/debugging/coredump.html

I tried testing the coredump module with the Bluetooth: Peripheral LBS sample in nRF Connect SDK v2.4.2 and was able to get it to work after I created this workaround in the flash driver:

diff --git a/drivers/mpsl/flash_sync/flash_sync_mpsl.c b/drivers/mpsl/flash_sync/flash_sync_mpsl.c
index eea9cadaa..c37ebd78c 100644
--- a/drivers/mpsl/flash_sync/flash_sync_mpsl.c
+++ b/drivers/mpsl/flash_sync/flash_sync_mpsl.c
@@ -140,9 +140,15 @@ void nrf_flash_sync_set_context(uint32_t duration)
 	_context.request_length_us = duration;
 }
 
+bool is_in_fault_isr(void)
+{
+	uint32_t isr = __get_IPSR();
+	return (isr >= 3 && isr <= 6);
+}
+
 bool nrf_flash_sync_is_required(void)
 {
-	return mpsl_is_initialized();
+	return mpsl_is_initialized() && !is_in_fault_isr();
 }
 
 int nrf_flash_sync_exe(struct flash_op_desc *op_desc)

Summary of changes made to the peripheral LBS sample to enable and test the Coredump module

Added the following lines to the prj.conf file:

# Enable coredump with flash backend
CONFIG_SHELL=y
CONFIG_DEBUG_COREDUMP_SHELL=y
CONFIG_DEBUG_COREDUMP=y
CONFIG_DEBUG_COREDUMP_BACKEND_FLASH_PARTITION=y
CONFIG_DEBUG_COREDUMP_MEMORY_DUMP_MIN=y
CONFIG_DEBUG_COREDUMP_MEMORY_DUMP_LINKER_RAM=n
# Must be disabled to allow flash write from exception handler.
# https://github.com/zephyrproject-rtos/zephyr/issues/59116
CONFIG_ASSERT=n

And the following code in main.c to raise a fault when the button is pressed (Button 1 on DK)

static void do_fault(void)
{
	*(uint32_t *) 0xFFFFFFFF = 1;
}

static void button_changed(uint32_t button_state, uint32_t has_changed)
{
	if (has_changed & USER_BUTTON) {
		uint32_t user_button_state = button_state & USER_BUTTON;

		bt_lbs_send_button_state(user_button_state);
		app_button_state = user_button_state ? true : false;
		do_fault();
		
	}
}

And lastly, the Devictree overlay to allocate the coredump partition in flash (this is for the nRF52840):

&flash0 {
    partitions {
        storage_partition: partition@f8000 {
            reg = <0x000f8000 0x00004000>;
        };
        coredump_partition: partition@fc000 {
			label = "coredump_partition";
            reg = <0x000fc000 0x00004000>;
        };

    };
};

Testing

1. After the fault has been triggered, connect a serial terminal to the board to access the stored coredump via the shell

2. Copy the coredump data to a text file. E.g., coredump.log. Then perform step 1 to 4 here https://docs.zephyrproject.org/latest/services/debugging/coredump.html#example

Result

peripheral_lbs_coredump_flash$ arm-zephyr-eabi-gdb build/zephyr/zephyr.elf 
GNU gdb (Zephyr SDK 0.16.0) 12.1
Copyright (C) 2022 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "--host=x86_64-build_pc-linux-gnu --target=arm-zephyr-eabi".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://github.com/zephyrproject-rtos/sdk-ng/issues>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from build/zephyr/zephyr.elf...
(gdb) target remote localhost:1234
Remote debugging using localhost:1234
0x000113b0 in button_changed (button_state=<optimized out>, has_changed=<optimized out>) at ../src/main.c:177
177     }
(gdb) bt
#0  0x000113b0 in button_changed (button_state=<optimized out>, has_changed=<optimized out>) at ../src/main.c:177
#1  0x00000000 in ?? ()
(gdb) info registers 
r0             0xfffffff3          -13
r1             0x1                 1
r2             0x1                 1
r3             0xfffff000          -4096
r4             0x0                 0
r5             0x0                 0
r6             0x0                 0
r7             0x0                 0
r8             0x0                 0
r9             0x0                 0
r10            0x0                 0
r11            0x0                 0
r12            0xffffffff          -1
sp             0x2000ab18          0x2000ab18 <sys_work_q_stack+2008>
lr             0x113b1             70577
pc             0x113b0             0x113b0 <button_changed+28>
xpsr           0x1000000           16777216
(gdb)

Project

devzone.nordicsemi.com/.../peripheral_5F00_lbs_5F00_coredump_5F00_flash.zip

RE: Saving coredumps to external flash

KLarocqueEMFluids — Thu, 28 Sep 2023 12:21:27 GMT

Do you guys plan on implementing it or for now we have to implement our own code to do that

RE: Saving coredumps to external flash

Vidar Berg — Thu, 28 Sep 2023 11:56:14 GMT

Memfault is not intended to replace the debug support in the SDK. Instead, it serves as an additional option for those who want remote device management without setting up and managing their own cloud solution. I've updated my internal feature request to highlight the need for a way to save core dumps to flash.

Documentation for the Core dump module can be found here: https://developer.nordicsemi.com/nRF_Connect_SDK/doc/2.4.2/zephyr/services/debugging/coredump.html. There is currently no flash backend to save the output to a flash partition.

RE: Saving coredumps to external flash

KLarocqueEMFluids — Wed, 27 Sep 2023 14:19:28 GMT

Thanks you for that. I will look into it. For some reasons there so little documentation on coredump and the coredump documentation flash partition tool from zephyr seems to not dump anything when I force an hardfault. I hate that Nordic is forcing you with Memfault instead of trying to finding a solution for offline debugging

RE: Saving coredumps to external flash

Vidar Berg — Thu, 20 Jul 2023 10:11:35 GMT

No problem! This is an internal system only, but I will update you on the progress here if there is any. I'm not sure how soon they will be able to prioritize this task.

RE: Saving coredumps to external flash

motters — Thu, 20 Jul 2023 08:31:58 GMT

Thanks. Apologies late on this!

Would be great to know if this feature does go anywhere. Is there anyway to track it's progress or rejection? Or is it an internal system only.

Yes, i might try giving the nrfx drivers a go. It'd be really useful to collect this data.

RE: Saving coredumps to external flash

Vidar Berg — Thu, 29 Jun 2023 09:50:26 GMT

Thank you for taking the time to provide this feedback. I will report it as a feature request internally.

[quote user="motters"]It would still be good to be able to write directly to flash/QSPI. As this way we could take a full memory dump.[/quote]

I found that we support writing core dumps to the internal flash in the Memfault module, which can be found here: https://github.com/nrfconnect/sdk-nrf/blob/main/modules/memfault-firmware-sdk/memfault_flash_coredump_storage.c . I suppose the same approach could work with QSPI (i.e. use the nrfx driver directly instead of the Zephyr NOR driver).

RE: Saving coredumps to external flash

motters — Thu, 29 Jun 2023 09:25:48 GMT

I've got this working, personally i think is area is lacking in nRF. With everyone shipping thousands of IoT devices with no remote debugging capability seems silly. Surely we shouldn't all keep re-inventing the wheel. I'll see if i get time to write a module which can implement everything.

It would still be good to be able to write directly to flash/QSPI. As this way we could take a full memory dump.

Anyways, few pointers for anyone else interested:

If your using nrf52 there's a really good example of how to use retained memory here: https://github.com/nrfconnect/sdk-zephyr/blob/main/samples/boards/nrf/system_off/src/retained.c

To get the ESF you can override Zephyr's fatal function and simply store the values in retained memory (as the above example shows).

void k_sys_fatal_error_handler(unsigned int reason, const z_arch_esf_t *esf_input)

To save the core dump you'll have to create a file called "coredump_backend_empty.c" using this as a template https://github.com/zephyrproject-rtos/zephyr/blob/main/tests/subsys/debug/coredump_backends/src/coredump_backend_empty.c

You can then simply "memcpy" the coredump in function:

static void coredump_empty_backend_buffer_output(uint8_t *c, size_t buflen)

I use the following settings to keep the coredump small enough for RAM retention

CONFIG_DEBUG_COREDUMP=y
CONFIG_DEBUG_COREDUMP_BACKEND_OTHER=y
CONFIG_DEBUG_COREDUMP_MEMORY_DUMP_LINKER_RAM=n
CONFIG_DEBUG_COREDUMP_SHELL=y
CONFIG_DEBUG_COREDUMP_MEMORY_DUMP_MIN=y

Hope that helps someone.

RE: Saving coredumps to external flash

motters — Wed, 21 Jun 2023 16:49:37 GMT

Hi,

Yes i suppose thats a good solution. I was looking at Zephyr's new flash simulator https://github.com/zephyrproject-rtos/zephyr/blob/main/drivers/flash/flash_simulator.c but its v3.4 which nrf-sdk doesn't support yet.

My only concern with this was the sram usage if we were to take a whole dump and not just the pointers.

I'll have a play with this tomorrow.

Cheers

RE: Saving coredumps to external flash

Vidar Berg — Wed, 21 Jun 2023 11:51:53 GMT

Hello,

I don't think it's a good idea to rely on Zephyr drivers such as QSPI NOR when you are in the fault handler, because you don't know what state the system will be in. Could it be an alternative to load the coredump to a "no init" section and have it written to flash on subsequent reboot?

I've used this approach when debugging WDT timeouts in the past (the WDT does not give you enough time to write anything to flash before resetting):

__noinit static z_arch_esf_t esf;
__noinit static uint32_t esf_crc;
void dump_stack(uint32_t *p_msp)
{
    /* Store stack frame along with a checksum value to the __noinit section in RAM  */
    memcpy(&esf, p_msp, sizeof(esf));
    esf_crc = crc32_ieee((uint8_t *)&esf, sizeof(esf));
    /* Wait for the impending Watchdog reset */
    while (1);
}
#if WDT_ALLOW_CALLBACK
static void wdt_callback(const struct device *wdt_dev, int channel_id)
{
    /* Get current stack frame from the process stack. 
     * 
     * TODO: implement logic to determine if the application
     * was running in thread or handler mode prior to the WDT interrupt.
     * For handler mode we would have to use the main stack pointer instead. 
     */
    __ASM(" mrs r0, psp          \n"
          " ldr r3, = dump_stack \n"
          " bx r3                \n");
}
#endif /* WDT_ALLOW_CALLBACK */
/* To be called on startup to check if a new exception frame has been stored by our wdt_callback() */
void wdt_startup_check(void)
{
    uint32_t computed_crc = crc32_ieee((uint8_t *)&esf, sizeof(esf));
    if (computed_crc == esf_crc) {
        printk("Exception stack frame:\n\r");
        printk("r0/a1:  0x%08x  r1/a2:  0x%08x  r2/a3:  0x%08x\n\r", esf.basic.a1,
            esf.basic.a2, esf.basic.a3);
        printk("r3/a4:  0x%08x r12/ip:  0x%08x r14/lr:  0x%08x\n\r", esf.basic.a4,
            esf.basic.ip, esf.basic.lr);
        printk("xpsr:  0x%08x  pc: 0x%08x\n\r", esf.basic.xpsr, esf.basic.pc);
        esf_crc = 0; // Invalidate CRC
    } else {
        printk("CRC mismatch\n\r");
    }
}

Cheers,

Vidar