having issues with saving coredump to flash or at all

RE: having issues with saving coredump to flash or at all

ziv123 — Mon, 27 Oct 2025 11:43:21 GMT

well, feeling a little neglected here 🙃🤷🏻‍♂️

anyway things i figured out so far:

1. what is saved to flash is actually a binary, but when using " nefjprog --memrd " to read it and save to a file it is converted to ascci hex (this can also be varified by the size of the saved file which is larger then the actual flash partition). to save the binary from flash this commands can be used:

JLinkExe -device nRF52840_xxAA -if swd -speed 4000 -autoconnect 1

followed by : savebin coredump.bin,0xED000,0x8000 from within JLink.

2. following instruction for coredump debug, here https://docs.zephyrproject.org/latest/services/debugging/coredump.html#:~:text=Run%20the%20core%20dump%20serial%20log%20converter%3A or here https://docs.nordicsemi.com/bundle/ncs-2.8.0/page/zephyr/services/debugging/coredump.html#:~:text=Run%20the%20core%20dump%20serial%20log%20converter%3A

the py script coredump_serial_log_parser.py is designed to take a .log file (not sure where i get it, maybe by copying my rtt output remove the timestamp and log file name etc. all the way to '#CD' and save to a file 🤔 ) .. any way, since i know how to get a binary i don't need this converter which is not really working with normal coredump cause normal coredump is missing the "BEGIN" bytes or the "#CD" and also contain a header before the CD initials 'ZE' .. so ..i can take my binary coredump parse the first 4 words to the header, get the size of coredump and then saved this size of bytes from after the header .. the reason that the bin file should not contain the header is that coredump_gdbserver.py calls for log_parser.py which expects to find 'ZE' in the first line.

p.s. in the CD saved to flash the bytes seems to be in reverse order from the way it seems in the log prints in the rtt, i think it is just a matter of how the tools present the ascii cause the bin file without any reverse operation is read by the log_parser.py and 'ZE' is found.

so far this is what i got working and understood.. Vidar Berg please correct me if i am wrong somewhere there.

the issue i am running this "/opt/zephyr-sdk/arm-zephyr-eabi/bin/arm-zephyr-eabi-gdb build/zephyr/zephyr.elf" (instead of non existing <path to SDK>/x86_64-zephyr-elf/bin/x86_64-zephyr-elf-gdb ) and trying to connect, then i get this error

(gdb)  target remote localhost:1234
localhost:1234
: cannot resolve name: Servname not supported for ai_socktype
localhost:1234
: No such file or directory.

even though coredump_gdbserver.py is running on another terminal

any idea why ? update - seems like it was related to order of things server must run before opening the gdb client even if the attempt to conect inside gdb is after the server start running ..

are there more debug options other then "info registers" and "bt" in order to get more information on running threads and stack ?

i'll mention that i am working with a coredump bin which is generated with normal building mechanism (

CONFIG_DEBUG_COREDUMP=y,

CONFIG_DEBUG_COREDUMP_BACKEND_FLASH_PARTITION=y,

CONFIG_DEBUG_COREDUMP_MEMORY_DUMP_THREADS=y,

CONFIG_ASSERT=n )

and i'll also mention that it is very strange to me that there is no clear tool or support for flash saved CD parsing / debugging

hope to read you soon

best regards

Ziv

RE: having issues with saving coredump to flash or at all

ziv123 — Mon, 20 Oct 2025 07:41:14 GMT

hi Vidar

well thanks for the patient and the sample

i found the issue with the PR of coredump_backend_nrf_flash_partition.c that caused the data to be saved partially and managed to save CD to flash 🤗.. however according to git i see this bug is already fixed (so i'll keep searching for other opportunities for me to contribute 🙃 )

basically it was this lines

for (uint32_t i = 0; i < size; i += sizeof(uint32_t)) {
		nrfx_nvmc_word_write(PARTITION_OFFSET + offset + i, UNALIGNED_GET(&data[i]));
	}
	
	data is uint8_t* type and jumping 4 bytes each time in the loop it losses 3 byts of data 
	so my solution and also in git was to cast it like so :
	
	nrfx_nvmc_word_write(PARTITION_OFFSET + offset + i, UNALIGNED_GET((const uint32_t *)&data[i]));

now i am a bit struggling with decoding of the coredump .. there are some python scripts in zephyr for that but they don't seem to work so well for example i get this error when running coredump_gdbserver.py

python3 coredump_gdbserver.py /../app/build_b/app/zephyr/zephyr.elf /../coredump/coredump_flash.bin 
Traceback (most recent call last):
  File "/.../ncs/v2.8.0/zephyr/scripts/coredump/coredump_gdbserver.py", line 14, in <module>
    from coredump_parser.elf_parser import CoredumpElfFile
  File "/.../ncs/v2.8.0/zephyr/scripts/coredump/coredump_parser/elf_parser.py", line 10, in <module>
    import elftools
ModuleNotFoundError: No module named 'elftools'

update: the above can be solve by installing "pip install pyelftools" (there are still errors cause i am not using a log file but a saved bytes hex string like)

is there a script like coredump_serial_log_parser.py that accepts the flash saved bytes instead of a log file to make it binary or is there a way to save CD initially as binary (this can probably save space on flash and less data to transfer OTA)?

another thing is big/little endien .. i notice that in the log the bytes are saved in this order ' 5a450100' (taken from log example in https://docs.zephyrproject.org/latest/services/debugging/coredump.html#:~:text=5a4501000100050000000000)

but in the file i read from flash they are saved like this 0002455A what is the correct way of byte order for it to be parsed ?

i would actually like to have one script that does all (giving it the zephyr.elf and coredump and an output file) and it will output a human readable parsing of coredump .. wonder if there is something like that ??

p.s. if you like i can open the CD parsing questions on a new thread and close this one

hope to read you soon

best regards

Ziv

RE: having issues with saving coredump to flash or at all

Vidar Berg — Thu, 16 Oct 2025 12:38:19 GMT

Yes, if the coredump backend triggers a fault or an assertion program will crash again before reaching your error handler. You can imagine what happens if the error handler calls something that triggers another error and keeps invoking itself. But you should be able to catch faults occurring in the error handler if you debug your code in VS code. A cpu lockup in debug mode will not trigger a reset but instead halt the program:

[quote user="ziv123"]2. i tried to add logs in nrf/modules/memfault-firmware-sdk/memfault_flash_coredump_storage.c to follow how memfault saves the coredump but when memfault is on i do not get any logs, even when i add LOG_PANIC() i just get the first call to z_arm_fault and after that it resets so i am not even sure it gets to the call of k_sys_fatal_error_handler() and if not, then which crash api memfault use to overwrite and take control of the flow ? [/quote]

The logger will not work in the case of a cpu lockup so it is not possible to rely on logging in this case.

[quote user="ziv123"]seems like it is loggs latency and not an actual interrupt being called while coredump operation is on[/quote]

When using LOG_MODE_DEFERRED logs are buffered and processed in the logger thread or when you call LOG_PANIC()

EDIT: adding a test sample I created store the stack frame and error reason (you can add other things here too) from the error handler to ram in case you are interested. This is without CD and it will therefore have a lower memory footprint.

devzone.nordicsemi.com/.../crash_5F00_save.zip

RE: having issues with saving coredump to flash or at all

ziv123 — Wed, 15 Oct 2025 11:15:07 GMT

[quote userid="4240" url="~/f/nordic-q-a/124173/having-issues-with-saving-coredump-to-flash-or-at-all/551451"]Please post a code snippet showing how you have implemented this function.[/quote]

i have managed to get it to overwrite, it basically overwrote before as well just it seems that i don't always get to that function being called and this is what i tried to ask here :

[quote userid="90084" url="~/f/nordic-q-a/124173/having-issues-with-saving-coredump-to-flash-or-at-all/551393"]1. according to log ahead, it seems that z_arm_fault precede z_fatal_error() which calls for coredump() and only after that calls for k_sys_fatal_error_handler() .. so i don't understand if memfault's implementation is overwriting this function to handle the crash data collection, it is done after coredump so why does it not get stuck even when asserts are enabled .... is it related to CONFIG_DEBUG_COREDUMP_BACKEND_OTHER though i don't see it used in coredump_backend_flash_partition.c ? or if i want to get to that handler i need to CONFIG_DEBUG_COREDUMP=n and only then i will get to this handler before system hults ? and in it i can call for the coredump save ? ?[/quote]

2. i tried to add logs in nrf/modules/memfault-firmware-sdk/memfault_flash_coredump_storage.c to follow how memfault saves the coredump but when memfault is on i do not get any logs, even when i add LOG_PANIC() i just get the first call to z_arm_fault and after that it resets so i am not even sure it gets to the call of k_sys_fatal_error_handler() and if not, then which crash api memfault use to overwrite and take control of the flow ?

p.s.

[quote userid="4240" url="~/f/nordic-q-a/124173/having-issues-with-saving-coredump-to-flash-or-at-all/551451"]I expect all but zero latency interrupts to be masked at this point.[/quote]

seems like it is loggs latency and not an actual interrupt being called while coredump operation is on

hope to read you soon

best regards

Ziv

RE: having issues with saving coredump to flash or at all

Vidar Berg — Tue, 14 Oct 2025 13:54:01 GMT

[quote userid="90084" url="~/f/nordic-q-a/124173/having-issues-with-saving-coredump-to-flash-or-at-all/551393"]1. when i try to implement my k_sys_fatal_error_handler() in my main it does not overwrite anything, i still get to zephyr error handling and my error handler is never called. what am i missing ?[/quote]

Please post a code snippet showing how you have implemented this function. You can also check the zephyr.map file for your build to confirm if you were able to redefine the function.

[quote userid="90084" url="~/f/nordic-q-a/124173/having-issues-with-saving-coredump-to-flash-or-at-all/551393"]2. when working with assert enabled , according to the log at crash i am entering z_arm_fault() twice .. any idea why it is so ? [/quote]

As mentioned earlier, the CD flash backend is using semaphores (with timeouts) which will raise an assert when invoked from an interrupt context.

[quote userid="90084" url="~/f/nordic-q-a/124173/having-issues-with-saving-coredump-to-flash-or-at-all/551393"]3. when using the coredump_backend_nrf_flash_partition.c (i added some prints in it) i see that in the middle of coredump work i get the print for the event of the button push (i put the crash at start of button push handler) this is strange i think .. any idea why it suddenly pops ? [/quote]

I expect all but zero latency interrupts to be masked at this point.

RE: having issues with saving coredump to flash or at all

ziv123 — Tue, 14 Oct 2025 10:07:54 GMT

hi Vidar

[quote userid="4240" url="~/f/nordic-q-a/124173/having-issues-with-saving-coredump-to-flash-or-at-all/550093"]You need to use LOG_PANIC() to be able to flush the log buffer from the fault handler. [/quote]

thanks for the LOG_PANIC() it helped me follow what goes in a crash scenario but it brings this questions:

1. according to log ahead, it seems that z_arm_fault precede z_fatal_error() which calls for coredump() and only after that calls for k_sys_fatal_error_handler() .. so i don't understand if memfault's implementation is overwriting this function to handle the crash data collection, it is done after coredump so why does it not get stuck even when asserts are enabled .... is it related to CONFIG_DEBUG_COREDUMP_BACKEND_OTHER though i don't see it used in coredump_backend_flash_partition.c ? or if i want to get to that handler i need to CONFIG_DEBUG_COREDUMP=n and only then i will get to this handler before system hults ? and in it i can call for the coredump save ? ?

2. when working with assert enabled , according to the log at crash i am entering z_arm_fault() twice .. any idea why it is so ?

this is the logs:

with assert enable crash trigger by fault bus:
00> [00000006] <inf> AUGU_PUSH_B[00000007] <inf> os: In z_arm_fault
00> [00000007] <err> os: ***** BUS FAULT *****
00> [00000007] <err> os:   Imprecise data bus error
....
00> [00000007] <err> os: Faulting instruction address (r15/pc): 0x000xxxx
00> [00000008] <err> os: in z_fatal_error
00> [00000008] <err> os: >>> ZEPHYR FATAL ERROR 26: Unknown error on CPU 0
00> [00000008] <err> os: Current thread: 0xFFFFFFFF (augu_work_q)
00> ASSERTION FAIL [((arch_is_in_isr() == 0) || ((timeout).ticks == (((k_timeout_t) {0})).ticks))] @ WEST_TOPDIR/zephyr/kernel/sem.c:136
00>   
00> [00000008] <inf> os: In z_arm_fault
00> [00000008] <err> os: ***** HARD FAULT *****
00> [00000008] <err> os:   Fault escalation (see below)
00> [00000008] <err> os: ARCH_EXCEPT with reason 4
00> 
00> [00000008] [1;31m<err> os: ....
00> [00000008] <err> os: Faulting instruction address (r15/pc): 0x000XXXX
00> [00000008] <err> os: in z_fatal_error
00> [00000008] [1;31m<err> os: >>> ZEPHYR FATAL ERROR 4: Kernel panic on CPU 0
00> [00000008] <err> os: Fault during interrupt handling
00> 
00> [00000008] <err> os: Current thread: 0xFFFFFFFF (augu_work_q)
00> ASSERTION FAIL [((arch_is_in_isr() == 0) || ((timeout).ticks == (((k_timeout_t) {0})).ticks))] @ WEST_TOPDIR/zephyr/kernel/sem.c:136

3. when using the coredump_backend_nrf_flash_partition.c (i added some prints in it) i see that in the middle of coredump work i get the print for the event of the button push (i put the crash at start of button push handler) this is strange i think .. any idea why it suddenly pops ?

4. when i use default coredump flow (coredump_backend_flash_partition.c) with CONFIG_ASSERT=n, and i try to crash with k_panic or k_oops, i still fail in saving coredump because of a timeout error at stream_flash_init(..) .. so it seems like default coredump is only supported when BUS FAULT errors ( ZEPHYR FATAL ERROR 26: Unknown error on CPU) .. this is an issue because when an end device in the field get stuck on some scenario i want to reset it in that scenario immediately and not loos this device or waste energy and time in a watchdog feed mechanism .. is there a build in solution for handling dead end scenarios in the end device ? (obviously i want to know that the device failed and what happened before and how it got to this point this is why i need the coredump, usually those scenarios can be device fails to bond or some indication of memory corruption that happened)

5. i still haven't figured out why data isn't saved to flash properly in my use of coredump_backend_nrf_flash_partition.c adding prints in it i do see valid coredump values but for some reason it does not seem to be written to flash partition properly .. will update on that (i also took it to my app instead of using it from within ncs (this way i can keep ncs clean and it is easier if i need to modify the module)

hope to read you soon

best regards

Ziv

RE: having issues with saving coredump to flash or at all

Vidar Berg — Wed, 01 Oct 2025 06:50:13 GMT

You can find the end address of RAM programmatically using symbols generated by the partition manager in pm_config.h, for example, PM_SRAM_PRIMARY_END_ADDRESS. Then use this as the start address when copying in the crash data.

RE: having issues with saving coredump to flash or at all

Tudor B. — Tue, 30 Sep 2025 10:22:13 GMT

Thank you for the explanation.

Beyond the configuration:

&sram0 {
   reg = <0x20000000 DT_SIZE_K(458752) - 150;
};

How would I set the variables in those 150 bytes that are reserved?

RE: having issues with saving coredump to flash or at all

Vidar Berg — Tue, 30 Sep 2025 09:05:52 GMT

The "missing" 64k is allocated to the shared RAM region and is used for communication with the network core.

RE: having issues with saving coredump to flash or at all

Tudor B. — Tue, 30 Sep 2025 08:51:02 GMT

Sorry Ziv, it seems Vidar brought up a valid point.

Going with your example Vidar, I would foresee something like:

&sram0 {
   reg = <0x20000000 DT_SIZE_K(458752) - 150;
};

This brings an interesting question: why don't I have the full 512Kb RAM?

I set the RAM config with 448Kb total since whatever configs I use, I get:

[650/650] Linking C executable zephyr/zephyr.elf
Memory region         Used Size  Region Size  %age Used
           FLASH:      402024 B         3 MB     12.78%
             RAM:      190868 B       448 KB     41.61%
        IDT_LIST:          0 GB        32 KB      0.00%

I don't have TFM or MCUBoot enabled. Since currently we're using nrf5340dk and nrf7002dk Where are 64Kb disappearing to?

RE: having issues with saving coredump to flash or at all

Vidar Berg — Tue, 30 Sep 2025 07:06:16 GMT

Thanks for the update Tudor B. . I’m glad to see you got it working. Just want to add that the noinit section may become overwritten on startup if you include a bootloader. A solution is to place the data at the end of RAM and reduce the RAM size in the sram0 devicetree node to ensure the area remains unallocated.

&sram0 {
   reg = <0x20000000 DT_SIZE_K(<total ram size in kB) - <size of area you want to set aside for crash data>>;
};

[quote userid="90084" url="~/f/nordic-q-a/124173/having-issues-with-saving-coredump-to-flash-or-at-all/549986"]any idea what is the reason for that ?[/quote]

Hard to say what's causing this. I suggest you try this in a minimal sample in the latest SDK to see if you are able to reproduce the same.

[quote userid="90084" url="~/f/nordic-q-a/124173/having-issues-with-saving-coredump-to-flash-or-at-all/549986"]plus, i would really appreciate if you can give some clarity on what i asked in prev message regarding the weak api that is not overwritten and about the order in which things are happening at crash and what about all the asserts in zephyr code if assert i[/quote]

You need to use LOG_PANIC() to be able to flush the log buffer from the fault handler.

RE: having issues with saving coredump to flash or at all

ziv123 — Mon, 29 Sep 2025 09:21:56 GMT

Hi Vidar

I took the approach of the merged solution you found, for using nrf apis to collect the CD instead of zephyr's, - https://github.com/nrfconnect/sdk-nrf/pull/21418/files#diff-fcf14c4b7b34fe7a11916195871ae66a59be87a395f28db73e345ebdc828085b and integrated it in my ncs ..

it seems to be working fine with assertion and i get the crash log as expected

but .. when i read the flash area where the CD was saved i do not see what i expect - the initials 0x5a,0x45 are missing and the whole thing looks like it is missing some data .. in every 4 bytes it seems to be writing only one with a value and the rest are zeros (the core dump i see when using memfault does not look like this)

any idea what is the reason for that ?

plus, i would really appreciate if you can give some clarity on what i asked in prev message regarding the weak api that is not overwritten and about the order in which things are happening at crash and what about all the asserts in zephyr code if assert is disabled 🙏

Tudor B. i you were successful, i think if you have farther questions on the solution you adopt it might be better to open a new thread since we took different approaches for CD collection 🙏

hope to read you soon

best regards

Ziv

RE: having issues with saving coredump to flash or at all

Tudor B. — Sun, 28 Sep 2025 17:09:45 GMT

Hey Vidar.

I finished implementing your proposal and in the end it worked like a charm! The implementation:

#include <zephyr/arch/cpu.h>
#include <zephyr/sys/reboot.h>
#include <zephyr/fatal.h>

#define LOG_LEVEL CONFIG_LOG_DEFAULT_LEVEL
#include <zephyr/logging/log.h>
#include <zephyr/logging/log_ctrl.h>
LOG_MODULE_REGISTER(aaa);

__noinit static struct arch_esf esf_keep;
__noinit static uint32_t esf_crc = 0;

extern void sys_arch_reboot(int type);

void k_sys_fatal_error_handler(unsigned int reason, const struct arch_esf *esf)
{
    (void)irq_lock();

    // From Nordic DevZone example
	/* Store stack frame along with a checksum value to the __noinit section in RAM  */
    memcpy(&esf_keep, esf, sizeof(*esf));
    LOG_ERR("AAA! Faulting instruction address (r15/pc): 0x%08x", esf_keep.basic.pc);
    esf_crc = crc32_ieee((uint8_t *)&esf_keep, sizeof(esf_keep));

    // From fatal_error.c
	LOG_ERR("Resetting system");
	LOG_PANIC();
	sys_arch_reboot(0);

	CODE_UNREACHABLE;
}

// generate fault
static mp_obj_t mp_do_fault(void)
{
	*(uint32_t *) 0xFFFFFFFF = 1;
}
static MP_DEFINE_CONST_FUN_OBJ_0(mp_do_fault_obj, mp_do_fault);

// check fault and print stored stack frame
static mp_obj_t mp_check_val(void)
{
    // esf_crc =0;
	printk("Stored CRC val is: ~%ld~ \r\n", esf_crc);

    uint32_t computed_crc = crc32_ieee((uint8_t *)&esf_keep, sizeof(esf_keep));
    printk("Computed CRC val is: ~%ld~ \r\n", computed_crc);

    if (computed_crc == esf_crc) {
        printk("Exception stack frame:\n\r");
        printk("r0/a1:  0x%08x  r1/a2:  0x%08x  r2/a3:  0x%08x\n\r", esf_keep.basic.a1,
            esf_keep.basic.a2, esf_keep.basic.a3);
        printk("r3/a4:  0x%08x r12/ip:  0x%08x r14/lr:  0x%08x\n\r", esf_keep.basic.a4,
            esf_keep.basic.ip, esf_keep.basic.lr);
        printk("xpsr:  0x%08x  pc: 0x%08x\n\r", esf_keep.basic.xpsr, esf_keep.basic.pc);
        esf_crc = 0; // Invalidate CRC
    } else {
        printk("CRC mismatch\n\r");
    }

    return mp_const_none;
}
static MP_DEFINE_CONST_FUN_OBJ_0(mp_check_val_obj, mp_check_val);

And actually running it:

*** Booting nRF Connect SDK v3.0.0-3bfc46578e42 ***
*** Using Zephyr OS v4.0.99-a0e545cb437a ***
MicroPython v1.24.0-preview.485.ge5af5faf8.dirty on 2025-09-28; zephyr-nrf5340dk with nrf5340
Type "help()" for more information.
>>>
>>>
>>> import aaa as a
>>>
>>>
>>> a.dofault()
[00:00:08.744,354] <err> os: ***** USAGE FAULT *****
[00:00:08.750,000] <err> os:   Unaligned memory access
[00:00:08.760,162] <err> os: r0/a1:  0x00000000  r1/a2:  0x00000000  r2/a3:  0x00000001
[00:00:08.768,890] <err> os: r3/a4:  0xfffff000 r12/ip:  0x00041e43 r14/lr:  0x0002452d
[00:00:08.777,648] <err> os:  xpsr:  0x29100000
[00:00:08.782,928] <err> os: s[ 0]:  0x20004a28  s[ 1]:  0x00042e3b  s[ 2]:  0x00042e1f  s[ 3]:  0x20022088
[00:00:08.793,426] <err> os: s[ 4]:  0x0004f580  s[ 5]:  0x00000157  s[ 6]:  0x0004f4f8  s[ 7]:  0x00021477
[00:00:08.803,924] <err> os: s[ 8]:  0x0004f4f8  s[ 9]:  0x00000157  s[10]:  0x20022088  s[11]:  0x20006987
[00:00:08.814,422] <err> os: s[12]:  0x20022088  s[13]:  0x2000698b  s[14]:  0x0004e038  s[15]:  0x00041e51
[00:00:08.824,890] <err> os: fpscr:  0x20022088
[00:00:08.830,139] <err> os: Faulting instruction address (r15/pc): 0x0002a0d8
[00:00:08.838,104] <err> os: >>> ZEPHYR FATAL ERROR 31: Unknown error on CPU 0
[00:00:08.846,069] <err> os: Current thread: 0x20001c10 (mp_main)
[00:00:08.852,905] <err> aaa: AAA! Faulting instruction address (r15/pc): 0x0002a0d8
*** Booting nRF Connect SDK v3.0.0-3bfc46578e42 ***
*** Using Zephyr OS v4.0.99-a0e545cb437a ***
MicroPython v1.24.0-preview.485.ge5af5faf8.dirty on 2025-09-28; zephyr-nrf5340dk with nrf5340
Type "help()" for more information.
>>>
>>>
>>> import aaa as a
>>>
>>>
>>> a.checkval()
Stored CRC val is: ~396985589~
Computed CRC val is: ~396985589~
Exception stack frame:
r0/a1:  0x00000000  r1/a2:  0x00000000  r2/a3:  0x00000001
r3/a4:  0xfffff000 r12/ip:  0x00041e43 r14/lr:  0x0002452d
xpsr:  0x29100000  pc: 0x0002a0d8

Note that we're using MicroPython in our project. But the relevant C code excerpts can be easily extracted.

RE: having issues with saving coredump to flash or at all

Tudor B. — Thu, 25 Sep 2025 17:18:48 GMT

Well, what Vidar suggests is NOT using Coredump with RAM, but rather using some variables declared in the no_init area to save the crash log data in them and then to check them when you start up again.

[quote userid="90084" url="~/f/nordic-q-a/124173/having-issues-with-saving-coredump-to-flash-or-at-all/549820"]thanks, but our application is already using most of available RAM so giving some of it up just for saving crashes is not a valid option [/quote]

I get it. We're also REALLY tight on RAM. We didn't hit the limit yet, but I expect we will in the next few months/ soon. But that's also the point! Enabling and using the Coredump feature costs you WAAAAY more ROM and RAM than simply doing his suggestion, which takes up ~70-80 bytes. 😎

Note: this whole "RAM implementation" is based on the two variables that he mentioned in that 2 year old ticket:

__noinit static z_arch_esf_t esf;
__noinit static uint32_t esf_crc;

I tried checking out "z_arch_esf_t" to see its size so I can get a rough estimate of the amount of RAM used. This led me down a small rabbit hole, which took me to: https://docs.zephyrproject.org/latest/releases/migration-guide-3.7.html

Where we find this phrase:

Migration guide to Zephyr v3.7.0
This document describes the changes required when migrating your application from Zephyr v3.6.0 to Zephyr v3.7.0.

...

Kernel
All architectures are now required to define the new struct arch_esf, which describes the members of a stack frame. This new struct replaces the named struct z_arch_esf_t. (GitHub #73593)

The named struct z_arch_esf_t is now deprecated. Use struct arch_esf instead. (GitHub #73593)

The header file include/zephyr/arch/arch_interface.h has been moved from include/zephyr/sys/ into include/zephyr/arch/. Out-of-tree source files will need to update the include path. (GitHub #64987)

So those previous two lines turn into:

__noinit static arch_esf esf;
__noinit static uint32_t esf_crc;

Keep in mind that we have NCS version 3.0.0, in which Zephyr has version v4.0.99-a0e545cb437a. You mentioned you're using v2.8.0 and I'm not sure which Zephyr version comes with that version of the NCS. So you gotta check if your Zephyr is >= v3.7.0 for the above to apply.

I doubt Coredump beats this given that the serial CLI logs it generates fill up a file with ~35kb of text.

RE: having issues with saving coredump to flash or at all

ziv123 — Thu, 25 Sep 2025 14:02:19 GMT

[quote userid="4240" url="~/f/nordic-q-a/124173/having-issues-with-saving-coredump-to-flash-or-at-all/549783"]Please read my previous comments where I try to explain what the issue is[/quote]

i read the issue here https://github.com/zephyrproject-rtos/zephyr/issues/59116 and the solution here https://github.com/nrfconnect/sdk-nrf/pull/21418 (which i think i may try to integrate into my ncs v2.8.0) untill we will update to ncs v3.1.1)

what i still don't understand is the order of things :

if i disable asserts then i see i am entering the 'z_arm_fault()' and after that to 'z_fatal_error()' which inside it call for coredump() and then to k_sys_fatal_error_handler() so how come it can be overwriten if when i enable asserts i am not getting into z_arm_fault() or z_fatal_error() ? ..

if i enable asserts and i am trying to use coredump even for logs (

CONFIG_DEBUG_COREDUMP_BACKEND_LOGGING or,

CONFIG_DEBUG_COREDUMP_MEMORY_DUMP_LINKER_RAM

) and not for flash i just don't see anything .. does the assertion trigger some hardware before zephyr apis can even act ? if so, why then when coredump is not configured then i do get into 'z_arm_fault()' and 'z_fatal_error()' when i have an assert and asserts are enabled ?

plus why if assert is enabled i don't get to zephyr apis even if i crash it with k_panic(), for example ?

sorry for being a nag about this 🙏 but i am really trying to understand the order of which things are happening when crash occurs 🙏

p.s. inside zephyr and ncs there is also a vast use of asserts .. what happning in those asserts when assert is disabled are he checks just skipped ? is it a valid practice to have assert enabled just for "on table" development and have a build with no asserts for deployed devices ?

i tried to overwrite the crash with my own implementation of this :

void k_sys_fatal_error_handler(unsigned int reason, const struct arch_esf *esf){
    LOG_ERR(">>> HardFault trapped in app override!\n");

    /* Option 1: loop forever (easy for breakpoints) */
    while (1) {
        __NOP();
    }
     
    /* Option 2: chain to Zephyr’s internal handler */
    // extern void z_arm_fault(uint32_t reason, const z_arch_esf_t *esf);
    // z_arm_fault(K_ERR_CPU_EXCEPTION, NULL);
}

but i did not see that it is actually overwriting something .. i don't see it's prints and i do see that z_fatal_error prints

[quote userid="145037" url="~/f/nordic-q-a/124173/having-issues-with-saving-coredump-to-flash-or-at-all/549788"]There is a special no_init area of the RAM[/quote]

thanks, but our application is already using most of available RAM so giving some of it up just for saving crashes is not a valid option

best regards

Ziv

RE: having issues with saving coredump to flash or at all

Tudor B. — Thu, 25 Sep 2025 10:34:00 GMT

Hey Ziv.

There is a special no_init area of the RAM which is persistent through a hot (soft) reset. Hot reset = reset command to the MCU; cold reset = complete power cycle. There is a CONFIG_ option for the system to do a hot reset in case of stack overflows/ hard faults -> the no_init area of the RAM is kept. 😉

RE: having issues with saving coredump to flash or at all

ziv123 — Thu, 25 Sep 2025 10:21:14 GMT

[quote userid="4240" url="~/f/nordic-q-a/124173/having-issues-with-saving-coredump-to-flash-or-at-all/549783"]tore relevant information to RAM from the k_sys_fatal_error_handler(). What you do with the RAM content on subsequent reboot is up to you. You can store it to flash, transfer it over BLE, etc.[/quote]

i think there is something fundamental i am missing here .. as far as i know ,whatever i save in RAM to some variable or whatever, is gone after reset, right ? . so, if i do not use logs and can only get info on the crash from the device via OTA, after it resets back to normal, then how saving things to RAM help me ?

RE: having issues with saving coredump to flash or at all

Vidar Berg — Thu, 25 Sep 2025 09:17:24 GMT

[quote userid="90084" url="~/f/nordic-q-a/124173/having-issues-with-saving-coredump-to-flash-or-at-all/549757"]we use assert a lot in our code and also zephyr uses it internally so this is why it is wird for me and also why i try to avoid it plus i don't seem to be getting to[/quote]

Please read my previous comments where I try to explain what the issue is. If you are going to have ASSERTs enabled you must use the flash storage backend introduced by the commit I linked.

[quote userid="90084" url="~/f/nordic-q-a/124173/having-issues-with-saving-coredump-to-flash-or-at-all/549757"]i can not use RAM for saving logs or CD since the devices in the field are configured with logs disabled and i need[/quote]

My suggestion is to not use the CD functionality at all but rather store relevant information to RAM from the k_sys_fatal_error_handler(). What you do with the RAM content on subsequent reboot is up to you. You can store it to flash, transfer it over BLE, etc.

RE: having issues with saving coredump to flash or at all

Tudor B. — Thu, 25 Sep 2025 06:59:16 GMT

Vidar Berg Well, we save the CD, then we reset, we get the reset reason, check the ext flash for CD's presence, save a correlated file with reset reason next to it. Something like:

crash_nr_231_CD.txt

crash_nr_231_reset_reason.txt

This also brings another question: can we also save other things the moment the hard fault happens? Sensor states, total runtime, battery level, etc?

But again, since you said that CD can't really save to external -> should we simply override the implementation for k_sys_fatal_error_handler() and disable CD completely?

ziv123 I didn't not ignore your point ziv, but this ties in to my main question to Vidar: can CD write to external? Is CD extendable to be able to write to external and write custom data? If not, then the best course of action is to make the mechanism ourselves inside k_sys_fatal_error_handler()?

RE: having issues with saving coredump to flash or at all

ziv123 — Thu, 25 Sep 2025 06:58:44 GMT

hi Vidar

[quote userid="4240" url="~/f/nordic-q-a/124173/having-issues-with-saving-coredump-to-flash-or-at-all/549382"]What is relevant here is whether you have CONFIG_ASSERT enabled in your build not[/quote]

we use assert a lot in our code and also zephyr uses it internally so this is why it is wird for me and also why i try to avoid it plus i don't seem to be getting to the writing of CD to flash at the moment .. anyway i'll try to run with it disabled and see

i can not use RAM for saving logs or CD since the devices in the field are configured with logs disabled and i need to know what happens there in a retrospective, after they reset and reconnected with a node

i don't mind using the internal flash or the external (external will be more comfortable obviously but internal will work for now as well)

[quote userid="4240" url="~/f/nordic-q-a/124173/having-issues-with-saving-coredump-to-flash-or-at-all/549382"]You can simply redefine the weakly defined k_sys_fatal_error_handler() function in your application[/quote]

the api seems to have fatal error reason and exception context .. are those filled automatically ?

just to be sure if i overwrite this api then i do not have to CONFIG_ASSERT=n ?

p.s. this api seems to be called from inside z_fatal_error which i don't thing i am getting into since i both don't see its prints and putting a breakpoint at the beginning of it when debugging did not stop there

RE: having issues with saving coredump to flash or at all

ziv123 — Thu, 25 Sep 2025 06:46:36 GMT

i suggested this for you, before .. we use it in main when app re starts

uint32_t reset_reason = nrf_power_resetreas_get(NRF_POWER);

RE: having issues with saving coredump to flash or at all

Vidar Berg — Thu, 25 Sep 2025 06:43:32 GMT

As mentioned earlier, writing to external flash requires a custom bare metal SPI flash driver that does not rely on any Zephyr primitives and can operate from the hardfault interrupt context. For internal flash, we already have a driver, see the PR linked earlier.

[quote userid="145037" url="~/f/nordic-q-a/124173/having-issues-with-saving-coredump-to-flash-or-at-all/549722"]For me, external is preferred since we'll also want to save the reset reason and have it all in one place[/quote]

I'm not sure what you have in mind here. How would you include the reset reason in the CD.

RE: having issues with saving coredump to flash or at all

Tudor B. — Wed, 24 Sep 2025 16:15:19 GMT

Thank you for the hasty reply Vidar.

Well, given your answer, could you provide us a minimal sample project with the necessary changes to be able to save the CD to the internal/ external flash? For me, external is preferred since we'll also want to save the reset reason and have it all in one place. and I saw that the logging CD generates ~34kb of data. Dunno what Zhiv prefers.

RE: having issues with saving coredump to flash or at all

Vidar Berg — Wed, 24 Sep 2025 16:06:21 GMT

The r14/lr register will tell you were the call was made from. But yes, there might of be cases were even a minimal CD will provide relevant debug information that's not shown in the crash log.

RE: having issues with saving coredump to flash or at all

Tudor B. — Wed, 24 Sep 2025 15:38:37 GMT

Well, given a hypothetical scenario where we have a function "myFunc(int a)" and it's used in 7 different files in our project and the crash log just says that "myFunc(int a)" is the Faulting instruction, with the crash log we can't really figure out what caused the error, but given that the coredump provides a stack trace, we can.