Saving coredump to flash

RE: Saving coredump to flash

Sigurd — Thu, 28 Sep 2023 12:08:11 GMT

Answered by Vidar here: RE: Saving coredumps to external flash

RE: Saving coredump to flash

KLarocqueEMFluids — Wed, 27 Sep 2023 15:40:57 GMT

No thats not an option not everyone can afford that out of the box. We should have something that is native to the chips. Its a shame that Nordic chips cant implement that.

RE: Saving coredump to flash

Konstantin Klitenik — Thu, 12 May 2022 13:48:13 GMT

Thanks, Sigurd. We will go with Memfault.

RE: Saving coredump to flash

Sigurd — Thu, 12 May 2022 13:33:04 GMT

Hi,

I got some feedback from the team. We have not heard of anyone using Zephyr's built-in coredump functionality on Nordic ICs (or Arm in general), though it does seem to be supported.
Most nRF Connect customers use memfault instead, would that be an option?

https://developer.nordicsemi.com/nRF_Connect_SDK/doc/latest/nrf/libraries/others/memfault_ncs.html

(Otherwise we will investigate using Zephyr's coredump)

Memfault's integration supports storing coredumps in flash:
CONFIG_MEMFAULT_NCS_INTERNAL_FLASH_BACKED_COREDUMP

Note that memfault has its own coredump handling, it doesn't use the upstream Zephyr one.

RE: Saving coredump to flash

Sigurd — Thu, 05 May 2022 14:00:46 GMT

Hi,

I'm checking with the developers.

RE: Saving coredump to flash

joebambino — Fri, 29 Apr 2022 19:33:35 GMT

Yes, Sigurd , has there been any follow up?

RE: Saving coredump to flash

Konstantin Klitenik — Fri, 29 Apr 2022 15:13:09 GMT

Any update on this? Is there a plan to support coredump to flash with BT enabled?

Sigurd

RE: Saving coredump to flash

Konstantin Klitenik — Thu, 21 Apr 2022 20:24:50 GMT

After more digging, it appears that the issue is with how nrf uses k_sem_take(). According to Zephyr docs (https://docs.zephyrproject.org/3.0.0/reference/kernel/synchronization/semaphores.html), if k_sem_take is called from ISR, it must use K_NO_WAIT for timeout.

However, NRF SDK uses K_FOREVER or actual timeout values in various places that are involved in the coredump. Because coredump is happening in the ISR, those timeout values are not valid. An assert gets triggered, or if asserts are disabled, I imagine Zephyr is unhappy, which is why we fail to take the semaphore.

__ASSERT(((arch_is_in_isr() == false) ||

K_TIMEOUT_EQ(timeout, K_NO_WAIT)), "");

Same issue exists with mutexes. nrf_flash_sync_exe() acquires a mutex but mutex lock has this assert

__ASSERT(!arch_is_in_isr(), "mutexes cannot be used inside ISRs");

RE: Saving coredump to flash

Konstantin Klitenik — Thu, 21 Apr 2022 18:31:54 GMT

[quote userid="15146" url="~/f/nordic-q-a/87148/saving-coredump-to-flash/364304#364304"]Seems like it's stream_flash_init() that returns an error-code. Could you print the error-code, and post it here? E.g. change the line to something like this to print the value:[/quote]

I traced the issue down to nrf_flash_sync_exe() in flash_sync_mpsl.c. The error happens when that function tries to obtain the semaphore (with my slight modification to print the status):

int status = k_sem_take(&_context.timeout_sem, K_MSEC(FLASH_TIMEOUT_MS));

LOG_ERR("Sem take result: %d (%ld)\n", status, FLASH_TIMEOUT_MS);

if (status < 0) {

LOG_ERR("timeout");

atomic_set(&_context.timeout_occured, 1);

}

This results in EAGAIN:

<err> flash_sync_mpsl: Sem take result: -11 (22963)

I tried to play with the number of slots. I've tried 0, 1, and 2. All result in the same error.

CONFIG_MPSL_TIMESLOT_SESSION_COUNT=2

RE: Saving coredump to flash

Sigurd — Thu, 21 Apr 2022 16:47:49 GMT

[quote user=""]NRF SDK 1.6.1.[/quote][quote user="ST-Kon"]coredump_flash_backend_start() adds an offset for the header but does not adjust the size but the same amount[/quote]

I see there was a fix here recently(fixed in NCS v1.9.x), see these 2 commits:

https://github.com/nrfconnect/sdk-zephyr/commit/1dc74d70f265ab753b9ef3f00450426bc2ccf296

https://github.com/nrfconnect/sdk-zephyr/commit/d5520f2e59b7d6d65bb8f4b6e848df1d46e95836

RE: Saving coredump to flash

Konstantin Klitenik — Thu, 21 Apr 2022 16:45:05 GMT

OK, the CRC error was being caused because my defined partition size of 4k and the shell thread stack size (where I was triggering the crash from) was also 4k. This caused incomplete coredump from being written to flash. In such a case, the CRC is not being calculated correctly, it's always off by 0x86. This probably should be fixed, even if incomplete coredump is saved.

Once I increased the partition to be 8k, I was able to successfully read it using coredump print.

So we just need to figure out how to make this work with BT enabled.

RE: Saving coredump to flash

Konstantin Klitenik — Thu, 21 Apr 2022 15:47:06 GMT

The code is ETIMEDOUT 116.

Interestingly, if I disable Bluetooth:

CONFIG_BT=n

CONFIG_BT_PERIPHERAL=n

CONFIG_BT_CTLR=n

Then I no longer see the timeout errors. It then complains about invalid parameter. Tracing that I found that because I put the coredump partition at the end of flash:

coredump_partition:

address: 0xff000

size: 0x1000

region: flash_primary

coredump_flash_backend_start() adds an offset for the header but does not adjust the size but the same amount. This causes stream_flash_init() to fail with invalid parameter because the start address (which now includes the offset) plus size of partition is past the end of flash.

offset = backend_ctx.flash_area->fa_off;

offset += ROUND_UP(sizeof(struct flash_hdr_t),

FLASH_WRITE_SIZE);

After adjusting the size by the offset (backend_ctx.flash_area->fa_size - 0x10,), I do see the coredump being written to flash. However, printing coredump fails because it fails CRC check. I'm currently trying to figure out why CRC in the header doesn't match read CRC.

In any case, we need to figure out why there is a timeout if BT is enabled. There also needs to be a bug filed to fix the size adjustment due to offset.

RE: Saving coredump to flash

Sigurd — Thu, 21 Apr 2022 15:39:16 GMT

[quote user="ST-Kon"]00> E: Cannot start coredump![/quote]

I see this line is being printed here: https://github.com/nrfconnect/sdk-zephyr/blob/v2.7.99-ncs1/subsys/debug/coredump/coredump_backend_flash_partition.c#L339

Seems like it's stream_flash_init() that returns an error-code. Could you print the error-code, and post it here? E.g. change the line to something like this to print the value:

LOG_ERR("Cannot start coredump! return-value: %d",ret);

RE: Saving coredump to flash

Konstantin Klitenik — Thu, 21 Apr 2022 13:14:53 GMT

Our application does use Bluetooth. I don't have a sample application without Bluetooth that I can try on our board or even the DK board.

We already had most of these set. The only difference was LOG_MODE. I updated LOG_MODE to be minimal but that did not make any difference. Still getting:

00> =====> Crashing system... (this is expected crash)
00> E: r0/a1: 0x00000003 r1/a2: 0x20013680 r2/a3: 0x00000040
00> E: r3/a4: 0x00000000 r12/ip: 0x00000000 r14/lr: 0x0002532b
00> E: xpsr: 0x41000000
00> E: Faulting instruction address (r15/pc): 0x00042352
00> E: >>> ZEPHYR FATAL ERROR 3: Kernel oops on CPU 0
00> E: Current thread: 0x200031b0 (shell_rtt)

// This is where it's trying to dump the core to flash but is failing
00> E: timeout
00> E: Cannot start coredump!

RE: Saving coredump to flash

Sigurd — Thu, 21 Apr 2022 08:45:07 GMT

Hi,

You only see this issue on samples that uses Bluetooth?

Could you try setting these configs? (taken from this .config file)

CONFIG_LOG=y
CONFIG_LOG_MODE_MINIMAL=y
CONFIG_DEBUG_COREDUMP=y
CONFIG_DEBUG_COREDUMP_BACKEND_FLASH_PARTITION=y
CONFIG_MP_NUM_CPUS=1
CONFIG_FLASH=y
CONFIG_DEBUG_COREDUMP_MEMORY_DUMP_MIN=y
CONFIG_DEBUG_COREDUMP_MEMORY_DUMP_LINKER_RAM=n