FDS garbage collection

RE: FDS garbage collection

vincent_chan — Mon, 04 Dec 2023 17:06:04 GMT

Hi, I think I am experiencing the exact issue. Do you find a solution for this issue?

RE: FDS garbage collection

Einar Thorsrud — Fri, 17 Sep 2021 13:37:47 GMT

Yes, in that case interrupt priorities should not be a problem.

RE: FDS garbage collection

emb_dev — Fri, 17 Sep 2021 12:27:57 GMT

We have the default priorities for SD and priority 6 for our irq handlers, so the requirements are met, right?

RE: FDS garbage collection

Einar Thorsrud — Fri, 17 Sep 2021 11:51:01 GMT

That depends on the priority of the interrupt where you make the FDS calls. See this post.

RE: FDS garbage collection

emb_dev — Thu, 16 Sep 2021 21:46:53 GMT

Hi, Einar! Thanks for the answers.

Yes, if deferred logging is disabled, we can see that queue_process() is called. The stack above is from the paused state after the "freeze".

We discovered that fds_gc() causes looping via event reentry in queue_process() (fds.c:1486, nRF5_SDK_17.0.2) if the number of freeable words is less than certain amount (~500). After that loop we got corrupted program flash.

Moving fds_gc call to the main loop helped. We do have some flash writes in irq handlers, but all of them have priorities > 4. So maybe not all the events allow to call fds_gs()? Now we can call it during FDS_EVT_WRITE, FDS_EVT_DEL_RECORD, FDS_EVT_DEL_FILE and FDS_EVT_GC events.

RE: FDS garbage collection

Einar Thorsrud — Thu, 16 Sep 2021 09:48:17 GMT

Hi,

[quote user="emb_dev"]We are running fds_gc() exclusively form fds event handler. Only one event handler is registered by fds_register. Can it be the RC of the issue? Should we move the call to the main loop?[/quote]

No, that should be OK.

[quote user="emb_dev"]NRF_LOG_ERROR in queue_start never logs anything, so we suggest that queue_process is not launched. Fault handlers are also not called.[/quote]

To get logs that are queued before the error, but not processed in the main loop you could disable deferred logging. That way logs will be processed and output immediately.

What can you find by debugging if you pause and check the stack pointer (when testing without optimization). Where is it?

RE: FDS garbage collection

emb_dev — Wed, 15 Sep 2021 14:38:46 GMT

We are running fds_gc() exclusively form fds event handler. Only one event handler is registered by fds_register. Can it be the RC of the issue? Should we move the call to the main loop?

NRF_LOG_ERROR in queue_start never logs anything, so we suggest that queue_process is not launched. Fault handlers are also not called.

Update: stack looks good, no corruption. Seems like fds_gc() triggers some looping :

page_identify fds.c:222
page_identify fds.c:214
fds_stat fds.c:2172
flash_inspect flash_utils.c:180  <== our handler calling fds_gc, fds_stat
event_send fds.c:109
queue_process fds.c:1487
event_send nrf_fstorage_sd.c:193
nrf_fstorage_sys_evt_handler nrf_fstorage_sd.c:559
nrf_sdh_soc_evts_poll nrf_sdh_soc.c:101
nrf_sdh_evts_poll nrf_sdh.c:358
<signal handler called> 0x00000000ffffffe9
serial_tx nrf_log_backend_uart.c:144
nrf_fprintf_buffer_flush nrf_fprintf.c:56
postfix_process nrf_log_str_formatter.c:150
nrf_log_std_entry_process nrf_log_str_formatter.c:192
...

RE: FDS garbage collection

Einar Thorsrud — Wed, 15 Sep 2021 13:01:00 GMT

I see. If you make a debug build and pause execution, what does the call stack look like then? Have you followed the operation further than queue_start()? Does it enter queue_process()? What then? You need to trace this down to where something actually happens (if it looks like a "freeze", perhaps a busy wait, or perhaps some issue with interrupt priorities causes an event to never happen.

Do you by any chance make the FDS API calls from a high interrupt priority?

RE: FDS garbage collection

emb_dev — Wed, 15 Sep 2021 12:53:04 GMT

Hi,

the device calls fds_gc() and there we added some logs for debugging:

ret_code_t fds_gc(void)
{
    fds_op_t * p_op;
    nrf_atfifo_item_put_t iput_ctx;
    NRF_LOG_ERROR("GC 0")
    if (!m_flags.initialized)
    {
        return FDS_ERR_NOT_INITIALIZED;
    }
    NRF_LOG_ERROR("GC 1")
    p_op = queue_buf_get(&iput_ctx);
    NRF_LOG_ERROR("GC 2")
    if (p_op == NULL)
    {
        return FDS_ERR_NO_SPACE_IN_QUEUES;
    }
    NRF_LOG_ERROR("GC 3")
    p_op->op_code = FDS_OP_GC;
    queue_buf_store(&iput_ctx);
    NRF_LOG_ERROR("GC 4")
    if (m_gc.state != GC_BEGIN)
    {
        // Resume GC by retrying the last step.
        m_gc.resume = true;
        NRF_LOG_ERROR("GC 5")
    }
    NRF_LOG_ERROR("GC 6")
    queue_start();
    NRF_LOG_ERROR("GC 7")
    return NRF_SUCCESS;
}

static void queue_start(void)
{
    if (!nrf_atomic_u32_fetch_add(&m_queued_op_cnt, 1))
    {
        NRF_LOG_ERROR("GC queue")
        queue_process(NRF_SUCCESS);
    }
}

Then the log output looks like this:

[00:01:39.000,000] <error> Flash: corruption: 1 valid records: 22 dirty records: 24 largest contig: 2048 freeable words: 6264 used words: 4629
[00:01:39.000,000] <error> Flash: corruption: 1 valid records: 23 dirty records: 24 largest contig: 2048 freeable words: 6264 used words: 4629
[00:01:39.000,000] <error> Flash: corruption: 1 valid records: 23 dirty records: 25 largest contig: 2048 freeable words: 6268 used words: 4629
[00:01:43.000,000] <error> app: GC queue
[00:01:43.000,000] <error> Flash: corruption: 1 valid records: 24 dirty records: 25 largest contig: 2048 freeable words: 6268 used words: 5260
[00:01:43.000,000] <error> app: GC 0
[00:01:43.000,000] <error> app: GC 1
[00:01:43.000,000] <error> app: GC 2
[00:01:43.000,000] <error> app: GC 3
[00:01:43.000,000] <error> app: GC 4
[00:01:43.000,000] <error> app: GC 6

Also the error check does not result in a reboot:

APP_ERROR_CHECK(fds_gc());

RE: FDS garbage collection

Einar Thorsrud — Wed, 15 Sep 2021 12:40:55 GMT

Hi,

Have you done any debugging to see what state the device is in when it "freezes"? That should be the first step in order to understand what has happened.