Storing sensor data as circular buffer in non volatile memory

[Artnia]

Hey guys,

I want to store sensor data as circular buffer in non volatile memory. I saw
https://docs.nordicsemi.com/bundle/zephyr-apis-latest/page/group_file_system_storage.html and https://docs.nordicsemi.com/bundle/ncs-latest/page/zephyr/services/storage/index.html. But researching here in the devzone I can see that users have issues like here https://devzone.nordicsemi.com/f/nordic-q-a/110506/nrf5340-fcb-flash-circular-buffer-on-external-flash/530085. 

I was trying to use NVS but it seems that it overwrites data as stated here https://github.com/nrfconnect/sdk-zephyr/blob/a1d9a8db0ea4bbee2e69feec1140e6ce20e62f5e/samples/subsys/nvs/src/main.c#L30
But of course I need the complete history of my data.

So my question is what is currently the best approach for this?

Best

[Hieu]

Hi Artnia,

You can use nvs_read_hist() to read past values. However, the past values are still gone when the current flash page for NVS is full, and a garbage collection and page switch is done.

Other than that, you store the entire circular buffer that you wish to track as the object. Please keep in mind though that this increase flash write size, shortening the flash erase cycle, and consequently shorten the flash life span.

Best,

Hieu

[Artnia]

Hi Hieu,

thanks. And in general which module/subsystem/etc would you suggest for storing measurement data non volatile in circular way (newest overwrites oldest)?

Best

[Hieu]

Hi Artnia,

Most persistent storage solution over flash works the same way as NVS, and has the same limitation that history is completely removed upon garbage collection. I am not aware of any other module/subsystem/etc that behave differently.

An option is to write your own version of NCS, or a subsystem based on it, that upon switching flash page, not only copy the newest value, but also copy a configurable number of past values.

If you do that, you could also consider making it a contribution to NVS/the Zephyr project.

Best,

Hieu

[Artnia]

Hi Hieu,

thanks for the info.
I guess FCB could be useful for my case: https://docs.nordicsemi.com/bundle/ncs-latest/page/zephyr/services/storage/fcb/fcb.html

FCB splits partition into sectors and fills sectors sequentially. Then once all sectors full, oldest sector is erased and becomes the active one (=rotation). This way FIFO is achieved and not all data but "just" old data get erased. That is the theory. But I am facing an issue with it. First sector 0 is filled up which is ok. Then once sector 0 is full FCB starts with sector 1. At this point there is still plenty of free space over all and no rotation happened yet. But for some reason the data in sector 1 is garbage and also data in sector 0 is modified. Which I both didn't expect.

Attached please find a minimal (hello world) FCB example for nrf9160DK I created based on
https://github.com/zephyrproject-rtos/zephyr/tree/main/subsys/fs/fcb and https://devzone.nordicsemi.com/f/nordic-q-a/110506/nrf5340-fcb-flash-circular-buffer-on-external-flash/530085. Also I added a log file "sector_issue.log". Since the terminal window cant hold much information, the log is split in two parts:

• Up to line 261: Init and first entries in sector 0 (message counter 10)
• From line 261: Sector 0 get full and sector 1 is written(message counter 44)
  Notice how sector 1 has garbage data and sector 0 is modified

hello_world_fcb.zip

[Michal]

Hello,

I took over this case from my colleague. Thank you very much for the minimal example, I will try to test that in the following days.

Best regards,

Michal

[Artnia]

Hello Michal,

thank you. Did you already have the chance to check it?

[Michal]

Sorry, not yet.

I will test it tomorrow.

Best regards,

Michal

[Artnia]

Hello Michal,

thanks

[Michal]

Hello,

Sorry for late reply, I have been testing it and trying to understand the failure mode.

Just a small request, could you add a small delay while printing out the log somewhere, since it is too fast for the built-in VCOM and is dropping logs at full speed?

I think you may also want to ask about FCB on the Zephyr discord as well, since it's a generic Zephyr OS service, people there may have an idea about the issue. 

https://docs.zephyrproject.org/latest/project/communication.html

Best regards,

Michal

[Artnia]

Hi Michal,

thanks for the reply.

I added a small delay in fcb_sector_walk_cb() but the issue seems to persist. Since you have the sources, could you double check?

 

static int fcb_sector_walk_cb(struct fcb_entry_ctx *entry_ctx, void *arg)
{
    uint16_t sector_no = *(uint16_t *)arg;
    struct FCB_sensor_data data;
    const struct flash_sector *sector = entry_ctx->loc.fe_sector;
    // Find sector index
    int idx = -1;
    for (int i = 0; i < FCB_SECTOR_COUNT; i++) {
        if (sector == &fcb_instance_sectors[i]) {
            idx = i;
            break;
        }
    }
    if (idx == sector_no) {
        // Print entry offset and length
        LOG_INF("[walk] sector %u: entry offset=0x%08lx, len=%u", sector_no, (unsigned long)entry_ctx->loc.fe_data_off, entry_ctx->loc.fe_data_len);
        // Read raw bytes for debug
        uint8_t raw[sizeof(struct FCB_sensor_data)] = {0};
        int rc = flash_area_read(entry_ctx->fap, entry_ctx->loc.fe_data_off, raw, entry_ctx->loc.fe_data_len);
        if (rc) {
            LOG_ERR("Failed to read entry with error %d", rc);
        } else {
            // Print raw bytes
            // char hexbuf[3 * sizeof(raw) + 1];
            // int hexlen = 0;
            // for (int i = 0; i < entry_ctx->loc.fe_data_len && i < (int)sizeof(raw); i++) {
            //     hexlen += snprintf(&hexbuf[hexlen], sizeof(hexbuf) - hexlen, "%02X ", raw[i]);
            // }
            // hexbuf[hexlen] = '\0';
            // LOG_INF("[walk] sector %u: raw: %s", sector_no, hexbuf);
            // Also decode as struct if possible
            if (entry_ctx->loc.fe_data_len == sizeof(struct FCB_sensor_data)) {
                memcpy(&data, raw, sizeof(struct FCB_sensor_data));
                LOG_INF("[walk] sector %u: sensor_ts: %llu, message_counter: %u, temp_deg_c: %d, humidity_percent: %u, pressure_kpa: %u",
                    sector_no,
                    data.payload.sensor_ts,
                    data.payload.message_counter,
                    data.payload.temp_deg_c,
                    data.payload.humidity_percent,
                    data.payload.pressure_kpa
                );
            }
            k_sleep(K_MSEC(100)); // Small delay for log readability
        }
    }
    return 0;
}