Storing sensor data as circular buffer in non volatile memory

[Artnia]

Hey guys,

I want to store sensor data as circular buffer in non volatile memory. I saw
https://docs.nordicsemi.com/bundle/zephyr-apis-latest/page/group_file_system_storage.html and https://docs.nordicsemi.com/bundle/ncs-latest/page/zephyr/services/storage/index.html. But researching here in the devzone I can see that users have issues like here https://devzone.nordicsemi.com/f/nordic-q-a/110506/nrf5340-fcb-flash-circular-buffer-on-external-flash/530085. 

I was trying to use NVS but it seems that it overwrites data as stated here https://github.com/nrfconnect/sdk-zephyr/blob/a1d9a8db0ea4bbee2e69feec1140e6ce20e62f5e/samples/subsys/nvs/src/main.c#L30
But of course I need the complete history of my data.

So my question is what is currently the best approach for this?

Best

[Hieu]

Hi Artnia,

You can use nvs_read_hist() to read past values. However, the past values are still gone when the current flash page for NVS is full, and a garbage collection and page switch is done.

Other than that, you store the entire circular buffer that you wish to track as the object. Please keep in mind though that this increase flash write size, shortening the flash erase cycle, and consequently shorten the flash life span.

Best,

Hieu

[Artnia]

Hi Hieu,

thanks. And in general which module/subsystem/etc would you suggest for storing measurement data non volatile in circular way (newest overwrites oldest)?

Best

[Hieu]

Hi Artnia,

Most persistent storage solution over flash works the same way as NVS, and has the same limitation that history is completely removed upon garbage collection. I am not aware of any other module/subsystem/etc that behave differently.

An option is to write your own version of NCS, or a subsystem based on it, that upon switching flash page, not only copy the newest value, but also copy a configurable number of past values.

If you do that, you could also consider making it a contribution to NVS/the Zephyr project.

Best,

Hieu

[Artnia]

Hi Hieu,

thanks for the info.
I guess FCB could be useful for my case: https://docs.nordicsemi.com/bundle/ncs-latest/page/zephyr/services/storage/fcb/fcb.html

FCB splits partition into sectors and fills sectors sequentially. Then once all sectors full, oldest sector is erased and becomes the active one (=rotation). This way FIFO is achieved and not all data but "just" old data get erased. That is the theory. But I am facing an issue with it. First sector 0 is filled up which is ok. Then once sector 0 is full FCB starts with sector 1. At this point there is still plenty of free space over all and no rotation happened yet. But for some reason the data in sector 1 is garbage and also data in sector 0 is modified. Which I both didn't expect.

Attached please find a minimal (hello world) FCB example for nrf9160DK I created based on
https://github.com/zephyrproject-rtos/zephyr/tree/main/subsys/fs/fcb and https://devzone.nordicsemi.com/f/nordic-q-a/110506/nrf5340-fcb-flash-circular-buffer-on-external-flash/530085. Also I added a log file "sector_issue.log". Since the terminal window cant hold much information, the log is split in two parts:

• Up to line 261: Init and first entries in sector 0 (message counter 10)
• From line 261: Sector 0 get full and sector 1 is written(message counter 44)
  Notice how sector 1 has garbage data and sector 0 is modified

hello_world_fcb.zip

[Michal]

Hello,

I took over this case from my colleague. Thank you very much for the minimal example, I will try to test that in the following days.

Best regards,

Michal