Storing sensor data as circular buffer in non volatile memory

[Artnia]

Hey guys,

I want to store sensor data as circular buffer in non volatile memory. I saw
https://docs.nordicsemi.com/bundle/zephyr-apis-latest/page/group_file_system_storage.html and https://docs.nordicsemi.com/bundle/ncs-latest/page/zephyr/services/storage/index.html. But researching here in the devzone I can see that users have issues like here https://devzone.nordicsemi.com/f/nordic-q-a/110506/nrf5340-fcb-flash-circular-buffer-on-external-flash/530085. 

I was trying to use NVS but it seems that it overwrites data as stated here https://github.com/nrfconnect/sdk-zephyr/blob/a1d9a8db0ea4bbee2e69feec1140e6ce20e62f5e/samples/subsys/nvs/src/main.c#L30
But of course I need the complete history of my data.

So my question is what is currently the best approach for this?

Best

[Hieu]

Hi Artnia,

You can use nvs_read_hist() to read past values. However, the past values are still gone when the current flash page for NVS is full, and a garbage collection and page switch is done.

Other than that, you store the entire circular buffer that you wish to track as the object. Please keep in mind though that this increase flash write size, shortening the flash erase cycle, and consequently shorten the flash life span.

Best,

Hieu

[Artnia]

Hi Hieu,

thanks. And in general which module/subsystem/etc would you suggest for storing measurement data non volatile in circular way (newest overwrites oldest)?

Best

[Hieu]

Hi Artnia,

Michal is out of office so I will take over this case again. Let me sync up and come back to you this week.

Hieu

[Hieu]

Hi Artnia,

I reproduced the "issue" on a nRF52840 DK. I observe that:

• After the sector change to sector 1, each write to the new sector causes the oldest line from sector 0 to disappear.
• sensor_ts and message counters should continue to increase. However, they both restarted at 0 after the sector change for some reason

I first suspect that the timestamp restarting to 0 might be indicative that the logic is reading from the wrong place. However, doubling the sleep duration to 200ms also causes the timestamp to restart to ~1100 instead, so that isn't it.

Without knowing the intended design of the FCB, I can't conclude anything yet. But I can say now that getting to the bottom of this is likely time consuming... I will keep working at it though.

Out of curiosity, was there any issue that motivated you to add this line?
ncs_add_partition_manager_config(pm_static.yml)

pm_static.yml should be picked up by default, without this line. Not that it is any problem.

Hieu

[Artnia]

Hi Hieu,

thanks for the reply.
I cant remember 100% why I put it there. If this is very unusual, it could be worth trying without.

[Hieu]

Hi Artnia,

That line isn't it. I reproduced the issue without it and with a much simpler pm_static.yml, since I am doing it on the nRF52840 DK.

I will keep looking into this.

app:
  address: 0x0
  end_address: 0x0f0000
  region: flash_primary
  size: 0x0f0000
test_partition:
  address: 0xf0000
  end_address: 0xfa000
  region: flash_primary
  size: 0xa000
dummy_partition:
  address: 0xfa000
  end_address: 0x100000
  region: flash_primary
  size: 0x6000
sram_primary:
  address: 0x20000000
  end_address: 0x20040000
  region: sram_primary
  size: 0x40000

[Hieu]

Hi Artnia,

So the issue is that you are using fe_data_off. This offset is relative to the sector start address. 
Therefore, to get the correct flash area offset, you need to add fe_sector->fs_off.

This incorrect usage is why when you attempt to write entries in sector 1, you are actually writing to sector 0 and corrupting the records.

The FCB public header file also provide a convenient macro for this, FCB_ENTRY_FA_DATA_OFF(entry).

I still have some doubts on why the timestamp could be overwritten, since flash memory doesn't really allow overwriting... But right now, the correct fix is identified, so I won't look into that yet.

Hieu