Logging interval data into flash memory. FDS or fstorage? Wear levelling?

[coyodha]

Hi Nordic DevTeam,

I am using nRF52840 (1MB Flash, 256kB RAM). I would like to store a log data to flash memory that I can extract them to my Android phone later on when connected. I have concern about the 10,000 write cycle of the nrRF52, I have to write this frequently to the flash.

Let's say after I allocate bootloader, SD s140 library and application (total : ~250kB in flash).  I expect the device to last 5-6 years and I allocate 500kB just for logging purposes and continuously log data every 1 minute when the device is on in csv format : eg. "time, voltage, current, etc." (let's say 40 Bytes each minute). Before I save to flash, I will allocate this data in RAM first (maybe fixed-sized array of 50kB), and whenever the allocated RAM is almost full or the device is about to be turned off. I will write this RAM data into the flash page, using the FDS library provided it has the wear-levelling mechanism in the background. My understanding is the bigger the flash allocated to it, wear-levelling will be more effective as there are more probability to use unused space to store data.

Does my implementation make sense in this case considering the write cycle of the flash?

Would it be recommended to log and save logged data into internal flash memory or would I really need external flash memory like SD Card?

[Einar Thorsrud]

Hi,

Does my implementation make sense in this case considering the write cycle of the flash?

Yes, this makes sense. FDS has basic wear leveling, so in principle you will write equal amount of data to all the flash pages you have allocated for FDS.

Would it be recommended to log and save logged data into internal flash memory or would I really need external flash memory like SD Card?

If you calculate the amount of data you intend to write, and find that you will over the life cycle of the device, and find that you will not write more than 10.000 times the number of flash pages you have allocated (less one for swap), then there should not be any problem and you will be using the device within the limits of the specification. If not, then you may want to consider additional external storage.

[coyodha]

Hi Einar,

Thanks for your reply.

How do you use the FDS Storage with CLI example?

Do I need to connect it with putty?

I can't get any log reading from the SEGGER Debug Terminal even after I switched "NRF_LOG_ENABLED" to 1. Some of the other example has this "NRF_LOG_BACKEND_RTT_ENABLED" in their "sdk_config.h" . This example doesn't. Can you clarify what's the difference between the 2 settings?

[coyodha]

Hi Einar,

I found out that there are two ways of appending data to the flash.

Either you do it with fds_record_update() which will read, append and write to the flash by replacement into existing record id, or fds_record_write() which create a new record, and keeping the old one. 

devzone.nordicsemi.com/.../data-appending-flash-storage

What are the pros and cons on using one against the other?

My procedure is simple :

-. Allocate X amount of bytes in flash for data logging storage.

-. these data would be a comma separated value, eg :

(unix time, current, voltage, ....)

1620794473, 53, 10000, ..., ...\r\n

-. store all these data into RAM until it's full or power is about to be turned off then force it to flash.

-. when device is connected with phone, extract all these data and (optional) erase the flash data storage memory.

The way I imagine is that there will be a .csv file inside the flash storage called "data_log.txt", that everytime I need to append, I just opened this file and append a huge amount of string array into the file. This seems like a FATFS implementation, would it be recommended to use fatfs instead? (I am not sure whether the wear-levelling is implemented here)

I understand that fds is a file system with the record_id, file_id and key_id. It's kind of hard to imagine how I would do this with my ideal scenario.

But do let me know if there's any better way to do this.

[coyodha]

Hi Einar,

Sorry for asking too much questions.

I look up that  #define NRF_FSTORAGE_SD_MAX_WRITE_SIZE 4096.  

Since fds is built on top of fstorage,

If my data currently stored in the RAM exceeds 4096 (Eg. 50000 bytes), do I need to write the logic to queue the writing from RAM to flash? or is already taken account by the fds library?

[Einar Thorsrud]

Hi,

I will try to answer all questions.

coyodha said:

How do you use the FDS Storage with CLI example?

CLI best with UART (and not RTT), and specifically PuTTY is good (though other terminal emulators can also be used).

coyodha said:

Either you do it with fds_record_update() which will read, append and write to the flash by replacement into existing record id, or fds_record_write() which create a new record, and keeping the old one. 

fds_record_update() and fds_record_write() both write a new record. The only difference is that fds_record_update() writes a flag to invalidate the old record so that it can be removed by the next garbage collection. The flash wear will be identical as you anyway write a completely new record.

coyodha said:

-. store all these data into RAM until it's full or power is about to be turned off then force it to flash.

I did not comment on this before, but I see no reason to wait this long before writing to flash (though it does not cause problems wither, except risk of loosing more data in case of unexpected reset). The maximum size of a FDS record is the page size less overhead (2 words for a page tag and 3 words for record header), so absolute maximum record size is 1019 words = 4076 bytes.

coyodha said:

The way I imagine is that there will be a .csv file inside the flash storage called "data_log.txt", that everytime I need to append, I just opened this file and append a huge amount of string array into the file. This seems like a FATFS implementation, would it be recommended to use fatfs instead? (I am not sure whether the wear-levelling is implemented here)

How will you transfer this file to the PC? How you do this may determine how to do it. If you do it by BLE, then it does not matter much how it is organized on the nRF as yo until need to assemble it after it has been transferred. Instinctively I would prefer FDS as it is proven on the nRF and works well with the SoftDevice etc unless you have a good reason for using something else.

coyodha said:

If my data currently stored in the RAM exceeds 4096 (Eg. 50000 bytes), do I need to write the logic to queue the writing from RAM to flash? or is already taken account by the fds library?

FDS operates on records, so you will need to split your chunks in sensible record sizes. Remember that records cannot span pages so if you have say a record that is 600 words, you would waste almost half the page. So you should consider this. Also, if you use bonding then the peer manager will also store data in FDS, so you should consider other potential FDS users as well.

Regarding NRF_FSTORAGE_SD_MAX_WRITE_SIZE specifically this is a configuration for fstorage, and if you change this in your sdk_config.h it will also apply to FDS, but handled behind the schenes by fstorage. In practice this mean that you can set a lower value in order to make it easier for the SoftDevice to schedule flash write operations in between BLE activity etc. if it is difficult to have time for larger writes in one go. Normally you do not need to think about this.