Logging interval data into flash memory. FDS or fstorage? Wear levelling?

[coyodha]

Hi Nordic DevTeam,

I am using nRF52840 (1MB Flash, 256kB RAM). I would like to store a log data to flash memory that I can extract them to my Android phone later on when connected. I have concern about the 10,000 write cycle of the nrRF52, I have to write this frequently to the flash.

Let's say after I allocate bootloader, SD s140 library and application (total : ~250kB in flash).  I expect the device to last 5-6 years and I allocate 500kB just for logging purposes and continuously log data every 1 minute when the device is on in csv format : eg. "time, voltage, current, etc." (let's say 40 Bytes each minute). Before I save to flash, I will allocate this data in RAM first (maybe fixed-sized array of 50kB), and whenever the allocated RAM is almost full or the device is about to be turned off. I will write this RAM data into the flash page, using the FDS library provided it has the wear-levelling mechanism in the background. My understanding is the bigger the flash allocated to it, wear-levelling will be more effective as there are more probability to use unused space to store data.

Does my implementation make sense in this case considering the write cycle of the flash?

Would it be recommended to log and save logged data into internal flash memory or would I really need external flash memory like SD Card?

[coyodha]

Hi Einar,

Sorry for asking too much questions.

I look up that  #define NRF_FSTORAGE_SD_MAX_WRITE_SIZE 4096.  

Since fds is built on top of fstorage,

If my data currently stored in the RAM exceeds 4096 (Eg. 50000 bytes), do I need to write the logic to queue the writing from RAM to flash? or is already taken account by the fds library?

[Einar Thorsrud]

Hi,

I will try to answer all questions.

coyodha said:

How do you use the FDS Storage with CLI example?

CLI best with UART (and not RTT), and specifically PuTTY is good (though other terminal emulators can also be used).

coyodha said:

Either you do it with fds_record_update() which will read, append and write to the flash by replacement into existing record id, or fds_record_write() which create a new record, and keeping the old one. 

fds_record_update() and fds_record_write() both write a new record. The only difference is that fds_record_update() writes a flag to invalidate the old record so that it can be removed by the next garbage collection. The flash wear will be identical as you anyway write a completely new record.

coyodha said:

-. store all these data into RAM until it's full or power is about to be turned off then force it to flash.

I did not comment on this before, but I see no reason to wait this long before writing to flash (though it does not cause problems wither, except risk of loosing more data in case of unexpected reset). The maximum size of a FDS record is the page size less overhead (2 words for a page tag and 3 words for record header), so absolute maximum record size is 1019 words = 4076 bytes.

coyodha said:

The way I imagine is that there will be a .csv file inside the flash storage called "data_log.txt", that everytime I need to append, I just opened this file and append a huge amount of string array into the file. This seems like a FATFS implementation, would it be recommended to use fatfs instead? (I am not sure whether the wear-levelling is implemented here)

How will you transfer this file to the PC? How you do this may determine how to do it. If you do it by BLE, then it does not matter much how it is organized on the nRF as yo until need to assemble it after it has been transferred. Instinctively I would prefer FDS as it is proven on the nRF and works well with the SoftDevice etc unless you have a good reason for using something else.

coyodha said:

If my data currently stored in the RAM exceeds 4096 (Eg. 50000 bytes), do I need to write the logic to queue the writing from RAM to flash? or is already taken account by the fds library?

FDS operates on records, so you will need to split your chunks in sensible record sizes. Remember that records cannot span pages so if you have say a record that is 600 words, you would waste almost half the page. So you should consider this. Also, if you use bonding then the peer manager will also store data in FDS, so you should consider other potential FDS users as well.

Regarding NRF_FSTORAGE_SD_MAX_WRITE_SIZE specifically this is a configuration for fstorage, and if you change this in your sdk_config.h it will also apply to FDS, but handled behind the schenes by fstorage. In practice this mean that you can set a lower value in order to make it easier for the SoftDevice to schedule flash write operations in between BLE activity etc. if it is difficult to have time for larger writes in one go. Normally you do not need to think about this.

[Einar Thorsrud]

Hi,

coyodha said:

Seems like from what you said here, the second chunk of records will be stored in the SECOND Virtual Page, which means that the FIRST Virtual Page remaining space (4076 - 1000 = 3076 bytes) will be wasted ? 

No, it will be as you wrote in step 1-3. I was referring to words, which are 4 bytes. Basically, the algorithm in FDS searches from the beginning, and will write the record to the first location it finds where it fits. The point is that you will have fragmentation, and if you write records that are just above half a page size, you will waste just below a page size. So you should make sure that does not happen. This is a generic problem with most file systems though, and just something you should think about so that you avoid choosing bad record sizes as illustrated.

[coyodha]

Hi Einar,

I hope you have a good holiday!

Einar Thorsrud said:

The point is that you will have fragmentation, and if you write records that are just above half a page size, you will waste just below a page size.

Just to clarify my understanding of this, so you are saying that if I have records which have a size of say 600 words each, while a page max word is 1019 words. Since records can't span pages, you are saying that each record will occupy 1 FDS Virtual Page instead of appending to the location of the last saved flash data and wasting the (1019 - 600 = 409 words) inside each page?

So if you have occupied 1000 words on this first virtual page, and you want to append another 100 words into it, which of this scenario will happen:

1. Since 1019 words is maximum on 1 page, the next 100 words will be stored as 19 words in FIRST PAGE and the rest 81 words in SECOND PAGE.

or

2. Since records can't span pages, the remaining 19 words in the FIRST PAGE will be wasted and the 100 word will stored in SECOND PAGE.

The reason I asked this is because my record might not be fixed-sized all the time, so I will still need to consider the best size to do this.

[Einar Thorsrud]

Hi,

coyodha said:

2. Since records can't span pages, the remaining 19 words in the FIRST PAGE will be wasted and the 100 word will stored in SECOND PAGE.

Yes, this is how it is.

coyodha said:

Is there any advantage/disadvantage of increasing the FDS_VIRTUAL
_PAGE_SIZE? default is 1024 words (4-byte words), and the other option is 2048 words?

The advantage of a larger virtual page size would be possible larger records and less overhead. The downside is significant though, as the swap page (which is essentially wasted space) will have to be as large. So if you use more than one physical page as virtual page, you will effectively reduce the available bytes of flash you can use. So in practice the virtual flash page should be the same as the physical flash page.

[coyodha]

Hi Einar,

I also learned about this amounts of block allowed to be written in between erase -> nWrite = 181 or is it 2 (based on this data sheet) ?

https://devzone.nordicsemi.com/f/nordic-q-a/11357/nrf52-flash-programming-writes-in-a-block-between-erase

https://devzone.noricsemi.com/f/nordic-q-a/15798/flash-endurance-on-nrf52

Should I be worried about this? Is this handled by the nrf_fstorage_sd ?

My plan right now is I will allocate 500kB of flash memory (128 virtual pages) and write 3 types of data file :

1. 0x01 (CONFIG_FILE_ID) -> write 4B + 12B (header) with same key ID every 5 minutes (maybe I will write this data under flash reserved page so it doesn't get deleted and just perform fds_gc on this flash page)

2. 0x02 (DATA_FILE_ID_1) -> write 240B + 12B (header) with new key ID every 20 minutes

3. 0x03 (DATA_FILE_ID_2) -> write 6B + 12B (header) with new key ID every 15 minutes

Just let all these keep writing until the flash amount has no space, then perform erase flash for the entire allocated flash.

Would this be a reasonable approach if I expect the device to run for at least 5-6 years assuming it runs all the time until flash reach 10,000 erase/write cycles?

[Einar Thorsrud]

Hi,

This is handled by FDS, which does not write more than 2 times to the same word.  (And the only that is written to more than once is the record header when it is invalidated, which would be the second write).

The limitation to the number of writes per block is also not an issue, as each page is split in 8 blocks, and you can write up to 181 times per block (so per 0x1000 words / 8 = 512 words. Given that you have 3 words overhead per record it is not a limit it is possible to reach in practice.

The approach you describe here is sensible and will give a minimal flash wear.