Logging interval data into flash memory. FDS or fstorage? Wear levelling?

[coyodha]

Hi Nordic DevTeam,

I am using nRF52840 (1MB Flash, 256kB RAM). I would like to store a log data to flash memory that I can extract them to my Android phone later on when connected. I have concern about the 10,000 write cycle of the nrRF52, I have to write this frequently to the flash.

Let's say after I allocate bootloader, SD s140 library and application (total : ~250kB in flash).  I expect the device to last 5-6 years and I allocate 500kB just for logging purposes and continuously log data every 1 minute when the device is on in csv format : eg. "time, voltage, current, etc." (let's say 40 Bytes each minute). Before I save to flash, I will allocate this data in RAM first (maybe fixed-sized array of 50kB), and whenever the allocated RAM is almost full or the device is about to be turned off. I will write this RAM data into the flash page, using the FDS library provided it has the wear-levelling mechanism in the background. My understanding is the bigger the flash allocated to it, wear-levelling will be more effective as there are more probability to use unused space to store data.

Does my implementation make sense in this case considering the write cycle of the flash?

Would it be recommended to log and save logged data into internal flash memory or would I really need external flash memory like SD Card?

[coyodha]

Hi Einar,

Thanks for your detailed response!

So regarding the splitting of the records, let's say I allocate 3 FDS virtual pages and I have 3 chunk of records that I need to store in the flash: 

1st -> 1000 bytes

2nd -> 1500 bytes

3rd -> 2000 bytes

I would like this data to be referred as one FILE_ID and probably multiple keys (maybe like a count of the record stamp) to refer.

Are the events that are going to happen as following ?

1. The first 1000 bytes stored in the FIRST Virtual Page. (Key : 0x01)

2. The next 1500 bytes will be appended in the FIRST Virtual Page. (Key : 0x02)

3.  The last 2000 bytes will be split into (4076 - 2500 = 1576 in the FIRST Virtual Page) and the remaining 424 bytes in the SECOND Virtual Page. (Key : 0x03).

Einar Thorsrud said:

Remember that records cannot span pages so if you have say a record that is 600 words, you would waste almost half the page.

Seems like from what you said here, the second chunk of records will be stored in the SECOND Virtual Page, which means that the FIRST Virtual Page remaining space (4076 - 1000 = 3076 bytes) will be wasted ? 

[Einar Thorsrud]

Hi,

coyodha said:

Seems like from what you said here, the second chunk of records will be stored in the SECOND Virtual Page, which means that the FIRST Virtual Page remaining space (4076 - 1000 = 3076 bytes) will be wasted ? 

No, it will be as you wrote in step 1-3. I was referring to words, which are 4 bytes. Basically, the algorithm in FDS searches from the beginning, and will write the record to the first location it finds where it fits. The point is that you will have fragmentation, and if you write records that are just above half a page size, you will waste just below a page size. So you should make sure that does not happen. This is a generic problem with most file systems though, and just something you should think about so that you avoid choosing bad record sizes as illustrated.

[coyodha]

Hi Einar,

I hope you have a good holiday!

Einar Thorsrud said:

The point is that you will have fragmentation, and if you write records that are just above half a page size, you will waste just below a page size.

Just to clarify my understanding of this, so you are saying that if I have records which have a size of say 600 words each, while a page max word is 1019 words. Since records can't span pages, you are saying that each record will occupy 1 FDS Virtual Page instead of appending to the location of the last saved flash data and wasting the (1019 - 600 = 409 words) inside each page?

So if you have occupied 1000 words on this first virtual page, and you want to append another 100 words into it, which of this scenario will happen:

1. Since 1019 words is maximum on 1 page, the next 100 words will be stored as 19 words in FIRST PAGE and the rest 81 words in SECOND PAGE.

or

2. Since records can't span pages, the remaining 19 words in the FIRST PAGE will be wasted and the 100 word will stored in SECOND PAGE.

The reason I asked this is because my record might not be fixed-sized all the time, so I will still need to consider the best size to do this.

[coyodha]

Also,

Is there any advantage/disadvantage of increasing the FDS_VIRTUAL
_PAGE_SIZE? default is 1024 words (4-byte words), and the other option is 2048 words?

[Einar Thorsrud]

Hi,

coyodha said:

2. Since records can't span pages, the remaining 19 words in the FIRST PAGE will be wasted and the 100 word will stored in SECOND PAGE.

Yes, this is how it is.

coyodha said:

Is there any advantage/disadvantage of increasing the FDS_VIRTUAL
_PAGE_SIZE? default is 1024 words (4-byte words), and the other option is 2048 words?

The advantage of a larger virtual page size would be possible larger records and less overhead. The downside is significant though, as the swap page (which is essentially wasted space) will have to be as large. So if you use more than one physical page as virtual page, you will effectively reduce the available bytes of flash you can use. So in practice the virtual flash page should be the same as the physical flash page.