FDS read non sequential sdk 17

[AdityaS09]

Hello,

I am using one nrf52840 based SOC with nRF5_SDK_17.0.2_d674dde SDK for one of our project. I am trying to utilize internal 1MB of flash for storing some data. For that I am using FDS library. I am writing to flash on periodic basis in chunks of size > 220bytes. Till that time I will hold onto the data in local buffer. However if and when a request is made to get the data, first I am immediately writing all the data held data to the flash so that while reading back I can collect all the data from there itself. In such cases the last packet which is being immediately written may not be >220bytes. Data contains multiple sample of 14 bytes each and each sample contains timestamp.

Now the problem which I am facing is that while reading back from the flash, the data read is not in sequential manner i.e. the last packet which I read back is not the last packet which I stored in the flash. This happens mostly when the size of data is less than my maximum threshold which is case when I am writing the last packet when I data request comes. What I actually read is the last >220 bytes of packet. And my less than lets say 50 bytes of data which actually got written to the flash last is read somewhere in between the read attempts. So the data is not lost but it messes up the sequence of the data and timestamps related to it.

I am attaching a file for reference. This is an example set of data which is received from the flash. The data is received in the same sequence as shown in the file. If we try to plot the last column of timestamp we can observe some spikes in the value, which are actually not the spike but smaller set of data being misplaced in flash or while reading back. Those spikes are just the last set of data (which got written last)being read early and sent first.

CC_log_dataMis.csv 

Is there a reason behind such behavior from FDS? Is it because of some fragmentation in storage space which causes it store data in flash in where it find available space? Or am I missing something in my implementation?

[tesc]

Hi,

If you have stored all of the data with the same File ID + Record Key, then searching for that combination of File ID + Record Key will give you the records in arbitrary order.

On a fresh system where no records has been deleted and there has not been garbage collection, the records should mostly be in the same order as they were written, but there is no guarantee.

The "spike" that you see on the value sets on lines 261 through 266 in the csv, I assume, is a result of the following:

* The first 260 values were placed in the first FDS flash page, in records that were stored consecutively in flash.

* The 261th value (on line 267 in the csv) did not fit in the remaining space of the first FDS flash page, and was placed at the beginning of the second FDS flash page. Records in FDS cannot span multiple flash pages. The same happens for the following records (containing more values than can fit in the remaining space in the first FDS flash page.)

* Then, after the values on line 602 in the csv, a shorter record (with the values on lines 261 through 266) is to be written, and that record is short enough to fit in the remaining free space at the end of the first FDS flash page. Therefore it gets put there.

* The values after that are all in records that are too big to fit in the remaining gap at the end of the first FDS flash page, and so they are all placed consecutively in the second FDS flash page.

* When you read back the records, you get them in the order they are physically stored in flash.

Please note that although this is what happens with the current implementation of FDS, it is not safe to assume that this will always be how it happens. What is guaranteed by FDS, is that records are stored in flash, and that they can be searched and found using either File ID, Record Key, or a combination of both. The physical ordering in flash, and the order which they are found during search may change, and will usually change after garbage collection (as records are moved around.)

There are some options how to get the ordering correct:

Option 1: Order the data according to the timestamps.

Option 2: Use the same File ID for all the records, but use a different (increasing) Record Key, then:

2 a) Search for File ID when retrieving data, and sort the segments according to the Record Key. Or:

2 b) Keep track of which record was last read, and search for File ID + Record Key combinations, with increasing Record Key for each new record searched for.

Note for option 2 that Record Keys must be in the range 0x0001 through 0xBFFF. Record Keys outside of that range are reserved by the system (0x0000) or by Peer Manager (0xC000 through 0xFFFF.) This means you can reach 0xBFFF in the index (49151 in decimal) before you must go back to 0x0001 again.

Option 3: Add a serial number to each record, that gets incremented by one every time you store a new record, and use that for the ordering (instead of using the Record Key or the timestamps.) This may be an option if you don't want to manually handle the overflow behaviour of Record Keys, and expect to get more than 49150 records. Just choose a large enough integer data type to never run out of indexes.

There might, of course, be other options as well.

Regards,
Terje