RE: FDS read non sequential sdk 17

tesc — Fri, 28 May 2021 14:08:59 GMT

Hi,

If you have stored all of the data with the same File ID + Record Key, then searching for that combination of File ID + Record Key will give you the records in arbitrary order.

On a fresh system where no records has been deleted and there has not been garbage collection, the records should mostly be in the same order as they were written, but there is no guarantee.

The "spike" that you see on the value sets on lines 261 through 266 in the csv, I assume, is a result of the following:

* The first 260 values were placed in the first FDS flash page, in records that were stored consecutively in flash.

* The 261th value (on line 267 in the csv) did not fit in the remaining space of the first FDS flash page, and was placed at the beginning of the second FDS flash page. Records in FDS cannot span multiple flash pages. The same happens for the following records (containing more values than can fit in the remaining space in the first FDS flash page.)

* Then, after the values on line 602 in the csv, a shorter record (with the values on lines 261 through 266) is to be written, and that record is short enough to fit in the remaining free space at the end of the first FDS flash page. Therefore it gets put there.

* The values after that are all in records that are too big to fit in the remaining gap at the end of the first FDS flash page, and so they are all placed consecutively in the second FDS flash page.

* When you read back the records, you get them in the order they are physically stored in flash.

Please note that although this is what happens with the current implementation of FDS, it is not safe to assume that this will always be how it happens. What is guaranteed by FDS, is that records are stored in flash, and that they can be searched and found using either File ID, Record Key, or a combination of both. The physical ordering in flash, and the order which they are found during search may change, and will usually change after garbage collection (as records are moved around.)

There are some options how to get the ordering correct:

Option 1: Order the data according to the timestamps.

Option 2: Use the same File ID for all the records, but use a different (increasing) Record Key, then:

2 a) Search for File ID when retrieving data, and sort the segments according to the Record Key. Or:

2 b) Keep track of which record was last read, and search for File ID + Record Key combinations, with increasing Record Key for each new record searched for.

Note for option 2 that Record Keys must be in the range 0x0001 through 0xBFFF. Record Keys outside of that range are reserved by the system (0x0000) or by Peer Manager (0xC000 through 0xFFFF.) This means you can reach 0xBFFF in the index (49151 in decimal) before you must go back to 0x0001 again.

Option 3: Add a serial number to each record, that gets incremented by one every time you store a new record, and use that for the ordering (instead of using the Record Key or the timestamps.) This may be an option if you don't want to manually handle the overflow behaviour of Record Keys, and expect to get more than 49150 records. Just choose a large enough integer data type to never run out of indexes.

There might, of course, be other options as well.

Regards,
Terje