FDS Limit & max data size calculation

[Salman]

I am working on an application in which I need to store large amount of data in the flash. I have successfully integrated FDS Module in my application and I am able to store the data how ever I am a bit confuse regarding the storage limit.

My Questions are:

1) Does one record key data size is same as the page size which is 1024 words, if so then it can store up to 1024 words and 4096 Bytes in one record Key. 

2) lets say I have 300 KB of empty space in the flash so according to my calculation I can store up to 70 Pages of page size 1024 word amount of data in the flash, which is around 280 KB of data, remaining I can leave for the Garbage collector or may be for future changes in the code. Is this calculations correct?

3) How do we keep track of how much data is stored in a page?

4) Restrictions on Key and ID say that if Peer Manager is used the Value of Record Key from 0xC000 to 0xFFFF can not be used, and File ID 0xFFFF can not be used. How ever it does not say how much space(in Bytes) is required for them. This is a bit confusing.

5) While using FDS function I dont need to worry about the pages the FDS module handles it internally. Is this correct? If it is correct how can I know the allocated space left for storage? or do I need to keep track of how much data I have stored and then the MAX limit will be 280 KB (according to my calculations above)?

[Edvin]

Hello,

1) Due to a bug in FDS, I recomend you to not store single records larger than 1024 bytes. It has to do with garbage collection that will move the records to a free page. you can have multiple records with the same key.

2) I don't think there is any limit to how many pages you can have, but you need 1 swap page that is used during garbage collection. Of course, more pages results in a longer process during garbage collection, but I don't think there is a hard limit anywhere. If you intend to have a bootloader, remember that you must set aside flash for that as well in the top of the flash. In addition, if you intend to have a BLE bootloader, and you want the possibility to update the softdevice and bootloader itself, you must leave room for one extra bootloader+softdevice in flash, because these needs to fit next to the existing bootloader and softdevice during an update. The application can be updated after the SD+BL update, when the double SD+BL is replaced and the space is freed up.

3) You can check out the function fds_stat(), which will print some information on the status of the fds space. You can see how it is used in the SDK\examples\peripheral\flash_fds example. It will populate all the fds_stat_t parameters, which you can see in fds.h:

/**@brief   File system statistics. */
typedef struct
{
    uint16_t pages_available;   //!< The number of pages available.
    uint16_t open_records;      //!< The number of open records.
    uint16_t valid_records;     //!< The number of valid records.
    uint16_t dirty_records;     //!< The number of deleted ("dirty") records.
    uint16_t words_reserved;    //!< The number of words reserved by @ref fds_reserve().

    /**@brief The number of words written to flash, including those reserved for future writes. */
    uint16_t words_used;

    /**@brief The largest number of free contiguous words in the file system.
     *
     * This number indicates the largest record that can be stored by FDS.
     * It takes into account all reservations for future writes.
     */
    uint16_t largest_contig;

    /**@brief The largest number of words that can be reclaimed by garbage collection.
     *
     * The actual amount of space freed by garbage collection might be less than this value if
     * records are open while garbage collection is run.
     */
    uint16_t freeable_words;

    /**@brief Filesystem corruption has been detected.
     *
     * One or more corrupted records were detected. FDS will heal the filesystem automatically
     * next time garbage collection is run, but some data may be lost.
     *
     * @note: This flag is unrelated to CRC failures.
     */
    bool corruption;
} fds_stat_t;

It will say something about remaining space and number of pages, but it will not say how much space you have on each page. 

4) Check out the Record Layout on infocenter. You can see how much header data that is present per record (which is 12 bytes per record).  But it is correct as you say. The 0xC000 to 0xFFFF (FFFE actually) is reserved for the peer manager if you use this. And 0xFFFF is a blank (free) record. 

5) Yes. This is the main advantage of FDS. It will handle the internal addressing, page distribution and an even flash wear (flash write/erase cycles). As I mentioned, the fds_stat() gives you a status of the FDS at any point in time. The MAX limit will be set by the number of FDS pages you set in sdk_config.h.

I hope this answered some of your questions. 

Best regards,

Edvin

[Salman]

1) So due to bug in the FDS I can not use one record more then 1024 byte however the minimum size of one page is 1024 WORDS, does it means that the remaining 3072 bytes will not be used in that record and the space will be lost or FDS will going to use what ever number of bytes I write, and that will be memory consumed. In simple words if I use 1024 bytes out of 4096 bytes what will happen to the remaining 3072 bytes? Are the remaining bytes reserved and can not be used?

[Edvin]

Salman said:

So due to bug in the FDS I can not use one record more then 1024 byte however the minimum size of one page is 1024 WORDS, does it means that the remaining 3072 bytes will not be used

 No, that is not the case.

One flash page is an area in the flash that can hold multiple records. The limitation is that one record can't be larger than 1024 bytes, but you can have multiple records in one flash page. So you can either have 4 records with size 1024 (or a little smaller. There are some headers for each record. The header size is 10 bytes, which you can see here.

So as long as there is room for the next record on the same page as the previous, it will continue to write to that page. If there is e.g. 20 bytes left on a page (page 1), and you try to write a record of size 30, this will be written to the next page. If you after this want to write a record of size 20, then that will be written to page 1, which now will be full. 

FDS will handle all of this for you, so you don't need to worry about where the records are placed. And as I mentioned, you can use fds_stat() to check how much space you have used and how much space that is left at any time.

Best regards,

Edvin