Why are the records getting dirty in fds example

RE: Why are the records getting dirty in fds example

Embedded_Engineer — Thu, 02 Apr 2020 12:05:13 GMT

That was the case.

.data.length_words = (sizeof(four_word_cfg) + 3) / sizeof(uint32_t),

RE: Why are the records getting dirty in fds example

Jørgen Holmefjord — Thu, 02 Apr 2020 11:16:40 GMT

Are you sure you did only write one word? Note that the data.length_words parameter in the fds_record_t struct takes the number of words, not the number of bytes, to be written. If you use sizeof() or similar to get the size of the struct, you need to divide by 4.

I modified the FDS example to output stats.words_used and it seems to work correctly:

fds example:~$ stat
total pages:    3
total records:  2
valid records:  2
dirty records:  0
largest contig: 1022
freeable words: 0 (0 bytes)
words used:     12
fds example:~$ write 0xFF 0xFF TEST
writing record to flash...
file: 0xFF, key: 0xFF, "TEST", len: 4 bytes
<info> app: Event: FDS_EVT_WRITE received (NRF_SUCCESS)
<info> app: Record ID:  0x0003
<info> app: File ID:    0x00FF
<info> app: Record key: 0x00FF
fds example:~$ stat
total pages:    3
total records:  3
valid records:  3
dirty records:  0
largest contig: 1022
freeable words: 0 (0 bytes)
words used:     16

RE: Why are the records getting dirty in fds example

Embedded_Engineer — Thu, 02 Apr 2020 10:25:23 GMT

Refer the declaration below,

typedef struct
{
  uint32_t boot_count;
  
} minimal_configuration_t;

I am updating single word record in flash,

After record has been successfully updated I am comparing the words (stat.words_used) written in flash,

================================================================================
                  F L A S H   S T A T S  ( F D S )
================================================================================
| BEFORE | AFTER | DESCRIPTION
|-------------------------------------------------------------------------------
| 3      | 3     |   Pages Available.
| 0      | 0     |   Open records.
| 1      | 1     |   Valid records.
| 74     | 75    |   Dirty records (ready to be garbage collected).
| 7      | 7     |   Words reserved by fds_reserve().
| 680    | 687   |   Words written to flash.
| 1022   | 1022  |   Largest number of free contiguous words in file system.
| 662    | 669   |   Words that can be reclaimed by garbage.
| 0      | 0     |   Corrupted records.
================================================================================

Before: 680

After : 687

Difference = 7

Since the length of new record's header is 3 and length of the data i.e. boot_count is 1 word. Shouldn't the difference be 4 instead of 7?

RE: Why are the records getting dirty in fds example

Jørgen Holmefjord — Wed, 01 Apr 2020 13:29:33 GMT

That is almost correct. Each page contains a Page tag, which occupies the first two words. This limits the available words for each page to 1022. Also, note that each record has a 3-word header, which needs to be accounted for when calculating the available space for actual data.

RE: Why are the records getting dirty in fds example

Embedded_Engineer — Wed, 01 Apr 2020 13:07:10 GMT

Jørgen Holmefjord Thank you very much for the details.

[quote userid="14926" url="~/f/nordic-q-a/59746/why-are-the-records-getting-dirty-in-fds-example/242886"]You can call fds_stat() to get statistics about the available flash size[/quote]

Running stats yields,

================================================
        FLASH STATS (FDS)
================================================
3 pages Available.
0 open records.
1 valid records.
51 dirty records (ready to be garbage collected).
0 words reserved by fds_reserve().
472 words written to flash.
1022 largest number of free contiguous words in the file system.
459 words that can be reclaimed by garbage.
0 corrupted records

From flash_bounds_set() I see flash size as 0x3000,

flash_size = (FDS_PHY_PAGES * FDS_PHY_PAGE_SIZE * sizeof(uint32_t))

= 3 x 1024 x 4

= 12kB

FDS_VIRTUAL_PAGES is 3, where FDS_VIRTUAL_PAGE_SIZE is 1024. As you mentioned one of the page will be used as SWAP page. So we have two pages out of three and by one page we mean we have 1024 WORDS i.e. 4096 bytes. Is this Correct?

From the stats we see 472 words written to flash. Which leaves us with 2048 - 427 words?

RE: Why are the records getting dirty in fds example

Jørgen Holmefjord — Wed, 01 Apr 2020 09:41:37 GMT

Hi,

The purpose of creating a new record and then invalidating the old one is not solely to prevent data loss due to power loss, this is also a more effective way to store data in flash. Flash can only be erased a whole page at a time, meaning that deleting a record and writing a new one back to the same location would require the entire flash page to be erased in between. Erasing a flash page takes a long time, and the flash has a limited number of write/erase cycles (endurance) before it starts to fail. Spreading the writes out over the entire page will provide basic flash wear leveling, increase the lifetime of the flash.

The maximum data you can store before running garbage collection depends on the number of reserved pages for FDS (FDS_VIRTUAL_PAGES). Note that one of the pages (SWAP page) will be reserved for garbage collection. During garbage collection, valid records are copied to the SWAP page before the DATA page is erased. The old SWAP page is then promoted to a DATA page, while the old DATA page will become the new SWAP page.

You can call fds_stat() to get statistics about the available flash size. It is not recommended to run garbage collection too often, as this will increase the wear on flash. We recommend calling it only when FDS has filled all available space, or when it is getting close to full.

Best regards,
Jørgen

RE: Why are the records getting dirty in fds example

Embedded_Engineer — Wed, 01 Apr 2020 08:25:18 GMT

After going through fds functionality It seems that when a record is to be updated (fds_record_update), a new record is created and old record is invalided solely to prevent loss of data due to power loss in the middle of operation. A new record descriptor is returned for updated record.

Thus frequent updates will consume flash space and its a good idea to clean it up using garbage collection.

1. The size of record varies depending upon the data stored. What is the maximum data that can be stored after which garbage collection becomes necessary? Is it FDS_VIRTUAL_PAGE_SIZE per page?