FDS wear-leveling not working

[Adrian Eggenberger]

Hi

I've just done a few tests with the FDS system of the SDK12.2. My observation is, that the FDS doesn't properly use all assigned flash pages to reach a maximal lifetime.

I have a configuration with the space for the FDS is set to 4 pages. The FDS will use one page as SWAP and the other three pages as DATA pages. My code is writting approximatly 30 records a ~30Bytes to the FDS and randomly updates these records. Once one page is nearly full the garbage collector copies the valid data to the SWAP page and converts the previous SWAP page into a DATA page and the erased page becomes the new SWAP page. Once the new data page is nearly full the valid records are copied back to the SWAP page and so on. In fact only two of the four pages are used.

In my opinion a good flash manager should use all assigned flash pages to reach the maximal possible lifetime. This will allow the developer to add more space to the FDS to increase lifetime.

The problem is based in the fds.c in the function "write_space_reserve" this function always starts at the same position in the flash page list.

Any thoughts how this could be improved?

Adrian

[Adrian Eggenberger]

Hi Hung Your guess is correct. Our systems have mostly a lot of idle time. I personally prefer the strategy to clean up to flash storage when idle time is available to avoid the FDS is full case normally. I'm aware that it can happen, but it shouldn't be the regular case.

[Hung Bui]

I see. This could be an use case we should think about. When we make our library, we usually choose to take generic solution just to make the library simple and easy to use. Other customer may expect a solution to only call gb when it's full to avoid unexpected CPU occupation from the fds.

My suggestion for you is maybe only do gb when there is no free page . Meaning when you first moved to the last free page, you start the gb collection, this will avoid using only 2 pages for swap page.

[Adrian Eggenberger]

Your suggestion was my first though as well. I have a simulation of our device where I can see the flash usage. This allows a good view to how the FDS behaves. The problem with your proposed way is that the GC process of the FDS doesn't create empty pages in most cases. If the cleaned page has still some valid records in it the cleaned up page will contain the same record. This would end in too much garbage collection.

We used previously an own implementation for the FDS management that had a different approach for GC. The records of the page to clean up were moved to normal valid pages one by one. As soon as all records are moved the page was erased. A swap page was not necessary with this approach. With this implementation the strategy was to wait until no full page was available. The GC process checked what page has most wasted space and clean it up.

I understand and like the generic thinking when creating your libraries. With the FDS this might be a bit tricky as the generic way should include a lot of possible usages of the FDS in term of file size and other parameters. A guideline or a recommendation on "How to use" the FDS incl. its garbage collection would be really helpful. The documentation in the fds.h doesn't provide the needed information.

[Hung Bui]

The main point of doing a gc in our implementation is to rearrange the records in the page to make biggest free space on the same page. Not to have more free empty page. We try to keep the complexity at low level.

There is one small risk with your approach is to keep track of the moving process, how to recover when there is a power failure occur when gc is happening, to avoid duplicated data ?

From our point of view gc should only be called when the flash is nearly full (using fds_stat) this to make sure wear leveling will be effective and to avoid unnecessary flash activity. But I do agree this can be an issue if there are lots of data coming and we need to be quick to store them before overloading the RAM. We don't want to do gc at that point. But this can be avoided by checking with fds_stat.

[Adrian Eggenberger]

I know that there is a risk for power failure when moving records to other pages. In the solution we used previously this was ensured by a integrity field in the record header. The system could recover the data integrity after a power failure.

I agree that unnecessairy GC action should be avoided to get best wear leveling performance.