FDS wear-leveling not working

RE: FDS wear-leveling not working

Adrian Eggenberger — Mon, 09 Jan 2017 15:44:44 GMT

I know that there is a risk for power failure when moving records to other pages. In the solution we used previously this was ensured by a integrity field in the record header. The system could recover the data integrity after a power failure.

I agree that unnecessairy GC action should be avoided to get best wear leveling performance.

RE: FDS wear-leveling not working

Hung Bui — Mon, 09 Jan 2017 15:31:37 GMT

The main point of doing a gc in our implementation is to rearrange the records in the page to make biggest free space on the same page. Not to have more free empty page. We try to keep the complexity at low level.

There is one small risk with your approach is to keep track of the moving process, how to recover when there is a power failure occur when gc is happening, to avoid duplicated data ?

From our point of view gc should only be called when the flash is nearly full (using fds_stat) this to make sure wear leveling will be effective and to avoid unnecessary flash activity. But I do agree this can be an issue if there are lots of data coming and we need to be quick to store them before overloading the RAM. We don't want to do gc at that point. But this can be avoided by checking with fds_stat.

RE: FDS wear-leveling not working

Adrian Eggenberger — Mon, 09 Jan 2017 11:38:09 GMT

Your suggestion was my first though as well. I have a simulation of our device where I can see the flash usage. This allows a good view to how the FDS behaves. The problem with your proposed way is that the GC process of the FDS doesn't create empty pages in most cases. If the cleaned page has still some valid records in it the cleaned up page will contain the same record. This would end in too much garbage collection.

We used previously an own implementation for the FDS management that had a different approach for GC. The records of the page to clean up were moved to normal valid pages one by one. As soon as all records are moved the page was erased. A swap page was not necessary with this approach. With this implementation the strategy was to wait until no full page was available. The GC process checked what page has most wasted space and clean it up.

I understand and like the generic thinking when creating your libraries. With the FDS this might be a bit tricky as the generic way should include a lot of possible usages of the FDS in term of file size and other parameters. A guideline or a recommendation on "How to use" the FDS incl. its garbage collection would be really helpful. The documentation in the fds.h doesn't provide the needed information.

RE: FDS wear-leveling not working

Hung Bui — Mon, 09 Jan 2017 10:23:48 GMT

I see. This could be an use case we should think about. When we make our library, we usually choose to take generic solution just to make the library simple and easy to use. Other customer may expect a solution to only call gb when it's full to avoid unexpected CPU occupation from the fds.

My suggestion for you is maybe only do gb when there is no free page . Meaning when you first moved to the last free page, you start the gb collection, this will avoid using only 2 pages for swap page.

RE: FDS wear-leveling not working

Adrian Eggenberger — Fri, 06 Jan 2017 17:13:20 GMT

Hi Hung Your guess is correct. Our systems have mostly a lot of idle time. I personally prefer the strategy to clean up to flash storage when idle time is available to avoid the FDS is full case normally. I'm aware that it can happen, but it shouldn't be the regular case.

RE: FDS wear-leveling not working

Hung Bui — Fri, 06 Jan 2017 16:00:30 GMT

Hi Adrian,

Could you explain a little bit more on what you want to achieve with your strategy ? I guess you want to take advantage of the CPU idle time so you can do gb collection instead of waiting till the fds system is full and flash storing will have to wait for gb collection?

RE: FDS wear-leveling not working

Adrian Eggenberger — Fri, 06 Jan 2017 14:42:33 GMT

It could be correct that the wear leveling is ok if the GC is only called if the FDS is full. In my case I modified the FDS to follow a different strategy. I've added a function to the FDS that can be called if the system is idle. This function will check if it makes sense to cleanup one of the pages based on various thresholds. If the function returns true the classic fds_gc() may be called to cleanup the page.

RE: FDS wear-leveling not working

emdi — Mon, 19 Dec 2016 16:58:45 GMT

Hi,

if you run garbage collection when the flash is full then the swap page will be moved around as you would expect and wear leveling will be maximized.

RE: FDS wear-leveling not working

Adrian Eggenberger — Mon, 19 Dec 2016 15:36:48 GMT

Hi Hung In our solution the GC is called as soon as in the FDS the number of freeable_words exceeds a given threshold. This triggers a GC as soon as the system is idle. We are not really sure what is the best strategy for the GC call. Waiting until the error FDS_ERR_NO_SPACE_IN_FLASH is called seams also a bit odd in my opinion. What is your recommended way for calling GC?

RE: FDS wear-leveling not working

Hung Bui — Mon, 19 Dec 2016 10:09:49 GMT

Hi Adrian,

As I understand, the behaviour you described only happens if you call fds_gc() every time the first page is full. Actually I don't know how you can get an event when one page is full. We only have FDS_ERR_NO_SPACE_IN_FLASH event when all pages are full.

In the case when all pages are full, and fds_gc() is called by the main application, each page in the fds file system that got erased will be used as swap page.

So I think it won't be just 2 pages out of 4 will be used as swap page, but rotating around ?