FDS - missing swap page

RE: FDS - missing swap page

Rick N — Thu, 31 Aug 2023 15:54:21 GMT

My workaround was to check the flash pages prior to calling `fds_init`. If there were no swap pages (`FDS_PAGE_SWAP`) and no blank pages (`FDS_PAGE_ERASED` or `FDS_PAGE_UNDEFINED`) I would perform a recovery. Recovery would either erase a single data page, if a data page was blank (just header), or it would erase all flash storage pages (last resort, data is lost, but the MCU will recovery). Also, if a page is `FDS_PAGE_UNDEFINED`, then I would erase that page because it is unknown what is on that page and it cannot be trusted.

`fds_init` takes care of the rest.

RE: FDS - missing swap page

Remco Poelstra — Thu, 31 Aug 2023 11:32:11 GMT

Hi Rick N,

I'm experiencing the same problem with the FDS and I really like your take on solving this problem.

Do you perhaps have a more worked-out example of your solution?

Thanks in advance.

Kind regards,

Remco Poelstra

RE: FDS - missing swap page

Rick N — Thu, 08 Apr 2021 16:04:01 GMT

I am having the same issue. I have not been able to reliably reproduce it, however, in my case at least one of the DATA pages is empty. This is a recoverable state as a blank data page can be erased and tagged as a SWAP page.

I suggest adding a new enum value of PAGE_DATA_EMPTY

enum
{
    PAGE_ERASED     = 0x1,  // One or more erased pages found.
    PAGE_DATA       = 0x2,  // One or more data pages found.
    PAGE_SWAP_CLEAN = 0x4,  // A clean (empty) swap page was found.
    PAGE_SWAP_DIRTY = 0x8,  // A dirty (non-empty) swap page was found.
    PAGE_DATA_EMPTY = 0x10, // One or more empty data pages found.
};

and a new enum to fds_init_opts_t

    // The filesystem only contains data pages, but at least one data page is empty.
    // It is likely that the device powered off during GC. It is safe to discard (erase)
    // a empty data page, since there is nothing stored there.    
    DISCARD_EMPTY_DATA             = (PAGE_DATA | PAGE_DATA_EMPTY),

Handling this in a similar manner as DISCARD_SWAP (erase empty data page then init that page as a swap), will produce the same result.

RE: FDS - missing swap page

Edvin — Wed, 21 Oct 2020 14:56:58 GMT

Is there a reliable way of reproducing this?

BR,

Edvin

RE: FDS - missing swap page

Jason Wright — Wed, 21 Oct 2020 14:25:42 GMT

I have also experienced this issue with SDK 17.0.2. On one device calls to fds_init during boot returned the NO_SWAP error. Looking at the memory contents all the FDS pages were tagged as DATA.

RE: FDS - missing swap page

Min-Joon Kim — Fri, 21 Aug 2020 08:35:36 GMT

Edvin,

The flash dump was very simple.

All the three pages were tagged as data page.

Two pages had no record.

One page had some records that were very normal as I wrote.

When I changed the tag of one data page having no record to swap,

it worked well.

Thanks.

Best regards,

Min-Joon Kim.

RE: FDS - missing swap page

Edvin — Fri, 21 Aug 2020 07:33:49 GMT

Min-Joon Kim:
Please create a new ticket and upload the flash dump in that ticket.

[quote user="Min-Joon Kim"] (I couldn't dump the hex file before fds_init().)[/quote]

No need. Just dump it after the fds_init() that returns "no swap page". Perhaps it gives some information as to why it can't initialize.

BR,

Edvin

RE: FDS - missing swap page

Min-Joon Kim — Fri, 21 Aug 2020 05:33:54 GMT

Dear Edvin,

I've upgraded the SDK and tested.

Unfortunately, I still have the same problem, the no swap page issue.

The test procedure was as follows:

1. I run a code doing "write and garbage collection" continuously.

2. During the process, I turned off the power.

3. After a while, turned on.

4. Then, from time to time, I found "no swap page" after fds_init().

(I couldn't dump the hex file before fds_init().)

5. As you know, if there is no swap page, fds_init() looks not fully successful and we cannot access to FDS.
After that, bricked at the next boot.

Best regards,

Min-Joon Kim.

RE: FDS - missing swap page

Edvin — Tue, 29 Oct 2019 07:16:17 GMT

Hello Wojciech,

Thank you for coming back to us. I am sorry I didn't reach out, but it is hard to keep track of old cases with no activity. The bug (the cause) was actually found some time before the release of SDK16. The fix looks quite similar (if not identical). You can compare it to the fds.c file in SDK 16.

Best regards,

Edvin

RE: FDS - missing swap page

Wojciech Jasko — Fri, 25 Oct 2019 10:33:43 GMT

Hi,

Good news:) I have a way to reproduce it.

Given we have interrupted the GC procedure (power off) in the following state:

| page_address | page_type  |
| 0xEF000      | erased     |
| 0xF0000      | data       |
| 0xF1000      | swap_dirty |
| 0xF2000      | data       |
| 0xF3000      | data       |

When we initialized fds again it performs the following actions (PROMOTE_SWAP_INST):

tag 0xF1000 as the data
tag 0xF0000 as the swap (this fails due to NAND flash)
tag 0xEF000 as the data

When the erased page is initialized before swap page then m_gc.cur_page is not initialized correctly. This is my proposition:

            case FDS_PAGE_SWAP:
            {
                if (swap_set_but_not_found)
                {
                    m_pages[page].page_type    = FDS_PAGE_ERASED;
                    m_pages[page].p_addr       = m_swap_page.p_addr;
                    m_pages[page].write_offset = FDS_PAGE_TAG_SIZE;

                    m_gc.cur_page = page; // FIX
                    page++;
                }

                m_swap_page.p_addr = p_page_addr;
                // If the swap is promoted, this offset should be kept, otherwise,
                // it should be set to FDS_PAGE_TAG_SIZE.
                page_scan(p_page_addr, &m_swap_page.write_offset, NULL);

                ret |= (m_swap_page.write_offset == FDS_PAGE_TAG_SIZE) ?
                        PAGE_SWAP_CLEAN : PAGE_SWAP_DIRTY;
            } break;

It is fixing this case, but I am not sure if it does not break anything. Please, could you review it for me?

RE: FDS - missing swap page

Edvin — Mon, 18 Mar 2019 08:39:46 GMT

I understand. However, I would say that anything is better than just passing the NO_SWAP into APP_ERROR_CHECK, which will preform a reset, run into the same check after reboot, and continue in that loop.

Of course, not knowing what caused the issue in the first place is a bit worrying. If it is caused by a power cut during a certain point of Garbage Collection (GC), then you will probably have the situation where you have a data page that is blank. (However, I don't see any specific reason why this would have happened due to a power cut during GC.)

But if you want a workaround, it needs to end up getting a new swap page. Maybe you can modify the GC function to copy all valid (not dirty/marked as deleted) records from e.g. the first data page to the current data page. That is, have a special GC function that empties all the records on the first data page. When finished, delete this page and mark it as a swap page.

RE: FDS - missing swap page

Wojciech Jasko — Wed, 13 Mar 2019 19:47:10 GMT

Hi,
I started to patch fds last month as a result of these tests. These devices are not affected by my changes. I am pretty sure that your solution will work as a workaround. Anyway, there is still a bug somewhere. I am afraid that it may manifest itself in a different way (for example some operation on 0x0007D000 might be processed before reboot). In such case, this workaround will not be working.

All records seem to be fine. Actually, I have erased this page using jlink and after reset device started to work.

RE: FDS - missing swap page

Edvin — Wed, 13 Mar 2019 17:16:28 GMT

Helllo,

I have been looking at this ticket a lot today, together with Terje, who is handling one of the other tickets.

I have looked through a lot of documentation, done some testing by erasing pages, writing the magic word, but no tag and so on, but I have not found a way to reproduce ending up with all data pages and no swap page, which is what you are facing with this node.

So I have to ask, have you done any changes to the FDS module before programming these devices? Any fixes that were implemented beforehand that can affect the FDS behavior in any way?

That being said, it looks like somehow, the swap page has been tagged as a data page. In this page it is the first FDS page, at 0007D000.

A suggested patch could be that if you end up with the NO_SWAP return from fds_init(), anything is better than just falling into APP_ERROR_CHECK(NO_SWAP);

As a workaround you can go through your FDS pages, and look for a page that contains the magic word and the data tag but no other data (all 0xFFFF). If you find one of these in the NO_SWAP state, it would be safe to assume that this page can be deleted. You can delete this page, then run fds_init() again. Alternatively, an NVIC_SystemReset();

If the FDS starts up with one empty page and the rest data pages, it will make the empty page to a SWAP page (I have tested this).

What would be interesting is to see what happens if you erase this first page (from 0x0007D000) and reboot the application. Do you get any other errors at this point? Are the rest of the records valid, or do you get any information saying parts of the rest of the FDS are corrupted?

RE: FDS - missing swap page

Wojciech Jasko — Wed, 13 Mar 2019 10:33:16 GMT

Sorry, this is probably my mistake. I sent you too much.
We use 5 fds_pages. FLASH_FDS (rw) : ORIGIN = 0x0006F000, LENGTH = 0x5000
Address 0x74000 is bootloader's data.

I have uploaded decoded fds_dump https://drive.google.com/file/d/1R7gSYENEmZZ3l5Fyh95nZNbv7FaCW6TW/view?usp=sharing. Sorry for this mistake.

This issue is not connected with my other 2 tickets. One of them will result in corrupted Record ID and the second with FDS_PAGE_UNDEFINED. In both cases, I believe that I found a root cause and I need more review than help.

In this case, we have 5 times FDS_PAGE_DATA. I do not know why it happened.

This corrupted node had exactly 7 runs of GC. Usually tested nodes have between 4-10 runs of GC.

RE: FDS - missing swap page

Edvin — Wed, 13 Mar 2019 08:55:48 GMT

Hello Wojciech,

I see that there are some regions missing from your flashdump. Also, I see that the last data page is moved to a weird location, not page aligned. Do you have any idea of what is present in the flash in the area that is missing?

That is between 0x00073FFF and 0x0007544B. It should have been included "between" line 1025 and 1026 in the flashdump. especially interesting is what that is present on 0x00074000, since this should be the start of a page.

How many FDS pages do you use in your application? if there are more than 5, then 0x00075000 is also interresting. Actually, for N flash pages 0x0007(N+1)000 is interresting. The start of each page. Also, as I mentioned, I don't understand why DEADC0DE (DEC0ADDE) is present in a non-page-aligned place. Do you have a record where the actual data is DEADC0DE?

I see the internal ticket that Jørgen created from your original post here on DevZone. There is still no conclusion. I also see from one of the other tickets that you created that you have some theory on why you may have a missing swap page. It is an interesting theory, but without seeing what in the missing part of the flashdump, it is difficult to say whether this actually is the issue in your case.

Also, you say that the chip has been running for a few months. Do you have any estimation on how much and how often you write and erase your flash? Or in other words, how often you perform the Garbage Collection?

Best regards,
Edvin