OTA-DFU sometimes erases the application data unexpectedly

RE: OTA-DFU sometimes erases the application data unexpectedly

tesc — Thu, 24 Sep 2020 13:26:57 GMT

Hi,

We now have a patch for the newly found issue that may lead to no swap pages in FDS:

One file need to be changed, and the required change is as follows:

In <sdk folder>/components/libraries/fds/fds.c, in the function init_execute(), in the switch case FDS_OP_INIT_PROMOTE_SWAP, remove the lines:

            // When promoting the swap, keep the write_offset set by pages_init().
            ret = page_tag_write_data(m_swap_page.p_addr);

In the same switch case, add the following lines right before the break:

            // Promote the old swap page to data, but do this at the end
            // because we can re-enter this function; we must update have
            // updated the page in RAM before that.
            ret = page_tag_write_data(p_old_swap);

So that the full case FDS_OP_INIT_PROMOTE_SWAP now looks like this:

        case FDS_OP_INIT_PROMOTE_SWAP:
        {
            p_op->init.step       = FDS_OP_INIT_TAG_SWAP;

            uint16_t const         gc         = m_gc.cur_page;
            uint32_t const * const p_old_swap = m_swap_page.p_addr;

            // Execute the swap.
            m_swap_page.p_addr = m_pages[gc].p_addr;
            m_pages[gc].p_addr = p_old_swap;

            // Copy the offset from the swap to the new page.
            m_pages[gc].write_offset = m_swap_page.write_offset;
            m_swap_page.write_offset = FDS_PAGE_TAG_SIZE;

            m_pages[gc].page_type = FDS_PAGE_DATA;

            // Promote the old swap page to data, but do this at the end
            // because we can re-enter this function; we must update have
            // updated the page in RAM before that.
            ret = page_tag_write_data(p_old_swap);
        } break;

This fix is verified to work with nRF5 SDK v17.

RE: OTA-DFU sometimes erases the application data unexpectedly

keni3 — Thu, 17 Sep 2020 03:42:08 GMT

Hi Terje,

Thank you for your update.

I will wait for the fix you are working on! Glad to hear that there will be a cure to this issue.

About your question, I DO use the FDS API to whenever accessing the flash.

What I wrote may have been misleading because I used the word "flash page" instead of "flash record".

More specifically, I use fds_record_delete() to delete and clean the record and then use fds_record_write() to re-write the updated parameters. When deleting, fds_record_find() is used with the same FILE_ID and REC_KEY as set in the descriptor for fds_record_write().

Regards,

Keni

RE: OTA-DFU sometimes erases the application data unexpectedly

tesc — Wed, 16 Sep 2020 13:43:20 GMT

Hi,

Sorry for the very long delay from my side.

Yes, it answered the question, and it means the issues that you see are more likely to stem from unidentified wrongly handled corner cases in FDS.

Since my last reply, we have identified a potential issue that may leave the FDS pages without any swap page. It may be related, and I'll share the fix once we have a tested and verified solution.

After having reread the previous entries in this thread, I noticed the following step:

[quote user="keni3"](4) Delete the flash page where the parameters are stored and over-write with the updated values[/quote]

Does that mean you delete the flash page through other means than deleting the records through the FDS API? Deleting FDS flash pages manually may leave FDS in an invalid state. If you used FDS to store records, you should use the proper FDS API calls to delete or update those records as well. You should not write to the FDS flash pages in any other way than through using the FDS API, or else there is no guarantee to data integrity.

Alternatively, if it means this is a separate flash page from what is used for FDS, and both writing and erasing happens directly (not through FDS), then any issues there should not be related to FDS.

Regards,
Terje

RE: OTA-DFU sometimes erases the application data unexpectedly

keni3 — Fri, 28 Aug 2020 03:30:12 GMT

Hi Terje,

Sorry about the delay.

About your question regarding the power fail comparator, I asked my PCB designer and got the following answer:

We are not using any feature for the abnormal power detection.

The module is providing nRF52832 with 2.1V power generated from 3V battery by a linear regulator.

When this issue occurred the 2.1V power supply as well as 3V battery was maintained and so as during OTA-DFU.

It was observed that the supply power was never below VPOF (1.7V).

I hope this answers your question.

Thank you.

RE: OTA-DFU sometimes erases the application data unexpectedly

tesc — Mon, 24 Aug 2020 14:45:24 GMT

Hi,

I had another look at the pattern 2 case, and the "File ID" field for the record is 0xFFFF instead of 0x1111. In other words, it looks like that field was not written. (Flash is erased into a state of all 0xFF, and then the 0 bits can be written. To FDS, a File ID of 0xFFFF means "not a valid record, and by the way the rest of the flash page is empty." I do not recall to have witnessed this particular error situation before. It may of course be related, but I would like to look a bit more into it before I let it pass.

Do you, by any chance, use the Power fail comparator? When supply voltage falls below VPOF, NVMC operations (e.g. flash writing) will fail, silently. Flash is written one word at a time, and File ID together with CRC (which is 0xFFFF because it is not used) constitutes one word which is word aligned in flash. If the power fail comparator prevents that particular word from getting written, you would end up in that state.

Pattern 1, however, I have seen before. The page does start with the "FDS page" marker word, but the rest of the page is completely 0xFF. I do believe we have a patch for that, and I find it a bit strange that it is still an issue in nRF5 SDK v16. It arises from garbage collection getting interrupted by a reset at a particular point in the process. It should be handled by FDS recognizing the page as an FDS page and treat it the same as if it was completely empty. (That is, on initialization, to promote it to a Swap page if no other Swap page exist, or to a Data page if a Swap page exist already.) I am a bit surprised this was not patched in SDK v16. I will look into that as well.

Regards,
Terje

RE: OTA-DFU sometimes erases the application data unexpectedly

keni3 — Mon, 24 Aug 2020 07:35:13 GMT

Hi Terje,

Thank you very much for your detailed insight.

To share the information of what I have encountered, let me share what my program was doing:

(0) Initialize FDS

(1) Read 60 bytes of data from Flash, where several parameters are stored in sequence

(2) Check the parameters if they have any invalid values

(3)-(a) If the values look OK, do nothing and go to (5)

(3)-(b) If there is any parameter with invalid value, over-write with a default value

(4) Delete the flash page where the parameters are stored and over-write with the updated values

(5) Continue the initialization and the rest of the program

Before the fix was made, there was a wrong condition in checking the parameters and the program was always going through (3)-(b).

After fixing the condition, the program normally goes though (3)-(a), apparently reducing the probability of the issue from happening.

As the ultimate fix, as you have suggested, I should be removing the step (4) to avoid running garbage collection at start-up. I will consider how we can do it...

Another fact to share regarding this issue was that a simple reset or power cycling didn't trigger this issue. It only happened by the reset after Flash read by nRF Connect Programmer and/or DFU. It may suggest that this issue was related both reset and Flash access.

About the pattern #2, it remains to be mystery to me. As you suggested, it does look the data is normally stored, but the data couldn't be read by the program... As neither #1 or #2 happens any more after the fix, the factor to the issue may have been the same...

Regards,

Keni

RE: OTA-DFU sometimes erases the application data unexpectedly

tesc — Thu, 20 Aug 2020 11:38:59 GMT

Hi,

I had a more thorough look through the lines from the hex dump, and I see for instance that in one of the cases (pattern 1) there are two FDS data pages, but no FDS swap page. We have experienced some issues with FDS related to resets during garbage collection, although known issues should mostly have been fixed for SDK v16. In any case, I think this points in the direction of reset related issues.

Running garbage collection on startup (e.g. in initialization) is not a good idea, as resets often occur in close succession (e.g. battery powered with low power on battery leading to frequent brownouts, during automatic programming / testing situations, etc.) As garbage collection typically mean additional flash read/erase cycles, this contributes to flash wear. While garbage collection is designed not to break on reset, we did have some edge cases prior to SDK 16 where reset at a particular clock cycle during garbage collection would leave the device in a state similar to your "pattern 1" situation.

For "pattern 2", the first piece of data is {TAG_MAGIC, TAG_DATA}, no? Followed by the beginning of a record?

:10500000 DEC0ADDE FE011EF1 22220A00FFFFFFFF1F
# @00075000 (16 bytes)

Regards,
Terje

RE: OTA-DFU sometimes erases the application data unexpectedly

tesc — Thu, 20 Aug 2020 11:08:07 GMT

Hi,

Without knowing the details of the bug that you fixed, I cannot tell if it could be the cause or not. It is not unreasonable that fds_record_find() returns FDS_ERR_NOT_FOUND if the record that you search for gets deleted and rewritten close in time to that. Permanent loss of records _might_ be explained by erasing/rewriting records, if for instance resets occurs so that records are deleted, then reset happens before the record is written again.

There has been some other reports of strange behavior after DFU after initial programming was done with the nRF Connect Programmer app. It does seem to be related to resets. From what I understand, when programming one or multiple hex files using the nRF Connect Programmer app, there may be one or more resets of the device as part of that process, which means the device may run briefly in the middle of being programmed. If that turns out to be the case, then that could be related to what you have been experiencing, in combination with erasing/writing records on startup.

We are currently looking into the Programmer app issue. For now I recommend using nrfjprog, part of nRF Command Line Tools, for programming the device, and especially so if using DFU.

Regards,
Terje

RE: OTA-DFU sometimes erases the application data unexpectedly

keni3 — Tue, 18 Aug 2020 11:25:21 GMT

Update from my investigation on this issue.

I have checked the error returned from fds_record_find() at the initialization process and found out that it was returning FDS_ERR_NOT_FOUND error.

Also I found out that there was a bug in my initialization process where FDS delete and write accesses are made every time my application is initialized.

After removing this bug, the issue doesn't seem to occur any more as long as I tried so far.

Do you think this was the cause? If so, what could be causing the FDS_ERR_NOT_FOUND error and why was it happening only once in a few times?

Since I'm not yet 100% sure if the problem is really resolved, I would like to know what was going on in detail to nail down the real cause.

RE: OTA-DFU sometimes erases the application data unexpectedly

keni3 — Tue, 18 Aug 2020 01:15:10 GMT

Hi Terje,

Thank you so much for your advice.

About the SDK, we are using SDK ver16.0.0.

For the hex dump, I was using nRF Connect's programmer to view the changes of flash before and after the issue occurs.

As I checked the inside, I could see that the data was completely lost. I also noticed that this issue tend to happen more often when I tried to read the flash memory by using nRF Connect's programmer right after the DFU completed. So it may be the issue of the FDS usage at initialization of the program.

Please see below for what I saw from the flash:

The application is freshly written (Before the issue occurs)

:020000040007F3

:10500000DEC0ADDEFE011EF122220A0011110000F9
# @00075000 (16 bytes)
# {TAG_MAGIC, TAG_DATA, 0x000A2222, 0x00001111}

:105010000200000001000000080000001100000074
# @00075010 (16 bytes)

:10502000FFFF0000FFFF0F00FFFF0F000200000066
# @00075020 (16 bytes)

:0C5030000100000054050000000000001A
# @00075030 (12 bytes)

:08600000DEC0ADDEFF011EF160
# @00076000 (8 bytes)
# {TAG_MAGIC, TAG_SWAP}

:08700000DEC0ADDEFE011EF151
# @00077000 (8 bytes)
# {TAG_MAGIC, TAG_SWAP}

The application data (0x12345, 0x67890) is stored using FDS

:020000040007F3

:085000 00 DEC0ADDE FF011EF1 70
# @00075000 (8 bytes)
# {TAG_MAGIC, TAG_SWAP}

:10600000DEC0ADDEFE011EF122220A0011110000 E9
# @00076000 (16 bytes)
# {TAG_MAGIC, TAG_DATA, 0x000A2222, 0x00001111}

:106010000500000001000000080000001100000061
# @00076010 (16 bytes)
# {0x00000005, 0x00000001, 0x00000008, 0x00000011}

:10602000FFFF0000452301009078060002000000F9
# @00076020 (16 bytes)
# {0x0000FFFF, 0x00012345, 0x00067890, 0x00000002}
#              ~~~~~~~~~~  ~~~~~~~~~~

:0C6030000100000054050000000000000A
# @00076030 (8 bytes)
# {0x00000001, 0x00000554, 0x00000000}

:08700000DEC0ADDEFE011EF151
# @00077000 (8 bytes)
# {TAG_MAGIC, TAG_DATA}

The issue occurred (Pattern 1)

:020000040007F3

:08500000DEC0ADDEFE011EF171
# @00075000 (8 bytes)
# {TAG_MAGIC, TAG_DATA}

:04600000DEC0ADDE73
# @00076000 (4 bytes)
# {TAG_MAGIC}

:08700000DEC0ADDEFE011EF151
# @00077000 (8 bytes)
# {TAG_MAGIC, TAG_DATA}

The issue occurred (Pattern 2)

:020000040007F3

:10500000DEC0ADDEFE011EF122220A00FFFFFFFF1F
# @00075000 (16 bytes)

:10501000080000000100000008000000110000006E
# @00075010 (16 bytes)

:10502000FFFF000045230100907806000200000009
# @00075020 (16 bytes)

:0C5030000100000054050000000000001A
# @00075030 (16 bytes)

:08600000DEC0ADDEFF011EF160
# @00076000 (8 bytes)
# {TAG_MAGIC, TAG_SWAP}

:08700000DEC0ADDEFE011EF151
# @00077000 (8 bytes)
# {TAG_MAGIC, TAG_DATA}

Any suggestion/advice would be helpful...!

Regards,

Keni

RE: OTA-DFU sometimes erases the application data unexpectedly

tesc — Mon, 17 Aug 2020 09:21:25 GMT

Hi,

Do you have a flash dump from a device where this has happened? (Use the command "nrfjprog --readcode --readuicr dump.hex" to dump flash contents (including UICR registers) to a hex file named "dump.hex".) It should hopefully give some clues about what has gone wrong in flash. Especially if there has been written entries to all of the FDS flash pages before the update. If you also have the flash dump from before the failing DFU, that could be even better.

If you use the debug bootloader, which has logging, that log from the update might also give some clues. Please note however that the debug bootloader is larger in size than the non-debug bootloader, so the location of application data will differ.

What version of the nRF5 SDK are you using?

Regards,
Terje