FDS and write/erase flash cycles

RE: FDS and write/erase flash cycles

dgerman — Tue, 02 Feb 2021 12:50:02 GMT

A.P. Thank you. This is exactly what I wanted. Please check back in a day or so and I'll open to piece together something. Maybe even a "stand alone" that others could use to add to contribute data on this.

RE: FDS and write/erase flash cycles

A.P — Tue, 02 Feb 2021 10:23:49 GMT

Hi dgerman!
Not sure why, I wasn't able to answer you directly before, but now I am.
1. We first tested using FDS, but realized it was slowing us down, we moved to a lower level and used nrf_fstorage_read/write/erase functions.
2. All subsequent reads returned an incorrect value, yes. However, I suspect that if you happen to write a '1' to the bit that is stuck as '1', you'll get a correct value read back to you. so you've got a 50% chance to be ok ;)
3. I've added a skeletal structure of our tests in a separate comment. Hope it helps.

RE: FDS and write/erase flash cycles

A.P — Tue, 02 Feb 2021 10:21:07 GMT

Hi dgerman!

I'm afraid I haven't quite understood how to replay to specific messages, so I"m just pressing anywhere that says 'reply'. I hope we can be able to piece together the message flow postmortem.
At any rate, I've pieced together the portions of the test which are relevant to the Flash test as best I could. I'm not able to share our code as is, since it belongs to the company etc.
We first tested using FDS, but realised it was slowing us down, we moved to a lower level and used nrf_fstorage_read/write/erase functions.

#define IMG_SIZE 4096

__ALIGN(32) uint8_t img[IMG_SIZE] = {0};
__ALIGN(32) uint8_t img_copy[IMG_SIZE];

#define FLASH_START_ADDR 0x30000
bool write_finished = false, erase_finished = false;
void callback(nrf_fstorage_evt_t *p_evt)
{
    if (p_evt->id == NRF_FSTORAGE_EVT_WRITE_RESULT) {
        NRFX_LOG_INFO("flash write result event");
        write_finished = true;
    }
    if (p_evt->id == NRF_FSTORAGE_EVT_ERASE_RESULT) {
        NRFX_LOG_INFO("erase write result event");
        erase_finished = true;
    }
}
NRF_FSTORAGE_DEF(nrf_fstorage_t my_instance) = {
    .evt_handler = callback,
    .start_addr  = FLASH_START_ADDR,
    .end_addr    = FLASH_START_ADDR + 4096,
};


bool write_to_flash()
{
    ret_code_t rc = nrf_fstorage_write(&my_instance,     /* The instance to use. */
                                       FLASH_START_ADDR, /* The address in flash where to store the data. */
                                       img,              /* A pointer to the data. */
                                       IMG_SIZE,         /* Lenght of the data, in bytes. */
                                       NULL              /* Optional parameter, backend-dependent. */
    );
    if (rc == NRF_SUCCESS) {
        NRFX_LOG_INFO("flash write success");
        return true;
    } else {
        NRFX_LOG_ERROR("flash write failure");
        return false;
    }
}

bool read_from_flash()
{
    ret_code_t rc = nrf_fstorage_read(&my_instance,     /* The instance to use. */
                                      FLASH_START_ADDR, /* The address in flash where to read data from. */
                                      img_copy,         /* A buffer to copy the data into. */
                                      IMG_SIZE          /* Lenght of the data, in bytes. */
    );
    if (rc == NRF_SUCCESS) {
        NRFX_LOG_INFO("flash read success");
        for (int i = 0; i < IMG_SIZE; i++) {
            if (img[i] != img_copy[i]) {
                NRFX_LOG_ERROR("flash read unequal at index %d", i);
                return false;
            }
        }
        return true;
    } else {
        NRFX_LOG_ERROR("flash read success");
        return false;
    }
}

bool erase_from_flash()
{
    ret_code_t rc = nrf_fstorage_erase(&my_instance,     /* The instance to use. */
                                       FLASH_START_ADDR, /* The address of the flash pages to erase. */
                                       1,                /* The number of pages to erase. */
                                       NULL              /* Optional parameter, backend-dependent. */
    );
    if (rc == NRF_SUCCESS) {
        NRFX_LOG_INFO("flash erase success");
        return true;
    } else {
        NRFX_LOG_ERROR("flash erase failure");
        return false;
    }
}


/**@brief Function for application main entry.
 */
int main(void)
{
	  // Initialize.
    nrf_fstorage_init(&my_instance,     /* You fstorage instance, previously defined. */
                      &nrf_fstorage_sd, /* Name of the backend. */
                      NULL              /* Optional parameter, backend-dependant. */
    );
  
    log_init();

    // write_to_flash();
    // read_from_flash();
    // erase_from_flash();

    NRFX_LOG_INFO("%s Starting: %s %s (%s)\n", __func__, DEVICE_NAME, FIRMWARE_REV, MODEL_NUM);
    NRF_LOG_FLUSH();

	//TODO: write 0 value to whereever you save your results. I'll just call it LOG from now on.

    bool     write_failed = false, gc_failed = false, read_failed = false;
    uint32_t write_cnt = 0, gc_cnt = 0, read_cnt = 0, fail_run_cnt = 0;

    bool use_170_and_not_85 = true;
	//170 binary is 1010..
	// 85 binary is 0101..

    while (true) {
        // if something has failed, never change it. just write to LOG every once in a long while to make sure it is on
        if (write_failed || gc_failed || read_failed) {
            ++fail_run_cnt;
            if (fail_run_cnt < 3) {
                NRFX_LOG_ERROR("update fail LOG");
                //TODO: write to your LOG the reason you've failed and the count.
            }
        } else {
            // flip all the bits every flash write-gc-read attempt
            if (use_170_and_not_85) {
                for (uint32_t i = 0; i < IMG_SIZE; i++) {
                    img[i] = 170;
                }
                use_170_and_not_85 = false;
            } else {
                for (uint32_t i = 0; i < IMG_SIZE; i++) {
                    img[i] = 85;
                }
                use_170_and_not_85 = true;
            }
            ++write_cnt;
            write_finished = false;
            bool write_ret = write_to_flash();
            while (nrf_fstorage_is_busy(&my_instance) && !write_finished)
                ;
            if (write_ret) {
                ++read_cnt;
                bool read_ret = read_from_flash();

                if (read_ret) {
                    ++gc_cnt;
                    erase_finished = false;
                    bool gc_ret    = erase_from_flash();
                    while (nrf_fstorage_is_busy(&my_instance) && !erase_finished)
                        ;
                    if (gc_ret) {
                    } else {
                        NRFX_LOG_ERROR("flash gc #%llu failed!", gc_cnt);
                        gc_failed = true;
                    }
                } else {

                    NRFX_LOG_ERROR("flash read #%llu failed!", read_cnt);
                    read_failed = true;
                }

            } else {
                NRFX_LOG_ERROR("flash write #%llu failed!", write_cnt);
                write_failed = true;
            }

            if (write_cnt % 256 == 0) {
                //TODO: write to your LOG the the result thus far, as it hasn't failed yet.
            }
        }

        NRF_LOG_FLUSH();

        idle_state_handle();

        // nrf_delay_ms(10);

    } // end of for loop
}

This by no means is bug proof, nor have I tested that it compiles. Its just the skeletal structure of our testing.
I've added a 'TODO' comment where I saw relevant.

Please let me know if you find any problems in our logic, and do report on your findings once you have them.

good luck!

RE: FDS and write/erase flash cycles

dgerman — Mon, 01 Feb 2021 16:17:28 GMT

Was this test done using high level FDS. i.e fds_record_write , fds_record_find & fds_record_open ?

Does "All subsequent reads returned an incorrect value from the flash.' mean that once a failure occures no successful WRITES can be performed

RE: FDS and write/erase flash cycles

A.P — Thu, 28 Jan 2021 12:36:27 GMT

Hi!

[quote userid="4240" url="~/f/nordic-q-a/69935/fds-and-write-erase-flash-cycles/291774#291774"]Does this correspond to the number of erase cycles on one flash page?[/quote]

Yes it does.

[quote userid="4240" url="~/f/nordic-q-a/69935/fds-and-write-erase-flash-cycles/291774#291774"]I think you should account for that number when designing your product. E.g. by allocating as many flash pages as possible to improve wear leveling. [/quote]

Will do. In our app, with as many flash pages as possible, and the amount of data we expect to write to the flash, it is not realistic to expect the flash to reach the point of failure before the expected end of life for the product.
Should that change, we will simply verify the CRC on every write to the flash (write --> read --> verify CRC is correct) and notify when the flash has started to fail.
That should be enough for our purposes.

Thanks for the support!

BTW, if possible, I think we could benefit from a distribution graph, which I assume will be normal around X erase cycles, for the point at which the flash fails. This way we have both the absolute value (10,000) writes, and the average (X) so that we have a rough idea what to expect from the flash. That would require a few thousand devices to be sacrificed, which is outside the scope of what I'm willing to pay ;)

RE: FDS and write/erase flash cycles

Vidar Berg — Thu, 28 Jan 2021 12:23:01 GMT

Thanks for the update. Intereseting results!

[quote user="A.P"]

First chip failed after 790,397 cycles.

Second chip failed after 918,918 cycles.

Third chip failed after 1,102,080 cycles.

[/quote]

Does this correspond to the number of erase cycles on one flash page?

[quote user="A.P"]So, it seems we can expect around 900 thousand cycles on each page flash, if my 3 samples are in any way a valid sample pool.
Does Nordic have the distribution graph for flash fails per write-erase cycles anywhere?[/quote]

I am afraid I don't have much data on this, it's only that the endurance may be as low as 10 000 cycles as stated in the absoute maxium ratings section of the PS. And I think you should account for that number when designing your product. E.g. by allocating as many flash pages as possible to improve wear leveling.

RE: FDS and write/erase flash cycles

A.P — Tue, 26 Jan 2021 08:42:37 GMT

Hi all!
The interesting results are as follows:

We've run a flash write-read-erase test and limited the pages used so that the same page is written to, read from, and then erased. Each cycle wrote to the entire page with the following pattern:

A. Write 101010... read and compare. Erase.

B. Write 010101... read and compare. Erase.

C. Goto A.

We did this on 3 nRF52840s and found the following:

First chip failed after 790,397 cycles.

Second chip failed after 918,918 cycles.

Third chip failed after 1,102,080 cycles.

All subsequent reads returned an incorrect value from the flash.

So, it seems we can expect around 900 thousand cycles on each page flash, if my 3 samples are in any way a valid sample pool.
Does Nordic have the distribution graph for flash fails per write-erase cycles anywhere?

Is this expected to change in other chips? ex: nRF52833, nRF52820?

RE: FDS and write/erase flash cycles

A.P — Tue, 05 Jan 2021 15:02:51 GMT

Hi!

Thank you for the reply!
I'll set up a test, hopefully in the upcoming days, and see what I get.
If I get interesting results I'll post them here as a reply.

Thanks again!

RE: FDS and write/erase flash cycles

Vidar Berg — Tue, 05 Jan 2021 14:20:35 GMT

Hi,

I would not expect to get checksum failures during normal operation before the flash cells are starting to wear out, so I think any percentage is an indication that the flash is nearing its end of life. Maybe you can set up a stress test with FDS to see what actaully happens when you start exceeding 10 000 erase cycles?

RE: FDS and write/erase flash cycles

A.P — Tue, 05 Jan 2021 12:10:50 GMT

Hi Vidar!

Thank you for the reply!
I understand you can't sign off on this, but I still would like to get an input from you on the following:
Should I want to get some indication of whether or not a specific device flash is near its end of life, will checking the percentage of FDS_CRC_CHECK_ON_WRITE fails out of the total request be a reasonable indication?

Meaning that if I write 100 times to the flash, and get 20% CRC write failed responses, that would be reasonable of me to assume that the flash has reached its end of life? Again, I'm not looking for absolutes, but rather some indication of the state of the flash.

Thanks!

RE: FDS and write/erase flash cycles

Vidar Berg — Tue, 05 Jan 2021 10:53:10 GMT

Hi!

1. I'm not an expert on the physics of NOR flash, but I believe the common failure mode after flash cells has worn out is that single bits can get stuck at the value '1'. In other words, data will not get stored correctly if the write operation involves clearing of a stuck bit.

The read operation should return the value as it was stored in flash.

2. You can enable the FDS_CRC_CHECK_ON_READ configuration option to make FDS confirm the integrity of the read data. It will then return FDS_ERR_CRC_CHECK_FAILED back to the application if the readback is incorrect. But, as mentioned in point 1, I believe it's more common for the error to occur during the write, and repeating the read operation will not help in that case.

3. You can use FDS_CRC_CHECK_ON_WRITE to check if data is stored correctly, and if not, implement a re-try mechanism (FDS will automatically shift to the next available address on next attempt). The CRC check should only start to fail when you are writing to a section of flash that has started to wear out.

There is no HW support to keep track of the number write/erase cycles performed on each flash page. Please keep in mind that 10 000 cycles is the minimum value according to spec. The typical value might be considerably higher.

4. It provides basic wear leveling by distributing the flash usage across several flash pages. The more flash pages you allocate to FDS, the less often you have to run Garbage collection and erase flash.

Best regards,

Vidar