Urgently!! About flash access

RE: Urgently!! About flash access

mansfield — Thu, 19 May 2016 03:40:15 GMT

I replaced pstorage.c, psdorage.h and pstorage_platform.h files from SDK 11.0.0. And I have updated the flash more than 2500 times. The problem didn't reproduced any more. It seems that SDK 11.0.0 have fixed this problem.

RE: Urgently!! About flash access

mansfield — Wed, 03 Feb 2016 07:30:53 GMT

Thank you! I created a case on www.nordicsemi.com just now.

RE: Urgently!! About flash access

Stefan Birnir Sverrisson — Tue, 26 Jan 2016 08:49:22 GMT

So I suspect you are using pstorage since you talk about updates, am I correct? It would be good if I can look at your code. If you do not want to share it here, you can create a support case on www.nordicsemi.com (not public), submit your project and refer to this case

RE: Urgently!! About flash access

mansfield — Tue, 26 Jan 2016 05:16:51 GMT

Now I set the timer priority to 3. But I also found the update operation fail in sdk 8.1.1, if I write frequently(1 write+2 update per 10 second, the 2 updates are in the same page).

RE: Urgently!! About flash access

Stefan Birnir Sverrisson — Fri, 22 Jan 2016 08:26:42 GMT

No it is not fixed in 8.x.x. However, there is one more factor and that is the softdevices. For softdevices prior to S110 8.0.0, CPU is blocked during the whole BLE radio event, with S110 8.0.0 and later, CPU can be used by the application during the BLE radio event. This means that S110 8.0.0 provides better CPU utilization than its predecessors, and could make a difference in your case, but it could depend on your BLE connection/advertitising interval. It also depends on if you are doing a flash write only or if you are doing also flash erase or flash update. Flash update is a pstorage operation and includes flash erase operation among other things.

For details on CPU blocking, see my answer on this thread below the dotted line.

RE: Urgently!! About flash access

mansfield — Fri, 22 Jan 2016 04:18:32 GMT

Thank you very much! I have changed the priority of the timer to level 3. Now I'm migrating to sdk 8.1.1. Does this problem fixed in 8.1.1?

RE: Urgently!! About flash access

Stefan Birnir Sverrisson — Thu, 21 Jan 2016 15:25:36 GMT

Hi mansfield

The timer frequency is rather high and I suspect this is a CPU utilization problem. The flash operation most likely can not be scheduled with such high frequency Timer interrupts. Ongoing flash operation blocks the CPU.

The softdevice calls sd_flash_page_erase, sd_flash_protect and sd_flash_write are handled with ARM priority 2. When you have the timer interrupt priority high (arm priority 1) it has higher priority than the flash operation. When you have the timer interrupt priority low (arm priority 3) it has lower priority than the flash operation, which means a timer interrupt can't preempt a flash operation. The execution of the timer interrupt handler is therefore delayed until a flash operation completes. You could verify if this is the case by connecting a logical analyzer to a GPIO pin and toggle the GPIO pin every time the timer interrupt handler starts executing. If you see that it pauses for some time, that is the pstorage operation at work.

Update 18.5.2016 The conclusion is that there is a bug in pstorage library in SDK 8.1.0 and earlier SDKs that causes a race condition between the thread that calls the pstorage operation and the pstorage callback. The result is that occasionally the pstorage callback handler is not called by the pstorage module. This is evident when the pstorage calling thread is in the main context and CPU load is high and/or interrupt activity is high. Large BLE activity creates both high CPU and interrupt activity and can cause the bug to appear.

A workaround is to simply use pstorage module from nRF51 SDK 9.0.0 or more recent. You dont necessarily need to update your whole project to SDK 9.0.0+, you can just replace the pstorage module (pstorage.h, pstorage.c and pstorage_platform.h files) with pstorage module from SDK 9.0.0+. The pstorage module was refactored in SDK 9.0.0, which apparently removed the bug mentioned above. Pstorage is pretty much isolated module. It is not depending on other modules, so migrating only the pstorage module to SDK 9.0.0+ should not be risky. As far as I see, it is only interacting with the two softdevice API flash functions, sd_flash_page_erase and sd_flash_write.

Note: pstorage_platform.h is pstorage configuration file normally located in your project. If you have configured the pstorage_platform.h file in your project you must configure the new pstorage_platform.h from SDK 9.0.0+ accordingly.

Bug details:

Occasionally, callback was not received from pstorage module, making the pstorage module hang. I found out that the callback was actually received by the pstorage_sys_event_handler in pstorage.c, but still the cb_handler was never called in flash.c. That is because occasionally there was no flash access registered when he callback arrived, i.e.

m_cmd_queue.flash_access == false

making the callback handler think there is no ongoing flash operation, so the handler did nothing. The fault is that in cmd_process function in pstorage.c, which processes pstorage task in the pstorage queue, calls the relevant flash operation (e.g. sd_flash_write) and then sets the flash access flag at the bottom of the function. I suspect this causes a race condition between the cmd_process function to set

 m_cmd_queue.flash_access = true;

before the pstorage_sys_event_handler checks for the same flash access flag, i.e. with

if (m_cmd_queue.flash_access == true)
{
    // Process the callback
}

because if the flash access flag is not true, the pstorage_sys_event_handler will do nothing and the pstorage will hang.

I suspect this deadlock can only happen when the caller to the pstorage function is in the main context, i.e. with the lowest priority. When the caller has lowest priority, the flash operation callback can preempt the caller and check the flash access flag before the caller can set it. This scenario is more likely to happen when there is increased interrupt activity, i.e. from timers, BLE, or other, as all interrupts can preempt the caller if it runs in the main context.

RE: Urgently!! About flash access

Stefan Birnir Sverrisson — Thu, 21 Jan 2016 15:13:29 GMT

What puzzles me is that you mention update operation. If you use S110 flash API directly, it consists of sd_flash_page_erase, sd_flash_protect and sd_flash_write.

RE: Urgently!! About flash access

mansfield — Sat, 16 Jan 2016 07:07:16 GMT

I am using S110 flash API directly. Frequncy of Timer1 is 6k, and Timer2 is 1k. Some simple compute and io control are running in the interrupt. I didn't found failure during erase, since this operation runs not so much. So I found failure only at update and write operation.

RE: Urgently!! About flash access

Stefan Birnir Sverrisson — Fri, 15 Jan 2016 12:00:39 GMT

Hi mansfield

Im not sure why this fails for you with high priority timer. Are you using the pstorage module or are you using the S110 flash API directly? Can I ask what you are trying to achieve? What is the interrupt frequency of your timers? Does it only fail with flash writes? What about flash erase?