nRF52832 with SDK v11.0.0 -- System Randomly Freeze and Doesn't Recover via WDog

[Sami]

Hello,

Currently, we are working on a product which leverages the following:

NRF52832 (64KB RAM and 512 KB Flash)

SDK v11.0.0

Soft Device 132

It has been reported out in the field that our product is randomly "freezing". After weeks of FW debugging/testing/analysis, our team has yet to root cause the issue. 

Why is the watchdog (WDog) not overriding the system to reset it? If the CPU has a hard-fault, shouldn't the WDog priority be higher than a CPU hard-fault? Our hard-fault handlers have been updated to reset the system, however, the system remains stuck. The only way that we're able to recover the system is if we pull the battery out and plug it back in. Essentially, we need to make sure that the WDog is not being blocked by any other executing instructions of higher priority. Please advise on what to do. Is this a known issue? Are there other ways that we can recover the system from an indeterminant state besides a WDog reset or hard-fault handler?

Please note, our product does not have a hardware reset pin.

[bjorn-spockeli]

HI Sami, 

Have you been able to determine if the Hardfaults occur at the same location? Or are they occuring randomly ? We do have a nRF5 SDK v11.0.0: HardFault handling library that is very useful when it comes do debugging these issues. 

Are you enabling the WDT upon entering main()? Do you have a bootloader present on the nRF52832? Is your device entering System OFF? 

The WDT in the nRF52 Series does rely on an interrupt to reset the SoC , if the WDT times out then the system will be reset, regardless of the state of the CPU. The WDT is decoupled from the CPU to avoid the scenario that are describing. 

Best regards

Bjørn

[bjorn-spockeli]

Also which MDK version are you using in your code? SDK v11.0.0 uses v8.5.0, so if you have not updated the MDK to v8.9.0, then I strongly recommend adding the workaround for the 108 errata. 

[108] RAM: RAM content cannot be trusted upon waking up from System ON Idle or System OFF mode

[Sami]

We have not been able to reproduce the issue in-house, so we don't know if a hard-fault is occurring, let alone occurring in the same location. 

-Yes, we are enabling the WDT in main(), before we enter the executive task ("scheduler loop").

-Yes, we have a bootloader present on the nRF52832.

-Yes, the device enters System OFF once and once only, while it is sitting on the shelf. Once transitioned to System ON mode, it will never enter System OFF mode again. 

- We are using MDK v8.5.0, and we have not implemented the work-around. I became aware of Errata 108 last week, but wasn't sure how it was affecting our system, if at all. It sounds like RAM content can be deemed indeterminant if System OFF mode is ever entered and then woken up without the work-around?

[bjorn-spockeli]

Sami said:

- We are using MDK v8.5.0, and we have not implemented the work-around. I became aware of Errata 108 last week, but wasn't sure how it was affecting our system, if at all. It sounds like RAM content can be deemed indeterminant if System OFF mode is ever entered and then woken up without the work-around?

 Errata 108 may occur when the system is waking up from System ON idle as well, not just System OFF. So everytime you wakeup from System ON idle, there is a chance that RAM is not retained correctly, which may lead to random hardfaults if values in the stack is modified to a value outside the valid memory range. 

[Sami]

So I could conceivably run a test where I loop the system between "System ON idle" and system active in order to induce this hard fault? Could you provide context as to how I could induce this issue ?

[bjorn-spockeli]

Hi Sami,
the location of the bits that flip may vary from chip to chip, but the location seems to be fixed for every chip that do fail. Only a few bits (e.g. 1-5) spread across RAM.

The way we test the issue is that we initialize the RAM blocks not used by the stack to all 0's and then enter System ON Idle and wake up from a timer. We then examine the RAM blocks after wake-up and check if any bits have been flipped.  

Best regards

Bjørn

[Sami]

I created an external array:

uint32_t errata108Arr[10] = {0};

so that it lives in the .bss section (i.e., not the stack). Then, I put the system into System OFF mode, waited a few seconds, and then woke the system back up (System ON mode). The array still preserved its contents. Do I need to check all memory blocks outside of the stack in accordance with what you said: "the location of the bits that flip may vary from chip to chip."

[bjorn-spockeli]

Hi Sami,

Yes, you should create an array that covers as much of RAM as possible. 

I have attached the code we have been using internally to check if devices are affected. 

 errata_108_test_code.zip

Best regards

Bjørn

[Sami]

Thank you. If the bug, i.e., errata 108, does occur, could it lead to a "screen freeze"/"blank screen" that does not recover via WDog timeout reset? Also, are interrupts also frozen?