nRF52832 with SDK v11.0.0 -- System Randomly Freeze and Doesn't Recover via WDog

RE: nRF52832 with SDK v11.0.0 -- System Randomly Freeze and Doesn't Recover via WDog

bjorn-spockeli — Thu, 02 Apr 2020 07:28:52 GMT

HI Sami Balbaky,

if a RAM corruption occurs due to Errata 108 issue, then this can cause a HardFault if the CPU fetches a corrupt instruction that points to a memory location outside the nRF52832s memory area. and the HardFault exception will trigger HardFault_Handler in the application. The HardFault exception is always enabled and has a fixed priority, -1 (higher than other interrupts and exceptions), so while the CPU is in the Hardfault handler it will prevent other interrupts with lower priority from being serviced.

I took a closer look at default HardFault handler implementation in the system_nrf52.c file and this will just loop indefinitely. The HardFault_Handler in the system_nrf52.c file is a weak implementation

  .weak   HardFault_Handler
HardFault_Handler:
  b     .

Which basically is equivalent to

HardFault_Handler:
  b HardFault_Handler

So if you want the Hardfault exception to trigger a reset then you need to override the weak implementation and issue a SoftReset,i.e. NVIC_SystemReset().

However, the WDT will reset the device if it is running when the HardFault occurs. As discussed previously the WDT is independent of the CPU, it will trigger a reset regardless of the CPU state. When the HardFault occurs the CPU is basically just looping the HardFault handler.

Best regards

Bjørn

RE: nRF52832 with SDK v11.0.0 -- System Randomly Freeze and Doesn't Recover via WDog

Sami — Tue, 31 Mar 2020 19:34:36 GMT

Thank you. If the bug, i.e., errata 108, does occur, could it lead to a "screen freeze"/"blank screen" that does not recover via WDog timeout reset? Also, are interrupts also frozen?

RE: nRF52832 with SDK v11.0.0 -- System Randomly Freeze and Doesn't Recover via WDog

bjorn-spockeli — Thu, 12 Mar 2020 10:13:58 GMT

Hi Sami,

Yes, you should create an array that covers as much of RAM as possible.

I have attached the code we have been using internally to check if devices are affected.

devzone.nordicsemi.com/.../errata_5F00_108_5F00_test_5F00_code.zip

Best regards

Bjørn

RE: nRF52832 with SDK v11.0.0 -- System Randomly Freeze and Doesn't Recover via WDog

Sami — Tue, 10 Mar 2020 23:54:09 GMT

I created an external array:

uint32_t errata108Arr[10] = {0};

so that it lives in the .bss section (i.e., not the stack). Then, I put the system into System OFF mode, waited a few seconds, and then woke the system back up (System ON mode). The array still preserved its contents. Do I need to check all memory blocks outside of the stack in accordance with what you said: "the location of the bits that flip may vary from chip to chip."

RE: nRF52832 with SDK v11.0.0 -- System Randomly Freeze and Doesn't Recover via WDog

bjorn-spockeli — Wed, 04 Mar 2020 08:35:58 GMT

Hi Sami,
the location of the bits that flip may vary from chip to chip, but the location seems to be fixed for every chip that do fail. Only a few bits (e.g. 1-5) spread across RAM.

The way we test the issue is that we initialize the RAM blocks not used by the stack to all 0's and then enter System ON Idle and wake up from a timer. We then examine the RAM blocks after wake-up and check if any bits have been flipped.

Best regards

Bjørn

RE: nRF52832 with SDK v11.0.0 -- System Randomly Freeze and Doesn't Recover via WDog

Sami — Tue, 03 Mar 2020 18:14:14 GMT

So I could conceivably run a test where I loop the system between "System ON idle" and system active in order to induce this hard fault? Could you provide context as to how I could induce this issue ?

RE: nRF52832 with SDK v11.0.0 -- System Randomly Freeze and Doesn't Recover via WDog

bjorn-spockeli — Mon, 24 Feb 2020 12:05:51 GMT

[quote user="sami_cala"]- We are using MDK v8.5.0, and we have not implemented the work-around. I became aware of Errata 108 last week, but wasn't sure how it was affecting our system, if at all. It sounds like RAM content can be deemed indeterminant if System OFF mode is ever entered and then woken up without the work-around?[/quote]

Errata 108 may occur when the system is waking up from System ON idle as well, not just System OFF. So everytime you wakeup from System ON idle, there is a chance that RAM is not retained correctly, which may lead to random hardfaults if values in the stack is modified to a value outside the valid memory range.

RE: nRF52832 with SDK v11.0.0 -- System Randomly Freeze and Doesn't Recover via WDog

Sami — Fri, 21 Feb 2020 17:58:59 GMT

We have not been able to reproduce the issue in-house, so we don't know if a hard-fault is occurring, let alone occurring in the same location.

-Yes, we are enabling the WDT in main(), before we enter the executive task ("scheduler loop").

-Yes, we have a bootloader present on the nRF52832.

-Yes, the device enters System OFF once and once only, while it is sitting on the shelf. Once transitioned to System ON mode, it will never enter System OFF mode again.

- We are using MDK v8.5.0, and we have not implemented the work-around. I became aware of Errata 108 last week, but wasn't sure how it was affecting our system, if at all. It sounds like RAM content can be deemed indeterminant if System OFF mode is ever entered and then woken up without the work-around?

RE: nRF52832 with SDK v11.0.0 -- System Randomly Freeze and Doesn't Recover via WDog

bjorn-spockeli — Fri, 21 Feb 2020 13:45:26 GMT

Also which MDK version are you using in your code? SDK v11.0.0 uses v8.5.0, so if you have not updated the MDK to v8.9.0, then I strongly recommend adding the workaround for the 108 errata.

[108] RAM: RAM content cannot be trusted upon waking up from System ON Idle or System OFF mode

RE: nRF52832 with SDK v11.0.0 -- System Randomly Freeze and Doesn't Recover via WDog

bjorn-spockeli — Fri, 21 Feb 2020 13:42:53 GMT

HI Sami,

Have you been able to determine if the Hardfaults occur at the same location? Or are they occuring randomly ? We do have a nRF5 SDK v11.0.0: HardFault handling library that is very useful when it comes do debugging these issues.

Are you enabling the WDT upon entering main()? Do you have a bootloader present on the nRF52832? Is your device entering System OFF?

The WDT in the nRF52 Series does rely on an interrupt to reset the SoC , if the WDT times out then the system will be reset, regardless of the state of the CPU. The WDT is decoupled from the CPU to avoid the scenario that are describing.

Best regards

Bjørn