nRF52840: 0.4 mA extra sleep current after QSPI writes from RAM8, depending on code alignment

[tdickenson]

nrf52840, Zephyr, ncs 3.4.1

For a few months I have been tracking an intermittent issue with a product that is close to production release.

The symptom of this problem is an excess 0.4 mA current during sleep. The excess current was observed to start after a burst of application activity (including QSPI flash writes), and continues until the next Bluetooth radio event.

The current profile is flat. Mostly this problem does not occur. But on some builds it occurs nearly every sleep. Then making the smallest change to the firmware would cause the problem to vanish, making this

super-difficult to investigate.

With Claude's help I have been able to identify the circumstances where it occurs, and build a minimal example that runs on the development kit board.

The circumstances are:

* It requires a certain pattern of QSPI flash writes. The minimal example uses a pattern of three writes that is common to our application. The issue does not occur after a single flash write.

* The source of the flash write needs to be in RAM8 (RAM above 0x20010000). The issue does not occur if the writes are from a lower RAM address (RAM0 to RAM7).

* The write needs to occur from a thread whose stack is in RAM8. The issue does not occur if the stack is at a lower address.

* Code alignment is an issue too. Adding some non-executed NOP instructions to a function in `main.c` shifts the rest of the application linked after it. Shifting by 2 to 6 bytes affects whether the issue occurs. I have not investigated exactly which instruction is sensitive.

The excess current will last indefinitely, but it seems to be ended by another EasyDMA transfer. We have tried two:

* In the product it ends at the next radio event. In the example built with a Bluetooth beacon, on the DK, it ends at the next advertising event.

* In an earlier version of the example, encrypting one AES block on the ECB, which reads and writes through EasyDMA, ends it at once, whether the ECB's data was in RAM8 or below it.

It is drawn by the nRF52840 itself. On the DK, the current measurement covers the nRF52840's supply alone, without the QSPI flash: 400 µA at 3.0 V, against a 2 µA floor when the issue doesn't occur. The results for each build are in the example's README.

What I have done:

* Published a minimal example that reproduces the excess 0.4 mA on the development kit, and demonstrates the expected low sleep current with trivial program changes.

* Reproduced it with nRF Connect SDK v3.4.1 exactly as released, every repository at its manifest revision with no local changes.

* Confirmed the current difference is not due to any expected effect from the firmware. In the high-current state, read through the debugger: the QSPI peripheral is disabled with its pins in their sleep state, the flash is in deep power-down, and the peripheral enables, timers, PPI, GPIOTE, pin configuration and RAM power registers are the same as at the normal floor. It looks like this code is triggering some incorrect internal silicon state.

* Confirmed this is not anomaly 122. That errata documentation is light on details, it looks similar, but nrfx already applies the documented workaround at every disable (`nrf_qspi_disable()` in `hal/nrf_qspi.h`. Doubling up this workaround (in various ways) has no effect on this issue.

* Ruled out anomaly 276, which describes a similar 400 to 450 µA after the SAADC stops in scan mode: the example never enables the SAADC.

The chip on our module reads build code D0 (FICR INFO.VARIANT = AAD0), which is Revision 2

What I have not done:

* Confirmed that all of the code widgets written by Claude (for example the kconfig options to move stacks into different blocks of ram) have expected effect. But I have no specific reason to doubt.

At this point I have a workaround which involves copying every QSPI flash write through a buffer at a controlled RAM address. This appears to eliminate all problems, in the example and in our product. This workaround has proven solid (so far) so there is no urgency from our side, but it would be good to understand what is going on here.

Reproduction example is here: github.com/.../tailissue

[Pallavi Patil]

Hie 

I agree, I am not able to read your case since some of the sentences are ending abruptly.  Can you please create a new case with the same text. Can you paste the text in notepad/word and then copy the text to create the new ticket. It might help.

Regards

Pallavi

[tdickenson]

nrf52840, Zephyr, ncs 3.4.1

For a few months I have been tracking an intermittent issue with a product that is close to production release.

The symptom of this problem is an excess 0.4 mA current during sleep. The excess current was observed to start after a burst of application activity (including QSPI flash writes), and continues until the next Bluetooth radio event.

The current profile is flat. Mostly this problem does not occur. But on some builds it occurs nearly every sleep. Then making the smallest change to the firmware would cause the problem to vanish, making this

super-difficult to investigate.

With Claude's help I have been able to identify the circumstances where it occurs, and build a minimal example that runs on the development kit board.

The circumstances are:

* It requires a certain pattern of QSPI flash writes. The minimal example uses a pattern of three writes that is common to our application. The issue does not occur after a single flash write.

* The source of the flash write needs to be in RAM8 (RAM above 0x20010000). The issue does not occur if the writes are from a lower RAM address (RAM0 to RAM7).

* The write needs to occur from a thread whose stack is in RAM8. The issue does not occur if the stack is at a lower address.

* Code alignment is an issue too. Adding some non-executed NOP instructions to a function in `main.c` shifts the rest of the application linked after it. Shifting by 2 to 6 bytes affects whether the issue occurs. I have not investigated exactly which instruction is sensitive.

The excess current will last indefinitely, but it seems to be ended by another EasyDMA transfer. We have tried two:

* In the product it ends at the next radio event. In the example built with a Bluetooth beacon, on the DK, it ends at the next advertising event.

* In an earlier version of the example, encrypting one AES block on the ECB, which reads and writes through EasyDMA, ends it at once, whether the ECB's data was in RAM8 or below it.

It is drawn by the nRF52840 itself. On the DK, the current measurement covers the nRF52840's supply alone, without the QSPI flash: 400 µA at 3.0 V, against a 2 µA floor when the issue doesn't occur. The results for each build are in the example's README.

What I have done:

* Published a minimal example that reproduces the excess 0.4 mA on the development kit, and demonstrates the expected low sleep current with trivial program changes.

* Reproduced it with nRF Connect SDK v3.4.1 exactly as released, every repository at its manifest revision with no local changes.

* Confirmed the current difference is not due to any expected effect from the firmware. In the high-current state, read through the debugger: the QSPI peripheral is disabled with its pins in their sleep state, the flash is in deep power-down, and the peripheral enables, timers, PPI, GPIOTE, pin configuration and RAM power registers are the same as at the normal floor. It looks like this code is triggering some incorrect internal silicon state.

* Confirmed this is not anomaly 122. That errata documentation is light on details, it looks similar, but nrfx already applies the documented workaround at every disable (`nrf_qspi_disable()` in `hal/nrf_qspi.h`. Doubling up this workaround (in various ways) has no effect on this issue.

* Ruled out anomaly 276, which describes a similar 400 to 450 µA after the SAADC stops in scan mode: the example never enables the SAADC.

The chip on our module reads build code D0 (FICR INFO.VARIANT = AAD0), which is Revision 2

What I have not done:

* Confirmed that all of the code widgets written by Claude (for example the kconfig options to move stacks into different blocks of ram) have expected effect. But I have no specific reason to doubt.

At this point I have a workaround which involves copying every QSPI flash write through a buffer at a controlled RAM address. This appears to eliminate all problems, in the example and in our product. This workaround has proven solid (so far) so there is no urgency from our side, but it would be good to understand what is going on here.

Reproduction example is here: github.com/.../tailissue

[hmolesworth]

Interesting problem, and a very good description. There is an issue with sharing an AHB Slave between a peripheral DMA and the CPU stack or other peripheral DMA, which may possibly be related as (as far as I know) the QSPI may not handle bus stalls well in relation to hardware clock handling, unlike the nRF54L15. Any stack access (interrupts or CPU) or other higher-priority peripheral DMA access will take precedence over QSPI DMA access via the same AHB bus causing a bus stall which may not handle SCK well.

If two (or more) bus masters request access to the same slave device, the master with the highest
priority is granted the access first. Bus masters with lower priority are stalled until the higher priority master has completed its transaction.

There is a two stage AMLI (AHB Multi-Layer Interface) bus system in the nRF52840 and nRF52833 instead of one stage on the nRF52832 and this makes avoiding the stall on the nRF52840 more difficult to design around. The simple fix - assuming this is an issue in this case - is to separate QSPI AHB RAM from all other RAM which might be accessed during a transaction. Place the stack in any other RAM area, say RAM7; this would imply that placing both stack and QSPI DMA in (say) RAM7 would give an identical problem, but without knowing which other transactions can take place over shared RAM8 or shared RAM7 (whichever is chosen) that's hard to predict. I did some investigation on this issue when trying to precisely align multiple PWM outputs which was more difficult on the nRF52833 vs nRF52832. start-pwms-synchronous

The reduced code version mentions that the workaround for Errata 122 has been implemented, but supposedly two writes are required  in the workaround (010 and 054) whereas the notes on github only mention one  (054):

*(volatile uint32_t *)0x40029010ul = 1ul;
*(volatile uint32_t *)0x40029054ul = 1ul;

010 is TASKS_DEACTIVATE, so is this an errata documentation bug or not? I ask as the errata states the workaround should be implemented before deactivating- "Execute the following code before disabling QSPI"

Regarding the nops, clearly adding nops allow displacing code memory locations, and perhaps less obviously adding nops also displaces time of code-triggered events such as the start of a QSPI transfer. The 8-28-8 byte DMA bursts may happen to occur near another interrupt event such as a hardware timer (zephyr) such that a DMA access may or may not collide with a timing interrupt stack register push thus causing (or not) a bus stall. 64MHz code execution vs 16MHz peripheral DMA allows fine tuning of such an event. Separating stack and DMA AHB busses fixes that issue.