We are currently working with a system that is using both and external flash and the HW WDG from the NRF5340.
Recently we are facing the issue, that we cannot re-flash the device cleanly, because we get following error:
west flash
...
Device error: Memory access error at 0x20004edc.
Probably a memory protection issue. Probe access is Non-secure (Generic)
After re-trying the `west flash`, we end up with locked AP, so we need to `--recover` the device first, before being able to flash it again.
Debug access to the Network core is currently disabled. The Network core is either protected or not started, and it is not possible to start the Network core through the Application core: The Application core access port is currently closed. AP-Protect might be enabled (NotAvailableBecauseProtection)
The operation is unavailable for some of the devices because they have readback protection enabled. These devices might be possible to unlock using the `recover` subcommand. See `nrfutil device recover --help` for more information.
We tracked down the issue up to the point where we suspect following chain of events.
1. The normal firmware is running. It has armed the HW WDG (`CONFIG_TASK_WDT_HW_FALLBACK=y`). The HW WGD timeout is 7.7s
2. Now we want to flash the new SW with `west flash`
2.1 First the CPU gets halted --> This also halts the HW WDG 2.2 Now the internal flash gets erased
2.3 Now the external flash gets erased. This requires the QSPI loader to run --> CPU is running --> WDG is running
2.4 Now the external flash gets flashed. -->WDG running
2.5 Now the internal flash gets flashed
3. The HW watchdog expires at some point during step 2.4/2.5 (the erase/flash operation are taking longer than 7.7s)
4. The expired WDG causes a reset. The chip reboots at address 0x00 where no code is located after the erase --> No `SystemInit()` runs --> AP never opens up.
5. The debugger who tries to perform operations on the external flash fails due to the shown error.
When trying to re-flash the device with `west flash` the debugger instantly noticed the closed debug port and throws the 2nd error.
Here's a filtered log from `nrfutil-device.log` which is backing our hypothesis of the explained sequence:
12:44:09.698 Programming build/merged.hex
12:44:09.942 -> halt
12:44:11.400 Performing system reset
12:44:11.400 -> halt
12:44:11.502 Erasing pages touched by firmware
12:44:27.786 Erasing pages from external memory
12:44:27.800 -> halt
12:44:27.912 Initial stack: 0x2000a840 Entry point: 0x20000ff4
12:44:28.038 -> halt
12:44:28.039 -> go
12:44:30.638 Done with device; restoring run state
12:44:30.638 -> halt ()
12:44:30.644 Programming 11 segments
12:44:30.663 Programming 10 segments to flash
12:44:30.663 NVMC: Programming 10 segments
12:44:34.494 Programming 2 segments to xip
12:44:34.511 -> halt
12:44:34.620 Initial stack: 0x2000a840 Entry point: 0x20000ff4
12:44:34.744 -> halt
12:44:34.746 -> go
12:44:38.926 Failed to write 3972 bytes at 0x20004ee0: Failed to write 3972 bytes to memory address
12:44:38.981 Probably a TrustZone access or other protection error. The original error is Failed to
12:44:38.981 Done with device; restoring run state
12:44:38.981 -> halt
12:44:39.028 Error executing operations for 506000982: Device error: Memory access error at 0x20004e
We verified this behavior with
a) Flashing the device with `CONFIG_TASK_WDT_HW_FALLBACK=n`
b) Adding a well times pin-reset (pin-reset is apparently clearing the WDG. Soft/System/Hard-Reset are not clearing the WDG)
The reset has to be performed within the small window where the debugger halts the SW before it has armed the WDG
c) By adding a small sample, that is: Opening the AP, (Stopping the WDG), keeping a buys loop so that it can be halted by the Debugger
c') We also noticed, that we do not need the active stop of the WDG if we flash our "keeper-sw" with option `reset=RESET_PIN`
Therefore we are quite positive about the fact that the HW WDG is causing the error during XIP operations.
Now there's the catch:
This behavior started, when we were migrating NCS to NCS version 3.4.0
The HW watchdog was in a few months earlier. Therefore we are questioning, whether were some changes to the nrfutil in either the flash sequence or reset behavior that, is causing the error to pop up now.
Unfortunately I cannot reproduce this error on a dev-kit, since those are not quipped with an external flash, but in case you have one at hand, just make a minimal sample with the HW WDG running and a bunch of stuff on the ext-flash so that you hit a very high chance of the HW WDG expiring during the external flash write/erase operation.
So for this ticket, my wish would be to resolve those two questions:
1) Is our hypothesis correct? Can the HW WDG interfere while the QSPI loader is within a flash/erase operation of the external flash?
2) Why is this issue coming up after the NCS 3.4.0 migration? Where there any changes to either the flash sequence, the `nrf_app_keeper.hex` or the reset behavior whilst `west flash`
With our proprietary keeper sample, flashing is working fine again, but imo this issue should be solved within the toolchain itself, by either:
- Actively clearing the HW WDG before flashing
- Flashing with PIN reset to clear the HW WDG
- Changing the flash sequence to first fully erase/flash the internal flash so that the AP gets opened up even if a reset occurs.
But before I request changes, I would first like to hear your take on this issue and whether this is valid or whether we are having a bad setup that is causing this :)
Some more details about our project:
SOC: nrf5340 + nrf7002 (application core with TF-M, network core with `hci_ipc`)
Board: Custom board with 8MB QSPI NOR flash for XIP
NCS version: Fork of NCS 3.4.0 (only few smaller patches applied)
nrfutil version: 8.1.1
nrfutil-device version: 8.1.1
Relevant configs:
CONFIG_BUILD_WITH_TFM=y
CONFIG_NRF_APPROTECT_USE_UICR=y
CONFIG_NRF_SECURE_APPROTECT_USE_UICR=y
CONFIG_TASK_WDT_HW_FALLBACK=y
CONFIG_TASK_WDT_MIN_TIMEOUT=7500
CONFIG_TASK_WDT_HW_FALLBACK_DELAY=200
CONFIG_XIP=y
CONFIG_NORDIC_QSPI_NOR_XIP=y
If there is any way you know, which can help us prevent these errors, we would be really glad to hear about it.
In case you have any extra questions, feel free to ask.
Thank you very much in advance!
Best regards
Jaro
