Firmware Reset failing with MCUboot and WDT

RE: Firmware Reset failing with MCUboot and WDT

Vidar Berg — Mon, 04 Nov 2024 11:56:32 GMT

Hi,

Yes, but I would recommend using v2.5.3 to get the latest fixes available for that branch.

RE: Firmware Reset failing with MCUboot and WDT

hugzy123 — Mon, 04 Nov 2024 09:59:23 GMT

Hi Vidar,

Thanks for updating me.

From the ticket you attached, if I upgrade to at least SDK v2.5.0 I should be fine?

Otherwise, i'll create a new nrf_reset_network_force_off() function in my board file.

RE: Firmware Reset failing with MCUboot and WDT

Vidar Berg — Fri, 01 Nov 2024 12:37:18 GMT

Hi,

I realized today while working on this ticket nRF5340 softreset question that the workaround for Errata 161 was not backported to v2.4.4. I recommend adding the workaround in your board file.

EDIT: nrf53_errata_161() will return false here https://github.com/zephyrproject-rtos/hal_nordic/blob/427ee1a519e8a0844d0f78f7cbc8cdfc134ef00d/nrfx/hal/nrf_reset.h#L175C14-L175C30 due to a bug in your current MDK version, which means the workaround will not be applied even if you use nrf_reset_network_force_off(). To avoid having to patch the SDK, you can create your own nrf_reset_network_force_off() function to be used in your boards *_cpunet_reset.c file.

RE: Firmware Reset failing with MCUboot and WDT

hugzy123 — Thu, 24 Oct 2024 11:24:01 GMT

No worries, thanks for the quick reply!

RE: Firmware Reset failing with MCUboot and WDT

Vidar Berg — Thu, 24 Oct 2024 11:22:22 GMT

Hi,

Sorry, you're right, it's 16K. I was mixing it up with the nRF9160, which has 32K alignment requirement.

Best regards,

Vidar

RE: Firmware Reset failing with MCUboot and WDT

hugzy123 — Thu, 24 Oct 2024 11:15:04 GMT

Hi Vidar Berg ,

Just wanted to double check something, does the SPU definitely require 32K alignment?

The only reason I ask is because, in my ncs v2.4.4 build I can see

#define CONFIG_NRF_SPU_FLASH_REGION_SIZE 0x4000

which would be 16K.

I'm assuming CONFIG_NRF_SPU_FLASH_REGION_SIZE would be what is used for CONFIG_NRF_TRUSTZONE_FLASH_REGION_SIZE in https://docs.nordicsemi.com/bundle/ncs-latest/page/nrf/security/tfm.html#tf-m_partition_alignment_requirements, i'm just not sure if this would be 16k or 32k,

Thanks,

RE: Firmware Reset failing with MCUboot and WDT

hugzy123 — Wed, 09 Oct 2024 08:12:29 GMT

[quote userid="4240" url="~/f/nordic-q-a/114321/firmware-reset-failing-with-mcuboot-and-wdt/505336"]Did you use the same memory layout in both versions? The SPU requires 32K aligment: https://docs.nordicsemi.com/bundle/ncs-latest/page/nrf/security/tfm.html#tf-m_partition_alignment_requirements.[/quote]

Thank you, I will look into this.

[quote userid="4240" url="~/f/nordic-q-a/114321/firmware-reset-failing-with-mcuboot-and-wdt/505336"]I do not foresee any problems the workaround itself[/quote]

Great, we'll use this for now.

RE: Firmware Reset failing with MCUboot and WDT

Vidar Berg — Tue, 08 Oct 2024 12:11:45 GMT

[quote user="hugzy123"]Upgrading to a newer SDK does sound like a better fix, however, we've been struggling to get the partitions to confirm to the alignment rules in the newer SDK versions for the trusted firmware module.[/quote]

Did you use the same memory layout in both versions? The SPU requires 32K aligment: https://docs.nordicsemi.com/bundle/ncs-latest/page/nrf/security/tfm.html#tf-m_partition_alignment_requirements.

[quote user="hugzy123"]Thanks, we've verified this workaround in our current firmware. Do you know if there are any downsides to using this patch? [/quote]

I do not foresee any problems the workaround itself.

RE: Firmware Reset failing with MCUboot and WDT

hugzy123 — Tue, 08 Oct 2024 11:36:45 GMT

Thanks for your reply.

[quote userid="4240" url="~/f/nordic-q-a/114321/firmware-reset-failing-with-mcuboot-and-wdt/504823"]I have not been able to pinpoint the source of the flash access error, nor have I found any other reports of this event happening under similar conditions. However, upgrading to SDK v2.4.4 may resolve this issue as it includes several errata workarounds that were not present in SDK v2.2.0.[/quote]

Upgrading to a newer SDK does sound like a better fix, however, we've been struggling to get the partitions to confirm to the alignment rules in the newer SDK versions for the trusted firmware module.

[quote userid="4240" url="~/f/nordic-q-a/114321/firmware-reset-failing-with-mcuboot-and-wdt/504823"]A workaround for now may be to issue a soft reset from the SPU ISR in TF-M to avoid having to wait for the WDT timeout. For example, by adding the following to ncs/v2.2.0/modules/tee/tf-m/trusted-firmware-m:[/quote]

Thanks, we've verified this workaround in our current firmware. Do you know if there are any downsides to using this patch?

[quote userid="4240" url="~/f/nordic-q-a/114321/firmware-reset-failing-with-mcuboot-and-wdt/504823"]I think you are right. The specification does not guarantee the pin state in reset, only when going out of reset. I know this approach with using a GPIO to trigger pinreset worked with the nRF52840 but there are some differences in the reset mechanism between the nRF52840 and the nRF5340. [/quote]

We're looking into adding an IC we can use which would generate a pulse on the pinreset line from an output GPIO of the nRF module.

RE: Firmware Reset failing with MCUboot and WDT

Vidar Berg — Thu, 03 Oct 2024 12:10:28 GMT

I'm sorry for the delayed response. I've been working on debugging the code to identify what is triggering the SPU event. It turned out to be quite challenging because the security violation is not caused by the CPU but by another bus master.

I have not been able to pinpoint the source of the flash access error, nor have I found any other reports of this event happening under similar conditions. However, upgrading to SDK v2.4.4 may resolve this issue as it includes several errata workarounds that were not present in SDK v2.2.0.

A workaround for now may be to issue a soft reset from the SPU ISR in TF-M to avoid having to wait for the WDT timeout. For example, by adding the following to ncs/v2.2.0/modules/tee/tf-m/trusted-firmware-m:

diff --git a/platform/ext/common/faults.c b/platform/ext/common/faults.c
index eb87c971c..9124030c2 100644
--- a/platform/ext/common/faults.c
+++ b/platform/ext/common/faults.c
@@ -107,3 +107,8 @@ __attribute__((naked)) void UsageFault_Handler(void)
         "b         .                      \n"
     );
 }
+
+void SPU_IRQHandler(void)
+{
+    NVIC_SystemReset();
+}
\ No newline at end of file

[quote user="hugzy123"]It seems like because the reset pin never goes low again it doesn't complete a reset. You can see this using the "Boot Reset" button on the DK; if I hold the button down the device won't reset, It's only when I release the button does it reset.[/quote]

I think you are right. The specification does not guarantee the pin state in reset, only when going out of reset. I know this approach with using a GPIO to trigger pinreset worked with the nRF52840 but there are some differences in the reset mechanism between the nRF52840 and the nRF5340.

RE: Firmware Reset failing with MCUboot and WDT

hugzy123 — Fri, 27 Sep 2024 10:15:53 GMT

[quote userid="4240" url="~/f/nordic-q-a/114321/firmware-reset-failing-with-mcuboot-and-wdt/504031"]Thank you for this example. I'm able to reproduce the same here now, and it looks like the program hangs in the SPU_IRQHandler when the WD times out.[/quote]

Great to hear you can see it your side too.

[quote userid="4240" url="~/f/nordic-q-a/114321/firmware-reset-failing-with-mcuboot-and-wdt/504031"]This should work, but it is important that the GPIO is never becomes configured as an output low during boot. Is this GPIO configured in the code your prodived?[/quote]

It isn't in the demo I provided, however, it can be added easily.

On the DK you just need to add a solder bridge to SB43 so the reset can be controlled via the pin, then add a jumper between a gpio and the RESET pin.

I've used GPIO (0,26) below)

Then you can add the following function to the demo example which triggers a pin reset.

#define CONTROL_PIN NRF_GPIO_PIN_MAP(0,26) 
void pin_reset(void)
{
	 LOG_DBG("Resetting Device"); 

    // Configure the GPIO pin
   // nrfx_gpiote_out_config_t pin_config = NRFX_GPIOTE_CONFIG_OUT_SIMPLE(true);
    nrfx_gpiote_out_config_t pin_config =
    {
        .action = GPIOTE_CONFIG_POLARITY_LoToHi,
        .init_state = GPIOTE_CONFIG_OUTINIT_High,
        .task_pin = false,
    };
    nrfx_err_t err = nrfx_gpiote_out_init(CONTROL_PIN, &pin_config);
    if (err != NRFX_SUCCESS) 
        LOG_DBG("nrfx error 2 ox%X",err); 
    k_msleep(300);
    // Set the pin high
        // power up unused ram for mcuboot
    if (IS_ENABLED(CONFIG_RAM_POWER_DOWN_LIBRARY))
    {
        power_up_unused_ram();
    }
    nrfx_gpiote_out_clear(CONTROL_PIN);
	LOG_DBG("triggered reset pin"); //shouldn't get to here
}

In my testing I call pin_reset() in main after the 2 second delay and led initialisation.

It seems like because the reset pin never goes low again it doesn't complete a reset. You can see this using the "Boot Reset" button on the DK; if I hold the button down the device won't reset, It's only when I release the button does it reset.

RE: Firmware Reset failing with MCUboot and WDT

Vidar Berg — Thu, 26 Sep 2024 15:45:20 GMT

Thank you for this example. I'm able to reproduce the same here now, and it looks like the program hangs in the SPU_IRQHandler when the WD times out.

I will continue to debug this to try find out what is triggering the SPU IRQ.

[quote user="hugzy123"]

I also tried connecting the RESET pin directly to a GPIO to trigger a pin reset instead of a software reset. However, when I attempted this, our custom board freezes after setting the GPIO low, I think its because the RESET pin remains low.

[/quote]

This should work, but it is important that the GPIO is never becomes configured as an output low during boot. Is this GPIO configured in the code your prodived?

RE: Firmware Reset failing with MCUboot and WDT

hugzy123 — Thu, 26 Sep 2024 11:49:04 GMT

Do you know if it’s possible to trigger a pin reset using a GPIO on the same nRF53 controller?

RE: Firmware Reset failing with MCUboot and WDT

hugzy123 — Thu, 26 Sep 2024 11:40:52 GMT

Hi Vidar,

Apologies for the delayed response; I’ve been tied up with other projects. I’ve now returned to this and managed to reproduce the issue on the nRF53 DK. I’ve attached a .zip file containing the source code, board files, and overlays.

The firmware is designed to initialise and activate LED1 on the DK, stay active for 2 seconds, then reboot. This works as expected for the first ~4 reboots*, but after that, the device fails to reboot correctly. It hangs for around 30 seconds (until the WDT window elapses), at which point the watchdog timer kicks in and the device reboots properly. Strangely, this doesn’t happen when the RTT viewer is connected. The issue occurs regardless of whether I use sys_reboot or NVIC_SystemReset to trigger the reboot. The firmware is built using nRF SDK 2.2.0.

*The issue tends to occur between the 3rd and 5th reboot.

To compile and run the firmware:

Set nRF toolchain and SDK to NCS v2.2.0
Select "demo_boards_cpuapp_ns" as the board type
Include the overlay-rtt.conf Kconfig fragment
Add external_flash.overlay and power_enhance.overlay
Flash the firmware onto the DK
Don't connect to RTT! Instead, connect via serial interface to view logs through UART

Below is a screenshot of my build configuration from the nRF VS Code extension.

When the issue arises (typically after 3-5 reboots), the logs will freeze on the following output for about 30 seconds.

After the WDT has elapsed the device will reset and boot.

devzone.nordicsemi.com/.../demo_5F00_wdt_5F00_issue.zip

Let me know if you need anything more from me.

RE: Firmware Reset failing with MCUboot and WDT

Vidar Berg — Fri, 06 Sep 2024 12:07:13 GMT

The boot banner is printed during system initialization in bg_thread_main(), so normally it should appear once during the startup of mcuboot and then a second time when the application boots up.

Is it possible to attach a debugger to your custom board? If not, do you think it would be possible to reproduce this on a DK so I can try to debug it here?

RE: Firmware Reset failing with MCUboot and WDT

hugzy123 — Fri, 06 Sep 2024 08:45:20 GMT

[quote userid="4240" url="~/f/nordic-q-a/114321/firmware-reset-failing-with-mcuboot-and-wdt/500964"]I would have expected the "boot banner" to be printed before the WD reset if the system initialization in bg_thread_main() completed succesfully. [/quote]

There is a boot banner being printed, its actually printed three times; once just after the application reset, a second after the first bootloader sequence has finished and a third after the WDT reset and the second bootloader sequence.

[quote userid="4240" url="~/f/nordic-q-a/114321/firmware-reset-failing-with-mcuboot-and-wdt/500964"]Could you place a breakpoint at the call to z_sys_init_run_level(INIT_LEVEL_POST_KERNEL); and another at the call to z_sys_init_run_level(INIT_LEVEL_APPLICATION); to check if both are reached?[/quote]

I'm using UART so I can't use breakpoints, however, I have added multiple log messages and observed the application reaches just before main(); in the function bg_thread_main on the first failed reset attempt (before the WDT reset).

RE: Firmware Reset failing with MCUboot and WDT

Vidar Berg — Tue, 03 Sep 2024 11:55:04 GMT

Hi,

[quote user="hugzy123"]From what I can tell, it seems the application is hanging on the main() function inside bg_thread_main(). All the functions called before main() appear to be exiting without any issues.[/quote]

I would have expected the "boot banner" to be printed before the WD reset if the system initialization in bg_thread_main() completed succesfully.

https://github.com/nrfconnect/sdk-zephyr/blob/c0689b16ff127d1b71a4cc20310d476e1807e26e/kernel/init.c#L297

Could you place a breakpoint at the call to z_sys_init_run_level(INIT_LEVEL_POST_KERNEL); and another at the call to z_sys_init_run_level(INIT_LEVEL_APPLICATION); to check if both are reached?

RE: Firmware Reset failing with MCUboot and WDT

hugzy123 — Tue, 03 Sep 2024 08:17:33 GMT

Hi Vidar,

Thanks for your reply.

You're right that the "CONFIG_BOOT_WATCHDOG_FEED" option is included by default in the mcuboot configuration.

From what I can tell, it seems the application is hanging on the main() function inside bg_thread_main(). All the functions called before main() appear to be exiting without any issues.

RE: Firmware Reset failing with MCUboot and WDT

Vidar Berg — Fri, 30 Aug 2024 11:21:15 GMT

Hello,

The CONFIG_BOOT_WATCHDOG_FEED symbol should be selected by default when the build target is a nRF device. You can confirm this by if the symbol is set in the generated configuration file (build/mcuboot/zephyr/.conf). Either way, I would suggest that you try to make the WD timeout longer (e.g., 30 seconds), if you have not done so already. Then debug the device to see where it hangs when the program isn't reaching main().

Thanks,

Vidar