nRF54L15: MCUboot hangs forever in sys_clock_disable() — unbounded GRTC SYSCOUNTER read after nrfx_grtc_uninit()

Attachment: nrf54l15-wdt-grtc-repro.zip (sysbuild project, builds and reproduces on a stock nRF54L15 DK)

Environment

Summary

sys_clock_disable() in zephyr/drivers/timer/nrf_grtc_timer.c switches the GRTC SYSCOUNTER off before it releases the LFCLK. Releasing the LFCLK synchronously aborts the LFCLK RC-calibration backoff k_timer, and aborting a k_timer makes the kernel ask the system clock what time it is. That read targets the counter that was just switched off, returns SYSCOUNTERH.BUSY = 1 forever, and the driver's retry loop is unbounded — so MCUboot spins in it and never chain-loads the application. No reset recovers the device: the next boot runs the same code and wedges in the same place.

What the user sees

MCUboot prints its full log, ending in

I: Jumping to the first image slot

and then nothing. The application never starts. In the field this is triggered by a watchdog reset, which turns the watchdog from a recovery mechanism into a brick: every bite parks the unit in the bootloader until someone power-cycles it — and the power cycle does not help either once the build is affected.

Root cause

zephyr/drivers/timer/nrf_grtc_timer.c:510:

void sys_clock_disable(void)
{
    nrfx_grtc_uninit();          /* 1: clears MODE.SYSCOUNTEREN -> counter off */
#if defined(CONFIG_CLOCK_CONTROL_NRF)
    int err;
    struct onoff_manager *mgr =
        z_nrf_clock_control_get_onoff((clock_control_subsys_t)CLOCK_CONTROL_NRF_TYPE_LFCLK);

    err = onoff_release(mgr);    /* 2: releases the LFCLK */
    __ASSERT_NO_MSG(err >= 0);

    nrfx_coredep_delay_us(1000);
#endif
}

Step 1: nrfx_grtc_uninit() calls nrfy_grtc_sys_counter_set(NRF_GRTC, false), which clears MODE.SYSCOUNTEREN. The counter stops. (MCUboot already gets CONFIG_NRF_GRTC_TIMER_STOP_AT_UNINIT=n — Zephyr defaults it to y if !MCUBOOT — so no TASKS_STOP is issued, but that mitigation is ineffective: clearing MODE stops the counter just as well.)

Step 2 then re-enters the kernel:

onoff_release                              lib/utils/onoff.c:501
  -> onoff_stop                            drivers/clock_control/clock_control_nrf.c:548
  -> lfclk_stop
  -> z_nrf_clock_calibration_lfclk_stopped()
  -> k_timer_stop(&backoff_timer)          drivers/clock_control/nrf_clock_calibration.c:268
  -> z_impl_k_timer_stop                   kernel/timer.c:250
  -> z_abort_timeout                       kernel/timeout.c:159
  -> elapsed()                             kernel/timeout.c:70
  -> sys_clock_elapsed()                   drivers/timer/nrf_grtc_timer.c:496
  -> counter() -> nrfx_grtc_syscounter_get()
  -> nrfy_grtc_sys_counter_get()           modules/hal/nordic/nrfx/haly/nrfy_grtc.h:329

and the loop at the end of that chain has no bound:

do {
    counter = nrf_grtc_sys_counter_get(p_reg);
} while (counter & NRFY_GRTC_SYSCOUNTER_RETRY_MASK);   /* BUSY | OVERFLOW */

A sleeping SYSCOUNTER clears BUSY within about one 32 kHz cycle. A disabled one never does, so this spins forever.

Preconditions

1. CONFIG_MULTITHREADING=y in the MCUboot image. MCUboot defaults it to n, but real bootloaders enable it — most commonly because the Zephyr SPI NOR driver requires it when the secondary slot is on external flash. The flash driver itself is not involved; the hang reproduces with both slots in internal RRAM and no SPI at all.
2. LFCLK from the internal RC with calibration enabled (CONFIG_CLOCK_CONTROL_NRF_K32SRC_RC_CALIBRATION, default y) — i.e. any board without a 32.768 kHz crystal. This is what puts a k_timer in sys_clock_disable()'s path.
3. The first LFRC calibration completes within MCUboot's lifetime, so the backoff k_timer is armed when do_boot() runs. Measured to be true on all three boards above — MCUboot's lifetime is ~98 ms on the DK and DVK, ~51 ms on the custom board, and the calibration completes well inside that.
4. The backoff timer is the first (nearest-expiring) entry in the kernel timeout list. z_abort_timeout() only consults the clock on the is_first branch:

   if (sys_dnode_is_linked(&to->node)) {
           bool is_first = (to == first());
           remove_timeout(to);
           to->dticks = TIMEOUT_DTICKS_ABORTED;
           ret = 0;
           if (is_first) {
                   sys_clock_set_timeout(next_timeout(elapsed()), false);
           }
   }

Condition 4 is what decides whether a given build is affected, and it is invisible in any configuration review. With MCUboot's stock CONFIG_TIMESLICE_SIZE=20 the time-slice timeout is always nearer than the 4000 ms (CONFIG_CLOCK_CONTROL_NRF_CALIBRATION_PERIOD) backoff timer, so the read never happens. Remove the slice timeout and the hang is immediate and total.

Minimal reproduction (stock nRF54L15 DK, no patches)

From the attached project:

west build -d build -p -b nrf54l15dk/nrf54l15/cpuapp --sysbuild
west flash -d build

Expected: the application banner appears after each reset.
Actual: MCUboot prints Jumping to the first image slot and the board is dead — on the very first boot, and after every subsequent reset of any kind.

The project is deliberately minimal. The application arms watchdog0, feeds it six times, stops feeding, and prints the reset cause on each boot. The only board overlay disables the LFXO, which is what forces RC + calibration as on crystal-less hardware. sysbuild/mcuboot.conf sets exactly three things that matter:

CONFIG_MULTITHREADING=y     # what a real bootloader with external-flash DFU has
CONFIG_CLOCK_CONTROL_NRF_K32SRC_RC=y
CONFIG_TIMESLICE_SIZE=0     # makes the backoff timer first in the timeout list

Comment out CONFIG_TIMESLICE_SIZE=0 and rebuild to get the non-hanging baseline; nothing else about the build changes. That single line is the whole difference between "boots forever" and "never boots again".

The watchdog is not required. The hang occurs on the very first boot after a plain pin reset (RESETREAS = 0x1), with no application banner at all — so the application never reached wdt_setup() and the watchdog was never armed. The watchdog is in the reproducer only because it demonstrates why this matters in the field: a reset that was supposed to recover the unit re-enters the identical code path.

Console note: printk output is on the DK's second VCOM port (vcom 1), 115200 8N1.

Forensics on the hung board (SWD, nrfutil device)

(Addresses are for the application core in secure mode: SYSCOUNTER[2], domain index GRTC_IRQ_GROUP = 2. The SYSCOUNTER runs at 1 MHz, so its raw value is microseconds.)

What does and does not revive the counter

The nRF54L15 datasheet, GRTC > SYSCOUNTER > Recommendation on reading SYSCOUNTER, specifies four steps: set SYSCOUNTER[m].ACTIVE, wait until SYSCOUNTERH.BUSY is cleared, read SYSCOUNTERL/H, then clear ACTIVE. nrfy_grtc_sys_counter_get() implements steps 2 and 3 only — it never touches ACTIVE — and nrfx_grtc_uninit() separately clears MODE.AUTOEN, MODE.SYSCOUNTEREN and ACTIVE (the last via nrfy_grtc_sys_counter_active_set(NRF_GRTC, false)). Read back on the hung DK:

GRTC.MODE                 0x500E2510 = 0x00000000   AUTOEN = 0, SYSCOUNTEREN = 0
SYSCOUNTER[2].ACTIVE      0x500E2748 = 0x00000000
SYSCOUNTER[2].SYSCOUNTERH 0x500E2744 = 0x40000000   BUSY = 1, forever
GRTC.KEEPRUNNING          0x500E2508 = 0x00000000

It is tempting to conclude that the skipped step 1 is the cause. It is not — I tested that on the hung board rather than assuming it, poking one register at a time:

So the missing steps are a real deviation from the documented procedure, but implementing them would not have prevented this hang. Once the counter has been switched off, only TASKS_START revives it — which is why the ordering fix, not the read procedure, is the substantive one. The datasheet's remark on the same page that "the internal low frequency timer must be started for the proper operation of the SYSCOUNTER" points the same way: sys_clock_disable() tears that down while a clock read is still reachable.

Incidentally, poking TASKS_START on the hung board resumes the boot: MCUboot's pending read completes, it chainloads the application, the application arms and then starves the watchdog, and the resulting reset lands back in the identical hang — RESETREAS = 0x4 (DOG1), counter frozen at 97.9 ms again. That is the field failure loop in full.

Bounded probe: the defective read is universal

mcuboot-grtc-read-probe.patch in the attachment (against bootloader/mcuboot/boot/zephyr/main.c) performs the same SYSCOUNTER read immediately after sys_clock_disable(), but with a 10,000,000-iteration bound and a park loop instead of a silent hang. On all three boards, after a pin reset:

grtc_probe_mode  = 0x00000000    /* MODE cleared by nrfx_grtc_uninit()  */
grtc_probe_tries = 10000000      /* bound exhausted                     */
grtc_probe_sch   = 0x40000000    /* BUSY still set                      */

This isolates the defect from the timeout-ordering question entirely: any read of the SYSCOUNTER after nrfx_grtc_uninit() hangs, on every board tested, regardless of silicon revision or reset type.

Why applications appear immune

The application's own sys_clock_driver_init() → nrfx_grtc_init() → nrfy_grtc_prepare() triggers TASKS_START unconditionally, repairing the counter before any application code can read it. Only code that reads the system clock between sys_clock_disable() and the next TASKS_START is exposed — in practice, MCUboot's own cleanup.

Workarounds (verified against the deterministic case above)

Either of these in the MCUboot image config makes the board boot cleanly through every reset instead of hanging on the first one:

CONFIG_CLOCK_CONTROL_NRF_K32SRC_RC_CALIBRATION=n   # preferred

or CONFIG_MULTITHREADING=n, where the bootloader permits it (not an option with the secondary slot on SPI flash).

Neither is a fix — they remove the caller, not the unbounded read. Any other kernel timeout aborted after nrfx_grtc_uninit() hits the same wall.

Suggested fix

1. Ordering (the substantive one). In sys_clock_disable(), release the LFCLK before calling nrfx_grtc_uninit(). That stops the calibration timer while the SYSCOUNTER is still running, so the abort's clock read succeeds normally.
2. Fail-safe read (containment). Make the GRTC read path bound its retry or early-out once the driver is uninitialized (nrfx_grtc_init_check() already exists). A peripheral read that can spin forever on a register bit is a hazard regardless of who calls it — see case 126653 below, where the same loop wedges an application at PRE_KERNEL_1 for an entirely different reason. As measured above, no read-side change can obtain a value from a counter that has been switched off, so this converts a silent hang into a detectable error rather than preventing it.

The existing CONFIG_NRF_GRTC_TIMER_STOP_AT_UNINIT=n default for MCUboot suggests the ordering hazard was already noticed, but it is not sufficient, because nrfx_grtc_uninit() clears MODE.SYSCOUNTEREN regardless of that option.

Related DevZone cases

Checked against the current threads; none is a duplicate of this report, but two are adjacent enough to be worth pre-empting.

• 124602 — "GRTC resets after sys_reboot() when using MCUboot (NCS v3.1.0)". Different symptom (the counter value being cleared, not a hang), same uninit code path. The fix from that thread is present in v3.4.0 as CONFIG_NRF_GRTC_TIMER_STOP_AT_UNINIT / ..._CLEAR_AT_UNINIT defaulting to y if !MCUBOOT, and my MCUboot build has both at n. It does not prevent this hang, because nrfx_grtc_uninit() stops the counter by clearing MODE.SYSCOUNTEREN whether or not TASKS_STOP is issued. Please do not close this as covered by that fix.
• 126653 — "Getting internal clocks to run on nRF54L15". Different trigger, same unbounded loop: the reporter wedges in nrfx_grtc_syscounter_get() at nrfx_grtc.c:282 during PRE_KERNEL_1 device init, reached via sys_clock_cycle_get_32() from z_log_msg_commit() inside psa_crypto_init() — i.e. an application reading the SYSCOUNTER before the clock is up rather than after it is torn down. Evidence that the unbounded retry is a general hazard with more than one caller, which is the argument for fix 2.
• 124294 — "Watchdog timer resets my GRTC". Nordic confirmed the datasheet reset table is wrong: "the WDT is indeed supposed to reset the GRTC, so what you're seeing is the intended behavior. We will update the datasheet to reflect this as soon as we can." Note that the reset overview table in the current nRF54L15 datasheet still shows no x in the GRTC.SYSCOUNTER column for "Watchdog timer reset", so that correction does not appear to have landed yet. Unrelated to this hang, but it misleads anyone reasoning about GRTC state across a watchdog reset.
• 124693 — "mcu hang at nrf_grtc_sys_counter_low_get". Superficially similar, but not this bug: NCS 3.0.2, no MCUboot involved, and the reporter had selected CONFIG_CLOCK_CONTROL_NRF_K32SRC_SYNTH=y, which depends on the HFXO. Nordic diagnosed a 32 MHz crystal start-up problem on that board. Listed here only so it is not mistaken for a prior report of this issue.

Note on the attachment

The zip contains board overlays for nrf54l15dk, bl54l15u_dvk and bl54l15_dvk only. The third board in the tables above is a private custom design; its board files are deliberately not included and nothing in the project depends on them. The nrf54l15dk/nrf54l15/cpuapp target reproduces everything on stock hardware.