nRF52832 + NCS 2.5.1 HFXO Stuck on 100% of Time with no Zephyr/BT HFCLK Requester - MPSL Bug?

Summary

On a small fleet of identical nRF52832 units running the same firmware (NCS 2.5.1, SoftDevice Controller, BLE peripheral, LFCLK = RC with calibration), individual units intermittently latch into a state where the 32 MHz crystal (HFXO) runs continuously. Once latched it is permanent for the life of the power cycle (never recovers), and adds ~330–400 µA, flattening a 500 mAh cell in ~2 weeks instead of ~10 months. It is not reproducible on demand but recurs across units over time (3 different units so far).

We have observed the HFXO being on continuously by using an oscilloscope and reading the state directly. On a faulted unit HFXO is on 99.65% of the time, but `hfclk_users` in `clock_control_nrf.c` is `0x0`. My assumption is that this means neither the BLE controller nor any application/Zephyr `onoff` user is requesting HFCLK through the Zephyr clock-control path. I'm wondering if that points to a bug in the closed-source MPSL library. This looks like the same failure described in **DRGN-17014** ("HFCLK would stay active if it was turned on between timing events … during LFCLK calibration when using the RC oscillator as the LFCLK source"), which the changelog lists as fixed in NCS v1.9.0. We might be hitting a regression or a variant on v2.5.1?

Environment

  • SoC: nRF52832 (QFAA), single-mode BLE, peripheral role.
  • SDK: nRF Connect SDK v2.5.1 (Zephyr 3.4.99).
  • MPSL / SoftDevice Controller: binaries from **nrfxlib v2.5.1** (git tag `v2.5.1`, commit `ab72f33c86db7252dbf9a3ffec86c6b7fc6a9da7`).
  • Board: custom; reproduced/observed on production units.
  • Roles/stack: `CONFIG_BT_LL_SOFTDEVICE=y`, `CONFIG_MPSL=y`, `CONFIG_CLOCK_CONTROL_MPSL=y`.

Relevant configuration

# LFCLK on internal RC with periodic calibration against HFXO
CONFIG_CLOCK_CONTROL_NRF_K32SRC_RC=y
CONFIG_CLOCK_CONTROL_NRF_K32SRC_RC_CALIBRATION=y
CONFIG_CLOCK_CONTROL_NRF_CALIBRATION_LF_ALWAYS_ON=y
CONFIG_CLOCK_CONTROL_NRF_CALIBRATION_PERIOD=4000   # -> MPSL rc_ctiv=16 (4 s)
CONFIG_CLOCK_CONTROL_NRF_CALIBRATION_MAX_SKIP=1    # -> MPSL rc_temp_ctiv=2
CONFIG_CLOCK_CONTROL_NRF_CALIBRATION_TEMP_DIFF=2
CONFIG_CLOCK_CONTROL_NRF_ACCURACY=20               # 20 ppm


CONFIG_MPSL=y
CONFIG_CLOCK_CONTROL_MPSL=y
# (No LFXO fitted-in-firmware; LFCLK deliberately kept on RC. No MPSL timeslots used.)

The application makes no explicit HF-clock requests anywhere (e.g. no `clock_control_on`, `HFCLKSTART`, `sd_clock_hfclk_request`, `z_nrf_clock_control_hf_request`)

Symptom / impact

  • Excess current ~330–400 µA above the ~70 µA baseline; the 32 MHz crystal is visibly running continuously on a scope, and confirmed by register/RAM readback (below).
  • Permanent: once entered this bug survives entering the app's low-power/sleep state, survives the watchdog (which is being fed); only a full power cycle or reset clears it.
  • Per-device and intermittent: It appears on random units over days/weeks, not on all units, and not reproducible on demand. Consistent with a race that latches.

Direct evidence

We added a background sampler that reads `NRF_CLOCK->HFCLKSTAT` every 2 s and counts HFXO-on (SRC == Xtal) vs total samples, and we read `clock_control_nrf.c`'s `hfclk_users` from RAM via `nrfutil device read --direct`.

On the healthy unit HFCLK runs from the internal RC (HFINT) and HFXO is only pulsed ~0.7% (for radio events + the brief LFRC calibration). On the faulted unit HFXO is pinned on ~100%, yet `hfclk_users == 0`, so nothing above MPSL is requesting HFCLK.

The only remaining owner I can think of is MPSL's own clock management (LFRC calibration and/or radio HFCLK handling).

Parents Reply Children
  • Hi Simon,

    Thanks for the reply.

    We have only seen this on a custom board so far (10's of thousands of these produced and out in production). I understand how reproducing it on a DK (or not be able to) would be insightful -- I don't think it would be an easy task to port the firmware across but I'll consider it (it's not reliably reproducable even on our custom boards).

    R.e. errata 192, I see 

    #define USE_WORKAROUND_FOR_ANOMALY_192 1

    in nrfx_clock.c. But I don't think this is actually included in the build, and instead it is managed by the closed-source MPSL (I can't see if the MPSL is calling it but I assume so).

    I can confirm that the PR to fix the HFXO issue is in our Git tree.

    Regards, Geoff

  • Also, I just connected to a device with the fault via SWD and probed some registers. Here are the values:

    HFCLKSTAT    : 0x00010001  SRC=Xtal/HFXO  STATE=running
      HFCLKRUN     : 0x00000001
      LFCLKSTAT    : 0x00010000
      hfclk_users  : 0x00000000  BT=0 GENERIC=0
      HFXO duty    : 99.72%  (903680/906196)
      silicon      : part 0x00052832  variant 'AAE1' -> Rev 2
      device id    : 0xB6E1AB23A09E2999
      CLOCK INTEN  : 0x00000008  HFSTARTED=0 DONE=1 CTTO=0
      CLOCK events : DONE=0 CTTO=0
      NVIC (IRQ0)  : enabled=1 pending=0 active=0 prio=96
      TEMP         : DATARDY=0 irq_en=0  SWI5 irq_en=1

    The CLOCK_INTENSET = 0x08 is what I would expect -- the RC calibration wants to know when the process is DONE.

    Something that might be worthy to note is that our application code calls irq_lock() and locks interrupts for about 600 us every second. Given the radio ISRs cannot be blocked by this (they are priority 0), it would mean the MPSL's radio's ISRs would run but it's CLOCK ISRs would block if we happened to have locked interrupts at that point. I was wondering if this could cause a race condition in the MPSL?

  • This info might of interest also -- I attached a programmer to a faulty device and wrote TASKS_CAL = 1 to kick off a LF RC clock calibration sequence. This fixed the issue. The current dropped back down from the 450 uA or so to our normal 70 uA.

    Is this more evidence to suggest for some reason the LF RC clock calibration sequence is holding the HFXO on? On one thought perhaps, but then again it could be something else that is turning it on, and it's just that the calibration sequence is something that turns it back off again. Then again, the BT radio would be requesting the HFXO regulalry too and that doesn't seem to fix it.

Related