nRF52832 + NCS 2.5.1 HFXO Stuck on 100% of Time with no Zephyr/BT HFCLK Requester - MPSL Bug?

Summary

On a small fleet of identical nRF52832 units running the same firmware (NCS 2.5.1, SoftDevice Controller, BLE peripheral, LFCLK = RC with calibration), individual units intermittently latch into a state where the 32 MHz crystal (HFXO) runs continuously. Once latched it is permanent for the life of the power cycle (never recovers), and adds ~330–400 µA, flattening a 500 mAh cell in ~2 weeks instead of ~10 months. It is not reproducible on demand but recurs across units over time (3 different units so far).

We have observed the HFXO being on continuously by using an oscilloscope and reading the state directly. On a faulted unit HFXO is on 99.65% of the time, but `hfclk_users` in `clock_control_nrf.c` is `0x0`. My assumption is that this means neither the BLE controller nor any application/Zephyr `onoff` user is requesting HFCLK through the Zephyr clock-control path. I'm wondering if that points to a bug in the closed-source MPSL library. This looks like the same failure described in **DRGN-17014** ("HFCLK would stay active if it was turned on between timing events … during LFCLK calibration when using the RC oscillator as the LFCLK source"), which the changelog lists as fixed in NCS v1.9.0. We might be hitting a regression or a variant on v2.5.1?

Environment

  • SoC: nRF52832 (QFAA), single-mode BLE, peripheral role.
  • SDK: nRF Connect SDK v2.5.1 (Zephyr 3.4.99).
  • MPSL / SoftDevice Controller: binaries from **nrfxlib v2.5.1** (git tag `v2.5.1`, commit `ab72f33c86db7252dbf9a3ffec86c6b7fc6a9da7`).
  • Board: custom; reproduced/observed on production units.
  • Roles/stack: `CONFIG_BT_LL_SOFTDEVICE=y`, `CONFIG_MPSL=y`, `CONFIG_CLOCK_CONTROL_MPSL=y`.

Relevant configuration

# LFCLK on internal RC with periodic calibration against HFXO
CONFIG_CLOCK_CONTROL_NRF_K32SRC_RC=y
CONFIG_CLOCK_CONTROL_NRF_K32SRC_RC_CALIBRATION=y
CONFIG_CLOCK_CONTROL_NRF_CALIBRATION_LF_ALWAYS_ON=y
CONFIG_CLOCK_CONTROL_NRF_CALIBRATION_PERIOD=4000   # -> MPSL rc_ctiv=16 (4 s)
CONFIG_CLOCK_CONTROL_NRF_CALIBRATION_MAX_SKIP=1    # -> MPSL rc_temp_ctiv=2
CONFIG_CLOCK_CONTROL_NRF_CALIBRATION_TEMP_DIFF=2
CONFIG_CLOCK_CONTROL_NRF_ACCURACY=20               # 20 ppm


CONFIG_MPSL=y
CONFIG_CLOCK_CONTROL_MPSL=y
# (No LFXO fitted-in-firmware; LFCLK deliberately kept on RC. No MPSL timeslots used.)

The application makes no explicit HF-clock requests anywhere (e.g. no `clock_control_on`, `HFCLKSTART`, `sd_clock_hfclk_request`, `z_nrf_clock_control_hf_request`)

Symptom / impact

  • Excess current ~330–400 µA above the ~70 µA baseline; the 32 MHz crystal is visibly running continuously on a scope, and confirmed by register/RAM readback (below).
  • Permanent: once entered this bug survives entering the app's low-power/sleep state, survives the watchdog (which is being fed); only a full power cycle or reset clears it.
  • Per-device and intermittent: It appears on random units over days/weeks, not on all units, and not reproducible on demand. Consistent with a race that latches.

Direct evidence

We added a background sampler that reads `NRF_CLOCK->HFCLKSTAT` every 2 s and counts HFXO-on (SRC == Xtal) vs total samples, and we read `clock_control_nrf.c`'s `hfclk_users` from RAM via `nrfutil device read --direct`.

On the healthy unit HFCLK runs from the internal RC (HFINT) and HFXO is only pulsed ~0.7% (for radio events + the brief LFRC calibration). On the faulted unit HFXO is pinned on ~100%, yet `hfclk_users == 0`, so nothing above MPSL is requesting HFCLK.

The only remaining owner I can think of is MPSL's own clock management (LFRC calibration and/or radio HFCLK handling).

Parents
  • Hi Geoff

    Yes, I think this points to the HFXO being left on if "manually" doing a calibration sends the device back into the expected range. It does indeed seem strange that it stays on if the radio/SoftDevice controller uses it often. Does the radio/BLE keep working as expected when a device has gotten into this state, and the only out of order behavior is the HFXO running and current consumption increase?

    The fact that writing the calibration task fixes tells me the Softdevice controller might not be releasing the HFXO correctly since calibration stops it as expected, but I'll need to investigate some more and ask internally. Thank you for your patience and the thorough input here so far.

    Best regards,

    Simon

  • Hi Simon,

    Yes the radio/BLE works fine even with the device is in this stuck state (so does our application code). Nothing appears out of the ordinary except I can connect an oscilloscope to HFXO and see it is continuously running, and the current consumption is stuck around 450 uA rather than 70 uA (using your amazing Power Profiler Kit II to measure current!).

    Thanks for the update, I'm looking forward to hearing if you find anything internal to the Softdevice/MPSL. 

    Let me know if you want further debug information. If a device is stuck I can easily connect a programmer to it and read arbitrary memory to inspect register states e.t.c.

    Regards,

    Geoff

  • Hi Geoff

    I think I've found something. The MPSL requests the HFXO when doing LFRC calibration, but when MPSL is uninitialized it doesn't handle the clock DONE event and won't release the HFXO. This was fixed as of NCS 2.8.0 using the pull request here. https://github.com/nrfconnect/sdk-nrf/pull/16731, so in short, make sure MPSL does indeed release the clock when uninitialized. 

    Best regards,

    Simon

Reply Children
  • Hi  , thanks for the effort you've put in to dig around and find potential causes!

    I've looked at our code base, and I haven't found anywhere that the MPSL is disabled. There are no calls to `mpsl_lib_uninit()` or `bt_disable()` in our application, and CONFIG_MPSL_DYNAMIC_INTERRUPTS is not enabled in our build. On the Zephyr side calibration_work is still called every 4 seconds even on the faulty devices.

    I've checked our bootloader also and I don't think this uses the RC calibration part of the MPSL.

    Since we last talked I added "self-heal" functionality to the firmware on the 10 devices I had on hand. They have been running continuously for about a week now and I haven't yet recorded a "self-heal event" (i.e. the FW has not detected any that are stuck and called TASKS_CAL).

Related