nRF54L15 GRTC unable to wake chip at low temperatures

[mmdunc]

Hi,

We're in the product validation stage of development with nRF54L15, part of the validation is temperature cycling, we noticed that devices seem to be "stuck" in sleep mode at temperatures under -20 deg C. The chips remain in the low power state forever (even after heating back up), and it seems the GRTC passed through the latest compare value without waking the chip properly.

I was able to reproduce this exact issue with an nrf54l15dk and the broadcaster sample from nrf connect sdk v3.3.0. Steps to reproduce:

1. Compile bluetooth/broadcaster sample for nRF54l15dk/nrf54l15/cpuapp/ns and flash an nRF54L15dk
2. Place the board in a freezer set to maximum cold, this typically will allow it to reach under -20 in about 10 minutes
3. Notice the bluetooth advertising has stopped. (for debugging it may be helpful to change line 44 to  err = bt_le_adv_start(BT_LE_ADV_NCONN_IDENTITY, ad, ARRAY_SIZE(ad), NULL, 0); so the bluetooth address doesn't change).
   1. It may also be helpful to attach a PPK, as that also clearly shows the advertising current spikes stopped. I have plenty of screenshots of current behavior if helpful.

I then started a debug session, it seems the last GRTC compare value was passed by without waking the chip. Interestingly, it seems like upon reset the board typically re-enters this "stuck" state in less than 10 seconds (so long as it is cold).

(gdb) x/48wx 0x400e2520
0x400e2520:  0x00000000 0x00000000      0x00000000      0x00000000
0x400e2530:  0x00000000 0x00000000      0x00000000      0x00000000
0x400e2540:  0x00000000 0x00000000      0x00000000      0x00000000
0x400e2550:  0x00000000 0x00000000      0x00000000      0x00000000
0x400e2560:  0x00000000 0x00000000      0x00000000      0x00000000
0x400e2570:  0x00000000 0x00000000      0x00000000      0x00000000
0x400e2580:  0x6fdaef98 0x00000000      0x00000000      0x00000000
0x400e2590:  0x00000000 0x00000000      0x00000000      0x00000000
0x400e25a0:  0x6fcd6042 0x00000000      0x00000000      0x00000000
0x400e25b0:  0x6fcbc956 0x00000000      0x00000000      0x00000000
0x400e25c0:  0x00000000 0x00000000      0x00000000      0x00000000
0x400e25d0:  0x00000000 0x00000000      0x00000000      0x00000000
(gdb) x/16wx 0x400e2720
0x400e2720:  0x717467c5 0x00000000      0x00000000      0x00000000
0x400e2730:  0x71746fbc 0x00000000      0x00000000      0x00000000
0x400e2740:  0x7174781d 0x00000000      0x00000000      0x00000000
0x400e2750:  0x71747fc5 0x00000000      0x00000000      0x00000000

Interestingly, it seems the interrupt is pending in the NVIC, as well as the GRTC peripheral.

(gdb) x/wx 0xe000e11c
0xe000e11c:  0x00000028
(gdb)  x/wx 0xe000e21c
0xe000e21c:  0x00000028

(gdb) x/48wx 0x400e2300
0x400e2300:  0x00000000 0x00000000      0x00000000      0x00000000
0x400e2310:  0x00000040 0x00000040      0x00000040      0x00000040
0x400e2320:  0x00000000 0x00000000      0x00000000      0x00000000
0x400e2330:  0x00000500 0x00000500      0x00000500      0x00000100
0x400e2340:  0x00000000 0x00000000      0x00000000      0x00000000
0x400e2350:  0x00000000 0x00000000      0x00000000      0x00000000
0x400e2360:  0x00000000 0x00000000      0x00000000      0x00000000
0x400e2370:  0x00000000 0x00000000      0x00000000      0x00000000
0x400e2380:  0x00000000 0x00000000      0x00000000      0x00000000
0x400e2390:  0x00000000 0x00000000      0x00000000      0x00000000
0x400e23a0:  0x00000000 0x00000000      0x00000000      0x00000000
0x400e23b0:  0x00000000 0x00000000      0x00000000      0x00000000
(gdb) x/16wx 0x400e2100
0x400e2100:  0x00000000 0x00000000      0x00000000      0x00000000
0x400e2110:  0x00000000 0x00000000      0x00000001      0x00000000
0x400e2120:  0x00000001 0x00000001      0x00000000      0x00000000
0x400e2130:  0x00000000 0x00000000      0x00000000      0x00000000

We'd appreciate any help on this matter as this issue prevents the product from being viable.

[Priyanka]

Hi Duncan,

Could you give me a few more details?

1. What is the SoC revision that you are using?

2. Do you use MPSL in the failing configuration?

Could you try to test the same setup using a build where MPSL is enabled exactly as in the standard sample, without disabling or overriding the clock/MPSL-related configuration?

-Priyanka

[mmdunc]

Hi Priyanka,

1. Our product uses the BL54L15 rev 1, which I believe corresponds to nRF54L15 rev 1. The nRF54L15 devkits I was able to reproduce this issue on are rev 1 (QFAAB0) as well.

2. I am using MPSL in the failed configuration, I've attached the autoconf.h file from the build directory which shows CONFIG_MPSL=y and CONFIG_CLOCK_CONTROL_MPSL=y. I've made no changes to the bluetooth/broadcaster sample, there shouldn't be anything overriding MPSL configs as far as I can tell.
1638.autoconf.h

[Priyanka]

Hi Duncan,

Could you take a look at this errata: https://docs.nordicsemi.com/bundle/errata_nRF54L15_Rev1/page/ERR/nRF54L15/Rev1/latest/anomaly_L15_30.html 

The workaround for the errata is implemented in v3.0.0, but it is not enabled by default.

Could you try the following in order to enable it,

• CLOCK_CONTROL = y

• CLOCK_CONTROL_NRF_HFINT_CALIBRATION = y

• CLOCK_CONTROL_NRF_HFINT_CALIBRATION_PERIOD = xxx
  
  where xxx need to be chosen according to worst case scenario; by default this is set to 60 seconds, and the time may need shortening

 

In addition to this, you may need to increase the crystal startup time MSPL expects:

https://docs.nordicsemi.com/bundle/ncs-latest/page/kconfig/index.html#CONFIG_MPSL_HFCLK_LATENCY

 -Priyanka

[mmdunc]

Thanks for forwarding this. This didn't directly solve the issue, but it did lead me to find that building the sample for nrf54l15dk/nrf54l15/cpuapp instead of nrf54l15dk/nrf54l15/cpuapp/ns prevents the issue from happening. (In this functional build configuration the configs listed above aren't enabled)

The main difference between the two builds is the ns variant (which shows the anomaly) includes TFM, which is responsible for initializing the system since the application in non-secure mode, where many register settings are unavailable. We're continuing to look into this, I'm wondering if there might be another workaround may have been implemented in zephyr already, but doesn't get applied by TFM in the ns build variants?

To be clear, our product also uses TFM and ns build variants, which could explain the issue.

[Emil Lenngren]

The GRTC has both a low-frequency clock and a high-frequency clock; the second only being running while in active mode. See  RE: GRTC Periodic interval + nrfx for my previous post. When you put it into a freezer and don't have HFINT calibration activated, as done by MSPL, the high-frequency clock will desynchronize if the GRTC is runinng in active mode continuously for some time because the synchronization mechanism that otherwise runs every LFCLK tick, and keeps it synchronized with the low-frequency clock, only works if HFINT is not way too off. I guess theoretically, this could mean that when the GRTC goes to idle (i.e., turns off the high-frequency clock and schedules the internal CC register of the low-frequency clock when to wake up next time), it could schedule at a low-frequency tick that has already passed. But this requires that the GRTC has been active for not only a brief moment at a time, so that the desynchronization can actually make a difference, and that you then have some event scheduled not too far away in the future.