When frozen then reset, nRF52840 RC osc/RTC doesn't tick and LFCLKSRC, etc doesn't init correctly

RE: When frozen then reset, nRF52840 RC osc/RTC doesn't tick and LFCLKSRC, etc doesn't init correctly

Susheel Nuguru — Mon, 08 Jan 2024 12:42:05 GMT

rossquatch,

There are couple of debugging directions I can take from here but I would need your help to validate few things before I start to modify the DK I have on my desk.

External clock temperature coefficient with temperature.
I would like to remove the external clock issues from the picture. You said that when you freeze the chip (spray) then you are not effecting the external lfclk? I mean that the external osc is far enough in your board to be not effected by the temperature? Or I can rephrase this better, Are you sure that when the issue is happening then you have a valid stable clock input coming from the external osc?
K_Sleep being stuck
We have seen some issues with some RTC tick accuracy with temperature but did not see an issue with the freeze completely. Saying that, I think these accuracy tests we have done on other issues were done only with internal RC or a XTAL for LFCLK. So it is a bit of a new area for me to explore this issues with external OSC (not XTAL).
LFCLKSTAT register

The default values of the LFCLKSTAT register at boot time will always show that RC is running until the external clock is stable. We need to rule out that at low temperatures we still get the stable clock.

On my end, the default setup will be nRF52840 DK and that have an external crystal for the LFCLK. We have recently done some tests in cold temperature (-30 to -32 C) in the temperature chamber that there is no issue with the RTC tick being stalled but there was some issue with little drift. I do not want to do the same experiment again just to sniff the LFCLKSTAT unless I know that this is necessary to do.

RE: When frozen then reset, nRF52840 RC osc/RTC doesn't tick and LFCLKSRC, etc doesn't init correctly

Susheel Nuguru — Mon, 08 Jan 2024 08:52:16 GMT

rossquatch,

Happy New Year and sorry for the delays. I am back from the holidays and will be looking into this during next few days. I will be back to you with my observations in the coming days. If you have any more observations that you might have made, then please share it here.

RE: When frozen then reset, nRF52840 RC osc/RTC doesn't tick and LFCLKSRC, etc doesn't init correctly

Susheel Nuguru — Fri, 15 Dec 2023 07:18:25 GMT

rossquatch,

Most of us are starting our Christmas vacation from Monday, So I need to start this investigation after the Christmas. Sorry about the delay, but I am reserving days in my calendar after Christmas holidays to 100% focus on this investigation.

RE: When frozen then reset, nRF52840 RC osc/RTC doesn't tick and LFCLKSRC, etc doesn't init correctly

rossquatch — Wed, 13 Dec 2023 17:52:33 GMT

Thanks! I don't think the problem's specific to the Fanstel module. I think you should see the same problem with nRF52840s that are mounted directly to the board, provided they are the same ones Fanstel uses (which is detailed in previous posts above.)

The steps to reproduce the problem are also detailed in a previous post To recap, the steps to reproduce are: (make sure you reset the board the way I said... i.e. don't just cycle power. Cycling power may also cause the issue, but I haven't tested that.)

Cool a running board to -20C (we used a -18C freezer, or freeze spray with temp unknown)
Reset using either Segger Ozone/J-Link reset or the Zephyr reset function
Don't call k_sleep in your test application, it will likely hang when frozen
When application starts running, check LFCLKSRC (should be LFCLKSRC_EXT_OSC, but I get LFCLKSRC_RC)
Also check if clock register is ticking (I used the z_nrf_rtc_timer_get_ticks() function) I see the tick count remain the same until the board warms.
After the board warms up, z_nrf_rtc_timer_get_ticks shows the clock is ticking again, but it is still using the LFCLKSRC_RC and some of the other status registers are wrong

RE: When frozen then reset, nRF52840 RC osc/RTC doesn't tick and LFCLKSRC, etc doesn't init correctly

rossquatch — Wed, 13 Dec 2023 17:48:47 GMT

The post should be public, because it contains work-around code for a specific problem other developers maybe be experiencing.

You can buy modules direct from Fanstel ... they cost $8.50.
I provided a link above in a previous post where you can buy them.

I suggest you analyze the failure using nRF52840s there, I don't think it's specific to Fanstel's module, the problem is with the RC oscillator and LFCLKSRC selection hardware inside the nRF52840. I think this particular problem (which is quite specific) slipped through your testing.

RE: When frozen then reset, nRF52840 RC osc/RTC doesn't tick and LFCLKSRC, etc doesn't init correctly

Susheel Nuguru — Wed, 13 Dec 2023 13:43:51 GMT

rossquatch,

We will do some temperature tests on that chip version with a DK in cold temperatures. You might be onto something here and we will do our end of investigation. Nevertheless, like Jennifer says, we will not do those tests on Fanstel module but on our DK to see if we can isolate the issue to be at our end.

RE: When frozen then reset, nRF52840 RC osc/RTC doesn't tick and LFCLKSRC, etc doesn't init correctly

Jennifer Mendoza — Wed, 13 Dec 2023 12:40:47 GMT

Hi Ross,

Unfortunately, we do not have Fanstel modules to test and verify your failure. Without the failing module, we are unable to verify if it is an nRF5240 issue or something else inside the module.

I would recommend that you first check with Fanstel on these observed failures and see if they can analyze the issue and comment on your proposed work-around.

Until we have understood the root-cause of the abnormal operation at cold temperature, we are unable to comment on your proposed work-around.

A request has been made to make this case "public". Please note that it will be set back to "private" if Nordic will move forward with the device/module investigation.

Sincerely, Jennifer

RE: When frozen then reset, nRF52840 RC osc/RTC doesn't tick and LFCLKSRC, etc doesn't init correctly

rossquatch — Tue, 12 Dec 2023 16:56:00 GMT

The nRF52840 is on a BC840M Fanstel module --- I do not have access to the top of the nRF52840. Please see my previous replies for links to Fanstels site and more information.

The problem is with the LFCLKRC oscillator within the chip. It happens with every device we tested. We will not be sending you our devices as they're prototypes and we have a limited number. I suggest you test your chips using some of Nordic's development boards. If you read back through the replies I have already detailed how to reproduce the bug.

Also, why has this post been changed to "private?" There is a code work-around at the top of the post, which will be useful to other people that are having the same problem. I made the post public for that reason. Please change the post back to "public."

RE: When frozen then reset, nRF52840 RC osc/RTC doesn't tick and LFCLKSRC, etc doesn't init correctly

Jennifer Mendoza — Tue, 12 Dec 2023 14:55:15 GMT

Hi Ross,

This case is now transferred to Quality.

Could you indicate the Trace code of the failing nRF52840? This is to be found on the top mark of the device.

It would also help if you can send us 2~3 of your failing devices (if possible still board mounted) and we will try to reproduce the failures. Let me know if this is possible and I will give you the shipping instructions.

[quote user="rossquatch"]Every chip we tested fails. [/quote]

How many chips have you built and tested?

Sincerely, Jennifer Mendoza, Field Quality Manager

RE: When frozen then reset, nRF52840 RC osc/RTC doesn't tick and LFCLKSRC, etc doesn't init correctly

Susheel Nuguru — Tue, 12 Dec 2023 07:44:25 GMT

Issues with the external clocks on external boards most of the time turns out to be an issue with custom boards.

The LFCLKSTAT thing sounds wrong so we continue the investigation here at Nordic. We need to firmware that you tested this where we should see that k_sleep hangs when chip is too cold. I will transfer this to the Quality team. Normally customers send faulty boards to us (even if it is on the module), but this you can agree with the Quality team. I will write a note to them.

To the Quality team, the below is the main suspicion about the faulty behavior.

[quote user=""]// RSW - When the ARM is cold, then reset by Zephyr reset function, debugger reset, etc it will
// not switch the LFCLK source properly between the internal RC and external oscillator. The RC osc
// stays selected, but does not increment the timer count register (RC may or may not be oscillating)
// This causes k_sleeps to hang, because at least according to the ARM hw, there is no LFCLK timer tick[/quote]

RE: When frozen then reset, nRF52840 RC osc/RTC doesn't tick and LFCLKSRC, etc doesn't init correctly

rossquatch — Mon, 11 Dec 2023 21:14:02 GMT

Every chip we tested fails. You're the chip manufacturer, I don't need to send to you your own chips. 🤣

You can buy the Fanstel BC840M modules direct from Fanstel

If you really don't have any nRF52840-CKAA ARMs like what's on the Fanstel module. you can buy those from DIgikey

If you need to know specifically which nRF52840-CKAA uses on their BC840M, you can contact FANSTEL. According to their website they use the "Nordic nRF52840 CKAA (Rev. D silicon) with ARM Cortex M4F"

RE: When frozen then reset, nRF52840 RC osc/RTC doesn't tick and LFCLKSRC, etc doesn't init correctly

Susheel Nuguru — Mon, 11 Dec 2023 14:47:17 GMT

Sorry for delayed response rossquatch.

no, I do not know of any other way to wait for clock stopping.
If you want us to investigate this, then I can forward this ticket to Quality team and you can ship the failing chips to them?

RE: When frozen then reset, nRF52840 RC osc/RTC doesn't tick and LFCLKSRC, etc doesn't init correctly

rossquatch — Thu, 07 Dec 2023 22:05:31 GMT

Thanks for the feedback!

That was something I'd wanted to do and what you have there looks similar to the implementation in the nrf_clock_is_running HAL function

Unfortunately when I tried it just now, it didn't work in this specific situation:

When the system's at room temp, it works fine
When the system's frozen and reset, that loops hangs until the board warms. If I take that line with the check for the RUNNING bit back out, it goes back to working fine when frozen

From my original notes, it looked to me like the LFSTAT register state might be screwed up when the system was frozen, so that could be why the check for the RUNNING bit cleared isn't worked after a stop task.

In any event, do you know any other way to check/wait for the LFCLK stopping?

RE: When frozen then reset, nRF52840 RC osc/RTC doesn't tick and LFCLKSRC, etc doesn't init correctly

Susheel Nuguru — Thu, 07 Dec 2023 11:52:41 GMT

[quote user="rossquatch"]Really what I'd like is your feedback on my suggested workaround.[/quote]

Your fix seems harmless, so I do not think adding it will make anything wrong. Just a small suggestion after stopping the clock, it is better to wait.

   // RSW - they didn't explicitly stop the clock in the online post, but I think you
   // probably want to stop it before changing the source...?  The reference manual doesn't
   // say.  There is also no EVENTS_LFCLKSTOPPED flag.
   NRF_CLOCK->TASKS_LFCLKSTOP = 1;
   
   while((NRF_CLOCK->LFCLKSTAT &
             CLOCK_LFCLKSTAT_STATE_Msk) >> CLOCK_LFCLKSTAT_STATE_Pos);

RE: When frozen then reset, nRF52840 RC osc/RTC doesn't tick and LFCLKSRC, etc doesn't init correctly

rossquatch — Tue, 05 Dec 2023 23:23:22 GMT

It seems fairly clear to me it's the nRF chip, as the RC oscillator and LFCLK controller are internal to the chip, but I encourage you to investigate further. These are the steps to reproduce the issue on my system, which is running a fork of the v2.7.1 branch of Zephyr w/ a stock bootloader, configured to use an external 32.768Khz rail-to-rail external oscillator connected to P0.00/XCL1, with P0.01/XCL2 unconnected:

Cool a running board to -20C (we used a -18C freezer, or freeze spray with temp unknown)
Reset using either Segger Ozone/J-Link reset or the Zephyr reset function
Don't call k_sleep in your test application, it will likely hang when frozen
When application starts running, check LFCLKSRC (should be LFCLKSRC_EXT_OSC, but I get LFCLKSRC_RC)
Also check if clock register is ticking (I used the z_nrf_rtc_timer_get_ticks() function) I see the tick count remain the same until the board warms.
After the board warms up, z_nrf_rtc_timer_get_ticks shows the clock is ticking again, but it is still using the LFCLKSRC_RC and some of the other status registers are wrong

Running the same test at room temp, everything worked properly, LFCLKSRC is set to LFCLKSRC_EXT_OSC, clock is ticking, etc...

Really what I'd like is your feedback on my suggested workaround.

Thanks!

RE: When frozen then reset, nRF52840 RC osc/RTC doesn't tick and LFCLKSRC, etc doesn't init correctly

Susheel Nuguru — Tue, 05 Dec 2023 12:31:39 GMT

rossquatch,

I need some more time to do some internal investigation if we have seen something like this before. And then comment on the workaround you suggested.

It is still not clear if the issue is with the nRF Chip ( including ARM core) or on the Fanstel module. I will update you once I have some info on this.

RE: When frozen then reset, nRF52840 RC osc/RTC doesn't tick and LFCLKSRC, etc doesn't init correctly

rossquatch — Mon, 04 Dec 2023 19:27:14 GMT

I didn't say the chip was being tested at the -40C spec limit

Our freezer is set at -18C.
Every board tested in the freezer has this issue. They all work fine at room temp.
I do not have access to the chip's marking. The chip is on a Fanstel BC840M module, which is also rated for -40C. The areas I have issue with (the RC oscillator and LFCLK module) are all internal to the ARM

Looks to me like the RC oscillator may be not starting after reset at low temperatures. This is detailed to a great extent in my original post. If this is in fact the issue, I proposed a workaround and was looking for confirmation/comments/etc on the viability of that solution.

Thanks,
Ross

RE: When frozen then reset, nRF52840 RC osc/RTC doesn't tick and LFCLKSRC, etc doesn't init correctly

Susheel Nuguru — Mon, 04 Dec 2023 11:22:32 GMT

Hi Rossquatech,

This seems to be very close to what we specify as the absolute temperature range for this chip (-40 degree C).

Can you please confirm if my understanding is right?
So based on your description,

the chip is being tested at room temperature and at -40 Celsius.
Both of them have LFCLKSRC set to LFCLK and then switches to LF XTAL.

Is this being tested only in one board or is it reproducible on every board?

What is the lazer marking on the nRF52840 chip?

Have you tested this at -38 degree Celsius and see the same issues?