Softdevice Assert at PC=0x15810 (S132 7.2.0) / RTC clock drift when using timeslot

RE: Softdevice Assert at PC=0x15810 (S132 7.2.0) / RTC clock drift when using timeslot

Einar Thorsrud — Wed, 10 Mar 2021 13:37:51 GMT

Hi Mike,

[quote user="m.wagner"]Does that have any implications on product certification?[/quote]

No, the choice of clock source have no impact on qualification or certifications. So the question is basically if this works well in your product or not.

[quote user="m.wagner"]

It would surprise me if there were any significant temperature variations. There is a motor with all the necessary periphery on the board as well, but my testing was done without any motor activity whatsoever.

Also - I think I mentioned it earlier - while I was still running the RC oscillator with its default configuration, I did not observe any recalibration due to temperature drift. I.e. I traced NRF_CLOCK->EVENTS_DONE via PPI and did never observe recalibration occurring after 4s instead of 8s.

[/quote]

I see, thanks for confirming. I just wanted to rule out any temperature change effects. (You are of course also right that this should have triggered more calibrations in that case.)

Thanks,

Einar

RE: Softdevice Assert at PC=0x15810 (S132 7.2.0) / RTC clock drift when using timeslot

m.wagner — Tue, 09 Mar 2021 15:27:03 GMT

[quote userid="7377" url="~/f/nordic-q-a/70252/softdevice-assert-at-pc-0x15810-s132-7-2-0-rtc-clock-drift-when-using-timeslot/298725#298725"]The downside is that the SoftDevice is not tested with SYNT clock source, which is why we state that it is not supported.[/quote]

Does that have any implications on product certification?

[quote userid="7377" url="~/f/nordic-q-a/70252/softdevice-assert-at-pc-0x15810-s132-7-2-0-rtc-clock-drift-when-using-timeslot/298725#298725"]I would expect you should see a further improvement by using RC_CTIV 1 to calibrate even more often (and still TEMP_CTIV 0)?[/quote]

I assume so, but I'll have to test this in long-term tests. I deliberately tested with NRF_SDH_CLOCK_LF_RC_CTIV 8, to see if the issue still persists and intend to further reduce the value and see how it behaves.

[quote userid="7377" url="~/f/nordic-q-a/70252/softdevice-assert-at-pc-0x15810-s132-7-2-0-rtc-clock-drift-when-using-timeslot/298725#298725"]Without knowing your HW I would not think this could be the main issue. If the HFXO is significantly off you would not get BLE working (frequencies would be shifted).[/quote]

That confirms what I discussed with our HW engineer.

[quote userid="7377" url="~/f/nordic-q-a/70252/softdevice-assert-at-pc-0x15810-s132-7-2-0-rtc-clock-drift-when-using-timeslot/298725#298725"] if your product is a type of product that could see significant rapid temperature changes (like a lightbulb when it is being turned on or off). Do you see this issue mostly around times when temperature changes, or also when the temperature is roughly stable?[/quote]

RE: Softdevice Assert at PC=0x15810 (S132 7.2.0) / RTC clock drift when using timeslot

Einar Thorsrud — Tue, 09 Mar 2021 13:38:07 GMT

Hi Mike,

[quote user="m.wagner"]

Are there any downsides to using this apart from current consumption? For our setup with power supply and maximised radio uptime (i.e. HFCLK is always running), it seems like this is actually the "better" solution for this issue than using the RC oscillator.

Considering the symptoms of the issue, I think this should resolve it from a software POV.

[/quote]

The downside is that the SoftDevice is not tested with SYNT clock source, which is why we state that it is not supported. That said my understanding is that there is no reason to believe that there should be any issues with SYNT other than current consumption, so it could be a good choice as long as you accept the risk with it not being properly tested.

[quote user="m.wagner"]

In the meantime, unfortunately it looked like the unwanted drift still occurs even with #define NRF_SDH_CLOCK_LF_RC_CTIV 8 and NRF_SDH_CLOCK_LF_RC_TEMP_CTIV 0.

The frequency of it happening is massively reduced, though. While it seemed to happen once in 10 - 15mins with NRF_SDH_CLOCK_LF_RC_CTIV 16 and NRF_SDH_

CLOCK_LF_RC_TEMP_CTIV 2, it only happened about twice or thrice a day with the above configuration.

[/quote]

I would expect you should see a further improvement by using RC_CTIV 1 to calibrate even more often (and still TEMP_CTIV 0)?

[quote user="m.wagner"]I also looked into Erratum 192, which looked like it may be interfering here, but the fact that it happens less frequently with more frequent calibration seems to contradict this...[/quote]

The workaround for erratum 192 is uses in the SoftDevice you use (7.2.0), so it should not be relevant in this case.

[quote user="m.wagner"]

I do have one suspicion as to the cause of the issue: Is it possible that the cause could be a mismatch of the HFXO load capacitance? It does not look right to me, but I'll have to talk to our HW engineer. We do not seem to have any issues with BLE, though.

EDIT2:

It does indeed seem that C_pin_HFXO and C_PCB have been neglected when calculating the required capacitance. The XTAL requires a C_L of 6pF while we currently do have a C_L in the range of 8pF to 9pF (assuming C_PCB between 0pF and 2pF).

[/quote]

Without knowing your HW I would not think this could be the main issue. If the HFXO is significantly off you would not get BLE working (frequencies would be shifted). It is good to look into though.

I have one other question. As you do not care about current consumption I was wondering if your product is a type of product that could see significant rapid temperature changes (like a lightbulb when it is being turned on or off). Do you see this issue mostly around times when temperature changes, or also when the temperature is roughly stable?

RE: Softdevice Assert at PC=0x15810 (S132 7.2.0) / RTC clock drift when using timeslot

m.wagner — Tue, 09 Mar 2021 09:23:56 GMT

Hi Einar,

Sorry for the late response - I've been out of the office for a few days.

[quote userid="7377" url="~/f/nordic-q-a/70252/softdevice-assert-at-pc-0x15810-s132-7-2-0-rtc-clock-drift-when-using-timeslot/297465#297465"]It seems clear that this is clock related, but not exactly how. Did you by any chance test with NRF_CLOCK_LF_SRC_SYNTH?[/quote]

I did not yet test with NRF_CLOCK_LF_SRC_SYNTH, but will do such a test now.

Considering the symptoms of the issue, I think this should resolve it from a software POV.

Status Quo

In the meantime, unfortunately it looked like the unwanted drift still occurs even with #define NRF_SDH_CLOCK_LF_RC_CTIV 8 and NRF_SDH_CLOCK_LF_RC_TEMP_CTIV 0.

The frequency of it happening is massively reduced, though. While it seemed to happen once in 10 - 15mins with NRF_SDH_CLOCK_LF_RC_CTIV 16 and NRF_SDH_

CLOCK_LF_RC_TEMP_CTIV 2, it only happened about twice or thrice a day with the above configuration.

[quote userid="7377" url="~/f/nordic-q-a/70252/softdevice-assert-at-pc-0x15810-s132-7-2-0-rtc-clock-drift-when-using-timeslot/297465#297465"]Perhaps it is somehow related to erratum 146 though I do not immediately see how.[/quote]

The application should account for 500ppm RC clock drift, so I'm not sure this applies. Though could this drift be even larger?

I also looked into Erratum 192, which looked like it may be interfering here, but the fact that it happens less frequently with more frequent calibration seems to contradict this...

[quote userid="7377" url="~/f/nordic-q-a/70252/softdevice-assert-at-pc-0x15810-s132-7-2-0-rtc-clock-drift-when-using-timeslot/297465#297465"]Another thing... When initializing the mesh stack with mesh_stack_init(), you pass a mesh_stack_init_params_t instance which has a member core.lfclksrc. Is this correctly configured, with the correct accuracy?[/quote]

FYI: Since I delivered the Mesh-example I reverted back to working on our actual non-mesh application. We only use parts of the Mesh-stack, specifically the timeslot- and bearer_handler-modules and whatever is needed to support these.

I did verify multiple times that the accuracy-parameter passed into timeslot_init() is correct (500ppm), so the drift calculation should work with the intended results.

EDIT:

EDIT2:

Regards,

-mike

RE: Softdevice Assert at PC=0x15810 (S132 7.2.0) / RTC clock drift when using timeslot

Einar Thorsrud — Wed, 03 Mar 2021 11:16:51 GMT

Hi Mike,

[quote user="m.wagner"]So, unfortunately I'm not sure what to make of this. It looks like there is something wrong, but what?[/quote]

It seems clear that this is clock related, but not exactly how. Did you by any chance test with NRF_CLOCK_LF_SRC_SYNTH? In that case the LF clock and HF clock would use the same clock source so there will not be any drift. Perhaps it is somehow related to erratum 146 though I do not immediately see how.

Update: There is a confirmed bug in the SoftDevice where it does not account for drift between HF clock and LF clock in the internal scheduler. I do not immediately see how it relates to this specific case as the SoftDevice only use the RTC and no timer in timeslots, so it would be up to the application to handle this drift, which it does. There may be something here, though.

[quote user="m.wagner"]I'm not sure I agree entirely - at least from my "black box perspective": the softdevice still signals NRF_EVT_RADIO_SESSION_IDLE at this point, so it does react somehow at the time the timeslot should have ended. So it looks like it should know that the application did not respect the time.[/quote]

Yes, you are right. It is the applications responsibility to clean up so that it have not configured any interrupts that will happen after the end of the timeslot on the restricted peripherals (like TIMER0). I agree that is not clearly stated though.

[quote user="m.wagner"] I was more trying to find out when and how this is triggered (through software? PPI? from what interrupt priority? Is there any difference in handling if temperature variation triggers the calibration?[/quote]

The decision on whether to calibrate or not depends on configuration as discussed. In the implementation a calibration timer fires regularly and the temperature is checked. If temperature has changed more than the threshold, calibration will be performed. If not a check if calibration has been skipped for too long is done, and if it is, calibration is performed. The actual calibration is done in the same way regardless of why it was triggered.

The calibration itself consists of very little code, most importantly these steps:

Request the HFXO clock
Check if HW clock is running (it takes some time to start). If it is continue, if not configure an interrupt on the HFCLOCK started event so that calibration can start when clock is ready
Apply first part of workaround for erratum 192.
Trigger NRF_CLOCK->TASKS_CAL
Wait for calibration to finish signaled by an NRF_CLOCK->EVENTS_DONE.
Do last part of erratum 192 workaround
release HFXO clock

Another thing... When initializing the mesh stack with mesh_stack_init(), you pass a mesh_stack_init_params_t instance which has a member core.lfclksrc. Is this correctly configured, with the correct accuracy?

Einar

RE: Softdevice Assert at PC=0x15810 (S132 7.2.0) / RTC clock drift when using timeslot

m.wagner — Wed, 03 Mar 2021 08:06:45 GMT

Hi Einar

[quote userid="7377" url="~/f/nordic-q-a/70252/softdevice-assert-at-pc-0x15810-s132-7-2-0-rtc-clock-drift-when-using-timeslot/297149#297149"] I have not found any reports of similar issues with the LFRC. Using regular calibration it should stay within +-500 ppm. If there is an issue here I would expect many other customers seeing a set of other issues as well (for instance missing BLE timing etc.), so it is a bit strange. We have found the default configuration, which you have described, to be good enough to not cause problems while not calibrating too often (in order to save power). I am not able to explain why this does not seem to be the case here. If it was just one device then I would suspect an issue with that specific device, but that doe snot seem to be the case her based on earlier discussions - though there are variations. [/quote]

So, unfortunately I'm not sure what to make of this. It looks like there is something wrong, but what?

Could there be an issue with the custom module or environment? I can't imagine that there are temperature variations > 0.5°C within 4s - that would really be unexpected, but I'm not too knowledgeable about hardware these days...
We did not observe any unexpected behaviour during BLE connections - which sometimes are established for rather long durations! In fact, it looked like the application runs more stable while BLE is active.
As I said, it looks like PCA20020 shows this same occasional behaviour from time to time as well!

EDIT: I just reverted back to the recommended original LFCLK configuration and traced NRF_CLOCK->EVENTS_DONE and it does not look like any temperature drifts are detected, the calibration is done at the 8s interval throughout!

[quote userid="7377" url="~/f/nordic-q-a/70252/softdevice-assert-at-pc-0x15810-s132-7-2-0-rtc-clock-drift-when-using-timeslot/297149#297149"]I see your point, but that is not in line with how the timeslot API is designed. The application does not signal that the timeslot has ended (sd_radio_session_close() is only called when a session ends, i.e. when the app no longer needs timeslots), so the SoftDevice does not know that the app did not respect the time until something unexpectedly happens when it should not.[/quote]

I'm not sure I agree entirely - at least from my "black box perspective": the softdevice still signals NRF_EVT_RADIO_SESSION_IDLE at this point, so it does react somehow at the time the timeslot should have ended. So it looks like it should know that the application did not respect the time.

But if this is normal behaviour, I think it would be great if it were somehow documented! I don't think I've seen this documented in the Softdevice specification.

Of course the SD would assert as soon as the application accesses one of the MWU-protected peripherals or something similar.

[quote userid="7377" url="~/f/nordic-q-a/70252/softdevice-assert-at-pc-0x15810-s132-7-2-0-rtc-clock-drift-when-using-timeslot/297149#297149"]Calibration is done in hardware. The SoftDevice simply starts the HFXO and triggers the CAL task (NRF_CLOCK_TASK_CALL) to trigger calibration and HW does the rest.[/quote]

Sorry, I was not clear with my question. I'm aware of the NRF_CLOCK->TASK_CAL, but I was more trying to find out when and how this is triggered (through software? PPI? from what interrupt priority? Is there any difference in handling if temperature variation triggers the calibration?

I'm just trying to figure out if there are any negative impacts if the LF-clock is calibrated more often besides power consumption. (We are not running on battery and have the RADIO receiver enabled almost all of the time - so power consumption is not an issue.)

Sorry for the unending amount of questions, but this whole issue does bother us quite a bit, as we still lack understanding in cause and details. I'm really happy to have found the workaround with more frequent calibration, thogh, it's quite relieving to be honest, as it looks like even the most "sensitive" device we have around is now running rock-solid.

Thank you for your patience & support. Best regards,

-mike

RE: Softdevice Assert at PC=0x15810 (S132 7.2.0) / RTC clock drift when using timeslot

Einar Thorsrud — Tue, 02 Mar 2021 11:43:19 GMT

Hi Mike,

These are very interesting findings.

[quote user="m.wagner"]How do I determine the "correct" values for NRF_SDH_CLOCK_LF_RC_CTIV and NRF_SDH_CLOCK_LF_RC_TEMP_CTIV? Why were the defaults not appropriate?[/quote]

This is a good question. I have not found any reports of similar issues with the LFRC. Using regular calibration it should stay within +-500 ppm. If there is an issue here I would expect many other customers seeing a set of other issues as well (for instance missing BLE timing etc.), so it is a bit strange. We have found the default configuration, which you have described, to be good enough to not cause problems while not calibrating too often (in order to save power). I am not able to explain why this does not seem to be the case here. If it was just one device then I would suspect an issue with that specific device, but that doe snot seem to be the case her based on earlier discussions - though there are variations.

[quote user="m.wagner"]Why does the softdevice not assert once it knows that timeslot is actually overstretched, but only fire the IDLE event? The COMPARE0-IRQ of RTC0 is enabled and active, after all so the possibility for the assert is there! This looks like unexpected behaviour by the softdevice.[/quote]

I see your point, but that is not in line with how the timeslot API is designed. The application does not signal that the timeslot has ended (sd_radio_session_close() is only called when a session ends, i.e. when the app no longer needs timeslots), so the SoftDevice does not know that the app did not respect the time until something unexpectedly happens when it should not.

[quote user="m.wagner"]How is the RC oscillator calibrated? Does this run in hardware entirely or is there code running, taking care of this? Is this running in interrupt handlers and at what priority? With As far as I understand , a temperature measurement is required if NRF_SDH_CLOCK_LF_RC_TEMP_CTIV is >= 2. Would be great if you could elaborate on this![/quote]

Calibration is done in hardware. The SoftDevice simply starts the HFXO and triggers the CAL task (NRF_CLOCK_TASK_CALL) to trigger calibration and HW does the rest.

Einar

RE: Softdevice Assert at PC=0x15810 (S132 7.2.0) / RTC clock drift when using timeslot

m.wagner — Mon, 01 Mar 2021 16:34:29 GMT

Hi Einar,

I looked a bit more into the behaviour of the device when this issue occurs - and tried to implement a workaround, too.

Tracing RTC0->EVENTS_COMPARE[1]

I did trace RTC0->EVENTS_COMPARE[1] via PPI with a logic analyzer, because this seems to be the timing reference the softdevice uses to assert timeslots are ended in time.

It looks like the compare event fires significantly earlier than normal when the unexpected behaviour occurs even though the timeslot was extended to the exact same duration as every other timeslot before.

I noticed that during undisrupted/regular operation, the period of RTC0->EVENTS_COMPARE[1] is in the range of 10'000'457us to 10'000'486us, with a standard deviation of ~28us, whereas the compare-event that occurs right before the unexpeced IDLE event fires is raised after e.g. 9'999'971us, so 486us earlier than the earliest of the other compare events, which is an order of magnitude more.

It looks like NRF_EVT_RADIO_SESSION_IDLE is then invoked within around 60us of RTC0->EVENTS_COMPARE[1], so lookis like this is triggered from the RTC0 COMPARE1 interrupt handler.

See the following logic analyzer trace (attached as Saleae Logic 2 data file, too).

devzone.nordicsemi.com/.../sd_2D00_assert_2D00_rtc_2D00_debug_2D00_1.zip

Tracing RTC0->CC[1]
I also looked at how the RTC0->CC[1] value changes when timeslots are extended. The nRF SDK for Mesh timeslot extension algorithm initially (when invoking sd_radio_request()) tries to reserve a "short" timeslot of 14000us duration and then tries to match the previous length by extending it up to 10'000'000us (without BLE activity this will be achieved).

I tried to see whether there were any anomalies when calculating the CC[1] value for the RTC, but did not observe any. When receiving NRF_RADIO_CALLBACK_SIGNAL_TYPE_START, the difference between NRF_RTC0->COUNTER and NRF_RTC0->CC[1] is consistently at 459, corresponding to around 14'000us. After extending to 10s (NRF_RADIO_CALLBACK_SIGNAL_TYPE_EXTEND_SUCCEED), the difference is consistently at 327681, corresponding to ~10'000'000us. This is no different for the timeslot in which the observed issue occurs.

Looking at another logic analyzer trace where I also traced the LFCLK, (RTC0 TICK event via PPI), I came to the conclusion that the reason of the early compare event apparently seems to be RC clock drift that is larger than assumed.

The amount of ticks counted from about the time the timeslot extension concluded up to the point the RTC0 COMPARE1 event fires was consistently higher for the period after which NRF_EVT_RADIO_SESSION_IDLE is invoked, while the measured duration was consistently significantly shorter.

I attached the Logic trace (recorded with the below workaround active, so there are no Softdevice asserts happening...)

devzone.nordicsemi.com/.../sd_2D00_assert_2D00_measure_2D00_rtc_2D00_tick.zip

Workaround

I figured - since I can actually detect the unexpected NRF_EVT_RADIO_SESSION_IDLE, I could try a workaround where I just force stop all my timeslot activity (while disabling the MWU region that watches RADIO and TIMER0...) and re-order a new timeslot. It seems to be a viable workaround since in our application this issue seems to occur only or at least mostly while there is no BLE activity (advertising or connection) at all.

A bug in the timeslot_timer prevented the workaround to work from the get-go, but once I figured that out, it started to look promising.

I do not include the code here, since it's probably a bad idea to actually use it...

The actual solution

As it turns out, the actual solution to the problem seems to be way simpler and less hacky than my workaround. It looks like for some reason the "normal" values of NRF_SDH_CLOCK_LF_RC_CTIV (16, corresponding to 4 seconds) and NRF_SDH_CLOCK_LF_RC_TEMP_CTIV (2, corresponding to force recalibration every second interval - so I assume 8 seconds), seem to not result in an accurate enough LFCLK. I reduced NRF_SDH_CLOCK_LF_RC_CTIV to 4 (corresponding to every second) and NRF_SDH_CLOCK_LF_RC_TEMP_CTIV to 0 (corresponding to recalibration at every interval).

So far, this has been running stable even on the "wonkier" devices. So two questions remain:

How do I determine the "correct" values for NRF_SDH_CLOCK_LF_RC_CTIV and NRF_SDH_CLOCK_LF_RC_TEMP_CTIV? Why were the defaults not appropriate?
Why does the softdevice not assert once it knows that timeslot is actually overstretched, but only fire the IDLE event? The COMPARE0-IRQ of RTC0 is enabled and active, after all so the possibility for the assert is there! This looks like unexpected behaviour by the softdevice.

EDIT:

3. How is the RC oscillator calibrated? Does this run in hardware entirely or is there code running, taking care of this? Is this running in interrupt handlers and at what priority? With As far as I understand , a temperature measurement is required if NRF_SDH_CLOCK_LF_RC_TEMP_CTIV is >= 2. Would be great if you could elaborate on this!

RE: Softdevice Assert at PC=0x15810 (S132 7.2.0) / RTC clock drift when using timeslot

Einar Thorsrud — Fri, 26 Feb 2021 10:09:18 GMT

Hi Mike,

Thanks for the traces.

Regarding the assert at PC=0x24f24 that can happen if the application used too much time in the callback handler as the SoftDevice timing may be thrown off.

Einar

RE: Softdevice Assert at PC=0x15810 (S132 7.2.0) / RTC clock drift when using timeslot

m.wagner — Fri, 26 Feb 2021 09:45:30 GMT

Hi Einar,

Thanks. Just for the sake of completeness, I also attached the Saleae Logic 2 traces I screenshotted further up. The first one is from our application, the second one from the mesh example.

softdevice_assert_logic_traces.zip

Additionally, I'd like to ask what the Softdevice assert at PC=0x24f24 indicates? I assume this is the one that gets raised when the timeslot was ended too late? I very rarely see this one (as opposed to the "regular" one at PC=0x15810) and assume it can be averted by the workaround suggested above, increasing TIMESLOT_END_SAFETY_MARGIN_US.

Looking forward to the results of the team & thanks for your support!

-mike

RE: Softdevice Assert at PC=0x15810 (S132 7.2.0) / RTC clock drift when using timeslot

Einar Thorsrud — Thu, 25 Feb 2021 14:35:06 GMT

Hi Mike,

Thanks for the update. I will forward the information to the team.

RE: Softdevice Assert at PC=0x15810 (S132 7.2.0) / RTC clock drift when using timeslot

m.wagner — Wed, 24 Feb 2021 15:12:44 GMT

Hi Einar,

While trying to reproduce another timeslot-related issue (Softdevice event handler continuously called with NRF_EVT_RADIO_BLOCKED), I found another angle by which I seem to be able to reproduce the same basic issue.

Hung Bui, the support engineer in the linked devzone entry, convinced me to try to reproduce the other issue based on his nRF52 BLE - ESB Timeslot example. Doing this, I managed to reproduce this issue here, wehere the softdevice event handler unexpectedly gets called with NRF_EVT_RADIO_SESSION_IDLE on an nRF52DK (PCA10040).

Steps to reproduce with his example:

Get the linked example to run as per its instructions.
Attach the patch attached to this comment
Run the example
Connect a BLE central to the peripheral (advertising as "Nordic_UART").

This will result in NRF_EVT_RADIO_SESSION_IDLE being invoked within a few milliseconds after establishing the BLE connection.

The patch contains the following changes:

Added and enabled GPIOs for debugging:
- P0.03 - Signals whether device is in timeslot
- P0.04 - High while in radio callback (signal ID is indicated by repeated toggles)
- P0.28 - Set when callback_action set to NRF_RADIO_SIGNAL_CALLBACK_ACTION_EXTEND, cleared on EXTEND_SUCCEEDED or EXTEND_FAILED
- P0.29 - High while serving SIGNAL_TYPE_TIMER0
- P0.30 - High while in softdevice signal handler (signal ID is indicated by repeated toggles)
- P0.31 - High while serving TIMER0 interrupt where timeslot is ended.
- P0.25 - Indicates EXTEND_SUCCEEDED.
Reduced the length_us parameter in the radio request to 3800 in case a timeslot is BLOCKED or CANCELED.

devzone.nordicsemi.com/.../example_2D00_uart_2D00_esb_2D00_timeslot.patch

So, the issue occurs immediately if the length_us of the requested timeslot is reduced from 5000 to 3800. I did not play around with the values any further!

I hope this may help you in getting to the cause of this issue! In the meantime, I'll see if increasing the minimum length requested may resolve it in may application!

Best regards,

-mike

RE: Softdevice Assert at PC=0x15810 (S132 7.2.0) / RTC clock drift when using timeslot

Einar Thorsrud — Wed, 24 Feb 2021 07:57:14 GMT

Hi Mike,

This is a tricky one it seems. I cannot promise anything at this point but I have forwarded this to the team. I will let you know if they make any progress.

Einar

RE: Softdevice Assert at PC=0x15810 (S132 7.2.0) / RTC clock drift when using timeslot

m.wagner — Tue, 23 Feb 2021 16:50:03 GMT

Hi Einar,

I tried to subsequently increase the TIMESLOT_END_SAFETY_MARGIN_US until either not observing any Softdevice Asserts or reaching 1000us. Unfortunately, I did always observe the described softdevice assert at some point. It looks like increasing it reduces the probability of the asserts' occurence, though which is at least something.

Still hoping for a definitive solution!

Thank you & best regards,

-mike

RE: Softdevice Assert at PC=0x15810 (S132 7.2.0) / RTC clock drift when using timeslot

Einar Thorsrud — Mon, 22 Feb 2021 11:31:38 GMT

Hi Mike,

The team has looked more into this but have not got to the bottom of it. They suggest that you can try to they increase TIMESLOT_END_SAFETY_MARGIN_US in steps of 100 us upto 1000 us.

Einar

RE: Softdevice Assert at PC=0x15810 (S132 7.2.0) / RTC clock drift when using timeslot

Einar Thorsrud — Fri, 05 Feb 2021 13:23:57 GMT

Hi Mike,

I am sorry this has take some time. I have talked to the mesh team and they are looking into this. I will let you know when they make some progress or have some interesting observations to share.

Einar

RE: Softdevice Assert at PC=0x15810 (S132 7.2.0) / RTC clock drift when using timeslot

m.wagner — Fri, 05 Feb 2021 12:40:10 GMT

Hi Einar,

It's been 10 days since I heard from you and tought I should ask if there are any new findings or estimates as to when there could be!

The issue is somewhat urgent as it concerns a product which is running in pre-production testing and the frequent resets - all due to the described softdevice asserts - seriously impact the usability of the product.

Thank you for your help & best regards,

-mike

RE: Softdevice Assert at PC=0x15810 (S132 7.2.0) / RTC clock drift when using timeslot

Einar Thorsrud — Tue, 26 Jan 2021 10:29:38 GMT

Hi Mike,

Thank you for that input, that could be relevant. I have forwarded it to the mesh team.

Einar

RE: Softdevice Assert at PC=0x15810 (S132 7.2.0) / RTC clock drift when using timeslot

m.wagner — Mon, 25 Jan 2021 17:12:28 GMT

Hi Einar,

Just a small update: I'm not 100% certain, but it looks like the issue occurs more often on PCA20020 (Thingy:52) than on PCA10040 (nRF52 DK). I did only notice this when testing our application firmware; did not test with the mesh example I provided.

(Could the core issue possibly be related to the accuracy of the 32MHz crystal in use? PCA10040 has a 10ppm crystal, PCA20020 a 40ppm crystal and as I mentioned earlier, our custom module uses a 30ppm crystal. With both "lower accuracy" devices showing said behaviour more frequently.)

Hope this helps in narrowing down the issue.

Regards,

-mike

RE: Softdevice Assert at PC=0x15810 (S132 7.2.0) / RTC clock drift when using timeslot

m.wagner — Fri, 22 Jan 2021 14:39:44 GMT

Okay, thanks for the feedback!

RE: Softdevice Assert at PC=0x15810 (S132 7.2.0) / RTC clock drift when using timeslot

Einar Thorsrud — Fri, 22 Jan 2021 14:38:20 GMT

Hi Mike,

The mesh team has started to look at it, but they have not made any interesting observations yet nor found a solution. I will let you know when they make progress of if they have some time estimates.

Einar

RE: Softdevice Assert at PC=0x15810 (S132 7.2.0) / RTC clock drift when using timeslot

m.wagner — Fri, 22 Jan 2021 08:06:13 GMT

Hi Einar,

Do you have any news concerning this issue?

Regards

-mike

RE: Softdevice Assert at PC=0x15810 (S132 7.2.0) / RTC clock drift when using timeslot

Einar Thorsrud — Fri, 15 Jan 2021 19:51:43 GMT

Hi,

Thank you for all the details. I have not made any progress on this today, but I will ask the mesh team to look at it and let you know what they find.

RE: Softdevice Assert at PC=0x15810 (S132 7.2.0) / RTC clock drift when using timeslot

m.wagner — Thu, 14 Jan 2021 16:13:03 GMT

Hi Einar,

In the meantime I was able to reproduce this issue with nRF SDK for Mesh 4.2.0 and a slightly modified beaconing example.

I attached a patch file which will apply the modifications I made.

devzone.nordicsemi.com/.../mesh_5F00_sd_5F00_assert_5F00_proof.patch

Summary of my changes:

Adapted to my target hardware (disabled LEDs, buttons, use LF_SRC = RC)
Disabled beacon advertising and provisionee
Enabled and slightly modified the debug pins from core/debug_pins.h

The debug pins used in this configuration are:

P0.00 - (LF XTAL pin) - Signals whether device is in timeslot
P0.01 - (LF XTAL pin) - High while in timeslot signal handler
P0.06 - High while in BEARER_ACTION_TIMER_IRQHandler()
P0.08 - High while in app_error_fault_handler()
P0.09 - (NFCT pin) - High while TIMER0 IRQ is being served
P0.10 - (NFCT pin) - High while in timeslot.c's softdevice event handler
P0.30 - Toggles on TIMER0->EVENTS_COMPARE[0]
P0.31 - Set to high by RADIO->EVENTS_READY, to low by RADIO->EVENTS_DISABLED.

As you can see in the *.patch file attached, I ran the beaconing_nrf52832_xxAA_s132_7_0_1.emProject example in Segger Embedded Studio.

The device I tested with is a custommade/proprietary RF-Module with an nRF52832-QFAA, no LF crystal and a Murata XRCMD32M000FXP53R0 30ppm HF-XTAL.

Below you can see the corresponding trace I made with the logic analyser. As in the previous traces, around the point where the TIMER0->EVENTS_COMPARE[0] should fire, the softdevice signal handler is invoked with event NRF_EVT_RADIO_SESSION_IDLE. The next time the radio interrupt is triggered or something alike (the mesh's scanner module is still running at this point), the example crashes with Softdevice assert: 88044.

Concerning the frequency of this occurence: On this particular device I tested with, it seems to usually occur within the first 10 minutes of run time or so. On nRF52 DK devices (also configured to use LF_SRC = RC!), it seems to happen less frequently. Maybe once every couple of hours or so... So far, I have not yet been able to properly trace this on an nRF52 DK yet.

It would be great, if you could investigate this issue in more detail, as the smooth functioning of this timeslot code is essential to our application.

RE: Softdevice Assert at PC=0x15810 (S132 7.2.0) / RTC clock drift when using timeslot

m.wagner — Wed, 13 Jan 2021 16:54:26 GMT

Hi Einar,

No, it's definitely running on the HFXO.

In the meantime, I finally managed to look into it with a logic analyzer and could make an additional important finding: The issue occurs after the SoftDevice Event IRQ Handler has been invoked with NRF_EVT_RADIO_SESSION_IDLE.

This is a trace of the event. You see that around the time the TIMER0 interrupt should fire via the signal handler, instead the SoftDevice Event IRQ is invoked with NRF_EVT_RADIO_SESSION_IDLE.

Since this is unexpected and regular timeslot cleanup is omitted, my TIMER4 is still running on priority 0 and firing shortly after, triggering a RADIO IRQ via software.

The zoom in on the final part of the trace shows that the softdevice asserts immediately after said timer interrupt (confirming my previous assumptions):

To see what happens if the Interrupt causing the assertion is not running, I observe that... nothing happens. Nothing pertaining to timeslot activity is going on (see following trace):

Now I do not expect this event while a timeslot is ongoing - or better yet: the mesh team does not expect this event while a timeslot is ongoing. As I mentioned, the timeslot session is handled by a barely modified timeslot.c from nRF SDK for Mesh. The event handling:

case NRF_EVT_RADIO_SESSION_IDLE:
    /* As the SD events aren't handled atomically, we could potentially run into this scenario:
     * - stop the timeslot -> SESSION_IDLE event pending, session_state is OPEN
     * - start the timeslot again from the same context -> session_state is RUNNING
     * - The SESSION_IDLE event is handled.
     * Closing the radio session now leads to unexpected behavior, as the user restarted
     * the timeslot already, expecting it to remain active.
     */
    if (m_current_timeslot.session_state == TS_SESSION_STATE_OPEN)
    {
        ASSERT_NRF_ERROR_CODE(sd_radio_session_close(), TS_CORE_ERR_SESSION_CLOSE);
    }
    break;

The variable m_current_timeslot.session_state is set to TS_SESSION_STATE_RUNNING at this point, though - so nothing happens.

A search of the DevZone also lead me to a user that seemingly had a similar issue with the actual nRF SDK for Mesh running:

SOFTDEVICE: ASSERTION FAILED while running mesh.

Unfortunately this was not followed up.

To me it looks like some sort of race condition occuring in the softdevice. I'm not entirely sure what to make of the NRF_EVT_RADIO_SESSION_IDLE event in any case.