Upgrading through DFU Target results in MPSL Assertion (112, 2142)

[Peter Maxwell Warasila]

Using ncs 2.1.2 on nrf52840. Softdevice controller, bluetooth mesh and bluetooth gatt coexisting application.

I have been having an issue attempting to perform DFU using the dfu_target interface on our nrf52840. For whatever reason, an MPSL assertion is being raised during this process. It is more consistently raised the larger I make the buffer passed through dfu_target_mcuboot_set_buf(). I've enabled the maximum (debug) logging from all the MPSL sources in KConfig, but I have not seen any additional logs. CONFIG_SOC_FLASH_NRF_RADIO_SYNC_MPSL is enabled, so my understanding is that should deal with any synchronization required. A log is shown below.

[00:01:26.549,102] <dbg> STREAM_FLASH: stream_flash_erase_page: Erasing page at offset 0x0007e000
[00:01:27.718,017] <dbg> STREAM_FLASH: stream_flash_erase_page: Erasing page at offset 0x0007f000
[00:01:28.861,694] <dbg> STREAM_FLASH: stream_flash_erase_page: Erasing page at offset 0x00080000
[00:01:29.995,269] <dbg> STREAM_FLASH: stream_flash_erase_page: Erasing page at offset 0x00081000
[00:01:31.164,489] <dbg> STREAM_FLASH: stream_flash_erase_page: Erasing page at offset 0x00082000
[00:01:32.339,294] <dbg> STREAM_FLASH: stream_flash_erase_page: Erasing page at offset 0x00083000
[00:01:33.468,994] <dbg> STREAM_FLASH: stream_flash_erase_page: Erasing page at offset 0x00084000
[00:01:34.672,882] <dbg> STREAM_FLASH: stream_flash_erase_page: Erasing page at offset 0x00085000
[00:01:35.801,696] <dbg> STREAM_FLASH: stream_flash_erase_page: Erasing page at offset 0x00086000
[00:01:36.970,489] <dbg> STREAM_FLASH: stream_flash_erase_page: Erasing page at offset 0x00087000
[00:01:38.137,115] <dbg> STREAM_FLASH: stream_flash_erase_page: Erasing page at offset 0x00088000
[00:01:39.272,125] <dbg> STREAM_FLASH: stream_flash_erase_page: Erasing page at offset 0x00089000
[00:01:40.406,616] <dbg> STREAM_FLASH: stream_flash_erase_page: Erasing page at offset 0x0008a000
[00:01:41.575,500] <dbg> STREAM_FLASH: stream_flash_erase_page: Erasing page at offset 0x0008b000
[00:01:42.738,739] <dbg> STREAM_FLASH: stream_flash_erase_page: Erasing page at offset 0x0008c000
[00:01:43.869,415] <dbg> STREAM_FLASH: stream_flash_erase_page: Erasing page at offset 0x0008d000
[00:01:45.040,130] <dbg> STREAM_FLASH: stream_flash_erase_page: Erasing page at offset 0x0008e000
[00:01:46.189,117] <dbg> STREAM_FLASH: stream_flash_erase_page: Erasing page at offset 0x0008f000
[00:01:47.317,932] <dbg> STREAM_FLASH: stream_flash_erase_page: Erasing page at offset 0x00090000
[00:01:48.473,114] <dbg> STREAM_FLASH: stream_flash_erase_page: Erasing page at offset 0x00091000
[00:01:49.641,021] <dbg> STREAM_FLASH: stream_flash_erase_page: Erasing page at offset 0x00092000
[00:01:50.785,522] <dbg> STREAM_FLASH: stream_flash_erase_page: Erasing page at offset 0x00093000
[00:01:51.945,953] <dbg> STREAM_FLASH: stream_flash_erase_page: Erasing page at offset 0x00094000
[00:01:53.111,816] <dbg> STREAM_FLASH: stream_flash_erase_page: Erasing page at offset 0x00095000
[00:01:54.249,938] <dbg> STREAM_FLASH: stream_flash_erase_page: Erasing page at offset 0x00096000
[00:01:55.416,595] <dbg> STREAM_FLASH: stream_flash_erase_page: Erasing page at offset 0x00097000
[00:01:56.581,695] <dbg> STREAM_FLASH: stream_flash_erase_page: Erasing page at offset 0x00098000
[00:01:57.776,275] <dbg> STREAM_FLASH: stream_flash_erase_page: Erasing page at offset 0x00099000
[00:01:58.952,484] <dbg> STREAM_FLASH: stream_flash_erase_page: Erasing page at offset 0x0009a000
[00:02:00.088,195] <dbg> STREAM_FLASH: stream_flash_erase_page: Erasing page at offset 0x0009b000
[00:02:01.219,482] <dbg> STREAM_FLASH: stream_flash_erase_page: Erasing page at offset 0x0009c000
[00:02:02.381,866] <dbg> STREAM_FLASH: stream_flash_erase_page: Erasing page at offset 0x0009d000
[00:02:03.550,964] <dbg> STREAM_FLASH: stream_flash_erase_page: Erasing page at offset 0x0009e000
[00:02:04.718,780] <dbg> STREAM_FLASH: stream_flash_erase_page: Erasing page at offset 0x0009f000
[00:02:05.891,052] <dbg> STREAM_FLASH: stream_flash_erase_page: Erasing page at offset 0x000a0000
[00:02:07.029,693] <dbg> STREAM_FLASH: stream_flash_erase_page: Erasing page at offset 0x000a1000
[00:02:08.200,744] <dbg> STREAM_FLASH: stream_flash_erase_page: Erasing page at offset 0x000a2000
[00:02:09.368,774] <dbg> STREAM_FLASH: stream_flash_erase_page: Erasing page at offset 0x000a3000
[00:02:10.502,593] <dbg> STREAM_FLASH: stream_flash_erase_page: Erasing page at offset 0x000a4000
[00:02:11.673,065] <dbg> STREAM_FLASH: stream_flash_erase_page: Erasing page at offset 0x000a5000
[00:02:12.861,755] <dbg> STREAM_FLASH: stream_flash_erase_page: Erasing page at offset 0x000a6000
[00:02:13.996,246] <dbg> STREAM_FLASH: stream_flash_erase_page: Erasing page at offset 0x000a7000
[00:02:15.163,269] <dbg> STREAM_FLASH: stream_flash_erase_page: Erasing page at offset 0x000a8000
[00:02:16.329,406] <dbg> STREAM_FLASH: stream_flash_erase_page: Erasing page at offset 0x000a9000
[00:02:17.466,461] <dbg> STREAM_FLASH: stream_flash_erase_page: Erasing page at offset 0x000aa000
[00:02:18.637,481] <dbg> STREAM_FLASH: stream_flash_erase_page: Erasing page at offset 0x000ab000
[00:02:19.793,518] <dbg> STREAM_FLASH: stream_flash_erase_page: Erasing page at offset 0x000ac000
[00:02:20.917,968] <dbg> STREAM_FLASH: stream_flash_erase_page: Erasing page at offset 0x000ad000
[00:02:22.090,728] <dbg> STREAM_FLASH: stream_flash_erase_page: Erasing page at offset 0x000ae000
[00:02:23.252,349] <dbg> STREAM_FLASH: stream_flash_erase_page: Erasing page at offset 0x000af000
[00:02:24.394,104] <dbg> STREAM_FLASH: stream_flash_erase_page: Erasing page at offset 0x000b0000
[00:02:25.563,507] <dbg> STREAM_FLASH: stream_flash_erase_page: Erasing page at offset 0x000b1000
[00:02:26.725,860] <dbg> STREAM_FLASH: stream_flash_erase_page: Erasing page at offset 0x000b2000
[00:02:27.947,998] <dbg> STREAM_FLASH: stream_flash_erase_page: Erasing page at offset 0x000b3000
[00:02:29.115,051] <dbg> STREAM_FLASH: stream_flash_erase_page: Erasing page at offset 0x000b4000
[00:02:30.250,396] <dbg> STREAM_FLASH: stream_flash_erase_page: Erasing page at offset 0x000b5000
[00:02:31.415,008] <dbg> STREAM_FLASH: stream_flash_erase_page: Erasing page at offset 0x000b6000
[00:02:32.586,853] <dbg> STREAM_FLASH: stream_flash_erase_page: Erasing page at offset 0x000b7000
[00:02:33.717,224] <dbg> STREAM_FLASH: stream_flash_erase_page: Erasing page at offset 0x000b8000
[00:02:34.886,779] <dbg> STREAM_FLASH: stream_flash_erase_page: Erasing page at offset 0x000b9000
[00:02:36.052,154] <dbg> STREAM_FLASH: stream_flash_erase_page: Erasing page at offset 0x000ba000
[00:02:37.188,446] <dbg> STREAM_FLASH: stream_flash_erase_page: Erasing page at offset 0x000bb000
[00:02:38.356,781] <dbg> STREAM_FLASH: stream_flash_erase_page: Erasing page at offset 0x000bc000
[00:02:39.524,169] <dbg> STREAM_FLASH: stream_flash_erase_page: Erasing page at offset 0x000bd000
[00:02:40.659,301] <dbg> STREAM_FLASH: stream_flash_erase_page: Erasing page at offset 0x000be000
[00:02:41.822,753] <dbg> STREAM_FLASH: stream_flash_erase_page: Erasing page at offset 0x000bf000
[00:02:43.003,967] <dbg> STREAM_FLASH: stream_flash_erase_page: Erasing page at offset 0x000c0000
[00:02:44.174,316] <dbg> STREAM_FLASH: stream_flash_erase_page: Erasing page at offset 0x000c1000
[00:02:45.343,017] <dbg> STREAM_FLASH: stream_flash_erase_page: Erasing page at offset 0x000c2000
[00:02:46.472,106] <dbg> STREAM_FLASH: stream_flash_erase_page: Erasing page at offset 0x000c3000
[00:02:47.609,558] <dbg> STREAM_FLASH: stream_flash_erase_page: Erasing page at offset 0x000c4000
[00:02:48.774,597] <dbg> STREAM_FLASH: stream_flash_erase_page: Erasing page at offset 0x000c5000
[00:02:49.943,969] <dbg> STREAM_FLASH: stream_flash_erase_page: Erasing page at offset 0x000c6000
[00:02:51.076,110] <dbg> STREAM_FLASH: stream_flash_erase_page: Erasing page at offset 0x000c7000
[00:02:52.244,079] <dbg> STREAM_FLASH: stream_flash_erase_page: Erasing page at offset 0x000c8000
[00:02:53.388,488] <dbg> STREAM_FLASH: stream_flash_erase_page: Erasing page at offset 0x000c9000
[00:02:54.521,545] <dbg> STREAM_FLASH: stream_flash_erase_page: Erasing page at offset 0x000ca000
[00:02:55.689,453] <dbg> STREAM_FLASH: stream_flash_erase_page: Erasing page at offset 0x000cb000
[00:02:56.856,262] <dbg> STREAM_FLASH: stream_flash_erase_page: Erasing page at offset 0x000cc000
[00:02:57.995,483] <dbg> STREAM_FLASH: stream_flash_erase_page: Erasing page at offset 0x000cd000
[00:02:59.164,947] <dbg> STREAM_FLASH: stream_flash_erase_page: Erasing page at offset 0x000ce000
[00:03:00.331,756] <dbg> STREAM_FLASH: stream_flash_erase_page: Erasing page at offset 0x000cf000
[00:03:01.470,123] <dbg> STREAM_FLASH: stream_flash_erase_page: Erasing page at offset 0x000d0000
[00:03:02.640,655] <dbg> STREAM_FLASH: stream_flash_erase_page: Erasing page at offset 0x000d1000
[00:03:03.795,196] <dbg> STREAM_FLASH: stream_flash_erase_page: Erasing page at offset 0x000d2000
[00:03:04.929,748] <dbg> STREAM_FLASH: stream_flash_erase_page: Erasing page at offset 0x000d3000
[00:03:06.175,964] <dbg> STREAM_FLASH: stream_flash_erase_page: Erasing page at offset 0x000d4000
[00:03:07.311,157] <dbg> STREAM_FLASH: stream_flash_erase_page: Erasing page at offset 0x000d5000
[00:03:08.481,323] <dbg> STREAM_FLASH: stream_flash_erase_page: Erasing page at offset 0x000d6000
[00:03:09.646,759] <dbg> STREAM_FLASH: stream_flash_erase_page: Erasing page at offset 0x000d7000
[00:03:10.788,269] <dbg> STREAM_FLASH: stream_flash_erase_page: Erasing page at offset 0x000d8000
[00:03:10.908,721] <err> mpsl_init: MPSL ASSERT: 112, 2142
ASSERTION FAIL [z_spin_lock_valid(l)] @ WEST_TOPDIR/zephyr/include/zephyr/spinlock.h:142
        Recursive spinlock 0x2000a7f0
[00:03:10.908,813] <err> os: ***** HARD FAULT *****
[00:03:10.908,813] <err> os:   Fault escalation (see below)
[00:03:10.908,843] <err> os: ARCH_EXCEPT with reason 4

[00:03:10.908,843] <err> os: r0/a1:  0x00000004  r1/a2:  0x0000008e  r2/a3:  0x00000003
[00:03:10.908,874] <err> os: r3/a4:  0x0000008e r12/ip:  0x00000000 r14/lr:  0x0005d773
[00:03:10.908,874] <err> os:  xpsr:  0x61000018
[00:03:10.908,905] <err> os: Faulting instruction address (r15/pc): 0x000637dc
[00:03:10.908,935] <err> os: >>> ZEPHYR FATAL ERROR 4: Kernel panic on CPU 0
[00:03:10.908,935] <err> os: Fault during interrupt handling

[00:03:10.908,966] <err> os: Current thread: 0x20003c88 (MPSL Work)
[00:03:11.283,386] <err> fatal_error: Resetting system

Since the MPSL is completely opaque, I really have no idea what's going on here. My best guess is that some time critical behavior is being violated, but I am not sure what. I'm also not sure what to do about it, since the flash erase and write times are essentially out of my control.

[Peter Maxwell Warasila]

Hieu,

Of course. We started with the Mesh and Peripheral Coexistence sample. We replaced the light button service with our own service (which is fairly similar to the NUS), replaced the mesh model with our own model, added a serial connection to the host processor on our system, and enabled CONFIG_DFU_TARGET_MCUBOOT=y. We then use serial commands to write packets of the update image to flash. These are handled by an application thread which is reading from a ring buffer populated using the async serial API.

I want to add that we seem to be having issues with delayable work/timeouts in the async serial uarte driver and the bluetooth mesh extended advertiser. Essentially, it seems like the timeout takes an entire RTC counter wrap around to occur. I'm going to trial using the SysTick as a source to see if the behavior is within the RTC timer driver. It may be related, but I don't see the MPSL assertion when we are not writing to flash.

[Hieu]

Peter,

I plan to replicate your issue here and try some debugging. It seems I will make do with the Mesh and Peripheral Coexistence sample as is then.

Could you please give some more details on the "added a serial connection" part? Which peripheral (1/2/3) did you use, and what Kconfig and DTS config did you set it up with?

If you feel like it can be quite complicated to replicate your issue, please let me know if you want to send us your project.

[Peter Maxwell Warasila]

So I've had success today after switching from the nrf RTC timer to the systick for the system tick source. Is there a known issue with the RTC timer in ncs 2.1.2?

[Hieu]

Sorry but I am not clear about/familiar with what you changed here. Could you please give some further details?

[Peter Maxwell Warasila]

Essentially, I added the following to our prj.conf file. We had no issues with the debugger connected after making this change. However I'm now running into an issue where the kernel tick count is not being updated until the RTT is connected. Very strange. Either way, I would have expected the RTC timer to work correctly out of the box. However, there seems to be an issue where timeouts can be evaluated on the 2^24 - 1 tick after the current one, for whatever reason. It appears something like that must have been interfering with the MPSL as well, it would seem.

CONFIG_NRF_RTC_TIMER=n
CONFIG_CORTEX_M_SYSTICK=y
CONFIG_SYS_CLOCK_TICKS_PER_SEC=10000
CONFIG_SYS_CLOCK_HW_CYCLES_PER_SEC=64000000

[Hieu]

By "kernel tick count is not being updated," do you mean k_uptime_get() and k_uptime_ticks() return 0?

[Peter Maxwell Warasila]

Not exactly. Timers are not executed in real time. k_uptime_get() reflects that the ticks counted by the system are less and are proportional to the timer execution. For example, a 5 second timer will take ~40 seconds to execute, and k_uptime_get() between those two times reflects that the kernel ticks have only advanced 5 seconds.

Once RTT is connected to read log messages, the ticks advance normally. This appears to continue after disconnection of the RTT as well.

This is all specifically while using the SysTick as the timer driver.

[Hieu]

Hi Peter,

The behavior you described are very strange. I will need to forward this to our R&D team. I will do that on Monday for some better attention.

I am having some difficulties reproducing the initial issue on my end. I will see if I can give it some more tries this weekend, as that might help R&D help faster.

Hieu

[Peter Maxwell Warasila]

Hieu,

I have been doing some testing with the Zephyr community on this issue and I think we have a good idea of what was originally going on when using the nrf_rtc_timer.c driver for the system timer. Using tests/kernel/timer/timer_behavior with ncs 2.2.0, the timer train test fails on our hardware, the nrf52840dk_nrf52840 dev kit, and the ubx_bmd345eval_nrf52840 evaluation kit. In our application, whenever we saw what seemed to be a missed timeout, it looked like it was handled nearly one full counter rollover late (~512 seconds). This pointed to some missed rollover behavior.

I ran test/kernel/timer/timer_behavior using latest zephyr instead of the version packaged with ncs 2.2.0, and the timer train test passed. I then ran a diff between the latest zephyr and ncs 2.2.0 versions of nrf_rtc_timer.c and discovered that there are differences, and they seem to deal with exactly this sort of issue in terms of setting an alarm when the requested cc_val is in the past or the next tick. See diff below.

diff --git a/drivers/timer/nrf_rtc_timer.c b/C:/Users/Peter/ncs/v2.2.0/zephyr/drivers/timer/nrf_rtc_timer.c
index b3b2cadeae..a7587f3366 100644
--- a/drivers/timer/nrf_rtc_timer.c
+++ b/C:/Users/Peter/ncs/v2.2.0/zephyr/drivers/timer/nrf_rtc_timer.c
@@ -42,7 +42,6 @@ BUILD_ASSERT(CHAN_COUNT <= RTC_CH_COUNT, "Not enough compare channels");
 static volatile uint32_t overflow_cnt;
 static volatile uint64_t anchor;
 static uint64_t last_count;
-static bool sys_busy;
 
 struct z_nrf_rtc_timer_chan_data {
 	z_nrf_rtc_timer_compare_handler_t callback;
@@ -234,7 +233,6 @@ static uint32_t set_absolute_alarm(int32_t chan, uint32_t abs_val)
 	uint32_t now2;
 	uint32_t cc_val = abs_val & COUNTER_MAX;
 	uint32_t prev_cc = get_comparator(chan);
-	uint32_t tick_inc = 2;
 
 	do {
 		now = counter();
@@ -253,16 +251,12 @@ static uint32_t set_absolute_alarm(int32_t chan, uint32_t abs_val)
 			k_busy_wait(19);
 		}
 
-		/* RTC may not generate event if CC is set for 1 tick from now.
-		 * Because of that if requested cc_val is in the past or next tick,
-		 * set CC to further in future. Start with 2 ticks from now but
-		 * if that fails go even futher. It may fail if operation got
-		 * interrupted and RTC counter progressed or if optimization is
-		 * turned off.
+		/* If requested cc_val is in the past or next tick, set to 2
+		 * ticks from now. RTC may not generate event if CC is set for
+		 * 1 tick from now.
 		 */
 		if (counter_sub(cc_val, now + 2) > COUNTER_HALF_SPAN) {
-			cc_val = now + tick_inc;
-			tick_inc++;
+			cc_val = now + 2;
 		}
 
 		event_clear(chan);
@@ -548,41 +542,6 @@ void z_nrf_rtc_timer_chan_free(int32_t chan)
 }
 
 
-int z_nrf_rtc_timer_trigger_overflow(void)
-{
-	uint32_t mcu_critical_state;
-	int err = 0;
-
-	if (!IS_ENABLED(CONFIG_NRF_RTC_TIMER_TRIGGER_OVERFLOW) ||
-	    (CONFIG_NRF_RTC_TIMER_USER_CHAN_COUNT > 0)) {
-		return -ENOTSUP;
-	}
-
-	mcu_critical_state = full_int_lock();
-	if (sys_busy) {
-		err = -EBUSY;
-		goto bail;
-	}
-
-	if (counter() >= (COUNTER_SPAN - 100)) {
-		err = -EAGAIN;
-		goto bail;
-	}
-
-	nrf_rtc_task_trigger(RTC, NRF_RTC_TASK_TRIGGER_OVERFLOW);
-	k_busy_wait(80);
-
-	uint64_t now = z_nrf_rtc_timer_read();
-
-	if (err == 0) {
-		sys_clock_timeout_handler(0, now, NULL);
-	}
-bail:
-	full_int_unlock(mcu_critical_state);
-
-	return err;
-}
-
 void sys_clock_set_timeout(int32_t ticks, bool idle)
 {
 	ARG_UNUSED(idle);
@@ -594,10 +553,6 @@ void sys_clock_set_timeout(int32_t ticks, bool idle)
 
 	ticks = (ticks == K_TICKS_FOREVER) ? MAX_TICKS : ticks;
 	ticks = CLAMP(ticks - 1, 0, (int32_t)MAX_TICKS);
-	/* If timeout is set to max we assume that system is idle and timeout
-	 * is set to forever.
-	 */
-	sys_busy = (ticks < (MAX_TICKS - 1));
 
 	uint32_t unannounced = z_nrf_rtc_timer_read() - last_count;
 
@@ -677,7 +632,8 @@ static int sys_clock_driver_init(const struct device *dev)
 	}
 
 	uint32_t initial_timeout = IS_ENABLED(CONFIG_TICKLESS_KERNEL) ?
-		MAX_TICKS : (counter() + CYC_PER_TICK);
+		(COUNTER_HALF_SPAN - 1) :
+		(counter() + CYC_PER_TICK);
 
 	compare_set(0, initial_timeout, sys_clock_timeout_handler, NULL);
 

I'm not positive that this is the sole reason for this behavior, but it seems to be a likely candidate. There may be other differences in how ncs 2.2.0 deals with the rtc timer which I am unaware of.

As far as the issues with the SysTick before connecting the RTT, I'm still not sure what's going on. However, running the code on our dev kits did not show the same behavior, so we must have some sort of hardware issue causing that. Maybe there's a debug pin floating or something? I'm not sure what would cause that symptom.

[Peter Maxwell Warasila]

...back to square one it looks like. The test requirements for the timer tick train were reduced, specifically with platforms like nRF in mind. See diff below:

diff --git a/tests/kernel/timer/timer_behavior/src/tick_timer_train.c b/C:/Users/Peter/ncs/v2.2.0/zephyr/tests/kernel/timer/timer_behavior/src/tick_timer_train.c
index 27a90e7a86..368ff74a8a 100644
--- a/tests/kernel/timer/timer_behavior/src/tick_timer_train.c
+++ b/C:/Users/Peter/ncs/v2.2.0/zephyr/tests/kernel/timer/timer_behavior/src/tick_timer_train.c
@@ -132,13 +132,8 @@ ZTEST(timer_tick_train, test_one_tick_timer_train)
 		k_timer_stop(&timers[i].tm);
 	}
 
-	const uint32_t maximum_busy_loops = TEST_SECONDS * 4;
-	/* On some platforms, where the tick period is short, like on nRF
-	 * platforms where it is ~30 us, execution of the timer handlers
-	 * can take significant part of the CPU time, so accept if at least
-	 * one-third of possible busy loop iterations is actually performed.
-	 */
-	const uint32_t acceptable_busy_loops = maximum_busy_loops / 3;
+	const uint32_t expected_busy_loops = TEST_SECONDS*4;
+	const uint32_t acceptable_busy_loops = expected_busy_loops - expected_busy_loops/10;
 
 	zassert_true(busy_loops > acceptable_busy_loops,
 		     "Expected thread to run while 1 tick timers are firing");

That said it looks like the issue is at least partially mitigated in our application with the code pulled in from Zephyr, though not perfectly. I still get occasional timer issues (which are visible through pauses in the serial heartbeat from the radio).

The only workaround I've found so far is to significantly reduce CONFIG_SYS_CLOCK_TICKS_PER_SEC to a value of 2048 or lower.

[Hieu]

Hi Peter,

It seems you are making a lot more progress than I have.

I am a little confused about your newest message. Are you saying that the test with latest main only passes because it was "easier" for nRF chips?

I think the comment in the diff would explain why the test was made more passable, do you not agree?

If I understand correctly, you are saying that the SysTick problem still happens with the latest main branch, however, not as often. Is this right?

I will forward your findings to our R&D team and get further support from them. If there are changes between latest main branch and 2.2.0, it likely means that the topic is under development and has someone actively working on it, so that's a good sign.

---

On the other hand, it seems that you are now very invested into investigating the slowed SysTick. Are you intended on getting SysTick to work reliably when "CONFIG_NRF_RTC_TIMER=n" and considering that a workaround for the original failed DFU issue?

Hieu 

[Peter Maxwell Warasila]

Yes, I am saying it only now passes the timer_behavior test because the test's bar was lowered from 90% of expected busy loops to 33% of expected busy loops. This was done here, by a Nordic employee, clearly with limitations of the current tick defaults in mind. Accepting a 66% degradation in CPU processing time due to a high tick rate does not seem like a reasonable compromise to me. It indicates to me that the tick rate is too high for the capabilities of the core to handle the worst case tick IRQ demand. It's also worth noting that this test has also lowered the bar for all platform configurations, regardless of it's expected time between system ticks. Realistically, it looks like the nrf tests only degrade slightly higher than 10%. I would have preferred to see either 1) a dynamic test which considers the core clock relative to the timer interval (or perhaps uses a KConfig for the test application) or 2) an nRF/ncs specific change to this test. But I digress.

What is relevant is that, for one reason or another, timeouts are occasionally not being evaluated correctly in our application. Below is a screen capture from our Saleae logic analyzer trace of the serial bus, for example. Ignoring spurious probe noise which shows up on both TX/RX lines, the serial heartbeat from the radio (which is supposed to be sent every 5 seconds) is sometimes submitted to the work queue after around 634 seconds! What's interesting is that the heartbeat is sent with the current uptime in milliseconds as provided by k_uptime_ms(), and it nearly perfectly matches the time interval. I've also seen this behavior with the Bluetooth mesh extended advertiser delayable work, where it was evaluated almost exactly 512 seconds later than it should have been. This is what initially led me to be suspicious of the nRF RTC timer driver, as 512 seconds is exactly one rollover of the 24-bit counter. See the attached file below; I recommend searching for "Advertising stopped after" to get an idea where this issue seems to occur.

out_of_net_buf_ext_log_+ts+rec.txt

I have also run our application code on the ubx_bmd345eval_nrf52840 evaluation kit with a python script capturing the heartbeat intervals. With the default 32768 Hz tick rate, I still see this issue, albeit less frequently. That is captured in a JSON file attached below. The first number of each sample is the PC's measured time interval, and the second is the difference between the reported up times in each heartbeat. I am currently running a test with at 2048 Hz tick rate to see if the problem is indeed eliminated and will attach the results here when they are ready.

Intervals using 32768 Hz on ubx_bmd345eval_nrf52840:

[
    [
        5.00000786781311,
        5000
    ],
    [
        4.999892473220825,
        5000
    ],
    [
        5.0002923011779785,
        5000
    ],
    [
        4.998442649841309,
        5000
    ],
    [
        5.000606536865234,
        5000
    ],
    [
        4.999890089035034,
        5000
    ],
    [
        4.9995996952056885,
        5000
    ],
    [
        4.999560117721558,
        5000
    ],
    [
        5.000163316726685,
        5000
    ],
    [
        4.999593257904053,
        5000
    ],
    [
        5.000501871109009,
        5000
    ],
    [
        4.9981279373168945,
        5000
    ],
    [
        5.000679969787598,
        5000
    ],
    [
        4.999467849731445,
        5000
    ],
    [
        5.000025033950806,
        5000
    ],
    [
        4.999408721923828,
        5000
    ],
    [
        5.000142574310303,
        5000
    ],
    [
        4.999686002731323,
        5000
    ],
    [
        5.000153064727783,
        5000
    ],
    [
        4.999997138977051,
        5000
    ],
    [
        4.999417543411255,
        5000
    ],
    [
        4.999722242355347,
        5000
    ],
    [
        4.999395370483398,
        5000
    ],
    [
        5.000059366226196,
        5000
    ],
    [
        4.9989330768585205,
        5000
    ],
    [
        5.000201225280762,
        5000
    ],
    [
        5.000433683395386,
        5000
    ],
    [
        5.0001912117004395,
        5000
    ],
    [
        4.999042987823486,
        5000
    ],
    [
        4.999509334564209,
        5000
    ],
    [
        4.9999706745147705,
        5000
    ],
    [
        5.0001068115234375,
        5000
    ],
    [
        4.999145984649658,
        5000
    ],
    [
        5.0002281665802,
        5000
    ],
    [
        4.999783992767334,
        5000
    ],
    [
        5.000519514083862,
        5000
    ],
    [
        4.998447895050049,
        5000
    ],
    [
        5.000113248825073,
        5000
    ],
    [
        4.999787092208862,
        5000
    ],
    [
        5.000275611877441,
        5000
    ],
    [
        4.999713659286499,
        5000
    ],
    [
        4.999034881591797,
        5000
    ],
    [
        4.9994800090789795,
        5000
    ],
    [
        5.000818490982056,
        5000
    ],
    [
        4.999665021896362,
        5000
    ],
    [
        4.998891353607178,
        5000
    ],
    [
        5.0002124309539795,
        5000
    ],
    [
        4.999501705169678,
        5000
    ],
    [
        5.000338554382324,
        5000
    ],
    [
        4.999272584915161,
        5000
    ],
    [
        5.000291585922241,
        5000
    ],
    [
        5.000064134597778,
        5000
    ],
    [
        5.000003099441528,
        5000
    ],
    [
        4.999077320098877,
        5000
    ],
    [
        5.000140190124512,
        5000
    ],
    [
        4.999295711517334,
        5000
    ],
    [
        5.000777721405029,
        5000
    ],
    [
        4.998395681381226,
        5000
    ],
    [
        5.000201225280762,
        5000
    ],
    [
        4.999592065811157,
        5000
    ],
    [
        5.000410079956055,
        5000
    ],
    [
        5.0002217292785645,
        5000
    ],
    [
        4.9993736743927,
        5000
    ],
    [
        4.999011993408203,
        5000
    ],
    [
        5.0005943775177,
        5000
    ],
    [
        4.9991583824157715,
        5000
    ],
    [
        4.999165773391724,
        5000
    ],
    [
        5.000348806381226,
        5000
    ],
    [
        4.999607801437378,
        5000
    ],
    [
        5.0003838539123535,
        5000
    ],
    [
        4.998901844024658,
        5000
    ],
    [
        5.000285387039185,
        5000
    ],
    [
        4.9996113777160645,
        5000
    ],
    [
        5.00066351890564,
        5000
    ],
    [
        4.999006271362305,
        5000
    ],
    [
        4.999561309814453,
        5000
    ],
    [
        4.9999003410339355,
        5000
    ],
    [
        5.000120401382446,
        5000
    ],
    [
        4.999594688415527,
        5000
    ],
    [
        4.998834133148193,
        5000
    ],
    [
        5.000446319580078,
        5000
    ],
    [
        4.999768257141113,
        5000
    ],
    [
        5.000524997711182,
        5000
    ],
    [
        4.999152421951294,
        5000
    ],
    [
        4.9993531703948975,
        5000
    ],
    [
        4.999927997589111,
        5000
    ],
    [
        5.000149488449097,
        5000
    ],
    [
        4.999437093734741,
        5000
    ],
    [
        5.000006198883057,
        5000
    ],
    [
        4.999869108200073,
        5000
    ],
    [
        5.000195264816284,
        5000
    ],
    [
        4.998447895050049,
        5000
    ],
    [
        5.000227928161621,
        5000
    ],
    [
        4.999801874160767,
        5000
    ],
    [
        5.000373840332031,
        5000
    ],
    [
        4.998406410217285,
        5000
    ],
    [
        5.000024080276489,
        5000
    ],
    [
        5.00041937828064,
        5000
    ],
    [
        4.999928951263428,
        5000
    ],
    [
        4.9995269775390625,
        5000
    ],
    [
        4.999550819396973,
        5000
    ],
    [
        4.999730348587036,
        5000
    ],
    [
        5.000584363937378,
        5000
    ],
    [
        5.000080347061157,
        5000
    ],
    [
        4.998950004577637,
        5000
    ],
    [
        4.999688625335693,
        5000
    ],
    [
        5.000393390655518,
        5000
    ],
    [
        4.9998860359191895,
        5000
    ],
    [
        4.999183177947998,
        5000
    ],
    [
        5.000225782394409,
        5000
    ],
    [
        4.9996795654296875,
        5000
    ],
    [
        4.999835014343262,
        5000
    ],
    [
        4.999478578567505,
        5000
    ],
    [
        4.999918699264526,
        5000
    ],
    [
        4.999856948852539,
        5000
    ],
    [
        4.999530792236328,
        5000
    ],
    [
        4.999507188796997,
        5000
    ],
    [
        5.0002601146698,
        5000
    ],
    [
        4.999974727630615,
        5000
    ],
    [
        5.000051259994507,
        5000
    ],
    [
        4.9990458488464355,
        5000
    ],
    [
        4.999610900878906,
        5000
    ],
    [
        5.000080823898315,
        5000
    ],
    [
        4.999649286270142,
        5000
    ],
    [
        5.000235319137573,
        5000
    ],
    [
        4.998884677886963,
        5000
    ],
    [
        4.999875783920288,
        5000
    ],
    [
        5.000746488571167,
        5000
    ],
    [
        4.999794006347656,
        5000
    ],
    [
        4.999047040939331,
        5000
    ],
    [
        4.999541521072388,
        5000
    ],
    [
        5.000712633132935,
        5000
    ],
    [
        4.999730348587036,
        5000
    ],
    [
        4.999234676361084,
        5000
    ],
    [
        4.999918699264526,
        5000
    ],
    [
        5.0000224113464355,
        5000
    ],
    [
        4.999305725097656,
        5000
    ],
    [
        4.999229192733765,
        5000
    ],
    [
        5.000525236129761,
        5000
    ],
    [
        4.999495506286621,
        5000
    ],
    [
        5.000237226486206,
        5000
    ],
    [
        4.998842716217041,
        5000
    ],
    [
        4.999946594238281,
        5000
    ],
    [
        5.0001561641693115,
        5000
    ],
    [
        5.000150680541992,
        5000
    ],
    [
        5.00017786026001,
        5000
    ],
    [
        4.998617172241211,
        5000
    ],
    [
        5.000456809997559,
        5000
    ],
    [
        4.9995057582855225,
        5000
    ],
    [
        5.000543117523193,
        5000
    ],
    [
        4.998405694961548,
        5000
    ],
    [
        5.000838279724121,
        5000
    ],
    [
        4.999407529830933,
        5000
    ],
    [
        4.999753475189209,
        5000
    ],
    [
        4.999708652496338,
        5000
    ],
    [
        5.00007700920105,
        5000
    ],
    [
        4.999965667724609,
        5000
    ],
    [
        4.999128818511963,
        5000
    ],
    [
        5.000034809112549,
        5000
    ],
    [
        4.999045133590698,
        5000
    ],
    [
        5.000838041305542,
        5000
    ],
    [
        4.99973726272583,
        5000
    ],
    [
        4.999382495880127,
        5000
    ],
    [
        4.999497175216675,
        5000
    ],
    [
        4.999729156494141,
        5000
    ],
    [
        5.000466346740723,
        5000
    ],
    [
        4.999454975128174,
        5000
    ],
    [
        4.999583959579468,
        5000
    ],
    [
        4.999675512313843,
        5000
    ],
    [
        5.0005042552948,
        5000
    ],
    [
        5.000052452087402,
        5000
    ],
    [
        4.999109506607056,
        5000
    ],
    [
        5.000246286392212,
        5000
    ],
    [
        4.999303102493286,
        5000
    ],
    [
        5.000249147415161,
        5000
    ],
    [
        4.998788595199585,
        5000
    ],
    [
        5.0002827644348145,
        5000
    ],
    [
        4.999477386474609,
        5000
    ],
    [
        5.000129461288452,
        5000
    ],
    [
        4.999334096908569,
        5000
    ],
    [
        5.0004026889801025,
        5000
    ],
    [
        4.999779939651489,
        5000
    ],
    [
        4.999929666519165,
        5000
    ],
    [
        4.998950242996216,
        5000
    ],
    [
        5.000062465667725,
        5000
    ],
    [
        4.999585866928101,
        5000
    ],
    [
        5.000675916671753,
        5000
    ],
    [
        4.999506711959839,
        5000
    ],
    [
        4.999521017074585,
        5000
    ],
    [
        4.999315500259399,
        5000
    ],
    [
        5.000797271728516,
        5000
    ],
    [
        4.999695777893066,
        5000
    ],
    [
        4.998730659484863,
        5000
    ],
    [
        5.0004894733428955,
        5000
    ],
    [
        5.000037908554077,
        5000
    ],
    [
        4.99996280670166,
        5000
    ],
    [
        4.999008417129517,
        5000
    ],
    [
        4.9998650550842285,
        5000
    ],
    [
        4.999857425689697,
        5000
    ],
    [
        4.999720096588135,
        5000
    ],
    [
        4.999574661254883,
        5000
    ],
    [
        4.999628305435181,
        5000
    ],
    [
        4.999911308288574,
        5000
    ],
    [
        5.000624895095825,
        5000
    ],
    [
        4.9984447956085205,
        5000
    ],
    [
        5.000390291213989,
        5000
    ],
    [
        5.000351190567017,
        5000
    ],
    [
        4.999809741973877,
        5000
    ],
    [
        4.9986958503723145,
        5000
    ],
    [
        4.999743938446045,
        5000
    ],
    [
        5.000042915344238,
        5000
    ],
    [
        5.000046730041504,
        5000
    ],
    [
        5.0001380443573,
        5000
    ],
    [
        4.999756336212158,
        5000
    ],
    [
        4.99914813041687,
        5000
    ],
    [
        4.999988555908203,
        5000
    ],
    [
        5.000065803527832,
        5000
    ],
    [
        4.999700307846069,
        5000
    ],
    [
        5.000056028366089,
        5000
    ],
    [
        4.99993896484375,
        5000
    ],
    [
        4.99957013130188,
        5000
    ],
    [
        4.999075412750244,
        5000
    ],
    [
        5.00061559677124,
        5000
    ],
    [
        4.999451160430908,
        5000
    ],
    [
        5.0002641677856445,
        5000
    ],
    [
        4.998740911483765,
        5000
    ],
    [
        5.000253677368164,
        5000
    ],
    [
        5.000051975250244,
        5000
    ],
    [
        4.999419450759888,
        5000
    ],
    [
        5.000275611877441,
        5000
    ],
    [
        4.999470472335815,
        5000
    ],
    [
        4.999720811843872,
        5000
    ],
    [
        4.999848365783691,
        5000
    ],
    [
        5.000426769256592,
        5000
    ],
    [
        4.998729944229126,
        5000
    ],
    [
        4.999743223190308,
        5000
    ],
    [
        5.000311613082886,
        5000
    ],
    [
        5.000424146652222,
        5000
    ],
    [
        4.998492956161499,
        5000
    ],
    [
        5.000468969345093,
        5000
    ],
    [
        4.999454021453857,
        5000
    ],
    [
        5.000047445297241,
        5000
    ],
    [
        4.999524831771851,
        5000
    ],
    [
        5.0001068115234375,
        5000
    ],
    [
        5.00002121925354,
        5000
    ],
    [
        4.999492645263672,
        5000
    ],
    [
        4.999426364898682,
        5000
    ],
    [
        4.999473333358765,
        5000
    ],
    [
        4.999948263168335,
        5000
    ],
    [
        5.000321388244629,
        5000
    ],
    [
        4.999338626861572,
        5000
    ],
    [
        4.999585866928101,
        5000
    ],
    [
        4.999826908111572,
        5000
    ],
    [
        4.999790668487549,
        5000
    ],
    [
        5.001020908355713,
        5000
    ],
    [
        4.9987633228302,
        5000
    ],
    [
        4.999589920043945,
        5000
    ],
    [
        5.000807523727417,
        5000
    ],
    [
        4.999605178833008,
        5000
    ],
    [
        4.998522520065308,
        5000
    ],
    [
        5.000738620758057,
        5000
    ],
    [
        4.999855995178223,
        5000
    ],
    [
        4.999724388122559,
        5000
    ],
    [
        4.999260425567627,
        5000
    ],
    [
        4.999669551849365,
        5000
    ],
    [
        4.999863862991333,
        5000
    ],
    [
        4.999981880187988,
        5000
    ],
    [
        4.999429702758789,
        5000
    ],
    [
        5.000068426132202,
        5000
    ],
    [
        4.999510765075684,
        5000
    ],
    [
        5.000194072723389,
        5000
    ],
    [
        4.999698638916016,
        5000
    ],
    [
        4.999781847000122,
        5000
    ],
    [
        4.999695777893066,
        5000
    ],
    [
        5.000379323959351,
        5000
    ],
    [
        5.000000476837158,
        5000
    ],
    [
        4.9984495639801025,
        5000
    ],
    [
        4.999948501586914,
        5000
    ],
    [
        636.8890287876129,
        636917
    ],
    [
        3.0820329189300537,
        3083
    ],
    [
        5.000337600708008,
        5000
    ],
    [
        5.000260353088379,
        5000
    ],
    [
        4.999823808670044,
        5000
    ],
    [
        4.999367713928223,
        5000
    ],
    [
        4.999781847000122,
        5000
    ],
    [
        5.000135183334351,
        5000
    ],
    [
        4.999829530715942,
        5000
    ],
    [
        4.99984884262085,
        5000
    ],
    [
        4.999052047729492,
        5000
    ],
    [
        5.000414133071899,
        5000
    ],
    [
        4.999117612838745,
        5000
    ],
    [
        5.000683069229126,
        5000
    ],
    [
        4.999336004257202,
        5000
    ],
    [
        4.999671459197998,
        5000
    ],
    [
        4.9999494552612305,
        5000
    ],
    [
        4.999446153640747,
        5000
    ],
    [
        4.999696254730225,
        5000
    ],
    [
        4.999938726425171,
        5000
    ],
    [
        4.999707937240601,
        5000
    ],
    [
        4.999931573867798,
        5000
    ],
    [
        4.999536514282227,
        5000
    ],
    [
        5.000085830688477,
        5000
    ],
    [
        4.999351978302002,
        5000
    ],
    [
        5.000037908554077,
        5000
    ],
    [
        5.000456809997559,
        5000
    ],
    [
        4.998752593994141,
        5000
    ],
    [
        4.9996912479400635,
        5000
    ],
    [
        5.000510215759277,
        5000
    ],
    [
        4.999837875366211,
        5000
    ],
    [
        4.999040842056274,
        5000
    ],
    [
        5.000426292419434,
        5000
    ],
    [
        4.9998626708984375,
        5000
    ],
    [
        4.999671459197998,
        5000
    ],
    [
        4.998641729354858,
        5000
    ],
    [
        5.000102758407593,
        5000
    ],
    [
        4.99994969367981,
        5000
    ],
    [
        5.0001747608184814,
        5000
    ],
    [
        4.999313116073608,
        5000
    ],
    [
        5.000277042388916,
        5000
    ],
    [
        5.000351190567017,
        5000
    ],
    [
        4.999834775924683,
        5000
    ],
    [
        4.9987781047821045,
        5000
    ],
    [
        5.000091075897217,
        5000
    ],
    [
        5.0002851486206055,
        5000
    ],
    [
        4.999478101730347,
        5000
    ],
    [
        4.999932289123535,
        5000
    ],
    [
        4.999029636383057,
        5000
    ],
    [
        4.999979734420776,
        5000
    ],
    [
        5.000532627105713,
        5000
    ],
    [
        4.999862194061279,
        5000
    ],
    [
        4.999420166015625,
        5000
    ],
    [
        4.999359846115112,
        5000
    ],
    [
        5.000280857086182,
        5000
    ],
    [
        4.999864101409912,
        5000
    ],
    [
        4.99865460395813,
        5000
    ],
    [
        4.999961614608765,
        5000
    ],
    [
        5.000922441482544,
        5000
    ],
    [
        4.999605417251587,
        5000
    ],
    [
        4.99913215637207,
        5000
    ],
    [
        4.9997477531433105,
        5000
    ],
    [
        5.000269174575806,
        5000
    ],
    [
        5.000487327575684,
        5000
    ],
    [
        4.998516798019409,
        5000
    ],
    [
        4.9998393058776855,
        5000
    ],
    [
        4.99992299079895,
        5000
    ],
    [
        5.000013113021851,
        5000
    ],
    [
        4.999870300292969,
        5000
    ],
    [
        4.999304294586182,
        5000
    ],
    [
        5.000465393066406,
        5000
    ],
    [
        4.999225854873657,
        5000
    ],
    [
        5.000251531600952,
        5000
    ],
    [
        4.999366044998169,
        5000
    ],
    [
        4.999287843704224,
        5000
    ],
    [
        5.000896215438843,
        5000
    ],
    [
        4.999265909194946,
        5000
    ],
    [
        4.999325752258301,
        5000
    ],
    [
        4.999860763549805,
        5000
    ],
    [
        5.000483751296997,
        5000
    ],
    [
        5.000020980834961,
        5000
    ],
    [
        4.9986348152160645,
        5000
    ],
    [
        4.999919176101685,
        5000
    ],
    [
        5.000330924987793,
        5000
    ],
    [
        5.000048398971558,
        5000
    ],
    [
        4.998587369918823,
        5000
    ],
    [
        5.000669956207275,
        5000
    ],
    [
        5.000044107437134,
        5000
    ],
    [
        5.000235319137573,
        5000
    ],
    [
        4.999298095703125,
        5000
    ],
    [
        4.999484300613403,
        5000
    ],
    [
        4.999906778335571,
        5000
    ],
    [
        5.000282526016235,
        5000
    ],
    [
        4.999630689620972,
        5000
    ],
    [
        4.998751163482666,
        5000
    ],
    [
        5.000326871871948,
        5000
    ],
    [
        4.99950647354126,
        5000
    ],
    [
        5.000080585479736,
        5000
    ],
    [
        4.999605655670166,
        5000
    ],
    [
        5.000173807144165,
        5000
    ],
    [
        5.000293254852295,
        5000
    ],
    [
        4.999831199645996,
        5000
    ],
    [
        4.9987099170684814,
        5000
    ],
    [
        4.9996867179870605,
        5000
    ],
    [
        5.000548839569092,
        5000
    ],
    [
        4.999835252761841,
        5000
    ],
    [
        4.9992687702178955,
        5000
    ],
    [
        4.99923038482666,
        5000
    ],
    [
        5.000206708908081,
        5000
    ],
    [
        4.99983024597168,
        5000
    ],
    [
        5.000763177871704,
        5000
    ],
    [
        4.998412847518921,
        5000
    ],
    [
        5.000567674636841,
        5000
    ],
    [
        4.999671697616577,
        5000
    ],
    [
        5.000042915344238,
        5000
    ],
    [
        4.998529434204102,
        5000
    ],
    [
        5.000442743301392,
        5000
    ],
    [
        5.000338077545166,
        5000
    ],
    [
        4.999900579452515,
        5000
    ],
    [
        4.998663425445557,
        5000
    ],
    [
        5.000049591064453,
        5000
    ],
    [
        5.000338792800903,
        5000
    ],
    [
        4.999834775924683,
        5000
    ],
    [
        4.9990012645721436,
        5000
    ],
    [
        5.0004355907440186,
        5000
    ],
    [
        4.999286890029907,
        5000
    ],
    [
        5.000973224639893,
        5000
    ],
    [
        4.998126983642578,
        5000
    ],
    [
        5.000388145446777,
        5000
    ],
    [
        5.0003015995025635,
        5000
    ],
    [
        4.999643325805664,
        5000
    ],
    [
        4.999888896942139,
        5000
    ],
    [
        4.999300241470337,
        5000
    ],
    [
        4.999785661697388,
        5000
    ],
    [
        5.000237464904785,
        5000
    ],
    [
        5.000291109085083,
        5000
    ],
    [
        4.998249292373657,
        5000
    ],
    [
        4.99992561340332,
        5000
    ],
    [
        5.0004894733428955,
        5000
    ],
    [
        4.999765634536743,
        5000
    ],
    [
        4.99928092956543,
        5000
    ],
    [
        5.0001747608184814,
        5000
    ],
    [
        5.000177621841431,
        5000
    ],
    [
        4.999823093414307,
        5000
    ],
    [
        4.998515605926514,
        5000
    ],
    [
        5.000401496887207,
        5000
    ],
    [
        5.00002384185791,
        5000
    ],
    [
        4.999535083770752,
        5000
    ],
    [
        4.99943470954895,
        5000
    ],
    [
        5.000592470169067,
        5000
    ],
    [
        4.9994988441467285,
        5000
    ],
    [
        4.99987530708313,
        5000
    ],
    [
        5.000138998031616,
        5000
    ],
    [
        4.999119758605957,
        5000
    ],
    [
        5.0001420974731445,
        5000
    ],
    [
        4.999441862106323,
        5000
    ],
    [
        5.00056791305542,
        5000
    ],
    [
        4.999165296554565,
        5000
    ],
    [
        4.999694585800171,
        5000
    ],
    [
        5.00017786026001,
        5000
    ],
    [
        5.000013589859009,
        5000
    ],
    [
        4.998575925827026,
        5000
    ],
    [
        5.0007312297821045,
        5000
    ],
    [
        4.999139070510864,
        5000
    ],
    [
        5.000442743301392,
        5000
    ],
    [
        4.999703884124756,
        5000
    ],
    [
        4.999150037765503,
        5000
    ],
    [
        5.000226974487305,
        5000
    ],
    [
        4.999983310699463,
        5000
    ],
    [
        4.998989820480347,
        5000
    ],
    [
        4.99974513053894,
        5000
    ],
    [
        5.0008814334869385,
        5000
    ],
    [
        4.999301910400391,
        5000
    ],
    [
        5.000536918640137,
        5000
    ],
    [
        4.998614549636841,
        5000
    ],
    [
        4.999744415283203,
        5000
    ],
    [
        5.000869274139404,
        5000
    ],
    [
        4.999995708465576,
        5000
    ],
    [
        4.998985290527344,
        5000
    ],
    [
        5.000159025192261,
        5000
    ],
    [
        5.0000433921813965,
        5000
    ],
    [
        4.999175071716309,
        5000
    ],
    [
        4.999051094055176,
        5000
    ],
    [
        4.9999871253967285,
        5000
    ],
    [
        5.000340461730957,
        5000
    ],
    [
        5.000080347061157,
        5000
    ],
    [
        4.998725414276123,
        5000
    ],
    [
        5.000439405441284,
        5000
    ],
    [
        4.999997615814209,
        5000
    ],
    [
        4.999980211257935,
        5000
    ],
    [
        4.999311208724976,
        5000
    ],
    [
        4.999884843826294,
        5000
    ],
    [
        4.999344825744629,
        5000
    ],
    [
        5.000598430633545,
        5000
    ],
    [
        4.999750852584839,
        5000
    ],
    [
        4.999223709106445,
        5000
    ],
    [
        5.000009536743164,
        5000
    ],
    [
        4.999490261077881,
        5000
    ],
    [
        5.000641584396362,
        5000
    ],
    [
        4.999546766281128,
        5000
    ],
    [
        4.99970555305481,
        5000
    ],
    [
        4.999611139297485,
        5000
    ],
    [
        5.0003767013549805,
        5000
    ],
    [
        4.998433589935303,
        5000
    ],
    [
        5.000780344009399,
        5000
    ],
    [
        4.999439001083374,
        5000
    ],
    [
        5.000011920928955,
        5000
    ],
    [
        4.999355792999268,
        5000
    ],
    [
        4.999411344528198,
        5000
    ],
    [
        4.999973773956299,
        5000
    ],
    [
        5.000600814819336,
        5000
    ],
    [
        4.999113082885742,
        5000
    ],
    [
        4.99997878074646,
        5000
    ],
    [
        5.0001349449157715,
        5000
    ],
    [
        4.999794244766235,
        5000
    ],
    [
        4.999021291732788,
        5000
    ],
    [
        4.99968695640564,
        5000
    ],
    [
        5.000313997268677,
        5000
    ],
    [
        5.000327825546265,
        5000
    ],
    [
        4.999594688415527,
        5000
    ],
    [
        4.999081134796143,
        5000
    ],
    [
        5.000566720962524,
        5000
    ],
    [
        4.99901270866394,
        5000
    ],
    [
        5.000623941421509,
        5000
    ],
    [
        4.9993205070495605,
        5000
    ],
    [
        4.999823808670044,
        5000
    ],
    [
        5.000262260437012,
        5000
    ],
    [
        4.9990410804748535,
        5000
    ],
    [
        4.999787330627441,
        5000
    ],
    [
        4.999610185623169,
        5000
    ],
    [
        4.999841213226318,
        5000
    ],
    [
        5.000366926193237,
        5000
    ],
    [
        4.998941659927368,
        5000
    ],
    [
        5.00033974647522,
        5000
    ],
    [
        5.000210285186768,
        5000
    ],
    [
        4.999150037765503,
        5000
    ],
    [
        5.000491619110107,
        5000
    ],
    [
        4.999027490615845,
        5000
    ],
    [
        4.999830007553101,
        5000
    ],
    [
        5.0003252029418945,
        5000
    ],
    [
        4.999867916107178,
        5000
    ],
    [
        4.999136209487915,
        5000
    ],
    [
        4.999955654144287,
        5000
    ],
    [
        5.000166893005371,
        5000
    ],
    [
        4.9995341300964355,
        5000
    ],
    [
        4.99961519241333,
        5000
    ],
    [
        5.000062465667725,
        5000
    ],
    [
        4.999422788619995,
        5000
    ],
    [
        4.999914884567261,
        5000
    ],
    [
        4.998900651931763,
        5000
    ],
    [
        5.000208616256714,
        5000
    ],
    [
        5.000302076339722,
        5000
    ],
    [
        5.00014591217041,
        5000
    ],
    [
        4.998888969421387,
        5000
    ],
    [
        4.999921798706055,
        5000
    ],
    [
        5.000386476516724,
        5000
    ],
    [
        4.999639511108398,
        5000
    ],
    [
        4.998911142349243,
        5000
    ],
    [
        5.000208377838135,
        5000
    ],
    [
        5.000252723693848,
        5000
    ],
    [
        5.000210523605347,
        5000
    ],
    [
        4.999373435974121,
        5000
    ],
    [
        4.999133110046387,
        5000
    ],
    [
        4.999831914901733,
        5000
    ],
    [
        4.999850749969482,
        5000
    ],
    [
        5.000164270401001,
        5000
    ],
    [
        4.999741315841675,
        5000
    ],
    [
        4.999228000640869,
        5000
    ],
    [
        5.000456809997559,
        5000
    ],
    [
        5.000596046447754,
        5000
    ],
    [
        4.99839973449707,
        5000
    ],
    [
        5.000148773193359,
        5000
    ],
    [
        4.99965500831604,
        5000
    ],
    [
        5.000053882598877,
        5000
    ],
    [
        4.998895168304443,
        5000
    ],
    [
        5.000050067901611,
        5000
    ],
    [
        5.000303506851196,
        5000
    ],
    [
        5.0000293254852295,
        5000
    ],
    [
        4.999144077301025,
        5000
    ],
    [
        4.999522686004639,
        5000
    ],
    [
        5.0002663135528564,
        5000
    ],
    [
        5.000084400177002,
        5000
    ],
    [
        5.00032114982605,
        5000
    ],
    [
        4.998689889907837,
        5000
    ],
    [
        5.0002830028533936,
        5000
    ],
    [
        4.9993085861206055,
        5000
    ],
    [
        636.3163607120514,
        636343
    ],
    [
        3.6554884910583496,
        3657
    ],
    [
        5.000249862670898,
        5000
    ],
    [
        4.999905347824097,
        5000
    ],
    [
        5.000211238861084,
        5000
    ],
    [
        4.999354839324951,
        5000
    ],
    [
        4.999786853790283,
        5000
    ],
    [
        4.999751567840576,
        5000
    ],
    [
        5.000332593917847,
        5000
    ],
    [
        4.998870134353638,
        5000
    ],
    [
        5.000206708908081,
        5000
    ],
    [
        4.999920129776001,
        5000
    ],
    [
        4.999751567840576,
        5000
    ],
    [
        4.99937629699707,
        5000
    ],
    [
        5.0002148151397705,
        5000
    ],
    [
        4.999966144561768,
        5000
    ],
    [
        4.999672889709473,
        5000
    ],
    [
        4.9986371994018555,
        5000
    ],
    [
        5.000301361083984,
        5000
    ],
    [
        4.9999003410339355,
        5000
    ],
    [
        5.000427961349487,
        5000
    ],
    [
        4.999532222747803,
        5000
    ],
    [
        4.998964786529541,
        5000
    ],
    [
        5.000499248504639,
        5000
    ],
    [
        5.000067472457886,
        5000
    ],
    [
        4.999388217926025,
        5000
    ],
    [
        4.999027490615845,
        5000
    ],
    [
        5.000752925872803,
        5000
    ],
    [
        4.999595642089844,
        5000
    ],
    [
        5.000486135482788,
        5000
    ],
    [
        4.998483419418335,
        5000
    ],
    [
        4.999632835388184,
        5000
    ],
    [
        5.000821352005005,
        5000
    ],
    [
        4.9995598793029785,
        5000
    ],
    [
        4.998753309249878,
        5000
    ],
    [
        5.0006325244903564,
        5000
    ],
    [
        4.999967336654663,
        5000
    ],
    [
        5.000312328338623,
        5000
    ],
    [
        4.999006032943726,
        5000
    ],
    [
        4.99934720993042,
        5000
    ],
    [
        4.999794244766235,
        5000
    ],
    [
        5.000152349472046,
        5000
    ],
    [
        5.000694990158081,
        5000
    ],
    [
        4.9987876415252686,
        5000
    ],
    [
        5.00000262260437,
        5000
    ],
    [
        5.000102281570435,
        5000
    ],
    [
        5.000328779220581,
        5000
    ],
    [
        4.998444557189941,
        5000
    ],
    [
        4.999891519546509,
        5000
    ],
    [
        5.000558853149414,
        5000
    ],
    [
        4.999963760375977,
        5000
    ],
    [
        4.998340845108032,
        5000
    ],
    [
        5.001047849655151,
        5000
    ],
    [
        4.998988151550293,
        5000
    ],
    [
        5.000737190246582,
        5000
    ],
    [
        4.998931884765625,
        5000
    ],
    [
        5.000036239624023,
        5000
    ],
    [
        4.999727249145508,
        5000
    ],
    [
        4.999480724334717,
        5000
    ],
    [
        5.000019311904907,
        5000
    ],
    [
        4.999131917953491,
        5000
    ],
    [
        5.000488758087158,
        5000
    ],
    [
        5.000271558761597,
        5000
    ],
    [
        4.998656511306763,
        5000
    ],
    [
        4.999583005905151,
        5000
    ],
    [
        5.000743389129639,
        5000
    ],
    [
        4.999602794647217,
        5000
    ],
    [
        4.999783515930176,
        5000
    ],
    [
        4.99964165687561,
        5000
    ],
    [
        5.000222444534302,
        5000
    ],
    [
        4.999138355255127,
        5000
    ],
    [
        5.0001795291900635,
        5000
    ],
    [
        4.998929262161255,
        5000
    ],
    [
        5.000108480453491,
        5000
    ],
    [
        5.000651597976685,
        5000
    ],
    [
        4.9999682903289795,
        5000
    ],
    [
        4.9990925788879395,
        5000
    ],
    [
        4.999576807022095,
        5000
    ],
    [
        4.999866008758545,
        5000
    ],
    [
        4.999888896942139,
        5000
    ],
    [
        4.999758005142212,
        5000
    ],
    [
        5.00018310546875,
        5000
    ],
    [
        4.9994056224823,
        5000
    ],
    [
        4.999983310699463,
        5000
    ],
    [
        4.999077796936035,
        5000
    ],
    [
        5.00034236907959,
        5000
    ],
    [
        4.999321937561035,
        5000
    ],
    [
        5.0005176067352295,
        5000
    ],
    [
        5.000316858291626,
        5000
    ],
    [
        4.998827695846558,
        5000
    ],
    [
        5.000155210494995,
        5000
    ],
    [
        4.999558210372925,
        5000
    ],
    [
        5.00057578086853,
        5000
    ],
    [
        4.998554468154907,
        5000
    ],
    [
        4.999628305435181,
        5000
    ],
    [
        5.00003719329834,
        5000
    ],
    [
        5.000509262084961,
        5000
    ],
    [
        4.999160289764404,
        5000
    ],
    [
        4.9989964962005615,
        5000
    ],
    [
        5.00084376335144,
        5000
    ],
    [
        5.000133514404297,
        5000
    ],
    [
        4.9986066818237305,
        5000
    ],
    [
        5.000319004058838,
        5000
    ],
    [
        5.000329256057739,
        5000
    ],
    [
        4.999586820602417,
        5000
    ],
    [
        4.9988555908203125,
        5000
    ],
    [
        5.000322580337524,
        5000
    ],
    [
        4.999454021453857,
        5000
    ],
    [
        5.000901699066162,
        5000
    ],
    [
        4.998320579528809,
        5000
    ],
    [
        4.999921798706055,
        5000
    ],
    [
        5.000636100769043,
        5000
    ],
    [
        4.999727725982666,
        5000
    ],
    [
        5.000405550003052,
        5000
    ],
    [
        4.998570203781128,
        5000
    ],
    [
        5.000271558761597,
        5000
    ],
    [
        4.999852180480957,
        5000
    ],
    [
        5.000229120254517,
        5000
    ],
    [
        4.998852014541626,
        5000
    ],
    [
        5.000047445297241,
        5000
    ],
    [
        4.9997947216033936,
        5000
    ],
    [
        5.000460147857666,
        5000
    ],
    [
        4.998490810394287,
        5000
    ],
    [
        5.000138521194458,
        5000
    ],
    [
        4.999918222427368,
        5000
    ],
    [
        5.0000786781311035,
        5000
    ],
    [
        4.999413251876831,
        5000
    ],
    [
        4.999627113342285,
        5000
    ],
    [
        4.999938011169434,
        5000
    ],
    [
        4.99996280670166,
        5000
    ],
    [
        5.00024938583374,
        5000
    ],
    [
        4.999224424362183,
        5000
    ],
    [
        4.999935150146484,
        5000
    ],
    [
        5.000247955322266,
        5000
    ],
    [
        4.999599456787109,
        5000
    ],
    [
        4.999135732650757,
        5000
    ],
    [
        5.0000646114349365,
        5000
    ],
    [
        4.999821662902832,
        5000
    ],
    [
        5.000647068023682,
        5000
    ],
    [
        4.998924732208252,
        5000
    ],
    [
        4.999952554702759,
        5000
    ],
    [
        4.9999470710754395,
        5000
    ],
    [
        5.0000200271606445,
        5000
    ],
    [
        4.999142408370972,
        5000
    ],
    [
        4.999942302703857,
        5000
    ],
    [
        5.000203847885132,
        5000
    ],
    [
        4.999928712844849,
        5000
    ],
    [
        4.999221324920654,
        5000
    ],
    [
        4.999430418014526,
        5000
    ],
    [
        5.000194311141968,
        5000
    ],
    [
        5.000286102294922,
        5000
    ],
    [
        5.0000011920928955,
        5000
    ],
    [
        4.998668193817139,
        5000
    ],
    [
        5.000267028808594,
        5000
    ],
    [
        4.999549388885498,
        5000
    ],
    [
        5.000039100646973,
        5000
    ],
    [
        4.999657869338989,
        5000
    ],
    [
        4.999383449554443,
        5000
    ],
    [
        4.999888896942139,
        5000
    ],
    [
        5.000516414642334,
        5000
    ],
    [
        4.99915885925293,
        5000
    ],
    [
        4.999464988708496,
        5000
    ],
    [
        5.000397682189941,
        5000
    ],
    [
        4.999715566635132,
        5000
    ],
    [
        4.999831914901733,
        5000
    ],
    [
        4.99951434135437,
        5000
    ],
    [
        4.999591827392578,
        5000
    ],
    [
        5.000461578369141,
        5000
    ],
    [
        4.999060392379761,
        5000
    ],
    [
        5.0003275871276855,
        5000
    ],
    [
        4.9993674755096436,
        5000
    ],
    [
        5.000105619430542,
        5000
    ],
    [
        5.000211715698242,
        5000
    ],
    [
        4.998889207839966,
        5000
    ],
    [
        5.000512599945068,
        5000
    ],
    [
        4.9998791217803955,
        5000
    ],
    [
        4.999391317367554,
        5000
    ],
    [
        4.999399185180664,
        5000
    ],
    [
        4.999786376953125,
        5000
    ],
    [
        5.000709295272827,
        5000
    ],
    [
        4.999324798583984,
        5000
    ],
    [
        4.999567270278931,
        5000
    ],
    [
        4.999300718307495,
        5000
    ],
    [
        5.000501394271851,
        5000
    ],
    [
        4.999989986419678,
        5000
    ],
    [
        4.9995269775390625,
        5000
    ],
    [
        5.000016450881958,
        5000
    ],
    [
        4.999068021774292,
        5000
    ],
    [
        5.00047492980957,
        5000
    ],
    [
        4.999349594116211,
        5000
    ],
    [
        4.999691963195801,
        5000
    ],
    [
        5.000421762466431,
        5000
    ],
    [
        4.999462604522705,
        5000
    ],
    [
        4.998676538467407,
        5000
    ],
    [
        5.000605821609497,
        5000
    ],
    [
        5.000132322311401,
        5000
    ],
    [
        4.999704122543335,
        5000
    ],
    [
        4.99883246421814,
        5000
    ],
    [
        5.000712633132935,
        5000
    ],
    [
        4.999223947525024,
        5000
    ],
    [
        5.000154972076416,
        5000
    ],
    [
        5.0000269412994385,
        5000
    ],
    [
        4.999396085739136,
        5000
    ],
    [
        5.000020980834961,
        5000
    ],
    [
        5.000052452087402,
        5000
    ],
    [
        5.000305891036987,
        5000
    ],
    [
        4.998469591140747,
        5000
    ],
    [
        5.000498294830322,
        5000
    ],
    [
        5.000068187713623,
        5000
    ],
    [
        4.999488592147827,
        5000
    ],
    [
        4.999563694000244,
        5000
    ],
    [
        4.999671220779419,
        5000
    ],
    [
        5.000359535217285,
        5000
    ],
    [
        4.999935626983643,
        5000
    ]
]

I have not tested the SysTick on the current Zephyr main branch. Since reducing the value for CONFIG_SYS_CLOCK_TICKS_PER_SEC to 2048 seems to solve the timer delay issues I've been observing, I have been trying to focus on the nrf RTC issues which prevent it from operating as intended. The weird SysTick behavior is probably a hardware mistake on our side. I'd like to track it down, but I can't seem to reproduce it on dev kit hardware, so I think it would be hard for you to reproduce on your end.

Truthfully, I really don't know what's going on. In the past I put in log messages to verify that the Bluetooth mesh extended advertiser was requesting a reasonably short delay for the work; it never requested an interval of zero or one greater than tens of milliseconds. Yet, I have seen the work take hundreds of seconds to be operated on. This is a catastrophic bug that I can't believe exists in something like ncs. I'm hesitant to believe that it is, since we wouldn't be the only ones having this issue and from what I know ncs is used all over the place. I'd be happy to accept that it's an issue with our project configuration or hardware, but we're not doing anything the device wasn't advertised to do, and it seems to be reproducible on development hardware.