ESB link in PRX mode requires re-initialization after minutes with nRF54L15

RE: ESB link in PRX mode requires re-initialization after minutes with nRF54L15

galenchurch — Thu, 15 May 2025 17:39:33 GMT

Thanks for the results, that is good to hear.

We have also produced similar results with RevB and Rev1 DKs as you have (issues are not there) though there are also other variations to our custom board but I think we can be optimistic this will be resolved on our next spin and its isolated to RevA.

We are putting payloads in the ACK, we are not switching roles.

RE: ESB link in PRX mode requires re-initialization after minutes with nRF54L15

Johnny Nguyen — Thu, 15 May 2025 16:04:38 GMT

Hello,

I've tested overnight on both SDK 2.8.0 and SDK 3.0.0 on rev 1 and rev B devices and the issue did not present itself overnight. This was at maximum communication frequency possible. (This to me rules out a subtle change in the SDK potentially being the culprit and points more at the HW).

I do not think the slowed down comm frequency, which was one packet every 200 ms on the old chips, is acceptable for an ESB application -- it was slowed down due to invasive debugging in the ESB library to see what the radio state was when it locked up (but it never locked up even in 14+ hours at the slower rate)

Your stop gap seems acceptable until you can get back on the latest devices, but if it winds up giving grief you may need to restart the radio, I am not sure what is problematic on the old device.

Before your next boards are built, could you mock hardware specific functions that are for your board and just validate the ESB side with dummy data on a current DK?

The only other divergence between our test setups is potentially what you state about sending messages back to the PTX other than acks -- to clarify: are you flipping their roles for this, or putting payloads in the ack?

Best regards,

RE: ESB link in PRX mode requires re-initialization after minutes with nRF54L15

galenchurch — Wed, 14 May 2025 18:43:16 GMT

That's good to hear you can re-produce. I can speak with the team if reducing the comm frequency is possible. What freq did you find worked well?

We have RevA on our current custom boards and cannot replace until our next boards are built. It would be great to know if this will be a problem for latest versions (Rev1 or Rev2?)... i appreciate your continued testing.

RE: ESB link in PRX mode requires re-initialization after minutes with nRF54L15

galenchurch — Wed, 14 May 2025 18:35:08 GMT

1 - we are using CONFIG_SOC_NRF_FORCE_CONSTLAT already
2 - well the PRX is always the PRX but we do send some messages back to the PTX other then ack's. We are receiving transmissions at 12Hz and replying at 1Hz (from the problematic PRX)

One of my colleagues looked into migrating to v3.0.0 but was having some issues getting things to run on the EngA silicon (what is on our boards). I went through it and cherry picked things that looked like they might have an effect, specifically: https://github.com/nrfconnect/sdk-nrf/commit/be1549932bd0200a7d2da4c61e2c7b8132d514d2 (didn't improve anything really).

Lastly what i did do as a hack to improve things after noticing it was hung at the ack state is add the following:

static void on_radio_disabled_rx_ack(void)
{
	esb_fem_for_ack_rx();

	if (IS_ENABLED(CONFIG_ESB_FAST_SWITCHING)) {
		nrf_radio_shorts_set(NRF_RADIO, radio_shorts_common);
		nrf_radio_task_trigger(NRF_RADIO, NRF_RADIO_TASK_RXEN);
	} else {
		nrf_radio_shorts_set(NRF_RADIO, (radio_shorts_common |
						 NRF_RADIO_SHORT_DISABLED_TXEN_MASK));
	}

	update_rf_payload_format(esb_cfg.payload_length);

	nrf_radio_packetptr_set(NRF_RADIO, rx_payload_buffer);
	on_radio_disabled = on_radio_disabled_rx;

	esb_state = ESB_STATE_PRX;

#if defined(CONFIG_ESB_USE_PRX_ACK_TIMEOUT)

#if IS_EMPTY(CONFIG_ESB_PRX_ACK_TIMEOUT_US)
#error "No ESB_PRX_ACK_TIMEOUT_US provided but ESB_USE_PRX_ACK_TIMEOUT enabled"
#endif

	/* Configure timer to produce an ISR after retransmit_delay */
	nrf_timer_task_trigger(esb_timer.p_reg, NRF_TIMER_TASK_CLEAR);
	nrfx_timer_clear(&esb_timer);
	nrfx_timer_compare(&esb_timer, NRF_TIMER_CC_CHANNEL1,
		CONFIG_ESB_PRX_ACK_TIMEOUT_US, true);

	/* Configure PPI to start the timer when transmission ends */
	esb_ppi_for_wait_for_rx_set();

	nrf_timer_event_clear(esb_timer.p_reg, NRF_TIMER_EVENT_COMPARE1);

	on_timer_compare1 = on_timeout;

#endif

}

#if defined(CONFIG_ESB_USE_PRX_ACK_TIMEOUT)
static void on_timeout(){
	LOG_ERR("Timeout function called on_timeout");
	nrf_timer_int_disable(esb_timer.p_reg,  nrf_timer_compare_int_get(NRF_TIMER_CC_CHANNEL1));
	esb_ppi_for_wait_for_rx_clear();

	clear_events_restart_rx();
	LOG_ERR("Timeout function called on_timeout return");
}
#endif

For the most part this seems to be working to add a timeout to that state that will clear the events and restart rx via `clear_events_restart_rx()`. If you had any advice on any unforeseen implications that might occur here that would be very helpful. Additionally I'm not sure if there are other states the system will hang id need to add something like this too, but ill cross that bridge when i get there. Otherwise for now this seems like a good stopgap until we can migrate to v3.0.0 and Rev1/2 on our next spin of boards.

RE: ESB link in PRX mode requires re-initialization after minutes with nRF54L15

Johnny Nguyen — Wed, 14 May 2025 18:23:30 GMT

an update:
after leaving it running for hours on revB and NCS 2.8.0, I was seeing the PRX stuck lockup requiring a reset. I slowed down the ESB communication frequency and it did not lock up during extended time testing.

As an extra datapoint, I tested latest 54L15 on NCS 3.0.0 and was not able to replicate the failure, even at maximum transmission rate. I am still leaving this test running to see if it comes up.

Any reason you are locked on v2.8.0 and revA chips?

RE: ESB link in PRX mode requires re-initialization after minutes with nRF54L15

Johnny Nguyen — Wed, 14 May 2025 00:09:27 GMT

Hello,

Thanks for letting me know.

1- Out of curiosity, if you use TASKS_CONSTLAT, does the failure still present itself? You should be using CONSTLAT when the radio is enabled regardless. (You can use the accompanying TASKS_LOWPOWER when you are through with it, but for exploring the failure case let's keep it enabled.)

2- Is the PRX always a PRX?

Amanda touched on some useful debugging tools and info, you can also use PPI to observe the radio activity on IO pin toggles.

(Some examples: Add alternative to radio library ppi pair for something more flexible. · Issue #5 · droidecahedron/esb_multi, https://github.com/droidecahedron/esb_multi/blob/68694693cb49272320ade3253a0546d0029f6535/esb_prx_blefallback/src/main.c#L23-L49)

but it would mostly just show you when the radio is active/inactive so you could see missed packets and connection events.

And for more information around ESB, I find the following insightful:

Old nRF24 user guide: nRF24LE1 Product Specification v1.6

Blog by my colleague: (+) Intro to ShockBurst/Enhanced ShockBurst - Blogs - Nordic Blog - Nordic DevZone

Said colleague's FOSS example project: inductivekickback/rc_radio: Remote Control Radio library for nRF52 using Enhanced ShockBurst.

Please let me know what you find as you collect more logs around the failure case.

I am working on replicating it on my end -- but am unable to do so reliably. Is the only thing that causes this lock up just a long period of receiving packets?

As an extra datapoint, could you try current 54L15DKs and NCS v3.0.0 stock PRX/PTX operation and see if the failure persists there as well?

BR,

RE: ESB link in PRX mode requires re-initialization after minutes with nRF54L15

galenchurch — Tue, 13 May 2025 18:18:24 GMT

For the PRX I am using NCS v2.8.0 for our custom board with nRF54L15 EngA

RE: ESB link in PRX mode requires re-initialization after minutes with nRF54L15

Amanda Hsieh — Tue, 13 May 2025 16:20:02 GMT

What NCS version are you using?

RE: ESB link in PRX mode requires re-initialization after minutes with nRF54L15

galenchurch — Mon, 12 May 2025 17:11:09 GMT

Hi Amanda Hsieh

I have added more logging and I can see when the PRX side hangs after what i assume is sending an ack payload via `esb.c::on_radio_disabled_rx_ack()`. I know that function returns and normally we would get another event on `esb_radio_direct_irq_handler` to handle normal reception, but none occurs. There is no event triggering that ISR to be called again and therefor `esb.c::radio_irq_handler()` is never called. What could cause the system to be hung in this state?

Just to reiterate, if we reset the PRX everything returns to normal, i.e. no change to the PTX device is required to return to normal and we know the PTX is still transmitting.

Thank,

Galen

RE: ESB link in PRX mode requires re-initialization after minutes with nRF54L15

galenchurch — Thu, 08 May 2025 13:09:27 GMT

HiAmanda Hsieh , thanks for this initial info. I will add in some additional logging around the radio and let you know when i have some better information/results from that.

RE: ESB link in PRX mode requires re-initialization after minutes with nRF54L15

Amanda Hsieh — Thu, 08 May 2025 13:03:10 GMT

Hi,

There is no other documentation than: https://docs.nordicsemi.com/bundle/ncs-latest/page/nrf/protocols/esb/index.html and https://docs.nordicsemi.com/bundle/nrf-apis-latest/page/group_esb.html, unfortunately.

A way to see more of what's happening would be to use logging. The ESB subsystem only logs a few error messages, but you can add debug logs to see where the execution gets stuck.

Regarding the register that can be checked - ESB is not a driver nor a HW subsystem - it is a protocol implementation, so there is no dedicated register for ESB. However, you can check some radio registers to understand the issue better: e.g. State register: https://docs.nordicsemi.com/bundle/ps_nrf54L15/page/radio.html#ariaid-title102 or event registers.

Please provide the guidance or shared snippets of application code to help us reproduce the issue.

-Amanda H.

RE: ESB link in PRX mode requires re-initialization after minutes with nRF54L15

Amanda Hsieh — Wed, 07 May 2025 19:32:24 GMT

Hi,

I am checking with the team and will update the case when I collect enough information.

Regards,
Amanda H.