NCS Update to v2.5.2: MPSL Hard Fault on Advertisement

RE: NCS Update to v2.5.2: MPSL Hard Fault on Advertisement

Einar Thorsrud — Mon, 08 Apr 2024 12:30:01 GMT

Hi,

Thank you for sharing the details of your investigation and findings. I is good to know you solved it. (And to find that it was not due to a bug in the MPSL).

RE: NCS Update to v2.5.2: MPSL Hard Fault on Advertisement

Talisca — Mon, 08 Apr 2024 07:06:29 GMT

First of all, thanks for the tip with CONFIG_CLOCK_CONTROL_NRF_CALIBRATION_PERIOD.
This actually reduced the time from approx. 34 minutes to approx. 2 minutes.

I then started to systematically remove individual systems from our application.

It's embarrassing, but I found the error in our codebase.

From NCS v2.4 to v2.5, something also changed with the SPI drivers.
I have to mention that we are not using Zephyr's drivers, because the overhead and the event notification is not sufficient and fast enough for our time-critical application.
Therefore, we directly use the nrfx drivers for SPI.

Originally we had the following code for our initialization (snippet):

IRQ_DIRECT_CONNECT(SPI0_TWI0_IRQn, 0, nrfx_spis_0_irq_handler, 0);
irq_enable(SPI0_TWI0_IRQn);

spi_instance = NRFX_SPIS_INSTANCE(0);
nrfx_spis_config_t spi_config =
NRFX_SPIS_DEFAULT_CONFIG(spi_pins.sck, spi_pins.mosi, spi_pins.miso, spi_pins.csn);
spi_config.orc = 0x99u;
spi_config.def = 0x66u;
spi_config.mode = NRF_SPIS_MODE_3;
spi_config.bit_order = NRF_SPIS_BIT_ORDER_MSB_FIRST;
spi_config.miso_drive = NRF_GPIO_PIN_H0H1;

nrfx_err_t spi_err = nrfx_spis_init(&spi_instance, &spi_config, spi0_event, NULL);
if(spi_err != NRFX_SUCCESS)
{
BT_LOG_ERR("got spi error: %8x\n", spi_err);
}

Apparently something has changed in the declaration of the interrupt handler, which is why we have replaced the first line with:

IRQ_DIRECT_CONNECT(SPI0_TWI0_IRQn, 0, NRFX_SPIS_INST_HANDLER_GET(0), 0);

However, with NCS v2.5.2 we always received the following spi error during initialization:

<err> service: got spi error:  bad0005

Which indicates an invalid state of my SPI instance.
It seems that the SPI slave has now been pre-initialized by Zephyr or whatever - I don't know.

Therefore we added a call to nrfx_spis_uninit so that our code looked like this:

IRQ_DIRECT_CONNECT(SPI0_TWI0_IRQn, 0, NRFX_SPIS_INST_HANDLER_GET(0), 0);
irq_enable(SPI0_TWI0_IRQn);

nrfx_spis_uninit(&spi_instance);

spi_instance = NRFX_SPIS_INSTANCE(0);

...

nrfx_err_t spi_err = nrfx_spis_init(&spi_instance, &spi_config, spi0_event, NULL);

You will notice an error there, just as I did when removing the code parts, that nrfx_spis_uninit is called with an uninitialized instance.

My guess is that since we are using SPI0 as a slave, it is resetting the state of the correct instance, as the spi error then no longer occurs. Probably only other configurations are somewhat malformed.

After fixing the sequence, the MPSL Hard Fault no longer occurs.

It is nevertheless funny that none of our team noticed this error and SPI still simply worked.
It is also interesting that SPI was only initialized during the tests but never used and yet the error always occurred in combination with the advertisement.

RE: NCS Update to v2.5.2: MPSL Hard Fault on Advertisement

Einar Thorsrud — Thu, 04 Apr 2024 12:17:16 GMT

Hi,

Just a short follow-up from me while waiting for your input. We have reproduced the same assert in a simple application that just starts advertising, and then calls NVIC_DisableIRQ(POWER_CLOCK_IRQn). Our theory is that somethign similar to this happens in your application. (Maybe not exactly that Clock interrupt is disabled in NVIC, but something else that makes clock interrupt not get to the MPSL interrupt handler).

RE: NCS Update to v2.5.2: MPSL Hard Fault on Advertisement

Einar Thorsrud — Wed, 03 Apr 2024 13:48:00 GMT

Hi,

Thank you. We are looking forward to hopefully getting code from you that reproduce it. In the mean time, we have some additional questions:

Can you try to change CONFIG_CLOCK_CONTROL_NRF_CALIBRATION_PERIOD to 250 (from the default 4000) and see if the issue happens quicker then?
Can you conect with a debugger once the assert has happened, and use "west attach" then "run bt" and upload the resulting back trace here?
Is the application touching the POWR_CLOCK peripheral in any way, or modifiing the
POWER_CLOCK_IRQn in NVIC?

RE: NCS Update to v2.5.2: MPSL Hard Fault on Advertisement

Talisca — Wed, 03 Apr 2024 10:49:24 GMT

[quote userid="7377" url="~/f/nordic-q-a/109834/ncs-update-to-v2-5-2-mpsl-hard-fault-on-advertisement/476656"]Do you use a 32.768 kHz crystal or the LFRC as low frequency clock source?[/quote]

We use the internal RC as LFCLK.

[quote userid="7377" url="~/f/nordic-q-a/109834/ncs-update-to-v2-5-2-mpsl-hard-fault-on-advertisement/476656"]Is a connection established at any point, or is the device only starting and stopping advertising when this happens?[/quote]

No connections are established during these tests. We only use the nRF Connect app for android to check which advertisement data is sent from the device.
The advertisement itself works fine until the device crashes after approx. 34 minutes.

[quote userid="7377" url="~/f/nordic-q-a/109834/ncs-update-to-v2-5-2-mpsl-hard-fault-on-advertisement/476656"]Do you by any chance use the mpsl_clock API, for instance by calling mpsl_clock_hfclk_request() and/or mpsl_clock_hfclk_release()?[/quote]

We don't use any functionalities of the MPSL library, but this does not necessarily mean that none of the used interfaces may use them after all. We do not know that.

Among other things we use:

SPI Slave
UART
RTC
TIMER
External Flash via QSPI
Nordic UART Service (NUS)
Bootloader Chain
etc

[quote userid="7377" url="~/f/nordic-q-a/109834/ncs-update-to-v2-5-2-mpsl-hard-fault-on-advertisement/476656"]Is it possible for you to share code that reproduce this that we can test on our end?[/quote]

I will try to reproduce the error in a minimal project. You can imagine that this is not so easy, as I only know whether the problem exists after about 34 minutes.

I am also busy with other tasks, so it may well take until the end of next week.

RE: NCS Update to v2.5.2: MPSL Hard Fault on Advertisement

Einar Thorsrud — Tue, 02 Apr 2024 14:53:50 GMT

Hello,

We have started to look into this and have some questions:

Do you use a 32.768 kHz crystal or the LFRC as low frequency clock source?
Is a connection established at any point, or is the device only starting and stopping advertising when this happens?
Do you by any chance use the mpsl_clock API, for instance by calling mpsl_clock_hfclk_request() and/or mpsl_clock_hfclk_release()?
Is it possible for you to share code that reproduce this that we can test on our end?