HARD FAULT during xTaskResumeAll after ending a DFU session and disabling the Softdevice

RE: HARD FAULT during xTaskResumeAll after ending a DFU session and disabling the Softdevice

Susheel Nuguru — Fri, 27 May 2022 11:49:23 GMT

[quote user="Rafalino"] my workaround && p_instance->timer_id seems to work but I don't think this is the proper[/quote]

Good find, and why do you think your workaround is not proper?

I think your workaround is very proper to check if the timer_id is valid before attempting to access the API of that timer through that id.

RE: HARD FAULT during xTaskResumeAll after ending a DFU session and disabling the Softdevice

Rafalino — Thu, 26 May 2022 22:01:56 GMT

Hello Susheel,

I have put a break point at timer.c inside xTimerGenericCommand

at the point we call

xReturn = xQueueSendToBackFromISR( xTimerQueue, &xMessage, pxHigherPriorityTaskWoken );

Then I was able to look at the call stack.

My finding are :

We perform nrf_ble_scan_start and when it returns with BLE_GAP_EVT_DISCONNECTED we call again to nrf_ble_scan_start (should we call some cleaning function before ?)

Then, what I see from the call stack is:

ble_conn_params.c has on_disconnect event handler

conn_handle is 0 in my case and from

ble_conn_params_instance_t * p_instance  = instance_get(conn_handle);

we get p_instance but it has everything set to 0 including the timer id.

so although (p_instance != NULL) is true, we get a garbage instance->timer_id and

then app_timer_stop(p_instance->timer_id) leads to hard fault.

my workaround && p_instance->timer_id seems to work but I don't think this is the proper

solution.

What do you say?

static void on_disconnect(ble_evt_t const * p_ble_evt)
{
ret_code_t err_code;
uint16_t conn_handle = p_ble_evt->evt.gap_evt.conn_handle;
ble_conn_params_instance_t * p_instance = instance_get(conn_handle);

if (p_instance != NULL && p_instance->timer_id) /* Workaround */
{
// Stop timer if running
err_code = app_timer_stop(p_instance->timer_id);
if (err_code != NRF_SUCCESS)
{
send_error_evt(err_code);
}

instance_free(p_instance);
}
}

RE: HARD FAULT during xTaskResumeAll after ending a DFU session and disabling the Softdevice

Susheel Nuguru — Wed, 25 May 2022 17:53:44 GMT

If you already have the XTAL LFCLK, why use RC? you can change NRF_SDH_CLOCK_LF_SRC to 1.

If this does not fix, then I am starting to believe that it probably is not related to softdevice.
We need to focus back on the tasks at hand. You already confirmed that it is not a stack overflow,

If we can trust the hardfault stack frame which suggests that it is a timer task, So start with commenting out creation of timers one by one until xTaskResumeAll does not crash. If all the timer creation code is commented out and no application timer are left and still able to trigger a hardfault, then we can forget about the timer task and assume that hardfault stack frame is useless in terms of info it holds to debug.

move on by starting to comment out creating of tasks one by one until xTaskResumeAll does not create this hardfault, we would then atleast know which task resumption is creating this hardfault.

RE: HARD FAULT during xTaskResumeAll after ending a DFU session and disabling the Softdevice

Rafalino — Wed, 25 May 2022 14:47:22 GMT

Hello Susheel,

NRF_SDH_CLOCK_LF_SRC is indeed set to 0.

I have added delay of 200ms before xTaskResumeAll but it didn't help.

/* sd_is_enabled is changed to false once we know that the SD is disabled (on brg_dfu_state_obs handler)*/
while(sd_is_enabled)
{
NRF_LOG_PROCESS();
wlt_utils_watchdog_feed();
}

nrf_delay_ms(200);

if( !xTaskResumeAll () )
{
taskYIELD ();
}

Any more suggestions? 🤔

RE: HARD FAULT during xTaskResumeAll after ending a DFU session and disabling the Softdevice

Susheel Nuguru — Wed, 25 May 2022 12:58:11 GMT

What kind of clock the softdevice thinks it is using depends on the first parameter you give to sd_softdevice_enable. If you are using nrf_sdh_enable_request then the LFCLK the softdevice uses on the configuration you set with NRF_SDH_CLOCK_LF_SRC in the sdk_config.h file. If this is set to 0, then you are making the softdevice think that it is using the internal RC which makes me think that the calibration might

The while wait condition on sd_is_enabled will still not help if the issue is with the extra delay needed with clock calibration (will only be possible if you have set NRF_SDH_CLOCK_LF_SRC to 0.

If none of that are relavent, then try to put about 80-100ms of extra delay just before xTaskResumeAll to confirm to check if this issue is with clock calibration or not. Because the clock calibration should not take more than 100ms worst case before it disables the softdevice.

RE: HARD FAULT during xTaskResumeAll after ending a DFU session and disabling the Softdevice

Rafalino — Wed, 25 May 2022 11:41:33 GMT

[quote userid="6207" url="~/f/nordic-q-a/88015/hard-fault-during-xtaskresumeall-after-ending-a-dfu-session-and-disabling-the-softdevice/369491#369491"]Can you show me how you are waiting to know that the softdevice is disabled?[/quote]

This is the part of disabling the SD:

err_code = nrf_sdh_disable_request();
APP_ERROR_CHECK(err_code);

/* sd_is_enabled is changed to false once we know that the SD is disabled (on brg_dfu_state_obs handler)*/
while(sd_is_enabled)
{
NRF_LOG_PROCESS();
wlt_utils_watchdog_feed();
}

if( !xTaskResumeAll () )
{
taskYIELD ();
}

And here is the event handler :

static void brg_dfu_state_obs(nrf_sdh_state_evt_t state, void * p_context)
{
UNUSED_PARAMETER(p_context);

NRF_LOG_INFO("%s state = %d", __FUNCTION__, state);
if (state == NRF_SDH_EVT_STATE_DISABLED)
{
sd_is_enabled = false;
}

}

[quote userid="6207" url="~/f/nordic-q-a/88015/hard-fault-during-xtaskresumeall-after-ending-a-dfu-session-and-disabling-the-softdevice/369491#369491"]Are you using XTAL HFCLK and/or LFCLK? [/quote]

We are calling the following in our application before we even start the Softdevice:

nrf_drv_clock_hfclk_request(NULL);
nrf_drv_clock_lfclk_request(NULL);

Any recommendation ?

RE: HARD FAULT during xTaskResumeAll after ending a DFU session and disabling the Softdevice

Susheel Nuguru — Wed, 25 May 2022 10:49:21 GMT

Hmm, Interesting observation.

[quote user=""]When we finish the DFU process we call nrf_sdh_disable_request and wait until we know that the Softdevice is disabled.[/quote]

Can you show me how you are waiting to know that the softdevice is disabled?

In the examples of FreeRTOS we deliver with our SDK, the softdevice events are pulled in a task called "softdevice_task".
I am assuming that you are not using this task since you suspend all tasks just before enabling the softdevice. Which also makes me assume that all the activity of the softdevice and BLE is happening with a baremetal kind of event handling.

[quote user="Rafalino"]

It feels as if there is an internal connection timeout at the Softdevice that is triggered once there is a connection that we somehow don't clean when disable the Softdevice

Does that make sense ?

[/quote]

It seems like for some reason there is a need for larger delay that the wait you have in your application to know that the softdevice is disabled. Are you using XTAL HFCLK and/or LFCLK? or are you only using internal RC for the LFCLK? We have noticed that if you are using internal RC clocks and disable the softdevice in midst of internal clock calibration, then the softdevice disable function might return normally but takes longer than expected to actually disable the softdevice since it is busy calibrating internal clocks before actually servicing the disable softdevice request.

RE: HARD FAULT during xTaskResumeAll after ending a DFU session and disabling the Softdevice

Rafalino — Wed, 25 May 2022 09:20:56 GMT

Hey Susheel,

We are using nrf_libuarte but I have un-init that interface and disabled my UART task before enabling the Softdevice so I don't think that this is the case.

As for tickless disable - we are using configUSE_TICKLESS_IDLE = 0 by default.

What I see in my testing is that If suspend all my application tasks and then suspend the scheduler (via vTaskSuspendAll ), enable Softdevice and start BLE scanning before actually establishing BLE connection with the other chip, and at this point power off the remote chip so that the ble connection will fail, and then call disable Softdevice and and resume scheduler - I do NOT see the hard fault and I am able to resume my application tasks.

It feels as if there is an internal connection timeout at the Softdevice that is triggered once there is a connection that we somehow don't clean when disable the Softdevice

Does that make sense ?

Rafalino

RE: HARD FAULT during xTaskResumeAll after ending a DFU session and disabling the Softdevice

Susheel Nuguru — Tue, 24 May 2022 11:39:35 GMT

The issue is not with using the UARTE but using the libUARTE. If you are not using libuarte then this is not the same issue.

But if you are using LibUARTE, then there is a bug in the way nrf_libuarte where a macro to initialize nrf_libuarte_async_t structure, particularly p_app_timer_t is done incorrectly when using freertos (since app_timer_freertos.c uses different timer casting than the app_timer.c). The dirty hack was to set .end_val in the below lstruct to 0.

static app_timer_t xLibUarteCOM0_app_timer_data =
{
.end_val = 0xFFFFFFFFFFFFFFFFULL,
};

This is just to make the freertos app_timer initialization pass and then properly initialize these structures later.

If none of the above works, then I am thinking that this could be related to Tickless mode (if you have enabled it in FreeRTOSConfig.h file?)

If the chip is waking up from sleep at the time of you resuming all tasks, then it might be some how related to the port specific changes we did on the tick interrupt handling. Can you disable tickless mode to see if you can still reproduce it?

So far based on your investigation

1) You confirmed that there are no stack overflows caught when enabling stack overflow checks and increasing the timer stack.

2) You applied or will apply the work around given in the link if you are using nrf_libuarte library.

3) Try to see if tickless disable have the same effect.

RE: HARD FAULT during xTaskResumeAll after ending a DFU session and disabling the Softdevice

Rafalino — Tue, 24 May 2022 11:21:58 GMT

Hello Susheel,

I have added the code that you suggested but could not see much on the call stack :

As for the UARTE, we do use UARTE and I would like to add the workaround that you suggested but could not fully understand where to set p_xLibUarteCOM0_app_timer_data->end_val = 0x0ULL;

Can you clarify ?

RE: HARD FAULT during xTaskResumeAll after ending a DFU session and disabling the Softdevice

Susheel Nuguru — Tue, 24 May 2022 10:50:07 GMT

Then it might be possible that we are overlooking into the memory corruption direction. If this is not a memory corruption by stack overflow, then the application is somehow passing the wrong timerID.

You can write a small code snippet like below in prvProcessReceivedCommands just before uxListRemove

if (pxTimer->pvTimerID == 0x0x1E200000)
{
    static volatile uint32_t counter = 0;
    counter++;     // <-- Put a breakpoint at this line
}

Compile your code, flash and start the code in the debugger. Put the breakpoint at the "counter++" and run the application in an attempt to trigger the hardfault.

The debugger should halt at the breakpoint and now your function call stack should allow you to browse through the functions that lead to this breakpoint. Try to understand the context of how this value has been passed to pvTimerID.

Please note that there is known bug in the libuarte library when using FreeRTOS as the macros use to initialize libuarte instances initialize the app_timer_freertos instances wrongly (after an incompatible casting). Please check if you are affected by this.

RE: HARD FAULT during xTaskResumeAll after ending a DFU session and disabling the Softdevice

Rafalino — Tue, 24 May 2022 10:37:34 GMT

Hello Susheel,

I have increased configTIMER_TASK_STACK_DEPTH from 80 to 256 but still got the same hard fault .😒

I then set

#define configCHECK_FOR_STACK_OVERFLOW 1

and added

void vApplicationStackOverflowHook( xTaskHandle pxTask, signed char *pcTaskName )
{
while (1)
{
NRF_LOG_INFO(" %s %s", __FUNCTION__, pcTaskName);
}

}

But it didn't jump into the above hook and I got the same hard fault .

What are we missing ? 🤔

RE: HARD FAULT during xTaskResumeAll after ending a DFU session and disabling the Softdevice

Susheel Nuguru — Tue, 24 May 2022 08:21:45 GMT

[quote user="Rafalino"]app: xMessage.xMessageID = 8, pxTimer=0x0x00000A81 , pxTimer->pvTimerID=0x0x1E200000[/quote]

This address is not a valid RAM location. So I strongly suspecting a memory corruption. Since this is happening the Timer task context, I am assuming that your timer stack size is too small.

Try increasing configTIMER_TASK_STACK_DEPTH in your FreeRTOSConfig.h file to a bigger value for testing.

Also, Like I suggested in my previous reply. Enable stack overflow checks.

[quote user="Aryan"]You can enable stack overflow check to see if there are any stack overflows[/quote]

RE: HARD FAULT during xTaskResumeAll after ending a DFU session and disabling the Softdevice

Rafalino — Tue, 24 May 2022 07:21:14 GMT

Hello Sushell,

I have added xMessageID to the prints at prvProcessReceivedCommands

just before ( void ) uxListRemove( &( pxTimer->xTimerListItem ) );

app: xMessage.xMessageID = 6, pxTimer=0x0x2000CE98 , pxTimer->pvTimerID=0x0x2002D2D8
app: xMessage.xMessageID = 8, pxTimer=0x0x00000A81 , pxTimer->pvTimerID=0x0x1E200000

The last two messages are coming from ISR (message ID 6 and 8)

The problematic message is : #define tmrCOMMAND_STOP_FROM_ISR ( ( BaseType_t ) 8 )

Is there a chance that the the Softdevice has a timer ISR that triggers it ?

(Even though the Softdevice is already disabled at that stage).

Thanks in advance,

Rafalino

RE: HARD FAULT during xTaskResumeAll after ending a DFU session and disabling the Softdevice

Susheel Nuguru — Tue, 24 May 2022 05:29:05 GMT

Either it is a memory corruption or it is an attempt to remove an already removed timer.
You can enable stack overflow check to see if there are any stack overflows happening for the first suspicion.

You need to add more logs to find out if the last one is an attempt to delete an already deleted timer.

RE: HARD FAULT during xTaskResumeAll after ending a DFU session and disabling the Softdevice

Rafalino — Mon, 23 May 2022 12:48:45 GMT

Hello Sushel,

I have added prints at timers.c :

NRF_LOG_INFO(" pxNewTimer=0x%p , pxTimer->pvTimerID=0x%p ", pxNewTimer, pxNewTimer->pvTimerID);

at the end of xTimerCreate

NRF_LOG_INFO(" pxTimer=0x%p , pxTimer->pvTimerID=0x%p ", pxTimer, pxTimer->pvTimerID);

at prvProcessReceivedCommands just before ( void ) uxListRemove( &( pxTimer->xTimerListItem ) );

This is what we see:

<info> app: pxNewTimer=0x0x2000CE40 , pxTimer->pvTimerID=0x0x200169C8
<info> app: pxNewTimer=0x0x2000D0C0 , pxTimer->pvTimerID=0x0x200168F0
<info> app: pxNewTimer=0x0x2000D0F0 , pxTimer->pvTimerID=0x0x2002D310
<info> app: pxNewTimer=0x0x2000D120 , pxTimer->pvTimerID=0x0x2002D330

<info> app: pxTimer=0x0x2000D120 , pxTimer->pvTimerID=0x0x2002D330
<info> app: pxTimer=0x0x2000D120 , pxTimer->pvTimerID=0x0x2002D330
<info> app: pxTimer=0x0x2000D0F0 , pxTimer->pvTimerID=0x0x2002D310

Just before the hard fault :
<info> app: pxTimer=0x0x00000A81 , pxTimer->pvTimerID=0x0x1E200000

We see that all the timers are at address 0x0x2000DXXX

and the timer that we need to remove is in weird address 0x0x00000A81

What can cause this corruption ?

RE: HARD FAULT during xTaskResumeAll after ending a DFU session and disabling the Softdevice

Rafalino — Mon, 23 May 2022 12:03:33 GMT

Hello Susheel,

I am checking return value using APP_ERROR_CHECK(ERR_CODE)

From : ...nRF5_SDK\nRF5_SDK_17.0.2_d674dde\components\libraries\util\app_error.h

But I don't see anything special.

RE: HARD FAULT during xTaskResumeAll after ending a DFU session and disabling the Softdevice

Susheel Nuguru — Mon, 23 May 2022 10:19:12 GMT

Please make sure that you are handling all the return values correctly from the API especially app_timer_stop

RE: HARD FAULT during xTaskResumeAll after ending a DFU session and disabling the Softdevice

Susheel Nuguru — Mon, 23 May 2022 10:15:58 GMT

Hi Ori,

Thanks for your patience.
It looks to me that the pxTimer might be pointing to an invalid instance of timer here in the uxListRemove.
Can you put some logs in the timer to print the instances of the pxTimer.

This might not be directly related to DFU but might be related to the way you are suspending and resuming all tasks. My best guess right now, is that the timer instance or the list instance within the timer is somehow invalid or corrupted. If that is the case, we need to find out how and when that happened.

RE: HARD FAULT during xTaskResumeAll after ending a DFU session and disabling the Softdevice

Rafalino — Fri, 20 May 2022 10:58:49 GMT

Hello Edvin,

As you can see from the screen shot that I attached (that has the Call stack):

it is coming from list.c and it was called from timer.c :

/* The timer is in a list, remove it. */
( void ) uxListRemove( &( pxTimer->xTimerListItem ) );

I suspect some timer was not stopped/cleared by the Softdevice but I can't figure out which one.

Rafalino

RE: HARD FAULT during xTaskResumeAll after ending a DFU session and disabling the Softdevice

Edvin — Fri, 20 May 2022 10:21:53 GMT

Ah, I see. Thank you for clarifying

Can you reproduce it once more (reproduce it with your latest build, so that the addresses from the log are correct), and see what is found at the address that the log is claiming contains the hardfault. In your original post, that would be 0x00029350.

Then, make sure that the tool arm-none-eabi-addr2line is in the environment path on your computer. In my case, it is found in C:\Program Files (x86)\GNU Tools ARM Embedded\4.9 2015q3\bin\arm-none-eabi-addr2line.exe, so I added "C:\Program Files (x86)\GNU Tools ARM Embedded\4.9 2015q3\bin\" to my environment path. To check whether it is in your environment path or not, you can use the command "where arm-none-eabi-addr2line". If it replies with the destination, that is fine. If it is not found, you need to install it and/or add it to your path. It is part of the GNU ARM embedded toolchain.

Then, go to your build folder for your application. I see that you are using Segger Embedded Studio, so that would be the "Output" folder, and locate the .elf file. Using the ble_app_uart example for demonstration, you can use this command to find out what function that is located at the Hardfault address:

arm-none-eabi-addr2line -e Output\Release\Exe\ble_app_uart_pca10056_s140.elf 0x00029350

Does it point to list.c: 193? If so, can you try to set a breakpoint there to see what happens right before the hardfault?

RE: HARD FAULT during xTaskResumeAll after ending a DFU session and disabling the Softdevice

Rafalino — Thu, 19 May 2022 19:25:53 GMT

Hi Edvin,

Let me clarify 🙂

Our application uses the chip's RADIO to send BLE beacons at times that we want to control so during that part of the application we do not enable the Softdevice.

Since we have several chips in our setup , we wanted to use Nordic's DFU service to upgrade one chip from another one over the air.

Let's say chip A has version 3 and chip B has version 2.

So we send a command from chip A to chip B to move to bootloader (a special BLE beacon).

Then chip B reboots and stays at bootloader while chip A disables our primary application (by suspending most of its freeRTOS tasks and also the scheduler itself by calling vTaskSuspendAll ) and then enables the Softdevice in order to send application version 3 OTA to chip B.

Up until this part, everything works smoothly and we are able to send all the application over to chip B.

The problem occurs on chip A after we finish the OTA and we disable the Softdevice , resume our tasks and call xtaskresumeall to resume the scheduler.

During xtaskresumeall we get Hard fault :

Do you have any suggestions what can cause that ?

RE: HARD FAULT during xTaskResumeAll after ending a DFU session and disabling the Softdevice

Edvin — Thu, 19 May 2022 12:47:00 GMT

Oh, sorry. I thought you said you were using the softdevice in your application, but I see that you are not using the softdevice in the application. Only the Radio.

So your DFU procedure: When this issue occurs, is it after a successful DFU? Or was the DFU aborted? What bootloader are you using? Background DFU or "normal" DFU (where the transport happens in the bootloader)? Does the error happen after you reset the device as well, or only directly after a DFU?

BR,
Edvin

RE: HARD FAULT during xTaskResumeAll after ending a DFU session and disabling the Softdevice

Rafalino — Wed, 18 May 2022 13:24:51 GMT

Hello Edvin,

We did not try enabling the Softdevice after we disabled it.

Our Application uses the RADIO all the time so we can't have the Softdevice running while we run our application.

We make sure that the Softdevice is disabled by checking the Softdevice state using NRF_SDH_STATE_OBSERVER.

Rafalino.

RE: HARD FAULT during xTaskResumeAll after ending a DFU session and disabling the Softdevice

Edvin — Wed, 18 May 2022 13:14:15 GMT

Hello Rafalino,

Did you try enabling the softdevice again before you resume the rest of the threads? Does that change anything?

Best regards,

Edvin