Getting error NRF_ERROR_RESOURCES when using more then one notification and the Connection crashes

[danny habot]

Hi, 

I am using the MCU on a board with sensors that are sampled in a very high rate (~100Hz). 

I am having issues when trying to send the data with notification on more then one characteristic. 

The information is sent correctly for a short time (~30 seconds) and then the connection get stuck and unresponsive. 

When the connection "crashes" i start getting NRF_ERROR_RESOURCES  from sd_ble_gatts_hvx(*pointer_conn_handle, &hvx_params);

Initially i thought it is because the queue was very small (the default value of 1) however even after changing it to 10 it still does it.

I use the following code to change it: 

ble_cfg_t ble_cfg;
memset(&ble_cfg, 0, sizeof ble_cfg);
ble_cfg.conn_cfg.conn_cfg_tag = APP_BLE_CONN_CFG_TAG;
ble_cfg.conn_cfg.params.gatts_conn_cfg.hvn_tx_queue_size = 10;
err_code = sd_ble_cfg_set(BLE_CONN_CFG_GATTS, &ble_cfg, ram_start);

I have 2 questions:

1. how can i config the system to allow much more bandwidth as i will need several notification simultaneously?

2. how can i avoid crashes in case of overload? i saw this post but couldn't understand how to use it on my project as the data structs are different.

I am using NRF52832 (custom board) with freeRTOS (based on the HRS freeRTOS example) SDK15, and SD 6.0

BR, 

Danny

[Sigurd]

Hi,

danny habot said:

1. how can i config the system to allow much more bandwidth as i will need several notification simultaneously?

Please try to increase the event length(NRF_SDH_BLE_GAP_EVENT_LENGTH). See this post.

danny habot said:

2. how can i avoid crashes in case of overload? i saw this post but couldn't understand how to use it on my project as the data structs are different.

The quick and easy solution is just to wait for the function to return something else than NRF_ERROR_RESOURCES.

err_code = your_send_function / sd_ble_gatts_hvx

if ( (err_code != NRF_ERROR_INVALID_STATE) && (err_code != NRF_ERROR_RESOURCES) &&
	 (err_code != NRF_ERROR_NOT_FOUND) )
{
	APP_ERROR_CHECK(err_code);
}
} while (err_code == NRF_ERROR_RESOURCES);

You could also set a flag when you get NRF_ERROR_RESOURCES, and wait for the BLE_GATTS_EVT_HVN_TX_COMPLETE event to occur, and try again. This is done in the ble_app_att_mtu_throughput example. But try to increase the event_length and do the easy solution first.

Let me know how it goes.

BR,

Sigurd

[danny habot]

Hi Sigurd,

I have increased the NRF_SDH_BLE_GAP_EVENT_LENGTH from 6 to 100 but still no luck i still get NRF_ERROR_RESOURCES after several seconds. 

In addition i tried to add the while loop in order to retry the sd_ble_gatts_hvx it get stuck in the loop and never recover. 

It feels like the resource shortage cause the soft device to stop responding. in addition, i don't know if it is related, but when i connect to the device the sd_ble_gatts_hvx function returns 13313 which i couldn't find the meaning of..

i am sending 200 packets per second which shouldn't be a too large amount. 

[Sigurd]

danny habot said:

the only errors i get is if i try to do any new action and it get timeout

What do mean by action? Is it the connection that timeout, or do mean that you get the return code NRF_ERROR_TIMEOUT from a function?

Are you getting the event BLE_GATTS_EVT_HVN_TX_COMPLETE ?

If you could capture a sniffer trace, it would help us to see what is happening on-air.

You could also test the S132 v6.1.0, and see if you see any improvements with that version.

[Sigurd]

Hi again,

I talked with my colleague Bjørn, and we are waiting for the sniffer trace. We believe the root cause of this issue is caused by FreeRTOS. We will try to reproduce and verify this.

[danny habot]

Hi Sigurd, 

I sent to Bjørn the sniffer trace, please let me know if you need anything else from me. 

[bjorn-spockeli]

HI Danny, I am still looking into this and have yet to find the root cause yet. 

From the sniffer trace I see that the device is still sending empty packets, i.e. maintaining the link, hence we can rule out a SoftDevice issue as this confirms that the SoftDevice is still running properly. WHat I suspect is that there is something in our FreeRTOS port that results in the application locking up, resulting in sd_ble_gatts_hvx not being called. 

I have been using the ble_app_hrs_freertos example for testing, using a connection interval between 7.5ms and 12.5ms, and sending notifications with a 5ms interval. I have been using a hvn_tx_queue_size of 6 and a connection event length of 7.5ms(6*1.25 ms). 

I have seen significant improvement if the logger task is not running, i.e.  disable logging by setting NRF_LOG_ENABLED to 0 in the sdk_config.h file.  I still see the issue occurring after 10+ minutes, but it is far better than after <60s with logging enabled. 

Do you have logging enabled when you see this issue? If so could you try to disable the logging module,  perform another sniffer trace capture and see if this results in the issue occurring later than with logging enabled?

Also, I see that you have set hvn_tx_queue_size to 10, do you still call  sd_ble_gatts_hvx in a loop or do you keep track of how many notifications that have been queued and then only call sd_ble_gatts_hvx if you have gotten a BLE_GATTS_EVT_HVN_TX_COMPLETE  event indicating that you can queue a new notification? 

[danny habot]

Hi bjorn, 

Thanks for the update. 

regarding your questions:

1. we also based our project on the ble_app_hrs_freertos  so it is good to hear that the issue is not with the code that we added but with the original example.

2. the logs are disable in our project as we use ITM for printf instead of using the NRF_LOG module. so this does not help us unfortunately. However it means that it might happen when the system is more "loaded" with tasks as our project has many periodic activities that might take the resources that the log module take in your system. 

3. I am not checking if I got the event BLE_GATTS_EVT_HVN_TX_COMPLETE  however i did try to do it in my debugging process without success. can you send me a code snippet with this condition so that i would add it as a precaution? my current code for notification is:

case SENSOR_VAL_UUID:
if (len == SENSOR_VAL_SIZE)
{
//check if notification/indication is configured and send the data if it does.
if(*pointer_conn_handle != BLE_CONN_HANDLE_INVALID)
{
ble_gatts_hvx_params_t hvx_params;
memset(&hvx_params, 0, sizeof(hvx_params));

hvx_params.handle = Sensor_val_handles.value_handle;
hvx_params.type = BLE_GATT_HVX_NOTIFICATION;
hvx_params.offset = 0;
hvx_params.p_len = &value.len;
hvx_params.p_data = (uint8_t*)data;

sd_ble_gatts_hvx(*pointer_conn_handle, &hvx_params);
}
sd_ble_gatts_value_set(*pointer_conn_handle, Sensor_val_handles.value_handle, &value);
}

 

As you can see in previous posts i tried other precautions as well but to no avail so i am a little skeptic.

 

[bjorn-spockeli]

Hi Danny, 

how many tasks do you create in your project and which priority have you given them? I am starting to think that this is not a bug, but rather a consquence of how the FreeRTOS scheduler is configured and which priority the different tasks are given. I am basing this on another post, here is the link, on DevZone. 

FreeRTOS uses prioritised preemptive scheduling with time slicing. That means the RTOS scheduler will always run the highest priority task that is in the Ready state, and will switch between tasks of equal priority on every RTOS tick interrupt. If time slicing is disabled then the RTOS scheduler will still run the highest priority task that is in the Ready state, but will not switch between tasks of equal priority just because a tick interrupt has occurred.

Time slicing appears to be disabled in the ble_app_hrs_freertos example, so could you try to enable time slicing by setting the following define

#define configUSE_TIME_SLICING 1

in the /external/freertos/config folder? I have enabled it in my code and i have ran 30min+ traces and have not seen the issue occuring. 

Best regards

Bjørn 

[danny habot]

Hi bjorn, 

I also think that it is somewhat related to the FreeRTOS configuration however the task handling was one of the first things that i checked. 

My task run with priority 2 which shouldn't starve the SD. in addition, because the SD is not sending anything new it shouldn't starve my task as well (and it doesn't because the UI is still working correctly) . 

I tried to config it to use time slicing but it has no effect on the behavior i see. Verified it again now. 

Did you change anything else in the example? This is very weird because we have based our project on it. 

How can we continue? would it help if i send you a copy of my project with instructions how to reproduce? 

[bjorn-spockeli]

Hi Danny, can you check which priority the timer task(s) are given in the FreeRTOSConfig.h, i.e. configTIMER_TASK_PRIORITY, is it set to 2 as well?

I tried increasing the priority of the main (ble)task from 2 to 3, i.e. 

xTaskCreate(softdevice_task,
"BLE",
NRF_BLE_FREERTOS_SDH_TASK_STACK,
p_context,
3,
&m_softdevice_task);

and that seemed to impreove things on my end. Can you try that?

[danny habot]

Hi Bjorn, 

I have added more priority levels so now i have one for each task with the softdevice on the highest. It seems to improve this. I am still running tests and if all goes well i will close the ticket. 

Thanks for all the help and finger crossed i will have good news soon