Getting error NRF_ERROR_RESOURCES when using more then one notification and the Connection crashes

[danny habot]

Hi, 

I am using the MCU on a board with sensors that are sampled in a very high rate (~100Hz). 

I am having issues when trying to send the data with notification on more then one characteristic. 

The information is sent correctly for a short time (~30 seconds) and then the connection get stuck and unresponsive. 

When the connection "crashes" i start getting NRF_ERROR_RESOURCES  from sd_ble_gatts_hvx(*pointer_conn_handle, &hvx_params);

Initially i thought it is because the queue was very small (the default value of 1) however even after changing it to 10 it still does it.

I use the following code to change it: 

ble_cfg_t ble_cfg;
memset(&ble_cfg, 0, sizeof ble_cfg);
ble_cfg.conn_cfg.conn_cfg_tag = APP_BLE_CONN_CFG_TAG;
ble_cfg.conn_cfg.params.gatts_conn_cfg.hvn_tx_queue_size = 10;
err_code = sd_ble_cfg_set(BLE_CONN_CFG_GATTS, &ble_cfg, ram_start);

I have 2 questions:

1. how can i config the system to allow much more bandwidth as i will need several notification simultaneously?

2. how can i avoid crashes in case of overload? i saw this post but couldn't understand how to use it on my project as the data structs are different.

I am using NRF52832 (custom board) with freeRTOS (based on the HRS freeRTOS example) SDK15, and SD 6.0

BR, 

Danny

[Sigurd]

Hi,

danny habot said:

1. how can i config the system to allow much more bandwidth as i will need several notification simultaneously?

Please try to increase the event length(NRF_SDH_BLE_GAP_EVENT_LENGTH). See this post.

danny habot said:

2. how can i avoid crashes in case of overload? i saw this post but couldn't understand how to use it on my project as the data structs are different.

The quick and easy solution is just to wait for the function to return something else than NRF_ERROR_RESOURCES.

err_code = your_send_function / sd_ble_gatts_hvx

if ( (err_code != NRF_ERROR_INVALID_STATE) && (err_code != NRF_ERROR_RESOURCES) &&
	 (err_code != NRF_ERROR_NOT_FOUND) )
{
	APP_ERROR_CHECK(err_code);
}
} while (err_code == NRF_ERROR_RESOURCES);

You could also set a flag when you get NRF_ERROR_RESOURCES, and wait for the BLE_GATTS_EVT_HVN_TX_COMPLETE event to occur, and try again. This is done in the ble_app_att_mtu_throughput example. But try to increase the event_length and do the easy solution first.

Let me know how it goes.

BR,

Sigurd

[danny habot]

Hi Sigurd,

I have increased the NRF_SDH_BLE_GAP_EVENT_LENGTH from 6 to 100 but still no luck i still get NRF_ERROR_RESOURCES after several seconds. 

In addition i tried to add the while loop in order to retry the sd_ble_gatts_hvx it get stuck in the loop and never recover. 

It feels like the resource shortage cause the soft device to stop responding. in addition, i don't know if it is related, but when i connect to the device the sd_ble_gatts_hvx function returns 13313 which i couldn't find the meaning of..

i am sending 200 packets per second which shouldn't be a too large amount. 

[Sigurd]

danny habot said:

but when i connect to the device the sd_ble_gatts_hvx function returns 13313 which i couldn't find the meaning of..

13313 is 0x3401 in hex, and is known as the error code BLE_ERROR_GATTS_SYS_ATTR_MISSING.

As mentioned in this post, it probably occurs because you are trying to send a notification/indication to the client before the client has enabled the CCCD.

See this post for more information. This and this post might also be helpful.

[danny habot]

Hi, 

I am not very concerned with the BLE_ERROR_GATTS_SYS_ATTR_MISSING. 

The more concerning issue is the fact the connection crashes. basically the soft device stop responding and connection events aren't being handled.

I have tried what you suggested but got nothing.

the part of the notifications in my code is:

if(*pointer_conn_handle != BLE_CONN_HANDLE_INVALID)
{
    sd_ble_gatts_value_get(*pointer_conn_handle, Sensor_val_handles.cccd_handle, &cccd_data);
    if(*cccd_data.p_value)
    {
        ble_gatts_hvx_params_t hvx_params;
        memset(&hvx_params, 0, sizeof(hvx_params));
        rslt = 0;
        hvx_params.handle = Sensor_val_handles.value_handle;
        hvx_params.type = BLE_GATT_HVX_NOTIFICATION;
        hvx_params.offset = 0;
        hvx_params.p_len = &value.len;
        hvx_params.p_data = (uint8_t*)data;
        rslt = sd_ble_gatts_hvx(*pointer_conn_handle, &hvx_params);
        if (rslt != 0)
    {
}

When i am getting NRF_ERROR_RESOURCES the soft device get "stuck" and it does not respond anymore. 

this is a major issue and i must solve it. please let me know what else is required in order to solve this. 

thank you, 

[Sigurd]

Hi,

danny habot said:

The information is sent correctly for a short time (~30 seconds) and then the connection get stuck and unresponsive. 

Does it always happen after 30 seconds?

What connection parameters are you using?

danny habot said:

The more concerning issue is the fact the connection crashes. basically the soft device stop responding and connection events aren't being handled.

What happens when the "connection get stuck" ? Does the central/peer disconnect? Any error-code on the other side of the link?

danny habot said:

        if (rslt != 0)
    {
}

What are you doing when rslt != 0 ?

[danny habot]

Hi, 

1. no, it is not always 30 seconds... it changes from run to run. i couldn't find a pattern...

2. my connection parameters are connection interval of 7.5ms, slave latency of 2 and Connection supervisory timeout of 4 seconds.

3. That is the weird thing. the connection is just unresponsive. it does not get disconnected, and the device has not crashed as it still response to inputs correctly. i do not get any error from the central side. the only errors i get is if i try to do any new action and it get timeout. when it happen i get the NRF_ERROR_RESOURCES on the relevant characteristics.

4. i should have deleted it. when rslt !=0 i print the return value.

[Sigurd]

danny habot said:

the only errors i get is if i try to do any new action and it get timeout

What do mean by action? Is it the connection that timeout, or do mean that you get the return code NRF_ERROR_TIMEOUT from a function?

Are you getting the event BLE_GATTS_EVT_HVN_TX_COMPLETE ?

If you could capture a sniffer trace, it would help us to see what is happening on-air.

You could also test the S132 v6.1.0, and see if you see any improvements with that version.

[Sigurd]

Hi again,

I talked with my colleague Bjørn, and we are waiting for the sniffer trace. We believe the root cause of this issue is caused by FreeRTOS. We will try to reproduce and verify this.

[danny habot]

Hi Sigurd, 

I sent to Bjørn the sniffer trace, please let me know if you need anything else from me. 

[bjorn-spockeli]

HI Danny, I am still looking into this and have yet to find the root cause yet. 

From the sniffer trace I see that the device is still sending empty packets, i.e. maintaining the link, hence we can rule out a SoftDevice issue as this confirms that the SoftDevice is still running properly. WHat I suspect is that there is something in our FreeRTOS port that results in the application locking up, resulting in sd_ble_gatts_hvx not being called. 

I have been using the ble_app_hrs_freertos example for testing, using a connection interval between 7.5ms and 12.5ms, and sending notifications with a 5ms interval. I have been using a hvn_tx_queue_size of 6 and a connection event length of 7.5ms(6*1.25 ms). 

I have seen significant improvement if the logger task is not running, i.e.  disable logging by setting NRF_LOG_ENABLED to 0 in the sdk_config.h file.  I still see the issue occurring after 10+ minutes, but it is far better than after <60s with logging enabled. 

Do you have logging enabled when you see this issue? If so could you try to disable the logging module,  perform another sniffer trace capture and see if this results in the issue occurring later than with logging enabled?

Also, I see that you have set hvn_tx_queue_size to 10, do you still call  sd_ble_gatts_hvx in a loop or do you keep track of how many notifications that have been queued and then only call sd_ble_gatts_hvx if you have gotten a BLE_GATTS_EVT_HVN_TX_COMPLETE  event indicating that you can queue a new notification? 

[danny habot]

Hi bjorn, 

Thanks for the update. 

regarding your questions:

1. we also based our project on the ble_app_hrs_freertos  so it is good to hear that the issue is not with the code that we added but with the original example.

2. the logs are disable in our project as we use ITM for printf instead of using the NRF_LOG module. so this does not help us unfortunately. However it means that it might happen when the system is more "loaded" with tasks as our project has many periodic activities that might take the resources that the log module take in your system. 

3. I am not checking if I got the event BLE_GATTS_EVT_HVN_TX_COMPLETE  however i did try to do it in my debugging process without success. can you send me a code snippet with this condition so that i would add it as a precaution? my current code for notification is:

case SENSOR_VAL_UUID:
if (len == SENSOR_VAL_SIZE)
{
//check if notification/indication is configured and send the data if it does.
if(*pointer_conn_handle != BLE_CONN_HANDLE_INVALID)
{
ble_gatts_hvx_params_t hvx_params;
memset(&hvx_params, 0, sizeof(hvx_params));

hvx_params.handle = Sensor_val_handles.value_handle;
hvx_params.type = BLE_GATT_HVX_NOTIFICATION;
hvx_params.offset = 0;
hvx_params.p_len = &value.len;
hvx_params.p_data = (uint8_t*)data;

sd_ble_gatts_hvx(*pointer_conn_handle, &hvx_params);
}
sd_ble_gatts_value_set(*pointer_conn_handle, Sensor_val_handles.value_handle, &value);
}

 

As you can see in previous posts i tried other precautions as well but to no avail so i am a little skeptic.

 

[bjorn-spockeli]

Hi Danny, 

how many tasks do you create in your project and which priority have you given them? I am starting to think that this is not a bug, but rather a consquence of how the FreeRTOS scheduler is configured and which priority the different tasks are given. I am basing this on another post, here is the link, on DevZone. 

FreeRTOS uses prioritised preemptive scheduling with time slicing. That means the RTOS scheduler will always run the highest priority task that is in the Ready state, and will switch between tasks of equal priority on every RTOS tick interrupt. If time slicing is disabled then the RTOS scheduler will still run the highest priority task that is in the Ready state, but will not switch between tasks of equal priority just because a tick interrupt has occurred.

Time slicing appears to be disabled in the ble_app_hrs_freertos example, so could you try to enable time slicing by setting the following define

#define configUSE_TIME_SLICING 1

in the /external/freertos/config folder? I have enabled it in my code and i have ran 30min+ traces and have not seen the issue occuring. 

Best regards

Bjørn