Random BLE Disconnects / NRF52832 stops responding to Central

RE: Random BLE Disconnects / NRF52832 stops responding to Central

Hung Bui — Mon, 20 Feb 2023 09:35:27 GMT

Hi Madhuru,
Please create a new ticket for your issue. Please try using the sniffer to get more insight of the connection and send us the trace so we can analyze.

RE: Random BLE Disconnects / NRF52832 stops responding to Central

madhurubharath — Mon, 20 Feb 2023 05:16:21 GMT

i am facing same problem after connecting its disconnecting automatically from central device ,i tried simple example the thing i noticed is its disconnecting even with simple example but after 1-2 min its disconnecting.

RE: Random BLE Disconnects / NRF52832 stops responding to Central

Hung Bui — Thu, 22 Jul 2021 13:34:56 GMT

Please do the test and let us know the result.
From what I can tell from the log and from the disconnected reason, it suggesting that there could be RF noise or bad radio performance or out of sync oscillator would cause the issue.

RE: Random BLE Disconnects / NRF52832 stops responding to Central

nikolozka — Wed, 21 Jul 2021 13:49:14 GMT

We're using various phones as centrals. As I mentioned earlier, the issue has been quite hard to replicate under test conditions so we're relying on production devices in the field for data collection. a couple of our customers have our internall test app that logs all the data that we make available over BLE.

So central devices that the issue is confirmed to have happened on is a mid-range Samsung Galaxy (not 100% sure on the model), Iphone12, Iphone X, Iphone 11. Most Iphones are running some version of iOS 14 generally up to date. It seems like it is independant of the central though. There is no particular pattern to which central is better at triggering this.

1. Thanks for the ble_app_hrs example suggestion, this would be an interesting test that would rule out other bits of our FW malfunctioning. I might also try stripping out our code to leave only bluetooth related bits. But if the issue persists with ble_app_hrs example I'm assuming this would indicate a hardware / sync issue right?

2. testing on the DK is a bit problematic as we rely on a lot of peripherals in our FW so it would be an extremely stripped down version of our FW, which I suppose would crash only if there is something misconfigured about our connection parameters for instance.

If it is indeed a sync issue, would it be possible to improve this?

We are using internal RC with CTIV=16 TEMP_CTIV=2 & Accuracy=500ppm. would tweaking these parameters yield some improvements?

RE: Random BLE Disconnects / NRF52832 stops responding to Central

Hung Bui — Wed, 21 Jul 2021 13:27:31 GMT

Hi Niko,

Disconnection reason 0x08 would suggest that it could be an issue with the oscillator or the radio hardware. What I can think of is that the peripheral gradually goes out of sync or the radio hardware doesn't work as it should.

What you can do to test is:

1. Try testing a very simple application on the same board. ble_app_hrs for example and let it run and check if disconnection happens or not.

2. Try testing your firmware on a nRF52DK , to verify if it's an hardware issue or not.

Which central device are you using ? Have you tried to test the board with different central for comparison ?

RE: Random BLE Disconnects / NRF52832 stops responding to Central

nikolozka — Tue, 20 Jul 2021 19:47:02 GMT

Trying to keep this discussion alive. The disconnect reason is reported as 0x8 (Timeout) & the sd doesn't assert. The device doesn't do a softreset either

I've verified that the sd_assert will cause a softreset and is being caught in general by doing the following to force an assert:

_disable_irq()
nrf_delay_ms(1000)
_enable_irq

(The softdevice will assert because of timing issues)

As i'm keeping the reset reason & the error_code & line where the error occured when APP_ERROR_CHECK catches something I'm fairly confident that a reset doesn't occur.

would it be possible for the SD to die without issuing a call to app_error_fault_handler() <- this is the handler that nrf_sdh.c has defined in the nrf_sdh_enable_request call, and I have this weak error_handler redefined in main.c

I understand your advice to try debugging this, but it's not really helpfull. as I've tried debugging this from every angle i can imagine.

Please suggest some additional strategy for debugging this issue.

It's extremely frustrating

RE: Random BLE Disconnects / NRF52832 stops responding to Central

nikolozka — Tue, 20 Jul 2021 15:07:52 GMT

That's what I've been trying to do for 4 months already ;(

RE: Random BLE Disconnects / NRF52832 stops responding to Central

Hung Bui — Tue, 20 Jul 2021 15:02:15 GMT

No, in any case calling that shouldn't cause a disconnection. It's pretty common when testing throughput that we simply put an infinite loop of sd_ble_gatts_hvx() calls in the main loop. Please try to debug.

RE: Random BLE Disconnects / NRF52832 stops responding to Central

nikolozka — Tue, 20 Jul 2021 14:44:09 GMT

Thanks for clarifying that NRF_ERROR_RESOURCES is not critical return from sd_ble_gatt_hvx(). I will add exceptions to my error handling to ignore this.

After repeating the test overqueueing notifications indeed doesn't lead to the previously observed disconnects.

Is there another error that sd_ble_gatt_hvx() can return that is critical and would lead to problems if not handled appropriately? if It's not clear from my comment threads the return value of sd_ble_gatt_hvx() was previously completely ignored.

RE: Random BLE Disconnects / NRF52832 stops responding to Central

Hung Bui — Tue, 20 Jul 2021 14:24:50 GMT

Hi again,

Please try to debug and check if it reset or it disconnect.
If it reset please check the assert log.

If it disconnect please check the disconnect reason.

RE: Random BLE Disconnects / NRF52832 stops responding to Central

nikolozka — Tue, 20 Jul 2021 14:14:44 GMT

I managed to recreate the observed behavior by queueing notifications in a 5msec timer. The sniffer trace was exactly the same, the peripheral would stop responding for n=slave_latency amount of packets after which the connection was destroyed without an assertion / softreset.

RE: Random BLE Disconnects / NRF52832 stops responding to Central

Hung Bui — Tue, 20 Jul 2021 14:08:41 GMT

Hi Nikolozka,

I don't think over queueing the notification can cause such issue. The softdevice will just ignore (return NRF_ERROR_RESOURCES) then the queue is full.

From the sniffer trace it didn't suggest that the softdevice was under much stress.
I can see that the Slave sometimes stopped responding for a few packets, I assume it's the Slave Latency setting you chose ?

Have you tried to debug the peripheral and check does it has any assertion ? Or check what's the disconnection reason is when it disconnected (BLE_GAP_EVT_DISCONNECTED) ? I suspect that it got assertion and reset for some reason.

Have you tried to test the same application on a nRF52 DK ? (so that we can rule out any issue with the hardware) ?

Modifying BLE_GATTS_HVN_TX_QUEUE_SIZE_DEFAULT, wont make it transfer more data. You can try to increase NRF_SDH_BLE_GAP_EVENT_LENGTH but by default it's 320 and should cover the entire connection interval that you have.

RE: Random BLE Disconnects / NRF52832 stops responding to Central

nikolozka — Tue, 20 Jul 2021 12:08:22 GMT

While writing this post the following has occured to me,

[quote userid="95197" url="~/f/nordic-q-a/77658/random-ble-disconnects-nrf52832-stops-responding-to-central"]The firmware isn't going through a softreset when this issue occurs as we have a custom app_error check that log's errors to ram (unless SD asserts silently somewhere)[/quote]

I tried to go over the entire codebase (which I have inheritted in this somewhat broken state) to find if there were any instances of SoftDevice calls where the return values were not handled properly, And indeed, I deiscovered that there were. Namely all the returns from sd_ble_gatts_hvx() were being ignored.

After adding the relevant APP_ERROR_CHECK calls the firmware promptly started dying with error code 19 (NRF_ERROR_RESOURCES). Having done a bit of investigating the issue seems to be that the TX queue of the SD fills up (in particular under not so ideal RSSI conditions). My theory is that attempts to retransmit data that didn't get ACK'd prevent further updates on the characteristics.

My investigations have pointed me in the direction of two variables: ble_gatts_conn_cfg_t::hvn_tx_queue_size which is the effective TX queue size & NRF_SDH_BLE_GAP_EVENT_LENGTH which seems to be that value that SD looks at when internally configuring the aforementioned queue size.

I have the following follow-up questions:

what is the correct way of querying the hvn_tx_queue_size parameter? I qould expect there to be something along the lines of ble_gatts_conn_cfg_get but I couldn't find anything like that. My next attempt was to use sd_ble_opt_get() but I haven't managed to find the magical combination of arguments to get it to work. There is a method for checking queue utilization described in the documentation for sd_ble_gatts_hvx():

Store initial queue element count in a variable.
Decrement the variable, which stores the currently available queue element count, by one when a call to this function returns NRF_SUCCESS.
Increment the variable, which stores the current available queue element count, by the count variable in BLE_GATTS_EVT_HVN_TX_COMPLETE event.

It's not very transparent how the queue size is set so i'm not sure what value to use for initial queue element count.

The second question is if there is a way to mitigate this problem using the NRF_SDH_BLE_GAP_EVENT_LENGTH define? our system seems to be working just fine under 95% of conditions with our current value of 12 (corresponds to 15msec of on air time as far as i understand). how do I account for the other 5% of conditions where the queue is getting overloaded? or better yet how do i calculate the optimal value for this parameter.

FYI under the heaviest load we are transmitting about 1122bytes of data (including the preamble and all the overhead and characteristics being split into 2 etc...) with our current connection params of 60msec max_conn_interval and 15msec NRF_SDH_BLE_GAP_EVENT_LENGTH this should be well within the bandwidth. we're using 1M PHY and if i understand the connection parameters correctly this should correspond to 15/60 = 1/4 of max theoretical data rate = 250kBit/sec = 31.25KBytes. if my calculations are collect we're using 3-4% of the theoretically available bandwidth but we still encounter problems...

each of our characteristics would have to be retransmitted on avarage maybe >25 times for the SD to start becoming overwhelmed

I'd appreciate any suggestions as well as any corrections to my calculations...

RE: Random BLE Disconnects / NRF52832 stops responding to Central

hmolesworth — Mon, 19 Jul 2021 16:05:49 GMT

Oh .. and that's the SDK magic definition in use? I play safe ..

#define NRFX_CLOCK_CONFIG_LF_SRC         0
#define CLOCK_CONFIG_LF_SRC              0
#define NRF_SDH_CLOCK_LF_SRC             0

RE: Random BLE Disconnects / NRF52832 stops responding to Central

nikolozka — Mon, 19 Jul 2021 15:39:45 GMT

Thanks for the suggestion, however I'm using the Internal oscilator:

NRF_SDH_CLOCK_LF_SRC 0 (NRF_CLOCK_LF_SRC_RC)
NRF_SDH_CLOCK_LF_RC_CTIV 16
NRF_SDH_CLOCK_LF_RC_TEMP_CTIV 2
NRF_SDH_CLOCK_LF_ACCURACY 1 (500_PPM)

in other words its as relaxed as possible

RE: Random BLE Disconnects / NRF52832 stops responding to Central

hmolesworth — Mon, 19 Jul 2021 15:32:02 GMT

Long shot, but 20ppm is optimistic for some 32kHz crystals when coupled with load cap tolerance, board assembly (dirt, grease, paste) and local pressure/temperature/humidity and even lorry/truck- or earthquake-inspired vibrations. Given that BLE timing relies on this setting, perhaps try and relax it (double it) to see if that changes anything.

// <o> NRF_SDH_CLOCK_LF_ACCURACY  - External clock accuracy used in the LL to compute timing.
 
// <0=> NRF_CLOCK_LF_ACCURACY_250_PPM 
// <1=> NRF_CLOCK_LF_ACCURACY_500_PPM 
// <2=> NRF_CLOCK_LF_ACCURACY_150_PPM 
// <3=> NRF_CLOCK_LF_ACCURACY_100_PPM 
// <4=> NRF_CLOCK_LF_ACCURACY_75_PPM 
// <5=> NRF_CLOCK_LF_ACCURACY_50_PPM 
// <6=> NRF_CLOCK_LF_ACCURACY_30_PPM 
// <7=> NRF_CLOCK_LF_ACCURACY_20_PPM 
// <8=> NRF_CLOCK_LF_ACCURACY_10_PPM 
// <9=> NRF_CLOCK_LF_ACCURACY_5_PPM 
// <10=> NRF_CLOCK_LF_ACCURACY_2_PPM 
// <11=> NRF_CLOCK_LF_ACCURACY_1_PPM 

#ifndef NRF_SDH_CLOCK_LF_ACCURACY
#define NRF_SDH_CLOCK_LF_ACCURACY 7
#endif