Out of range catastrophe

RE: Out of range catastrophe

Simonr — Mon, 24 Jul 2023 08:53:07 GMT

That's great news Žiga

Glad we were able to get to the bottom of this issue. Thank you for letting us know! I'm closing this case then.

Best regards,

Simon

RE: Out of range catastrophe

Ziga Miklosic — Thu, 20 Jul 2023 11:48:27 GMT

Hi Simonr

I have a very good news!

We are now confident that the proposed solution solves the "out of range" problem. After doing a lot of tests there was no sign of devices blocking each other from being re-connected. Everything works as expected.

Thank you again for your time and help!

BR, Žiga

RE: Out of range catastrophe

Simonr — Tue, 04 Jul 2023 12:36:51 GMT

Aha! That makes sense, although I had no idea that the lack of a scan/connection timeout like this would result in "blocking" other devices from reconnecting. I'm sorry I didn't come with this suggestion already, but I really didn't think this would be the root cause, but now that you explain it it makes sense to me. Best of luck with testing, and let me know if you run into any issues on the way.

Best regards,

Simon

RE: Out of range catastrophe

Ziga Miklosic — Tue, 04 Jul 2023 10:01:11 GMT

I think I got it! But there is still a lot to be tested before celebration 🙂

PROBLEM

WHAT: Central device did not end connection initiation procedure, no timeout was implemented! Calling "sd_ble_gap_connect" with scan timeout of 0 will never timeouted!

WHY: After calling "sd_ble_gap_connect" additional advertising packet needs to be received before SD start to send connection request to peer device. In case of DevB adv packets were received very reliable and connection establishment was done immediately. In case of DevA that was on rangle limit its adv packet come very offset and very randomly. So between first and second adv packet it might pass couple of minutes or it might take indefinetly. Central was therefore blocked waiting for that second adv packet. And from outside it looked like it stop to scan. (Here I was misdirected...)

SOLUTION: Implemented timeout for connection initiation (when first calling "sd_ble_gap_connect") and at timeout event calling "sd_ble_gap_connect_cancel".

In picture below there is a sequence diagram of described scenarious for both devices:

Similar issues were discussed on several Nordic Dev Zone post, but unfortunatelly I wasn't come across them before:

1. https://devzone.nordicsemi.com/f/nordic-q-a/63088/it-is-possible-to-set-a-timeout-between-sd_ble_gap_connect-till-ble_gap_evt_connected-event

2. https://devzone.nordicsemi.com/f/nordic-q-a/99516/sd_ble_gap_connect-times-out-but-ble_gap_evt_timeout-never-fires

3. https://devzone.nordicsemi.com/f/nordic-q-a/10386/sd_ble_gap_connect-timeout-in-s120-central

What do you think of that?

BR, Žiga

RE: Out of range catastrophe

Ziga Miklosic — Mon, 03 Jul 2023 13:16:50 GMT

Hi Simonr,

[quote userid="75734" url="~/f/nordic-q-a/99913/out-of-range-catastrophe/434213"]I must say we're quite stumped at this, but wanted to let you know that we're still looking into this on our end as well.[/quote]

Please, don't give up on me! 😃

[quote userid="75734" url="~/f/nordic-q-a/99913/out-of-range-catastrophe/434213"]Just to clarify a few things here. Is it device A that takes up to 4 minutes to reconnect when it is moved back into range of the central device?[/quote]

Yes, device A is on a range limit.

[quote userid="75734" url="~/f/nordic-q-a/99913/out-of-range-catastrophe/434213"]Are you able to provide a sniffer trace here as well. Usually, the debug log isn't all that reliable in terms of timing events, and these are just the times the device reports these events to the debugger.[/quote]

I'll prepare sniffed files during that week. It is a bit difficult to reproduce such a scenario as there are many factors influencing the system but will do my best to capture that event.

[quote userid="75734" url="~/f/nordic-q-a/99913/out-of-range-catastrophe/434213"]1. Most likely due to multiple failed connection requests/responses and retries, as well as this link loss that seems to appear in the midst of the connection here, as well as "Scanning Started" over again. So I'm guessing that it's just at the edge of the range where it's able to be detected.[/quote]

That makes perfect sense, as the device causing the problems is at the very limit of the range.

Meanwhile I added also RSSI value to debug print beside "Connectable advertising report from FC:B4:D7:B8:28:4B". Debug message now looks like that, for example: "Connectable advertising report from FC:B4:D7:B8:28:4B with RSSI: -93 dBm". What I found out is that connection establishment with received signal strength higher than -90dBm is OK. Connection is established right away.

So, I added additional RSSI check to advertisement report event handler before initiating connection. See the code below:

////////////////////////////////////////////////////////////////////////////////
/**
*		BLE on advertisement report event handler 
*   
* @note     This function is being executed in main BLE stack callback!
*
* @param[in] 	p_ble_evt   - Pointer to BLE event informations
* @return 		void
*/
////////////////////////////////////////////////////////////////////////////////
static inline void ble_c_evt_on_adv_report(ble_evt_t const * p_ble_evt)
{
    static  uint8_t             man_data[32]    = { 0 };
            uint16_t            man_data_len    = 0;
             ble_c_obs_data_t   observer_data   = { 0 };

    // Get advertisment info
    const ble_gap_evt_adv_report_t * const p_adv_report = &( p_ble_evt->evt.gap_evt.params.adv_report );

    // Get advertisement type
    const ble_gap_adv_report_type_t * const p_adv_type = &( p_adv_report->type );

    // Get advertisement data and lenght
    const uint8_t * p_adv_data      = p_adv_report->data.p_data;
    const uint16_t  adv_data_len    = p_adv_report->data.len;

    // Advertisment status OK
    if ( BLE_GAP_ADV_DATA_STATUS_COMPLETE == p_adv_type->status )
    {
        // Get manufacturer data
        if ( eBLE_C_OK == ble_c_get_manufacturer_data( p_adv_data, adv_data_len, (uint8_t*) &man_data, &man_data_len ))
        {
            // Advertisment request for connection
            if ( 1U == p_adv_type->connectable )
            {
                BLE_C_DBG_PRINT( "Connectable advertising report from %02X:%02X:%02X:%02X:%02X:%02X with RSSI: %i dBm", p_adv_report->peer_addr.addr[5], p_adv_report->peer_addr.addr[4], p_adv_report->peer_addr.addr[3],
                                                                                                      p_adv_report->peer_addr.addr[2], p_adv_report->peer_addr.addr[1], p_adv_report->peer_addr.addr[0],
                                                                                                      p_adv_report->rssi );

                // Is this device already connected
                //if ( false == ble_c_check_dev_conn( p_adv_report->peer_addr.addr ))
                {                      
                    // Check for magic request
                    //if ( true == ble_c_check_magic_request( man_data, man_data_len ))

                    // TEST: Only for testing purposes!!! Remove later on!!!
                    if ( p_adv_report->rssi > -90 )
                    {
                        BLE_C_DBG_PRINT( "Connection initiated..." );

                        // Continue scanning
                        (void) sd_ble_gap_scan_start( NULL, &g_ble_c.scan_data );

                        // Stop scanning before connection
                        (void) sd_ble_gap_scan_stop();

                        // Connect to peer device
                        if ( NRF_SUCCESS != sd_ble_gap_connect( &p_adv_report->peer_addr, &g_ble_c.scan_params, &g_ble_c.conn_params, BLE_C_CONN_CFG_TAG ))
                        {
                            BLE_C_DBG_PRINT( "Attept to connect failed!" );
                        }
                    }
                }
            }

            // Advertisement only broadcast
            else
            {
                BLE_C_DBG_PRINT( "Broadcasting advertising report!" );

                // Assemble observer data packet
                observer_data.rssi = p_adv_report->rssi;
                observer_data.size = man_data_len;
                memcpy( &observer_data.data, &man_data, man_data_len );
                memcpy( &observer_data.mac, p_adv_report->peer_addr.addr, sizeof( observer_data.mac ));
            
                // Put data to observer data
                if ( eRING_BUFFER_OK != ring_buffer_add( g_ble_c.observer_buf, (ble_c_obs_data_t*) &observer_data ))
                {
                    // Buffer overflow!
                    BLE_C_DBG_PRINT( "Observer buffer overflow! Increse buffer size via \"BLE_C_OBSERVER_BUF_SIZE\" macro!");
                }
            }
        }
    }

    // Continue scanning
    (void) sd_ble_gap_scan_start( NULL, &g_ble_c.scan_data );
}

This is just playing around as I have no other clue how to fix the problem. So far, that filtering technique seems to be working ok, but I'm not confident to use that in production software. On the other hand, it also reduce the working distance of the BLE. What is your opinion on that "solution"?

I will provide you with wanted sniffed file in couple of days.

Thank you!

BR, Žiga

RE: Out of range catastrophe

Simonr — Mon, 03 Jul 2023 12:26:01 GMT

Hi again

I must say we're quite stumped at this, but wanted to let you know that we're still looking into this on our end as well.

Just to clarify a few things here. Is it device A that takes up to 4 minutes to reconnect when it is moved back into range of the central device? For it to take up to 4 minutes, the only thing that makes sense to us is if it is right on the edge of the range it can be detected, but in the 4 minute window you report here there also seem to be a connection loss in the meantime. So it seems like there are something happening in the meantime here. Are you able to provide a sniffer trace here as well. Usually, the debug log isn't all that reliable in terms of timing events, and these are just the times the device reports these events to the debugger.

To answer your questions:

1. Most likely due to multiple failed connection requests/responses and retries, as well as this link loss that seems to appear in the midst of the connection here, as well as "Scanning Started" over again. So I'm guessing that it's just at the edge of the range where it's able to be detected.

2. This is most likely due to failed connection requests and retries as well as advertisements not being picked up by the central when very far away.

3. Without knowing the details of the range between them and seeing a sniffer log, I don't see anything breaking the Bluetooth specification.

4. I will need a sniffer log capturing this behavior before commenting on what you could do to mitigate this issue I'm afraid.

Best regards,

Simon

RE: Out of range catastrophe

Ziga Miklosic — Fri, 30 Jun 2023 09:11:18 GMT

Hello Simonr,

I've done as you recommended, so removed all the logic regarding whitelist, see the advertisement report event handler:

////////////////////////////////////////////////////////////////////////////////
/**
*		BLE on advertisement report event handler 
*   
* @note     This function is being executed in main BLE stack callback!
*
* @param[in] 	p_ble_evt   - Pointer to BLE event informations
* @return 		void
*/
////////////////////////////////////////////////////////////////////////////////
static inline void ble_c_evt_on_adv_report(ble_evt_t const * p_ble_evt)
{
    static  uint8_t             man_data[32]    = { 0 };
            uint16_t            man_data_len    = 0;
             ble_c_obs_data_t   observer_data   = { 0 };

    // Get advertisment info
    const ble_gap_evt_adv_report_t * const p_adv_report = &( p_ble_evt->evt.gap_evt.params.adv_report );

    // Get advertisement type
    const ble_gap_adv_report_type_t * const p_adv_type = &( p_adv_report->type );

    // Get advertisement data and lenght
    const uint8_t * p_adv_data      = p_adv_report->data.p_data;
    const uint16_t  adv_data_len    = p_adv_report->data.len;

    // Advertisment status OK
    if ( BLE_GAP_ADV_DATA_STATUS_COMPLETE == p_adv_type->status )
    {
        // Get manufacturer data
        if ( eBLE_C_OK == ble_c_get_manufacturer_data( p_adv_data, adv_data_len, (uint8_t*) &man_data, &man_data_len ))
        {
            // Advertisment request for connection
            if ( 1U == p_adv_type->connectable )
            {
                BLE_C_DBG_PRINT( "Connectable advertising report from %02X:%02X:%02X:%02X:%02X:%02X", p_adv_report->peer_addr.addr[5], p_adv_report->peer_addr.addr[4], p_adv_report->peer_addr.addr[3],
                                                                                                      p_adv_report->peer_addr.addr[2], p_adv_report->peer_addr.addr[1], p_adv_report->peer_addr.addr[0] );

                // Is this device already connected
                //if ( false == ble_c_check_dev_conn( p_adv_report->peer_addr.addr ))
                {                      
                    // Check for magic request
                    //if ( true == ble_c_check_magic_request( man_data, man_data_len ))
                    {
                        BLE_C_DBG_PRINT( "Connection initiated..." );

                        // Continue scanning
                        (void) sd_ble_gap_scan_start( NULL, &g_ble_c.scan_data );

                        // Stop scanning before connection
                        (void) sd_ble_gap_scan_stop();

                        // Connect to peer device
                        if ( NRF_SUCCESS != sd_ble_gap_connect( &p_adv_report->peer_addr, &g_ble_c.scan_params, &g_ble_c.conn_params, BLE_C_CONN_CFG_TAG ))
                        {
                            BLE_C_DBG_PRINT( "Attept to connect failed!" );
                        }
                    }
                }
            }

            // Advertisement only broadcast
            else
            {
                BLE_C_DBG_PRINT( "Broadcasting advertising report!" );

                // Assemble observer data packet
                observer_data.rssi = p_adv_report->rssi;
                observer_data.size = man_data_len;
                memcpy( &observer_data.data, &man_data, man_data_len );
                memcpy( &observer_data.mac, p_adv_report->peer_addr.addr, sizeof( observer_data.mac ));
            
                // Put data to observer data
                if ( eRING_BUFFER_OK != ring_buffer_add( g_ble_c.observer_buf, (ble_c_obs_data_t*) &observer_data ))
                {
                    // Buffer overflow!
                    BLE_C_DBG_PRINT( "Observer buffer overflow! Increse buffer size via \"BLE_C_OBSERVER_BUF_SIZE\" macro!");
                }
            }
        }
    }

    // Continue scanning
    (void) sd_ble_gap_scan_start( NULL, &g_ble_c.scan_data );
}

After two days of testing the results are the same. Dev B still do not re-connect as DevA is blocking it for some jet unknown reason. As you can se from the code, I added additional debug message before calling "sd_ble_gap_connect" in order to confirm whitelist logic is disable correctly.

Furthermore I have new informations about the problem that might be useful for your team to further investigate/analyse the problem.

I observe that connectable advertising report is received and after that connection is not established right away. That stands only for peripheral devices that are not close to the central device (located at -85/-95 dBm). In some cases, there might be a time difference for couple of minute (worst case I measure was ~4min). During that time, other devices that are close to central do not get connected, are ignored (they are advertising, confirmed with nRF Sniffer).

I provided a logged file, where you can see the described scenario. Focus on two devices with the same naming convention and roles as we have had used in the post above:

DevA, MAC:E4:8E:30:81:78:D9, device that blocks DevB, located away from central
DevB, MAC: CE:AA:75:A4:1C:F1, device close to central, but blocked to connect

Look at the line between 61 and 88 inside attached debug log file from central device - "OutOfRange_Central_DebugLog__30_06_2023.txt":

Line 61, Time 07:28:41.313: BLE_C: Connectable advertising report from E4:8E:30:81:78:D9
Line 70, Time 07:32:37.449: New connection established! Link count: 4 (with E4:8E:30:81:78:D9 device)

(Blocked device with MAC CE:AA:75:A4:1C:F1 was advertising during that time. That device is close to central.)

3. Line 78, 07:32:37.638: BLE_C: Connectable advertising report from CE:AA:75:A4:1C:F1

4. Line 81, 07:32:38.737: BLE_C: New connection established! Link count: 5 (with CE:AA:75:A4:1C:F1)

Look at the times! Between connectable adv report of DevA and actual connection establishement took 4min. After DevA is connected, DevB gets connected immediately.

Questions

Why does it take more time for distand device to established connection?
Why is this connection est. delay only there for distand devices?
Is such behaviour expected and compliant with SIG BLE standard?
Can this be mitigated by any way by application SW?

Debug log file:

devzone.nordicsemi.com/.../OutOfRange_5F00_Central_5F00_DebugLog_5F005F00_30_5F00_06_5F00_2023.txt

I hope these additional observations might spur some idea within your team. Looking forward to your answer!

Thank you again for all the help!

BR, Žiga

RE: Out of range catastrophe

Ziga Miklosic — Tue, 27 Jun 2023 05:57:28 GMT

Hello Simonr,

sorry for late response. I will test your suggestions and will come back with results in a couple of days.

Thank you for all the help!

BR, Žiga

RE: Out of range catastrophe

Simonr — Fri, 16 Jun 2023 11:31:39 GMT

Hi Žiga

We just need you to try without the filtering, as that's what we deem is most likely causing the scanner not to look for device B. Our theory is that when device A goes out of range, the scanner is specifically scanning for the address/ID of device A, and won't care about any other devices until device A has reconnected (or a power reset is triggered for example).

Best regards,

Simon

RE: Out of range catastrophe

Ziga Miklosic — Thu, 15 Jun 2023 12:43:25 GMT

[quote userid="75734" url="~/f/nordic-q-a/99913/out-of-range-catastrophe/431215"]Can you try my suggestion on removing the whitelist implementation on your scanning device so we could narrow down whether the issue is in fact due to the whitelist implementation or not?[/quote]

As already said:

[quote userid="119194" url="~/f/nordic-q-a/99913/out-of-range-catastrophe/430654"]There is no filtering/whitelist problem at all. It is simple the central device stops to scan. The advertising report callback from Soft Device is not being invoked anymore. With the fact that peripheral device advertise correct packets with expected magic number for filtering, as inspected by nRF Sniffer + Wireshark.[/quote]

Problem is more fundamental and it lies one step before even reaching the whitelist logic. As stated the advertising report callback does not come any more, even though sniffing shows that peripheral is advertising.

Any idea why advertising report callback stops to trigger?

And again, when moving DevA back in range, everything goes to normal. Adv Reports start to come and other devices connects as expected.

[quote userid="75734" url="~/f/nordic-q-a/99913/out-of-range-catastrophe/431215"]It is the only thing we can think of that would cause behavior like this, but I'm not able to spot what's wrong reviewing your project as there are a lot of files and lines of code...[/quote]

I understand that...

BR, Žiga

RE: Out of range catastrophe

Simonr — Thu, 15 Jun 2023 11:04:53 GMT

Can you try my suggestion on removing the whitelist implementation on your scanning device so we could narrow down whether the issue is in fact due to the whitelist implementation or not? Then you can see whether the scanner is able to find device B in the same scenario or not.

It is the only thing we can think of that would cause behavior like this, but I'm not able to spot what's wrong reviewing your project as there are a lot of files and lines of code...

Best regards,

Simon

RE: Out of range catastrophe

Ziga Miklosic — Wed, 14 Jun 2023 07:31:34 GMT

Hi Simonr,

[quote userid="75734" url="~/f/nordic-q-a/99913/out-of-range-catastrophe/430900"]But if the central device stops scanning, it doesn't make sense that it is able to immediately reconnect if advertising device A moves back into range, which is still the case, correct?[/quote]

Yes, this is the case. I should have stated "central device stops to scan" in more precise way! It is more like: "Central device scan is blocked by out of range DevA". And from DevB perspective we can say: "Central device stops to scan", as effectively we observed that from DevB point of view.

[quote userid="75734" url="~/f/nordic-q-a/99913/out-of-range-catastrophe/430900"]Putting device A back into range wouldn't restart the scanning in any way, so it's more likely that the scanner can't scan for advertising packets other than from device A when device A was first moved out of range, which is likely why you don't see advertising reports from device B in your log.[/quote]

Correct.

[quote userid="75734" url="~/f/nordic-q-a/99913/out-of-range-catastrophe/430900"]I think the central is indeed scanning (since it reconnects immediately to device A when it is moved back into range), so to me it sounds like it's just not picking up advertising packets from device B in this scenario for some reason.[/quote]

Yes, I totally agree with you!

What can be the reason for that? And how can we mitigate that?

Overall, that is a major drawback to our products, so we would be very glad if we can come to the bottom of the problem.

BR, Žiga

RE: Out of range catastrophe

Simonr — Wed, 14 Jun 2023 07:14:55 GMT

Hi Žiga

But if the central device stops scanning, it doesn't make sense that it is able to immediately reconnect if advertising device A moves back into range, which is still the case, correct? Putting device A back into range wouldn't restart the scanning in any way, so it's more likely that the scanner can't scan for advertising packets other than from device A when device A was first moved out of range, which is likely why you don't see advertising reports from device B in your log. It's definitely advertising (from the sniffer traces) and I think the central is indeed scanning (since it reconnects immediately to device A when it is moved back into range), so to me it sounds like it's just not picking up advertising packets from device B in this scenario for some reason.

Best regards,

Simon

RE: Out of range catastrophe

Ziga Miklosic — Tue, 13 Jun 2023 08:34:04 GMT

Hi Simonr ,

[quote userid="75734" url="~/f/nordic-q-a/99913/out-of-range-catastrophe/429917"]So, the only thing we can see that could be the issue, is that you seem to call the sd_ble_gap_connect() function manually in your project instead of our scan module, where it's called from nrf_ble_scan_connect_with_target(). You can see the nrf_ble_scan.c/.h files for reference.[/quote]

Yes, but the program flow is the same. At advertising report checking for target device (whitelist), then performing connection by first calling sd_ble_gap_scan_stop() and then sd_ble_gap_connect(). After that continue scanning with sd_ble_gap_scan_start(NULL, ...) is called. Therefore our custom code and Nordic official scan module have a same sequence of SD API calls. So I would argue that custom implementation of that specific code shall not be a problem...

[quote userid="75734" url="~/f/nordic-q-a/99913/out-of-range-catastrophe/429917"]1. Enable logging that prints out scanned devices so you can see if device B is actually detected on the scanner side at all[/quote]

I have done that and capture all BLE related events on central side. There was very little code change, nothing that directly related to BLE. In following attachment you can find debug log:

devzone.nordicsemi.com/.../OutOfRange_5F00_Central_5F00_DebugLog_5F00_ManyDevice_5F00_Error_5F00_2_5F005F00_13_5F00_06_5F00_2023.txt

Measurement was done with single central device (same FW as discussed above in the post) and 8 peripheral devices, to mimic the real life scenario problem. Two of peripheral devices was out of range from begging of the test, but that is not important at that point. What I found out that after the random time during test, scanner on central side stops. So debug log "Connectable advertising report" stops. The peripheral device was in fact advertise in that time, as this was confirm by nRF Sniffer! But on central device advertising report callback did not come as there was no debug log saying "Connectable advertising report".

Conclusion

[quote userid="75734" url="~/f/nordic-q-a/99913/out-of-range-catastrophe/429917"]

Scanner can't scan for advertising packets from device B because of the whitelist implementation
Scanner finds advertising packets from device B but does not call sd_ble_gap_connect() due to some logic or filter (whitelist most likely)
Scanner calls sd_ble_gap_connect() for B, but the radio packet isn't sent for some reason.

[/quote]

None of the above is true.

There is no filtering/whitelist problem at all. It is simple the central device stops to scan. The advertising report callback from Soft Device is not being invoked anymore. With the fact that peripheral device advertise correct packets with expected magic number for filtering, as inspected by nRF Sniffer + Wireshark.

So there we have it, central device stops to scan. That must be te problem here as all facts point to that issue.

The question now is, why does that happens? Do we have some scan related software issues? Can you find any anomaly in our code implementation to manifest in scan problems?

Thank you again with all the help!

BR, Žiga

RE: Out of range catastrophe

Simonr — Thu, 08 Jun 2023 08:56:52 GMT

Hi again

So, the only thing we can see that could be the issue, is that you seem to call the sd_ble_gap_connect() function manually in your project instead of our scan module, where it's called from nrf_ble_scan_connect_with_target(). You can see the nrf_ble_scan.c/.h files for reference.

It's hard to tell without being able to debug on our end, but there must be that your project doesn't want to call sd_ble_gap_connect() when what you describe with device B trying to reconnect before device A occurs. A few things you could try to narrow down the issue:

1. Enable logging that prints out scanned devices so you can see if device B is actually detected on the scanner side at all.

2. Remove the whitelisting altogether on the scanner, and check whether or not you're then able to capture advertising and connect to device B in this scenario. That way we should be able to pin point the issue to be either of the following:

Scanner can't scan for advertising packets from device B because of the whitelist implementation
Scanner finds advertising packets from device B but does not call sd_ble_gap_connect() due to some logic or filter (whitelist most likely)
Scanner calls sd_ble_gap_connect() for B, but the radio packet isn't sent for some reason.

Best regards,

Simon

RE: Out of range catastrophe

Ziga Miklosic — Wed, 07 Jun 2023 05:55:18 GMT

Hi Simonr ,

thank you for not giving up!

[quote userid="75734" url="~/f/nordic-q-a/99913/out-of-range-catastrophe/429346"]I have asked for a second opinion internally to see if any of my colleagues can find what's wrong, as I expect I must have missed something. I hope to get back to you tomorrow with a second pair of eyes and some idea of what we're missing here.[/quote]

I'm eager to get more info and perhaps some solution ideas :)

[quote userid="75734" url="~/f/nordic-q-a/99913/out-of-range-catastrophe/429346"]Or if we could get a version of the project that builds for and can reproduce this issue on nRF52840 DKs it would be helpful for us to see this on our end as well.[/quote]

Unfortunately, both central & peripheral mock-up project are prepared for custom board.

I attached both project as a .zip file with a password (as it contains sensitive company data). I'll send you password over friend request. I hope you received it.

devzone.nordicsemi.com/.../RBD_5F00_Central_5F00_Mockup_5F005F00_DevZone.zip

devzone.nordicsemi.com/.../RBD_5F00_Mockup_5F005F00_DevZone.zip

Thanks again for all the help!

BR, Žiga

RE: Out of range catastrophe

Simonr — Mon, 05 Jun 2023 14:11:24 GMT

Hi again

I have looked at your project files, where the scanning and connection setup seems to be perfectly fine as far as I can tell, and the sniffer logs doesn't appear to act unexpectedly either (except the part where the central doesn't try to reconnect to Device B of course). At least this confirms that there must be the central side that is causing issues here. I have asked for a second opinion internally to see if any of my colleagues can find what's wrong, as I expect I must have missed something. I hope to get back to you tomorrow with a second pair of eyes and some idea of what we're missing here. Thank you for your patience again!

I know we've tested Coded PHY scanning, advertising and reconnecting similar to this on SoftDevice 7.2.0 and SDK v17.1.0 before so I don't think that where the problem lies either. Could you also upload your main file perhaps so we can get a better view of your project as a whole. Or if we could get a version of the project that builds for and can reproduce this issue on nRF52840 DKs it would be helpful for us to see this on our end as well.

Best regards,

Simon

RE: Out of range catastrophe

Ziga Miklosic — Thu, 01 Jun 2023 09:17:10 GMT

Hi Simonr ,

[quote userid="75734" url="~/f/nordic-q-a/99913/out-of-range-catastrophe/428446"]From the sniffer trace I hoped to see Device B as well, to see if it shows up as advertising at all when device A and then device B is moved out of range, and if there is in fact a connection request to device B made. But it seems like you only connect to Device A, then reconnect to it, is that correct?[/quote]

Yes, I filter out the device B as I wanted to show you also connection packets and therefore selected only DevA:

[quote userid="75734" url="~/f/nordic-q-a/99913/out-of-range-catastrophe/428446"]Do you see that Device B actually advertises, either on the central device, another central device, or sniffer trace after both devices are moved out of range, and device B is moved back into range? [/quote]

I repeated the measurement with two nRF52840-DK sniffers, one is observing DevA and other DevB. Measurement confirms that DevB is actually advertising.

A take a step further and logged connection state and RSSI of DevA & B simultaneously with Wireshark sniffer. There is some minor (less than 5sec) difference between start of logged file and start of sniffed file. As I was not able to start it all at the same time manually. So sniffed and logged data are soft of synchronised.

Sniffed files:

MAC Addresses:
Central: E2:1E:EA:19:77:C2
Peripheral DevA: DD:5F:43:78:58:0E
Peripheral DevB: CE:AA:75:A4:1C:F1

Same test actions were done as with previous measurement (look at my previous post).

devzone.nordicsemi.com/.../OutOfRange_5F00_DualTest_5F00_DevA_5F005F00_2_5F005F00_01_5F00_06_5F00_2023.pcapng

devzone.nordicsemi.com/.../OutOfRange_5F00_DualTest_5F00_DevB_5F005F00_2_5F005F00_01_5F00_06_5F00_2023.pcapng

Logged measurement connection state & RSSI:

Measurement conclusion:

DevB is advertising while DevA is still out of range. Cetral Dev is not initiating connection (not sending connection req) towards DevB. After DevA is moved back to range, both devices are re-connected and normal operation is re-established.

[quote userid="75734" url="~/f/nordic-q-a/99913/out-of-range-catastrophe/428446"]Do you get either of those two logs printed when device B gets back into range whenever it's not able to reconnect because of device A "still" being out of range?[/quote]

Yes, as confirmed with sniffed data, DevB is advertising when DevA is out of range and sort of "blocking" the connection with DevB. Those two logs are disabled as BLE_C_PRINTF is blocking in nature and therefore program might be blocked for too long in observer handler.

Other questions

- Did you have a time to look at the code? Do you maybe have any comments/improvements regarding SD API usage?

- As it is obvious Central device problem, what will be the next think to look at? Can it be that SDK is not properly configurated?

Thank you for all the help!

BR, Žiga

RE: Out of range catastrophe

Simonr — Wed, 31 May 2023 12:14:16 GMT

Hi again Žiga

From the sniffer trace I hoped to see Device B as well, to see if it shows up as advertising at all when device A and then device B is moved out of range, and if there is in fact a connection request to device B made. But it seems like you only connect to Device A, then reconnect to it, is that correct?

Do you see that Device B actually advertises, either on the central device, another central device, or sniffer trace after both devices are moved out of range, and device B is moved back into range?

I'm sorry, I did indeed miss the paragraph where you explained where to find your whitelist. From the following part of your ble_c_evt_on_adv_report();

// Advertisment status OK

    if ( BLE_GAP_ADV_DATA_STATUS_COMPLETE == p_adv_type->status )

    {

        // Get manufacturer data

        if ( eBLE_C_OK == ble_c_get_manufacturer_data( p_adv_data, adv_data_len, (uint8_t*) &man_data, &man_data_len ))

        {

            // Advertisment request for connection

            if ( 1U == p_adv_type->connectable )

            {

                BLE_C_DBG_PRINT( "Connectable advertising report!" );




                // Is this device already connected

                if ( false == ble_c_check_dev_conn( p_adv_report->peer_addr.addr ))

                {                      

                    // Check for magic request

                    if ( true == ble_c_check_magic_request( man_data, man_data_len ))

                    {

                        // Stop scanning before connection

                        (void) sd_ble_gap_scan_stop();




                        // Connect to peer device

                        if ( NRF_SUCCESS != sd_ble_gap_connect( &p_adv_report->peer_addr, &g_ble_c.scan_params, &g_ble_c.conn_params, BLE_C_CONN_CFG_TAG ))

                        {

                            BLE_C_DBG_PRINT( "Attept to connect failed!" );

                        }

                    }

                }

            }

Do you get either of those two logs printed when device B gets back into range whenever it's not able to reconnect because of device A "still" being out of range? Additional logging might be helpful to get a better picture of what's going on.

Best regards,

Simon

RE: Out of range catastrophe

Ziga Miklosic — Tue, 30 May 2023 05:57:03 GMT

Hello Simonr ,

Please find the sniffed BLE communication and test description (README.txt) in the attachement:

devzone.nordicsemi.com/.../README.txt

devzone.nordicsemi.com/.../OutOfRange_5F00_test_5F00_1_5F005F00_29_5F00_05_5F00_2023.pcapng

I also added packet comments to .pcapng file to easier track of major events.

I hope this will speed up solving the out of range problem!

BR, Žiga

RE: Out of range catastrophe

Ziga Miklosic — Mon, 29 May 2023 05:47:42 GMT

Sorry for late response, I've been out of office on friday!

Answers

[quote userid="75734" url="~/f/nordic-q-a/99913/out-of-range-catastrophe/427785"]Okay, and there is the same manufacturer data on all devices, so it shouldn't matter what manufacturer data the advertising devices transmit? Since I don't know how this custom whitelist is implemented my thoughts go to the central scanning for data that the first disconnected device (A) is advertising, and thus doesn't really "look for" the device B.[/quote]

Yes and no. All peripheral devices advertise manufacturer data, that is true. The content of manufacturer data on does matter, as based on valid data values connection is being initiated. Please, look at the attached code inside "ble_c.c" line 1175: ble_c_evt_on_adv_report

[quote userid="75734" url="~/f/nordic-q-a/99913/out-of-range-catastrophe/427785"]Any chance you can upload the file where the whitelist implementation is done so we can take a look at how you've implemented this?[/quote]

Simonr I have a feeling that you read my post/answers superficially, as I already upload code and pointed out where to look for whitelist implementation:

[quote userid="119194" url="~/f/nordic-q-a/99913/out-of-range-catastrophe/427527"]Central device is using whitelist, but custom implemented one. It simply check for manufacturer data inside connectable advertisement packet and if that data match it tries to connect to that peripheral (look at the attached code inside "ble_c.c" line 1175: ble_c_evt_on_adv_report).[/quote]

[quote userid="75734" url="~/f/nordic-q-a/99913/out-of-range-catastrophe/427785"]As of version 4.0.0 of the nRF Sniffer supports Coded PHY, so that should be fine.[/quote]

OK, great! Then I will take time to repeate experiment and prepare sniffed files. This might take some time for me. In meanwhile I would be very glad if we can continue to search for the obvious reason.

Thank you again for all the help. We really appreciate your help!

BR, Žiga

RE: Out of range catastrophe

Simonr — Fri, 26 May 2023 12:08:29 GMT

[quote user="ziga_m"]Central device is using whitelist, but custom implemented one. It simply check for manufacturer data inside connectable advertisement packet and if that data match it tries to connect to that peripheral (look at the attached code inside "ble_c.c" line 1175: ble_c_evt_on_adv_report).[/quote]

Okay, and there is the same manufacturer data on all devices, so it shouldn't matter what manufacturer data the advertising devices transmit? Since I don't know how this custom whitelist is implemented my thoughts go to the central scanning for data that the first disconnected device (A) is advertising, and thus doesn't really "look for" the device B. Any chance you can upload the file where the whitelist implementation is done so we can take a look at how you've implemented this?

As of version 4.0.0 of the nRF Sniffer supports Coded PHY, so that should be fine.

Best regards,

Simon

RE: Out of range catastrophe

Ziga Miklosic — Thu, 25 May 2023 12:37:47 GMT

Hello,

[quote userid="75734" url="~/f/nordic-q-a/99913/out-of-range-catastrophe/427509"]Can you explain a bit more on how the central device behaves when a peripheral is disconnected? [/quote]

As described, central device will start to scan right after disconnection of the peripheral device in order to obtain connection back:

[quote userid="119194" url="~/f/nordic-q-a/99913/out-of-range-catastrophe"]So, we prepare a simple mock-up test application to check out how the system will behave in real-life scenarios. Test application was very simple, central BLE device scans continuously and stops after all 8 connections are established. Peripheral BLE devices advertise when not in connection. With such approach system shall always converge to have all devices connected, as if one device gets disconnected advertising/scanning shall re-started and the drop connections should re-establish. Every second 64 bytes of dummy data were exchange between central and peripheral device based on server-initiate update (notification type). Tx power for advertising and connection was set to 8dBm. Connection interval was set to 1500ms, SlaveLatency to 2 and Supervision Timeout to 15000ms. Well, it works perfectly on a desk! [/quote]

[quote userid="75734" url="~/f/nordic-q-a/99913/out-of-range-catastrophe/427509"]To me it doesn't really seem like an issue on the peripheral, but rather with how the central device handles disconnects and reconnections.[/quote]

Yes, we also thinks that the problem is on central side, as said:

[quote userid="119194" url="~/f/nordic-q-a/99913/out-of-range-catastrophe"]We suspect that Device A (goes out of range first) block Device B in context of advertisement, as both devices are peripheral. But that is not the case as we done separate tests to eliminate that possibility, where the first disconnected device did not start advertising at disconnection. Same effect was observed, other devices did not re-connect and thus conclude that advertisement do not play role in that effect, meaning that the problem lives on Central device.[/quote]

[quote userid="75734" url="~/f/nordic-q-a/99913/out-of-range-catastrophe/427509"]It almost seems like when a device is disconnected, the central device will wait for that specific device to reconnect before allowing/searching for any other disconnected peripherals.[/quote]

Exactly! That behaviour is repeatable and easily reproduced. But note that only when data transmission is enabled! As said:

[quote userid="119194" url="~/f/nordic-q-a/99913/out-of-range-catastrophe"]

If we take Device A (one of the 8 peripheral devices, all in connection) out of range, it disconnects. If then moved back to range, it connects back without any problems - NORMAL EXPECTED OPERATION,
If we take Device A (one of the 8 peripheral devices, all in connection) out of range, it disconnects. If we then move Device B (one of the 8 peripheral devices, all beside Device A in connection) out of range, it disconnects. Then Device B does not re-connect when moved back to range. Only after moving Device A back to range, both Devices A & B gets re-connected. It is like Device A is blocking Device B from re-connectiong, regardless Device B is in range! - ABNORMAL OPERATION,
We observe that only first disconnected device (due to out of range) blocks other from re-connecting. Moving first disconnected device back to range triggers all other devices to re-connect,

We suspect that Device A (goes out of range first) block Device B in context of advertisement, as both devices are peripheral. But that is not the case as we done separate tests to eliminate that possibility, where the first disconnected device did not start advertising at disconnection. Same effect was observed, other devices did not re-connect and thus conclude that advertisement do not play role in that effect, meaning that the problem lives on Central device.

If we disable data transmission on Central Device and Device A and repeat point 2., Device B gets re-connected when moved back to range. In that test case Device A did not blocked Device B from re-connection.

[/quote]

[quote userid="75734" url="~/f/nordic-q-a/99913/out-of-range-catastrophe/427509"]Does the central scan for each specific device using a whitelist, or does it put the disconnected devices in a queue for example so that device B will not be considered to connect before device A is reconnected?[/quote]

Central device is using whitelist, but custom implemented one. It simply check for manufacturer data inside connectable advertisement packet and if that data match it tries to connect to that peripheral (look at the attached code inside "ble_c.c" line 1175: ble_c_evt_on_adv_report).

There is no mechanism implemented to block devices to re-connect in any way. This was also verified in described test (look at the test point 2, 3, 4). In normal conditions (all devices in range) everythink is working as expected.

[quote userid="75734" url="~/f/nordic-q-a/99913/out-of-range-catastrophe/427509"]Are you able to share a sniffer trace using a dedicated sniffer or the nRF Sniffer so we can see what messages are being sent over the air when this issue occurs.[/quote]

Unfortunately, we don't own BLE sniffer. Does nRF Sniffer support CODED PHY? I need to check it and will provide you with logged data. That will take some time for me, so expect to get answer beginning of next week.

Here is also our sdk_config and LL BLE driver code in case you find out some obvious problems:

devzone.nordicsemi.com/.../ble_5F00_c.c

devzone.nordicsemi.com/.../ble_5F00_c.h

devzone.nordicsemi.com/.../4477.sdk_5F00_config.h

I hope given information will be sufficient to start investigating of the problem. As we do not have any clue how to solve/mitigate that problem, we would be very glad to get any ideas/recommendations where to start🙂

Thank you for all the help!

BR, Žiga

RE: Out of range catastrophe

Simonr — Thu, 25 May 2023 11:47:37 GMT

I've read through the case, and this behavior seems very strange indeed. Thank you for the thorough explanation! Can you explain a bit more on how the central device behaves when a peripheral is disconnected? It almost seems like when a device is disconnected, the central device will wait for that specific device to reconnect before allowing/searching for any other disconnected peripherals.

Does the central scan for each specific device using a whitelist, or does it put the disconnected devices in a queue for example so that device B will not be considered to connect before device A is reconnected? To me it doesn't really seem like an issue on the peripheral, but rather with how the central device handles disconnects and reconnections.

Are you able to share a sniffer trace using a dedicated sniffer or the nRF Sniffer so we can see what messages are being sent over the air when this issue occurs.

Best regards,

Simon

RE: Out of range catastrophe

Simonr — Thu, 25 May 2023 11:28:58 GMT

Hello Žiga

I'm very sorry, but this case seem to have slipped through the cracks during the public holidays last week and we're only just now getting to it. I just wanted to let you know that I'm now looking into it and will get back to you later today or tomorrow when I've read and understood your case.

Thank you for your patience,

Simon