Central device claims to have Connected but no Connection event is fired on the peripheral

RE: Central device claims to have Connected but no Connection event is fired on the peripheral

Kenneth — Tue, 10 Mar 2020 10:53:18 GMT

[quote user="nvelozsavino"]I've been noticing that the issue is happening more often now (the battery voltage is lower), I'm not sure if this could be important but we are not using the DC-DC. Is there a minimum voltage to ensure that the flash writes properly?[/quote]

The flash can be read, written and erased over the entire supply voltage range yes. Though you may look into the stability of VDD if you think it is fluctuating a lot, becuse very large ripples (or if VDD step up) this can cause the power on reset to trigger, which will prevent any on-going flash operation or queue to finish.

You should also look into if you have any asserts or hardfaults, since those also typically will do a soft reset of the chip (default code of the error handlers), which also will prevent any on-going flash or queue to finish. For instance a flash erase operation can block the CPU execution with ~90ms, which may cause multiple interrupts to be pending for a long period of time.

Have you tried to dump the entire content of the flash before and after it is failing? Then you can for instance look at the flash page tags and record layout:
https://infocenter.nordicsemi.com/topic/com.nordic.infocenter.sdk5.v15.3.0/lib_fds_format.html#lib_fds_format_page

Please follow up on the: "Do you have an on-air sniffer log that show bonding between the two peers before the re-connection fails?".

RE: Central device claims to have Connected but no Connection event is fired on the peripheral

nvelozsavino — Fri, 06 Mar 2020 10:00:00 GMT

HI Kenneth

No, I don't have any logic to make sure the fstorage is not busy before going to sleep. I'll try to implement something like that to see if it helps.

I've been noticing that the issue is happening more often now (the battery voltage is lower), I'm not sure if this could be important but we are not using the DC-DC. Is there a minimum voltage to ensure that the flash writes properly?

If the problem is that the devices are not bonding, then 2 questions arise:

1. How they lose the bonding information? Both? central and peripheral? or is the central who lose the bonding information, correct me if I'm wrong but this could happen if the flash gets corrupted right?

2. The bonding information should be replaced when I delete the bonds on the central and reenable the whitelist on both devices, however, just deleting the bonds doesn't work, you have to do a flash erase in order to work again. As far as I understand deleting bonds is just invalidating the files stored by the peer manager in the fstorage. Again, my hypotesis still holds, if the flash is corrupted somehow, invalidating the files containing the bonds doesn't seem to work.

RE: Central device claims to have Connected but no Connection event is fired on the peripheral

Kenneth — Tue, 03 Mar 2020 13:36:11 GMT

[quote user="nvelozsavino"]I don't know if it could be that the central device goes to System OFF if there's no activity registered by an accelerometer, It might be possible that the device shuts down while a flash operation is in progress and this could cause the corruption, maybe between invalidating a file before adding the new updated one.[/quote]

Do you have any logic that make sure nrf_fstorage_is_busy(NULL) does not return true before you are going to sleep?

[quote user="nvelozsavino"]My guess is that somehow the flash gets corrupted, something related to the info of the peer saved by the peer manager.[/quote]

The problem with theory is that it would still connect if this was the case, only encryption would fail. I am thinking that the real problem here is that the two devices are not bonding at all. Do you have an on-air sniffer log that show bonding between the two peers before the re-connection fails?

RE: Central device claims to have Connected but no Connection event is fired on the peripheral

nvelozsavino — Tue, 03 Mar 2020 09:36:36 GMT

Hi Kenneth, I believe that the problem is definetly the Central, and I strongly doubt that there's something to do with noise, if that were the case, the probability of a successful connection should be higher than 0 after the problem manifest for the first time, what I've seen is that once the problem occours once, the central device never reconnects, other centrals can connect successfully to the same peripheral.

I've tested deleting all bonds on the central without success, the only way it can work again without having to re-flash the MCU is doing a clean of all the data on the flash (only data stored by the peer manager, and the app, not the actual running program)

My guess is that somehow the flash gets corrupted, something related to the info of the peer saved by the peer manager. I don't know if it could be that the central device goes to System OFF if there's no activity registered by an accelerometer, It might be possible that the device shuts down while a flash operation is in progress and this could cause the corruption, maybe between invalidating a file before adding the new updated one.

We had a while ago another issue with another hardware, also a central device, after about 100 different connections to different peripherals (allowing only 1 at the time in the whitelist), the central stop connecting anymore and a clean of the flash was required. We haven't had any new reports from the user about this problem again, but I don't know if it's because we added an option to factory reset the central which solve the issue. Our guess that time was that the flash filled and there was a problem with the garbage collector that damaged the peers information. That time we were using a different sdk and softdevice version (s132 v5) and a different hardware, but now that I remember, it was the same behavior

RE: Central device claims to have Connected but no Connection event is fired on the peripheral

Kenneth — Mon, 02 Mar 2020 14:35:25 GMT

I assume that the central is f5:c6:75:d8:88:50 and the peripheral is e5:fa:95:ec:dc:61.

From both logs I can see that the central send several scan request, but none are responded by a scan response by the peripheral. All the scan requests have invalid CRC according to the sniffer logs from the specific central, this lead me to believe you may have marginal hardware and/or some noise in the system. Is there any serial interfaces and/or gpios active with high drive strength that may impact the modulation of the scan request packet on the central? Have you done any DTM testing on your central hardware to verify modulation characteristics etc? Can you use an nRF52-DK for comparison?

Also I can see that other central devices (e.g. 74:33:b4:05:18:b9 and 76:d6:47:c6:8e:94) seems to be able to send scan request packets successfully (some errors, but not all) and receive scan response from the peripheral during this time. Have you tried to call connect from any of these centrals for comparison?

The connection request packets don't have any CRC errors based on the logs, so this leads me to believe that the whitelist on the peripheral may be wrong and/or some noise in the system, but of course difficult to know. Can you set a breakpoint in sd_ble_gap_whitelist_set() just to check that the addr_ptrs contain the central address (f5:c6:75:d8:88:50)?

Have you tried to call ble_advertising_restart_without_whitelist() to just check?

I find it slightly inconclusive yet if it's the peripheral or central side that is the issue.

Best regards,
Kenneth

RE: Central device claims to have Connected but no Connection event is fired on the peripheral

nvelozsavino — Fri, 28 Feb 2020 14:34:28 GMT

Hi Kenneth, today I got the error again, I'm attaching the sniffer traces where it fails

Within the file: devzone.nordicsemi.com/.../connection_5F00_unsuccessful_5F00_2.pcapng

at line 854 you can see the Central trying to connect and the peripheral doesn't respond, both using their whitelists

at line 2012 there's a connection again but this time both devices has their whitelist disabled, this time I can see that the peripheral is disconnecting, at line 2020

after that, you will find lots of attemps with the whitelist enabled, Then I did a factory reset (mass erase) on the peripheral and starting line 4163 you can see the connection attempt and the peripheral is disconnecting.

The problem is solved when I perform a mass erase on the central, as you can see in the successful file devzone.nordicsemi.com/.../connection_5F00_successful_5F00_3.pcapng

as you can see in line 981

RE: Central device claims to have Connected but no Connection event is fired on the peripheral

nvelozsavino — Wed, 19 Feb 2020 15:42:20 GMT

Hi Kenneth, no, it's not the same device, I have several peripherals and 4 master devices, it doesn't seem to be related with a specific hardware, I've seen the issue in all combinations.

However I'll try to reproduce both a successful connection and the issue with the same pair of devices

RE: Central device claims to have Connected but no Connection event is fired on the peripheral

Kenneth — Wed, 19 Feb 2020 15:28:15 GMT

But this is not the same devices?

Seems both the central and peripheral gap addresses are different between working and failing?

Are you changing the GAP addresses of peers here?

Kenneth

RE: Central device claims to have Connected but no Connection event is fired on the peripheral

nvelozsavino — Tue, 18 Feb 2020 16:30:18 GMT

Here's the trace when the connection is successful, the packet id of the connection request is 11199

devzone.nordicsemi.com/.../connection_5F00_successful.pcapng

RE: Central device claims to have Connected but no Connection event is fired on the peripheral

Kenneth — Tue, 18 Feb 2020 12:44:46 GMT

[quote user="nvelozsavino"]I've noticed that the problem happens when the central has more than one device in the whitelist,[/quote]

Do you have a sniffer log of failing and working? The next step would be to look at the content of the connection request packet from the central. The peripheral do not have any knowledge of the whitelist that may be present on the central, so this is indeed very odd.

Kenneth

RE: Central device claims to have Connected but no Connection event is fired on the peripheral

nvelozsavino — Tue, 18 Feb 2020 09:34:19 GMT

The MTU exchange shouldn't wait for the connection response?

The whitelist is enabled on the peripheral, but the central is a known device and it's within the whitelist of the peripheral

The peripheral is our own hardware, It's working just fine with other central using other softdevice s132 6.0.0

I doubt it has something to do with signal interference, the problem seems to be related with the flash storage on the central, because as soon as you do a mass erase on the central and disable the whitelist on the peripheral to allow the bond to be updated, it connects again. All this without changing anything in the environment.

Before you ask, I've already tried disabling the whitelist on the peripheral, The peripheral doesn't even know there's a connection request.

I've noticed that the problem happens when the central has more than one device in the whitelist,

RE: Central device claims to have Connected but no Connection event is fired on the peripheral

Kenneth — Mon, 17 Feb 2020 16:18:39 GMT

Everything is pointing to the connection request packet is not received (the advertisement timing indicate that the interval is not stopped at any time). The MTU exchange is just the first packet from the central, so that doesn't contradict that the connection request packet is not received.

Presuming that whitelist is not enabled on the peripheral, then the only other explanation is some noise preventing the connection request packet to be received on the peripheral. Is the peripheral here an nRF52-DK or your own hardware? Is there any serial interfaces active near the antenna that may prevent the packet from being received?

Best regards,
Kenneth

RE: Central device claims to have Connected but no Connection event is fired on the peripheral

nvelozsavino — Thu, 13 Feb 2020 09:43:38 GMT

I finally was able to get the sniffer traces, I can't see any connection response from the peripheral, however, the master still tries to negotiate the MTU

devzone.nordicsemi.com/.../traces_5F00_with_5F00_issue.pcapng

The peripheral address is: e6:9d:93:ce:2d:c0

and the master address is: e1:2e:95:29:49:6c

RE: Central device claims to have Connected but no Connection event is fired on the peripheral

Kenneth — Mon, 20 Jan 2020 16:24:46 GMT

I think the next step for debugging is to try download and use the nRF sniffer, so we may see if the connection request packet is indeed sent or not:

https://www.nordicsemi.com/Software-and-tools/Development-Tools/nRF-Sniffer-for-Bluetooth-LE

On the peripheral you may also monitor the whitelist, to see if it's updated at any time during, before or after advertisement/connection.

RE: Central device claims to have Connected but no Connection event is fired on the peripheral

nvelozsavino — Mon, 20 Jan 2020 15:19:30 GMT

Hello Keneth, yes, I'm using a whitelist on the peripheral, however, that's not the problem, there's no other central trying connecting, and if that would be the case, disabling the whitelist on the peripheral would allow the old central to connect again, however, even disabling the whitelist on the peripheral doesn't work unles I delete all bonds on the central device.

I've already tested the allowing repair, that's not the issue, the code doesn't even reach the event PM_EVT_CONN_SEC_CONFIG_REQ on the peripheral. before getting to that point it requires to fire at least BLE_GAP_EVT_CONNECTED or PM_EVT_BONDED_PEER_CONNECTED, whitch doesn't occour. It's like the connection request is not going out over the air, or the peripheral filter it somehow, is there a way to debug the Softdevice sd_ble_gap_connect step by step?

RE: Central device claims to have Connected but no Connection event is fired on the peripheral

Kenneth — Mon, 20 Jan 2020 15:05:38 GMT

It sounds very much like that you have some whitelist enabled/active on the peripheral, and if for instance the peripheral is bonded with >8 devices, then it is not able to include the "older" centrals in the whitelist. Hence the "old" central will not be able to connect.

It may also be because the central have deleted or erased it's bond data, in such case you need to allow re-pairing from the peripheral by adding the following code on the peripheral:

https://devzone.nordicsemi.com/f/nordic-q-a/37837/sdk15-paring-problem/145821#145821

I assume you are not altering the GAP address on the central or peripheral here.

RE: Central device claims to have Connected but no Connection event is fired on the peripheral

nvelozsavino — Mon, 20 Jan 2020 12:45:43 GMT

Hello Kenneth, as I say before, the connection works fine for a while, I can turn off, turn on, disconnect, reconnect several times, then suddenly, it stops working, no more connection to that particular peripheral, I don't believe it's something regarding interference, both devices are close, is not like it sometimes connects and sometimes it fail, when the issue happens, it never connect again. However, without moving anything, just deleting bonds or doing a mass erase it starts working again.

I'm going to try to sniff the packets to check if there's something wrong that I'm not seeing in the logs, however I need to get the nrf52 DK or the dongle to do it.

Summarizing:

1. After a fresh mass erase or a delete bonds, everything works fine, it always find the peripheral, connect, bonds, reconnects if disconnected, you can power off and on several times and everything works

2. Suddenly the peripheral doesn't connect anymore, the Central is able to see the advertising packets from the peripheral, however as soon as the central tries to connect, the BLE_GAP_EVT_CONNECTED is received in the Central device but the peripheral doesn't receive anything, shortly after the central disconnects

3. The peripheral is connectable through another central

4. Scanning and connecting for another peripheral works fine (creating and using a new bond and a new peer entry)

5. Deleting all bonds on the central and allowing the peripheral to refresh the bond information fix the connection issues

6. Repeat from step 1

I haven't count how many times the connection works before getting the issue, however I'm sure that it's more than at least 10 times, but probably a lot more.

RE: Central device claims to have Connected but no Connection event is fired on the peripheral

Kenneth — Mon, 20 Jan 2020 12:15:09 GMT

You are not writing anything about how frequently this occurs , but it is possible that the connection request packet from the central to the peripheral from time to time is lost (e.g. due to interference). In such case the central will first get an BLE_GAP_EVT_CONNECTED event, then shortly after get an BLE_GAP_EVT_DISCONNECTED event with disconnect reason BLE_HCI_CONN_FAILED_TO_BE_ESTABLISHED.

Best regards,
Kenneth

RE: Central device claims to have Connected but no Connection event is fired on the peripheral

nvelozsavino — Fri, 17 Jan 2020 15:28:39 GMT

Deleting the bonds seems to solve the issue, therefore I believe it's something related to the flash storage?

RE: Central device claims to have Connected but no Connection event is fired on the peripheral

nvelozsavino — Fri, 17 Jan 2020 14:41:19 GMT

I first thought that it was related with this issue:

https://devzone.nordicsemi.com/f/nordic-q-a/42384/connection-security-failed-role-central-in-ble_app_rscs_c/229637#229637

However I now believe it's the other way around, that issue may be caused by this odd behavior