Beacon output suddenly stops

RE: Beacon output suddenly stops

hiroiwas — Thu, 30 Jan 2025 04:13:41 GMT

Hello,

A tentative conclusion has been reached on the private ticket side, so I will post it here.

The issue occurred in the following scenario:

1. An error occurred in the initiator when connecting.
☞ Example: The peripheral address changed after sd_ble_gap_connect(), the peripheral was shutdown, deterioration of radio wave conditions, etc.

2. In SoftDevice, the initiator where the error occurred will not be able to accept the next ADV_IND.

3. Because the scan timeout is set to 0 (no timeout), the initiator error state continues.

4. Being unable to escape from the initiator causes the scheduler (time slot) inside the SoftDevice to lock, causing the SD to become unresponsive from the app's perspective.
☞ The SD becomes unresponsive, but the app continues to run.

5. If the time slot stops working, the next beacon cannot be sent.
☞ This was the first issue that encounted.

6. Since the RADIO time slot processing will no longer work, subsequent Radio notifications will also stop.
☞ A fail-safe measure that resets itself by monitoring the time when no response is received from the radio notification

This issue occurs because the peripheral address update in 1 causes 3, and also because 2 causes 3.

The solution is to set the ble_gap_scan_params_t::timeout (given by sd_ble_gap_connect) on the initiator where the malfunction occurs to a significant value other than 0, and to resume scanning when BLE_GAP_EVT_TIMEOUT of the BLE_GAP_TIMEOUT_SRC_CONN event occurs.

Regarding the SoftDevice fix in 2., we have received the answer that, given the state of the live cycle, unfortunately, it is highly unlikely that this issue will be fixed (see the nRF Connect SDK and nRF5 SDK statement given in 2021).

That's all.

RE: Beacon output suddenly stops

Einar Thorsrud — Mon, 05 Aug 2024 14:45:58 GMT

Hi,

That sounds good. let us continue the discussion in the private ticket for now.

RE: Beacon output suddenly stops

hiroiwas — Mon, 05 Aug 2024 11:06:25 GMT

Hello Einar,

A private ticket has been issued.
Thank you for your continued support until the issue is resolved.

Best regards.

RE: Beacon output suddenly stops

Einar Thorsrud — Fri, 02 Aug 2024 14:14:21 GMT

Hi Iwasaki,

[quote user="hiroiwas"]When providing this software, the internal code contains confidential security information, so I would like to ask that you not use it for anything other than bug analysis. Is that okay?
You can answer this ticket instead of signing an NDA.[/quote]

We handle all prive support cases confidentlially (not that this is a public case at the moment).

[quote user="hiroiwas"]Also, to run it, you will need a PC with a UART connection and two or more Android smartphones.[/quote]

Is it possibel to reproduce without Android devices? I should be able to obtain one or two for quick testing, though.

[quote user="hiroiwas"]If there are no problems with the above, I will send you the currently working set of the software, so please send the recipient's email address to my email address.[/quote]

Please make a private support ticket where you refer to this thread. There you can upload your code as a zip file.

If it is so that this issue is difficult to consisently and fairly quickly reproduce it would be prefrable if it is possible to reproduce it usign a simpler setup. A simple setup is also benefisial for narrowing down the root cause of the issue. (We can of course inspect the code and see if we see somehting suspicious also without having been able to reproduce it on our end).

Br,

Einar

RE: Beacon output suddenly stops

hiroiwas — Thu, 01 Aug 2024 11:01:38 GMT

Hello Einar,

I have something to confirm.

When providing this software, the internal code contains confidential security information, so I would like to ask that you not use it for anything other than bug analysis. Is that okay?
You can answer this ticket instead of signing an NDA.

Also, to run it, you will need a PC with a UART connection and two or more Android smartphones.

If there are no problems with the above, I will send you the currently working set of the software, so please send the recipient's email address to my email address.

If there are any problems, we will need to create a subset with limited functionality in order to reproduce the phenomenon, so this will take some time, including confirming that the phenomenon can be reproduced with the subset.

Best regards.

RE: Beacon output suddenly stops

Einar Thorsrud — Tue, 30 Jul 2024 10:32:51 GMT

Hi,

I see. I understand that this issue is difficult to reproduce, but are you ablet share a project that can reproduce this on a DK (given enough time)?

Regardgin the other thread that is using the nRF Connect SDK which is using a different Bluetooth stack (though the controller / link layer inherits components from the SoftDevice).

RE: Beacon output suddenly stops

hiroiwas — Tue, 30 Jul 2024 10:02:49 GMT

Hello Einar,

> This was fixed in SoftDevice version v7.3.0 though

Currently the SoftDevice version that is having the issue is v7.3.0.

> It is regardless important to process SoftDevice events in a timely manner.

The most time-consuming BLE event is the process between receiving NRF_BLE_SCAN_EVT_NOT_FOUND(SCAN_RSP) and calling sd_ble_gap_connect (a maximum of about 100 ms).
Specifically, what should be the upper limit for this event processing time?

> Do you know more about when this issue occurs?

This issues always occurred right after sd_ble_gap_connect in the NRF_BLE_SCAN_EVT_NOT_FOUND event.

> Could it be that the application have not pulled events in a long time due to other activity?

The logs at the time the problem occurred show no evidence of any events other than BLE being active.

In fact, no BLE events (<debug> nrf_sdh_ble: BLE event: xx) have been issued from the SD card since sd_ble_gap_connect at the time of the problem.

> And if you use the SoftDvice hanler with app scheduler,

No, SoftDevice events are set to use interrupts.

#define NRF_SDH_DISPATCH_MODEL 0
// <i> NRF_SDH_DISPATCH_MODEL_INTERRUPT: SoftDevice events are passed to the application from the interrupt context.

FYI, this issues seems to be the same.

Best regards.

RE: Beacon output suddenly stops

Einar Thorsrud — Tue, 30 Jul 2024 09:21:30 GMT

Hi,

There was an issue in some SoftDevice versions that coudl cause also important events to be lost if the applications does not pull/process evetns on time. This was fixed in SoftDevice version v7.3.0 though, so I wonder if you can confirm that that is what is used on the failing devices? (You mentionned v7.2.0 as well, where this issue still existed).

It is regardless important to process SoftDevice events in a timely manner. Do you know more about when this issue occurs? Could it be that the application have not pulled events in a long time due to other activity? And if you use the SoftDvice hanler with app scheduler, could it be that the app scheduler queue is too short, for instance?

RE: Beacon output suddenly stops

hiroiwas — Tue, 30 Jul 2024 04:07:46 GMT

Hello Einar,

> but I have pinged the devs looking at this internally, and will let you know if they have any updates.

> For now I do not have anything to add to what he wrote,

In this case, we have no option but to reset the device, and since the problem is occurring within the SoftDevice, we would appreciate your immediate response.

It's been two months since the problem first occurred and no progress has been made.

If there is any information you require, please let us know and we will do our best to provide it.

> You can uploade large files here on DevZone as long as the file type is allowed (and .zip is).

I previously uploaded Log of the incident.zip, but when I tried to upload the log this time, I was unable to do so.
So I wondered if there was a limit to the upload size, and inquired about it.

Best regards,

RE: Beacon output suddenly stops

Einar Thorsrud — Mon, 29 Jul 2024 12:37:22 GMT

Hi,

I have inerited this case back from Simon. For now I do not have anything to add to what he wrote, other than that if the problem with uploading logs is the file type, then I suggest putting the log in a zip file and uploading that. You can uploade large files here on DevZone as long as the file type is allowed (and .zip is).

Br,

Einar

RE: Beacon output suddenly stops

Simonr — Thu, 25 Jul 2024 12:31:41 GMT

I'm sorry you're being sent in a loop here, but now Hieu is out of office. I haven't had time to get into this case, but I have pinged the devs looking at this internally, and will let you know if they have any updates.

Einar who initially had this ticket will return on Monday July 29th and likely take back this case then. So sorry about all the delays.

Thus far I'm afraid there's no further updates from our side. You say that the full log can't be uploaded. Would it be possible to upload it with Google Drive or similar so we can download it on our end from there? Alternatively you can send it by mail to me, but for the mail address I'll reach out to you in direct messages if that's what you prefer.

And just a shot in the dark from me, how many devices has the beacon connected to when this issue occurs. Could it be that the maximum number of connections is reached, and that's why the advertiser won't start again, since it seems to happen after a connection event if I'm not mistaken.

One option you could try is to add a watchdog timer that will reset the device if no BLE events trig in X amount of time. Use the WDT example for reference here.

Best regards,

Simon

RE: Beacon output suddenly stops

hiroiwas — Mon, 22 Jul 2024 08:16:11 GMT

Hello Hieu,

Has there been any progress on the Nordic investigation since then?

It takes a while for this to happen, but it has happened a few times here.

In order to reduce the load of SoftDevice events during connection, I tried calling sd_ble_gap_adv_stop() just before sd_ble_gap_connect() and then restarting the beacon with ble_advertising_start() when BLE_GAP_EVT_DISCONNECTED, but the issue of RadioNotification stopping persisted.

The situation from the log when the problem occurred was exactly the same.

I also tried putting scan_stop just before or just after sd_ble_gap_connect, but I was unable to establish a connection.

At this point, we're running out of options, so are there any other effective methods you can think of?

Best regards.

RE: Beacon output suddenly stops

hiroiwas — Fri, 19 Jul 2024 06:53:36 GMT

Hello Hieu,

Since the problem occurred, I have attached the trimmed raw log.

00> ==SCAN_RSP== AdType:FF
00> 
00>  42 04 01 00 00 00 00 0B  55 55 55 55 00 DF C3 10  FF 0E 2F B7 F5 66 99 D6  54 0C EE C2 01
00> 00:07:46.5214  > Continue moving average calculation with RssiList[1]
00> 
00> > Addr:57:6A:99:32:FA:2B (Resolvable) RSSI:-49dBm(+0) raw:-49dBm
00> 
00> > Suppress re-auth: elapsed 1645ms.
00> 
00> <debug> nrf_sdh_ble: BLE event: 0x1D.
00> 
00> <debug> nrf_sdh_ble: BLE event: 0x1D.
00> 
00> <debug> nrf_sdh_ble: BLE event: 0x1D.
00> 
00> <debug> nrf_sdh_ble: BLE event: 0x1D.
00> 
00> <debug> nrf_sdh_ble: BLE event: 0x1D.
00> 
00> <debug> nrf_sdh_ble: BLE event: 0x1D.
00> 
00> <debug> nrf_sdh_ble: BLE event: 0x1D.
00> 
00> <debug> nrf_sdh_ble: BLE event: 0x1D.
00> 
00> <debug> nrf_sdh_ble: BLE event: 0x1D.
00> 
00> <debug> nrf_sdh_ble: BLE event: 0x1D.
00> 
00> <debug> nrf_sdh_ble: BLE event: 0x1D.
00> 
00> <debug> nrf_sdh_ble: BLE event: 0x1D.
00> 
00> <debug> nrf_sdh_ble: BLE event: 0x1D.
00> 
00> <debug> nrf_sdh_ble: BLE event: 0x1D.
00> 
00> <debug> nrf_sdh_ble: BLE event: 0x1D.
00> 
00> <debug> nrf_sdh_ble: BLE event: 0x1D.
00> 
00> <debug> nrf_sdh_ble: BLE event: 0x1D.
00> 
00> <debug> nrf_sdh_ble: BLE event: 0x1D.
00> 
00> <debug> nrf_sdh_ble: BLE event: 0x1D.
00> 
00> <debug> nrf_sdh_ble: BLE event: 0x1D.
00> 
00> <debug> nrf_sdh_ble: BLE event: 0x1D.
00> 
00> <debug> nrf_sdh_ble: BLE event: 0x1D.
00> 
00> <debug> nrf_sdh_ble: BLE event: 0x1D.
00> 
00> <debug> nrf_sdh_ble: BLE event: 0x1D.
00> 
00> <debug> nrf_sdh_ble: BLE event: 0x1D.
00> 
00> <debug> nrf_sdh_ble: BLE event: 0x1D.
00> 
00> <debug> nrf_sdh_ble: BLE event: 0x1D.
00> 
00> <debug> nrf_sdh_ble: BLE event: 0x1D.
00> 
00> <debug> nrf_sdh_ble: BLE event: 0x1D.
00> 
00> <debug> nrf_sdh_ble: BLE event: 0x1D.
00> 
00> <debug> nrf_sdh_ble: BLE event: 0x1D.
00> 
00> <debug> nrf_sdh_ble: BLE event: 0x1D.
00> 
00> <debug> nrf_sdh_ble: BLE event: 0x1D.
00> 
00> <debug> nrf_sdh_ble: BLE event: 0x1D.
00> 
00> <debug> nrf_sdh_ble: BLE event: 0x1D.
00> 
00> <debug> nrf_sdh_ble: BLE event: 0x1D.
00> 
00> <debug> nrf_sdh_ble: BLE event: 0x1D.
00> 
00> <debug> nrf_sdh_ble: BLE event: 0x1D.
00> 
00> <debug> nrf_sdh_ble: BLE event: 0x1D.
00> 
00> <debug> nrf_sdh_ble: BLE event: 0x1D.
00> 
00> <debug> nrf_sdh_ble: BLE event: 0x1D.
00> 
00> <debug> nrf_sdh_ble: BLE event: 0x1D.
00> 
00> <debug> nrf_sdh_ble: BLE event: 0x1D.
00> 
00> <debug> nrf_sdh_ble: BLE event: 0x1D.
00> 
00> <debug> nrf_sdh_ble: BLE event: 0x1D.
00> 
00> <debug> nrf_sdh_ble: BLE event: 0x1D.
00> 
00> <debug> nrf_sdh_ble: BLE event: 0x1D.
00> 
00> <debug> nrf_sdh_ble: BLE event: 0x1D.
00> 
00> <debug> nrf_sdh_ble: BLE event: 0x1D.
00> 
00> <debug> nrf_sdh_ble: BLE event: 0x1D.
00> 
00> <debug> nrf_sdh_ble: BLE event: 0x1D.
00> 
00> <debug> nrf_sdh_ble: BLE event: 0x1D.
00> 
00> <debug> nrf_sdh_ble: BLE event: 0x1D.
00> 
00> <debug> nrf_sdh_ble: BLE event: 0x1D.
00> 
00> <debug> nrf_sdh_ble: BLE event: 0x1D.
00> 
00> <debug> nrf_sdh_ble: BLE event: 0x1D.
00> 
00> <debug> nrf_sdh_ble: BLE event: 0x1D.
00> 
00> <debug> nrf_sdh_ble: BLE event: 0x1D.
00> 
00> > Code:55555555 UID:000-00000011 Rssi+0 Ver.1
00> 
00> 
00> ==SCAN_RSP== AdType:FF
00> 
00>  42 04 01 00 00 00 00 0B  55 55 55 55 00 1B B8 9E  63 04 B0 F4 F4 66 99 D6  55 0C EE C3 01
00> 00:07:46.7956  > Continue moving average calculation with RssiList[1]
00> 
00> > Addr:57:6A:99:32:FA:2B (Resolvable) RSSI:-48dBm(+0) raw:-48dBm
00> 
00> > Suppress re-auth: elapsed 1919ms.
00> 
00> <debug> nrf_sdh_ble: BLE event: 0x1D.
00> 
00> <debug> nrf_sdh_ble: BLE event: 0x1D.
00> 
00> <debug> nrf_sdh_ble: BLE event: 0x1D.
00> 
00> <debug> nrf_sdh_ble: BLE event: 0x1D.
00> 
00> <debug> nrf_sdh_ble: BLE event: 0x1D.
00> 
00> <debug> nrf_sdh_ble: BLE event: 0x1D.
00> 
00> <debug> nrf_sdh_ble: BLE event: 0x1D.
00> 
00> <debug> nrf_sdh_ble: BLE event: 0x1D.
00> 
00> <debug> nrf_sdh_ble: BLE event: 0x1D.
00> 
00> <debug> nrf_sdh_ble: BLE event: 0x1D.
00> 
00> <debug> nrf_sdh_ble: BLE event: 0x1D.
00> 
00> <debug> nrf_sdh_ble: BLE event: 0x1D.
00> 
00> 00:07:46.8769  > 55555555-000-00000011/71:32:4C:9B:07:56 (Resolvable) was removed from the authentication suppression list, can auth: elapsed 2000ms.
00> 
00> 
00> <debug> nrf_sdh_ble: BLE event: 0x1D.
00> 
00> <debug> nrf_sdh_ble: BLE event: 0x1D.
00> 
00> <debug> nrf_sdh_ble: BLE event: 0x1D.
00> 
00> <debug> nrf_sdh_ble: BLE event: 0x1D.
00> 
00> <debug> nrf_sdh_ble: BLE event: 0x1D.
00> 
00> <debug> nrf_sdh_ble: BLE event: 0x1D.
00> 
00> <debug> nrf_sdh_ble: BLE event: 0x1D.
00> 
00> <debug> nrf_sdh_ble: BLE event: 0x1D.
00> 
00> <debug> nrf_sdh_ble: BLE event: 0x1D.
00> 
00> <debug> nrf_sdh_ble: BLE event: 0x1D.
00> 
00> <debug> nrf_sdh_ble: BLE event: 0x1D.
00> 
00> <debug> nrf_sdh_ble: BLE event: 0x1D.
00> 
00> <debug> nrf_sdh_ble: BLE event: 0x1D.
00> 
00> <debug> nrf_sdh_ble: BLE event: 0x1D.
00> 
00> <debug> nrf_sdh_ble: BLE event: 0x1D.
00> 
00> <debug> nrf_sdh_ble: BLE event: 0x1D.
00> 
00> > Code:55555555 UID:000-00000011 Rssi+0 Ver.1
00> 
00> 
00> ==SCAN_RSP== AdType:FF
00> 
00>  42 04 01 00 00 00 00 0B  55 55 55 55 00 1B B8 9E  63 04 B0 F4 F4 66 99 D6  55 0C EE C3 01
00> 00:07:46.9318  > Continue moving average calculation with RssiList[1]
00> 
00> > Addr:57:6A:99:32:FA:2B (Resolvable) RSSI:-44dBm(+0) raw:-44dBm
00> 
00> > TicketSeedBase:(16)
00>  55 55 55 55 00 00 00 00  0B 66 8C E7 92 00 00 00
00> > AccInfo[0] Accumulate.
00> 
00> > - Authed #105 at Fri Jul 19 02:58:33 2024.
00> 
00> > - Addr: 57:6A:99:32:FA:2B (Resolvable) (000-00000011)
00> 
00> > - RSSI: -44dBm(+0)
00> 
00> > - Auth: 1
00> 
00> +++ CERT OK +++ (RSSI: -44)
00> 
00> 
00> > Device has been added to the authentication suppression list, can NOT auth.
00> 
00> 
00> 00:07:46.9358  > All Smartphone Suppression timer start.
00> 
00> *** SCAN STOP ***
00> 
00> 
00> @@@ nrf_ble_scan_stop(1041)->sd_ble_gap_scan_stop
00> UART-Tx:(41)
00>  F2 01 10 10 11 00 05 55  55 55 51 60 00 00 00 00  00 00 00 11 2C 00 00 00  00 0B FF EE C3 1B B8 9E
00>  63 04 55 55 55 55 00 01  00
00> UART-Rx:(2)
00>  F3 01
00> <debug> ser_conn: [SD_CALL]:SD_CALL_UNKNOWN
00> 
00> 00:07:47.0361  > All Smartphone Suppression timer has ended.
00> 
00> @@@ nrf_ble_scan_stop(1041)->sd_ble_gap_scan_stop
00> <debug> ble_scan: Scanning parameters have been changed successfully
00> 
00> *** SCAN START ***
00> 
00> 
00> @@@ nrf_ble_scan_stop(1041)->sd_ble_gap_scan_stop
00> <debug> ble_scan: Scanning
00> 
00> <debug> nrf_sdh_ble: BLE event: 0x1D.
00> 
00> <debug> nrf_sdh_ble: BLE event: 0x1D.
00> 
00> <debug> nrf_sdh_ble: BLE event: 0x1D.
00> 
00> <debug> nrf_sdh_ble: BLE event: 0x1D.
00> 
00> <debug> nrf_sdh_ble: BLE event: 0x1D.
00> 
00> <debug> nrf_sdh_ble: BLE event: 0x1D.
00> 
00> <debug> nrf_sdh_ble: BLE event: 0x1D.
00> 
00> <debug> nrf_sdh_ble: BLE event: 0x1D.
00> 
00> <debug> nrf_sdh_ble: BLE event: 0x1D.
00> 
00> <debug> nrf_sdh_ble: BLE event: 0x1D.
00> 
00> <debug> nrf_sdh_ble: BLE event: 0x1D.
00> 
00> <debug> nrf_sdh_ble: BLE event: 0x1D.
00> 
00> <debug> nrf_sdh_ble: BLE event: 0x1D.
00> 
00> <debug> nrf_sdh_ble: BLE event: 0x1D.
00> 
00> <debug> nrf_sdh_ble: BLE event: 0x1D.
00> 
00> <debug> nrf_sdh_ble: BLE event: 0x1D.
00> 
00> <debug> nrf_sdh_ble: BLE event: 0x1D.
00> 
00> <debug> nrf_sdh_ble: BLE event: 0x1D.
00> 
00> <debug> nrf_sdh_ble: BLE event: 0x1D.
00> 
00> <debug> nrf_sdh_ble: BLE event: 0x1D.
00> 
00> <debug> nrf_sdh_ble: BLE event: 0x1D.
00> 
00> <debug> nrf_sdh_ble: BLE event: 0x1D.
00> 
00> > Code:55555555 UID:000-00000011 Rssi+0 Ver.1
00> 
00> 
00> ==SCAN_RSP== AdType:FF
00> 
00>  42 04 01 00 00 00 00 0B  55 55 55 55 00 1B B8 9E  63 04 B0 F4 F4 66 99 D6  55 0C EE C3 01
00> 00:07:47.2010  > Continue moving average calculation with RssiList[1]
00> 
00> > Addr:57:6A:99:32:FA:2B (Resolvable) RSSI:-46dBm(+0) raw:-46dBm
00> 
00> > Suppress re-auth: elapsed 266ms.
00> 
00> 00:07:47.2018  > 1BACC0DE-50273-09551615/71:32:4C:9B:07:56 (Resolvable) was removed from the rssi moving average buffering list, calc end: elapsed 1001ms.
00> 
00> 
00> <debug> nrf_sdh_ble: BLE event: 0x1D.
00> 
00> <debug> nrf_sdh_ble: BLE event: 0x1D.
00> 
00> <debug> nrf_sdh_ble: BLE event: 0x1D.
00> 
00> <debug> nrf_sdh_ble: BLE event: 0x1D.
00> 
00> <debug> nrf_sdh_ble: BLE event: 0x1D.
00> 
00> <debug> nrf_sdh_ble: BLE event: 0x1D.
00> 
00> <debug> nrf_sdh_ble: BLE event: 0x1D.
00> 
00> <debug> nrf_sdh_ble: BLE event: 0x1D.
00> 
00> 
00> ==SCAN_RSP== AdType:FF
00> 
00>  4C 00 01 00 00 00 00 0B  55 55 00 53 00 45 00 43  4F 02 4D
00> 00:07:47.2170  > Start moving average calculation with RssiList[0]
00> 
00> > Addr:71:32:4C:9B:07:56 (Resolvable) RSSI:-42dBm(+0) raw:-42dBm
00> 
00> > No accumulated.
00> 
00> 
00> @@@ scan_connect_judgement(688)->sd_ble_gap_connect
00> +++ Connecting +++
00> 
00> 00:07:47.4373  > Auth Buzzer timeout.
00> 
00> 00:07:48.2013  > 55555555-000-00000011/57:6A:99:32:FA:2B (Resolvable) was removed from the rssi moving average buffering list, calc end: elapsed 1000ms.
00> 
00> 
00> 00:07:48.2173  > 1BACC0DE-50273-09551615/71:32:4C:9B:07:56 (Resolvable) was removed from the rssi moving average buffering list, calc end: elapsed 1000ms.
00> 
00> 
00> *** No radio notification for a while(1000ms).
00> 
00> @@@ nrf_ble_scan_stop(1041)->sd_ble_gap_scan_stop
00> <debug> ble_scan: Scanning parameters have been changed successfully
00> 
00> *** SCAN START ***
00> 
00> 
00> @@@ nrf_ble_scan_stop(1041)->sd_ble_gap_scan_stop
00> <debug> ble_scan: Scanning
00> 
00> @@@ MdlBLE_vd_AdvertisingStopReq(2666)->sd_ble_gap_adv_stop
00> @@@ ble_radio_notification_init(87)->sd_radio_notification_cfg_set
00> @@@ flags_set(309)->sd_ble_gap_adv_set_configure
00> @@@ ble_advertising_start(754)->sd_ble_gap_adv_set_configure
00> sd_ble_gap_adv_start() returned NRF_SUCCESS.00:07:48.2378  BLE_ADV_EVT_IDLE(0)
00> 00:07:48.9356  > 55555555-000-00000011/57:6A:99:32:FA:2B (Resolvable) was removed from the authentication suppression list, can auth: elapsed 2000ms.
00> 
00> 
00> *** No radio notification for a while(2000ms).
00> 
00> @@@ nrf_ble_scan_stop(1041)->sd_ble_gap_scan_stop
00> <debug> ble_scan: Scanning parameters have been changed successfully
00> 
00> *** SCAN START ***
00> 
00> 
00> @@@ nrf_ble_scan_stop(1041)->sd_ble_gap_scan_stop
00> <debug> ble_scan: Scanning
00> 
00> @@@ MdlBLE_vd_AdvertisingStopReq(2666)->sd_ble_gap_adv_stop
00> @@@ ble_radio_notification_init(87)->sd_radio_notification_cfg_set
00> @@@ flags_set(309)->sd_ble_gap_adv_set_configure
00> @@@ ble_advertising_start(754)->sd_ble_gap_adv_set_configure
00> @@@ ble_advertising_start(759)->sd_ble_gap_adv_start
00> sd_ble_gap_adv_start() returned NRF_SUCCESS.00:07:49.2378  BLE_ADV_EVT_IDLE(0)

The addition of SoftDevice events is causing a large number of logs to be output.

As an aside, this causes nrf_log to hang and the same log to be output repeatedly, so I have to manually reset it each time.
Reference link: devzone.nordicsemi.com/.../nrf_log-output-sometimes-goes-crazy
It looks like this issue is still not resolved.

The problem begins at L.322, one second before L.346 "*** No radio notification", and it seems that the SoftDevice is unresponsive and locked in response to sd_ble_gap_connect.

Please let me know if there is any other information you need.

p.s.
I tried to upload a log, but even after compressing it, I got the message "File type is not allowed" and it didn't work.
I think there is a size limit, but how large can I upload?

Best regards.

RE: Beacon output suddenly stops

Hieu — Tue, 16 Jul 2024 07:16:40 GMT

[quote user="hiroiwas"]> At the end of your SoftDevice event handler, could you please also add a default case handling where it just quickly log what the value of the event type is?

Does this mean that the following log output is enabled?

nrf_sdh_soc.c(87) NRF_LOG_DEBUG("SoC event: 0x%x.", evt_id);

Or do you mean adding an observer that logs?[/quote]

I am not certain if the event polling in nrf_sdh_soc.c is the same, but I figure it doesn't hurt to enable it too.

What I meant is to add handling to your equivalent of the ble_evt_handler in the ble_app_uart example.

Looking forwards to your further test results.

Best regards,

Hieu

RE: Beacon output suddenly stops

hiroiwas — Tue, 16 Jul 2024 04:31:15 GMT

Hello Hieu,

> Your latest log doesn't highlight what sd_ble_gap_adv_start() returns. Are we sure that it returns 0?

The return value of SoftDevice was not output.
I will add an output and check it.

> Also, I don't see a scan starting attempted.

sd_ble_gap_scan_start() is deter because of the large number of log outputs.
SD_BLE_GAP_ADV_START() is unconditionally called after nrf_ble_scan_stop() in nrf_ble_scan_start(), so I think there is no doubt.

> BLE_GAP_TIMEOUT_SRC_CONN. Has this case been checked?

On the log, it was confirmed that BLE_GAP_TIMEOUT_SRC_CONN has never occurred as an event.
Currently, BLE_GAP_TIMEOUT_SRC_CONN does nothing except to log, but is it better to cut and re -scan?

> you might want to bind RADIO events to GPIOTE output with PPI and keep track of it instead.

Does it mean monitoring the Radio status using a logic analyzer, etc., without using the Radio Notification event?

Okay, I'll try it.

> At the end of your SoftDevice event handler, could you please also add a default case handling where it just quickly log what the value of the event type is?

Does this mean that the following log output is enabled?

nrf_sdh_soc.c(87) NRF_LOG_DEBUG("SoC event: 0x%x.", evt_id);

Or do you mean adding an observer that logs?

> Also, I highly recommend using the Logger Module with NRF_LOG_DEFERRED,

Thanks for the advice, I wasn't using that, I'll set it.

> Adding timestamp, no matter how basic, also helps getting more information into the log.

Log it as NRF_LOG_INIT( app_timer_cnt_get );

I will report back if the problem occurs again.

Best regards.

RE: Beacon output suddenly stops

Hieu — Mon, 15 Jul 2024 21:55:27 GMT

Hi hiroiwas,

Thank you for the detailed information.

I highly doubt that the SoftDevice drops event, and even if it does, I strongly believe it cannot be something important like BLE_GAT_EVT_CONNECTED. Since the data you have provided so far does indicate this, I will start an internal query with the past SoftDevice team to be extra sure.

Your latest log doesn't highlight what sd_ble_gap_adv_start() returns. Are we sure that it returns 0?

Also, I don't see a scan starting attempted.

I noticed that here we have only been looking into BLE_GAP_EVT_CONNECTED. However, that is not the only outcome from a connection establishment attempt. The attempt could also fail with BLE_GAP_EVT_TIMEOUT with ble_gap_evt_timeout_t::src == BLE_GAP_TIMEOUT_SRC_CONN. Has this case been checked?

Finally, when event drops are being suspected, using Radio Notification might make the matter worse.

For the purpose of resetting the device as a fail-safe, then I agree that you need Radio Notification for that.

But for the purpose of debugging, if we consider our goal resolving the issue completely, and be free of Radio Notification, then you might want to bind RADIO events to GPIOTE output with PPI and keep track of it instead. I have some sample snippets here: https://gist.github.com/hieuv/06fb6b9609c59e3a57bb62340ccaf242.

Best regards,

Hieu

P.s: At the end of your SoftDevice event handler, could you please also add a default case handling where it just quickly log what the value of the event type is? This helps us check if there are any events we haven't accounted for.

Also, I highly recommend using the Logger Module with NRF_LOG_DEFERRED, if you are not using it yet. Adding timestamp, no matter how basic, also helps getting more information into the log.

RE: Beacon output suddenly stops

hiroiwas — Thu, 11 Jul 2024 00:37:52 GMT

Hello.

I was able to confirm No radio notification in the code that implemented the restart of SCAN and ADVERTISE.

As a result, I found that restarting at the point where the problem occurred was ineffective.

I have attached the log from just before the problem occurred below.

00> ==SCAN_RSP== AdType:FF
00> > Addr:60:4D:3C:3E:F0:A4 (Resolvable) RSSI:-41dBm(+0) raw:-41dBm
00> 
00> 
00> @@@ scan_connect_judgement(687)->sd_ble_gap_connect
00> +++ Connecting +++
00> 
00> @@@ on_connected_evt(160)->sd_ble_gattc_exchange_mtu_request
00> @@@ data_length_update(89)->sd_ble_gap_data_length_update
00> 03:29:33.6818  NRF_BLE_SCAN_EVT_CONNECTED(6)
00> Connected to target : (7)
00> 03:29:33.6830  BLE_GAP_EVT_CONNECTED(16)
00> > conn_handle=0, conn_count=1
00> 
00> @@@ request_process(359)->sd_ble_gattc_primary_services_discover
00> @@@ queue_process(215)->sd_ble_gattc_primary_services_discover
00> ble_db_discovery_start: err_code=0
00> 
00> @@@ ble_evt_handler(1050)->sd_ble_gap_rssi_start
00> [APP] CONNECTING
00> 
00> @@@ queue_process(215)->sd_ble_gattc_primary_services_discover
00> 03:29:33.7282  BLE_GATTC_EVT_EXCHANGE_MTU_RSP(58)
00> 03:29:33.7432  BLE_GAP_EVT_DATA_LENGTH_UPDATE(36)
00> @@@ request_process(366)->sd_ble_gattc_characteristics_discover
00> 03:29:33.7582  BLE_GATTC_EVT_PRIM_SRVC_DISC_RSP(48)
00> @@@ request_process(366)->sd_ble_gattc_characteristics_discover
00> @@@ request_process(372)->sd_ble_gattc_descriptors_discover
00> @@@ request_process(372)->sd_ble_gattc_descriptors_discover
00> 03:29:33.8931  BLE_DB_DISCOVERY_COMPLETE(0)
00> 03:29:33.8933  BLE_NUS_C_EVT_DISCOVERY_COMPLETE(0)
00> 03:29:33.8935  CMNBLE_vd_EventConnected(2381):  60:4D:3C:3E:F0:A4 (Resolvable)  elapsed(ms): 210
00> 
00> @@@ request_process(353)->sd_ble_gattc_write
00> 03:29:33.8943  BLE_DB_DISCOVERY_AVAILABLE(3)
00> 
00> > LOCK conn_handle=-1->0
00> 
00> @@@ request_process(353)->sd_ble_gattc_write
00> @@@ queue_process(208)->sd_ble_gattc_write
00> 
00> @@@ queue_process(208)->sd_ble_gattc_write
00> 
00> @@@ on_hvx(135)->sd_ble_gattc_hv_confirm
00> 
00> @@@ nrf_ble_scan_stop(1041)->sd_ble_gap_scan_stop
00> 
00> @@@ request_process(353)->sd_ble_gattc_write
00> 
00> @@@ MdlBLE_en_Disconnect(1801)->sd_ble_gap_disconnect
00> 
00> 03:29:34.0880  BLE_GAP_EVT_DISCONNECTED(17)
00> > conn_handle=0, conn_count=0, reason=0x16
00> 
00> @@@ ble_evt_handler(1062)->sd_ble_gap_rssi_stop
00> > UNLOCK conn_handle=0
00> 
00> @@@ nrf_ble_scan_stop(1041)->sd_ble_gap_scan_stop
00> *** SCAN START ***
00> 
00> @@@ nrf_ble_scan_stop(1041)->sd_ble_gap_scan_stop
00> 
00> 
00> ==SCAN_RSP== AdType:FF
00> > Addr:50:EE:9C:69:95:7A (Resolvable) RSSI:-47dBm(+0) raw:-47dBm
00> 
00> 
00> ==SCAN_RSP== AdType:FF
00> > Addr:50:EE:9C:69:95:7A (Resolvable) RSSI:-49dBm(+0) raw:-49dBm
00> 
00> 
00> ==SCAN_RSP== AdType:FF
00> > Addr:60:4D:3C:3E:F0:A4 (Resolvable) RSSI:-47dBm(+0) raw:-47dBm
00> 
00> 
00> ==SCAN_RSP== AdType:FF
00> > Addr:50:EE:9C:69:95:7A (Resolvable) RSSI:-50dBm(+0) raw:-50dBm
00> 
00> 
00> ==SCAN_RSP== AdType:FF
00> > Addr:60:4D:3C:3E:F0:A4 (Resolvable) RSSI:-47dBm(+0) raw:-47dBm
00> 
00> 
00> ==SCAN_RSP== AdType:FF
00> > Addr:60:4D:3C:3E:F0:A4 (Resolvable) RSSI:-41dBm(+0) raw:-41dBm
00> 
00> @@@ scan_connect_judgement(687)->sd_ble_gap_connect
00> +++ Connecting +++
00> 
00> 
00> *** No radio notification for a while(1000ms).
00> 
00> @@@ nrf_ble_scan_stop(1041)->sd_ble_gap_scan_stop
00> *** SCAN START ***
00> 
00> 
00> @@@ nrf_ble_scan_stop(1041)->sd_ble_gap_scan_stop
00> @@@ MdlBLE_vd_AdvertisingStopReq(2570)->sd_ble_gap_adv_stop
00> @@@ ble_radio_notification_init(87)->sd_radio_notification_cfg_set
00> @@@ flags_set(309)->sd_ble_gap_adv_set_configure
00> @@@ ble_advertising_start(754)->sd_ble_gap_adv_set_configure
00> @@@ ble_advertising_start(759)->sd_ble_gap_adv_start
00> 03:29:35.6688  BLE_ADV_EVT_IDLE(0)
00> 03:29:36.6667  > Auth Buzzer timeout.
00> 
00> *** No radio notification for a while(2000ms).
00> 
00> @@@ nrf_ble_scan_stop(1041)->sd_ble_gap_scan_stop
00> *** SCAN START ***
00> 
00> 
00> @@@ nrf_ble_scan_stop(1041)->sd_ble_gap_scan_stop
00> @@@ MdlBLE_vd_AdvertisingStopReq(2570)->sd_ble_gap_adv_stop
00> @@@ ble_radio_notification_init(87)->sd_radio_notification_cfg_set
00> @@@ flags_set(309)->sd_ble_gap_adv_set_configure
00> @@@ ble_advertising_start(754)->sd_ble_gap_adv_set_configure
00> @@@ ble_advertising_start(759)->sd_ble_gap_adv_start
00> 03:29:36.6688  BLE_ADV_EVT_IDLE(0)
00> *** No radio notification for a while(3000ms).
00> 
00> @@@ nrf_ble_scan_stop(1041)->sd_ble_gap_scan_stop
00> *** SCAN START ***
00> 
00> 
00> @@@ nrf_ble_scan_stop(1041)->sd_ble_gap_scan_stop
00> @@@ MdlBLE_vd_AdvertisingStopReq(2570)->sd_ble_gap_adv_stop
00> @@@ ble_radio_notification_init(87)->sd_radio_notification_cfg_set
00> @@@ flags_set(309)->sd_ble_gap_adv_set_configure
00> @@@ ble_advertising_start(754)->sd_ble_gap_adv_set_configure
00> @@@ ble_advertising_start(759)->sd_ble_gap_adv_start
00> 03:29:37.6687  BLE_ADV_EVT_IDLE(0)
00> *** No radio notification for a while(4000ms).
00> 
00> @@@ nrf_ble_scan_stop(1041)->sd_ble_gap_scan_stop
00> *** SCAN START ***
00> 
00> 
00> @@@ nrf_ble_scan_stop(1041)->sd_ble_gap_scan_stop
00> @@@ MdlBLE_vd_AdvertisingStopReq(2570)->sd_ble_gap_adv_stop
00> @@@ ble_radio_notification_init(87)->sd_radio_notification_cfg_set
00> @@@ flags_set(309)->sd_ble_gap_adv_set_configure
00> @@@ ble_advertising_start(754)->sd_ble_gap_adv_set_configure
00> @@@ ble_advertising_start(759)->sd_ble_gap_adv_start
00> 03:29:38.6688  BLE_ADV_EVT_IDLE(0)
00> *** No radio notification for a while(5000ms).
00> 
00> @@@ nrf_ble_scan_stop(1041)->sd_ble_gap_scan_stop
00> *** SCAN START ***
00> 
00> 
00> @@@ nrf_ble_scan_stop(1041)->sd_ble_gap_scan_stop
00> @@@ MdlBLE_vd_AdvertisingStopReq(2570)->sd_ble_gap_adv_stop
00> @@@ ble_radio_notification_init(87)->sd_radio_notification_cfg_set
00> @@@ flags_set(309)->sd_ble_gap_adv_set_configure
00> @@@ ble_advertising_start(754)->sd_ble_gap_adv_set_configure
00> @@@ ble_advertising_start(759)->sd_ble_gap_adv_start
00> 03:29:39.6688  BLE_ADV_EVT_IDLE(0)
00> *** No radio notification for a while(6000ms).
00> 
00> @@@ nrf_ble_scan_stop(1041)->sd_ble_gap_scan_stop
00> *** SCAN START ***
00> 
00> 
00> @@@ nrf_ble_scan_stop(1041)->sd_ble_gap_scan_stop
00> @@@ MdlBLE_vd_AdvertisingStopReq(2570)->sd_ble_gap_adv_stop
00> @@@ ble_radio_notification_init(87)->sd_radio_notification_cfg_set
00> @@@ flags_set(309)->sd_ble_gap_adv_set_configure
00> @@@ ble_advertising_start(754)->sd_ble_gap_adv_set_configure
00> @@@ ble_advertising_start(759)->sd_ble_gap_adv_start
00> 03:29:40.6690  BLE_ADV_EVT_IDLE(0)
00> *** No radio notification for a while(7000ms).
00> 
00> @@@ nrf_ble_scan_stop(1041)->sd_ble_gap_scan_stop
00> *** SCAN START ***
00> 
00> 
00> @@@ nrf_ble_scan_stop(1041)->sd_ble_gap_scan_stop
00> @@@ MdlBLE_vd_AdvertisingStopReq(2570)->sd_ble_gap_adv_stop
00> @@@ ble_radio_notification_init(87)->sd_radio_notification_cfg_set
00> @@@ flags_set(309)->sd_ble_gap_adv_set_configure
00> @@@ ble_advertising_start(754)->sd_ble_gap_adv_set_configure
00> @@@ ble_advertising_start(759)->sd_ble_gap_adv_start
00> 03:29:41.6689  BLE_ADV_EVT_IDLE(0)
00> *** No radio notification for a while(8000ms).
00> 
00> @@@ nrf_ble_scan_stop(1041)->sd_ble_gap_scan_stop
00> *** SCAN START ***
00> 
00> 
00> @@@ nrf_ble_scan_stop(1041)->sd_ble_gap_scan_stop
00> @@@ MdlBLE_vd_AdvertisingStopReq(2570)->sd_ble_gap_adv_stop
00> @@@ ble_radio_notification_init(87)->sd_radio_notification_cfg_set
00> @@@ flags_set(309)->sd_ble_gap_adv_set_configure
00> @@@ ble_advertising_start(754)->sd_ble_gap_adv_set_configure
00> @@@ ble_advertising_start(759)->sd_ble_gap_adv_start
00> 03:29:42.6689  BLE_ADV_EVT_IDLE(0)
00> *** No radio notification for a while(9000ms).
00> 
00> @@@ nrf_ble_scan_stop(1041)->sd_ble_gap_scan_stop
00> *** SCAN START ***
00> 
00> 
00> @@@ nrf_ble_scan_stop(1041)->sd_ble_gap_scan_stop
00> @@@ MdlBLE_vd_AdvertisingStopReq(2570)->sd_ble_gap_adv_stop
00> @@@ ble_radio_notification_init(87)->sd_radio_notification_cfg_set
00> @@@ flags_set(309)->sd_ble_gap_adv_set_configure
00> @@@ ble_advertising_start(754)->sd_ble_gap_adv_set_configure
00> @@@ ble_advertising_start(759)->sd_ble_gap_adv_start
00> 03:29:43.6689  BLE_ADV_EVT_IDLE(0)
00> *** No radio notification for a while(10000ms).
00> 
00> @@@ nrf_ble_scan_stop(1041)->sd_ble_gap_scan_stop
00> *** SCAN START ***
00> 
00> 
00> @@@ nrf_ble_scan_stop(1041)->sd_ble_gap_scan_stop
00> @@@ MdlBLE_vd_AdvertisingStopReq(2570)->sd_ble_gap_adv_stop
00> @@@ ble_radio_notification_init(87)->sd_radio_notification_cfg_set
00> @@@ flags_set(309)->sd_ble_gap_adv_set_configure
00> @@@ ble_advertising_start(754)->sd_ble_gap_adv_set_configure
00> @@@ ble_advertising_start(759)->sd_ble_gap_adv_start
00> 03:29:44.6689  BLE_ADV_EVT_IDLE(0)
00> *** No radio notification for a while(11000ms).
00> 
00> @@@ nrf_ble_scan_stop(1041)->sd_ble_gap_scan_stop
00> *** SCAN START ***
00> 
00> 
00> @@@ nrf_ble_scan_stop(1041)->sd_ble_gap_scan_stop
00> @@@ MdlBLE_vd_AdvertisingStopReq(2570)->sd_ble_gap_adv_stop
00> @@@ ble_radio_notification_init(87)->sd_radio_notification_cfg_set
00> @@@ flags_set(309)->sd_ble_gap_adv_set_configure
00> @@@ ble_advertising_start(754)->sd_ble_gap_adv_set_configure
00> @@@ ble_advertising_start(759)->sd_ble_gap_adv_start
00> 03:29:45.6689  BLE_ADV_EVT_IDLE(0)
00> *** No radio notification for a while(12000ms).
00> 
00> @@@ nrf_ble_scan_stop(1041)->sd_ble_gap_scan_stop
00> *** SCAN START ***
00> 
00> 
00> @@@ nrf_ble_scan_stop(1041)->sd_ble_gap_scan_stop
00> @@@ MdlBLE_vd_AdvertisingStopReq(2570)->sd_ble_gap_adv_stop
00> @@@ ble_radio_notification_init(87)->sd_radio_notification_cfg_set
00> @@@ flags_set(309)->sd_ble_gap_adv_set_configure
00> @@@ ble_advertising_start(754)->sd_ble_gap_adv_set_configure
00> @@@ ble_advertising_start(759)->sd_ble_gap_adv_start
00> 03:29:46.6689  BLE_ADV_EVT_IDLE(0)
00> *** No radio notification for a while(13000ms).
00> 
00> @@@ nrf_ble_scan_stop(1041)->sd_ble_gap_scan_stop
00> *** SCAN START ***
00> 
00> 
00> @@@ nrf_ble_scan_stop(1041)->sd_ble_gap_scan_stop
00> @@@ MdlBLE_vd_AdvertisingStopReq(2570)->sd_ble_gap_adv_stop
00> @@@ ble_radio_notification_init(87)->sd_radio_notification_cfg_set
00> @@@ flags_set(309)->sd_ble_gap_adv_set_configure
00> @@@ ble_advertising_start(754)->sd_ble_gap_adv_set_configure
00> @@@ ble_advertising_start(759)->sd_ble_gap_adv_start
00> 03:29:47.6689  BLE_ADV_EVT_IDLE(0)
00> *** No radio notification for a while(14000ms).
00> 
00> @@@ nrf_ble_scan_stop(1041)->sd_ble_gap_scan_stop
00> *** SCAN START ***
00> 
00> 
00> @@@ nrf_ble_scan_stop(1041)->sd_ble_gap_scan_stop
00> @@@ MdlBLE_vd_AdvertisingStopReq(2570)->sd_ble_gap_adv_stop
00> @@@ ble_radio_notification_init(87)->sd_radio_notification_cfg_set
00> @@@ flags_set(309)->sd_ble_gap_adv_set_configure
00> @@@ ble_advertising_start(754)->sd_ble_gap_adv_set_configure
00> @@@ ble_advertising_start(759)->sd_ble_gap_adv_start
00> 03:29:48.6689  BLE_ADV_EVT_IDLE(0)
00> *** No radio notification for a while(15000ms).
00> 
00> @@@ nrf_ble_scan_stop(1041)->sd_ble_gap_scan_stop
00> *** SCAN START ***
00> 
00> 
00> @@@ nrf_ble_scan_stop(1041)->sd_ble_gap_scan_stop
00> @@@ MdlBLE_vd_AdvertisingStopReq(2570)->sd_ble_gap_adv_stop
00> @@@ ble_radio_notification_init(87)->sd_radio_notification_cfg_set
00> @@@ flags_set(309)->sd_ble_gap_adv_set_configure
00> @@@ ble_advertising_start(754)->sd_ble_gap_adv_set_configure
00> @@@ ble_advertising_start(759)->sd_ble_gap_adv_start
00> 03:29:49.6690  BLE_ADV_EVT_IDLE(0)
00> *** No radio notification for a while(16000ms).
00> 
00> @@@ nrf_ble_scan_stop(1041)->sd_ble_gap_scan_stop
00> *** SCAN START ***
00> 
00> 
00> @@@ nrf_ble_scan_stop(1041)->sd_ble_gap_scan_stop
00> @@@ MdlBLE_vd_AdvertisingStopReq(2570)->sd_ble_gap_adv_stop
00> @@@ ble_radio_notification_init(87)->sd_radio_notification_cfg_set
00> @@@ flags_set(309)->sd_ble_gap_adv_set_configure
00> @@@ ble_advertising_start(754)->sd_ble_gap_adv_set_configure
00> @@@ ble_advertising_start(759)->sd_ble_gap_adv_start
00> 03:29:50.6690  BLE_ADV_EVT_IDLE(0)
00> *** No radio notification for a while(17000ms).
00> 
00> @@@ nrf_ble_scan_stop(1041)->sd_ble_gap_scan_stop
00> *** SCAN START ***
00> 
00> 
00> @@@ nrf_ble_scan_stop(1041)->sd_ble_gap_scan_stop
00> @@@ MdlBLE_vd_AdvertisingStopReq(2570)->sd_ble_gap_adv_stop
00> @@@ ble_radio_notification_init(87)->sd_radio_notification_cfg_set
00> @@@ flags_set(309)->sd_ble_gap_adv_set_configure
00> @@@ ble_advertising_start(754)->sd_ble_gap_adv_set_configure
00> @@@ ble_advertising_start(759)->sd_ble_gap_adv_start
00> 03:29:51.6691  BLE_ADV_EVT_IDLE(0)
00> *** No radio notification for a while(18000ms).
00> 
00> @@@ nrf_ble_scan_stop(1041)->sd_ble_gap_scan_stop
00> *** SCAN START ***
00> 
00> 
00> @@@ nrf_ble_scan_stop(1041)->sd_ble_gap_scan_stop
00> @@@ MdlBLE_vd_AdvertisingStopReq(2570)->sd_ble_gap_adv_stop
00> @@@ ble_radio_notification_init(87)->sd_radio_notification_cfg_set
00> @@@ flags_set(309)->sd_ble_gap_adv_set_configure
00> @@@ ble_advertising_start(754)->sd_ble_gap_adv_set_configure
00> @@@ ble_advertising_start(759)->sd_ble_gap_adv_start
00> 03:29:52.6690  BLE_ADV_EVT_IDLE(0)
00> *** No radio notification for a while(19000ms).
00> 
00> @@@ nrf_ble_scan_stop(1041)->sd_ble_gap_scan_stop
00> *** SCAN START ***
00> 
00> 
00> @@@ nrf_ble_scan_stop(1041)->sd_ble_gap_scan_stop
00> @@@ MdlBLE_vd_AdvertisingStopReq(2570)->sd_ble_gap_adv_stop
00> @@@ ble_radio_notification_init(87)->sd_radio_notification_cfg_set
00> @@@ flags_set(309)->sd_ble_gap_adv_set_configure
00> @@@ ble_advertising_start(754)->sd_ble_gap_adv_set_configure
00> @@@ ble_advertising_start(759)->sd_ble_gap_adv_start
00> 03:29:53.6690  BLE_ADV_EVT_IDLE(0)
00> *** No radio notification for a while(20000ms).
00> 
00> @@@ nrf_ble_scan_stop(1041)->sd_ble_gap_scan_stop
00> *** SCAN START ***
00> 
00> 
00> @@@ nrf_ble_scan_stop(1041)->sd_ble_gap_scan_stop
00> @@@ MdlBLE_vd_AdvertisingStopReq(2570)->sd_ble_gap_adv_stop
00> @@@ ble_radio_notification_init(87)->sd_radio_notification_cfg_set
00> @@@ flags_set(309)->sd_ble_gap_adv_set_configure
00> @@@ ble_advertising_start(754)->sd_ble_gap_adv_set_configure
00> @@@ ble_advertising_start(759)->sd_ble_gap_adv_start
00> 03:29:54.6690  BLE_ADV_EVT_IDLE(0)
00> *** No radio notification for a while(21000ms).
00> 
00> @@@ nrf_ble_scan_stop(1041)->sd_ble_gap_scan_stop
00> *** SCAN START ***
00> 
00> 
00> @@@ nrf_ble_scan_stop(1041)->sd_ble_gap_scan_stop
00> @@@ MdlBLE_vd_AdvertisingStopReq(2570)->sd_ble_gap_adv_stop
00> @@@ ble_radio_notification_init(87)->sd_radio_notification_cfg_set
00> @@@ flags_set(309)->sd_ble_gap_adv_set_configure
00> @@@ ble_advertising_start(754)->sd_ble_gap_adv_set_configure
00> @@@ ble_advertising_start(759)->sd_ble_gap_adv_start
00> 03:29:55.6690  BLE_ADV_EVT_IDLE(0)
00> *** No radio notification for a while(22000ms).
00> 
00> @@@ nrf_ble_scan_stop(1041)->sd_ble_gap_scan_stop
00> *** SCAN START ***
00> 
00> 
00> @@@ nrf_ble_scan_stop(1041)->sd_ble_gap_scan_stop
00> @@@ MdlBLE_vd_AdvertisingStopReq(2570)->sd_ble_gap_adv_stop
00> @@@ ble_radio_notification_init(87)->sd_radio_notification_cfg_set
00> @@@ flags_set(309)->sd_ble_gap_adv_set_configure
00> @@@ ble_advertising_start(754)->sd_ble_gap_adv_set_configure
00> @@@ ble_advertising_start(759)->sd_ble_gap_adv_start
00> 03:29:56.6691  BLE_ADV_EVT_IDLE(0)
00> *** No radio notification for a while(23000ms).
00> 
00> @@@ nrf_ble_scan_stop(1041)->sd_ble_gap_scan_stop
00> *** SCAN START ***
00> 
00> 
00> @@@ nrf_ble_scan_stop(1041)->sd_ble_gap_scan_stop
00> @@@ MdlBLE_vd_AdvertisingStopReq(2570)->sd_ble_gap_adv_stop
00> @@@ ble_radio_notification_init(87)->sd_radio_notification_cfg_set
00> @@@ flags_set(309)->sd_ble_gap_adv_set_configure
00> @@@ ble_advertising_start(754)->sd_ble_gap_adv_set_configure
00> @@@ ble_advertising_start(759)->sd_ble_gap_adv_start
00> 03:29:57.6691  BLE_ADV_EVT_IDLE(0)
00> *** No radio notification for a while(24000ms).
00> 
00> @@@ nrf_ble_scan_stop(1041)->sd_ble_gap_scan_stop
00> *** SCAN START ***
00> 
00> 
00> @@@ nrf_ble_scan_stop(1041)->sd_ble_gap_scan_stop
00> @@@ MdlBLE_vd_AdvertisingStopReq(2570)->sd_ble_gap_adv_stop
00> @@@ ble_radio_notification_init(87)->sd_radio_notification_cfg_set
00> @@@ flags_set(309)->sd_ble_gap_adv_set_configure
00> @@@ ble_advertising_start(754)->sd_ble_gap_adv_set_configure
00> @@@ ble_advertising_start(759)->sd_ble_gap_adv_start
00> 03:29:58.6691  BLE_ADV_EVT_IDLE(0)
00> *** No radio notification for a while(25000ms).
00> 
00> @@@ nrf_ble_scan_stop(1041)->sd_ble_gap_scan_stop
00> *** SCAN START ***
00> 
00> 
00> @@@ nrf_ble_scan_stop(1041)->sd_ble_gap_scan_stop
00> @@@ MdlBLE_vd_AdvertisingStopReq(2570)->sd_ble_gap_adv_stop
00> @@@ ble_radio_notification_init(87)->sd_radio_notification_cfg_set
00> @@@ flags_set(309)->sd_ble_gap_adv_set_configure
00> @@@ ble_advertising_start(754)->sd_ble_gap_adv_set_configure
00> @@@ ble_advertising_start(759)->sd_ble_gap_adv_start
00> 03:29:59.6691  BLE_ADV_EVT_IDLE(0)
00> *** No radio notification for a while(26000ms).
00> 
00> @@@ nrf_ble_scan_stop(1041)->sd_ble_gap_scan_stop
00> *** SCAN START ***
00> 
00> 
00> @@@ nrf_ble_scan_stop(1041)->sd_ble_gap_scan_stop
00> @@@ MdlBLE_vd_AdvertisingStopReq(2570)->sd_ble_gap_adv_stop
00> @@@ ble_radio_notification_init(87)->sd_radio_notification_cfg_set
00> @@@ flags_set(309)->sd_ble_gap_adv_set_configure
00> @@@ ble_advertising_start(754)->sd_ble_gap_adv_set_configure
00> @@@ ble_advertising_start(759)->sd_ble_gap_adv_start
00> 03:30:00.6691  BLE_ADV_EVT_IDLE(0)
00> *** No radio notification for a while(27000ms).
00> 
00> @@@ nrf_ble_scan_stop(1041)->sd_ble_gap_scan_stop
00> *** SCAN START ***
00> 
00> 
00> @@@ nrf_ble_scan_stop(1041)->sd_ble_gap_scan_stop
00> @@@ MdlBLE_vd_AdvertisingStopReq(2570)->sd_ble_gap_adv_stop
00> @@@ ble_radio_notification_init(87)->sd_radio_notification_cfg_set
00> @@@ flags_set(309)->sd_ble_gap_adv_set_configure
00> @@@ ble_advertising_start(754)->sd_ble_gap_adv_set_configure
00> @@@ ble_advertising_start(759)->sd_ble_gap_adv_start
00> 03:30:01.6691  BLE_ADV_EVT_IDLE(0)
00> *** No radio notification for a while(28000ms).
00> 
00> @@@ nrf_ble_scan_stop(1041)->sd_ble_gap_scan_stop
00> *** SCAN START ***
00> 
00> 
00> @@@ nrf_ble_scan_stop(1041)->sd_ble_gap_scan_stop
00> @@@ MdlBLE_vd_AdvertisingStopReq(2570)->sd_ble_gap_adv_stop
00> @@@ ble_radio_notification_init(87)->sd_radio_notification_cfg_set
00> @@@ flags_set(309)->sd_ble_gap_adv_set_configure
00> @@@ ble_advertising_start(754)->sd_ble_gap_adv_set_configure
00> @@@ ble_advertising_start(759)->sd_ble_gap_adv_start
00> 03:30:02.6692  BLE_ADV_EVT_IDLE(0)
00> *** No radio notification for a while(29000ms).
00> 
00> @@@ nrf_ble_scan_stop(1041)->sd_ble_gap_scan_stop
00> *** SCAN START ***
00> 
00> 
00> @@@ nrf_ble_scan_stop(1041)->sd_ble_gap_scan_stop
00> @@@ MdlBLE_vd_AdvertisingStopReq(2570)->sd_ble_gap_adv_stop
00> @@@ ble_radio_notification_init(87)->sd_radio_notification_cfg_set
00> @@@ flags_set(309)->sd_ble_gap_adv_set_configure
00> @@@ ble_advertising_start(754)->sd_ble_gap_adv_set_configure
00> @@@ ble_advertising_start(759)->sd_ble_gap_adv_start
00> 03:30:03.6691  BLE_ADV_EVT_IDLE(0)
00> <error> app: ERROR 0 [NRF_SUCCESS] at D:\SVN\BLE_MockUp\BLE_Module\src\Middleware\MdlBLE.c:1532
00> 
00> PC at: 0x000357CF
00> 
00> <error> app: End of error report
00> 

# Logging stopped @ 11 Jul 2024 09:00:29

When this happens, the only thing that can be thought of is that the SoftDevice is locked.

The only other thing that can be done is to try stopping and restarting the SoftDevice itself, but since it is not clear what state the SoftDevice is in, a self-reset may be more reliable.

Best regards.

RE: Beacon output suddenly stops

hiroiwas — Tue, 09 Jul 2024 09:27:14 GMT

Hello, Edvin.

> So your beacon is also scanning, and connecting to other devices from time to time, is that correct?

I have drawn a simple sequence chart of the system.

The device emits a beacon (ADV_NONCONN_IND) at a 100 ms cycle and opens a 20 ms scan window at 20 ms intervals.
The iPhone catches the device's beacon and initiates ADV_IND.
The device uses the ADV_IND and SCAN_RSP information to make a connection request.
After connecting and completing the series of communications, it disconnects from the device, resumes scanning, and waits for the next ADV_IND.

Beacons in the peripheral role are essentially configured when the device is first powered on and then continue to transmit signals continuously.

In the central role, it receives ADV_IND and SCAN_RSP and makes decisions such as connection.
Only one device can be connected at a time to the central, and a scanner that is stopped by CONNECT_IND will be restarted after disconnection.

> The advertisements have stopped at the end of this graph, right?

Yes, the radio notification stops after the graph ends and resets after 30 seconds.

> It may happen that if there are too many events coming from the softdevice, that an event is not correctly forwarded to the application.

This is important information.
Does this mean that the SoftDevice is dropping events?

If so, this could be the reason why BLE_GAP_EVT_CONNECTED is not raised and SCAN remains stopped even though SCAN is stopped after sd_ble_gap_connect.

However, why does the peripheral role beacon also stop working at the same time?

Since it's not something the app can fix, I think this may be a problem with the SoftDevice.

Are there any parameters to increase the SoftDevice's event queue, or options to limit the types of events that are fired?

> When this occurs, and you detect it, is it still possible to start advertising? What happens if you try to do so?

Since we believe the issue is with the SoftDevice, we've set it to auto-reset if radio notifications stop for 30 seconds as a safety measure.

In addition to this, if radio notifications stop for 1 second, we'll try to resume scanning and advertising if that's possible to recover.

It will take some time to reproduce, but I will report the results once they are known.

Best regards.

RE: Beacon output suddenly stops

Edvin — Fri, 05 Jul 2024 10:31:21 GMT

Hello,

Sorry for the late reply. Einar is out of of office, so I will try to look into it.

I am sorry, but I am a bit confused (and there is a lot of information here). So your beacon is also scanning, and connecting to other devices from time to time, is that correct?

Will another central also connect to the beacon's advertisement at some point?

How do you set up your advertising? Does it restart advertising after a connection?

[quote user="hiroiwas"]This means that the ad will likely continue playing until the radio is turned off.[/quote]

What do you mean by this? The advertisements have stopped at the end of this graph, right?

[quote user="hiroiwas"]Judging from these logs, I think the reason why radio stopped is due to a conflict within SoftDevice. Is this correct?[/quote]

It may happen that if there are too many events coming from the softdevice, that an event is not correctly forwarded to the application. In some cases, that can be e.g. the disconnected event.

When this occurs, and you detect it, is it still possible to start advertising? What happens if you try to do so?

Best regards,

Edvin

RE: Beacon output suddenly stops

hiroiwas — Thu, 04 Jul 2024 04:58:14 GMT

Additional information.

After that, I added logs and tried to reproduce the issue several times, and all of the logs at the time the problem occurred recorded that there were no subsequent BLE events after sd_ble_gap_scan_start after sd_ble_gap_connect.

Also, in the radio notification event received at the same time, the argument radio_active was notified as 0 twice, and radio_active did not become 1 after that.

I thought that in Radio Notification, radio_active would alternate between 0 and 1, but is that wrong?

I have attached a chart capturing radio_active just before the radio stopped using data sampling.

This means that the ad will likely continue playing until the radio is turned off.

Judging from these logs, I think the reason why radio stopped is due to a conflict within SoftDevice. Is this correct?

Best regards.

RE: Beacon output suddenly stops

hiroiwas — Mon, 01 Jul 2024 09:29:17 GMT

Hello, Einar.

I was able to capture the log when the phenomenon occurred.

In conclusion, the point at which advertising stopped and switched is unrelated, and the problem appears to be occurring due to the overlapping timing of sd_ble_gap_scan_start during nrf_ble_scan_on_adv_report and the stop of scanning in the SoftDevice after sd_ble_gap_connect.

00> @@@ nrf_ble_scan_on_adv_report(949)->sd_ble_gap_scan_start
00> @@@ nrf_ble_scan_on_adv_report(949)->sd_ble_gap_scan_start
00> @@@ nrf_ble_scan_on_adv_report(949)->sd_ble_gap_scan_start
00> @@@ nrf_ble_scan_on_adv_report(949)->sd_ble_gap_scan_start
00> 
00> ==SCAN_RSP== AdType:FF
00>  4C 00 01 00 00 00 00 0B  55 55 00 53 00 45 00 43  4F 02 4D
00> 
00> > Addr:59:D7:5E:C8:C4:3B (Resolvable) RSSI:-36dBm(+0) raw:-36dBm
00> 
00> @@@ scan_connect_judgement(661)->sd_ble_gap_connect
00> +++ Connecting +++
00> 
00> @@@ nrf_ble_scan_on_adv_report(949)->sd_ble_gap_scan_start
00> 
00> 01:12:37.9172  > 1BACC0DE-50273-09551615/59:D7:5E:C8:C4:3B (Resolvable) was removed from the rssi moving average buffering list, calc end: elapsed 1000ms.
00> 
00> *** No radio notification for a while(30000ms).

So here are my questions.

In SoftDevice, the scanner stops after sd_ble_gap_connect, but at what point does it stop?
After CONNECT_IND? After that delay?

The scanner also stops at BLE_GAP_EVT_ADV_REPORT when an advertisement is received, but at what point does it stop?
After receiving ADV_IND? Before the ADV_REPORT event?

From the log, nrf_ble_scan_on_adv_report appears only once after sd_ble_gap_connect, but when is scan_start accepted here?

For your reference, we have attached below the logs showing successful connection and disconnection.

00> @@@ nrf_ble_scan_on_adv_report(949)->sd_ble_gap_scan_start
00> @@@ nrf_ble_scan_on_adv_report(949)->sd_ble_gap_scan_start
00> @@@ nrf_ble_scan_on_adv_report(949)->sd_ble_gap_scan_start
00> @@@ nrf_ble_scan_on_adv_report(949)->sd_ble_gap_scan_start
00> @@@ nrf_ble_scan_on_adv_report(949)->sd_ble_gap_scan_start
00> 
00> ==SCAN_RSP== AdType:FF
00> 
00>  4C 00 01 00 00 00 00 0B  55 55 00 53 00 45 00 43  4F 02 4D
00> 01:12:28.2542  > Continue moving average calculation with RssiList[0]
00> 
00> > Addr:59:D7:5E:C8:C4:3B (Resolvable) RSSI:-37dBm(+0) raw:-37dBm
00> 
00> @@@ scan_connect_judgement(661)->sd_ble_gap_connect
00> +++ Connecting +++
00> 
00> @@@ nrf_ble_scan_on_adv_report(949)->sd_ble_gap_scan_start
00> @@@ on_connected_evt(160)->sd_ble_gattc_exchange_mtu_request
00> @@@ data_length_update(89)->sd_ble_gap_data_length_update
00> 01:12:28.4926  NRF_BLE_SCAN_EVT_CONNECTED(6)
00> > conn_handle=0, conn_count=1
00> 
00> @@@ request_process(359)->sd_ble_gattc_primary_services_discover
00> @@@ queue_process(215)->sd_ble_gattc_primary_services_discover
00> ble_db_discovery_start: err_code=0
00> 
00> @@@ ble_evt_handler(1024)->sd_ble_gap_rssi_start
00> 
00> @@@ queue_process(215)->sd_ble_gattc_primary_services_discover
00> 01:12:28.5242  BLE_GATTC_EVT_EXCHANGE_MTU_RSP(58)
00> 01:12:28.5392  BLE_GAP_EVT_DATA_LENGTH_UPDATE(36)
00> @@@ request_process(366)->sd_ble_gattc_characteristics_discover
00> 01:12:28.5692  BLE_GATTC_EVT_PRIM_SRVC_DISC_RSP(48)
00> @@@ request_process(366)->sd_ble_gattc_characteristics_discover
00> @@@ request_process(372)->sd_ble_gattc_descriptors_discover
00> @@@ request_process(372)->sd_ble_gattc_descriptors_discover
00> 01:12:28.7040  BLE_DB_DISCOVERY_COMPLETE(0)
00> 01:12:28.7043  BLE_NUS_C_EVT_DISCOVERY_COMPLETE(0)
00> 
00> @@@ request_process(353)->sd_ble_gattc_write
00> 01:12:28.7052  BLE_DB_DISCOVERY_AVAILABLE(3)
00> [APP] CONNECTED
00> 

    === Part omitted ===

00> @@@ MdlBLE_en_Disconnect(1711)->sd_ble_gap_disconnect
00> 01:12:29.8440  BLE_GAP_EVT_DISCONNECTED(17)
00> > conn_handle=0, conn_count=0, reason=0x16
00> 
00> @@@ ble_evt_handler(1036)->sd_ble_gap_rssi_stop
00> 
00> @@@ MdlBLE_vd_ConnectStartReq(1614)->sd_ble_gap_tx_power_set
00> @@@ nrf_ble_scan_stop(1041)->sd_ble_gap_scan_stop
00> *** SCAN START ***
00> 
00> @@@ nrf_ble_scan_stop(1041)->sd_ble_gap_scan_stop
00> 
00> @@@ nrf_ble_scan_on_adv_report(949)->sd_ble_gap_scan_start
00> @@@ nrf_ble_scan_on_adv_report(949)->sd_ble_gap_scan_start
00> @@@ nrf_ble_scan_on_adv_report(949)->sd_ble_gap_scan_start
00> @@@ nrf_ble_scan_on_adv_report(949)->sd_ble_gap_scan_start
00> @@@ nrf_ble_scan_on_adv_report(949)->sd_ble_gap_scan_start

Does this information help you determine the cause and solution?

Incidentally, the beacon (ADV_NONCONN_IND) has not been stopped since the parameters were first set, so it was not possible to determine from the log when it stopped.

Best regards.

RE: Beacon output suddenly stops

Einar Thorsrud — Thu, 27 Jun 2024 13:16:20 GMT

[quote user="hiroiwas"]I have already done that, and the log I attached last time shows that no events occurred in radio notification for 30 seconds.[/quote]

I see. Then that could work as a recovery mechanism: if you have not had an radio activity for x amount of time, reset to recover. But this will not address the root cause.

[quote user="hiroiwas"]Just to be sure, I will set the advertising output of the beacon (ADV_NONCONN_IND) and then check whether the issue occurs with software that does not stop it thereafter.[/quote]

Sond good. Becase advertising really should not stop by itself (unless there is a timeout, and if so you get an event). And I have not found any other reports of such an issue. So I thin it is hightly likely that the application disables or does not restart advertising when it should.

[quote user="hiroiwas"]Is there anything else I can do besides monitoring the radio notification to get a clue as to what might be going on?[/quote]

I would add extensive loggin everywhere you stop or start advertising, and also log all BLE events (you oculd just log the event ID). Then you could go back in the log when advertising unexpectedly stopped and check if there are any events you did not process or if you did stop (or not start) advertising undexpectedly.

RE: Beacon output suddenly stops

hiroiwas — Thu, 27 Jun 2024 08:05:26 GMT

Hello, Einar.

> To monitor, perhaps you could use an app_timer instance and reset it every time you get a radio notification?

I have already done that, and the log I attached last time shows that no events occurred in radio notification for 30 seconds.

> if there could be an issue with advertising not being restarted after a timeout or a connection (attempt). Could that be the case?

There are places where advertising is stopped for switching, but when I check the log, it appears that it is not passing through there.

Just to be sure, I will set the advertising output of the beacon (ADV_NONCONN_IND) and then check whether the issue occurs with software that does not stop it thereafter.

The SoftDevice is designed to stop scanning on connection, but resume scanning on disconnection.

Is there anything else I can do besides monitoring the radio notification to get a clue as to what might be going on?

Best regards.

RE: Beacon output suddenly stops

Einar Thorsrud — Mon, 24 Jun 2024 14:19:57 GMT

Hi,

[quote user="hiroiwas"]I believe that asserts that occur in SoftDevices can be trapped with interrupts such as app_error_handler or HardFault, just like in the SDK. Is that correct?[/quote]

Yes, that is correct. If you are using the default error handler from the SDK, this would also detect SoftDevice asserts (and reset for release builds).

[quote user="hiroiwas"]The main loop continues to run after the Radio Notification has stopped, and still responds on the UART, so it doesn't appear to be asserted.[/quote]

I see.

[quote user="hiroiwas"]What should I monitor to determine why Radio Notification has stopped?[/quote]

To monitor, perhaps you could use an app_timer instance and reset it every time you get a radio notification? If it does not come in time, the app_timer would time out and the timeout handler run, so that you can handle it.

[quote user="hiroiwas"]How can I tell if the SoftDevice is working properly?[/quote]

Generally, the SoftDevice shoud not behave like this. You can moitor things like I sugested, and reset if you detect an issue (you could also use a watchdog instead of a n app timer, and feed the watchdog every time you get a radio notification. And if not, the watchdog would reset).

[quote user="hiroiwas"]Can you get an idea of the operating status of the SoftDevice from the RAM dump I sent?[/quote]

No, unfortuantely not.

[quote user="hiroiwas"]

To be honest, I'm at a loss as to what to do, so I'd appreciate it if you could tell me how to identify the cause.

[/quote]

This is not an expected issue. To be honest, the SoftDevice is very well tested I do not recall seeing such an issue before. I would be interested in a verification that your application does not stop advertising, or alternatively, if there could be an issue with advertising not being restarted after a timeout or a connection (attempt). Could that be the case? Have you done anything to verify this?

RE: Beacon output suddenly stops

hiroiwas — Sat, 22 Jun 2024 01:10:26 GMT

Hello, Einar.

I confirmed that there is no radio activity by checking that the radio notification event callback is not received for a certain period of time (30sec).

I believe that asserts that occur in SoftDevices can be trapped with interrupts such as app_error_handler or HardFault, just like in the SDK. Is that correct?

The main loop continues to run after the Radio Notification has stopped, and still responds on the UART, so it doesn't appear to be asserted.

What should I monitor to determine why Radio Notification has stopped?

How can I tell if the SoftDevice is working properly?

Can you get an idea of the operating status of the SoftDevice from the RAM dump I sent?

To be honest, I'm at a loss as to what to do, so I'd appreciate it if you could tell me how to identify the cause.

Best regards.