Connection failure - device gets stuck sending a notification in an infinite loop

RE: Connection failure - device gets stuck sending a notification in an infinite loop

Susheel Nuguru — Mon, 11 Mar 2019 08:19:00 GMT

I tried to run your example again and I can see that 50k bytes of data has been transferred by your send_notify_task_func. Look at the PC value that came out of the while(n > 0) loop below

Doesn't this mean that the data is being sent by notifications?

The connection does not fail either.

RE: Connection failure - device gets stuck sending a notification in an infinite loop

Paolo D — Thu, 07 Mar 2019 19:07:13 GMT

Hello,

I did apply the nrf_sdh_freertos.c bug fix to the SDK and that did not fix the problem.

I have also encountered the same problem again, this is what has changed since my previous report:

Running my application on a custom nRF52832 board instead of PCA10040
Application is sending actual data read out of external flash memory instead of dummy data like my simplified error_demo example
SDK is updated to nRF5_SDK_15.3.0_59ac345 (now includes the nrf_sdh_freertos.c bug fix)
SoftDevice is S132 v6.1.1
Toolchain is GCC ARM Embedded 7 2018-q2-update running on Windows 10
pc-ble-driver-js is v2.5.1 running under nodejs 64-bit v10.15.1

The problem still only occurs when the debugger is disconnected. Connecting the J-Link and launching the RTT Viewer fixes the problem.

I have attached two more nRF Sniffer logs that show the link layer getting into a bad state:

devzone.nordicsemi.com/.../fail1.pcapng

In fail1.pcapng, everything is working properly until packet 11003. Before that point there are a lot of lost packets being re-sent but they always get through eventually.

Packet 11004 is the first time the slave sends the packet (notification whose payload starts with 0x3a) which will eventually be re-sent in an infinite loop. It has SN=1 and NESN=0.

In packet 11005 the master NACKs it by responding with SN=1 and NESN=1.

In packet 11006 the slave resends it with SN=1 and NESN=0.

In packet 11007 the master ACKs it by responding with SN=0 and NESN=1.

In packet 11008 the slave resends it again even though the master just ACKed, and from this point onwards the slave just keeps sending the same notification again in an infinite loop.

The slave keeps sending the same packet, with SN=1 and NESN toggling between 0 and 1. The master will not ACK the packet by responding with NESN=0. It always responds with NESN=1 indicating that it wants the slave to resend the packet.

devzone.nordicsemi.com/.../fail2.pcapng

In fail2.pcapng everything works until packet 3106.

Packet 3106 is the first time the slave sends the notification (payload begins with 0x0e) which will be re-sent in an infinite loop.

In packet 3107 the master NACKs it.

The sniffer shows packet 3108 as invalid CRC. It looks like a bit got flipped in the ATT opcode changing it from 0x1b to the invalid value 0x1a.

In packet 3109 the master NACKs packet 3108 (?)

In packet 3110 the slave retransmits packet 3106 again and from this point onwards the devices are stuck in an infinite loop.

From packet 3110 onwards the slave keeps sending the same packet over and over again, always with SN=0 and toggling NESN between 0 and 1. The master keeps responding, each time with its SN matching the previous slave packet's NESN. But it never acknowledges the packet by changing its NESN to 1, it always sends NESN=0 telling the slave to re-send it.

In both these cases it looks like the problem exists within the master, not the slave. The slave keeps sending the same packet over and over again, which looks valid according to the sniffer. And in the packets it sends, the master is correctly setting its SN bit according to the last NESN bit from the slave, so it seems like the master must be seeing the packets from the slave. But the master will not acknowledge the packet by setting its NESN bit to the opposite of the SN bit in the packet the slave keeps sending repeatedly.

Maybe the problem is similar to this one? https://devzone.nordicsemi.com/f/nordic-q-a/17797/sniffer-detects-ll-infinite-loop

RE: Connection failure - device gets stuck sending a notification in an infinite loop

Susheel Nuguru — Mon, 29 Oct 2018 07:37:51 GMT

In the post i mentioned to replace xTaskResumeFromISR/vTaskSuspend in softdevice_task with vTaskNotifyGiveFromISR/ulTaskNotifyTake. Did you do that? Because I cannot reproduce any disconnects with this.

RE: Connection failure - device gets stuck sending a notification in an infinite loop

Susheel Nuguru — Wed, 24 Oct 2018 16:54:50 GMT

OK, if you get a disconnected event, then this is definitely different. I still recommend you not to use suspend/resume as there is definitely a corner case there and you should not take chances.

I will give it a try with your project to see if i can reproduce it, if I did, then it is easier for me to debug.

RE: Connection failure - device gets stuck sending a notification in an infinite loop

Paolo D — Wed, 24 Oct 2018 14:17:18 GMT

It still behaves exactly the same with that fix. When I was experimenting yesterday I had also tried replacing vTaskSuspend(NULL) with vTaskDelay(1) in softdevice_task and that also didn't make any difference.

The application isn't deadlocked, because even after it gets into the bad state where the connection is stuck, if the connection is lost the application does get the BLE_GAP_DISCONNECTED_EVT event from the stack.

RE: Connection failure - device gets stuck sending a notification in an infinite loop

Susheel Nuguru — Wed, 24 Oct 2018 13:16:48 GMT

just found one race condition that could cause deadlock in the application.

Looking at your description it seems possible that this case is hit by it if we manage to loose one softdevice notifying an event.

Please apply the solution given in this post and see if the race condition is removed.