BLE_GATTC_EVT_TIMEOUT: Possible causes?

RE: BLE_GATTC_EVT_TIMEOUT: Possible causes?

Marius Heil — Fri, 24 Feb 2017 13:40:05 GMT

I don' use authorization. I will open a support case for this in a few minutes. It is really hard to simplify the code but I could point you to the classes that make are involved in this incident.

RE: BLE_GATTC_EVT_TIMEOUT: Possible causes?

Hung Bui — Fri, 24 Feb 2017 13:22:01 GMT

Hi Marius,

The issue you described here might not be the same as what you observed in the original question. In the trace you provided in the question the softdevice actually stopped sending packet when in your newer trace the softdevice still sent empty packet until the connection terminated.

They may not be the same problem.

Regarding the new trace. I do agree that GATTC_EVT_TIMEOUT is unexpected and could be an issue with softdevice. I suspect it could be confused when there are multiple connections and are doing lots of reliable write.

Do you use authorization for handle 0x000e or not ? If you use authorization, we can think of an issue on the application code that delayed the reply for the write request.

About those packets that you're suspecting:

61 <===> 53

Packet 15363: We don't see a response for the write request earlier could be because it was sent in the previous connection event but the sniffer couldn't capture it. You can see the delta time of 20ms instead of 10ms between 15365 and 15364. Implying that there must be a packet in between. The SN and NESN also confirm this. More likely just the sniffer couldn't capture it.

But I do see the connection termination came 30 seconds after these packets, there must be something wrong there.

54 <==> 507

Another glitch with the sniffer I suspect, the "Connection oriented channel" packet actually just a corrupted Write response. You can see the CRC mismatch and the content of this packet is almost identical with the normal write response. SN and NESN doesn't show that there was retransmission or anything.

But then again the connection terminated 30 seconds after this, it's hard to explain.

We may need to have your code to test here, I can setup 8 beacons. I would prefer to have a simplified version of your code that can show the issue, so it's easier to dig up the code.

RE: BLE_GATTC_EVT_TIMEOUT: Possible causes?

Marius Heil — Fri, 24 Feb 2017 11:57:54 GMT

I added an answer instead of a comment, i hope that's ok.

RE: BLE_GATTC_EVT_TIMEOUT: Possible causes?

Marius Heil — Fri, 24 Feb 2017 10:48:54 GMT

Alright, I have to add an "answer" to describe the problem because the comments are too restricted.

I did a test with 8 devices that form a scatternet and they are all working normally. The error happens on all of these dongles. I used S130 v2.0.1
I logged all output from these 8 nodes on UART
I used 4 Sniffers to log connections
I managed to catch a few connections where the failure appeared.
I did not test with only two devices because I think it will take a very long time and maybe the failure will never appear.
Please forget about the sniffer trace discussed above because it might be another issue.

61 <===> 53

This is Log A from node 61 (Master, BBBBP, DD:D3:16:3A:6C:1B) This node has another connection to Node 62 which is not shown.

This is Log B from node 53 (Slave, BBBBF, EF:45:B0:9A:69:77) This node has no other connections.

This is the Wireshark log from this connection:

Please note that there is an offset of about 18 seconds between my log and wireshark. But the screenshots show the same events.

Found in Beaconlog
- @708sec, node 61 stops sending packets and has a GATTC_EVT_TIMEOUT 30 sec later because of no ACK
- Last packet from 61 to 53 is 69:3E:00:00:00:03:02:04:C1:01:AB:F8:01:BE:FB:01:AF:FC:01:BD
- The next packet (69:C9:01:B1:A3:01:BB:3D:00:B7:C0:01:C1:C8:01:AD:F7:01:BE:F9) is not ACKED
Conclusion from Wireshark dump:
- Packet 15363 from Master is a WRITE_REQ but the SoftDevice does not wait for an Ack
- Packet 15365 from Master is the next WRITE_REQ with the next data, but should not have been sent
- Slave sends an ACK in 15368
- Master sends another WRITE_REQ in 15396 which is also the last packet that is reported as sent by the stack, but an ACK is never received

I attached the Wireshark log here:53 BBBBF EF.45.pcapng

Could this error result from my code or is this an error from the SoftDevice?

I could send you my firmware, but you need a few beacons (I used 8) to reproduce the error in a short time. I had 8 disconnections of this type in 30 minutes. I am currently tracking down another of these losses to find out if it follows the exact same pattern.

EDIT: I was able to track down another incident. The beacon logs look the same, at one point, the communication stalls. In the Wireshark log, it looks similar but there is a packet that was not seen in the other dump:

54 <==> 507

Here the download: 507 BBBTK D0.9C.pcapng

Here are the notes:

packet 21252@1072sec: CONNECT REQ
packet 36095@1800sec: Master sends another WRITE_REQ without having received a WRITE_RSP
- Instead the Slave sent a Connection oriented channel??
packet 36619@1830sec: LL_TERMINATE which was triggered by my implementation as a reaction to the GATTC_EVT_TIMEOUT
Found in log
- 54 is connected to 508 additionaly
- 507 is connected to 61 additionaly
- 54 receives the GATTC_EVT_TIMEOUT

For the sake of completenes, here are my node logs:

Node 54:

Node 507:

RE: BLE_GATTC_EVT_TIMEOUT: Possible causes?

Hung Bui — Thu, 23 Feb 2017 09:40:21 GMT

Hi Marius,

Could you summarize the issue to make it a little bit clearer.

We are dealing with the issue with just one particular board or it happens on other boards also ? How often do you see the issue ?

We need to narrow down the issue and find a way to reproduce it in house here.

If you test with only 1 pair of devices (one connection at a time) can you reproduce the issue ?

Please try to let the disconnection occur normally on the peripheral side and check the disconnected reason.

If you have, please send other sniffer trace that captured the issue when it happens. Then we could be able to find a pattern. I suspecting it was the issue when a write response comes with a write request in a same connection event.

Regarding your question, if you are too slow to pull all event from event buffer, the softdevice will NACK the packet. But there will still be packets from slave, not just stalled like what we see here.

RE: BLE_GATTC_EVT_TIMEOUT: Possible causes?

Marius Heil — Wed, 22 Feb 2017 20:50:55 GMT

What would happen if I were to slow to fetch all events from the event buffer? Would a BLE_GATTS_EVT_WRITE get lost inbetween and would never be answered by the SoftDevice? I have added two logs here from two connected nodes. Each of the nodes has other connections, but I filtered the log to show only the communication between the two. (TX_COMPLETE is generic though).

Log A: http://i.imgur.com/bgdIyoN.png

Log B: http://i.imgur.com/Bi7FMXZ.png

At one point, node A sends a packet (marked with (4) at the beginning of the line), but node B does never receive this packet. Nor does it receive any BLE_GATTS_EVT_WRITE event. The WRITE_RSP after (4) is just the current event that is being processed, it is not a write response to this very packet. I read all coming events.

After this, the connection is ended 30 seconds later with a BLE_GATTC_EVT_TIMEOUT. Any clues how I could proceed?

RE: BLE_GATTC_EVT_TIMEOUT: Possible causes?

Marius Heil — Wed, 22 Feb 2017 17:35:30 GMT

Nothing special from the outside, a developer dongle just like the others, but even after reflashing, it has these kind of connections problems which I cannot figure out.

I only have to plug it in and have another beacon around and they will connect, but the connection will break.

I will test with one pair of devices tomorrow and post the results. I do the disconnection manually after two seconds because that is part of the handshake. So I get local and remote host disconnect. I could remove this timeout and see what the disconnect reason will be, but I gues it will be a GATTC_EVT_TIMEOUT, the same problem that I have with my other beacons.

I do not use slave latency.

I do not use special interrupts, I use Uart and Button interrupts, but these are lower priority.

I do also pull all events and other following events are processed happily. I have many logs now which I am analyzing.

RE: BLE_GATTC_EVT_TIMEOUT: Possible causes?

Hung Bui — Wed, 22 Feb 2017 12:58:54 GMT

Hi Marius,

You mentioned I do have one beacon that will produce a similar fault very realiable what is special aabout that beacon ?

Is it easy to reproduce the issue on the beacon ?

If you only test with 1 pair of device would that happen?

If there is no assertion or hardfault, I assume you will receive DISCONNECTED event , could you check which is the disconnected reason?

Do you have any slave latency on the peripheral side ?

Do you have any interrupt or other thing that may block the BLE events to be handled in the application ? If you failed in pulling out the BLE event from the stack, it may run out of buffer and can't process the write request.

Regarding your question, the softdevice always prepares what it plan to send in the next connection event prior to the event. In this case, the slave has nothing to send to the master, so it simply sends empty PDU. You can find which packet are in same connection event by looking either at the channel the packets are in, or you can look at the delta time end to start between packets.

If you can provide a simplified code that can reproduce the issue, it would be very helpful

RE: BLE_GATTC_EVT_TIMEOUT: Possible causes?

Marius Heil — Wed, 22 Feb 2017 10:35:24 GMT

Hi, i uploaded it here:

www112.zippyshare.com/.../file.html

The slave did not crash, there was no hardfault or anything. I had my UART logger open. The slave ran into a handshake timeout (the packets you are seeing are my custom handshake) and then it continued to connect to other devices with the same result.

Interesting, why are 154 and 155 in the same event if only an empty PDU is sent? I use one handle for communication purpose. Whatever is written to this handle is used as a kind of serial-data transfer. It will trigger a handler on the node that will process the message.

RE: BLE_GATTC_EVT_TIMEOUT: Possible causes?

Hung Bui — Wed, 22 Feb 2017 10:28:47 GMT

Hi Marius,

Could you upload the full trace ?

From the screenshot what I can see is that the Slave stalled after the a few ms after the request. I suspecting a hardfault crashed the Slave and it might be reset at that point already.

EmptyPDU at packet 155 is normal. Not that #154 and #155 are in the same connection event. The slave can only response to the write request in the next connection event.

There is one interesting thing is that #154 and #161 both are write request to handle 0x0e but one is to the handle on the server on the Slave and the other is to the server on the Master, respectively.

I would need the full sniffer trace to investigate more. I suggest you to check if there is any assertion or hardfault occurs.