Reponse time of WRITE WITH RESPONSE command

RE: Reponse time of WRITE WITH RESPONSE command

bjorn-spockeli — Wed, 14 Nov 2018 14:49:59 GMT

Happy to hear that you're not getting the events in close succession and I apologize that I didnt see the cause of the issue until now :)

RE: Reponse time of WRITE WITH RESPONSE command

Tielman — Wed, 14 Nov 2018 14:42:37 GMT

Great, thanks! Changing NRF_SDH_BLE_GAP_EVENT_LENGTH to 5 does force tevent=6.25ms and the events are delivered in close succession.

Thanks! This marathon session left me with traces coming out of my ears, but I am glad it is resolved 🙂

Best Regards

Tielman

RE: Reponse time of WRITE WITH RESPONSE command

bjorn-spockeli — Wed, 14 Nov 2018 14:23:56 GMT

Yes, I think that you are absolutly correct. I just assumed that there were packets missing, i.e. that we didn't see the empty packets transmitted from the central in the trace.

So what I think is going on is that SoftDevice scheduler on the Central is allocating the maximum event length for each connection, i.e. connection event length == connection interval, since the NRF_SDH_BLE_GAP_EVENT_LENGTH is set to 320(400ms). If we take a look at Multilink scheduling with eight connections as a Central and minimum intervall, then we see that the minimum connection interval for C0( and other 7 connections) will be 8*connection event length, which in your case will be 8*30ms = 240ms

So setting the NRF_SDH_BLE_GAP_EVENT_LENGTH to 5, i.e. a connection event length of 6.25ms, the minimum connection interval is 5*1.25ms*8=50ms, should resolve the issue.

Best regards

Bjørn

RE: Reponse time of WRITE WITH RESPONSE command

Tielman — Wed, 14 Nov 2018 13:12:25 GMT

Hi Bjørn

Thanks for the detailed response.

From my side, I will change NRF_SDH_BLE_GAP_EVENT_LENGTH to 6 and see what happens to the timing. It will be great if the fix is that simple and I'll let you know what the result is.

[quote userid="7571" url="~/f/nordic-q-a/40044/reponse-time-of-write-with-response-command/157276"]However, the sniffer trace indicates that the Write response is sent after 6-8 connection events, so it is either the peripheral that takes longer to send the Write Response or it is missing 6-8 connection events[/quote]

Forgive me, but I believe you are not interpreting the trace correctly. A WRITE_RESPONSE is never sent from a peripheral as the first comms on the connection event. It is always after the master sends an Data Channel PDU to start the communications. As per BLE spec Vol 6 Part B 4.5.1:

"The start of a connection event is called an anchor point. At the anchor point, a
master shall start to transmit a Data Channel PDU to the slave. ... The slave
listens for the packet sent by its master at the anchor point."

What you pointed out with no packets being sent between the master or slave for the 6-8 connection events points to the softdevice not sending the Data Channel PDU as per Spec:

"Each connection event contains at least one packet sent by the master."

Here is a breakdown per packet from 1302 in the trace

(Connection EVT 1)

1302 - Master send Write Request

1303 - Slave ack in same connection event

No data for 6-8 connection events

(Connection EVT x)

1304 - Master initiales comms after 300ms with the empty PDU

1305 - Slave send WR_RSP in same connection event

You can look through the entire trace, but it is always the master that initiates the communications on a new connection event. If the master does not, the peripheral does not respond. You can therefor see that it is the master that starts the comms (after skipping 6-8 connection events) only 300ms later for the peripheral to then send the WR_RSP.

Regards

Tielman

RE: Reponse time of WRITE WITH RESPONSE command

bjorn-spockeli — Wed, 14 Nov 2018 12:05:56 GMT

Hi Tielman,

the event counter in the sniffer trace shows that the Write Response is sent from the peripheral 6-8 connection events, i.e. 180 msec(6*30) to 240 msec (8*30) after the Write Request is sent from the central. As you have pointed out he peripheral timing should not change when additional peripherals are connected to the central as the central schedules the connection events for each connection so that all connections have communicate with the central every 30msec. However, the sniffer trace indicates that the Write response is sent after 6-8 connection events, so it is either the peripheral that takes longer to send the Write Response or it is missing 6-8 connection events. The latter should be visible in the sniffer trace, i.e. we should see multiple Write Response packets for each Write Request.

From the first comment (Nov 1st) it seems that with one peripheral connected the Write Response is sent on the next connection event, i.e. 30msec after the Write Request is sent from the central. The connection event length (NRF_SDH_BLE_GAP_EVENT_LENGTH) defines the maximum length of a connection event in 1.25msec units, i.e. if its set to 320, then the maximum connection event length is 400msec. However, the connection event will close when the transmit buffer is empty, i.e if there is only one packet to be sent, then the connection event is ended and the central moves on to the next connection event for the next peripheral.

The write request is a total of 49 bytes on-air, which at 1Mbps will take 390us, so sending 8 Write Requests will take 3.1360 msec , in addition you need to add the time it takes to change between TX and RX for each event (about 40us) and changing frequency of the radio between the event( 140us ), So in total is should be somewhere in the 3.1360msec + 8*40us + 8*140us ~ 4.60 milliseconds area., which is leagues away from the max event length of 400msec.

My suggestion would be to reduce the NRF_SDH_BLE_GAP_EVENT_LENGTH to 6, i.e. a connection event length of 7.5ms and see if that removes the delays. If not, is it possible to run the central and peripheral code on nRF52840 DKs? I could then reproduce the issue here and sniff the on-air packets with a Ellisys Bluetooth protocol analyzer to get the complete picture with no missing packets.

Best regards

Bjørn

RE: Reponse time of WRITE WITH RESPONSE command

Tielman — Wed, 14 Nov 2018 07:24:45 GMT

Hi Bjørn

1) 30ms connection interval - You are correct, thanks for pointing this out. My peripherals were configured to use 50ms, but I found the the BLE_CONN_INT_MAX_MS on the master side is set to 30 so the connections are created at 30ms and never updated. (I will get back to this)

2) Your assumption that the peripheral is responsible for the delay in response does not make sense since the peripheral cannot be aware of other peripherals connecting to the master and can therefore not "decide" to wait longer before sending a response when there are more peripherals connected to the master.

I reiterate the issue: The peripheral response DOES arrive in a timely manner when only one peripheral is connected. As soon as a second is connected, the time from transmitting the WR to receiving the first connections EVT increases, but also that there is a 30ms space between receiving the EVT from each of the other connections. In other words, when sending a WR to 8 different connections at the same time, the first event arrives roughly 300ms later, but all events from the other connections take an extra 30ms to arrive making the total time to wait for receiving the events from all 8 connections to 8x30ms=240ms.

Since you pointed out the my connection interval is 30ms (which matches the connection event length seen as 30ms), I changed my connection interval to 50ms and then to 100ms just to see that the connection event length also increases to 50ms and then 100ms. Now it takes even longer to receive all the events.

The softdevice appears to use the connection interval for the connection event length.

Regards

Tielman

RE: Reponse time of WRITE WITH RESPONSE command

bjorn-spockeli — Tue, 13 Nov 2018 14:15:39 GMT

Hi Tielman,

I apologize for the late reply, the SoftDevice needed some time to get back to me.

Ok, so from connection request in the trace we see that the connection interval is set to 30msec, the transmit window is set to 6.5msec and slave latency to 0. The Connection Request is followed up by a MTU Request, where the central asks for it to be set to 247 and the Peripheral replies with 23, i.e. the MTU size for the connection is set to 23.

The first thing they suspected was that the messages were fragmented, e.g. the data sent with the write response did not fit within the MTU of 23. However, this appears to not be the case for the packets after packet no 1301. As far as I can see the value is a 16 byte number, i.e. 19byte in total with the ATT header, which is well within the 23 byte MTU.

Taking a closer look at the trace you attached and the packets sent after packet 1301. From the event counter it seems that the peripheral is sending the Write Response 6-8 connection events after the Write request is sent from the central, but there are packets missing from the trace as the event counter states that there has been 6-8 connection events between the Write Request and Write Response, but in the trace we only see one or two Master-Slave message exchanges(i.e. one or two connection events). This is a known issue with the current version of the sniffer

So that the delay is at the Link Layer, i.e. the packet is not sent from the peripheral before 6-8 connection events have passed, and not between the reception of the Write Response and signaling of the BLE_GATTC_EVT_WRITE_RSP event in the S140 on the central side. Rather its the peripheral that takes its time to send the Write response.

I think an idea would be to look at the timing on the peripheral side, for instance that you toogle a GPIO when you get the BLE_GATTS_EVT_RW_AUTHORIZE_REQUEST on the peripheral and then toogle the same GPIO prior to calling sd_ble_gatts_rw_authorize_reply() which sends the Write Response? That way we could find out if the delay is added in the peripheral application or by the SD.

It would also be interesting to get a complete trace, so I was wondering if you have access to a commercial Bluetooth Protocol sniffer (e.g. Ellisys, Frontline or similar)?

BEst regards

Bjørn

RE: Reponse time of WRITE WITH RESPONSE command

Tielman — Fri, 09 Nov 2018 15:57:16 GMT

Hi Bjørn

I will wait for the softdevice team's response, thanks.

However, the 30ms mentioned above is not between events for 1 peripheral, it is between the consecutive connections. In other words,

Connection 1 receives EVT, 30ms later connection 2 receives EVT, 30ms later connection 3 receives EVT, etc...Also, the 30ms stays the same for 2 connections or for 8 so the 7.5ms x 4 does not seem relevant at all. Also, it does not appear that the connection event length is 7.5ms unless there is another delay that pushes event scheduling to 30ms instead of servicing each connection every 7.5ms.

From what I see each connection is serviced (Connection_count x 30ms) and not (Connection_count x 7.5ms)

But I will await the response from the team

Regards

Tielman

RE: Reponse time of WRITE WITH RESPONSE command

bjorn-spockeli — Fri, 09 Nov 2018 15:18:20 GMT

Hi Tielman,

I would expect the SoftDevice to issue the BLE_GATTC_EVT_WRITE_RSP to the application as soon as connection event has passed, i.e. after Connection Event Length ms after the connection event started and before the next connection interval. 200ms between the reception of the ATT Write Response and the SoftDevice issuing the BLE_GATTC_EVT_WRITE_RSP to the application does not sound reasonable, when you have a connection interval of 50ms. I have raised this question with the SoftDevice team and will come back to you with their comments.

As for the questions in your previous comment

1) Well, in the sniffer trace you are only monitoring the traffic between one peripheral and the central. Keep in mind that you also have 4 other peripherals connected at the same time, so if the Connection Event Length is 7.5ms then the Central will schedule 7.5ms for each connection. This means that the central will use 4*7.5ms = 30ms to talk to the 4 other peripherals before talking to the monitored peripheral again.

2) NRF_SDH_BLE_GAP_EVENT_LENGTH == Connection Event Length. It's the same thing.

Best regards

Bjørn

RE: Reponse time of WRITE WITH RESPONSE command

Tielman — Mon, 05 Nov 2018 12:19:05 GMT

Hi Bjørn

Attached is a sniffer trace between the master and one of the peripherals using Sniffer V2 and Wireshark.

From packet 0 to 400 is peripheral advertising
From packet 400 to 1050 (roughly) they are connected by not talking (only one connection)
From packet 1050 five more peripherals connected, but with peripheral 1 still not talking
From packet 1300 (roughly), the monitored peripheral and central communicate

devzone.nordicsemi.com/.../Pico-intervals-2.pcapng

From what I could see, the timing corresponded exactly to my previous findings, except that it looks like the peripheral sends the WRITE_RSP in the same connection interval as when it received the write command from the central and also appears tp respond with a notification in the same event (all 150us apart). So is the softdevice receiving the WRITE_RSP event and possibly also a notification from the peripheral in the same 1ms, but only giving that back to the application layer >200ms later?

Let me know your thoughts

Thanks

RE: Reponse time of WRITE WITH RESPONSE command

Tielman — Mon, 05 Nov 2018 09:58:23 GMT

Hi Bjørn

I have not changed those parameters in the config file, but here is what they are configured as:

#define NRF_SDH_BLE_GAP_EVENT_LENGTH 320

#define NRF_SDH_BLE_GATT_MAX_MTU_SIZE 247

#define NRF_SDH_BLE_GAP_DATA_LENGTH 27

My scan window is 50ms and interval is 100ms, though when I have 8 connections, scanning is stopped so this will only apply to the connection graphs above with less than 8 connections.

Thanks for the links. I see there is some optimization I need to do in terms of my connection intervals and scanning windows and intervals. Regardless, and to understand the timing better:

1) You gave an example Connection Event Length of 7.5ms

[quote userid="7571" url="~/f/nordic-q-a/40044/reponse-time-of-write-with-response-command/155793"]connection event length off 6*1.25ms = 7.5ms[/quote]

, however, from my graphs, it looks like my Tevent-Cx = 30ms since the events arrive 30ms apart? Why so slow and is there a way to reduce this?

2) I honestly don't understand what NRF_SDH_BLE_GAP_EVENT_LENGTH is for. Infocentre gives the following description

"The time set aside for this connection on every connection interval in 1.25 ms units"

Does this have a part to play in the connection event length?

I will try and get my packet sniffer working and let you know what I find.

RE: Reponse time of WRITE WITH RESPONSE command

bjorn-spockeli — Fri, 02 Nov 2018 15:13:09 GMT

Hi Tielman,

Are you able to capture a sniffer trace of the on-air packets using a BLE protocol sniffer e.g. Ellisys or using the nRF Sniffer v2 ? It would be very useful in debugging the issue.

Also have you altered the NRF_SDH_BLE_GAP_EVENT_LENGTH, NRF_SDH_BLE_GATT_MAX_MTU_SIZE or NRF_SDH_BLE_GAP_DATA_LENGTH values in sdk_config.h?

The S140 in a central role will schedule the connections as described in the Connection timing as a Central section in the S140 SoftDevice Specification. In addition to the scheduling the connection events, the central will also add the scanner event after all the connection events, see Figure 3. Scanner timing - one or more connections as a Central in the Primary channel scanner timing section.

In short, if all the write request are send at the same time then the last connection that is scheduled be sent after N*Connection Event Length, where N is the number of connections. So with 50ms and a connection event length off 6*1.25ms = 7.5ms, then the last Write request will be sent 8*7.5ms = >60 milliseconds after the first connection event. If you're scanning at the same time, then the scan window will have to be added to the N*Connection Event Length value.

Best regard

Bjørn

RE: Reponse time of WRITE WITH RESPONSE command

Tielman — Thu, 01 Nov 2018 10:12:36 GMT

To add more details to this...

I am using FreeRTOS to start with.

Then, I have done some more controlled test in the following way:

1) Connections: 1; Send 1 message and resend the message on every WRITE_RSP event

Result: WRITE_RSP event arrival times after sending message: 210ms, 60ms, 120ms, 150ms, 60ms ... (perfect 30ms windows)

(IMAGE: Looking at the bottom two rows where there are 'i' in: First row is calling sd_ble_gattc_write(); 2nd row is getting BLE_GATTC_EVT_WRITE_RSP)

2) Connections: 2; Send only 1 message on each connection at the same time, not to be repeated on EVT

Result: First connection's WR_RSP arrives 130ms after send, 2nd connection's WR_RSP arrives another 30ms later

3) Same test as in (2) but with 4 connections

Result: First connection's WR_RSP arrives 300ms later, followed by each of the other connections' WR_RSP an extra 30ms later

4) Same test as (2) & (3) but with 8 connections

Result: Interestingly the first WR_RSP arrives 250ms later followed by the WR_RSP for of the other connections 30ms appart (so a total of 450ms to receive all WR_RSP events)

5) Connections: 8; Send 1 message per connection at the same time and reply to each WR_RSP event with another message as it arrives

Result: First WR_RSP event arrives 250ms after sending WRITE with 30ms spacing between each connection's WR_RSP event, however, the next set of events arrive 250ms after the last WR_RSP event was received. This means that for the second set of sending of WRITE_WITH_RESPONSE commands sent, their corresponding event arrives 470ms later

6) Connections: 8; Send only 1 message on only one connection and reply to the WR_RSP event (So only 1 connection is talking)

Result: Almost xact same timing as per (5), however, only one connection is talking. Time to event from first write is 300ms. Time to every other event after a write is 470ms

So with 8 connections and only one device talking, it takes 470ms just to send one WRITE_WITH_REPONSE and get the corresponding WR_RSP event.

1) What is the softdevice design that explains this?

2) Does each connection receive a 30ms (or something) timeslot in which data is sent and received?

3) Is there a SDK setting or something that will allow me to change this to not have such a slow turnaround time for WRITE_WITH_RESPONSE issues when having multiple connections? We are planning on supporting 20 connections so this might be a painful experience if it cannot be changed.

Thanks