Synchronize timestamps between SoC and mobile app

[alejandrohc]

Background

I am attempting to build a system in which a Nordic NRF52840 SoC, is connected to a mobile app (iOS / Android) via BLE and both devices play back audio-visual feedback in a synchronized manner. Both devices are able to take care of their own playback so not much data needs to be exchanged while playing.

What I am having trouble with is to synchronize the SoC's timestamp with the app reliably, so that the start of the playback can happen at exactly the same time.

Setup

• Mobile device is master and SoC is the peripheral
• Mobile and SoC are connected and bonded
• Multiple SoCs could be connected to the mobile device and the system should still work reliably
• The connection interval is set to 30ms
  
• Packet transmission from the SoC to the master is done through Notifications
• Looking at the arrival timestamps on the master by using Android's Bluetooth HCI Snoop Log with Wireshark and logging the values on XCode for iOS

Goals

• 5ms synchronization accuracy
• Measured by looking at how the offset between the SoC's and master's timestamps changes across multiple synchronizations (* Figure 1)

Approach

• There is a radio notification interrupt set up on the SoC, that triggers 5500µs before connection events
• At that point, I store the current timestamp +5500µs (the timestamp at the time of the assumed packet transmission)
• As enqueueing BT packets on the software interrupt is a bad idea, the main loop calls sd_ble_gatts_hvx() on its next iteration to send the previously calculated timestamp
• Master keeps track of the SoC's timestamp individually by calculating an offset with its own timestamp (native monotonic clock)
• I send the timestamp from the SoC to the master multiple times, calculating the difference between their clocks' timestamp offsets to see if they are always the same, as they should be, since the clocks run independently in parallel

* Figure 1: Expected timestamp offset across synchronizations

Questions

1. Due to the nature of BLE, there seems to be no way to guarantee that the timestamp sync packet is sent out with the next connection event
   • It is assumed that there are enough resources to enqueue the packet, when there is a NRF_SUCCESS return code produced by sd_ble_gatts_hvx()
   • Q1: Is this assumption correct?
2. I've seen a random difference between the timestamp offsets for one device, which is sometimes within 20ms and sometimes seems to be multiples of the connection interval (up to 4 times 30ms)
   
   • Q2: What could be the cause of the smaller errors? I think that maybe the transmission window is the culprit here
     • Please note that the packages are being received with some milliseconds of difference between them, according to the data in the HCI Snoop Log (* Figure 2)
   • Q3: How could it be that some packages are being received a couple of connection events after I enqueued them if there seems to be enough space in the queue and I did not enqueue other packets?
3. As part of the tests, I've enqueued more than 6 packets multiple times and the way that the errors are reported on the SoC is not always consistent when using the same device as master i.e. sometimes I can only enqueue 4 or up to 7 packages with a result of NRF_SUCCESS
   • Q4: What's the reason for this behavior?
4. Since I only need to synchronize the timestamps once per connection, I could do this by placing the timestamp in the SoC's advertisement data, as mentioned in this other entry in the forum
   • Q5: Do you recommend this approach, is it faster or more reliable than the regular notifications or indication?
     
   • Q6: Are there other alternatives to tackle this?
5. On the same note as point no. 4: I've seen that iOS keeps this advertisement data cached and it can be rather unreliable for me to have the updated version on the master
   • On top of that, as I rely on the ANCS for notifying push notifications to the peripheral, iOS would automatically connect to the SoC, without me being able to grab the advertisement data beforehand
   • Q7: Would it be possible for the SoC to advertise the data even after it has established a bonded connection to the master?
6. When looking at the current approach:
   • Q8: Is this a reasonable approach? What kind of accuracy can be expected?
   • Q9: What about delays on phone side, is there anything in particular that I should take special care of?
   • Q10: Are there any further resources on the topic of timestamp synchronization that you would recommend?

* Figure 2: Timestamp difference between received notifications as logged on Android OS

[Einar Thorsrud]

Hi,

A1: Yes. If sd_ble_gatts_hvx() returns NRF_SUCCESS, the packet has been queued. It will return NRF_ERROR_RESOURCES if the queue is full.

A2: The notifications are transmitted int the order there were transmitted. But if you also send other BLE data, there may be some other packets that were queued before, and if so, these will be transmitted before. This is the most likely explanation for variation within a connection event.

A3: The most probable reason for seeing delays which are multiple of a connection interval is packet loss, which leads to retransmission the next connection event (and so on, until the packet is successfully transferred or there is a supervision timeout). It could also be that if you are transmitting a lot of data and/or the link does not support "enough" packets per connection event, the packet is delayed until the next event in this case as well.

A4: Is this with the exact same firmware on the nRF side? The number of packets you can queue depends on the SoftDevice configuration. If not, then perhaps there is a problem with your test, for instance, that packets are popped from the queue while you are filling it (in the middle of a connection event), and therefore you can fill more? But in another case, you are doing the test outside a connection event? This is just one possible explanation.

A5: I am not in a position to recommend one or the other. However, putting the timestamp in the advertisement packet also has the limitation that there is a time from when you read the timestamp to when you transmit the advertisement. A particularly important point here is that advertisement packets are transmitted with a pseudo-random 0-10 ms delay, which may not be acceptable.

A6: No good/obvious alternatives. Since you are communicating with a phone, you have to use standard BLE, so using, for instance, ESB (which has been recommended to some customers before) is not an option in this case.

A7: Yes. The nRF is capable of handling several roles concurrently, and advertising while maintaining a connection is no problem.

A8: Yes, using notifications in a connection seems sensible, and looks like the best approach in my mind. The accuracy depends on several factors. You will get the best accuracy if you don't transmit other BLE data and you syncronize the clocks regularly. Then you can filter out synchronization packets that have a jump in the time since they are probably bad (a result of packet loss). Inaccuracies due to other packets being queued before is a bit more difficult to predict, but perhaps you could make sure that you never queue other data until after having queued the synchronization notification.

A9: This one is difficult. It can vary from phone to phone and may change over time (with SW updates). So I cannot make any promises on the phone side. We have also seen earlier that some phones may change the order of the notifications. But this is unfortunately out of our control. 

A10: Not really. You may find Wireless timer synchronization among nRF5 devices interesting, as it describes the problems you are seeing. But the proposed solution cannot be used between an nRF and a phone since the phones do not support ESB.

[alejandrohc]

Hi Einar, thank you for your responses, they're very informative.

A1: Great to know I'm on the right track.

A3: I see, did not consider that the protocol is retrying the transmission of packages for me. 

Q11: Is there any other way of transmitting data without the protocol automatically retransmitting lost packages? This way I could send a series of packages, where the one with the lowest offset wins. Dropped and therefore outdated packages would then not further clutter the queue either (I am thinking of something like UDP vs. TCP in the web).

A4: Yes, it is the exact same firmware on the SoC. Q12: How can one pop packages from the queue? That would help me to cancel the message if it's too late already.

Q13: Do you have any news about the potential way of knowing which characteristic sent out a packet? With that information I could notify the mobile phone if the packet was sent out too late.

A5: Q14: Is that pseudo-random delay exclusive to the advertisement packets or are notifications also susceptible to that due to the transmission window?

A6, A7: Interesting.
A8: Will definitely try to not transmit packets from the SoC to the phone during the timestamp synchronisations. Q13: Should I also make sure no packets are transmitted from the phone to the SoC? That would be harder to do, particularly in iOS since I don't control its services ANCS and CTS for push notifications and time correspondingly.
A9: Some phones changing the order of the notifications sounds critical, will then not assume that the packets are always sent in a specific order. I think this is not an issue for Indication Q14: right?

A10: That's a very interesting post indeed, I've looked at it in detail. In an ideal world, I'd be able to develop a BLE dongle what connects over USB to the mobile device and also runs a Nordic Semi SoC with a custom radio notification protocol... but that's a dream for another time.

[Einar Thorsrud]

Hi,

A11: No. There is no standard way, so there is no way of doing this together with a phone.

A12: It is not possible to pop packets from the Tx queue on the nRF side. Once the packets are queued the will be sent at some point (unless there is a connection supervision timeout).

A13: No updates. It is still not supported by the SoftDevice.

A14: The pseudo-random delay is only used for advertisement packets, not for connection events.

A15(13): It depends on the link configuration (the packets per connection depends on several factors such as packet lengths, event duration, and buffer sizes). You can only control the nRF side, as the BLE stack on the phone is less configurable. So ideally "Yes", and you should remember that even if you test with one phone, a different phone can behave differently.

A16(14): I have not heard about it being an issue with indications, and it seems less likely since the ack has to come the whole way round before a new packet is transmitted. But in any case, this is just a strange behavior we have seen a few times within the phone, and not something that you can observe on the BLE link. It is difficult to predict what may happen on the API between the application and BLE stack on the phone side since that is totally out of our control.

[alejandrohc]

Hi Einar,

Q17:

• Do you think that dynamically reducing the connection interval for the timestamp sync process (to something close to 7ms) would help?
• I know that it is now also supported by iOS, but am afraid that it would cause more packets to get lost / not arrive during that smaller time frame.

Other than that, all of my questions have been addressed and I thank you for your helpful advice.

[Einar Thorsrud]

A17: Reducing the connection interval would reduce the delay introduced by packet loss, so it could help in that way. I do not see any other benefit. Regarding packet loss it depends. The center may have a harder time scheduling events the more frequent they are, so in that sense, the likelihood of packet loss may be higher. But I do not see any other problems, except for the obvious increase in average power consumption.