Dropped packets with bt_nus_send

RE: Dropped packets with bt_nus_send

hmolesworth — Tue, 13 Jun 2023 15:32:24 GMT

Agreed; though in our case the number of disconnects exactly tracked the number of lost packets. The implication is that a packet could be queued while a disconnect was in progress and subsequently discarded perhaps when the connection was re-established. I forget now quite where I found that the stack knowingly discards queued data on a disconnect/connect, but there was something in the documentation.

RE: Dropped packets with bt_nus_send

Susheel Nuguru — Tue, 13 Jun 2023 04:47:11 GMT

Yes, you are right, it can end in an endless loop. I will edit the code snippet to exit after 10 attempts.

RE: Dropped packets with bt_nus_send

Susheel Nuguru — Tue, 13 Jun 2023 04:44:34 GMT

How would be correlate mBleConnectionCounter to the number of lost packets? Maybe we can use this counter to tell the application of any possible data loss. Also, We cannot assume that every disconnect in such a scenario would definitely translate to data loss.

RE: Dropped packets with bt_nus_send

hmolesworth — Mon, 12 Jun 2023 16:03:11 GMT

Perhaps the user clockis could track the disconnect/connects; we found this helpful as it exactly corresponded to the number of missing packets. Something like this though this example is of course for the SDK not Zephyr

static void ble_evt_handler(ble_evt_t const * p_ble_evt, void * p_context)
{
    switch (p_ble_evt->header.evt_id)
    {
        case BLE_GAP_EVT_CONNECTED:
            ...
            // Indicate number of connections for checking against missing packets
            mBleConnectionCounter++;

RE: Dropped packets with bt_nus_send

clockis — Mon, 12 Jun 2023 08:11:53 GMT

Yes, it was indeed the handling of the Message Queue, thank you very much for your support and feedback!

RE: Dropped packets with bt_nus_send

Susheel Nuguru — Mon, 12 Jun 2023 08:02:00 GMT

I think handling your message queue incorrectly was the issue here based on the code snippet you provided. I am glad that it is solved now. Please mark my reply as verified so that others from this forum who have similar issue can be benefited..

RE: Dropped packets with bt_nus_send

Susheel Nuguru — Mon, 12 Jun 2023 08:00:08 GMT

Hugh, yes, that is one possible scenario where the uart data can be lost when you have a disconnect/connect sequence happening at the end of BLE range, but I did not think that this was the case as the description made me think that the data loss happens even without disconnect. Not sure how to handle such scenario when the softdevice just discards the packets. Seems like there is no error handling we can do here.

RE: Dropped packets with bt_nus_send

hmolesworth — Fri, 09 Jun 2023 16:56:19 GMT

I didn't read this whole thread but something which might assist your basic lost packet issue: When a BLE disconnect/reconnect event occurs the data successfully sent to the Softdevice by the uart service handler but not yet physically transmitted in full via BLE using the uart code is simply discarded by the Nordic Softdevice without notification to the uart code. In our case this produces a single lost packet (we transmit in packets) for each disconnect followed by a connect, This happens more often near the edge of the BLE range.

The only workaround we know is to query Central for any missing packets (or Central simply informs Peripheral of any missing packets using an embedded packet sequence number) and re-transmit those missing packets. This is a pain; if you find a simpler solution we would be happy to hear of it.

RE: Dropped packets with bt_nus_send

clockis — Fri, 09 Jun 2023 15:57:27 GMT

Actually, after a few modifications I have implemented this in my application and it seems to eliminate any further dropped packets.

Thnk you very much for the support if you have any other comments for improvements or any other thoughts you have on this it would be greatly appreciated, otherwise I will be closing this ticket on Monday that I will have finalized my implementation

RE: Dropped packets with bt_nus_send

clockis — Fri, 09 Jun 2023 14:41:25 GMT

Because the Zephyr documentation suggests this implementation and also I do not want to make the producer threads wait.
I will try this though.

Wouldnt this result in an endless loop if the device disconnects while the producer thread is stuck in this loop? Something like this would also cause the watchdog of the producer thread to trigger.

I just tried it and it makes the device unresponsive when it is on the edge of the Bluetooth range and it just freezes and stops transmitting, and after disconnecting the ble_write_thread and producer thread report 60 - 120 seconds of execution ( I am profiling the execution of the bt_nus_send and the producer thread )

RE: Dropped packets with bt_nus_send

Susheel Nuguru — Fri, 09 Jun 2023 14:11:33 GMT

purging a message queue might cause data loss in your case, when the message queue buffer is not empty. Why do you want to purge the message queue? I would suggest something like below

void ble_transmit_data( ble_data_t ble_data )
{
    uint8_t retry_count = 0;
    
	while ((k_msgq_put(&ble_msgq, &ble_data, K_NO_WAIT) != 0) && (retry_count++ < 10)) {
		/* Sleep for sometime. How long, is application specific */
		k_msleep(10);
	}
	
	if(retry_count == 10) { LOG_WRN("Failed to add data to message queue"); }
}

RE: Dropped packets with bt_nus_send

clockis — Fri, 09 Jun 2023 13:37:10 GMT

Hello Susheel,

Thank you very much for the thorough and immediate response.

Your previous message pointed me to look at the error handling when using k_msgq_put to queue my data to the Message Queue by adding the code below:

while (k_msgq_put(&ble_msgq, &ble_data, K_NO_WAIT) != 0) {
	/* message queue is full: purge old data & try again */
	k_msgq_purge(&ble_msgq);
}

Instead of just k_msgq_put on its own

k_msgq_put(&ble_msgq, &ble_data, K_NO_WAIT);

After changing this, no packets were dropped even when I was close to the edge of the BLE range and had poor signal strength (although about 4% of messages were still getting dropped but that seems like a separate issue).

The behavior was as your described and the BLE stack indeed cannot drop messages regardless of its connection settings (supervision timeout, ACL_BUF size, etc.). So it definitely was causing the issue I was seeing as messages were not being queued correctly in the message queue and did not arrive correctly in the ble_write_thread and thus were not fed to the bt_nus_send.

Sorry for spamming with a double post I did it just before trying the modification mentioned above so it was premature.

Now I am facing a different issue but since this might not be entirely relevant I will open a separate ticket for that.

Meanwhile, I am attaching some code snippets of the producer/consumer threads I have implemented and how they are used in case you can reproduce or spot any issues with my implementation.

Producer thread:

File: producer.c

void producer_thread (void)
{
    for (;;) {
    
        send_data();
        
        k_msleep(1000);
        
    }
}

uint8_t data[25][30]; //Data are already stored in a buffer

void send_data ( void )
{
    ble_data_t ble_data;
    
    for (i=0;i<25;i++)
    {
        //Load data to ble_data
        memcpy(ble_data.data , data[i], len[i]);
        ble_data.len = len[i];
        
        ble_transmit_data(ble_data);
    }
}

Consumer thread:

File: consumer.c

typedef struct ble_data_t {
    uint8_t data[30];
    uint16_t len;
} ble_data_t;

void ble_transmit_data( ble_data_t ble_data )
{
	while (k_msgq_put(&ble_msgq, &ble_data, K_NO_WAIT) != 0) {
		/* message queue is full: purge old data & try again */
		k_msgq_purge(&ble_msgq);
	}
}

void ble_write_thread (void) //The consumer thread
{
    ret_code_t err_code;
    
    for (;;) {
    
        /* Wait indefinitely for data to be sent over bluetooth */
		if (k_msgq_get( &ble_msgq, &buf , K_FOREVER))
		{
			LOG_ERR("Failed to get data from ble_msgq");
			continue;
		}
		
		err_code = bt_nus_send(NULL, buf.data, buf.len);
		if (err_code) {
			LOG_WRN("Failed to send data over BLE connection");
		}
    
    }
}

Thank you again for your thorough feedback it is very helpful!

Best regards,

Stavros

RE: Dropped packets with bt_nus_send

Susheel Nuguru — Fri, 09 Jun 2023 12:11:29 GMT

[quote user="clockis"]Q 1. Given this what is the best way for a producer thread to pass data to this consumer thread that runs bt_nus_send? Zephyr FIFO? Zephyr Message Queue? and how should they be written as code if you could provide a very simple generic sample code?[/quote]

I think before we even dive deeper into checking which data passing mechanism is best suited here, we first need to confirm that this is infact a producer/consumer problem that we are seeing. There is absolutely nothing wrong using the message queues, but it can matter how you are using it and how you are handling any errors you get while passing messages (due to queue being full or timedout).

Since you are doing this in the ble_write_thread and not in the bluetooth callbacks, I do not think this is caused by the possible blocking nature of bt_gatt_notify_cb.

Can you help me reproduce this or give me enough code snippets both in the producer and consumer so that I can attempt to make something similar to your use case?

What you are seeing could also happen due to application not handling errors with message queues or if there is any other race condition in the context where you get raw data and how you queue and send them in the notification.

[quote user="clockis"]Also the description above regarding the internal buffers being ovewritten was explained to me by a Zephyr community member on the Zephyr discord in detail, so I am just quoting their words and their explanation for it[/quote]

Could you please provide a link to this. I am surprised that I did not hear about this, but if you provide a link to this discussion and if it is still relevant to the SDK version you are using, then I will also keep focusing on the possible bug in the BLE stack handling. For now, my focus is mostly on application.

RE: Dropped packets with bt_nus_send

clockis — Fri, 09 Jun 2023 09:46:19 GMT

Hello Susheel,

Please excuse me for double posting but if it is possible for you I would like an answer to a more general question first before you respond to my previous comment because my issue is more general as well.

How should someone send data using bt_nus_send assuming having a producer thread that periodically generates data and I assume a consumer thread that waits for that data and calls the bt_nus_send (just like the ble_write_thread in the peripheral_uart sample)? I am assuming a consumer thread that waits on the data and feeds it to bt_nus_send is the most ideal case for an application with multiple threads that want to send data to Bluetooth (if not please suggest another way).

Q 1. Given this what is the best way for a producer thread to pass data to this consumer thread that runs bt_nus_send? Zephyr FIFO? Zephyr Message Queue? and how should they be written as code if you could provide a very simple generic sample code?

I am inquiring about this because I have tried using FIFOs & Message Queue and I still get dropped messages

Q 2. Also when the device reaches close to the edge of the range and transmission slows down and ultimately disconnects because it's out of range, when I get back close and connect it again, it still transmits at a very slow speed like it's still far away, what could be causing this?

Thank you very much for your patience and support and I look forward to hearing from you!

Best regards,

Stavros

RE: Dropped packets with bt_nus_send

clockis — Thu, 08 Jun 2023 11:41:23 GMT

Hello Susheel,

I have already performed the tests from the side of the application by checking the queued data sent to the bt_nus_send function and they are complete as expected and we are not receiving any errors from bt_nus_send or other messages.

Besides, this dropping of packets does not happen when the device is close to the mobile phone (BLE central using the nRF Connect app for Android as well as the nRD Connect app for Windows) it only happens when approaching the edge of the Bluetooth range.

When the device is close to the BLE central the transmission is complete, and successful, and all data arrive as expected, so to me, this shows that they're queued correctly as this mechanism is not related to the BLE range/signal strength, etc.

I am using a Zephyr MSGQ to wait for data in the ble_write_thread and this works perfectly. Queueing the data to the ble_write_thread with the MSGQ is irrelevant to the BLE stack. I don't see how this mechanism could be affected by the Bluetooth range or closeness of our device and the BLE central.

I will however try to perform the BLE sniffing and I will inform you asap but right now I am very much constrained in time and I am not sure when I will have the chance to do this.

[quote userid="6207" url="~/f/nordic-q-a/100276/dropped-packets-with-bt_nus_send/429920"]

clockis said:
But when it is on the edge of the BLE range and the connection is poor it keeps retrying to send the same message for a long time while we are still feeding/queueing new data to it(which it queues in its internal BLE stack buffers for transmission). Which at some point overwrites older data that were queued (in the internal BLE stack buffers) but not sent thus resulting in dropped packets (packets that were fed to bt_nus_send successfully but never sent to the BLE central because they got overwritten by newer data).

I do not think that this should be possible to happen that the BLE stack internal buffers are somehow overwritten. Atleast I have not heard of this happening.

[/quote]

Also the description above regarding the internal buffers being ovewritten was explained to me by a Zephyr community member on the Zephyr discord in detail, so I am just quoting their words and their explanation for it. Also I have seen it mentioned in other DevZone tickets:

bt_nus_send takes time to execute

Execution time of bt_nus_send

Thank you very much for your support! If you require any more info please let me know ( I will be doing the sniffing tests but if you could provide some feedback based on this it would be very helpful and greatly appreciated )

Best regards,

Stavros

RE: Dropped packets with bt_nus_send

Susheel Nuguru — Thu, 08 Jun 2023 08:59:42 GMT

Hi Stavros,

Karl requested me to take over. I am looking at the information that you gave as below

[quote user="clockis"]But when it is on the edge of the BLE range and the connection is poor it keeps retrying to send the same message for a long time while we are still feeding/queueing new data to it(which it queues in its internal BLE stack buffers for transmission). Which at some point overwrites older data that were queued (in the internal BLE stack buffers) but not sent thus resulting in dropped packets (packets that were fed to bt_nus_send successfully but never sent to the BLE central because they got overwritten by newer data).[/quote]

I do not think that this should be possible to happen that the BLE stack internal buffers are somehow overwritten. Atleast I have not heard of this happening.

I am assuming that you have double checked that your application is actually queuing full data using bt_nus_send (maybe print all data queued?) and you have confirmed here that you do not receive any error while sending notification. You also confirmed that the peer sees dropped packets. So the only thing that is remaining is to see the air traffic data. Can you please see it in the sniffer, that the notification data sent to the peer has missing data (dropped packets or overwritten data within the BLE stack)? If you queued data correctly and if you are not able to see few data it in the BLE sniffer, then something is happening within the BLE stack and we can focus our debugging direction into debugging the BLE stack issue.

But I am thinking this could be a producer and consumer problem in the application itself. Before we could dive deeper into two different directions, please provide the data sent and BLE sniffer so that we can narrow the problem to either application specific or the BLE stack specific.

RE: Dropped packets with bt_nus_send

Karl Ylvisaker — Wed, 07 Jun 2023 19:00:00 GMT

Hello Stavros,

Thank you for your patience with this.

[quote user="clockis"]Excuse me for the incomplete information.[/quote]

No need to apologize, I will just ask if I have any questions.

Thank you for the clarifications.

I will get back to you tomorrow with an update.

Best regards,
Karl

RE: Dropped packets with bt_nus_send

clockis — Wed, 07 Jun 2023 09:22:27 GMT

Hello Karl,

Kind reminder if you have any updates on this as this is a very important issue for our application.

Thank you very much

Best regards,

Stavros

RE: Dropped packets with bt_nus_send

clockis — Fri, 02 Jun 2023 15:33:45 GMT

Hello Karl,

If you require any additional information or clarifications or if you have any updates on this please let me know.

Also please note that I am using a ring buffer as I am using the Zephyr Message Queue(which is implemented as a ring buffer) to pass the generated data from the producer thread to the consumer(ble_write_thread).

Kind regards,

Stavros

RE: Dropped packets with bt_nus_send

clockis — Wed, 31 May 2023 13:20:48 GMT

Hello Karl,

Excuse me for the incomplete information.

Up until the change that requires the transmission of additional data we are transferring 341 bytes/second of data separated in separate 24 messages of various lengths that are queued to the ble_write_thread ( like the one used in peripheral_uart sample) that uses a Zephyr Message Queue to feed the data to the bt_nus_send.

As mentioned above the ble_write_thread just waits on the Zephyr Message Queue and feeds the incoming data to the bt_nus_send just like in the peripheral_uart sample.

With the new requirement we want to ideally send 21 x 341 = 7161 bytes/second and the hard requirement is to have NO dropped messages at all.

As I have understood after speaking with multiple Zephyr community members when called the bt_nus_send queues the data fed to it for transmission in the internal BLE stack's buffers and the BLE stack is responsible to try and send them until it succeeds or the supervision timeout expires.

But when it is on the edge of the BLE range and the connection is poor it keeps retrying to send the same message for a long time while we are still feeding/queueing new data to it(which it queues in its internal BLE stack buffers for transmission). Which at some point overwrites older data that were queued (in the internal BLE stack buffers) but not sent thus resulting in dropped packets (packets that were fed to bt_nus_send successfully but never sent to the BLE central because they got overwritten by newer data).

bt_nus_send never returns an error message during this behavior( so no error handling is possible ) it always returns successfully but is executing/blocking for tens of milliseconds when this dropping of the packets happens(I am observing the execution time of the bt_nus_send (in the ble_write_thread.

I have tried using the maximum number of BLE stack's buffers (CONFIG_BT_BUF_ACL_TX_COUNT=255) and the minimum supervision timeout (100 ms) so that the BLE stack can queue/buffer more data as well as disconnect when the transmission duration is too long but I still get some dropped packets when the device is in the edge of its range.

Best regards,

Stavros

RE: Dropped packets with bt_nus_send

Karl Ylvisaker — Wed, 31 May 2023 07:06:34 GMT

Hello Stavros,

The best approach to handle this will depend on your applications requirements and constraints, could you elaborate on the requirements to your data transfer?
How much data will you generate, and how quickly will you do so, that you will be sending to the phone?

In any case, the best way to approach this is to implement a local error handling after the call to queue the data for sending over the NUS service, so that you can handle it in the case that the data fails to queue due to the buffer being full.
To handle this you could for example implement a ring-buffer, which you feed into the NUS service.
Alternatively, you could handle the data that fails to queue in another way - for instance, if it is not important for your application that every packet of data makes it across the link you could discard this data just as well. I mention both these approaches since I do not know enough about your application to know which one fits your use-case.

Best regards,
Karl