Connection failure when sending and receiving data simultaneously with SoftDevice 6.0 and SDK 15

RE: Connection failure when sending and receiving data simultaneously with SoftDevice 6.0 and SDK 15

JRRSoftware — Mon, 03 Sep 2018 18:51:23 GMT

I think the subject shoudn't be closed yet. I'm happy that Bruce found the solution for his problem but mine still isn't solved yet.
Did you checked the sample code I provided some time ago that replicates the issue?

If not, here is the link:
https://drive.google.com/file/d/1cXsduRdlnS_2xqVA3PXpB3KtY_KcU9nZ/view?usp=sharing

It clearly shows that assertion isn't thrown when it should.

Jack

RE: Connection failure when sending and receiving data simultaneously with SoftDevice 6.0 and SDK 15

Susheel Nuguru — Mon, 03 Sep 2018 16:24:21 GMT

Thanks for letting us know Bruce

RE: Connection failure when sending and receiving data simultaneously with SoftDevice 6.0 and SDK 15

owl-wy7n — Thu, 30 Aug 2018 23:49:02 GMT

Here is an update on our end.

Ends up we didn't need to reimplement our ISR. Our mobile app developer found a bug in his code that explained our remaining symptoms.

I guess it's OK for an ISR to run for 50 uS or so.

Thanks for the help!

RE: Connection failure when sending and receiving data simultaneously with SoftDevice 6.0 and SDK 15

owl-wy7n — Wed, 29 Aug 2018 00:46:31 GMT

Hello Aryan,

I've been away on a very welcome vacation. I'm now back to work on this issue.

Our ISR is running at priority 6.

This is lower than all the softdevice interrupts. But, by definition, it is higher priority than the ble task. It preempts the ble task, thus causing the same symptoms as described by JRRSoftware. In his case it was a FreeRTOS task that was preempting the ble task. In our case it is this ISR.

Our plan is to reimplement our ISR to do its time consuming work in a task that is lower priority than the ble task. All the ISR will do is wake the bottom-half ISR task. I predict that this will fix the problem. I'll let you know what I find.

Thanks!

Bruce

RE: Connection failure when sending and receiving data simultaneously with SoftDevice 6.0 and SDK 15

JRRSoftware — Mon, 20 Aug 2018 09:56:50 GMT

Do we have any update?

RE: Connection failure when sending and receiving data simultaneously with SoftDevice 6.0 and SDK 15

JRRSoftware — Mon, 13 Aug 2018 11:14:35 GMT

Yes, you can reproduce the issue using my test program. The IRQ priority for hardware timer is set to 6.
Try to run my test program without any modifications. All "definse's" are set to values that cause the issue on my dev boards. The settings are as follows:

#define ISR_WORKLOAD_MODE 2 // simulate workload on DUMMY side
#define ISR_IRQ_PRIORITY 6 //IRQ priority for harware timer used to trigger the workload ISR
#define ISR_WORKLOAD_EVERY 10 //trigger workload ISR every 10 ms
#define WORKLOAD_TIMER_LOOP_COUNT 10000 // workload is a loop counting to 10 000

Alternatively, you can keep pressing (very fast) first button (index 0) to simulate the workload on both sides (tester/dummy) and if you are lycky enough, you will reproduce the issue that way too.

ps: I'm aware that IRQa 0,1,4 can NOT be used. It was rather my hint for Bruce, and confirmation that they really cannot be used 🙂

Jack

RE: Connection failure when sending and receiving data simultaneously with SoftDevice 6.0 and SDK 15

Susheel Nuguru — Mon, 13 Aug 2018 10:24:41 GMT

Hi JRRSoftware,

good experiment, but you cannot use priorities number 0, 1 and 4 in your test as those will guarantee undefined behavior.

In any case, I think it is always wise to keep ISR as short as possible. Were you be able to reproduce the issue without using priorities 0,1 and 4?

RE: Connection failure when sending and receiving data simultaneously with SoftDevice 6.0 and SDK 15

Susheel Nuguru — Mon, 13 Aug 2018 10:16:19 GMT

Hello Bruce,

The softdevice will always run on highest priority for most time critical tasks and the application will never use this priority

The app can use next higher priority (priority 2 and 3) as you can see in the the pic below.

If your app ISR is using these higher priority rather than (priority 5, 6 and 7), then you are blocking non critical softdevice activities. This is legal as long as your connection is not depending on your app replying to the peer's query. And this can be only notified by softdevice to the app using the SWI2 interrupt using Priority 4. If your app is using priority 2 and 3, long enough to block this notification from softdevice, the peer can in some circumstances deem to understand that the connection is either lost or will intentionally terminate the connection as no response.

It is hard to define "too long" in this. I think that it depends on if there are any procedures that can happen requiring you app to respond within certain time. Encryption could be one example.

RE: Connection failure when sending and receiving data simultaneously with SoftDevice 6.0 and SDK 15

JRRSoftware — Sun, 12 Aug 2018 15:01:43 GMT

Dear Aryan and Bruce,

After many hours of investigations I think I managed to solve this problem.

I say “I think” because I need to carry on further tests to be 100% sure. But so far my program works.

The most important finding and observation is:

In some very specific circumstances, the sd_ble_gattc_write() function does NOT throw assertion when it should do. There are situations where assertion seems to be hidden or some how is not propagated properly, which leads to the “connection failure” described in this topic.

I’ve prepared a sample test program, where you can simulate different scenarios for the above.

It is actually the 3rd version and the link is available here:

https://drive.google.com/file/d/1cXsduRdlnS_2xqVA3PXpB3KtY_KcU9nZ/view?usp=sharing

I found several cases where the “connection issue” might popup:

Task priorities
Yes, Aryan was right in one of his previous replies. It is important to set task priorities right in the RTOS environment or use the “configUSE_TIME_SLICING” option.
High priority ISR
BLE stack seems to be very sensitive for a heavy workload inside an ISR function with a high priority. It is very easy to reproduce the problem with hardware timer of which the IRQ priority is set to 0 (highest) and the workload is “just” a loop counting to 400 only!
Like this:
for (int i=0;i<400;i++) {};
Low priority ISR
It is still possible to reproduce the problem with low priority ISR. In my case, the workload loop must count to 10000 (for (int i=0;i<10000;i++) {};)
and you probably need to wait a bit (while running my test program) until the problem occurs.
Volatile variables
Probably my most important finding that helped me to solve the issue.
I found an object (a buffer with control properties) that has been shared among 2 RTOS tasks and was NOT declared as “volatile”. I still do not fully understand why it has such a big impact to the BLE stack (and the sd_ble_gattc_write function) causing the “connection failure” to popup very occasionally and so hard to catch.

I think there are few questions that still need to be answered. For example, why assertion is not throwed by the sd_ble_gattc_write() when it should be?

Hopefully, my test program can help to answer them.

In the “main.c” file you will find the following defines that controls the program:

#define ISR_WORKLOAD_MODE

0 - no workload called from ISR (timer) function

1 - workload called on TESTER side

2 - workload called on DUMMY (other) side

3 - workload called on both sides

#define ISR_IRQ_PRIORITY

0-7 are the ISR (hardware timer) priorities. Use 0 for high and 7 for low priority. If set to 0, the WORKLOAD_TIMER_LOOP_COUNT can be set to 400, which is enough to reproduce the issue.

#define ISR_WORKLOAD_EVERY

This is time in ms that tells how often the ISR workload need to be run.

#define WORKLOAD_BUTTON_LOOP_COUNT

Workload “heaviness” when called from button event.

#define WORKLOAD_TIMER_LOOP_COUNT

Workload “heaviness” when called from hardware timer.

By the way, you can replicate the problem by pressing the first button fast enough (on the PCA 10040 dev boards).

As mentioned before, the test program is based on the “ble_app_att_mtu_throughput” sample code from the SDK.

RE: Connection failure when sending and receiving data simultaneously with SoftDevice 6.0 and SDK 15

owl-wy7n — Sat, 11 Aug 2018 00:47:43 GMT

Hello Aryan,

We are experiencing a very similar problem.

In our case our task priorities are correct (the softdevice task is the highest priority). Our failure is caused by an ISR that is starving the softdevice task. That ISR is consuming more than 50% of available CPU cycles.

We are in process of refactoring the design to correct this. In doing so, questions have come up that we hope you can answer.

I'll explain...

We have one prototype ISR implementation that consumes very close to 0% of available CPU cycles. That version seems to resolve the problem described in this forum. Alas, this version is not optimal for our application. We need to do more work in the ISR.

We have made another prototype ISR implementation that is better for our application. That version consumes about 5% of the CPU. Alas, it appears that the connection problem is back with this version.

So here are the questions...

Is the problem caused by the % of CPU stolen from the softdevice task? Or is it the duration? E.g. our ISR implementation that consumes %50 of the CPU only keeps the CPU for about 5 uS, but does so every 10 uS. The implementation that consumes %5 of the CPU keeps the CPU for about 50 uS and does so once every 1 mS or so.

My guess is that both are bad. We will refactor once again and put most of the 50 uS processing in a task that is lower priority than the softdevice task. My guess is this will fix things.

The question though, is how long is "too long" for a user ISR to preempt the softdevice task? It appears that 50 uS is too long.

Comments?

Thanks!

Bruce

RE: Connection failure when sending and receiving data simultaneously with SoftDevice 6.0 and SDK 15

Susheel Nuguru — Tue, 31 Jul 2018 10:01:02 GMT

Hi, Very good question, but is not very straight forward to answer

If you understand your system very well, then you should be able to tell the timing constraints of each action to be performed by your application. Very strict timing requirement on tasks gives them high priority compared to the others.

In your application where you seem to send the audio data, your application gives high importance on being able to transmit data (by notifications). So the task that sends hvx data could be high in priority. At the same time you should see that the high priority tasks suspends once in a while to give low priority tasks chance to run (else low priority task will never run in FreeRTOS). Safest way if you do not have clear understanding of inter dependency of tasks in your system is to enable time slicing and use same priority on tasks.

I also did not see it very efficient that the fact that you used timer to send a notification (and only one notification at a time) instead of TX_COMPLETE event (and sending as much as you can in that event). But maybe you did this just an example to demonstrate the problem and your real application does it differently.

RE: Connection failure when sending and receiving data simultaneously with SoftDevice 6.0 and SDK 15

JRRSoftware — Tue, 31 Jul 2018 09:37:40 GMT

Hi Aryan

I think we are very close to the final solution for this problem.
Your suggestions helped a lot and I managed to run the application whithout the issue described.

However, what is your suggestion about task priorities in this case?
It wasn't my intention to run all tasks at the same priority level.
I would rather assign different priority level for each task than use the "time slicing" option (configUSE_TIME_SLICING) on.

Do you think this approach is right?
Would it be a good practice, if SDH task has the highest priority in the entire application? (in the SDK code, the nrf_sdh_freertos_init() function sets the SDH task priority to 2 ...which is fairy low).

Or maybe you will advise the same priorities for all tasks and use "time slicing" option on?

RE: Connection failure when sending and receiving data simultaneously with SoftDevice 6.0 and SDK 15

Susheel Nuguru — Tue, 31 Jul 2018 07:31:48 GMT

setting the timeslicing to 1 has fixed the problem in the project you attached.

RE: Connection failure when sending and receiving data simultaneously with SoftDevice 6.0 and SDK 15

Susheel Nuguru — Tue, 31 Jul 2018 07:30:08 GMT

Hi JRRSoftware,

I was able to run some tests and i found it very clear that your timer deamon task at priority (2) was starving your dummy task and softdevice task at the same priority.

#define configTIMER_TASK_PRIORITY ( 2 )

Remember that in FreeRTOS configuring the kernel is very important to suit your needs. Since you have many "runnable" state tasks at the same time with same priority, FreeRTOS scheduler will always choose one task to run and starve the rest as long as the first task suspends itself. The reason is that you have set the timeslicing of equal priority tasks to 0. Your configuration for this is as below

#define configUSE_TIME_SLICING 0

Quoting the text from FreeRTOS documentation

configUSE_TIME_SLICING

By default (if configUSE_TIME_SLICING is not defined, or if configUSE_TIME_SLICING is defined as 1) FreeRTOS uses prioritised preemptive scheduling with time slicing. That means the RTOS scheduler will always run the highest priority task that is in the Ready state, and will switch between tasks of equal priority on every RTOS tick interrupt. If configUSE_TIME_SLICING is set to 0 then the RTOS scheduler will still run the highest priority task that is in the Ready state, but will not switch between tasks of equal priority just because a tick interrupt has occurred.

So if you set the timeslicing to 1 and leave the preemption to 1, then you should not see this problem.

configUSE_PREEMPTION	1
configUSE_TIME_SLICING	1

I guess some timing has changed with softdevice in few microseconds with relation to the notification for us to be able to trigger this corner case. Never the less, please choose your task priorities very wisely, they are very crucial part of your application design.

RE: Connection failure when sending and receiving data simultaneously with SoftDevice 6.0 and SDK 15

JRRSoftware — Tue, 31 Jul 2018 07:25:36 GMT

Hi Aryan,

Thanks for your reply.

Answering your questions in short:

In my code, as well as in the provided example, changing tasks priorities does not make any difference. As far as I understand how RTOS works, in my configuration the task priorities are used only to queue tasks in the right order. Please note that the “configUSE_TIME_SLICING” is set to 0.
The problem popped out once we upgraded SoftDevice and SDK to the latest versions
I’m not sure if the interrupt model for SDH can be mixed with RTOS tasks. I’m worried about unexpected behaviour.

I’m looking forward to your further investigations.

RE: Connection failure when sending and receiving data simultaneously with SoftDevice 6.0 and SDK 15

Susheel Nuguru — Mon, 30 Jul 2018 20:04:46 GMT

Hi,

Thanks Hung for trying to look into this case.

Sorry JRRSoftware for the delays. I have read all your investigation and thank you for doing a very detail analysis.

I can see that you narrowed down the problem to the configuration of NRF_SDH_DISPATCH_MODEL.

Hung has given a very good answer on why are getting NORESOURCE error. I want to get few answers from my investigation tomorrow

The design of your task priorities compared with softdevice task.
What has changed between the software versions for this behavior to pop out. If everything else remained same, then this error should have appeared in old SD and SDK version, i need to understand exactly what changed.
I will look into your example tomorrow and see if the interrupt model for SDH suits your needs or not.

We really appreciate your patience , you definitely are debugging into right direction and gave me some good start.

RE: Connection failure when sending and receiving data simultaneously with SoftDevice 6.0 and SDK 15

JRRSoftware — Mon, 23 Jul 2018 06:34:57 GMT

Thanks. I look forward for further updates.

RE: Connection failure when sending and receiving data simultaneously with SoftDevice 6.0 and SDK 15

Hung Bui — Fri, 20 Jul 2018 14:51:50 GMT

Hi,

Sorry for late update, due to the summer holiday, most of our expert in FreeRTOS are on vacation so the progress of the case is quite slow.

I captured a sniffer trace and seems that when the issue happens the Master can't ACK the write command from the Slave and keep NACKing. This results in the NORESOURCE issue. I assume in your code you sent (the last one) you send data both way ? It's not very clear on the trace dues to the sniffer issue.

This then comes to what you mentioned, how the application pulls events out from the softdevice. If the events are not pulling out from the softdevice fast enough, the buffer will be full and it will start NACKing incoming packet. But using NRF_SDH_DISPATCH_MODEL_INTERRUPT may not be the best when using with FreeRTOS as far as I understand.

We need to find out why adding a heavy task, at the same time having heavy BLE traffic causing the application fail to pull the events. We will have more resource to look into the case next week. We will update you what we find.

RE: Connection failure when sending and receiving data simultaneously with SoftDevice 6.0 and SDK 15

JRRSoftware — Thu, 19 Jul 2018 07:54:55 GMT

Hi, Do we have any progress?

RE: Connection failure when sending and receiving data simultaneously with SoftDevice 6.0 and SDK 15

JRRSoftware — Mon, 16 Jul 2018 16:41:29 GMT

Hi,

I did further investigation and the problem is somehow related to RTOS. It occurs only when NRF_SDH_DISPATCH_MODEL is set as NRF_SDH_DISPATCH_MODEL_POLLING.

Moreover, once SD_EVT_IRQHandler() is triggered and softdevice_task() is handled immediatelly - there is NO such issue.
But if other task is doing something and softdevice_task() cannot be handled immediatelly, then sometimes (but not always) the above isse is happening.

I'm not sure if I can use NRF_SDH_DISPATCH_MODEL_INTERRUPT together with RTOS. For now it seems to be the only solution. What is your advice?

RE: Connection failure when sending and receiving data simultaneously with SoftDevice 6.0 and SDK 15

Hung Bui — Mon, 16 Jul 2018 16:20:58 GMT

Hi,

I just tried it here and can reproduce the issue. We will start the investigation and get back to you when we have an update.

RE: Connection failure when sending and receiving data simultaneously with SoftDevice 6.0 and SDK 15

JRRSoftware — Fri, 13 Jul 2018 12:44:22 GMT

Hi again,

Answering your questions:

Unfortunately, your suggestion regarding the line 239 in the port_cmsis_systick.c does not make any difference
There are 2 functions that may throw the NRF_ERROR_RESOURCES error in this case:
sd_ble_gatts_hvx and sd_ble_gattc_write. However, as mentioned in my previous post, bi-directional data transfer is not necessary to reproduce the problem. It is enough to send data one way only. And this is what I do in the second version of my test case (link below).
When the issue occur on the device (doesn’t matter if on Master or Slave) it starts to be “blind and deaf” for BT events, data and notifications. However, other functions seems to work normally (there is no assertion, no hard fault, no stack overflow etc). You can reproduce this using my test case and watch log lines still printed from both sides.
Also, there are no high priority tasks in the provided examples. Furthermore, changing priorities for the dummy task (simulation workload) and SDH tasks does not make any difference either.

Here is the simplified code example (JRR_DeadlockTest_v2) to reproduce the problem:
drive.google.com/.../view

You can easily compare this code with original ble_app_att_mtu_throughput ad see there are only few changes. I use 2 dev boards (PCA 10040) to reproduce the problem. Usually, it takes few seconds from the data transfer start but sometimes need to run the test again.

The simulated workload is a task with a loop counting to 16000 every 5ms.

Any other suggestions?

RE: Connection failure when sending and receiving data simultaneously with SoftDevice 6.0 and SDK 15

Hung Bui — Fri, 13 Jul 2018 08:49:36 GMT

Thanks for providing us the example code. We unfortunately under reduced staff, so we couldn't try to test this right away.

But could you try at line 239 in SDKv15\external\freertos\portable\CMSIS\nrf52\port_cmsis_systick.c to change #if 0 to #if 1 ? This is a bug fix added to reduce power consumption but may have a side effect.

You mentioned NRF_ERROR_RESOURCES show on both side, what would throw that error on the central side ? Could you findout which function throwing that error ?

You mentioned the central doesn't throw disconnected event even though you turn off the client. It sound like the central actually got an assertion/hardfault. Have you check if it still functional normally ? like the main loop still executed?

Also, please make sure that you are not doing something in APP_HIGH priority for too long, and too often, doing that you may block the softdevice from throwing events, including DISCONNECTED event.

RE: Connection failure when sending and receiving data simultaneously with SoftDevice 6.0 and SDK 15

JRRSoftware — Thu, 12 Jul 2018 23:33:48 GMT

Hi,

I’ve managed to reproduce the problem using modified example code from the SDK.

The source code is available here:

https://drive.google.com/file/d/11sXPxutC-bAeOTJMpi_ByRIw5-pwclDq/view?usp=sharing

I’ve modified the ble_app_att_mtu_throughput example to use FreeRTOS.

A dummy RTOS task emulates a workload. In this case it is a loop counting to 100000 every 10 ms (roughly). It is more likely to trigger the issue if there is a serie of “short” workloads. If there are “heavy” tasks, but not triggered so often, the issue appears less likely.

To replicate the problem,

Put the JRR_DeadlockTest folder next to the ble_app_att_mtu_throughput (inside SDK_15\examples\ble_central_and_peripheral\experimental).
Follow the steps described in the instruction for the ble_app_att_mtu_throughput example.

Additional log lines shows how many times sending data ended successfully or threw an NRF_ERROR_RESOURCES error. After few seconds you should see that no data has been sent successfully (all ended throwing an error). A screenshot below shows this situation:

Red arrow shows the point where the BT connection failed. Each time it happens in slightly different point. Sometimes sooner sometimes later. No assertion and no disconnection event will be triggered when this situation happen (on TESTER device).

The key facts to reproduce the problem are:

Use FreeRTOS,
Emulate workload in a task that is triggered fairly often,
Send data using timer (or a task). Do not send data as a response for the BLE_GATTS_EVT_HVN_TX_COMPLETE event
Try to “flood” the BT characteristic by sending too much data.
Sending data bi-directionally is not necessary (as I thougt originally). It is possible to reproduce the problem by sending data only one way.

I hope it helps. Do you have any suggestions so far?

RE: Connection failure when sending and receiving data simultaneously with SoftDevice 6.0 and SDK 15

Hung Bui — Thu, 12 Jul 2018 07:41:27 GMT

Hi JRR,

There could be some bug fixes to avoid deadlock actually causing more deadlock. We will need to reproduce the issue here.

Could you provide a simplified version can on nRF52DK, sending dummy data that can reproduce the issue so we can test here ?

How it usually take for you to see the issue ? if testing in less noisy environment would it make a significant difference ?