s140 pc-ble-driver 4.0.0 freezes when receiving too many data

[jpo38]

I have a nRF52840-PCA10059 dongle.

I have a PCA10040 dev boeard on which I programmed the ble_app_hrs_freertos_pca10040_s132 example after I modified HEART_RATE_MEAS_INTERVAL and BATTERY_LEVEL_MEAS_INTERVAL variables (lines 100 and 95 of main.c) with value 10 (10ms for 100Hz).

If I connect this fast "Nordic_HRM" using nRF Connect (it programmed my dongle with API v3 SoftDevice, I can connect and get notifications from the device.

If I connect this fast "Nordic_HRM" using heart_rate_collector_v3 or heart_rate_collector_v6 example programs (compiled from https://github.com/NordicSemiconductor/pc-ble-driver/tree/v4.0.0), then I only get a few notifications and then the app freezes.

When it's frozen, using a debugger I can see that UartBoost::readHandler is not called anymore. If I press Enter (supposed to toggle notifications OFF), I see written

"Setting HRM CCCD"

I see:

UartBoost::readHandler receives: c0 10 0 0 f0 c0 c0 d6 6e 0 bc 1 9c 11 0 0 0 61 3e c0
SerializationTransport::readHandler receives: 1 9c 11 0 0 0

But "Received write response." never shows up. Looks like the soft device freezed and does not send my anything.

When using heart_rate_collector_v2 with a old nRF51 dongle, I observe no freezing.

When using heart_rate_collector_v3, I kept the SoftDevice programmed by "nRF Connect". Also tried with connectivity_4.0.0_usb_with_s132_3.1.0.hex, same behaviour observed.

When using heart_rate_collector_v6, I used connectivity_4.0.0_usb_with_s140_6.1.0.hex.

"Nordic HRM" unmodified device (low sampling rate) does not freeze.

Any ides what could cause this? What does nRF Connect make differently than heart_rate_collector sample?

[Jørgen Holmefjord]

Hi,

Could you check which connection interval nRF Connect and the HR collector example is connected to the peripheral with? If you decrease the sample interval without lowering the connection interval, you might fill up the notification buffers, resulting in the device getting an error code and crashing. Can you also post the log output from the HR collector example that is output to the terminal when you run it?

Best regards,
Jørgen

[Jørgen Holmefjord]

Can you provide the modified HRS example, or did you only change the two parameters? If you could also provide a sniffer trace between the two connected devices with both pc-ble-driver and nRF Connect, that would also be helpful.

[jpo38]

Also, as the PC load seems to make the problem occur, could you try again with heart_rate_collector C program and just add a sleep in on_hvx function. If you make the sleep duration increase (5ms, 10ms, 20ms...) this will "simulate" a slow computer and hopefully this will make the dongle freeze occur on your system?

[jpo38]

Hello Jørgen,

Did you get a chance to reproduce the issue?

I tried to work on this issue today. It's true that it is hard to reproduce it using "nRF Connect". I took a new laptop (faster than mine) and I was unable to reproduce the issue.

However, with heart_rate_collector sample program, taken from https://github.com/NordicSemiconductor/pc-ble-driver/blob/master/examples/heart_rate_collector/main.c and unmodified:

- In Release, I could not reproduce the problem
- In Debug, I could not reproduce the problem

Then I commented sd_rpc_log_handler_severity_filter_set line (to have more logging, slowing down acquisition processus):

- In Release, I could not reproduce the problem
- In Debug, I could not reproduce the problem

Now I added "Sleep(5)" at the beginning of the log_handler function:

- In Release, I could reproduce the problem, after some seconds, I stop receiving data
- In Debug, I could reproduce the problem, after some seconds, I stop receiving data

Hopefully, you can reproduce that now. As we receive data every 10ms, it should be OK to sleep 5ms. And even, sleeping more than 10ms should work as well, this should not freeze the softdevice,it may loose some data being unable to handle them fast enough, but it should not freeze.

Please let me know if you can reproduce.

Kind regards,

Jean

[Jørgen Holmefjord]

I have still not been able to reproduce this. What did you include to use sleep? Are you sure that log_handler is only being called once every connection interval?

Have you tried debugging the application on the nRF to see if it reports any error codes, that could explain the behavior?

Can you provide the sample application in source code, rather than hex, for debugging?

[jpo38]

Hello Jørgen,

Here are answers to all your questions:

- To have "Sleep(5);" work you simply need to include "Windows.h". If you could not add the Sleep statement, then you could not test the change I recommanded on Apr-16 to reproduce the issue? Did you? Hopefuly with that you can reproduce it.

- I can't tell how many times log_handler is called per connection interval. I don't have this information, I just see when it's invoked. You'll see this in the console once you reproduced the problem as commeted above.

- Yes we did debug the app. No error detected here.

- Attached is a zip with the source of the app. It's based on your example, we only modified two lines (in components\ble\ble_services\ble_hrs\ble_hrs.c and examples\ble_peripheral\ble_app_hrs_freertos\main.c), flagged by "//MODIF SF" comment). Project to open is examples\ble_peripheral\ble_app_hrs_freertos\pca10040\s132\arm5_no_packs\ble_app_hrs_freertos_pca10040_s132.uvprojx.

Please let me know if combining this app with a Sleep(5) makes it possible for you to reproduce the issue.

Kind regards,

Jean Porcherot 

ble_app_hrs_freertos_SDK15_3.zip

[Jørgen Holmefjord]

I was missing the capital S in Sleep. However, I did not manage to reproduce this with 5 ms sleep. I can try to increase it further, but of course I would expect some buffers to be filling up if you are not able to handle the data fast enough. My guess would be that either the application or connectivity firmware asserts at some point when you are not able to process the data on your slow machine.

[jpo38]

Probably, please give it a try with a higher sleep, at least to see if you can reproduce the issue.

In this situation where the PC is not "fast enough" (or the device is actually too fast), I would expect some data to be lost (meaning that the driver, instead of freezing, should just not propagate the data to higher levels). This is how a nRF51 dongle with SD API v2 works, the PC may recieve data at a lower rate than 100Hz (some notifications are being "lost" at some point, probably because a buffer is full), however it never freezes. With nRF52 dongle with SD API v3, the fact that the buffer gets full is not handled correctly and the driver freezes for good. That's what we hope could be fixed.

[jpo38]

Hello Jørgen,

I have a very fast desktop machine (Win10), I can constantly reproduce the issue here with heart_rate_collector_sd_api_v3. I don't modify any code as I downloaded pc-ble-driver 4.1.0, it has heart_rate_collector_sd_api_v3.exe compiled, I run it, it freezes after I receive less than 20 notifications. I could reproduce this again and again many times today.

I also have a laptop, slower (Win7), but not that slow (it's a Corei7). It's on this laptop that I had to add Sleep statement to make the issue occur more constantely. Else it's hard to get the freeze.

If the PC is too slow to process data (which is not the root cause of the issue I guess, as it constantly fails on my fastest computer), there should be no freezing anyway. With SD API v2 and a nRF51 dongle, the system does not freeze, but you "loose" some data (un to 40%): meaning that the driver is smart enough to ignore data when some internal buffer is full....but remaining responsive. With SD API v3 or v6, the driver freezes, this is not good.

I'm wondering if we could consider having any kind of local support. As this is only reprodcutible on our computers (but on ALL of them!). What about having someone from Nordic come and have a look, is that something possible? Or us coming to you with the computer, then we can do more testing.

Kind regards,

Jean Porcherot

[Jørgen Holmefjord]

I tested with increased Sleep (up to 100), but I still cannot make the device freeze. Can you zip and upload the entire pc-ble-driver directory with compiled driver and application that I can test with?

[jpo38]

As I mentioned in my previous post, I can constantely reproduce this issue even without compiling/modifying any code at all. Just download nrf-ble-driver-4.1.1-win_x86_64.zip or nrf-ble-driver-4.1.0-win_x86_64.zip from https://github.com/NordicSemiconductor/pc-ble-driver/releases, unzip and run heart_rate_collector_sd_api_v3.exe with dongle port name as first parameter.

After it found the device, I hit "Enter", I start getting notifications, after a few seconds, it eventually stops receiving data (sometimes it works for good for minutes, but often it freezes within the 3 first seconds). Then when I press "Enter" again, it reports "Setting HRM CCCD" but not "Received write response.".

Please let me know what you experience with those executables.

Kind regards,

Jean

[Jørgen Holmefjord]

Ah, I did not catch that you downloaded the release from your previous post. I am not able to reproduce it with the release either, I have attached the output log: pc-ble-driver-example.log. This has been running for ~40 minutes now.

What versions are the dongle and DK that you run the application on?