s140 pc-ble-driver 4.0.0 freezes when receiving too many data

RE: s140 pc-ble-driver 4.0.0 freezes when receiving too many data

jpo38 — Fri, 28 Feb 2020 08:21:06 GMT

Hello,

I have updates. We just noticed that having "nRF Connect - Bluetooth Low Energy" app opened (even if not connected to the dongle, just opened in Windows) makes everything work better:

- The connection is prefectly stable. I can receive notifications for hours, while it used to freeze after a few seconds
- The services dicovery is much faster

As soon as I close "nRF Connect - Bluetooth Low Energy", my application stops receiving notifications an I observe the freeze.

I bet this explains why the problem was hard to reproduce, and probably also why you could not reproduce because maybe you have nRF Connect often opened.

Apparently, "nRF Connect - Bluetooth Low Energy" does something in the background that makes the dongle and/or the soft device be more stable. What could it be?

See this ther post:

https://devzone.nordicsemi.com/f/nordic-q-a/58470/why-having-nrf-connect---bluetooth-low-energy-open-makes-my-connection-faster-and-more-stable

RE: s140 pc-ble-driver 4.0.0 freezes when receiving too many data

jpo38 — Fri, 11 Oct 2019 06:58:01 GMT

Hi,

Have you made any progress on this issue?

Kind regards,

Jean

RE: s140 pc-ble-driver 4.0.0 freezes when receiving too many data

jpo38 — Tue, 13 Aug 2019 07:01:52 GMT

Thanks for the update.

And what about this similar issue you mentioned: https://devzone.nordicsemi.com/f/nordic-q-a/42661/nrf-52840-hang-out-and-no-more-notification-connectivity-v3-s132-usb/192546#192546. Any progress on that? It looks similar.

The behaviour you observe (receiving a lot of 20bytes packets for a few seconds and later slowing down) is what I observe when things work right, but in almost all situations, it freezes during the first second. And yes, the fact that it slows down is due to parameters being updated.

I'm convinced that if you test this on a freshly installed Win10 you should be able to reproduce the issue, if not, then the hardware may be different (of configured/programmed differently).

Kind regards,

Jean

RE: s140 pc-ble-driver 4.0.0 freezes when receiving too many data

Jørgen Holmefjord — Mon, 12 Aug 2019 19:14:09 GMT

Hi,

Sorry for the very slow response to your request. Unfortunately, I do not have a freshly installed Windows PC to test with, but I will try that if I find one available soon. I did a test on a different computer, but neither on this one, I'm able to reproduce your issue. I had some of the tools installed from before, so the setup is not completely identical, but I did follow your exact steps to run the test.

I can confirm that I receive 20 bytes with nRF Connect and that I receive much more frequent notifications than one second. After a while, the notifications slows down, but I believe that this is caused by the connection parameters being updated.

Best regards,
Jørgen

RE: s140 pc-ble-driver 4.0.0 freezes when receiving too many data

jpo38 — Mon, 17 Jun 2019 14:45:52 GMT

Hello Jørgen,

As you cannot reproduce the problem and as I can on many different computers here, I was wondering if there could be a problem with our Windows setup (like using wrong drivers, or having the antivirus, firewall mess things up or whatever).

So I took a different computer, formatted the drive and did two tests:

- Installed Windows 7 (Enterprise with SP1 x64)
- Simply installed nrfconnectsetup260.exe (to get the good driver and have the dongle appear in Device Manager, it show up as "nRF Connect USB CDC ACM" driver info reports "1.0.1.0" "11/05/2018")
- Installed vc_redist.x64_2015.exe as nrf-ble-driver uses VS2015 runtimes
- Downloaded nrf-ble-driver-4.1.1-win_x86_64.zip from github
- Unzipped it, ran heart_rate_collector_sd_api_v3.exe COM3
- Pressed Enter, got ~15 notifications and nothing more: so I could reproduce the bug

Again, formatted the drive and:

- Installed Windows 10 (Enterprise version_1703)
- Did not even installed nrfconnectsetup260.exe (as the dongle appears directly in Device Manager)
- Installed vc_redist.x64_2015.exe as nrf-ble-driver uses VS2015 runtimes
- Downloaded nrf-ble-driver-4.1.1-win_x86_64.zip from github
- Unzipped it, ran heart_rate_collector_sd_api_v3.exe COM3
- Pressed Enter, got ~15 notifications and nothing more: so I could reproduce the bug

So there is definitely a problem with this driver and it does not come from my setup here.

- Either your machine has something installed/configured that makes it work better than mine
- Either I did not explain you correctly how to reproduce the issue

Could you please test from a freshly installed Windows (following the steps above)?
Also, can you confirm that when you start notifications with the device from nRF Connect, it sends you frame of data of 20bytes long. And can you also confirm that when you run heart_rate_collector_sd_api_v3.exe program you get lots of data during one second, not only one per second as the standard "Nordic_HRM" app behaves?

Kind regards,

Jean

RE: s140 pc-ble-driver 4.0.0 freezes when receiving too many data

jpo38 — Tue, 14 May 2019 14:20:12 GMT

Unfortnately, I'm not allowed to do that....

RE: s140 pc-ble-driver 4.0.0 freezes when receiving too many data

Jørgen Holmefjord — Tue, 14 May 2019 13:26:02 GMT

I'm see the same info, but with driver version 10.0.17134.1. Could you check if you are able to update the driver?

RE: s140 pc-ble-driver 4.0.0 freezes when receiving too many data

jpo38 — Tue, 14 May 2019 06:26:11 GMT

Its written 1.0.0 on my dongle, for the DK, it was installed by nRF Connect, it reports "Connectivity firmware version: ble-connectivity 0.1.0+Aug-14-2018-15-12-51. SoftDevice API version: 3. Baud rate: 1000000."

I'm having more difficulties to reproduce the problem those days. After a reboot, I see the freezing one or two times, then I can't reproduce them anymore. I'm wondering if Windows could give the process more priority at some point.

By the way, could this be a driver issue? When I plug the dongle, it appears in device manager as "nRF Connect USB CDC ACM", pilot used is usbser.sys (from system32/DRIVERS), Microsoft version 10.0.16299.334 (WinBuild.160101.0800). Are you using the same?

RE: s140 pc-ble-driver 4.0.0 freezes when receiving too many data

Jørgen Holmefjord — Mon, 13 May 2019 20:51:50 GMT

I had the example running over the weekend, it had still not failed this morning. Which revisions are the Dongle and the DK, are they both v1.x.x?

RE: s140 pc-ble-driver 4.0.0 freezes when receiving too many data

jpo38 — Fri, 10 May 2019 14:59:31 GMT

For me too sometimes it works for minutes with no crash, but sometime it constantly freezes. The dongle is a PCA10059 with SD APIv3, it was programmed by nRF connect itself first time I started it, the app is a PCA10040 with hex file uploaded to this post.

RE: s140 pc-ble-driver 4.0.0 freezes when receiving too many data

Jørgen Holmefjord — Fri, 10 May 2019 14:54:53 GMT

Ah, I did not catch that you downloaded the release from your previous post. I am not able to reproduce it with the release either, I have attached the output log: devzone.nordicsemi.com/.../pc_2D00_ble_2D00_driver_2D00_example.log. This has been running for ~40 minutes now.

What versions are the dongle and DK that you run the application on?

RE: s140 pc-ble-driver 4.0.0 freezes when receiving too many data

jpo38 — Fri, 10 May 2019 13:53:12 GMT

As I mentioned in my previous post, I can constantely reproduce this issue even without compiling/modifying any code at all. Just download nrf-ble-driver-4.1.1-win_x86_64.zip or nrf-ble-driver-4.1.0-win_x86_64.zip from https://github.com/NordicSemiconductor/pc-ble-driver/releases, unzip and run heart_rate_collector_sd_api_v3.exe with dongle port name as first parameter.

After it found the device, I hit "Enter", I start getting notifications, after a few seconds, it eventually stops receiving data (sometimes it works for good for minutes, but often it freezes within the 3 first seconds). Then when I press "Enter" again, it reports "Setting HRM CCCD" but not "Received write response.".

Please let me know what you experience with those executables.

Kind regards,

Jean

RE: s140 pc-ble-driver 4.0.0 freezes when receiving too many data

Jørgen Holmefjord — Fri, 10 May 2019 13:37:27 GMT

I tested with increased Sleep (up to 100), but I still cannot make the device freeze. Can you zip and upload the entire pc-ble-driver directory with compiled driver and application that I can test with?

RE: s140 pc-ble-driver 4.0.0 freezes when receiving too many data

jpo38 — Thu, 02 May 2019 14:03:24 GMT

Hello Jørgen,

I have a very fast desktop machine (Win10), I can constantly reproduce the issue here with heart_rate_collector_sd_api_v3. I don't modify any code as I downloaded pc-ble-driver 4.1.0, it has heart_rate_collector_sd_api_v3.exe compiled, I run it, it freezes after I receive less than 20 notifications. I could reproduce this again and again many times today.

I also have a laptop, slower (Win7), but not that slow (it's a Corei7). It's on this laptop that I had to add Sleep statement to make the issue occur more constantely. Else it's hard to get the freeze.

If the PC is too slow to process data (which is not the root cause of the issue I guess, as it constantly fails on my fastest computer), there should be no freezing anyway. With SD API v2 and a nRF51 dongle, the system does not freeze, but you "loose" some data (un to 40%): meaning that the driver is smart enough to ignore data when some internal buffer is full....but remaining responsive. With SD API v3 or v6, the driver freezes, this is not good.

I'm wondering if we could consider having any kind of local support. As this is only reprodcutible on our computers (but on ALL of them!). What about having someone from Nordic come and have a look, is that something possible? Or us coming to you with the computer, then we can do more testing.

Kind regards,

Jean Porcherot

RE: s140 pc-ble-driver 4.0.0 freezes when receiving too many data

jpo38 — Tue, 30 Apr 2019 12:23:17 GMT

Probably, please give it a try with a higher sleep, at least to see if you can reproduce the issue.

In this situation where the PC is not "fast enough" (or the device is actually too fast), I would expect some data to be lost (meaning that the driver, instead of freezing, should just not propagate the data to higher levels). This is how a nRF51 dongle with SD API v2 works, the PC may recieve data at a lower rate than 100Hz (some notifications are being "lost" at some point, probably because a buffer is full), however it never freezes. With nRF52 dongle with SD API v3, the fact that the buffer gets full is not handled correctly and the driver freezes for good. That's what we hope could be fixed.

RE: s140 pc-ble-driver 4.0.0 freezes when receiving too many data

Jørgen Holmefjord — Tue, 30 Apr 2019 12:15:19 GMT

I was missing the capital S in Sleep. However, I did not manage to reproduce this with 5 ms sleep. I can try to increase it further, but of course I would expect some buffers to be filling up if you are not able to handle the data fast enough. My guess would be that either the application or connectivity firmware asserts at some point when you are not able to process the data on your slow machine.

RE: s140 pc-ble-driver 4.0.0 freezes when receiving too many data

jpo38 — Mon, 29 Apr 2019 13:48:18 GMT

Hello Jørgen,

Here are answers to all your questions:

- To have "Sleep(5);" work you simply need to include "Windows.h". If you could not add the Sleep statement, then you could not test the change I recommanded on Apr-16 to reproduce the issue? Did you? Hopefuly with that you can reproduce it.

- I can't tell how many times log_handler is called per connection interval. I don't have this information, I just see when it's invoked. You'll see this in the console once you reproduced the problem as commeted above.

- Yes we did debug the app. No error detected here.

- Attached is a zip with the source of the app. It's based on your example, we only modified two lines (in components\ble\ble_services\ble_hrs\ble_hrs.c and examples\ble_peripheral\ble_app_hrs_freertos\main.c), flagged by "//MODIF SF" comment). Project to open is examples\ble_peripheral\ble_app_hrs_freertos\pca10040\s132\arm5_no_packs\ble_app_hrs_freertos_pca10040_s132.uvprojx.

Please let me know if combining this app with a Sleep(5) makes it possible for you to reproduce the issue.

Kind regards,

Jean Porcherot

devzone.nordicsemi.com/.../ble_5F00_app_5F00_hrs_5F00_freertos_5F00_SDK15_5F00_3.zip

RE: s140 pc-ble-driver 4.0.0 freezes when receiving too many data

Jørgen Holmefjord — Tue, 23 Apr 2019 17:19:59 GMT

I have still not been able to reproduce this. What did you include to use sleep? Are you sure that log_handler is only being called once every connection interval?

Have you tried debugging the application on the nRF to see if it reports any error codes, that could explain the behavior?

Can you provide the sample application in source code, rather than hex, for debugging?

RE: s140 pc-ble-driver 4.0.0 freezes when receiving too many data

jpo38 — Tue, 16 Apr 2019 07:16:45 GMT

Hello Jørgen,

Did you get a chance to reproduce the issue?

I tried to work on this issue today. It's true that it is hard to reproduce it using "nRF Connect". I took a new laptop (faster than mine) and I was unable to reproduce the issue.

However, with heart_rate_collector sample program, taken from https://github.com/NordicSemiconductor/pc-ble-driver/blob/master/examples/heart_rate_collector/main.c and unmodified:

- In Release, I could not reproduce the problem
- In Debug, I could not reproduce the problem

Then I commented sd_rpc_log_handler_severity_filter_set line (to have more logging, slowing down acquisition processus):

- In Release, I could not reproduce the problem
- In Debug, I could not reproduce the problem

Now I added "Sleep(5)" at the beginning of the log_handler function:

- In Release, I could reproduce the problem, after some seconds, I stop receiving data
- In Debug, I could reproduce the problem, after some seconds, I stop receiving data

Hopefully, you can reproduce that now. As we receive data every 10ms, it should be OK to sleep 5ms. And even, sleeping more than 10ms should work as well, this should not freeze the softdevice,it may loose some data being unable to handle them fast enough, but it should not freeze.

Please let me know if you can reproduce.

Kind regards,

Jean

RE: s140 pc-ble-driver 4.0.0 freezes when receiving too many data

jpo38 — Thu, 11 Apr 2019 14:30:59 GMT

Also, as the PC load seems to make the problem occur, could you try again with heart_rate_collector C program and just add a sleep in on_hvx function. If you make the sleep duration increase (5ms, 10ms, 20ms...) this will "simulate" a slow computer and hopefully this will make the dongle freeze occur on your system?

RE: s140 pc-ble-driver 4.0.0 freezes when receiving too many data

jpo38 — Tue, 09 Apr 2019 18:58:26 GMT

By the way, we speeded up the device to send data at 250Hz instead of 100Hz. Then the freezing occurs even more regularily. Specially when the PC is loaded (start new app, compile some code...).

By attaching a debugger, we see the device is never frozen or crashed. It's really the dongle that appears to freeze at some point.

RE: s140 pc-ble-driver 4.0.0 freezes when receiving too many data

jpo38 — Fri, 05 Apr 2019 14:57:35 GMT

Hello,

What I mean by 20bytes is that a regular HRM device sends a few (2 or 3 I don't recall) bytes per notification. With the .hex file I sent (attached to this case), it sends 20 bytes (for example 0x00 0x00 0x00 0x00 0x8C 0x56 0x00 0x20 0x7D 0xB4 0x02 0x00 0xC8 0x61 0x00 0x20 0xFB 0x6B 0x02 0x00) per notification. This is an additional change we did in the code (we send more data, 20 bytes, AND more often, at 100Hz).

I don't know what yo umean by "0 output". For me, when a connect such device in nRF connect. I enable notification, then I get a few notifications and after a few 1às seconds, or probably minutes, notifications stop and then I can't exchange with the device even if it is still reported as being connected by nRF connect.

Jean

RE: s140 pc-ble-driver 4.0.0 freezes when receiving too many data

Jørgen Holmefjord — Fri, 05 Apr 2019 14:28:46 GMT

I did not manage to reproduce it with sd_api_v3 either. I do get a lot of 0 output on the terminal (and 180 occasionally), is this the 20 bytes you are talking about? I will try to check with sniffer on Monday if the packets on-air are 100Hz and 20 bytes.

RE: s140 pc-ble-driver 4.0.0 freezes when receiving too many data

jpo38 — Thu, 04 Apr 2019 06:43:28 GMT

This is really surprising.

Are you receiving HRM at 100Hz with blocks of 20bytes?

Kind regards,

Jean

RE: s140 pc-ble-driver 4.0.0 freezes when receiving too many data

Jørgen Holmefjord — Wed, 03 Apr 2019 15:28:55 GMT

Unfortunately, I have not been able to reproduce this issue yet. I just ran your application together with sd_api_v6 for ~1 hour without any unresponsiveness. I will try again with sd_api_v3 now.