s140 pc-ble-driver 4.0.0 freezes when receiving too many data

[jpo38]

I have a nRF52840-PCA10059 dongle.

I have a PCA10040 dev boeard on which I programmed the ble_app_hrs_freertos_pca10040_s132 example after I modified HEART_RATE_MEAS_INTERVAL and BATTERY_LEVEL_MEAS_INTERVAL variables (lines 100 and 95 of main.c) with value 10 (10ms for 100Hz).

If I connect this fast "Nordic_HRM" using nRF Connect (it programmed my dongle with API v3 SoftDevice, I can connect and get notifications from the device.

If I connect this fast "Nordic_HRM" using heart_rate_collector_v3 or heart_rate_collector_v6 example programs (compiled from https://github.com/NordicSemiconductor/pc-ble-driver/tree/v4.0.0), then I only get a few notifications and then the app freezes.

When it's frozen, using a debugger I can see that UartBoost::readHandler is not called anymore. If I press Enter (supposed to toggle notifications OFF), I see written

"Setting HRM CCCD"

I see:

UartBoost::readHandler receives: c0 10 0 0 f0 c0 c0 d6 6e 0 bc 1 9c 11 0 0 0 61 3e c0
SerializationTransport::readHandler receives: 1 9c 11 0 0 0

But "Received write response." never shows up. Looks like the soft device freezed and does not send my anything.

When using heart_rate_collector_v2 with a old nRF51 dongle, I observe no freezing.

When using heart_rate_collector_v3, I kept the SoftDevice programmed by "nRF Connect". Also tried with connectivity_4.0.0_usb_with_s132_3.1.0.hex, same behaviour observed.

When using heart_rate_collector_v6, I used connectivity_4.0.0_usb_with_s140_6.1.0.hex.

"Nordic HRM" unmodified device (low sampling rate) does not freeze.

Any ides what could cause this? What does nRF Connect make differently than heart_rate_collector sample?

[Jørgen Holmefjord]

Hi,

Could you check which connection interval nRF Connect and the HR collector example is connected to the peripheral with? If you decrease the sample interval without lowering the connection interval, you might fill up the notification buffers, resulting in the device getting an error code and crashing. Can you also post the log output from the HR collector example that is output to the terminal when you run it?

Best regards,
Jørgen

[Jørgen Holmefjord]

Can you provide the modified HRS example, or did you only change the two parameters? If you could also provide a sniffer trace between the two connected devices with both pc-ble-driver and nRF Connect, that would also be helpful.

[jpo38]

Project was compiled from SDK14.2. I attach you the hex file. It's programmed on a PCA10040 board.

I tested this again with "NRF Connect", after a fresh reboot, I constantly see the notifications stop and then device not repsonding within less than 5 minutes after I started Heart rate notifications.

Note that I just realized that we did not only change the sampling rate, we also changed the amount of data being sent: it's always 20 bytes, when regular HRM sends 2 or 3 bytes only. Sorry that I forgot to mention that.

We just tested with SDK 15.3.0 with a PCA10056 board. Just changed the sample rate, did not change the amount of data being sent. The connection was more stable. But after ~30mn, same problem: stopped receiving notifications and could not do anything with the device (no reply to read/write characterisctis/notifications requests)... 

6332.nrf52832_xxaa.hex

[jpo38]

Hello,

Could you finally reproduce the issue ? 

Kind regards,

Jean 

[Jørgen Holmefjord]

Unfortunately, I have not been able to reproduce this issue yet. I just ran your application together with sd_api_v6 for ~1 hour without any unresponsiveness. I will try again with sd_api_v3 now.

[jpo38]

This is really surprising.

Are you receiving HRM at 100Hz with blocks of 20bytes?

Kind regards,

Jean

[Jørgen Holmefjord]

I did not manage to reproduce it with sd_api_v3 either. I do get a lot of 0 output on the terminal (and 180 occasionally), is this the 20 bytes you are talking about? I will try to check with sniffer on Monday if the packets on-air are 100Hz and 20 bytes.

[jpo38]

Hello,

What I mean by 20bytes is that a regular HRM device sends a few (2 or 3 I don't recall) bytes per notification. With the .hex file I sent (attached to this case), it sends 20 bytes (for example 0x00 0x00 0x00 0x00 0x8C 0x56 0x00 0x20 0x7D 0xB4 0x02 0x00 0xC8 0x61 0x00 0x20 0xFB 0x6B 0x02 0x00) per notification. This is an additional change we did in the code (we send more data, 20 bytes, AND more often, at 100Hz).

I don't know what yo umean by "0 output". For me, when a connect such device in nRF connect. I enable notification, then I get a few notifications and after a few 1às seconds, or probably minutes, notifications stop and then I can't exchange with the device even if it is still reported as being connected by nRF connect.

Jean

[jpo38]

By the way, we speeded up the device to send data at 250Hz instead of 100Hz. Then the freezing occurs even more regularily. Specially when the PC is loaded (start new app, compile some code...).

By attaching a debugger, we see the device is never frozen or crashed. It's really the dongle that appears to freeze at some point.

[jpo38]

Also, as the PC load seems to make the problem occur, could you try again with heart_rate_collector C program and just add a sleep in on_hvx function. If you make the sleep duration increase (5ms, 10ms, 20ms...) this will "simulate" a slow computer and hopefully this will make the dongle freeze occur on your system?

[jpo38]

Hello Jørgen,

Did you get a chance to reproduce the issue?

I tried to work on this issue today. It's true that it is hard to reproduce it using "nRF Connect". I took a new laptop (faster than mine) and I was unable to reproduce the issue.

However, with heart_rate_collector sample program, taken from https://github.com/NordicSemiconductor/pc-ble-driver/blob/master/examples/heart_rate_collector/main.c and unmodified:

- In Release, I could not reproduce the problem
- In Debug, I could not reproduce the problem

Then I commented sd_rpc_log_handler_severity_filter_set line (to have more logging, slowing down acquisition processus):

- In Release, I could not reproduce the problem
- In Debug, I could not reproduce the problem

Now I added "Sleep(5)" at the beginning of the log_handler function:

- In Release, I could reproduce the problem, after some seconds, I stop receiving data
- In Debug, I could reproduce the problem, after some seconds, I stop receiving data

Hopefully, you can reproduce that now. As we receive data every 10ms, it should be OK to sleep 5ms. And even, sleeping more than 10ms should work as well, this should not freeze the softdevice,it may loose some data being unable to handle them fast enough, but it should not freeze.

Please let me know if you can reproduce.

Kind regards,

Jean

[Jørgen Holmefjord]

I have still not been able to reproduce this. What did you include to use sleep? Are you sure that log_handler is only being called once every connection interval?

Have you tried debugging the application on the nRF to see if it reports any error codes, that could explain the behavior?

Can you provide the sample application in source code, rather than hex, for debugging?