Product crash with nrf51822 peripheral s130 2.0.1 : how to get logs or more information ?

RE: Product crash with nrf51822 peripheral s130 2.0.1 : how to get logs or more information ?

Julien Cosmo — Wed, 26 Oct 2022 09:35:57 GMT

I used the HARDFAULT_HANDLER, and a new release is on its way.

I think we can clode this ticket.

Thanks again !

RE: Product crash with nrf51822 peripheral s130 2.0.1 : how to get logs or more information ?

Hung Bui — Mon, 24 Oct 2022 07:44:54 GMT

Hi Julien,
Yes you are correct by default hardfault handler will make it stay in an infinite loop. You can have a look at this guide.

You can either enable HARDFAULT_HANDLER module in sdk_config.h (you would need to include hardfault_implementation.c file)

Of implement your own HardFault_Handler to overwrite the weak definition of it in ses_startup_nrf52.s (if you are using SES)

RE: Product crash with nrf51822 peripheral s130 2.0.1 : how to get logs or more information ?

Julien Cosmo — Fri, 21 Oct 2022 15:00:20 GMT

Ok, thanks.

I reproduced a similar issue while in debug using a low stack size setting to see what's happens, and I can see I get an "hardfault"

From what I see, the hardfault produce an infinite loop (while others faults produce a reset by default).

I guess for future use, it'll be a good idea for me to change this behavior, and have a reset too in case of hardfault...

This infinite loop by default is for debug purpose, am I right ?

RE: Product crash with nrf51822 peripheral s130 2.0.1 : how to get logs or more information ?

Hung Bui — Fri, 21 Oct 2022 11:27:09 GMT

Hi Julien,
Unfortunately I don't know. But it's better not to go under what required by the spec.

I am not sure not using bond can reduce the peak number of byte the stack need. I suspect it doesn't.

RE: Product crash with nrf51822 peripheral s130 2.0.1 : how to get logs or more information ?

Julien Cosmo — Fri, 21 Oct 2022 09:52:39 GMT

Thanks again, we successfully reached 1944 bytes for the stack without major modifications, and we are still testing it (so far no crash).

About the stack size for the softdevice, so you know if the stack use is decreased (below 1536 bytes) if the bonding is not used ?

RE: Product crash with nrf51822 peripheral s130 2.0.1 : how to get logs or more information ?

Hung Bui — Fri, 21 Oct 2022 08:04:34 GMT

Hi Julien,
It could be the root cause. I wouldn't give just 1536 bytes but slightly more, maybe 2kB. You may want to run some memory analysis to know how much application's stack required. For example: https://www.segger.com/products/development-tools/embedded-studio/technology/code-analysis/#stack-usage

RE: Product crash with nrf51822 peripheral s130 2.0.1 : how to get logs or more information ?

Julien Cosmo — Thu, 20 Oct 2022 14:34:35 GMT

Thanks !

The current configuration is (only) 1280 bytes for the stack size. We are also testing a new version with 1792 bytes (it's difficult to go further, we use a lot of RAM, but we can't reproduce on this version so far). Our previous firmware had 2KB.

We do not use heap (set to 0)

So it may be the root cause of our crash ?

RE: Product crash with nrf51822 peripheral s130 2.0.1 : how to get logs or more information ?

Hung Bui — Thu, 20 Oct 2022 14:14:37 GMT

Hi Julien,
Could you give the exact call stack configuration you have in your project ?
S130 requires at least 1536 bytes. The stack configuration in your project is shared between softdevice and application. So it also depends on how many bytes your application using. Have you tried to reduce heap and increase stack ?
By default we don't use HCI mem pool in most of our examples so I don't think it's an issue. I assume you don't use it in your application ?

RE: Product crash with nrf51822 peripheral s130 2.0.1 : how to get logs or more information ?

Julien Cosmo — Thu, 20 Oct 2022 10:11:40 GMT

We are trying to reproduce with debug and/or logs, but it's very difficult, and we didn't managed so far.

One tip: we were able to reproduce the issue with all notifications deactivated. So it seems it's not linked to notifications. And the trace show exactly the same stop in the the same service request.

One question : do you think that an issue in the stack allocated size may produce this kind of crash ? Due to ram limitation (we are pretty full on RAM side), we needed to reduce the stack size compare to previous firmware version.

Also here is the sdk config we used, I don't know if this could help : devzone.nordicsemi.com/.../7635.sdk_5F00_config.h

We are wondering if HCI_MEM_POOL_ENABLED set to 0 may have a link to our problem.

Another question : Where can I find more details on what's happening during the service discovery process ?

Thanks again !

Julien

RE: Product crash with nrf51822 peripheral s130 2.0.1 : how to get logs or more information ?

Hung Bui — Wed, 19 Oct 2022 16:03:43 GMT

Hi Julien,

Thanks for the sniffer. Yes it looked like a crash right after a read by type request. But what causing this we don't know.
If you managed to find a way to reproduce the issue please try to debug and check the log.

You can also take more sniffer trace with a central that you have more control over (for example nRF Connect Dekstop on PC or nRF5 as a central) so that you can control when you want to do service discovery. By having that we can be sure if it was because of service discovery or not.
Usually service discovery is handled by the softdevice and should not cause any issue.

RE: Product crash with nrf51822 peripheral s130 2.0.1 : how to get logs or more information ?

Julien Cosmo — Wed, 19 Oct 2022 13:33:49 GMT

I'm more thinking about a bad parameter we send to the softdevice... Or a race condition somewhere. But it's still very unclear to me.

Here is a sniffer trace we just manage to get !

devzone.nordicsemi.com/.../capture-crash-ride.pcapng

After a series of connection/restart, the last command send by nrfconnect (at the end of the file) seems to be:

6743 1148.847139 Master_0xaf9a959f ATT 37 Sent Read By Type Request, GATT Characteristic Declaration, Handles: 0x000c..0x0013 Slave_0xaf9a959f

There is no answer, and the peripheral device is then totally blocked.

RE: Product crash with nrf51822 peripheral s130 2.0.1 : how to get logs or more information ?

Hung Bui — Wed, 19 Oct 2022 13:19:36 GMT

Hi Julien,
No, not really. The softdevice S130 is pretty robust and it's been using in millions of devices for many years (almost 10 years). I am not aware of any report about the softdevice being crashed and keep the CPU in an infinite loop, no. If there is any crash, it would either throw a fault or go straight to a hardfault and reset.

Regarding the log, it seems that the connection is terminated before discovery completed but it's hard to tell what exactly wrong here. If you can capture a sniffer trace when the issue happen it would be very useful.

RE: Product crash with nrf51822 peripheral s130 2.0.1 : how to get logs or more information ?

Julien Cosmo — Wed, 19 Oct 2022 12:59:29 GMT

Thanks for you answer !

Currently we can not reproduce the issue with the log (it's very difficult to reproduce), but we are still trying.

On the application side, everything is blocked (software-blinking leds stopped for example - even if PWMs are still active) - without a reset.

Thanks about the softdevice_fault_handler.

But as far as I can see, softdevice_fault_handler or other faults will generate a reset (I did not changed app_error_fault_handler, and I'm not in DEBUG) - and it's not the case here.

Have you ever heard about a possible crash or infinite loop during the Discovering services on the periph side ? It may be linked to a bonding process (bonding should not append, but it's not forbidden and not correctly managed either) and/or a cold start (it seems for us easiest to reproduce the issue after a cold start) and/or a possible notification send (at the same time ?)

We are thinking about an issue there because logs from nrfconnect seems to show a time out on the discovering services request :

V	17:49:23.129	Connecting to EA:A3:51:3B:C8:AB...
D	17:49:23.130	gatt = device.connectGatt(autoConnect = false, TRANSPORT_LE, preferred PHY = LE 1M)
D	17:49:23.625	[Broadcast] Action received: android.bluetooth.device.action.ACL_CONNECTED
D	17:49:23.640	[Callback] Connection state changed with status: 0 and new state: CONNECTED (2)
I	17:49:23.640	Connected to EA:A3:51:3B:C8:AB
V	17:49:23.640	Discovering services...
D	17:49:23.640	gatt.discoverServices()
I	17:49:23.965	Connection parameters updated (interval: 7.5ms, latency: 0, timeout: 5000ms)
D	17:49:29.145	[Callback] Connection state changed with status: 0 and new state: DISCONNECTED (0)
I	17:49:29.145	Disconnected
D	17:49:29.173	[Broadcast] Action received: android.bluetooth.device.action.ACL_DISCONNECTED

Regards,

Julien

RE: Product crash with nrf51822 peripheral s130 2.0.1 : how to get logs or more information ?

Hung Bui — Wed, 19 Oct 2022 10:21:43 GMT

Hi Julien,

Do you see anything in the log ? Was the application continue to work (without BLE) after the fault happened ?

In most case if the softdevice crash or has a fault it will call the softdevice_fault_handler() function with the info and fault ID about the fault.
You may need to define app_error_fault_handler() so that it overwrite the weak declaration in the app_error_weak.c if you want to print more log. But if you can reproduce the issue you can just put a breakpoint inside app_error_fault_handler() in app_error_weak.c.

If you have a look inside nrf_sdm.h you can find NRF_FAULT_ID_SD_RANGE_START it's where the fault ID for softdevice start.

RE: Product crash with nrf51822 peripheral s130 2.0.1 : how to get logs or more information ?

Julien Cosmo — Tue, 18 Oct 2022 16:13:50 GMT

One update : the crash may occurs during the "Discovering services" request (send by the central - here a phone using nrfconnect). It's also possible that a notification occurs pretty fast after the connection, and so maybe during the discovering service (I don't know if it's possible, and/or may result to a crash on the periph side).