Connection hangs/drops randomly without triggering callback

[i_4556]

Hello and thanks for reading.

I have a critical problem with my device, and seeing as the problem is BLE related I figured I would ask your help as well. We are using the NRF5340 as a peripheral. There are two operating modes; in one, the peripheral acts as an HID mouse; a computer connects, we perform pairing (security level 2), and then the device sends movements and clicks based on on-board sensors. In the other, we use a custom service to send back live data from those same sensors (basically NUS).

We have tested both modes, using an Amazon tablet running Android, an IOS phone, and windows 10, and in all modes and operating systems, we have the same critical failure. After a random amount of time, typically around 4-5 minutes, the connection seems to mysteriously hang.

On the Central side, it shows as disconnected. So for example when the device is acting as a mouse, windows will move the device from connected to paired, and the cursor stops moving. When we try our custom service, we stop getting data and on windows the program we use enters it's disconnect routine.

On the peripheral side, its like the connection slipped out in the night without leaving a note. The disconnect is never called, and even with BT_DEBUG enabled I never get the message telling me the connection has disconnected with the reason. I know the NRF hasnt reset or something because I continue to get log messages in the console, shortly followed by warnings about overflowing buffers (because without the connection active theres nowhere for the data to go).

Thinking that maybe the gathering data thread was just monopolizing the processor or something, I tried connecting with our application, but never sending the command to start gathering data. The windows application just sits there and receives the heartbeat message every 5 seconds. The problem persists.

At this point I need to reach out for help, as we need to get this fixed quickly so we can start final testing and shipping. Essentially our bug has the connection being abandoned after a random amount of time, central disconnects, and then the disconnect callback is never called. The desired behavior is that we dont disconnect unexpectedly.

We are using Visual studio code and SDK 2.5.0 with zephyr if that helps.

Looking around on the forums, one thought is that maybe we dont have our clock tree set up perfectly or something? We are using an external oscillator for the HF clock but with the internal capacitors, and for the LF clock we need the internal resonator and capacitors. How _should_ our project config look for that configuration? Right now I have the following, which at least seems to be at the correct tick rate; a 1ms delay takes one ms:

In the network core:

#clock config

CONFIG_CLOCK_CONTROL_NRF_K32SRC_RC=y

CONFIG_CLOCK_CONTROL_NRF_K32SRC_RC_CALIBRATION=y

CONFIG_CLOCK_CONTROL_NRF_K32SRC_500PPM=y

And in the main core:

#SYSTEM & CLOCK

CONFIG_CLOCK_CONTROL_NRF_K32SRC_RC=y

Here is my full config file:

#GPIO
CONFIG_GPIO=y
CONFIG_PINCTRL=y
# CONFIG_DK_LIBRARY=y

#UART & SERIAL
CONFIG_SERIAL=y
CONFIG_NEWLIB_LIBC=y
CONFIG_NEWLIB_LIBC_FLOAT_PRINTF=y

#SYSTEM & CLOCK
CONFIG_CLOCK_CONTROL_NRF_K32SRC_RC=y
CONFIG_MAIN_STACK_SIZE=16384
CONFIG_LOG=y
CONFIG_SYSTEM_WORKQUEUE_STACK_SIZE=8192

#CPP
CONFIG_CPP=y
CONFIG_REQUIRES_FULL_LIBCPP=y
CONFIG_STD_CPP20=y

#i2C
CONFIG_I2C=y

#spi
CONFIG_SPI=y

#flash
CONFIG_SETTINGS=y
CONFIG_FLASH=y
CONFIG_FLASH_PAGE_LAYOUT=y
CONFIG_NVS=y
CONFIG_FLASH_MAP=y
CONFIG_MPU_ALLOW_FLASH_WRITE=y
CONFIG_FILE_SYSTEM=y
CONFIG_SOC_FLASH_NRF=y

#math
CONFIG_CMSIS_DSP=y
CONFIG_NEWLIB_LIBC=y
# CONFIG_FPU=y
# CONFIG_CMSIS_DSP_TRANSFORM=y
# CONFIG_TFM_ENABLE_CP10CP11=y
CONFIG_CMSIS_DSP_FILTERING=y

#BLE
CONFIG_BT=y
CONFIG_BT_SETTINGS=y
CONFIG_BT_GATT_CLIENT=y
CONFIG_BT_PERIPHERAL=y
# CONFIG_BT_DEVICE_NAME="AAVAANRF" #Set dynamically in BLE_COMMANDS.h
CONFIG_BT_DEBUG_LOG=y
CONFIG_BT_CENTRAL=y
CONFIG_BT_BROADCASTER=y
CONFIG_BT_OBSERVER=y

CONFIG_BT_DEVICE_NAME_DYNAMIC=y
CONFIG_BT_DEVICE_NAME_MAX=30

#NUS
CONFIG_BT_NUS=y
CONFIG_BT_NUS_AUTHEN=n

#DFU
CONFIG_BOOTLOADER_MCUBOOT=y
CONFIG_NCS_SAMPLE_MCUMGR_BT_OTA_DFU=y




#BT BATTERY SERVICE (BAS)
CONFIG_BT_BAS=y

#DIS
CONFIG_BT_DIS=y
CONFIG_BT_DIS_PNP=y
CONFIG_BT_DIS_MANUF="AAVAA"
CONFIG_BT_DIS_PNP_VID_SRC=2
CONFIG_BT_DIS_PNP_VID=0x1915
CONFIG_BT_DIS_PNP_PID=0xEEEE
CONFIG_BT_DIS_PNP_VER=0x0100

#BT HID
CONFIG_NCS_SAMPLES_DEFAULTS=y

CONFIG_BT_MAX_CONN=6
CONFIG_BT_MAX_PAIRED=20
CONFIG_BT_SMP=y
CONFIG_BT_L2CAP_TX_BUF_COUNT=5
CONFIG_BT_DEVICE_APPEARANCE=962

CONFIG_BT_HIDS=y
CONFIG_BT_HIDS_MAX_CLIENT_COUNT=2
CONFIG_BT_HIDS_DEFAULT_PERM_RW_ENCRYPT=y
CONFIG_BT_GATT_UUID16_POOL_SIZE=40
CONFIG_BT_GATT_CHRC_POOL_SIZE=20

CONFIG_BT_CONN_CTX=y

#2MEG PHY & MTU
# CONFIG_BT_CTLR_PHY_2M=y
# CONFIG_BT_AUTO_PHY_UPDATE=y
# CONFIG_BT_USER_PHY_UPDATE=y

CONFIG_BT_BUF_ACL_RX_SIZE=256
CONFIG_BT_L2CAP_TX_MTU=252

CONFIG_BT_PERIPHERAL_PREF_MIN_INT=6
CONFIG_BT_PERIPHERAL_PREF_MAX_INT=24
CONFIG_BT_PERIPHERAL_PREF_LATENCY=0
CONFIG_BT_PERIPHERAL_PREF_TIMEOUT=400

#ADC
CONFIG_ADC=y
CONFIG_ADC_ASYNC=y

#timer
CONFIG_COUNTER=y

I can post any other information as needed.

Thanks for your time!

[Simonr]

Hi

This issue seems very strange to be honest, and I don't really have some working theories of what this could be caused by based on this description. Are you also able to reproduce this with an nRF5340 DK running this application? It would also be helpful to see a sniffer trace so we can get some info of what's happening over-the-air. The clock configuration looks okay to me, but it does seem like the radio on the nRF5340 side stops transmitting all of a sudden. Does a disconnect occur eventually, when the connection times out with the central device?

Best regards,

Simon

[i_4556]

Hello and thank you for your answer. I am testing with the dev board now.

In the meantime, a thought occurred to me; I am not using the watchdog timer, in either the network or main core. Are they disabled by default, or do I need to manually disable them?

When the problem occurs, it does seem to happen around the 4 minute mark, and I am wondering if there is something significant about that timeframe.

I have also tested powering the device from USB rather than battery, and the problem persists, so I dont think the problem is related to a weak battery.

EDIT: I have tested with the sniffer, please find the file attached. Skip to packet 35876, that is the last good packet. It really looks like the network core just stops suddenly for some reason... It never stops/disconnects either, even twenty minutes later my application core still thinks it has a good connection...

nrf5340_netwrok_error.pcapng

EDIT2: I may have gotten it working, in that in ten 5 minute tests I have experienced zero errors. Unfortunatly I made a bunch of changes so I dont know what did it, but heres what I did:

1) Homogenized the clock configs in both the main and network core, they now both contain

CONFIG_CLOCK_CONTROL_NRF_K32SRC_RC=y

CONFIG_CLOCK_CONTROL_NRF_K32SRC_RC_CALIBRATION=y

CONFIG_CLOCK_CONTROL_NRF_K32SRC_500PPM=y

CONFIG_SOC_HFXO_CAP_EXTERNAL=y

CONFIG_SOC_HFXO_CAP_INTERNAL=n

2) Added the following magic line of code to the network core config

CONFIG_RESET_ON_FATAL_ERROR=n

3) Removed the ability to have multiple centrals connect to my peripheral. for now we arent using this feature, so not a huge deal. In my network core config I commented out

# CONFIG_BT_CTLR_SDC_PERIPHERAL_COUNT=3

4) Removed the ability to store pairing information. I do need to add this feature back in at some point, but for now in my disconnect callback I have

bt_unpair(BT_ID_DEFAULT, BT_ADDR_LE_ANY);

Does any of this make sense, could any of this plausibly be the problem?