Central connection attempt times out the peripheral connection

We have an nRF5340 device that maintains a peripheral connection to a central device and then makes several of its own central connections to other peripherals.  We ran into a strange issue where this device attempted to form a central connection, but after 1 second (CONFIG_BT_CREATE_CONN_TIMEOUT) a connection failure was reported with reason 0x02 (BT_HCI_ERR_UNKNOWN_CONN_ID).  Three seconds later (4 seconds after the initial connection attempt which is our supervisor timeout), we got notified that our peripheral connection disconnected with reason 0x08 (BT_HCI_ERR_CONN_TIMEOUT).  The central device also reported the reason as 0x08 (BT_HCI_ERR_CONN_TIMEOUT).  After this timeout, the device was able to advertise and regain its peripheral connection quickly.  There were no radio errors reported in the log.

Each time we see this issue, it happens with the same pattern: a central connection is requested, is unsuccessful and then the peripheral connection times out.  It does not happen with every connection attempt.  Sometimes it happens quickly and other times it can take 30min to reproduce it.

Since we are not actively trying to close the connection we think the issue must be lower in the radio code.  One item we wanted to double check was the behavior of the CLOCK_CONTROL_NRF_ACCURACY_PPM setting.  As I understand it, this setting is used to determine the sleep time and receive window for peripheral connections  For a device that just connects to a central it is easy to visualize the impact.  However, for a device that also behaves as a central while maintaining a peripheral connection it is harder to understand the impact of this setting and how it is used by the scheduler?  Can you shed some light on that?  Could an incorrect PPM setting result in the behavior we are seeing?

Most crystals have a 20ppm accuracy at 25C, but then lose accuracy considerably at temperature up to 150ppm.  Should we be using the nominal accuracy for the PPM configuration, or should we be taking into account the worst case ppm over temperature and with crystal aging?

Thanks.

Parents
  • We were hopeful the PPM change fixed the issue, but one of the teams was able to reproduce this issue even with the change.  We are attempting find a better way to reproduce the issue and try to isolate what condition causes it to occur.  We are open to other suggestions why the connection would appear to just go away.

  • Hello,

    I am sorry for the late reply.

    Is it possible to share some sniffer trace of the connection that is dropping out? Also, if possible, also a sniffer trace of the other connection at the same time (two different traces).

    Best regards,

    Edvin

  • Hello,

    I am sorry for the late reply. 

    I don't recall seeing this behavior in this SDK version (v2.6.6). The first thing that comes to mind was anomaly 165. It may sound like you want to try the network core monitor. 

    Enable it with CONFIG_NET_CORE_MONITOR=y on both cores.
    By default, the __weak handler only prints information about the event. To react to a reset or freeze, for example by recovering or rebooting, you must override ncm_net_core_event_handler().

    One example is to add this to main:

    #include "net_core_monitor.h"
    ...
    /* This is the override for the __weak handler. */
    void ncm_net_core_event_handler(enum ncm_event_type event, uint32_t reset_reas)
    {
       switch (event) {
       case NCM_EVT_NET_CORE_RESET:
          printk("The network core reset.\n");
          /* do something */
          break;
       case NCM_EVT_NET_CORE_FREEZE:
          printk("The network core is not responding.\n");
          /* do something */
          break;
       }
    }

    Both /*do something*/ should be replaced by full device resets, as there is no real graceful recovery from this kind of events. But first and foremost, it would be interesting to see whether you see these events at all.

    Best regards,

    Edvin

  • We do have CONFIG_NET_CORE_MONITOR enabled in our project already since we have had net core asserts in the past that have been hard to reproduce.  This particular issue does not cause any assert from the net core.  It looks like the connection just goes away.  From what we can tell, all active connections get disconnected, and then it has no problem reconnecting after that without special intervention.

    It does not appear to occur if the connections are made and static.  It seems to occur mostly when a new connection is being made, but we believe we have seen it also occur when a connection was forcefully disconnected.

    We have a test setup where we constantly make and break connections, and have not been able to reproduce the issue that way, so there is some sort of additional component and not just connecting and disconnecting.

  • It does sound like you are already doing this, but just to make sure, you are monitoring the ncm_net_core_event_handler() events, right?

    umr_engnr said:
    It does not appear to occur if the connections are made and static.

    What do you mean by this? 

    Best regards,

    Edvin

  • I can confirm that we are monitoring ncm_net_core_event_handler() and have not seen it get called.

    The project I am working on consists of a main hub device which connects to several stationary devices, then there are non-stationary devices that roam between the stationary devices.  What we have seen is that sometimes when roaming one of the non-stationary devices will lose all its connections.  After a supervisor timeout, it attempts to reconnect to the hub and does so just fine.  Unfortunately, this disconnect results in a poor user experience and we need to fix it.

    My comment "It does not appear to occur if the connections are made and static." was meant to relay that if the system is connected and operational but no devices are roaming then the issue never appears to happen regardless how much time it is left to run.  So it appears to be due to the connections being connected or disconnected.

    Additionally, we have setup a test where the roaming devices are forced to connect and disconnect continuously.  We can leave that test running for days without issue.  So their appears to be more to the issue than just making/breaking connections.

    The issue sometimes occurs on the first roaming jump, but other times can take hours.  To me this indicates some sort of timing issue where all the stars align to cause the issue.

    Since the BLE stack reports that the disconnection occurred due to a supervisor timeout and we get no net core warnings, it seems to me that maybe the scheduler timing is getting confused.  Since we are having an issue reproducing the issue on command, I was hoping you could assist me by reasoning through what could possibly cause a condition like this which would help us attempt to reproduce it more reliably.  Some thoughts we had:

    1. Corruption of the scheduler data resulting in missing connection events.

    2. Radio lockup due to resource contention that is remedied once the connections disconnect.

  • Hello, and thanks for the description.

    Without being able to reproduce the issue, it is difficult to say the exact cause.

    What does come to mind, without it being an nRF5340-specific cause is that we have seen occasions where disconnected events are being dropped, but this only occurs if the SoftDevice Controller's event queue is full. The SoftDevice Controller does try to prioritize the disconnected event, so it is rare. Could it be that you are spending a lot of time in any of the other BLE events, causing the SoftDevice Controller's event queue to fill up in the meantime?

    If that is the case, we would recommend leaving the events quickly, storing the data from them somewhere, and processing them outside the event handler, to allow new events from the softdevice to reach the application layer.

    I know this is all a bit high-level, but hopefully it gives some pointers on where to look. If you are able to replicate the issue, we would be very interested in reproducing the issue locally to investigate the cause of the missing disconnected event.

    BR,
    Edvin

Reply
  • Hello, and thanks for the description.

    Without being able to reproduce the issue, it is difficult to say the exact cause.

    What does come to mind, without it being an nRF5340-specific cause is that we have seen occasions where disconnected events are being dropped, but this only occurs if the SoftDevice Controller's event queue is full. The SoftDevice Controller does try to prioritize the disconnected event, so it is rare. Could it be that you are spending a lot of time in any of the other BLE events, causing the SoftDevice Controller's event queue to fill up in the meantime?

    If that is the case, we would recommend leaving the events quickly, storing the data from them somewhere, and processing them outside the event handler, to allow new events from the softdevice to reach the application layer.

    I know this is all a bit high-level, but hopefully it gives some pointers on where to look. If you are able to replicate the issue, we would be very interested in reproducing the issue locally to investigate the cause of the missing disconnected event.

    BR,
    Edvin

Children
No Data
Related