Connection hangs/drops randomly without triggering callback

RE: Connection hangs/drops randomly without triggering callback

i_4556 — Wed, 22 May 2024 13:38:28 GMT

Hello,

Thank you for all your help. I will open new tickets for the other issues.

The main reason I suggest making the clock config carry over implicitly is that its a huge trap for new programmers that creates a weird and hard-to-replicate bug. I fully understand that multi-image development is challenging and complicated in many weird and wonderful ways.

Thanks for your consideration!

RE: Connection hangs/drops randomly without triggering callback

Simonr — Wed, 22 May 2024 07:26:57 GMT

Sorry about the late reply here, but we've had a few bank holidays in Norway.

[quote user="i_4556"]maybe NRF should consider revising this in a future version?[/quote]

This is noted and I'll put up a feature request internally on this. The problem is it is two separate images where this must be set which is why it's needed twice as of now I believe. Yes, you should also set this in the MCUBoot image config IIRC.

For the crystal load capacitors we recommend following the following formula where CL is the load capacitance in the crystal datasheet and Ccap is what the optimal capacitor value in our designs would be: Ccap = 2 * CL - 4pF. Removing them to use the internal caps should be perfectly fine, and the settings seem correct.

Your other questions are not really related to the initial ticket, so I'd recommend you create separate tickets for the other questions. If the initial issue reappears it will be easier to follow up in this case if we don't have multiple topics ongoing at the same time. Most peripheral samples support connecting to multiple devices, but in order to connect to multiple at the same time you need to make sure it starts advertising again after connecting to a central.

Best regards,

Simon

RE: Connection hangs/drops randomly without triggering callback

i_4556 — Fri, 17 May 2024 14:03:34 GMT

Hello and thank you for your reply. I am a little surprised that I need to set the clock config twice, in both the app and net cores; this feels like a recipe for mistakes and a landmine for new developers; maybe NRF should consider revising this in a future version? I assume I should also make the same change in my MCUBoot confing too then?

Our oscillator is this one: https://www.mouser.ca/datasheet/2/122/ECX-1048-1115630.pdf for which we have added a set of 10pF ceramic caps. I am not the hardware developer, so I cant really comment on how they were chosen. I know the hardware dev. is threatening to remove them to save space and instead use the internal caps on the NRF; in that case I would need to make the following changes right:

CONFIG_SOC_HFXO_CAP_EXTERNAL=n
CONFIG_SOC_HFXO_CAP_INTERNAL=y
SOC_HFXO_CAP_INT_VALUE_X2=20

Per your suggestions, I have rolled back the change to the CONFIG_RESET_ON_FATAL_ERROR, and everything is still working. We have done more tests, including one overnight, and everything seems to be working.

Thank you for all your help. While you are here, if you dont mind, do you have an example for configuring a peripheral to accept multiple central connections? We need our device to accept to simultaneous centrals.

Also, I seem to be getting some incorrect performance from the flash memory with NVS, in that NVS needs to be primed or something before it works properly. I have been testing with this code:

void Prime_Flash()
{
    //now prime the file system, ive no idea why we need this but it works...
    const float test_value = 0xDEADBEEF; //random number, doesnt actually matter
    float read_value = 0;
    int i = 0;
    while ((i < 25) && (test_value != read_value)) //usually takes 12 tries
    {
        NUS_Send("Write %u %i \n", i, nvs_write(&fs, TEST_REGISTER_ID, &test_value, sizeof(test_value)));
        // nvs_write(&fs, TEST_REGISTER_ID, &test_value, sizeof(test_value));
        k_msleep(10);
        NUS_Send("Read %u %i \n", i, nvs_read(&fs, TEST_REGISTER_ID, &read_value, sizeof(read_value)));
        // nvs_read(&fs, TEST_REGISTER_ID, &read_value, sizeof(read_value));
        k_msleep(10);
        i++;
    }
    NUS_Send("Flash ready took %i tries \n",i);
}

And I get the following results:

Write 0 4
Read 0 -2
Write 1 4
Read 1 -2
Write 2 4
Read 2 -2
Write 3 4
Read 3 -2
Write 4 4
Read 4 -2
Write 5 4
Read 5 -2
Write 6 4
Read 6 4
Write 7 4
Read 7 4
Write 8 4
Read 8 4
Write 9 4
Read 9 4
Write 10 4
Read 10 4
Write 11 4
Read 11 4
Write 12 4
Read 12 4
Flash ready took 13 tries

This happens consistently, where it takes 13 tries before the flash can be used properly. As near as I can tell, once we get that Flash ready then everything works fine until the next reset.

I initialize the flash with:

void Initialise_Flash()
{
    //prepare flash memory
    fs.flash_device = NVS_PARTITION_DEVICE;
    if (!device_is_ready(fs.flash_device)) 
        printk("Flash device %s is not ready\n", fs.flash_device->name);
    fs.offset = NVS_PARTITION_OFFSET;
    flash_get_page_info_by_offs(fs.flash_device, fs.offset, &info);
    fs.sector_size = info.size;
    fs.sector_count = FLASH_SECTOR_COUNT;
    nvs_mount(&fs);
}

And it seems to work, I get the message

I: 2 Sectors of 4096 bytes
I: alloc wra: 0, f40
I: data wra: 0, 54

I doubt this is the expected behavior; do you have any suggestions what I am doing wrong?

Thanks!

RE: Connection hangs/drops randomly without triggering callback

Simonr — Wed, 15 May 2024 06:17:18 GMT

Glad to hear you might have fixed the issue. My guess is setting HFXO_CAP_EXTERNAL=y in both cores is what has fixed this, as without it the network cores might have drifted which would explain the issue occurring not right away but after a while (4-5 minutes). You should still do more testing and over longer periods than just 5 minutes. What are the external capacitor values set to the HF crystal you're using, and can you upload the HF crystal datasheet so I can double check that the capacitor values make sense?

Also, the CONFIG_RESET_ON_FATAL_ERROR=n should only be used for debugging, as if the device runs into a fatal error it will stop entirely and not reset and try to get back up and running. When this is set, you can debug and see where the device gets stuck, but it's not recommended to be used in release products.

As for the watchdog, you should see a watchdog reset occur if the watchdog is triggered. The default value for it is not ~4 minutes if I recall correctly, so I think that is a dead end.

Let me know how your testing goes, and if it still persists I'll do a proper review of the sniffer trace as well.

Best regards,

Simon

RE: Connection hangs/drops randomly without triggering callback

i_4556 — Tue, 14 May 2024 14:41:22 GMT

Hello and thank you for your answer. I am testing with the dev board now.

In the meantime, a thought occurred to me; I am not using the watchdog timer, in either the network or main core. Are they disabled by default, or do I need to manually disable them?

When the problem occurs, it does seem to happen around the 4 minute mark, and I am wondering if there is something significant about that timeframe.

I have also tested powering the device from USB rather than battery, and the problem persists, so I dont think the problem is related to a weak battery.

EDIT: I have tested with the sniffer, please find the file attached. Skip to packet 35876, that is the last good packet. It really looks like the network core just stops suddenly for some reason... It never stops/disconnects either, even twenty minutes later my application core still thinks it has a good connection...

devzone.nordicsemi.com/.../nrf5340_5F00_netwrok_5F00_error.pcapng

EDIT2: I may have gotten it working, in that in ten 5 minute tests I have experienced zero errors. Unfortunatly I made a bunch of changes so I dont know what did it, but heres what I did:

1) Homogenized the clock configs in both the main and network core, they now both contain

CONFIG_CLOCK_CONTROL_NRF_K32SRC_RC=y
CONFIG_CLOCK_CONTROL_NRF_K32SRC_RC_CALIBRATION=y
CONFIG_CLOCK_CONTROL_NRF_K32SRC_500PPM=y

CONFIG_SOC_HFXO_CAP_EXTERNAL=y
CONFIG_SOC_HFXO_CAP_INTERNAL=n

2) Added the following magic line of code to the network core config

CONFIG_RESET_ON_FATAL_ERROR=n

3) Removed the ability to have multiple centrals connect to my peripheral. for now we arent using this feature, so not a huge deal. In my network core config I commented out

# CONFIG_BT_CTLR_SDC_PERIPHERAL_COUNT=3

4) Removed the ability to store pairing information. I do need to add this feature back in at some point, but for now in my disconnect callback I have

bt_unpair(BT_ID_DEFAULT, BT_ADDR_LE_ANY);

Does any of this make sense, could any of this plausibly be the problem?

RE: Connection hangs/drops randomly without triggering callback

Simonr — Tue, 14 May 2024 12:00:55 GMT

This issue seems very strange to be honest, and I don't really have some working theories of what this could be caused by based on this description. Are you also able to reproduce this with an nRF5340 DK running this application? It would also be helpful to see a sniffer trace so we can get some info of what's happening over-the-air. The clock configuration looks okay to me, but it does seem like the radio on the nRF5340 side stops transmitting all of a sudden. Does a disconnect occur eventually, when the connection times out with the central device?

Best regards,

Simon