Unusual Wi-Fi behavior with net_sock and net_mgmt requests

[BENBF]

Hello,

We are doing some Wi-Fi connection tests on our custom board equipped with the nRF5340 and nRF7002 chips, and at some point during our tests, we encountered a strange error that we do not understand nor know how to fix.

The test goes as follows:

1. Start up the Wi-Fi with net_if_up()
2. Request a connection to the Wi-Fi net_mgmt NET_REQUEST_WIFI_CONNECT
3. Once connected, we send a small packet to a server
4. Disconnect from the Wi-Fi with net_mgmt NET_REQUEST_WIFI_DISCONNECT
5. Shut down the WiFi with net_if_down()
6. Wait 5 minutes and then repeat from Step 1

Now, our tests run smoothly at the beginning, always able to connect to the same Wi-Fi, send a packet, shut down, etc. However, after hours of doing it (about 23 hours), the Wi-Fi starts to fail more often. Initially, it sometimes fails the connection by giving the error -116 within the Wi-Fi connection callback. Then it will always fail with the error -116, and ultimately the net_mgmt Wi-Fi connect request will fail, returning -1. The following log is displayed: "<err> net_sock: invalid access on sock 6 by thread 0x20001ce8".

We began analyzing the behavior of threads to see if thread 0x20001ce8 might have been leaking, but we didn't find anything unusual compared to previous instances when the Wi-Fi connection worked.

Here's an overview of our connection request function:

int8_t wifi_connect(void)
{
    struct net_if *interface = net_if_get_default();
    static struct wifi_connect_req_params connection_parameters;

    printk("<Inf> Connection requested\n");
    wifi_configuration_reader(&connection_parameters);
    int status = net_mgmt(NET_REQUEST_WIFI_CONNECT, interface,
                          &connection_parameters,
                          sizeof(struct wifi_connect_req_params));
    if (status) {
        printk("<Err> Connection request failed (%d)\n", status);
        return -ENOEXEC;
    }
    return 0;
}

We tried replacing all references to `net_if_get_default()` with `net_if_get_first_wifi()`, but it did not prevent the bug from occurring.

We couldn't find many references to this encountered issue, and we hope the information we provided is sufficient for you to assist us. Please reach out to us if you need more information.

Thank you for your time and have a good day!

[Charlie]

Hi Benjamin,

Thanks for checking with us about this issue. As you mentioned, memory leaking is highly suspect.

For the next steps, there are two things worth to try:

1. Can you test without step 3 and comment related codes about this step? This will help to confirm the issue is actually caused by Wi-Fi.

2. Could you try to change to another AP or just reset AP when this issue happens?

Additional debugging information can be collect by following this document page: Debugging — nRF Connect SDK 2.6.1 documentation (nordicsemi.com)

Best regards,

Charlie

[BENBF]

Hello,

We are reaching out to you again as we had the opportunity to create a test environment with your criteria that was running during this weekend. Unfortunately, even without step 3, the bug happened again about 10 hours after the start of the test. We launched another test on another card on the same AP at the same time, with the only difference being that instead of connecting every 5 minutes, it connected every hour. This test ran smoothly without any bugs or issues.

As for trying to launch with a debugger, we could not do that as we are conducting very long tests and cannot be there to catch the error in real time. We are sorry for that.

We hope this bit of information could help you assist us. Thank you for your time and have a great day.

[Charlie]

Ok, have you done any test on nRF7002DK? Could you share your testing codes? 

I can try to test your codes with nRF7002DK to see if I can repeat this issue.

[BENBF]

Hello,

I apologize for the delay. Our codebase is very large, and bundling it to run on the nRF7002DK was quite tedious. However, we were able to recreate the bug with the code that I will attach to this message. It took some time for the bug to appear, but we eventually managed to replicate it. I will also include an example of the logs we extracted during the test, which might be helpful.

Even though the code is small, it can be a bit dense to understand. We apologize for that. To give you the most faithful representation of our real code, we had to import our general algorithm. This might seem unnecessary in this context, as it doesn't contain the entirety of the code, but we hope it doesn't cause too much confusion.

We greatly appreciate your support. Please feel free to reach out if you notice anything odd or have any questions. Thank you very much, and have a great day.

out

bugHunt.zip