Persistent (-1) error in POSIX connect() function for TCP reconnection

RE: Persistent (-1) error in POSIX connect() function for TCP reconnection

brown27 — Tue, 07 Nov 2023 21:27:53 GMT

Yes, I am using MCUBoot. I will be on leave shortly, so I will create a new ticket for the v2.5.0 issue when I return in a few weeks.

RE: Persistent (-1) error in POSIX connect() function for TCP reconnection

Maria Gilje — Tue, 07 Nov 2023 13:56:13 GMT

Are you using MCUBoot? In the image trailer section of the MCUBoot Bootloader documentation there is an overview of what values the magic, image-ok, and copy-done fields can have to enter different states. Your case falls into State IV.

I'm not very familiar with MCUBoot, so my recommendation is that you create a new ticket where you share your setup and another engineer can help you more efficiently.

Another reason to open a new ticket is that your original issue was resolved by the upgrade to NCS v2.5.0.

I will help you to the best of my ability if it's not an option for you to create a new ticket.

Best regards,

Maria

RE: Persistent (-1) error in POSIX connect() function for TCP reconnection

brown27 — Mon, 06 Nov 2023 16:46:06 GMT

Thank you, I have switched to NCS v2.5.0 and zephyr SDK v0.16.1 based on your recommendation. I am able to compile and flash the device, but it gets stuck "Jumping to the first image slot."

This seems to be a common issue involving TF-M and SPM, but I have been unable to resolve it. Do you have any recommendations for resolution of this hang-up? Thanks.

RE: Persistent (-1) error in POSIX connect() function for TCP reconnection

Maria Gilje — Mon, 06 Nov 2023 12:28:30 GMT

Hello,

One of our developers has looked into this and has given some feedback.

Your close/open/connect sequence is done correctly.

We need to check for packet leaks:

Enable CONFIG_NET_SHELL=y and CONFIG_NET_DEBUG_NET_PKT_ALLOC=y

Use the net allocs shell command to check for packet leaks.

Use the net conn shell command to check for net_context leaks.

The following is more for information, because it is hard to say if it is related without more information from the shell commands. There was a recent fix for a bug in the TCP handshake (handling RST packets). This was fixed in NCS v2.5.0, and the relevant commit is found here: https://github.com/nrfconnect/sdk-zephyr/commit/1907327297f59993d41f2f7f5af50968b0316289. The bug could cause unexpected behaviour, so it could be worth to switch to NCS v2.5.0 or cherry pick the commit I linked.

Best regards,

Maria

RE: Persistent (-1) error in POSIX connect() function for TCP reconnection

brown27 — Wed, 01 Nov 2023 15:54:56 GMT

I believe my client is restarting the connection request completely each time.

After connect() returns the 116 timeout error, the socket is closed via "close(sock)". The next time a connection is attempted, the application starts from the top with "sock = socket(AF_INET6, SOCK_STREAM, IPPROTO_TCP);" followed by "err = connect(sock, &brok, sizeof(struct sockaddr_in6));".

RE: Persistent (-1) error in POSIX connect() function for TCP reconnection

Maria Gilje — Wed, 01 Nov 2023 15:07:06 GMT

Hello,

Sorry, I haven't been able to find a solution yet. I have a suspicion that you need to set up your client to restart the connection request completely after the SYN messages stops. But I am not sure.

I am looking for the right people to help me with this. Thank you for your patience.

Best regards,

Maria

RE: Persistent (-1) error in POSIX connect() function for TCP reconnection

brown27 — Fri, 27 Oct 2023 19:27:22 GMT

Hi Maria Gilje ,

Did the timeout error and/or my other troubleshooting give you any ideas for next steps?

Thanks.

RE: Persistent (-1) error in POSIX connect() function for TCP reconnection

brown27 — Wed, 25 Oct 2023 19:16:11 GMT

Thank you for the recommendation. I have added #include <errno.h> in this .c file and changed the code around the connect() call as follows:

int errnum;

err = connect(sock, &brok, sizeof(struct sockaddr_in6));
errnum = errno; //errnum currently unused, set just in case it's needed later on
if (err<0)
{
    LOG_ERR("Failed to connect to host for HTTP POST, errno %d. Returning NULL pointer.", errno);
    (void)close(sock);
    return NULL;
}

I also set CONFIG_NET_TCP_LOG_LEVEL_DBG=y for additional information. The resulting error is 116, which appears to be a timeout error. However, these SYN messages still disappear in the Wireshark logs after some time. I trust the Wireshark logs because the device does not reconnect to the server while these SYN messages are missing. If I restart the server while these SYN attempts are still seen in the Wireshark logs, the device reconnects successfully.

I forgot to mention that I also have these configs set (in addition to the configs I listed in my original post) for a fall-back, non-OpenThread form of communication:

CONFIG_MBEDTLS_CIPHER_MODE_CBC=y
CONFIG_MBEDTLS_CIPHER_MODE_CTR=y
CONFIG_MBEDTLS_LIBRARY_NRF_SECURITY=y
CONFIG_MBEDTLS_ENABLE_HEAP=Y
CONFIG_MBEDTLS_HEAP_SIZE=10240
CONFIG_NORDIC_SECURITY_BACKEND=y

RE: Persistent (-1) error in POSIX connect() function for TCP reconnection

Maria Gilje — Wed, 25 Oct 2023 13:32:01 GMT

Hello,

Sorry about the delay.

Per the documentation for connect(), the return value being -1 only indicates that an error has occurred. To find out which error which caused the call to fail, we need to know the value of errno after connect() has returned with an error. Make sure to save the value to another variable before doing any other function calls.

My suggestion for how to do this:

You need to include errno.h and change the variable which is printed in the LOG_ERR on line 100 to be errno instead of err.

You can then find the relevant error code in zephyr/lib/libc/minimal/include/errno.h.

RE: Persistent (-1) error in POSIX connect() function for TCP reconnection

brown27 — Mon, 23 Oct 2023 16:58:39 GMT

Hello,

Were you able to look into this further? This is one of the final issues in our development process.

I just observed an interesting transmission sequence that could help with troubleshooting. The device successfully sent a large number (~650) of locally stored messages to the server. In the middle of clearing this local message backlog, the device started to exhibit the behavior in my original post. This is interesting because the server never went offline and messages were sending successfully until the error was encountered abruptly. There were a few failed transmissions in the middle of the successful sends, but the those messages were always sent successfully on the first retry. Maybe I have a buffer that is filling or sockets that aren't being closed?

^See original post for code that generates this error message. This error occurs for each of the DNS lookups in the bottom-most blue block of Wireshark messages

RE: Persistent (-1) error in POSIX connect() function for TCP reconnection

brown27 — Fri, 20 Oct 2023 17:54:49 GMT

I updated my application to use the Connect SDK v2.4.2 and kept the zephyr TCP implementation. I still experience the same problematic behavior.

I will try updating to the OpenThread TCP implementation to see if that is any better.

Any guidance on resolving the zephyr implementation issue in the meantime would be greatly appreciated!

RE: Persistent (-1) error in POSIX connect() function for TCP reconnection

brown27 — Thu, 19 Oct 2023 15:12:02 GMT

Which nRF Connect SDK version are you using? v2.3.0

Is your project based on a sample? It is based on the TCP/TLS connections in the nrf/subsys/net/lib/azure_iot_hub nRF Connect library. I had my application working great with TLS and a nearly identical code structure. Then I scaled it back to TCP for various reasons.

Have you set up a maximum amount of repetitions for a SYN message on your nRF52840 device? Yes, I generate messages in 1-minute intervals. The device attempts to transmit each message 5 times before moving it to local storage for future transmission.

How is your Thread border router configured? I used the configuration instructions below with the SRP client and upstream DNS functionality enabled. I had no border router issues when doing the TLS version of my application. Thread Border Router - Bidirectional IPv6 Connectivity and DNS-Based Service Discovery | OpenThread

Is your intention to use the Zephyr implementation of TCP instead of the TCP implementation in OpenThread? That was originally my intention because we are using Thread 1.2 and need something released for production. If the nRF Connect SDK was to support Thread 1.3 TCP for production, that might be an option.

RE: Persistent (-1) error in POSIX connect() function for TCP reconnection

Maria Gilje — Thu, 19 Oct 2023 14:27:50 GMT

Hello,

I need some more information from you. Please answer all of the questions below.

Which nRF Connect SDK version are you using?

Is your project based on a sample?

Have you set up a maximum amount of repetitions for a SYN message on your nRF52840 device? You could try to add a restart of SYN message sending to the application which is triggered after a timeout or external input, f.ex.

How is your Thread border router configured?

You have CONFIG_OPENTHREAD_TCP_ENABLE=n in your .conf file. Is your intention to use the Zephyr implementation of TCP instead of the TCP implementation in OpenThread? In Thread 1.3, support for TCP was introduced which makes TCP more efficient in an IEEE 802.15.4 network.

Best regards,

Maria