Socket API "send" hangs when sending multiple TCP messages

RE: Socket API "send" hangs when sending multiple TCP messages

Carl Richard — Thu, 10 Jun 2021 14:16:22 GMT

Hello again, Omer!

Apologies for the slow response from me. I hope your testing has gone well! To use the net management API you must include net_mgmt.h and enable it in prj.conf by adding CONFIG_NET_MGMT=y. To obtain "iface" the developers suggest that the easiest way will be to use net_if_get_ieee802154() function (available in net/net_if.h).

Hope this is clear and please reach out if you face any other issues.

Best regards,
Carl Richard

RE: Socket API "send" hangs when sending multiple TCP messages

Omer Farooq — Tue, 01 Jun 2021 17:07:57 GMT

Hi Carl,

Thanks very much for your reply and help. I am using NCS v1.3.0, and now I have switched to NCS v1.5.0. In my initial tests, the application is no longer hanging after I produce an executable using NCS v1.5.0. I will do more rigorous testing, and if I found something I will get in touch.

I tried to use net_mgmt(NET_REQUEST_IEEE802154_SET_ACK, iface, NULL, 0) to enable L2 ACKs, however I got compile time errors. One of the error is "NET_REQUEST_IEEE802154_SET_ACK" not defined. Could you please tell me what header files I need to include so the "NET_REQUEST_IEEE802154_SET_ACK" can be found? Second, could you please also tell me how I can obtain "iface" on nRF52840 DK so that I can pass it to the net_mgmt function?

Once again thank you very much for your help.

Best regards,

Omer

RE: Socket API "send" hangs when sending multiple TCP messages

Carl Richard — Wed, 26 May 2021 14:53:50 GMT

Hello again, Omer!

We have not managed to reproduce your issue with the applications you attached. Our tests were done using NCS v1.5.0. What version are you running?

The developers also noted that setting the ACK should be done differently when using the nRF driver for 802.15.4. To do this in runtime you need to o configure the L2 so that it sets the ACK request bit in the outgoing frames:

net_mgmt(NET_REQUEST_IEEE802154_SET_ACK, iface, NULL, 0);

For testing purposes this can also be done using the ieee802154 shell command: ieee802154 ack set

Could you also share the HW version of your nRF52840 DKs and the file "<project_root>/zephyr/include/generated/autoconf.h"?

Best regards,
Carl Richard

RE: Socket API "send" hangs when sending multiple TCP messages

Omer Farooq — Thu, 20 May 2021 12:53:40 GMT

Hi Carl,

Thanks a lot for the detailed response. I was using UDP for the echo client/server application, and yes reliability can be an issue. However, I did enable layer 2 ACKs by having the following in proj.conf file, "CONFIG_NET_L2_IEEE802154_ACK_REPLY=y". Based on zephyr documentation I think this is the way to enable layer 2 ACKs, right? With all the modifications to the proj.conf and overlay files that you have suggested in your recent answer the problem still exists.

Second, the original application that I am developing is based on TCP. TCP is a reliable protocol, i.e., if the message gets lost then TCP will automatically retransmit the message. Hence, the hanging issue is not due to lost packets.

Third, I modified echo client/server application so that the applications use TCP instead of UDP for communication. Again, I also incorporated the changes you have suggested in the proj.conf and overlap files, however the hanging issue still exists (with this message I am attaching all the sources). For a few minutes the messages are exchanged, however afterwards the system hangs.

I suspect there might be an issue new "net_pkt buffer_allocation". May be buffers are not corrected deallocated, and it might be causing the issue while buffer allocation. My suggestion would be to run the attached pieces of codes, and your team may identify the problem.

Best regards,

Omerdevzone.nordicsemi.com/.../my_5F00_echo_5F00_server.zip devzone.nordicsemi.com/.../my_5F00_echo_5F00_client.zip

RE: Socket API "send" hangs when sending multiple TCP messages

Carl Richard — Wed, 19 May 2021 14:08:56 GMT

Hi again, Omer!

I've gotten some feedback from the developers. Listing it here:

For many devices the chosen entropy node is Cryptocell by default. This can immensely slow down the applications. You can change this by adding nrf52840dk_nrf52840.overlay files containing the code below in your applications.

/ {
	/*
	* In some default configurations within the nRF Connect SDK,
	* e.g. on nRF52840, the chosen zephyr,entropy node is &cryptocell.
	* This devicetree overlay ensures that default is overridden wherever it
	* is set, as this application uses the RNG node for entropy exclusively.
	*/
	chosen {
		zephyr,entropy = &rng;
	};
};

They suggest using the applying the changes below to the overlay-802154.conf of the applications to ensure correct operation on the nRF devices.

diff --git a/samples/net/sockets/echo_client/overlay-802154.conf b/samples/net/sockets/echo_client/overlay-802154.conf
index 2fc07cf685..5707ebbffe 100644
--- a/samples/net/sockets/echo_client/overlay-802154.conf
+++ b/samples/net/sockets/echo_client/overlay-802154.conf
@@ -1,7 +1,8 @@
 CONFIG_BT=n
 
 # Disable TCP and IPv4 (TCP disabled to avoid heavy traffic)
-CONFIG_NET_TCP=n
+CONFIG_NET_TCP=y
+CONFIG_NET_UDP=n
 CONFIG_NET_IPV4=n
 
 CONFIG_NET_CONFIG_NEED_IPV6=y
@@ -14,5 +15,6 @@ CONFIG_NET_CONFIG_PEER_IPV6_ADDR="2001:db8::1"
 CONFIG_NET_L2_IEEE802154=y
 CONFIG_NET_L2_IEEE802154_SHELL=y
 CONFIG_NET_L2_IEEE802154_LOG_LEVEL_INF=y
+CONFIG_NET_L2_IEEE802154_FRAGMENT_REASS_CACHE_SIZE=8
 
 CONFIG_NET_CONFIG_IEEE802154_CHANNEL=26

Lastly, they noticed (at least for you echo client/server test) that may lead to hangs/instability. Quoting:

Their client app, after sending the UDP packet enters the recv() function in a blocking manner, w/o any timeout configured. This means that the client assumes that it'll always receive a response from the server. This is not a correct approach in general, but it's even worse in lossy networks like 802.15.4, where it's not that uncommon that either request or reponse may be lost. Especially that Zephyr's 802.15.4 MAC does not use ACK mechanism by default (have to be enabled explicitly in the config file).

Please make sure to take the above into account!

If anything is unclear, please reach out!

Best regards,
Carl Richard

RE: Socket API "send" hangs when sending multiple TCP messages

Omer Farooq — Tue, 18 May 2021 15:28:23 GMT

Hi Carl,

Thanks for the message. I did try to increase the buffers you have mentioned, however I still observe the same behavior.

TCP/UDP: In my own application, I am using TCP and my application hangs. In the echo client server application, I am using UDP, and these applications also hang. Therefore, we cannot say that there is no issue with TCP/UDP implementation.

Best regards,

Omer

RE: Socket API "send" hangs when sending multiple TCP messages

Carl Richard — Tue, 18 May 2021 12:25:58 GMT

Hi again!

The developers are looking into this now. In the meantime, could you try to increase the following buffers and see if that give you better results?

CONFIG_NET_PKT_RX_COUNT
CONFIG_NET_PKT_TX_COUNT
CONFIG_NET_BUF_RX_COUNT
CONFIG_NET_BUF_TX_COUNT

And for some clarification; I see that TCP isn't enabled in the echo samples you tested last. Hence, we can assume that this isn't related to the TCP/UDP operation and rather some lower level issue. Agree?

Best regards,
Carl Richard

RE: Socket API "send" hangs when sending multiple TCP messages

Carl Richard — Fri, 14 May 2021 08:20:10 GMT

Hello again, Omer!

Apologies for the delayed answer and thanks for the additional information! It seems like this might be a bug on our side. I'll report this internally and will get back to you soon.

Best regards,
Carl Richard

RE: Socket API "send" hangs when sending multiple TCP messages

Omer Farooq — Fri, 07 May 2021 15:48:55 GMT

Hi Carl,

I have tested echo-client/server example by transmitting data packets in a loop. I can now confirm that the blocking behavior is reproduced in the echo-client/server example as well. I have attached the source files and proj.conf file. Please note that I am using nRF Connect SDK Version 1.3.0, however I have also tested on over nRF Connect Version 1.5.0. devzone.nordicsemi.com/.../6472.prj.conf devzone.nordicsemi.com/.../echo_2D00_client.c devzone.nordicsemi.com/.../echo_2D00_server.c

RE: Socket API "send" hangs when sending multiple TCP messages

Omer Farooq — Fri, 07 May 2021 12:48:59 GMT

Thanks for your reply. Unfortunately, I will not be able to share my application code here due to restrictions.

The problem nearly starts to happen after the same number of messages, but the number is not exactly the same very time. Can I suggest you to please use net/sockets echo client server example and transmit messages in an loop, you may reproduce the same problem?

Best regards,

Omer

RE: Socket API "send" hangs when sending multiple TCP messages

Carl Richard — Fri, 07 May 2021 11:57:29 GMT

Thanks for the logs! I couldn't manage to find any obvious problems in them. Could you share the applications here so that I can have a look?

This could possible be an overflow issue. Does it happen after the same number of messages every time?

Best regards,
Carl Richard

RE: Socket API "send" hangs when sending multiple TCP messages

Omer Farooq — Tue, 04 May 2021 09:37:37 GMT

Client log.

devzone.nordicsemi.com/.../1055.clientlog.txt

RE: Socket API "send" hangs when sending multiple TCP messages

Omer Farooq — Tue, 04 May 2021 09:36:55 GMT

Hi Carl,

Thanks for the reply. I did add the CONFIGs to my proj.conf files for the client and the server applications. My application works the following way.

- Once a user presses button 3 on nrf52840 dk the communication starts, however as we now have CONFIG_ASSERT=y in the proj.conf file, my application resets when I press the button on the client application please see attached screenshot. This does not happen for the server application.

If I remove CONFIG_ASSERT=y, the above does not happen. In order to reproduce the erroneous scenario, I removed the assert config, and execute the client server applications. Please see attached logs. You can see at the end of the logs server application requested data, client did send the data, however server is stuck. devzone.nordicsemi.com/.../serverlog.txt devzone.nordicsemi.com/.../clientlog.txt

RE: Socket API "send" hangs when sending multiple TCP messages

Carl Richard — Tue, 04 May 2021 07:44:04 GMT

Hi again!

Understood. Could you add the following to your prj.conf and share any eventual output with me?

CONFIG_ASSERT=y
CONFIG_SERIAL=y

CONFIG_LOG=y
CONFIG_LOG_IMMEDIATE=y
CONFIG_LOG_BACKEND_SHOW_COLOR=n

CONFIG_NET_LOG=y

#CONFIG_NET_IF_LOG_LEVEL_DBG=y
#CONFIG_NET_L2_ETHERNET_LOG_LEVEL_DBG=y

CONFIG_NET_SOCKETS_LOG_LEVEL_DBG=y
CONFIG_NET_CONN_LOG_LEVEL_DBG=y
CONFIG_NET_CONTEXT_LOG_LEVEL_DBG=y

CONFIG_NET_TCP_LOG_LEVEL_DBG=y

Best regards,
Carl Richard

RE: Socket API "send" hangs when sending multiple TCP messages

Omer Farooq — Mon, 03 May 2021 17:31:28 GMT

Hi Carl,

Regarding logs - I am obtaining information by using printk and displaying the information on terminal. I am relatively new to Zephyr, therefore I do not know whether Zephyr stores some log files. If so, I would really appreciate if you could also tell me where those log files are stored.

Code Sample: I built my code using the socket client server example available at zephyr/samples/net/sockets.

I am using nRf Connect SDK.

Best regards,

Omer

RE: Socket API "send" hangs when sending multiple TCP messages

Carl Richard — Mon, 03 May 2021 12:14:44 GMT

Hello, Omer!

Are you able to provide any logs from the devices? I would also like information about which code your samples are based on, and whether you are using the nRF Connect SDK or the nRF5 SDK.

Best regards,
Carl Richard