nRF7002-DK TCP: Data buffer allocation failed

[iotdeveloper]

Hello,

I am solving a problem with nRF7002-DK, I am using TCP sockets for communication. MTU can have up to 6800 bytes, unfortunately, both in STA and AP mode, the connection is interrupted and the following message is displayed in the terminal:

[00:25:28.479,705] <err> net_pkt: Data buffer (1133) allocation failed.
[00:25:28.486,968] <err> net_tcp: conn: 0x200624d8 packet allocation failed, len=1085
[00:25:28.495,452] <err> net_tcp: TCP failed to allocate buffer in retransmission
[00:25:28.604,309] <err> net_pkt: Data buffer (1413) allocation failed.
[00:25:28.611,572] <err> net_tcp: conn: 0x20062738 packet allocation failed, len=1365
[00:25:28.620,056] <err> net_tcp: TCP failed to allocate buffer in retransmission
[00:25:28.934,387] <err> net_pkt: Data buffer (1413) allocation failed.
[00:25:28.941,650] <err> net_tcp: conn: 0x20062738 packet allocation failed, len=1365
[00:25:28.950,134] <err> net_tcp: TCP failed to allocate buffer in retransmission
[00:25:29.072,448] <err> net_pkt: Data buffer (1260) allocation failed.
[00:25:29.079,742] <err> net_tcp: conn: 0x200624d8 packet allocation failed, len=1212
[00:25:29.088,226] <err> net_tcp: TCP failed to allocate buffer in retransmission

Sometimes the connection is restored, sometimes it drops completely. Please advise what to do about this. I am attaching prj.conf.

Thank you.

#
# Copyright (c) 2022 Nordic Semiconductor ASA
#
# SPDX-License-Identifier: LicenseRef-Nordic-5-Clause
#

# General
CONFIG_LOG=y
CONFIG_ASSERT=y
CONFIG_MAIN_STACK_SIZE=2048
CONFIG_TEST_RANDOM_GENERATOR=y
CONFIG_RESET_ON_FATAL_ERROR=n

CONFIG_CONSOLE=y
CONFIG_UART_CONSOLE=y
CONFIG_SERIAL=y


CONFIG_UART_ASYNC_API=y
CONFIG_UART_INTERRUPT_DRIVEN=y
CONFIG_UART_1_ASYNC=y
CONFIG_UART_1_INTERRUPT_DRIVEN=n
CONFIG_UART_1_NRF_HW_ASYNC=y
CONFIG_UART_1_NRF_HW_ASYNC_TIMER=2
CONFIG_UART_1_NRF_ASYNC_LOW_POWER=y
CONFIG_NRFX_TIMER2=y

# Logging configuration
CONFIG_LOG_DEFAULT_LEVEL=3
CONFIG_LOG_MODE_IMMEDIATE=y

CONFIG_LOG_BUFFER_SIZE=2048
# ==========================================

CONFIG_ZVFS_OPEN_MAX=32


# WiFi Configuration - STA MODE
CONFIG_WIFI=y
CONFIG_WIFI_NRF70=y

# WPA supplicant pro STA mode
CONFIG_WIFI_NM_WPA_SUPPLICANT=y
CONFIG_WIFI_READY_LIB=y

# Logging pro WiFi
CONFIG_WIFI_NM_WPA_SUPPLICANT_LOG_LEVEL_INF=y

# Networking
CONFIG_NETWORKING=y
CONFIG_NET_CONNECTION_MANAGER=y
CONFIG_NET_SOCKETS=y
CONFIG_NET_LOG=y
CONFIG_NET_IPV4=y
CONFIG_NET_UDP=y
CONFIG_NET_TCP=y

CONFIG_POSIX_API=y

# Network buffer sizes
CONFIG_NET_PKT_RX_COUNT=98
CONFIG_NET_PKT_TX_COUNT=74
CONFIG_NET_BUF_RX_COUNT=130
CONFIG_NET_BUF_TX_COUNT=100


# Below section is the primary contributor to SRAM and is currently
# tuned for performance, but this will be revisited in the future.
CONFIG_NRF70_RX_NUM_BUFS=20

CONFIG_NRF_WIFI_CTRL_HEAP_SIZE=30000
CONFIG_NRF_WIFI_DATA_HEAP_SIZE=50000
CONFIG_NET_TC_TX_COUNT=1

CONFIG_NET_IF_UNICAST_IPV4_ADDR_COUNT=1

CONFIG_NET_MAX_CONN=10
CONFIG_NET_MAX_CONTEXTS=64
CONFIG_NET_CONTEXT_SYNC_RECV=y

CONFIG_INIT_STACKS=y

CONFIG_NET_L2_ETHERNET=y

# Network configuration pro STA (DHCP klient místo statické IP)
CONFIG_NET_CONFIG_SETTINGS=y
CONFIG_NET_CONFIG_INIT_TIMEOUT=0
CONFIG_NET_DHCPV4=y

CONFIG_NET_SOCKETS_POLL_MAX=10

# Memories
CONFIG_MAIN_STACK_SIZE=4096
CONFIG_NET_TX_STACK_SIZE=8192
CONFIG_NET_RX_STACK_SIZE=8192
CONFIG_MBEDTLS_HEAP_SIZE=512

# Debugging
CONFIG_STACK_SENTINEL=y
CONFIG_DEBUG_COREDUMP=y
CONFIG_DEBUG_COREDUMP_BACKEND_LOGGING=y
CONFIG_DEBUG_COREDUMP_MEMORY_DUMP_MIN=y
CONFIG_SHELL_CMDS_RESIZE=n

# Kernel options
CONFIG_ENTROPY_GENERATOR=y

# Logging
CONFIG_LOG=y
CONFIG_LOG_BUFFER_SIZE=2048
CONFIG_POSIX_CLOCK=y

# printing of scan results puts pressure on queues in new locking
# design in net_mgmt. So, use a higher timeout for a crowded
# environment.
CONFIG_NET_MGMT_EVENT_QUEUE_TIMEOUT=5000

CONFIG_SYSTEM_WORKQUEUE_STACK_SIZE=8192
CONFIG_NET_TCP_WORKQ_STACK_SIZE=8192

[Marte Myrvold]

Hi,

Please try increasing the following in prj.conf:

CONFIG_NET_PKT_TX_COUNT
CONFIG_NET_PKT_RX_COUNT
CONFIG_NET_BUF_TX_COUNT
CONFIG_NET_BUF_RX_COUNT
CONFIG_NET_BUF_DATA_SIZE
CONFIG_HEAP_MEM_POOL_SIZE

Best regards,
Marte

[iotdeveloper]

Hello,

I was trying to increase the parameters values but still I get sometimes the TCP buffer allocation errors. Is there any way to print to console how much space is used/free by network buffers for solution-debugging?

Thanks

[Marte Myrvold]

Hi,

You can use Zephyr shell to get runtime statistics and debug information, which lets you see buffer usage.

Add these in prj.conf to enable this:

CONFIG_NET_BUF_POOL_USAGE=y
CONFIG_NET_DEBUG_NET_PKT_ALLOC=y
CONFIG_NET_STATISTICS=y
CONFIG_NET_SHELL=y

Then you can use the net stats and net mem commands to get statistics about network packets, buffers, and memory usage.

Best regards,
Marte

[iotdeveloper]

Okay, I added this a used the commands, but could you explain me the meaning of the printed output please? Can I somewhere observer the free memory for network buffer value?

I can see that there are many free buffers left, but despite that, I encountered an allocation error there.

uart:~$ net> net mem
 mem
Fragment length 256 bytes
Network buffer pools:
Address         Total   Avail   Name
0x20009178      30      25      RX
0x200091b8      20      15      TX
0x20009520      26      26      RX DATA (rx_bufs)
0x20009560      20      16      TX DATA (tx_bufs)
[00:01:00.855,224] <inf> tcp_server: Buffer status: KMB=0/3 free, MODBUS=2/2 free, Active: KMB=3 MODBUS=0
uart:~$ > net allocs
net allocs
Network memory allocations

memory          Status  Pool    Function alloc -> freed
0x20065d50/1     used      RX   tcp_conn_alloc():2106
0x200658cc/1     used      TX   tcp_conn_alloc():2114
0x20065d94/1     used      RX   tcp_conn_alloc():2106
0x20065888/1     used      TX   tcp_conn_alloc():2114
0x20063a80/1     used   TDATA   ethernet_fill_header():584
0x20065e1c/1     used      RX   tcp_conn_alloc():2106
0x20065844/1     used      TX   tcp_conn_alloc():2114
0x20065e60/1     used      RX   tcp_conn_alloc():2106
0x20065954/1     used      TX   tcp_conn_alloc():2114
0x20065dd8/1     used      RX   tcp_conn_alloc():2106
0x20065800/1     used      TX   tcp_conn_alloc():2114
0x20063a80       free   TDATA   ethernet_fill_header():584 -> ethernet_send():809
0x20065aa8       free      TX   tcp_out_ext():1551 -> ethernet_send():809
0x20065aa8       free      TX   tcp_out_ext():1551 -> ethernet_send():809
0x20065a20       free      TX   tcp_out_ext():1551 -> ethernet_send():809
0x20065910       free      TX   tcp_out_ext():1551 -> ethernet_send():809
0x20065ee8       free      RX   net_pkt_from_nbuf():490 -> zsock_recv_stream_immediate():1299
0x20065a20       free      TX   tcp_out_ext():1551 -> ethernet_send():809
0x20065a20       free      TX   tcp_out_ext():1551 -> ethernet_send():809


uart:~$ > net stats
net stats

Interface 0x200095a0 (WiFi) [1]
===============================
IPv6 recv      0        sent    6       drop    0       forwarded       0
IPv6 ND recv   0        sent    4       drop    0
IPv6 MLD recv  0        sent    2       drop    0
IPv4 recv      4206     sent    7546    drop    84      forwarded       0
IP vhlerr      6        hblener 0       lblener 0
IP fragerr     0        chkerr  0       protoer 6
ICMP recv      0        sent    6       drop    79
ICMP typeer    0        chkerr  0
IGMP recv      1        sent    0       drop    0
UDP recv       2        sent    3       drop    84
UDP chkerr     0
TCP bytes recv 58089    sent    1764688 resent  2170
TCP seg recv   10243    sent    5001    drop    0
TCP seg resent 3        chkerr  0       ackerr  0
TCP seg rsterr 0        rst     0
TCP conn drop  0        connrst 0
TCP pkt drop   0
Bytes received 302020
Bytes sent     2174242
Processing err 17
uart:~$

Thank you

[Marte Myrvold]

Hi,

The net mem command shows the total and available buffers for each pool. The net allocs command helps you track which functions are allocating and freeing buffers. Lastly, the net stats command provides drop and error counters.

iotdeveloper said:

I can see that there are many free buffers left, but despite that, I encountered an allocation error there.

It could be that other related resources (such as data buffers, packet buffers, or heap memory) are exhausted. Have you tried increasing CONFIG_HEAP_MEM_POOL_SIZE?

Can you share what you have configured these configs as?

CONFIG_NET_PKT_TX_COUNT
CONFIG_NET_PKT_RX_COUNT
CONFIG_NET_BUF_TX_COUNT
CONFIG_NET_BUF_RX_COUNT
CONFIG_NET_BUF_DATA_SIZE
CONFIG_HEAP_MEM_POOL_SIZE

Best regards,
Marte

[iotdeveloper]

Hello, yes, I have also tried increasing CONFIG_HEAP_MEM_POOL_SIZE.

I am sending the complete prj. conf here, but I am mainly dealing with a lack of RAM—if I increase some parameters, I can no longer compile the program; as it is, it is at the compilation limit (99.5% RAM used).

#
# Copyright (c) 2022 Nordic Semiconductor ASA
#
# SPDX-License-Identifier: LicenseRef-Nordic-5-Clause
#

# General
CONFIG_LOG=y
CONFIG_ASSERT=y
CONFIG_MAIN_STACK_SIZE=2048
CONFIG_TEST_RANDOM_GENERATOR=y
CONFIG_RESET_ON_FATAL_ERROR=n

# UART Console
CONFIG_CONSOLE=y
CONFIG_UART_CONSOLE=y
CONFIG_SERIAL=y


CONFIG_UART_ASYNC_API=y

# UART1 async
CONFIG_UART_INTERRUPT_DRIVEN=y

CONFIG_UART_1_ASYNC=y
CONFIG_UART_1_INTERRUPT_DRIVEN=n
CONFIG_UART_1_NRF_HW_ASYNC=y
CONFIG_UART_1_NRF_HW_ASYNC_TIMER=2
CONFIG_UART_1_NRF_ASYNC_LOW_POWER=y
CONFIG_NRFX_TIMER2=y

# Logging configuration
CONFIG_LOG_DEFAULT_LEVEL=3
CONFIG_LOG_MODE_IMMEDIATE=y


CONFIG_LOG_BUFFER_SIZE=2048

CONFIG_ZVFS_OPEN_MAX=32


# WiFi Configuration - STA MODE
CONFIG_WIFI=y
CONFIG_WIFI_NRF70=y

# WPA supplicant pro STA mode
CONFIG_WIFI_NM_WPA_SUPPLICANT=y
CONFIG_WIFI_READY_LIB=y

# Logging pro WiFi
CONFIG_WIFI_NM_WPA_SUPPLICANT_LOG_LEVEL_INF=y

# Networking
CONFIG_NETWORKING=y
CONFIG_NET_CONNECTION_MANAGER=y
CONFIG_NET_SOCKETS=y
CONFIG_NET_LOG=y
CONFIG_NET_IPV4=y
CONFIG_NET_UDP=y
CONFIG_NET_TCP=y

CONFIG_POSIX_API=y


CONFIG_NET_PKT_RX_COUNT=72
CONFIG_NET_PKT_TX_COUNT=78
CONFIG_NET_BUF_RX_COUNT=48
CONFIG_NET_BUF_TX_COUNT=38

CONFIG_NET_BUF_FIXED_DATA_SIZE=y
CONFIG_NET_BUF_DATA_SIZE=256

# Below section is the primary contributor to SRAM and is currently
# tuned for performance, but this will be revisited in the future.
CONFIG_NRF70_RX_NUM_BUFS=30

CONFIG_HEAP_MEM_POOL_SIZE=34000
CONFIG_HEAP_MEM_POOL_IGNORE_MIN=y
CONFIG_NRF_WIFI_CTRL_HEAP_SIZE=38000
CONFIG_NRF_WIFI_DATA_HEAP_SIZE=85000
CONFIG_NET_TC_TX_COUNT=1

CONFIG_NET_IF_UNICAST_IPV4_ADDR_COUNT=1

CONFIG_NET_MAX_CONN=10
CONFIG_NET_MAX_CONTEXTS=64
CONFIG_NET_CONTEXT_SYNC_RECV=y

CONFIG_INIT_STACKS=y

CONFIG_NET_L2_ETHERNET=y

# Network configuration
CONFIG_NET_CONFIG_SETTINGS=y
CONFIG_NET_CONFIG_INIT_TIMEOUT=0
CONFIG_NET_DHCPV4=y

CONFIG_NET_SOCKETS_POLL_MAX=10

# Memories
CONFIG_MAIN_STACK_SIZE=4096
CONFIG_NET_TX_STACK_SIZE=4096
CONFIG_NET_RX_STACK_SIZE=4096
CONFIG_MBEDTLS_HEAP_SIZE=512

# Debugging
CONFIG_STACK_SENTINEL=y
CONFIG_DEBUG_COREDUMP=y
CONFIG_DEBUG_COREDUMP_BACKEND_LOGGING=y
CONFIG_DEBUG_COREDUMP_MEMORY_DUMP_MIN=y
CONFIG_SHELL_CMDS_RESIZE=n

# Kernel options
CONFIG_ENTROPY_GENERATOR=y

# Logging
CONFIG_LOG=y
CONFIG_LOG_BUFFER_SIZE=2048
CONFIG_POSIX_CLOCK=y

# printing of scan results puts pressure on queues in new locking
# design in net_mgmt. So, use a higher timeout for a crowded
# environment.
CONFIG_NET_MGMT_EVENT_QUEUE_TIMEOUT=5000


CONFIG_SYSTEM_WORKQUEUE_STACK_SIZE=6144
CONFIG_NET_TCP_WORKQ_STACK_SIZE=6144


#Memory stats
CONFIG_SYS_MEM_BLOCKS=y
CONFIG_THREAD_STACK_INFO=y
CONFIG_THREAD_RUNTIME_STATS=y
CONFIG_SYS_HEAP_RUNTIME_STATS=y


CONFIG_NET_BUF_POOL_USAGE=y
CONFIG_NET_DEBUG_NET_PKT_ALLOC=y
CONFIG_NET_STATISTICS=y
CONFIG_NET_SHELL=y

CONFIG_NET_STATISTICS_IPV4=y
CONFIG_NET_STATISTICS_TCP=y
CONFIG_NET_STATISTICS_WIFI=y

Thank you

[Marte Myrvold]

Hi,

34000 can be too low for CONFIG_HEAP_MEM_POOL_SIZE. I recommend trying to increase this and see if that helps. 
If you have problems with RAM usage, I recommend looking at Memory footprint optimization and Wi-Fi stack configuration and performance to see how you can reduce your application's memory usage.

Best regards,
Marte