nrf7002dk: raw tx performance

[rulle]

Hi,

I am trying to optimise for raw TX throughput. I am running the raw_tx_packet sample. I am using a nrf7002-dk and running v3.1.99 with the following configuration changes to the prj.conf:

# APPLICATION
CONFIG_RAW_TX_PKT_SAMPLE_INTER_FRAME_DELAY_MS=1
CONFIG_RAW_TX_PKT_SAMPLE_TX_MODE_FIXED=y
CONFIG_RAW_TX_PKT_SAMPLE_FIXED_NUM_PACKETS=10000
CONFIG_RAW_TX_PKT_SAMPLE_QUEUE_NUM=3

# RATE
CONFIG_RAW_TX_PKT_SAMPLE_RATE_VALUE=5
CONFIG_RAW_TX_PKT_SAMPLE_RATE_FLAGS=2

# DRIVER
CONFIG_NRF70_MAX_TX_AGGREGATION=1

# CHANNEL
CONFIG_RAW_TX_PKT_SAMPLE_CHANNEL=52

I use the "overlay-tx-prio.conf' from your documentation to maximize tx performance

# CONFIG_NRF70_AP_MODE=n
# CONFIG_NRF70_P2P_MODE=n
CONFIG_NET_PKT_TX_COUNT=32
CONFIG_NET_PKT_RX_COUNT=10
CONFIG_NET_BUF_TX_COUNT=64
CONFIG_NET_BUF_RX_COUNT=10
CONFIG_NRF70_RX_NUM_BUFS=10
CONFIG_NRF_WIFI_CTRL_HEAP_SIZE=20000
CONFIG_NRF_WIFI_DATA_HEAP_SIZE=199856
CONFIG_SPEED_OPTIMIZATIONS=y
CONFIG_NRF70_QSPI_LOW_POWER=n
CONFIG_NRF70_UTIL=n
CONFIG_NRF70_MAX_TX_AGGREGATION=9
CONFIG_NRF70_MAX_TX_TOKENS=12
CONFIG_ZVFS_OPEN_MAX=20
CONFIG_NET_SOCKETS_POLL_MAX=20
# Sockets: select + poll take up more stack
CONFIG_MAIN_STACK_SIZE=5200
CONFIG_SHELL_STACK_SIZE=5200
CONFIG_NET_MGMT_EVENT_STACK_SIZE=4600

CONFIG_NET_PKT_BUF_TX_DATA_POOL_SIZE=25000
CONFIG_NET_PKT_BUF_RX_DATA_POOL_SIZE=15000

I am interested in sending slitghtly larger packets than the sample so I set

#define BEACON_PAYLOAD_LENGTH 512

I have also made some small differences to the sample code to improve performance like setting the nrf5340 in 128MHz mode by

nrfx_clock_divider_set(NRF_CLOCK_DOMAIN_HFCLK, NRF_CLOCK_HFCLK_DIV_1);

and also using an aligned buffer for the "test_frame". Also, to be able to get more than 1000 packets out per second I have changed

k_msleep(CONFIG_RAW_TX_PKT_SAMPLE_INTER_FRAME_DELAY_MS);

to

k_usleep(various_different_values)

I am still not able to get more than 1000 packets/second out. This corresponds roughly to 600*8*1000 ~  4.8Mbits/second. I am testing on a basically empty channel so there should not be too much contention or interference, and I am also using the voice class for my traffic so CW should not be large. The theoretical max throughput on MCS 5 is roughly 50 Mbits/second. I don't need that much but was hoping to get above 10Mbits/second. Is this a limitation in the nrf5340 or is there any else I can do increase the throughput?

Best regards,
Erik

[Håkon Alseth]

Hi Erik,

 

Could you try to yield instead, and move the k_msleep() function if sendto() returns < 0?

Proposal:

		for (int i = 0; i < num_pkts; i++) {
			memcpy(test_frame + sizeof(struct raw_tx_pkt_header),
			       &test_beacon_frame, sizeof(test_beacon_frame));
			ret = sendto(sockfd, test_frame, buf_length, 0,
					(struct sockaddr *)&sa, sizeof(sa));
			if (ret < 0) {
				LOG_ERR("Unable to send beacon frame: %s", strerror(errno));
				num_failures++;
				k_msleep(1);
			}
			
			increment_seq_control();
			k_yield();
		}

 

I am seeing approx. 700 ms per 1000 packets with this loop, 512 bytes, and your proposed configs. If you tune the payload size towards the MTU size, you will likely get even better throughput.

 

Kind regards,

Håkon

[rulle]

Hi,

Thanks for the reply.

The loop goes faster that way but I don't think I get all the packets out on the air. On RX side in one experiment i received about 6700 packets. It could be that my receiver is not keeping up but I also see error messages like:

[00:00:05.000,762] .<wrn> net_if: Skipped 2242 messages.[

[00:00:05.000,793]  <wrn> net_if: iface 1 pkt 0x2003f3f8 send failure status -105.[0m

There are no socket errors though. Is there any way to know for sure that a packet has been sent on the air by the WiFi chip?
 

// Best regards,
Erik

[Håkon Alseth]

Hi,

 

You are receiving ENOBUFS back:

https://github.com/nrfconnect/sdk-zephyr/blob/main/lib/libc/minimal/include/errno.h#L89

 

I am seeing this output (note: BEACON_PAYLOAD_LENGTH 1280, ie. near the MTU size of 1500):

*** Booting nRF Connect SDK v3.1.0-6c6e5b32496e ***
*** Using Zephyr OS v4.1.99-1612683d4010 ***
[00:00:00.469,665] <inf> net_config: Initializing network
[00:00:00.469,696] <inf> net_config: Waiting interface 1 (0x20001548) to be up...
[00:00:00.469,879] <inf> net_config: IPv4 address: 192.168.1.99
[00:00:00.469,909] <inf> net_config: Running dhcpv4 client...
[00:00:00.471,008] <inf> wifi_supplicant: wpa_supplicant initialized
[00:00:00.473,541] <inf> raw_tx_packet: TX Injection mode enabled
[00:00:00.477,050] <inf> raw_tx_packet: Wi-Fi channel set to 52
[00:00:00.477,142] <inf> raw_tx_packet: Sending 10000 number of raw tx packets
[00:00:05.477,355] <wrn> net_if: Skipped 7 messages
[00:00:05.477,355] <wrn> net_if: iface 1 pkt 0x20058618 send failure status -1
[00:00:08.891,082] <inf> raw_tx_packet: Sent 10000 packets with 0 failures on socket

I've tried different sizes, but just see a difference in timing, rather than the send failure that you're getting. And then I realize that I am not testing exactly the same as you, which I am sorry for not noticing until now!

 

With ncs main, I do get the same issue as you:

*** Booting nRF Connect SDK v3.1.99-e30d0f86f581 ***
*** Using Zephyr OS v4.1.99-1d73d0e3cb5f ***
[00:00:00.472,564] <inf> net_config: Initializing network
[00:00:00.472,595] <inf> net_config: Waiting interface 1 (0x20001148) to be up...
[00:00:00.472,778] <inf> net_config: IPv4 address: 192.168.1.99
[00:00:00.472,808] <inf> net_config: Running dhcpv4 client...
[00:00:00.473,907] <inf> wifi_supplicant: wpa_supplicant initialized
[00:00:00.476,470] <inf> raw_tx_packet: TX Injection mode enabled
[00:00:00.479,980] <inf> raw_tx_packet: Wi-Fi channel set to 52
[00:00:00.480,072] <inf> raw_tx_packet: Sending 10000 number of raw tx packets
[00:00:05.002,288] <wrn> net_if: Skipped 1236 messages
[00:00:05.002,288] <wrn> net_if: iface 1 pkt 0x20058238 send failure status -105
[00:00:07.223,571] <inf> raw_tx_packet: Sent 10000 packets with 0 failures on socket
[00:00:11.480,560] <wrn> net_if: Skipped 694 messages
[00:00:11.480,590] <wrn> net_if: iface 1 pkt 0x20058238 send failure status -1

 

Isolating this, to testing on ncs v3.1.0, with this fix cherry-picked:

https://github.com/zephyrproject-rtos/nrf_wifi/pull/73

 

Shows the exact same output as on main. I will check internally related to this behavior and get back to you within a few business days.

 

Kind regards,

Håkon

[Håkon Alseth]

Hi,

 

Håkon Alseth said:

I will check internally related to this behavior and get back to you within a few business days.

Related to the "skipped 'n' messages" print, this is the explanation when discussing with the wifi team:

All Wi-Fi samples use CONFIG_NET_TC_TX_COUNT=1, so, the connection between socket and driver is decoupled at TC layer in networking stack, and any errors down the TC layer are not propagated to the socket like in this case (nRF70 driver is dropping due to heavy pumping rate and pending Q is full in the driver).

This is a bug from our side, meaning that it is expected that the POSIX call shall return the appropriate ret / errno.

 

Wrt. the performance; there seems to be a side effect with the out-of-order bugfix, meaning that this impacts the throughput negatively.

I will report both these issues internally.

 

Kind regards,

Håkon

[rulle]

Thanks. If you make any changes to main I can test them. Is it OK to leave the ticket open until then?


Best regards,
Erik