net_arp: Memory fault in in arp_entry_cleanup

RE: net_arp: Memory fault in in arp_entry_cleanup

Elfving — Thu, 30 Nov 2023 12:27:04 GMT

Hello again Daniel, and sorry about the delay here.

I had a talk with a developer working on this who agree that this doesn't look right. It seems that we do ARP cleanup in case of TX error, but not in other cases (like out-of-buf issue on header allocation). This would cause dereferencing the packet on the ARP queue. We'll try to reproduce and provide a fix upstream.

Quick update:
This PR should fix the double packet dereference and ARP packet leak in case of out-of-mem issues. And another thing is that the minimum amount of net buffers configured for sending a UDP are 3. I've looked into why we don't enforce a minimum of something like 3, but what the minimum is differs a lot depending on your configuration.

Regards,

Elfving

RE: net_arp: Memory fault in in arp_entry_cleanup

Daniel K — Mon, 20 Nov 2023 19:35:44 GMT

I had noticed that net_pkt_clone() was used in a couple of places, but possibly unchecked...
In zephyr/subsys/net/ip/tp.c:

struct net_pkt *tp_pkt_clone(struct net_pkt *pkt, const char *file, int line)
{
	struct tp_pkt *tp_pkt = k_malloc(sizeof(struct tp_pkt));

	pkt = net_pkt_clone(pkt, K_NO_WAIT);

	tp_pkt->pkt = pkt;
	tp_pkt->file = file;
	tp_pkt->line = line;

In zephyr/subsys/net/ip/net_if.c:

		verdict = net_if_l2(iface)->recv(iface, pkt);
		if (verdict == NET_CONTINUE) {
			new_pkt = net_pkt_clone(pkt, K_NO_WAIT);
		} else {
			new_pkt = net_pkt_ref(pkt);
		}

		/* L2 has modified the buffer starting point, it is easier
		 * to re-initialize the cursor rather than updating it.
		 */
		net_pkt_cursor_init(new_pkt);

RE: net_arp: Memory fault in in arp_entry_cleanup

Daniel K — Mon, 20 Nov 2023 17:11:41 GMT

Hi sorry for the late reply. Increasing it fixes the immediate problem, but my concern is that there is some case of unchecked allocation failure causing memory corruption. In that case, no matter how large the buffer is, there is always a possibility of memory corruption and crash.

RE: net_arp: Memory fault in in arp_entry_cleanup

Elfving — Wed, 08 Nov 2023 12:14:40 GMT

Hello Daniel,

Yeah, I guess it makes sense that some packets are released when the buffers are that low. I've seen some similar cases to this before.

For TX, the buffers have to twice that of packets (Each Ethernet TX frame occupies two buffers). Ie. for your case:

CONFIG_NET_BUF_RX_COUNT=6
CONFIG_NET_BUF_TX_COUNT=12

But try to increase the buffers even more, eg. 50 and 100. For performance testing you could for instance have a look at overlay-zperf.conf that has optimized settings for this.

I am not exactly seeing why a buffer that low should make it crash, though the default seems to be 16. So 6 is a bit low. Does increasing it fix all the issues for you?

Regards,

Elfving

RE: net_arp: Memory fault in in arp_entry_cleanup

Daniel K — Thu, 02 Nov 2023 18:38:23 GMT

I found that increasing CONFIG_NET_BUF_TX_COUNT from 2 to 8 fixes the issue, although it's concerning it doesn't have any enforced minimum.

I suppose somewhere a packet is being allocated unsuccessfully and not being checked. It doesn't fix the actual root cause of the crash though.

RE: net_arp: Memory fault in in arp_entry_cleanup

Daniel K — Thu, 02 Nov 2023 15:58:42 GMT

I think there has to be a memory corruption issue. For testing purposes, I changed the while loop in arp_entry_cleanup() to skip over entries that are probably corrupt just to see if it would still transmit:

		while (!k_fifo_is_empty(&entry->pending_queue)) {
			pkt = k_fifo_get(&entry->pending_queue, K_FOREVER);
			atomic_val_t ref = atomic_get(&pkt->atomic_ref);
			if (ref == 1) {
				NET_INFO("Releasing pending pkt %p (ref %ld)", pkt, ref - 1);
				net_pkt_unref(pkt);
			} else {
				NET_INFO("Skip released pending pkt %p (ref %ld)", pkt, ref - 1);
			}
		}

Seems like the packets are getting corrupt, as the ref counts make no sense:

[00:00:13.444,824] <inf> app: Sending 1 B to 192.168.0.4:49229
[00:00:15.445,190] <inf> net_arp: Skip released pending pkt 0x2003aaa4 (ref -1)
[00:00:15.445,220] <inf> net_arp: Skip released pending pkt 0x2003aa64 (ref 1391567922)
[00:00:15.445,251] <inf> net_arp: Skip released pending pkt 0x2003aa24 (ref -794600311)
[00:00:15.445,251] <inf> net_arp: Skip released pending pkt 0x2003a9e4 (ref -1327495242)
[00:00:15.445,281] <inf> net_arp: Skip released pending pkt 0x2003a9a4 (ref -1091356391)
[00:00:15.445,281] <inf> net_arp: Skip released pending pkt 0x2003a964 (ref -2070176309)
[00:00:15.445,312] <inf> net_arp: Skip released pending pkt 0x2003a924 (ref 885680368)
[00:00:15.445,312] <inf> net_arp: Skip released pending pkt 0x2003a8e4 (ref 1893167851)
[00:00:15.445,343] <inf> net_arp: Skip released pending pkt 0x2003a8a4 (ref -1807458317)
[00:00:15.445,373] <inf> net_arp: Skip released pending pkt 0x2003a864 (ref -1093865732)
[00:00:15.445,373] <inf> net_arp: Skip released pending pkt 0x2003a824 (ref -1744161751)
[00:00:15.445,404] <inf> net_arp: Skip released pending pkt 0x2003a7e4 (ref -187367831)
[00:00:15.445,404] <inf> net_arp: Skip released pending pkt 0x2003a7a4 (ref 175564112)

I think this leads to a memory leak, because eventually it starts failing every time to send a packet.

[00:04:43.773,345] <inf> app: Sending 1 B to 192.168.0.4:49229
[00:04:43.873,718] <err> net_pkt: Data buffer (28) allocation failed.

I would guess that this means the packets must have actually NOT been freed, and were ignored instead of being sent.

A Wireshark capture of ethernet packets only shows an XID broadcast, and the DHCP exchange; no other packets being sent.

RE: net_arp: Memory fault in in arp_entry_cleanup

Daniel K — Thu, 02 Nov 2023 13:45:42 GMT

Oh, one other thing. I had added a log entry in arp_prepare() where the packet is added to the pending_queue:

            NET_DBG("New pkt %p dst %s",
                pending, net_sprint_ipv4_addr(&entry->ip));

RE: net_arp: Memory fault in in arp_entry_cleanup

Elfving — Thu, 02 Nov 2023 12:54:38 GMT

Hello Daniel,

Just letting you know that I am looking into it.

Regards,

Elfving