Getting CONFIG_HEAP_MEM_POOL_SIZE under control (minimising)

[liteyear]

After discovering that MBEDTLS requires at least 48000 bytes of heap to function, I've had to desperately find some spare bytes to conduct a TLS session over WiFi using a nRF5340 (with nRF7002 for WiFi).

The biggest single allocation is from CONFIG_HEAP_MEM_POOL_SIZE. By default it is 153000. Why? I can't figure out where this number comes from, or how to figure that out.

Note I'm using ncs 2.5.2, so that's before all the HEAP_MEM_POOL_ADD_SIZE_* stuff. All I can find is defaults of about 4096 for Bluetooth, and dramatically, 98304 for WiFi on some non-Nordic boards. So can't see why I'd need 153000, yet if I reduce it to 100000 it fails.

Is there any opportunity to shave some bytes off, or is the figure quite accurate?

[Amanda Hsieh]

Hi,

The biggest single allocation is from CONFIG_HEAP_MEM_POOL_SIZE. By default it is 153000. Why? I can't figure out where this number comes from, or how to figure that out.

The `CONFIG_HEAP_MEM_POOL_SIZE` configuration option specifies the size of the heap memory pool. The default value of 153000 bytes you mentioned is likely set to accommodate your application's various subsystems and modules that require heap memory.

The heap memory pool is used by different parts of the system, including drivers, libraries, and other subsystems. Each of these can have different heap memory requirements, and the total heap memory pool size needs to be large enough to accommodate all of them.

Reducing the `CONFIG_HEAP_MEM_POOL_SIZE` to 100000 bytes might cause the system to run out of heap memory, leading to failures. This is because the total memory required by all the subsystems and modules in your application exceeds this reduced heap memory pool size.

Finding opportunities to shave off some bytes would require a careful analysis of your application's memory usage. You would need to identify which parts of the system are using the heap memory and how much they are using. This could involve profiling your application's memory usage or reviewing the source code of the subsystems and modules to understand their memory requirements. Maybe this post could help. Also, see the Defining the Heap Memory Pool doc. 

Regards,
Amanda H.

[liteyear]

Okay, you can safely assume all of that is all already well known.

What about my questions? Let me make them clearer:

1. Where does this number comes from?
2. How would I figure out where it comes from?
3. Is the figure quite accurate, or is it an overestimate with room to spare?
   

Perhaps it would help to understand what I'm asking to know that I never call k_malloc in my application, and we have no third party libraries other than ncs.

[Amanda Hsieh]

Hi, 

I am contacting the team for the figure and will be back when I collect enough information. 

[Amanda Hsieh]

Hi, 

The 153000 number came from experimental evidence. This has subsequently been optimized in Wi-Fi samples based on configuration and actual traffic requirements of the respective samples.

 The main consumers of the the heap is the Wi-Fi driver and mbedTLS (based on configuration). The Wi-Fi driver usage is divided into:

• Control path: 40960 bytes. This is fixed.

• Data path: Heap usage is determined by the number of buffers (TX and RX) provisioned. 

The overall heap usage can be reduced by reducing the number of buffers at the cost of overall throughput that can be achieved. More details can be found at: nRF Wi-Fi driver performance and memory fine-tuning controls doc.

 Please note that the following section Usage profiles in the same page:

 gives an idea about how various configurations affect the throughputs. However please note the kernel heap size is incorrectly the same for all profiles right now and is due to get fixed in the upcoming releases.

 To understand whether the heap can be reduced in your case,  can we get the following information:

1. What is the sample that is being used?

2. .config​ to understand the configuration

3. Traffic & throughput requirements

-Amanda H.

[liteyear]

Thank you for the relevant info. It still doesn't check out though - neither the mbedTLS heap nor the net buffers are part of the mempool. See, for example:



Here the mempool is the fixed 153000 bytes we've been talking about.



And here's where the net buffers live. I've been able to make huge (10's of kB) reductions here by adjusting the CONFIG_NET_PKT_* and CONFIG_NET_BUF_* settings.

Likewise, mbed_tls heap is over on the left, and I've confirmed that adjusting this with CONFIG_MBEDTLS_HEAP_SIZE directs affects it and the ability of the mbedtls driver to allocate dynamic memory.

It's plausible, I suppose, that the nrf700x driver uses the mempool for dynamic allocation, in addition to the static allocations nrf/modules/hostap and nrf/drivers/wifi/nrf700x that are over on the left of the memory report diagram. But is there anyway to get visibility of this? If not, then unfortunately it's useless information at the moment.

I can't find any settings, on that page or elsewhere, that make a difference to the mempool. But to be fair I'm rather sick of this wild goose chase, so many have given up too early.

Yes I've already discovered the profiles don't make a difference, thanks for the confirmation.

Thanks to your guidance, I text searched the entire ncs directory and found the source of "153000"! Sure enough, it's the default set by the nrf700x driver here: nrf/drivers/wifi/nrf700x/Kconfig:348. I still don't know of a better way to discover these things other than text searching the entire tree.

To answer your questions:

1. Sample? This stirs up my growing fear that ncs is only for demos! We're writing a custom application, for commercial purposes. So no sample. But at various times we've drawn from lots of samples, like "Wi-Fi scan", some BLE samples and some OTAA samples. So for example we're using CONFIG_BT_GATT_CLIENT, CONFIG_NET_TCP, CONFIG_NET_CONFIG_SETTINGS, CONFIG_MBEDTLS, CONFIG_WIFI_NRF700X, CONFIG_WIFI_CREDENTIALS_BACKEND_SETTINGS, CONFIG_MQTT_LIB, CONFIG_BOOTLOADER_MCUBOOT, CONFIG_MCUMGR_TRANSPORT_BT, and many others.
2. .config attached. I had to try a few rename options to get the forum to allow it! 3364.dotconfig.txt.
3. At this stage, we can compromise on through put. If we get a single stream of say 10Mb/s, we might be able to get away with it. We don't have much need for parallel connections, and the traffic is intermittent.

Thanks for your on-going help.

[Amanda Hsieh]

Hi,

liteyear said:

It still doesn't check out though - neither the mbedTLS heap nor the net buffers are part of the mempool.

You are right. MbedTLS either uses a dedicated heap or uses libc heap, the reference to kernel heap was incorrect, it was an overlook from our side.

Packet buffers occupy memory at both networking stack and at driver level, the comment refers to packet buffers at driver level that contribute to CONFIG_HEAP_MEM_POOL_SIZE, the packet buffers at networking stack are separate and contribute to static RAM.

 So, if networking stack buffers and buffer sizes have been reduced, then corresponding driver buffers and buffer sizes can also be reduced.

 As per the .config​ shared, we could recommend

 # To avoid packet loss, should match incoming rate rather than CONFIG_NET_PKT_RX_COUNT=4

CONFIG_NRF700X_RX_NUM_BUFS=16

# As CONFIG_NET_PKT_TX_COUNT=4, we don't need aggregation,

 CONFIG_NRF700X_MAX_TX_AGGREGATION=1

# This is tricky, even though CONFIG_NET_BUF_DATA_SIZE=128 if an APP sends larger packet it can span
# multiple buffers, but  the app payload should fit in below settings + 52 bytes overhead, assuming 1024 bytes max app payload

 CONFIG_NRF700X_TX_MAX_DATA_SIZE=1100

 CONFG_NRF700X_RX_MAX_DATA_SIZE=1100

 e.g., samples: wifi: sta: Reduce kernel heap by krish2718 · Pull Request #16250 · nrfconnect/sdk-nrf reduces from 153600->90000.

liteyear said:

Sample? This stirs up my growing fear that ncs is only for demos! We're writing a custom application, for commercial purposes. So no sample.

The intention behind this question was to understand the use case and if possible replicate build at our end.

-Amanda H.

[liteyear]

Thank Amanda, I appreciate your patience and willingness to get this right.

That PR from last week is particularly interesting!

We've had to move on - we don't want to be the first team to publicly shine a light on these details. As embedded developers, resource management is important to us, so we're going to find another way in the meantime and may return to this in the future. The details you've revealed will certainly help those future efforts.

Kind regards.