Getting CONFIG_HEAP_MEM_POOL_SIZE under control (minimising)

[liteyear]

After discovering that MBEDTLS requires at least 48000 bytes of heap to function, I've had to desperately find some spare bytes to conduct a TLS session over WiFi using a nRF5340 (with nRF7002 for WiFi).

The biggest single allocation is from CONFIG_HEAP_MEM_POOL_SIZE. By default it is 153000. Why? I can't figure out where this number comes from, or how to figure that out.

Note I'm using ncs 2.5.2, so that's before all the HEAP_MEM_POOL_ADD_SIZE_* stuff. All I can find is defaults of about 4096 for Bluetooth, and dramatically, 98304 for WiFi on some non-Nordic boards. So can't see why I'd need 153000, yet if I reduce it to 100000 it fails.

Is there any opportunity to shave some bytes off, or is the figure quite accurate?

[Amanda Hsieh]

Hi,

The biggest single allocation is from CONFIG_HEAP_MEM_POOL_SIZE. By default it is 153000. Why? I can't figure out where this number comes from, or how to figure that out.

The `CONFIG_HEAP_MEM_POOL_SIZE` configuration option specifies the size of the heap memory pool. The default value of 153000 bytes you mentioned is likely set to accommodate your application's various subsystems and modules that require heap memory.

The heap memory pool is used by different parts of the system, including drivers, libraries, and other subsystems. Each of these can have different heap memory requirements, and the total heap memory pool size needs to be large enough to accommodate all of them.

Reducing the `CONFIG_HEAP_MEM_POOL_SIZE` to 100000 bytes might cause the system to run out of heap memory, leading to failures. This is because the total memory required by all the subsystems and modules in your application exceeds this reduced heap memory pool size.

Finding opportunities to shave off some bytes would require a careful analysis of your application's memory usage. You would need to identify which parts of the system are using the heap memory and how much they are using. This could involve profiling your application's memory usage or reviewing the source code of the subsystems and modules to understand their memory requirements. Maybe this post could help. Also, see the Defining the Heap Memory Pool doc. 

Regards,
Amanda H.

[liteyear]

Okay, you can safely assume all of that is all already well known.

What about my questions? Let me make them clearer:

1. Where does this number comes from?
2. How would I figure out where it comes from?
3. Is the figure quite accurate, or is it an overestimate with room to spare?
   

Perhaps it would help to understand what I'm asking to know that I never call k_malloc in my application, and we have no third party libraries other than ncs.

[Amanda Hsieh]

liteyear said:

• Where does this number comes from?
• How would I figure out where it comes from?
• Is the figure quite accurate, or is it an overestimate with room to spare?

The size of the heap memory pool is specified using the CONFIG_HEAP_MEM_POOL_SIZE configuration option.

By default, the heap memory pool size is zero bytes. This value instructs the kernel not to define the heap memory pool object. The maximum size is limited by the amount of available memory in the system. The project build will fail in the link stage if the size specified can not be supported.

In addition, each subsystem (board, driver, library, etc) can set a custom requirement by defining a Kconfig option with the prefix HEAP_MEM_POOL_ADD_SIZE_ (this value is in bytes). If multiple subsystems specify custom values, the sum of these will be used as the minimum requirement. If the application tries to set a value that’s less than the minimum value, this will be ignored and the minimum value will be used instead.

To force a smaller than minimum value to be used, the application may enable the CONFIG_HEAP_MEM_POOL_IGNORE_MIN option. This can be useful when optimizing the heap size and the minimum requirement can be more accurately determined for a specific application.

see the Defining the Heap Memory Pool doc. 

-Amanda H.

[liteyear]

None of that is relevant to my question. Most of it is simply wrong or doesn't apply. Please assume I've read the documentation, over and over again, before posting.

Can you look into the details in my question? Here they are again:

• I'm using a nRF5340 (with nRF7002 for WiFi).
• I'm using ncs 2.5.2, so that's before all the HEAP_MEM_POOL_ADD_SIZE_* stuff.
• By default CONFIG_HEAP_MEM_POOL_SIZE is 153000. Why?

Quoting the Zephyr documentation doesn't help - something specific is happening in ncs 2.5.2 with the nrf5340, and this is when the documentation often turns out to be misleading.

[Amanda Hsieh]

Hi, 

I am contacting the team for the figure and will be back when I collect enough information. 

[Amanda Hsieh]

Hi, 

The 153000 number came from experimental evidence. This has subsequently been optimized in Wi-Fi samples based on configuration and actual traffic requirements of the respective samples.

 The main consumers of the the heap is the Wi-Fi driver and mbedTLS (based on configuration). The Wi-Fi driver usage is divided into:

• Control path: 40960 bytes. This is fixed.

• Data path: Heap usage is determined by the number of buffers (TX and RX) provisioned. 

The overall heap usage can be reduced by reducing the number of buffers at the cost of overall throughput that can be achieved. More details can be found at: nRF Wi-Fi driver performance and memory fine-tuning controls doc.

 Please note that the following section Usage profiles in the same page:

 gives an idea about how various configurations affect the throughputs. However please note the kernel heap size is incorrectly the same for all profiles right now and is due to get fixed in the upcoming releases.

 To understand whether the heap can be reduced in your case,  can we get the following information:

1. What is the sample that is being used?

2. .config​ to understand the configuration

3. Traffic & throughput requirements

-Amanda H.

[liteyear]

Thank you for the relevant info. It still doesn't check out though - neither the mbedTLS heap nor the net buffers are part of the mempool. See, for example:



Here the mempool is the fixed 153000 bytes we've been talking about.



And here's where the net buffers live. I've been able to make huge (10's of kB) reductions here by adjusting the CONFIG_NET_PKT_* and CONFIG_NET_BUF_* settings.

Likewise, mbed_tls heap is over on the left, and I've confirmed that adjusting this with CONFIG_MBEDTLS_HEAP_SIZE directs affects it and the ability of the mbedtls driver to allocate dynamic memory.

It's plausible, I suppose, that the nrf700x driver uses the mempool for dynamic allocation, in addition to the static allocations nrf/modules/hostap and nrf/drivers/wifi/nrf700x that are over on the left of the memory report diagram. But is there anyway to get visibility of this? If not, then unfortunately it's useless information at the moment.

I can't find any settings, on that page or elsewhere, that make a difference to the mempool. But to be fair I'm rather sick of this wild goose chase, so many have given up too early.

Yes I've already discovered the profiles don't make a difference, thanks for the confirmation.

Thanks to your guidance, I text searched the entire ncs directory and found the source of "153000"! Sure enough, it's the default set by the nrf700x driver here: nrf/drivers/wifi/nrf700x/Kconfig:348. I still don't know of a better way to discover these things other than text searching the entire tree.

To answer your questions:

1. Sample? This stirs up my growing fear that ncs is only for demos! We're writing a custom application, for commercial purposes. So no sample. But at various times we've drawn from lots of samples, like "Wi-Fi scan", some BLE samples and some OTAA samples. So for example we're using CONFIG_BT_GATT_CLIENT, CONFIG_NET_TCP, CONFIG_NET_CONFIG_SETTINGS, CONFIG_MBEDTLS, CONFIG_WIFI_NRF700X, CONFIG_WIFI_CREDENTIALS_BACKEND_SETTINGS, CONFIG_MQTT_LIB, CONFIG_BOOTLOADER_MCUBOOT, CONFIG_MCUMGR_TRANSPORT_BT, and many others.
2. .config attached. I had to try a few rename options to get the forum to allow it! 3364.dotconfig.txt.
3. At this stage, we can compromise on through put. If we get a single stream of say 10Mb/s, we might be able to get away with it. We don't have much need for parallel connections, and the traffic is intermittent.

Thanks for your on-going help.