How to get stats for heap size usage when using nRF Connect SDK with CONFIG_COMMON_LIBC_MALLOC=y

[BrianW]

As recommended by the documentation I am using the malloc/free in the common libc branch.

This code (in zephyr/lib/libc/common/source/stdlib/malloc.c) uses sys_heap_init() with a struct sys_heap that it creates (in my config with CONFIG_COMMON_LIBC_MALLOC_ARENA_SIZE=-1) using a heap_base just after _end, and running to the end of RAM. (around 300K).

I would like to be able to log the heap current usage and high water mark during the run of the firmware.

I have found the posts about enabling CONFIG_SYS_HEAP_RUNTIME_STATS and using code like this:

    uint32_t heap_free = 0;

    uint32_t heap_used = 0;

    uint32_t heap_max_used = 0;

#ifdef CONFIG_SYS_HEAP_RUNTIME_STATS

    extern struct k_heap _system_heap;

    struct sys_memory_stats stats;

    sys_heap_runtime_stats_get(&_system_heap.heap, &stats);

    heap_used = (uint32_t)stats.allocated_bytes;

    heap_max_used = (uint32_t)stats.max_allocated_bytes;

    heap_free = (uint32_t)stats.free_bytes;

#endif

    log_info("System Stats: Heap %d at boot, now : free %u, used %u, max used %u. Malloc says total alloc todate is %u", _boot_heap_size, heap_free, heap_used, heap_max_used, _allocated_total);

(note I calculate _boot_heap_size  myself with the same calculation used by malloc_prepare(), and track _allocated_total in my wrapper round malloc)

The values for stats.allocated_bytes, stats.max_allocated_bytes, stats.free_bytes are not at all as expected eg:

[00:57:03.272,064] <inf> base: System Stats: Heap 251192 at boot, now : free 3380, used 632, max used 912. Malloc says total alloc todate is 165901

Is this because the 'k_heap_system_heap' is not the same heap used by malloc (seems to be Z_LIBC_DATA static struct sys_heap z_malloc_heap; in malloc.c)?

How can i get the 'real' heap used by malloc in this case (without hacking malloc.c to make it not be static...)?

btw, sys_heap_print_info(&_system_heap.heap,true); gives me the same log output...

thanks

[BrianW]

A hacky workaround:

at line 131 of zephyr/lib/libc/common/source/stdlib/malloc.c, remove the 'static' keyword so it reads:

Z_LIBC_DATA  struct sys_heap z_malloc_heap;

then get the stats on this structure:

    extern struct sys_heap z_malloc_heap;

    struct sys_memory_stats stats;

    sys_heap_runtime_stats_get(&z_malloc_heap, &stats);

and you get the stats for the actual heap used by common libc malloc/free.

But I can't see a way to get this structure without modifting the core zephyr code....

[Håkon Alseth]

Hi!

 

I was blocked by the exact same issue as you.

 

You will need to enable user space (CONFIG_USERSPACE=y) to expose the libc heap:

extern struct k_mem_partition z_malloc_partition;

This gives me access to the picolibc internals:

(gdb) print z_malloc_partition
$1 = {
  start = 0x20000000,
  size = 0x2720,
  attr = {
    rbar = 0x3,
    mair_idx = 0x1
  }
}

However, if you do not really need to enable userspace, I would rather recommend that you use the kernel heap management, ie. k_* prefixed functions.

 

Kind regards,

Håkon

[BrianW]

Håkon Alseth said:

However, if you do not really need to enable userspace, I would rather recommend that you use the kernel heap management, ie. k_* prefixed functions.

Ok... you mean not have 

CONFIG_COMMON_LIBC_MALLOC=y

 in prj.conf?

and then call k_malloc/k_free directly in my code instead of malloc/free?

k_malloc uses  extern struct k_heap _system_heap then?

How is the size of this configured (eg to be rest of free RAM)?

[Håkon Alseth]

Hi,

 

You can either statically define your own heap:

https://docs.zephyrproject.org/latest/kernel/memory_management/heap.html#heap-listener

 

Or you can use the system heap, configured via CONFIG_HEAP_MEM_POOL_SIZE, where you can use the k_*alloc / k_free functions directly.

 

Setting up a heap listener can be done using the heap listener API:

https://docs.zephyrproject.org/latest/doxygen/html/group__heap__listener__apis.html

 

There is a test for this function that shows how it is setup:

https://github.com/zephyrproject-rtos/zephyr//blob/v3.7.0/tests/lib/heap/src/main.c#L416

 

Kind regards,

Håkon

[BrianW]

Håkon Alseth said:

Or you can use the system heap, configured via CONFIG_HEAP_MEM_POOL_SIZE, where you can use the k_*alloc / k_free functions directly.

Ok. How would I get other code that uses libc malloc/free to use the same system heap?