No heap space for incoming notifications

[Ron Segal]

We have an application running on an nRF9160 development board (shortly to be ported to a production board), which listens on a serial link for sensor data, which is then sent via udp/dtls, via NB-IoT.

The development board is connected to a serial terminal for diagnostics.

After several messages have been sent there's a warning message printed out on the console:

"W: No heap space for incoming notification: +CSCON: 0"

or

"W: No heap space for incoming notification: +CSCON: 1"

I've tried doubling heap space and also system workqueue stack size in prj.conf 

# Heap and stacks

CONFIG_HEAP_MEM_POOL_SIZE=4096

CONFIG_MAIN_STACK_SIZE=4096

CONFIG_SYSTEM_WORKQUEUE_STACK_SIZE=4096

However this has made no difference.

There is no apparent impact on the application itself, but I would of course prefer to properly handle whatever is causing the warning.

Help with this would be appreciated. Thanks.

[Achim Kraus]

Heap is that part of the RAM, which is used for alloc/free.

Stack is that part, where the function local variables and sometimes the parameters (if not passed in register) are allocated and freed on return of that function.

Though the error message is "No heap space", I would assume, that it requires to use a larger value for "CONFIG_HEAP_MEM_POOL_SIZE".

I checked it with a app I use, there it is "CONFIG_HEAP_MEM_POOL_SIZE=16384".

[Ron Segal]

Thanks Achim, had tried doubling heap space previously in prj.conf, then upped it again from 4096 to 16384. Pity but the warning still appears, again after a few udp/dtls transmissions. Also tried increasing main stack and workqueue stack but again no change. Any other ideas?

[Achim Kraus]

I use NCS 2.0.0, fw 1.3.2 and my app (https://github.com/boaks/zephyr-coaps-client) runs quite long (now for more than 7 week, exchanging a message each hour, so currently about 920 exchanges).

That uses only plain UDP and adds DTLS CID using Eclipse/tinydtls.

So, I see two other causes.

- something in the DTLS stack (I don't think so).

- a memory leak somewhere in the app (malloc's without free).

I also tried to add some heap-diagnose to my app, but until now, not with success, (see Zephyr - Discussions - Access heap statistic)

[Achim Kraus]

One pitfall may be the logging. There are many modes and details. If the logging backend works deferred (CONFIG_LOG_MODE_DEFERRED), then sometimes parts of the passed parameters ("%s") must be copied to the heap. if that happens "fast/excessive", that may cause also some issues. But I'm not sure, if that uses the same heap as malloc or something else. 

[Achim Kraus]

There is also a CONFIG_NRF_MODEM_LIB_HEAP_SIZE, see nrf/lib/nrf_modem_lib/Kconfig.modemlib, read the docu there. Maybe that requires more than the 1k (range 512-4096).

[Ron Segal]

Wow, thanks ever so much for giving this some serious consideration Achim.  A memory leak is of course more than likely, despite no obvious candidates having walked through the code many times. Also surprising in that case that is doesn't eventually lead to a failure.  Or maybe it simply hasn't run enough times yet. Will have a go with Valgrind, see if that throws up anything suspicious. Will check logging. Also, didn't know about the modem heap size parameter, thanks for that, will try increasing it first opportunity then reply again later.