No heap space for incoming notifications

[Ron Segal]

We have an application running on an nRF9160 development board (shortly to be ported to a production board), which listens on a serial link for sensor data, which is then sent via udp/dtls, via NB-IoT.

The development board is connected to a serial terminal for diagnostics.

After several messages have been sent there's a warning message printed out on the console:

"W: No heap space for incoming notification: +CSCON: 0"

or

"W: No heap space for incoming notification: +CSCON: 1"

I've tried doubling heap space and also system workqueue stack size in prj.conf 

# Heap and stacks

CONFIG_HEAP_MEM_POOL_SIZE=4096

CONFIG_MAIN_STACK_SIZE=4096

CONFIG_SYSTEM_WORKQUEUE_STACK_SIZE=4096

However this has made no difference.

There is no apparent impact on the application itself, but I would of course prefer to properly handle whatever is causing the warning.

Help with this would be appreciated. Thanks.

[Achim Kraus]

Heap is that part of the RAM, which is used for alloc/free.

Stack is that part, where the function local variables and sometimes the parameters (if not passed in register) are allocated and freed on return of that function.

Though the error message is "No heap space", I would assume, that it requires to use a larger value for "CONFIG_HEAP_MEM_POOL_SIZE".

I checked it with a app I use, there it is "CONFIG_HEAP_MEM_POOL_SIZE=16384".

[Ron Segal]

Thanks Achim, had tried doubling heap space previously in prj.conf, then upped it again from 4096 to 16384. Pity but the warning still appears, again after a few udp/dtls transmissions. Also tried increasing main stack and workqueue stack but again no change. Any other ideas?

[Achim Kraus]

I searched for "No heap space for incoming notification" and found it in "nrf/lib/at_monitor/at_monitor.c". That at_monitor contains a Kconfig and that:

config AT_MONITOR_HEAP_SIZE
    int "Heap size for notifications"
    range 64 2048
    default 256

Interesting, how many different heaps are used.

On a first look, at_monitor copies all received data and put that into a fifo, where a monitor tasks takes the data, calls the handler, and frees the data.

I guess, depending on what you do in the handler callback, that may lead to a shortage in that heap.

[Ron Segal]

Thanks for the detective work Achim, problem solved.

After adding CONFIG_AT_MONITOR_HEAP_SIZE=1024 to prj.conf the warning message no longer appears.  Much appreciated.

[Achim Kraus]

You're welcome.

(It's cool, just spend few moments for a hint, and let you do the time intensive tests :-) .)

[Ron Segal]

Hello again Achim. Pity but I spoke too soon about increasing monitoring heap size being the solution, as this morning there were 3 more of the same warnings waiting. It's pretty clear that these warnings are happening whilst the worker function is blocked on the message queue read indicated above, where it is waiting for messages to be generated as a result of characters arriving over the uart that is linked to an external mcu.  Although increasing the heap size has significantly reduced the frequency of their appearance, it does seem like some sort of memory leak is occurring after all.

Messages are pushed onto the message queue in the uart callback function as follows:

case UART_RX_RDY:
		printk("Received data %d bytes\r\n", evt->data.rx.len);
		rxevt.length = evt->data.rx.len;
		//** Copy data from dma rx buffer to message queue buffer
		memcpy(rxevt.data, &evt->data.rx.buf[evt->data.rx.offset], evt->data.rx.len); // copy received bytes to message queue
		
		printk("rxevt data is: ");
		for (int i = 0; i < rxevt.length; i++) printk("%02X",rxevt.data[i]);
		printk(" length: %lu\r\n",rxevt.length);

		//** push base uart received buffer on message queue
		while (k_msgq_put(&receive_event_msq, &rxevt, K_NO_WAIT) != 0)  {
            //** message queue is full: purge old data & try again 
            k_msgq_purge(&receive_event_msq);
        }

		//** Disabling the uart causes it to be enabled again below
		uart_rx_disable(baseuart);
		
		break;

Nothing else is happening other than waiting for this case statement to be invoked as a result of characters arriving.  So maybe an under the hood issue after all?  

[Ron Segal]

Hi Achim.  

Anyway, after the additional warning messages were received over night, sending characters to the application now generates this slightly different warning:

W: No heap space for incoming notification: +CEREG: 1,"C920","000E9166",9,,,"00000000","00000110"

then on a couple more character sends:

W: No heap space for incoming notification: +CSCON: 0

W: No heap space for incoming notification: +CEREG: 1,"C920","000E9166",9,,,"000

Terminal disconnected itself.

Then again after reconnecting the terminal, printing after every incoming uart message and at intervals thereafter

W: No heap space for incoming notification: +CSCON: 0