No heap space for incoming notifications

[Ron Segal]

We have an application running on an nRF9160 development board (shortly to be ported to a production board), which listens on a serial link for sensor data, which is then sent via udp/dtls, via NB-IoT.

The development board is connected to a serial terminal for diagnostics.

After several messages have been sent there's a warning message printed out on the console:

"W: No heap space for incoming notification: +CSCON: 0"

or

"W: No heap space for incoming notification: +CSCON: 1"

I've tried doubling heap space and also system workqueue stack size in prj.conf 

# Heap and stacks

CONFIG_HEAP_MEM_POOL_SIZE=4096

CONFIG_MAIN_STACK_SIZE=4096

CONFIG_SYSTEM_WORKQUEUE_STACK_SIZE=4096

However this has made no difference.

There is no apparent impact on the application itself, but I would of course prefer to properly handle whatever is causing the warning.

Help with this would be appreciated. Thanks.

[Achim Kraus]

Heap is that part of the RAM, which is used for alloc/free.

Stack is that part, where the function local variables and sometimes the parameters (if not passed in register) are allocated and freed on return of that function.

Though the error message is "No heap space", I would assume, that it requires to use a larger value for "CONFIG_HEAP_MEM_POOL_SIZE".

I checked it with a app I use, there it is "CONFIG_HEAP_MEM_POOL_SIZE=16384".

[Ron Segal]

Thanks Achim, had tried doubling heap space previously in prj.conf, then upped it again from 4096 to 16384. Pity but the warning still appears, again after a few udp/dtls transmissions. Also tried increasing main stack and workqueue stack but again no change. Any other ideas?

[Achim Kraus]

You're welcome.

(It's cool, just spend few moments for a hint, and let you do the time intensive tests :-) .)

[Ron Segal]

Hello again Achim. Pity but I spoke too soon about increasing monitoring heap size being the solution, as this morning there were 3 more of the same warnings waiting. It's pretty clear that these warnings are happening whilst the worker function is blocked on the message queue read indicated above, where it is waiting for messages to be generated as a result of characters arriving over the uart that is linked to an external mcu.  Although increasing the heap size has significantly reduced the frequency of their appearance, it does seem like some sort of memory leak is occurring after all.

Messages are pushed onto the message queue in the uart callback function as follows:

case UART_RX_RDY:
		printk("Received data %d bytes\r\n", evt->data.rx.len);
		rxevt.length = evt->data.rx.len;
		//** Copy data from dma rx buffer to message queue buffer
		memcpy(rxevt.data, &evt->data.rx.buf[evt->data.rx.offset], evt->data.rx.len); // copy received bytes to message queue
		
		printk("rxevt data is: ");
		for (int i = 0; i < rxevt.length; i++) printk("%02X",rxevt.data[i]);
		printk(" length: %lu\r\n",rxevt.length);

		//** push base uart received buffer on message queue
		while (k_msgq_put(&receive_event_msq, &rxevt, K_NO_WAIT) != 0)  {
            //** message queue is full: purge old data & try again 
            k_msgq_purge(&receive_event_msq);
        }

		//** Disabling the uart causes it to be enabled again below
		uart_rx_disable(baseuart);
		
		break;

Nothing else is happening other than waiting for this case statement to be invoked as a result of characters arriving.  So maybe an under the hood issue after all?  

[Ron Segal]

Hi Achim.  

Anyway, after the additional warning messages were received over night, sending characters to the application now generates this slightly different warning:

W: No heap space for incoming notification: +CEREG: 1,"C920","000E9166",9,,,"00000000","00000110"

then on a couple more character sends:

W: No heap space for incoming notification: +CSCON: 0

W: No heap space for incoming notification: +CEREG: 1,"C920","000E9166",9,,,"000

Terminal disconnected itself.

Then again after reconnecting the terminal, printing after every incoming uart message and at intervals thereafter

W: No heap space for incoming notification: +CSCON: 0

[Achim Kraus]

Hi Ron,

my understanding of "at_monitor" is:

- it reads some notification and queues them for later processing (at_monitor_dispatch)

- based on the messages in the queue, it calls the registered callbacks (at_monitor_task)

If these callbacks take too long, then "at_monitor_dispatch" will run out-of-heap.

I don't know, how long your callback takes, I would start to add some time-measurement to see, if there are cases, where it takes longer and then start to analyze them.

From analyzing the source code (if the snippet is that code in the callback), I see all the "printk" (at least 7-9 I would put in #ifdef and disable them). And there is a second queue. If that is also too full, then that pushs back the waits. The comment above "k_msgq_purge" indicates, that it should not be an issue (and dropping messages is really OK), but maybe calling "k_msgq_purge" takes longer as expected.

[Ron Segal]

Thanks Achim,  

Have just made all the printk statements conditional, which should have been done anyway to switch them off for a release version.

Thought it had resolved the issue but another heap space warning just popped up!

Print statement also don't explain the out of heap warning messages generated overnight when there was no input/output character activity, with the worker function blocked waiting for input.  Or why increasing the heap significantly did not solve the problem anyway, which tends to indicate that the heap somehow isn't being entirely garbage collected. 

The warning is annoying but otherwise appears to be benign, though there could be bad side effects over time.

[Achim Kraus]

Seems to be complicated and irritating. From my side, I have no idea left.

[Ron Segal]

Ok, understood Achim. Thanks for your efforts in trying to track the problem down.  Will report back if I do eventually discover a root cause.