No heap space for incoming notifications

[Ron Segal]

We have an application running on an nRF9160 development board (shortly to be ported to a production board), which listens on a serial link for sensor data, which is then sent via udp/dtls, via NB-IoT.

The development board is connected to a serial terminal for diagnostics.

After several messages have been sent there's a warning message printed out on the console:

"W: No heap space for incoming notification: +CSCON: 0"

or

"W: No heap space for incoming notification: +CSCON: 1"

I've tried doubling heap space and also system workqueue stack size in prj.conf 

# Heap and stacks

CONFIG_HEAP_MEM_POOL_SIZE=4096

CONFIG_MAIN_STACK_SIZE=4096

CONFIG_SYSTEM_WORKQUEUE_STACK_SIZE=4096

However this has made no difference.

There is no apparent impact on the application itself, but I would of course prefer to properly handle whatever is causing the warning.

Help with this would be appreciated. Thanks.

[Hakon]

Hello,

are you using the system workqueue in your application? It's possible that the system workqueue is running tasks that are blocking the at monitor from running so the at notification fifo won't clear out. Have you checked this?

[Ron Segal]

Thanks for the reply, appreciate your picking this up.

Apart from the work task waiting for messages to arrive in a message queue:

k_msgq_get(&receive_event_msq, &rxevt, K_FOREVER);

There are no other tasks that have been started by me to run on the system work queue.

New to Zephyr, I've been assuming that waiting on a k_msqq_get would automatically yield to allow other tasks on the system work queue to run.  Is that incorrect?

[Achim Kraus]

The system work queue is a job queue. A job is added with k_work_submit as in at_monitor.c,

    k_work_submit(&at_monitor_work);

What is suspicious is the

static void server_transmission_work_fn(struct k_work *work)

in your snippet. Because that's usually a function assigned to such a job. e.g.

static K_WORK_DEFINE(server_transmission_work, server_transmission_work_fn);

and

    k_work_submit(&server_transmission_work);

[Ron Segal]

Hi Achim,

server_transmission_worker_init();  // initialise the udp worker
	baseuart_connect_async(baseuart); // Also creates dma buffer storage
	//## Kick off main transmission worker here
 	k_work_submit(&server_transmission_work);  // uses system workthread

Yes, in main() a system workqueue worker task to transmit/receive data over dtls/NB-IoT is initialised, a uart connection to another mcu is also initialised, then the system workqueue transmission worker task is started. This waits on k_msgq_get for messages that are created on the queue by bytes coming across the uart link by the (simple) uart dma interrupt routine that marshals the bytes into a message pushed onto the message queue. Those messages are then sent via dtls/NB-IoT to a dtls2mqtt gateway with responses being sent back across the uart link.

Am quite open to modifying this design if there is a better approach, or if it is preferable that the work is done in an application workqueue rather than the system one.

[Achim Kraus]

I'm mainly a java developer, so I'm not that used to zephyr.

As far as I understand zephyr and the idea of a job-queue, it's no good practice to wait in such a job.

But you may wait in you main-thread, or you may use an own thread, which then is able to wait. 

[Ron Segal]

Thanks, was aware that the system queue thread isn't to be blocked for any significant length of time.  I'd assumed that a k_msgq_get would implement an automatic yield but maybe this is wrong.  Apart from using a different thread, another solution might be to create an 'automatic yield' by waiting on the message queue with a k_msqq_get with a short timeout period (rather than waiting forever), then call yield, then loop back to k_msqq_get and so on, only exiting the loop when a message is received. Will perhaps try that simple change anyway, see what happens.

[Achim Kraus]

A short timeout in k_msgq_get will be a polling. Maybe working.

There is also some more sophisticated function (e.g. Events).

A thread is not that complicated and changing that job into a thread should not take too long.

Anyway, it's you to decide.

[Ron Segal]

Can't see a downside to polling in this case as nothing else needs to be done in user land and it isn't possible to miss a message.  However, I may try using a different thread at least to accumulate more practical experience with Zephyr. Anyway, thanks, this has been really helpful. Will report back later on results. Cheers Ron.

[Achim Kraus]

That polling causes energy consumption, a wait not.

But, yes, check a short polling interval and we will see, if that helps.