No heap space for incoming notifications

RE: No heap space for incoming notifications

Ron Segal — Thu, 29 Sep 2022 17:52:05 GMT

Running for more than 12 hours now with no warning messages. Clearly the problem is solved.

Am including code here in case it helps others.

#define TRANSMISSION_STACK_SIZE 1024
#define TRANSMISSION_PRIORITY 5

K_THREAD_STACK_DEFINE(transmission_stack_area, TRANSMISSION_STACK_SIZE); // define memory for application workqueue

struct k_work_q transmission_work_q; // application workqueue

static struct k_work server_transmission_work; // A work Q element - infinite loop that receives, parses and acts on message events

...


k_work_queue_init(&transmission_work_q); // intialise workqueue

// start workqueue
k_work_queue_start(&transmission_work_q, transmission_stack_area,
                   K_THREAD_STACK_SIZEOF(transmission_stack_area), TRANSMISSION_PRIORITY,
                   NULL);

k_work_init(&server_transmission_work, server_transmission_work_fn);  // initialise worker task - points to function that does the work

// k_work_submit(&server_transmission_work);  // uses system workqueue
k_work_submit_to_queue(&transmission_work_q, &server_transmission_work); // worker task uses application workqueue

Initially when the stack size of the workqueue was set to 512 the device panicked. At 1024 it is running perfectly.

Thanks again. Cheer Ron.

RE: No heap space for incoming notifications

Achim Kraus — Thu, 29 Sep 2022 05:44:40 GMT

Great news!

RE: No heap space for incoming notifications

Ron Segal — Thu, 29 Sep 2022 05:10:15 GMT

In the end, it was as you indicated Achim, ridiculously simple, in about 4 lines of code, to create another workqueue thread and assign the work task to that with no other changes. Since doing that a few hours ago the application has been almost continuously running with no warnings. Will see what happens overnight, then all being well close this question .. again! Cheers Ron.

RE: No heap space for incoming notifications

Achim Kraus — Mon, 26 Sep 2022 20:16:09 GMT

That polling causes energy consumption, a wait not.

But, yes, check a short polling interval and we will see, if that helps.

RE: No heap space for incoming notifications

Ron Segal — Mon, 26 Sep 2022 20:14:08 GMT

Can't see a downside to polling in this case as nothing else needs to be done in user land and it isn't possible to miss a message. However, I may try using a different thread at least to accumulate more practical experience with Zephyr. Anyway, thanks, this has been really helpful. Will report back later on results. Cheers Ron.

RE: No heap space for incoming notifications

Achim Kraus — Mon, 26 Sep 2022 20:02:27 GMT

A short timeout in k_msgq_get will be a polling. Maybe working.

There is also some more sophisticated function (e.g. Events).

A thread is not that complicated and changing that job into a thread should not take too long.

Anyway, it's you to decide.

RE: No heap space for incoming notifications

Ron Segal — Mon, 26 Sep 2022 19:49:25 GMT

Thanks, was aware that the system queue thread isn't to be blocked for any significant length of time. I'd assumed that a k_msgq_get would implement an automatic yield but maybe this is wrong. Apart from using a different thread, another solution might be to create an 'automatic yield' by waiting on the message queue with a k_msqq_get with a short timeout period (rather than waiting forever), then call yield, then loop back to k_msqq_get and so on, only exiting the loop when a message is received. Will perhaps try that simple change anyway, see what happens.

RE: No heap space for incoming notifications

Achim Kraus — Mon, 26 Sep 2022 19:33:56 GMT

I'm mainly a java developer, so I'm not that used to zephyr.

As far as I understand zephyr and the idea of a job-queue, it's no good practice to wait in such a job.

But you may wait in you main-thread, or you may use an own thread, which then is able to wait.

RE: No heap space for incoming notifications

Ron Segal — Mon, 26 Sep 2022 19:25:50 GMT

Hi Achim,

server_transmission_worker_init();  // initialise the udp worker
	baseuart_connect_async(baseuart); // Also creates dma buffer storage
	//## Kick off main transmission worker here
 	k_work_submit(&server_transmission_work);  // uses system workthread

Yes, in main() a system workqueue worker task to transmit/receive data over dtls/NB-IoT is initialised, a uart connection to another mcu is also initialised, then the system workqueue transmission worker task is started. This waits on k_msgq_get for messages that are created on the queue by bytes coming across the uart link by the (simple) uart dma interrupt routine that marshals the bytes into a message pushed onto the message queue. Those messages are then sent via dtls/NB-IoT to a dtls2mqtt gateway with responses being sent back across the uart link.

Am quite open to modifying this design if there is a better approach, or if it is preferable that the work is done in an application workqueue rather than the system one.

RE: No heap space for incoming notifications

Achim Kraus — Mon, 26 Sep 2022 18:42:04 GMT

The system work queue is a job queue. A job is added with k_work_submit as in at_monitor.c,

k_work_submit(&at_monitor_work);

What is suspicious is the

static void server_transmission_work_fn(struct k_work *work)

in your snippet. Because that's usually a function assigned to such a job. e.g.

static K_WORK_DEFINE(server_transmission_work, server_transmission_work_fn);

and

k_work_submit(&server_transmission_work);

RE: No heap space for incoming notifications

Ron Segal — Mon, 26 Sep 2022 18:26:19 GMT

Thanks for the reply, appreciate your picking this up.

Apart from the work task waiting for messages to arrive in a message queue:

k_msgq_get(&receive_event_msq, &rxevt, K_FOREVER);

There are no other tasks that have been started by me to run on the system work queue.

New to Zephyr, I've been assuming that waiting on a k_msqq_get would automatically yield to allow other tasks on the system work queue to run. Is that incorrect?

RE: No heap space for incoming notifications

Hakon — Mon, 26 Sep 2022 16:06:03 GMT

Hello,

are you using the system workqueue in your application? It's possible that the system workqueue is running tasks that are blocking the at monitor from running so the at notification fifo won't clear out. Have you checked this?

RE: No heap space for incoming notifications

Ron Segal — Wed, 14 Sep 2022 18:51:33 GMT

Ok, understood Achim. Thanks for your efforts in trying to track the problem down. Will report back if I do eventually discover a root cause.

RE: No heap space for incoming notifications

Achim Kraus — Wed, 14 Sep 2022 11:27:21 GMT

Seems to be complicated and irritating. From my side, I have no idea left.

RE: No heap space for incoming notifications

Ron Segal — Wed, 14 Sep 2022 06:36:53 GMT

Thanks Achim,

Have just made all the printk statements conditional, which should have been done anyway to switch them off for a release version.

Thought it had resolved the issue but another heap space warning just popped up!

Print statement also don't explain the out of heap warning messages generated overnight when there was no input/output character activity, with the worker function blocked waiting for input. Or why increasing the heap significantly did not solve the problem anyway, which tends to indicate that the heap somehow isn't being entirely garbage collected.

The warning is annoying but otherwise appears to be benign, though there could be bad side effects over time.

RE: No heap space for incoming notifications

Achim Kraus — Tue, 13 Sep 2022 14:48:07 GMT

Hi Ron,

my understanding of "at_monitor" is:

- it reads some notification and queues them for later processing (at_monitor_dispatch)

- based on the messages in the queue, it calls the registered callbacks (at_monitor_task)

If these callbacks take too long, then "at_monitor_dispatch" will run out-of-heap.

I don't know, how long your callback takes, I would start to add some time-measurement to see, if there are cases, where it takes longer and then start to analyze them.

From analyzing the source code (if the snippet is that code in the callback), I see all the "printk" (at least 7-9 I would put in #ifdef and disable them). And there is a second queue. If that is also too full, then that pushs back the waits. The comment above "k_msgq_purge" indicates, that it should not be an issue (and dropping messages is really OK), but maybe calling "k_msgq_purge" takes longer as expected.

RE: No heap space for incoming notifications

Ron Segal — Mon, 12 Sep 2022 19:01:25 GMT

Hi Achim.

Anyway, after the additional warning messages were received over night, sending characters to the application now generates this slightly different warning:

W: No heap space for incoming notification: +CEREG: 1,"C920","000E9166",9,,,"00000000","00000110"

then on a couple more character sends:

W: No heap space for incoming notification: +CSCON: 0

W: No heap space for incoming notification: +CEREG: 1,"C920","000E9166",9,,,"000

Terminal disconnected itself.

Then again after reconnecting the terminal, printing after every incoming uart message and at intervals thereafter

W: No heap space for incoming notification: +CSCON: 0

RE: No heap space for incoming notifications

Ron Segal — Mon, 12 Sep 2022 18:47:00 GMT

Hello again Achim. Pity but I spoke too soon about increasing monitoring heap size being the solution, as this morning there were 3 more of the same warnings waiting. It's pretty clear that these warnings are happening whilst the worker function is blocked on the message queue read indicated above, where it is waiting for messages to be generated as a result of characters arriving over the uart that is linked to an external mcu. Although increasing the heap size has significantly reduced the frequency of their appearance, it does seem like some sort of memory leak is occurring after all.

Messages are pushed onto the message queue in the uart callback function as follows:

case UART_RX_RDY:
		printk("Received data %d bytes\r\n", evt->data.rx.len);
		rxevt.length = evt->data.rx.len;
		//** Copy data from dma rx buffer to message queue buffer
		memcpy(rxevt.data, &evt->data.rx.buf[evt->data.rx.offset], evt->data.rx.len); // copy received bytes to message queue
		
		printk("rxevt data is: ");
		for (int i = 0; i < rxevt.length; i++) printk("%02X",rxevt.data[i]);
		printk(" length: %lu\r\n",rxevt.length);

		//** push base uart received buffer on message queue
		while (k_msgq_put(&receive_event_msq, &rxevt, K_NO_WAIT) != 0)  {
            //** message queue is full: purge old data & try again 
            k_msgq_purge(&receive_event_msq);
        }

		//** Disabling the uart causes it to be enabled again below
		uart_rx_disable(baseuart);
		
		break;

Nothing else is happening other than waiting for this case statement to be invoked as a result of characters arriving. So maybe an under the hood issue after all?

RE: No heap space for incoming notifications

Achim Kraus — Mon, 12 Sep 2022 10:13:23 GMT

You're welcome.

(It's cool, just spend few moments for a hint, and let you do the time intensive tests :-) .)

RE: No heap space for incoming notifications

Ron Segal — Mon, 12 Sep 2022 10:11:49 GMT

Thanks for the detective work Achim, problem solved.

After adding CONFIG_AT_MONITOR_HEAP_SIZE=1024 to prj.conf the warning message no longer appears. Much appreciated.

RE: No heap space for incoming notifications

Achim Kraus — Mon, 12 Sep 2022 07:19:16 GMT

I searched for "No heap space for incoming notification" and found it in "nrf/lib/at_monitor/at_monitor.c". That at_monitor contains a Kconfig and that:

config AT_MONITOR_HEAP_SIZE
   int "Heap size for notifications"
   range 64 2048
   default 256

Interesting, how many different heaps are used.

On a first look, at_monitor copies all received data and put that into a fifo, where a monitor tasks takes the data, calls the handler, and frees the data.

I guess, depending on what you do in the handler callback, that may lead to a shortage in that heap.

RE: No heap space for incoming notifications

Ron Segal — Sun, 11 Sep 2022 22:09:57 GMT

(Of course valgrind isn't applicable, was still in Linux dev mode!).

Have tried changing CONFIG_NRF_MODEM_LIB_HEAP_SIZE, so memory allocation in prj.conf is now:

# Heap and stacks

CONFIG_HEAP_MEM_POOL_SIZE=16384

CONFIG_MAIN_STACK_SIZE=4096

CONFIG_SYSTEM_WORKQUEUE_STACK_SIZE=4096

CONFIG_NRF_MODEM_LIB_HEAP_SIZE=2048

No change.

Have tried setting CONFIG_LOG_MODE_DEFERRED to y and n in prj.conf

No change, apart from the warning being in bold with a timestamp when set to y

The application itself is very simple. Just one system workqueue worker function running in a loop that waits for and reads messages from an application mcu from a uart via a short message queue, transmits them via dtls/NB-IoT, then transmits replies via the uart back to the application mcu. There are no mallocs in the application. The message queue is created once in main as is the worker function.

The warning - W: No heap space for incoming notification: +CSCON: 0

Seems to occur during the wait for incoming uart messages, before any data to send arrives from the application mcu via the uart:

k_msgq_get(&receive_event_msq, &rxevt, K_FOREVER);

That queue is only 4 elements max, with each q element event buffer at 48 bytes.

The entire worker function is reproduced below in case you can spot anything suspicious:

static void server_transmission_work_fn(struct k_work *work)
{
	int err;
	struct receive_event rxevt; //** NB - receive_event struct data buffer is currently size 96 in sockets.h
	int received;
	uint8_t recv_buf[DOWNLINK_RCV_BUF_SIZE];

	for(;;) { // infinite loop

		printk("\r\n>>> server_transmission_work_fn started <<<\r\n");

		//** read message to transmit from the message queue
		//** NB - k_forever waits until a message arrives in the queue
		k_msgq_get(&receive_event_msq, &rxevt, K_FOREVER);
		
		printk("Base board receive buffer to send is: %s\r\n", rxevt.data);

		printk("Transmitting UDP/IP payload of %d bytes to the ", rxevt.length + UDP_IP_HEADER_SIZE);
		printk("IP address %s, port number %d\n", CONFIG_UDP_SERVER_ADDRESS_STATIC, CONFIG_UDP_SERVER_PORT);
		printk("Message buffer to send is: ");	
		for (int i = 0; i < rxevt.length; i++) printk("%02X",rxevt.data[i]);
		printk("\r\n");

		/*******************************************************************/

		//** NB - need to reconnect (create socket) for each transmission, disconnect after receive timeout
		//** or on error
		err = server_connect();
		if (err) {
			printk("Not able to connect to UDP server\n");
			goto exitloop;
		}
		else {
			printk("Connected to UDP Server\r\n");
		}

		/******************************************************************/

		// if (endndx > 0) { // endndx is the length of data to send - packet found if endndx > 0
		if (rxevt.length > 0) {
			// err = send(client_fd,&rxevt.data[startndx],endndx, 0);
			err = send(client_fd, rxevt.data, rxevt.length, 0);
			if (err < 0) {
				printk("Failed to transmit UDP packet, %d\n", errno);
				goto exitloop;
			}
			else {
				printk("Packet transmitted\r\n");
			}

			memset(recv_buf,0, sizeof(recv_buf));
			//** Receive udp return message
			received = recv(client_fd, recv_buf, sizeof(recv_buf), MSG_WAITALL); //NB - If MSG_DONTWAIT flag used recv returns err and no message

			if (received <= 0) {
				printk("No udp reply\r\n");
				//** Saved outgoing message should be resent up to n times
				//** By injecting into base uart receive message queue  
				if (sendretrycount < MAXSENDRETRIES) {
					//** push base uart received buffer back into message queue
					while (k_msgq_put(&receive_event_msq, &rxevt, K_NO_WAIT) != 0)  {
						//** message queue is full: purge old data & try again 
						k_msgq_purge(&receive_event_msq);
					}
					sendretrycount += 1; // increment retry count
				}
			}
			else {
				sendretrycount = 0;
				printf("udp reply is: ");
				for (int i = 0; i < received; i++) printk("%02X",recv_buf[i]);
				printk(" length: %d\r\n", received);

				//** Send receive message to base board via dma transmit
				uart_tx(baseuart,recv_buf, received, SYS_FOREVER_MS);	
			}
		}
		else {
			printk("Failed to find valid packet in receive buffer\r\n");
		}

	exitloop:
		// ** Disconnect udp socket - closes client_fd
		server_disconnect( );

	}// end infinite for(;;)

}

Thanks again, Ron.

RE: No heap space for incoming notifications

Ron Segal — Sun, 11 Sep 2022 09:03:08 GMT

Wow, thanks ever so much for giving this some serious consideration Achim. A memory leak is of course more than likely, despite no obvious candidates having walked through the code many times. Also surprising in that case that is doesn't eventually lead to a failure. Or maybe it simply hasn't run enough times yet. Will have a go with Valgrind, see if that throws up anything suspicious. Will check logging. Also, didn't know about the modem heap size parameter, thanks for that, will try increasing it first opportunity then reply again later.

RE: No heap space for incoming notifications

Achim Kraus — Sun, 11 Sep 2022 07:30:57 GMT

There is also a CONFIG_NRF_MODEM_LIB_HEAP_SIZE, see nrf/lib/nrf_modem_lib/Kconfig.modemlib, read the docu there. Maybe that requires more than the 1k (range 512-4096).

RE: No heap space for incoming notifications

Achim Kraus — Sun, 11 Sep 2022 06:53:06 GMT

One pitfall may be the logging. There are many modes and details. If the logging backend works deferred (CONFIG_LOG_MODE_DEFERRED), then sometimes parts of the passed parameters ("%s") must be copied to the heap. if that happens "fast/excessive", that may cause also some issues. But I'm not sure, if that uses the same heap as malloc or something else.