packet_buffer_reserve() causes fault during long term test

[choehyunho]

Hi teams,

I am developing with Mesh SDK 4.0.0, and suffers from random fault during long-term test.

The last readable callstack position is the below.

NRF_MESH_ASSERT(m_get_packet(p_buffer, p_buffer->head)->packet_state !=
                    PACKET_BUFFER_MEM_STATE_RESERVED);

The callstack is QDEC_IRQHandler() -> bearer_event_handler() -> flag_event_cb() -> fire_timers() -> iv_update_timer_handler() -> iv_update_timeout_counter_store() -> mesh_config_entry_set() -> entry_store() -> dirty_entries_process() -> mesh_config_backend_store() -> mesh_config_backend_record_write() -> flash_manager_entry_alloc() -> reserve_action_buffer() -> packet_buffer_reserve()

This device is heavily loaded to send some data gathered from mesh network to local ethernet.

I am not sure exactly what kind of exception happens because I am not so familiar to SEGGER embedded studio.

Can someone explain what is the above error, and give some advice how to debug it?

[Hung Bui]

Hi Choe, 

Could you tell how long in to the test when that happened ? 
I suspect that it has something to do with the IV index update. 
How often do you send data from a node (the one send the most) ? Could you give some more information about your test setup ? 

[choehyunho]

I have 1 element & 6 models in it.

3 models are servers and the other 3 models are clients. and 2 out of 3 server models are publishing every 10 minutes, and 1 model publishes every 10 seconds.

3 client models are receiving each server messages including messages from the own node, and then send them to the ethernet with some specific format.

Can these information be helpful to figure out the problem?

BTW, I modified logging and send them to UART instead of RTT.

When I send logging back to RTT, the frequency of this error is much lower or seems to be disappeared.

I am trying to reproduce this error in RTT logging mode, but until now it doesn't occur.

The UART function may impact on the performance or integrity of Mesh?

[Hung Bui]

Could you clarify that you only has one 1 node in this mesh network ? And the 3 servers are sending the message to the 3 client internally on the same node ? 
If it's the case what's exactly the purpose of doing so ? 
What did you printout on the UART log ? 

[choehyunho]

No, I have several nodes.

Multiple nodes roles only as servers. These nodes are sending periodic publish messages to all nodes.

And there is one or two nodes which roles both server and client. This node is gathering some data from itself and others and send them via ethernet.

The server-only nodes works fine with UART logging, but control node(server & client) node has a quite a heavy load, and the symptom happens in this node.

And for the UART log, I added some code to mesh/core/log.c and send all messages to UART.

[Hung Bui]

Hi Choe, 

Thanks for the information. I have reported this to our team. I will let you know if we have any update.