SDK code is getting hanged in do-while loop with NRF_ERROR_NO_MEM

[Durgesh Tanuku]

Hi, I am beginner for nRF51822. We are using SDK v10 and S120.

We have an IOT Gateway with it's processor interfaced with nRF51822 radio through UART. We have flashed nRF51822 radio with ble_central application with our changes. The radio on Gateway connects to sensors and reads required data.

If there is a single sensor, the flow is working as expected as only single connection is established. If there are multiple sensors, the processor on GW is establing simultaneous connections (Maximum of 8) by sending ble_gap_connect request to the radio. Then multiple ble_gattc_read requests are being sent to read corresponding data from different sensors. In this scenario we have observed that radio code is getting hanged occassionally and not responding to further gattc_read commands.

I have tried to debug the issue by enabling RTT logs. Found that the code is getting blocked in do-while loop which is calling ser_hal_transport_tx_pkt_alloc().

do
{
err_code = ser_hal_transport_tx_pkt_alloc(&p_tx_buf, (uint16_t *)&tx_buf_len);
}
while (NRF_ERROR_NO_MEM == err_code);

ser_hal_transport_tx_pkt_alloc() was returning NRF_ERROR_NO_MEM as m_tx_state was set to HAL_TRANSP_TX_STATE_TX_TRANSMITTING. I have tried to understing state machine of m_tx_state. But couldn't figure out what was causing m_tx_state to stuck is single state.

What could cause this issue and had anyone faced similar issue ?

[Edvin]

Hello,

It looks like there are no free buffer space in the packet processing. It will probably return this from time to time. Are you sure that it is always stuck there, and that it doesn't eventually return NRF_SUCCESS, and then the next packet is returning NRF_ERROR_NO_MEM? 

It may also be that there simply are too many packets coming in. Can you, just for testing purposes, try to increase the connection interval from your application chip?

Best regards,

Edvin

[Durgesh Tanuku]

Hi Edvin,

Yes, It is always stuck there and not returning forever. 

I have increased the connection interval from 40ms to 128ms. Still no luck. Facing the same issue. 

Regards,

Durgesh

[Edvin]

Ok. Thank you for confirming.

What serial transport platform do you use for your serialization? Are the nRF and other µcontroller connected via SPI, UART or TWI?

If it is UART, is flow control enabled?

Best regards,

Edvin

[Durgesh Tanuku]

Hi Edvin,

Thanks for your suggestion. 

We are using UART and flow control was not enabled. I have enabled Hardware flow control in both application and host chips. But still facing the issue.

[Edvin]

Durgesh Tanuku said:

We are using UART and flow control was not enabled.

 

Ok. That was what I wanted to check, because if for some reason flow control was enabled, and the other chip doesn't allow the nRF to send the message. That doesn't seem to be the case.

I am looking a bit in the serialization project now. m_tx_state is only set to HAL_TRANSP_TX_STATE_TRANSMITTING on line 443 in ser_hal_transport.c.

Are you able to get any information on what message that is the last one that is set?

The two other places that HAL_TRANSP_TX_STATE_TRANSMITTING is used is in ser_hal_transport.c on line 115 and 260. in phy_events_handler(). Can you see if phy_events_handler is called after it is stuck in NRF_ERROR_NO_MEM? And if so, what type of event is it?

Best regards,

Edvin

[Durgesh Tanuku]

Hi Edvin,

In another instance of the similar issue, code was stuck in NRF_ERROR_NO_MEM due to HAL_TRANSP_TX_STATE_TX_ALLOCATED state of the buffer. 

Below is the corresponding GDB stack trace. 

#0 SEGGER_RTT_vprintf (BufferIndex=BufferIndex@entry=0, sFormat=0x26a77 "M : %d\r\n", sFormat@entry=0x2000329a <m_ble_evt_buffer_size> "H",
pParamList=0x20003f1c, pParamList@entry=0x20003f04)
at git_working_directory/nordic_sdk_10/components/drivers_ext/segger_rtt/SEGGER_RTT_printf.c:439
#1 0x00026232 in log_rtt_printf (terminal_index=terminal_index@entry=0, format_msg=0x26a68 "NRF_ERROR_NO_MEM : %d\r\n")
at git_working_directory/nordic_sdk_10/components/libraries/util/nrf_log.c:39
#2 0x00023970 in ser_hal_transport_tx_pkt_alloc (pp_memory=pp_memory@entry=0x20003f38, p_num_of_bytes=p_num_of_bytes@entry=0x20003f3c)
at git_working_directory/nordic_sdk_10/components/serialization/common/transport/ser_hal_transport.c:406
#3 0x000234b0 in ser_conn_ble_event_encoder (p_event_data=p_event_data@entry=0x20002c34 <BLE_EVT_BUFFER.6626>, event_size=<optimized out>)
at git_working_directory/nordic_sdk_10/components/serialization/connectivity/ser_conn_event_encoder.c:41
#4 0x0002360a in ser_conn_ble_event_handle (p_ble_evt=0x20002c34 <BLE_EVT_BUFFER.6626>)
at git_working_directory/nordic_sdk_10/components/serialization/connectivity/ser_conn_handlers.c:165
#5 0x0002568c in intern_softdevice_events_execute ()
at git_working_directory/nordic_sdk_10/components/softdevice/common/softdevice_handler/softdevice_handler.c:138
#6 0x0002578c in intern_softdevice_events_execute ()
at git_working_directory/nordic_sdk_10/components/softdevice/common/softdevice_handler/softdevice_handler.c:77
#7 SWI2_IRQHandler ()
at git_working_directory/nordic_sdk_10/components/softdevice/common/softdevice_handler/softdevice_handler.c:333
#8 <signal handler called>
#9 0x000265b6 in memcpy ()
#10 0x0001f79a in ble_gattc_evt_read_rsp_enc (p_event=0x20002d4c <APP_SCHED_BUF.6623+208>, event_len=<optimized out>, p_buf=<optimized out>,
p_buf_len=0x20003fcc)
at git_working_directory/nordic_sdk_10/components/serialization/connectivity/codecs/s120/serializers/ble_gattc_evt_read_rsp.c:43
#11 0x0001d3cc in ble_event_enc (p_event=p_event@entry=0x20002d4c <APP_SCHED_BUF.6623+208>, event_len=event_len@entry=0,
p_buf=<optimized out>, p_buf_len=p_buf_len@entry=0x20003fcc)
at git_working_directory/nordic_sdk_10/components/serialization/connectivity/codecs/s120/serializers/ble_event.c:126
#12 0x000234d2 in ser_conn_ble_event_encoder (p_event_data=0x20002d4c <APP_SCHED_BUF.6623+208>, event_size=<optimized out>)
at git_working_directory/nordic_sdk_10/components/serialization/connectivity/ser_conn_event_encoder.c:51
#13 0x00024908 in app_sched_execute ()
at git_working_directory/nordic_sdk_10/components/libraries/scheduler/app_scheduler_serconn.c:246
#14 0x00024ea0 in main () at git_working_directory/nordic_sdk_10/examples/ble_central/ble_connectivity/main.c:76

From the stack trace it seems that in frame 12 (in file ser_conn_event_encoder.c) ser_hal_transport_tx_pkt_alloc() was called and m_tx_state was set to HAL_TRANSP_TX_STATE_TX_ALLOCATED. After that, before sending this packet to transmit SWI2_IRQHandler () was invoked in frame 8. In interrupt handler context it is again trying to allocate buffer and got stuck there.

So is there any way to deffer SWI2_IRQHandler () interrupt ?