Nrf9160 nrf_modem_library ASYNC send hang.

[optical]

Hello,

i'm interfacing with the nrf_modem_lib using posix socket API with offloading. The send is called separately from a dedicated thread to avoid any hanging of the rest of the application. 
However, i have seen some anomalies during operation of the application. Which is most likely cause by nrf_modem_library actively waiting for buffer flushing.

I wanted to ask what are condition for the Asynchronous send to block other thread than the calling one ?
The issue happened during low quality signal and in connection stage of the modem on NB-IoT network. The blocking it self would not be an issue if it blocked only the calling thread.
How is it possible that the call to the nrf_modem_library hangs the entire application including other threads ?  This may cause to trip the hardware watchdog or other issues. In my case the modem hanged for 10s. How can i prevent such state ?

[14:03:16.292,000] <dbg> dt_coap_client: Id: 41617, T0: 280029, T: 2827, Tries: 5, Ack: 0, pending: 0x200105f0
[14:03:16.292,000] <wrn> dt_coap_client: Retransmittion Next Try: 5654ms
[14:03:16.293,000] <dbg> dt_coap_client: Retransmittion schedule: 4188ms
[14:03:16.293,000] <dbg> dt_socket_manager: Send handler: 1
[14:03:16.293,000] <dbg> dt_socket_manager: Msg fifo: 1
[14:03:16.293,000] <dbg> dt_socket_manager: Sending data: 64 - (Call to send)
[14:03:27.096,000] <dbg> dt_socket_manager: Resend issue detected. 500 (Log after send fails due to network not connected and internal buffers of the modem are full)
[14:03:27.108,000] <inf> dt_app_flight: Processing Location Data: Alt: 345
[14:03:27.108,000] <inf> dt_app_flight: Time: 15.08.2022 12:01:58.841, timestamp: 1660564918841
[14:03:27.145,000] <inf> dt_mgmt: System heap Alloc: 292, Free: 3708

The issue is not trivially reproducible therefore i cannot provide modem traces. Modem version: nrf9160_1.3.1. NCS 2.0.0

[Charlie]

Hi Tomas,

Thanks for reporting this issue. It is highly likely I will request help from our MFW development to investigate this issue further. 

1) Can you update to the latest MFW1.3.2 for your application? We expect you to use the latest MFW for any further tests.

2) You mentioned "the issue happened during low quality signal and in connection stage of the modem on NB-IoT network" and "the issue is not trivially reproducible." Does this mean the issue will mostly not happen even when the LTE signal quality is low?

3) Do you have any chance to see a similar issue on the existing samples? If not, could you provide a minimal sample of your codes so we can review and try to reproduce the issue locally? I can set this case to private to protect your sharing if you think it is OK.

Best regards,

Charlie 

[optical]

1) - I will update to the latest MFW 1.3.2 a try to reproduce the issue again.
2) - I tested the low signal quality by disconnecting external antenna from our board. The device in this state can still connect to network it is just slower.

3) - The application, which we have is a part of large codebase. We will be unable to share the whole sample. The application sends UDP ~200 byte packages every 3s, and once per 5s ~64 byte package. This should be reproducible by slightly modifying the UDP sample for nrf9160.

After the tests with the 1.3.2 MFW we might try to create sample. 
I would like to ask you whether you can find(from MFW development) a situation when the MFW, whether there are some situations where the nrf_modem_lib might hang other threads than the callers.