race in modemlib _req_forward()

our application is occasionally getting stuck in a tcp (tls) send. it's not permanently stuck since things like our cell provider's 'rnetwork reset' gets things moving along.

a little reverse engineering (and waiting for the hang) leads me to _req_forward() from libmodem.a called from nrf_sendto(). this loooks like:

int32_t _req_forward(int32_t arg1, int32_t shared, int32_t* buf)
  *(uint32_t*)(shared + 0xc) = buf;
  *(uint32_t*)buf = 0xbc00c1ee;
  int32_t timeout = -1;
  rpc_client_msg_send(4, arg1, shared);
  int32_t err;
  do {
    err = *(uint32_t*)buf;
    if (err != 0xbc00c1ee)
      return rpc_error_to_nrf_errno(err);
    err = nrf_modem_os_timedwait(2, &timeout);
  } while (err == 0);
  if (err < 0)
    return -err;
  return err;

which calls into the zephyr support code


int32_t nrf_modem_os_timedwait(uint32_t context, int32_t *timeout)
  struct sleeping_thread thread;
...
  if (*timeout < 0)
    *timeout = SYS_FOREVER_MS;
...
  sleeping_thread_init(&thread, context);
  if (!sleeping_thread_add(&thread))
    return 0;
  k_sem_take(&thread.sem, SYS_TIMEOUT_MS(*timeout));
...

where sleeping_init() sets up the semaphore, sleeping_add() adds it to a list (w/ interrupts disabled), and then waits on the semapore. this semaphore is signalled at interrupt time from


void nrf_modem_os_event_notify(uint32_t context)
  atomic_inc(&rpc_event_cnt);
  SYS_SLIST_FOR_EACH_CONTAINER(&sleeping_threads, thread, node)
    /* Wake sleeping thread if context of the thread matches, is 0 or the notify
     * context is 0.
     */
  if ((thread->context == context) || (context == 0) || (thread->context == 0))
    k_sem_give(&thread->sem);

however, there's a window from when _req_forward() calls rpc_client_msg_send() and when nrf_modem_os_timedwait() makes the thread available to the isr to be signalled. hoisting the irq lock from sleeping_add() into nrf_modem_os_timedwait() helps our case but i guess there's still a theorhetical race (w/o fully knowing what the modem is off doing for this rpc and the state of the stream)

Related