race in modemlib _req_forward()

our application is occasionally getting stuck in a tcp (tls) send. it's not permanently stuck since things like our cell provider's 'rnetwork reset' gets things moving along.

a little reverse engineering (and waiting for the hang) leads me to _req_forward() from libmodem.a called from nrf_sendto(). this loooks like:

int32_t _req_forward(int32_t arg1, int32_t shared, int32_t* buf)
  *(uint32_t*)(shared + 0xc) = buf;
  *(uint32_t*)buf = 0xbc00c1ee;
  int32_t timeout = -1;
  rpc_client_msg_send(4, arg1, shared);
  int32_t err;
  do {
    err = *(uint32_t*)buf;
    if (err != 0xbc00c1ee)
      return rpc_error_to_nrf_errno(err);
    err = nrf_modem_os_timedwait(2, &timeout);
  } while (err == 0);
  if (err < 0)
    return -err;
  return err;

which calls into the zephyr support code


int32_t nrf_modem_os_timedwait(uint32_t context, int32_t *timeout)
  struct sleeping_thread thread;
...
  if (*timeout < 0)
    *timeout = SYS_FOREVER_MS;
...
  sleeping_thread_init(&thread, context);
  if (!sleeping_thread_add(&thread))
    return 0;
  k_sem_take(&thread.sem, SYS_TIMEOUT_MS(*timeout));
...

where sleeping_init() sets up the semaphore, sleeping_add() adds it to a list (w/ interrupts disabled), and then waits on the semapore. this semaphore is signalled at interrupt time from


void nrf_modem_os_event_notify(uint32_t context)
  atomic_inc(&rpc_event_cnt);
  SYS_SLIST_FOR_EACH_CONTAINER(&sleeping_threads, thread, node)
    /* Wake sleeping thread if context of the thread matches, is 0 or the notify
     * context is 0.
     */
  if ((thread->context == context) || (context == 0) || (thread->context == 0))
    k_sem_give(&thread->sem);

however, there's a window from when _req_forward() calls rpc_client_msg_send() and when nrf_modem_os_timedwait() makes the thread available to the isr to be signalled. hoisting the irq lock from sleeping_add() into nrf_modem_os_timedwait() helps our case but i guess there's still a theorhetical race (w/o fully knowing what the modem is off doing for this rpc and the state of the stream)

Parents
  • Hello, could you please provide more information on what LTE device you are running, and what version of the modem FW? Also, what application/sample are you running, and what version of the nRF Connect SDK is it based on? Is this on a custom HW design or a Nordic development kit? Do you have any logs to share with us? A modem trace would be very beneficial to pinpoint exactly where the device gets stuck, please see Capture trace data from an nRF91 modem using nrfutil trace or Capturing modem trace data with Cellular Monitor app.

    Thanks!

    Kind regards,
    Øyvind

  • > could you please provide more information on what LTE device you are running, and what version of the modem FW?

    sorry, i thought i filed this in the nrf9151  categry, but we're using the 2.0.4 fw w/ the 2.7.0 sdk.

    >Is this on a custom HW design or a Nordic development kit? Do you have any logs to share with us?

    right now we're using the circuitdojo feather nrf9151 but we're using the four serial devices for sensors (what we're sending over the lte connection) so cant give either serial or modem traces.

    to get to the point we're at (waiting on a semaphore in one thread and having missed the notification that would signal the semaphore) i have a couple devices running (it's sort of rare and seems related to the load that i havent tracked down) and attach in the debugger. i can see (w/ a modification to the modem_os_xxx functions) that things like the timestamp / rpc count / context when we started waiting was 'just' before we started waiting forever. this might be a coincidence but static analysis seems to show a race window.

Reply
  • > could you please provide more information on what LTE device you are running, and what version of the modem FW?

    sorry, i thought i filed this in the nrf9151  categry, but we're using the 2.0.4 fw w/ the 2.7.0 sdk.

    >Is this on a custom HW design or a Nordic development kit? Do you have any logs to share with us?

    right now we're using the circuitdojo feather nrf9151 but we're using the four serial devices for sensors (what we're sending over the lte connection) so cant give either serial or modem traces.

    to get to the point we're at (waiting on a semaphore in one thread and having missed the notification that would signal the semaphore) i have a couple devices running (it's sort of rare and seems related to the load that i havent tracked down) and attach in the debugger. i can see (w/ a modification to the modem_os_xxx functions) that things like the timestamp / rpc count / context when we started waiting was 'just' before we started waiting forever. this might be a coincidence but static analysis seems to show a race window.

Children
No Data
Related