DATA LOSS

[Rachita Reddy]

---

Support Ticket Draft: Severe PPP RX Buffer Overflow on nRF5340 Gateway (NCS v2.5.0)

Subject: Persistent net_ppp: Rx buffer doesn't have enough space Errors on nRF5340 Gateway (Data Loss)

Environment:

• SDK/Version: nRF Connect SDK (NCS) v2.5.0

• Gateway Board: nRF5340 DK / Custom Board (AP/Net Core)

• Node Device: nRF52 Series

• Connectivity: Cellular (GSM/PPP, Quectel Modem, APN: airtelgprs.com)

• Application Role: BLE Observer/Gateway, forwarding data to AWS IoT Core.

---

1. Problem Description

Our nRF5340 gateway, which is simultaneously running BLE Observer (receiving high-rate data from nRF52 nodes) and Cellular PPP (sending data to AWS), is experiencing severe data loss and network instability.

The device console is constantly saturated with the following PPP buffer error, indicating the kernel cannot process incoming cellular traffic fast enough, possibly due to buffer exhaustion or a blocked thread:

<err> net_ppp: Rx buffer doesn't have enough space. Bytes pending: 1, written: 0

This persistent error is blocking essential network functions, leading to the SNTP clock synchronization failure and subsequent AWS connection refusal.

2. Configuration and Concurrency Status

We have implemented extensive thread-safety measures and buffer increases:

• Concurrency Fixed: We replaced a blocking k_mutex with a dedicated struct k_spinlock to protect shared Flash indices (readIndex, writeIndex), isolating slow Flash I/O from the kernel and high-priority workqueues.

• Buffer Attempts: We have tried increasing standard buffer counts (e.g., CONFIG_NET_PKT_RX_COUNT=256, CONFIG_NET_BUF_RX_COUNT=256, CONFIG_NET_BUF_DATA_SIZE=1500).

3. Hypothesis & Critical Questions

Since increasing the standard buffer pools failed, we suspect the issue lies in resource allocation specific to the nRF5340's multi-core architecture or the PPP processing thread priority.

1. Thread Priority: We believe the PPP RX processing thread may be running at a priority too low to compete with other Application Core threads (like our BLE observer/Flash workqueue, which uses the system workqueue). What is the recommended priority for the PPP RX thread to prevent starvation?

2. Core Memory: Is it possible that the buffers are being depleted on the Network Core (Net Core) before they even reach the Application Core's memory management, and are there specific Kconfig options (outside the standard CONFIG_NET_BUF_* group) to allocate more buffers or memory specifically for the Net Core's PPP function in NCS v2.5.0?

3. PPP RX Stack: Is CONFIG_NET_PPP_RX_STACK_SIZE=8192 an adequate stack size for high-traffic scenarios on the nRF5340, or should this be increased further?

Thank you for your assistance in diagnosing this severe memory/concurrency issue on the nRF5340 gateway.

[dejans]

Hi,

Thread Priority: We believe the PPP RX processing thread may be running at a priority too low to compete with other Application Core threads (like our BLE observer/Flash workqueue, which uses the system workqueue). What is the recommended priority for the PPP RX thread to prevent starvation?

After thread has been started, it is possible to change its initial priority if this is needed. This is described in thread priority.

Core Memory: Is it possible that the buffers are being depleted on the Network Core (Net Core) before they even reach the Application Core's memory management, and are there specific Kconfig options (outside the standard CONFIG_NET_BUF_* group) to allocate more buffers or memory specifically for the Net Core's PPP function in NCS v2.5.0?

Regarding buffers, please refer to network buffer configuration options. 

PPP RX Stack: Is CONFIG_NET_PPP_RX_STACK_SIZE=8192 an adequate stack size for high-traffic scenarios on the nRF5340, or should this be increased further?

It is generally advisable to check if stack sizes are appropriate using thread analyzer. 

Best regards,
Dejan