RE: Cellular and WiFi Stack Management on a custom nRF9160 board

kris_01 — Mon, 23 Sep 2024 07:56:16 GMT

Additionally, we received an answer from rlubos on Zephyr's discord:

What you described in the devzone ticket did sound like a bug , as with SO_BINDTODEVICE being used with dispatcher, socket priorities should not be used at all, given proper interface names are provided (which seems to be true in your case), so it shouldn't be needed to change priorities.

It turned out that in NCS 2.5.0 we indeed had a bug, incorrect macro was used to register the offload socket implementation, so the dispatcher wrongly treated the nRF91 LTE interface as a native one. This was fixed a few months ago in sdk-nrf:
github.com/.../f26d29161dd3553ee68ca7ed77704933a58ad994

Cherry-picking this commit into v2.5.0-branch (or switching to NCS 2.6.0) should resolve the issue you see.

RE: Cellular and WiFi Stack Management on a custom nRF9160 board

kris_01 — Fri, 16 Aug 2024 07:57:52 GMT

Hi Simon, we made some progress on this.

We are now able to use the WiFi and LTE interfaces simultaneously, at least with non-TLS sockets.

After reading through the Zephyr documentation, we learned about the
socket dispatcher, which enables the application to specify which network implementation a socket should be bound to.

As shown in the first example in the above-linked documentation, a socket first needs to be created
Then, the SO_BINDTODEVICE option, with the interface name, is set with setsockopt.

The names are not arbitrary, but rather the names of the network interfaces. For WiFi, the name is
wlan0 and for LTE, the name is net0.

With that information, we started integrating the offloaded socket dispatcher into the
nrf/samples/net/udp sample, as it seemed to be the simplest sample to begin with.

I will post the code below, but here are the important changes:

- CONFIG_NET_SOCKETS_OFFLOAD and CONFIG_NET_SOCKETS_OFFLOAD_DISPATCHER must be enabled.
- The CONFIG_NET_IF_MAX_IPV4_COUNT must be set to 2, since we have two interfaces which require
their IPs.
- Various CONFIG_WIFI_* and CONFIG_WIFI_ESP_AT_* options must be enabled.
- You can't use the NET_EVENT_L4_CONNECTED event as a trigger to create both WiFi and LTE
sockets, since it is emitted only once when the first interface is connected. Instead the
NET_EVENT_IF_UP event must be used, as it is emitted for each interface.

We thought that the above changes would be enough, however, during our initial tests we saw that:

- If only esp32 wifi and socket dispatcher are enabled, then the setsockopt and the following
connect calls are successful.
- If only nRF91's LTE and socket dispatcher are enabled, then the setsockopt and the following
connect calls are successful.
- If all three are enabled and the setsockopt is given the wlan0 interface, then the nRF91
socket implementation is used instead.

The last case was very tricky to figure out and we had to dig into the Zephyr source code to find
out what exactly was happening.

By using the debugger and inspecting the
socket_dispatcher.c we found out that:

- When the setsockopt is called with the SO_BINDTODEVICE option and the wlan0 interface name
the following functions get called:

- sock_dispatch_setsockopt_vmeth
- sock_dispatch_native
- sock_dispatch_find

- sock_dispatch_find iterates through registered sockets. The order of the sockets is determined
by their priority that was given when they were registered with NET_SOCKET_REGISTER.
- nRF91's LTE uses an offloaded socket interface, which has a default priority of 40.
- ESP32 AT uses a native Zephyr socket interface, which has a default priority of 50.

This means that sock_dispatch_find will always return the nRF91's LTE socket implementation first.

The solution was to set the CONFIG_NET_SOCKETS_PRIORITY_DEFAULT to a value lower than 40, so that
the native Zephyr socket implementation is used by default.

I find it interesting that around line 300 there is a check that decides how to handle the socket
creation based on whether the interface is native or offloaded. Even if we are requesting a socket
with a native interface, the offloaded interfaces can be used if their priority is higher.

I feel like this is a bug, but I am not sure.

We will attempt to do the same with secure sockets (TLS/DTLS).

Below are all the relevant files, if someone uses this don't forget to adjust the overlay and the
WiFi SSID and password. Keep in mind, we used the nrf/samples/net/udp (NCS version 2.5.0) as a basis for this.

main.c:

#include <zephyr/kernel.h>
#include <zephyr/logging/log.h>
#include <zephyr/logging/log_ctrl.h>
#include <zephyr/sys/reboot.h>

#include <zephyr/net/conn_mgr_connectivity.h>
#include <zephyr/net/conn_mgr_monitor.h>
#include <zephyr/net/net_if.h>
#include <zephyr/net/socket.h>
#include <zephyr/net/wifi_mgmt.h>

#include <stdio.h>

LOG_MODULE_REGISTER(udp_sample, CONFIG_UDP_SAMPLE_LOG_LEVEL);

#define UDP_IP_HEADER_SIZE 28

#define CONN_LAYER_EVENT_MASK (NET_EVENT_CONN_IF_FATAL_ERROR)

static struct net_mgmt_event_callback conn_cb;
static struct net_mgmt_event_callback iface_up_cb;

static int lte_fd;
static int wifi_fd;

static struct sockaddr_storage host_addr;

/* Forward declarations */
static void wifi_work_fn(struct k_work *work);

/* Work item used to schedule periodic UDP transmissions. */
static K_WORK_DELAYABLE_DEFINE(wifi_work, wifi_work_fn);

static void lte_work_fn(struct k_work *work);

static K_WORK_DELAYABLE_DEFINE(lte_work, lte_work_fn);

static void wifi_work_fn(struct k_work *work)
{
	int err;
	char buffer[] = "wifi";

	LOG_INF("WIFI: Transmitting UDP/IP payload of %d bytes to the "
		"IP address %s, port number %d",
		sizeof(buffer) + UDP_IP_HEADER_SIZE, CONFIG_SERVER_ADDRESS, CONFIG_SERVER_PORT);

	err = send(wifi_fd, buffer, sizeof(buffer), 0);
	if (err < 0) {
		LOG_ERR("Failed to transmit UDP packet, %d", -errno);
		return;
	}

	(void)k_work_reschedule(&wifi_work, K_SECONDS(5));
}

static void lte_work_fn(struct k_work *work)
{
	int err;
	char buffer[] = "lte";

	LOG_INF("LTE: Transmitting UDP/IP payload of %d bytes to the "
		"IP address %s, port number %d",
		sizeof(buffer) + UDP_IP_HEADER_SIZE, CONFIG_SERVER_ADDRESS, CONFIG_SERVER_PORT);

	err = send(lte_fd, buffer, sizeof(buffer), 0);
	if (err < 0) {
		LOG_ERR("Failed to transmit UDP packet, %d", -errno);
		return;
	}

	(void)k_work_reschedule(&lte_work, K_SECONDS(5));
}

static int server_addr_construct(void)
{
	int err;

	struct sockaddr_in *server4 = ((struct sockaddr_in *)&host_addr);

	server4->sin_family = AF_INET;
	server4->sin_port = htons(CONFIG_SERVER_PORT);

	err = inet_pton(AF_INET, CONFIG_SERVER_ADDRESS, &server4->sin_addr);
	if (err < 0) {
		LOG_ERR("inet_pton, error: %d", -errno);
		return err;
	}

	return 0;
}

static int prv_socket_create(struct net_if *iface)
{
	LOG_WRN("Create socket for iface %s", iface->config.name);

	int fd = socket(AF_INET, SOCK_DGRAM, IPPROTO_UDP);
	if (fd < 0) {
		LOG_ERR("Failed to create UDP socket: %d", -errno);
		return -1;
	}

	struct ifreq ifreq = {.ifr_name = {0}};

	strncpy(ifreq.ifr_name, iface->config.name, sizeof(ifreq.ifr_name));

	setsockopt(fd, SOL_SOCKET, SO_BINDTODEVICE, &ifreq, sizeof(ifreq));

	return fd;
}

static int prv_connect(int fd)
{
	int err;

	err = connect(fd, (struct sockaddr *)&host_addr, sizeof(struct sockaddr_in));
	if (err < 0) {
		LOG_ERR("Connect, error: %d", -errno);
		return err;
	}

	return 0;
}

static void iface_up_handler(struct net_mgmt_event_callback *cb, uint32_t mgmt_event,
			     struct net_if *iface)
{
	LOG_WRN("Iface name %s", iface->config.name);

	int fd = prv_socket_create(iface);

	if (mgmt_event == NET_EVENT_IF_UP) {
		if (strcmp(iface->config.name, "wlan0") == 0) {
			wifi_fd = fd;
			prv_connect(wifi_fd);
			k_work_reschedule(&wifi_work, K_NO_WAIT);
		} else if (strcmp(iface->config.name, "net0") == 0) {
			lte_fd = fd;
			prv_connect(lte_fd);
			k_work_reschedule(&lte_work, K_NO_WAIT);
		}
	}
}

static void connectivity_event_handler(struct net_mgmt_event_callback *cb, uint32_t event,
				       struct net_if *iface)
{
	if (event == NET_EVENT_CONN_IF_FATAL_ERROR) {
		LOG_ERR("NET_EVENT_CONN_IF_FATAL_ERROR");
		return;
	}
}

int wifi_connect(char *ap_name, char *psk)
{
	int err;
	struct net_if *iface = net_if_get_default();

	/* Set up connection parameters */
	static struct wifi_connect_req_params cnx_params;
	cnx_params.ssid = (uint8_t *)ap_name;
	cnx_params.ssid_length = strlen(ap_name);

	/* No need to select channel, so pass any */
	cnx_params.channel = WIFI_CHANNEL_ANY;

	/* if passkey is NULL, the AP is unsecured */
	if (!psk) {
		cnx_params.security = WIFI_SECURITY_TYPE_NONE;
	} else {
		cnx_params.psk = (uint8_t *)psk;
		cnx_params.psk_length = strlen(psk);
		cnx_params.security = WIFI_SECURITY_TYPE_PSK;
	}

	err = net_mgmt(NET_REQUEST_WIFI_CONNECT, iface, &cnx_params,
		       sizeof(struct wifi_connect_req_params));
	if (err) {
		LOG_ERR("Connection request failed, err: %d", err);
		return -ENOEXEC;
	}

	return 0;
}

int main(void)
{
	int err;

	LOG_INF("UDP sample has started");

	net_mgmt_init_event_callback(&conn_cb, connectivity_event_handler, CONN_LAYER_EVENT_MASK);
	net_mgmt_add_event_callback(&conn_cb);

	net_mgmt_init_event_callback(&iface_up_cb, iface_up_handler, NET_EVENT_IF_UP);
	net_mgmt_add_event_callback(&iface_up_cb);

	err = server_addr_construct();
	if (err) {
		LOG_INF("server_addr_construct, error: %d", err);
		return err;
	}

	LOG_INF("Bringing network interface up and connecting to the network");

	wifi_connect(CONFIG_AP_SSID, CONFIG_AP_PSK);

	err = conn_mgr_all_if_up(true);
	if (err) {
		LOG_ERR("conn_mgr_all_if_up, error: %d", err);
		return err;
	}

	return 0;
}

prj.conf:

# General configurations
CONFIG_LOG=y
CONFIG_LOG_INFO_COLOR_GREEN=y
CONFIG_CONSOLE=y
CONFIG_RTT_CONSOLE=y
CONFIG_USE_SEGGER_RTT=y

# CONFIG_SHELL=y
# CONFIG_SHELL_LOG_BACKEND=y
# CONFIG_SHELL_BACKEND_RTT=y
# CONFIG_SEGGER_RTT_BUFFER_SIZE_DOWN=128

CONFIG_DEBUG_OPTIMIZATIONS=y
CONFIG_DEBUG_THREAD_INFO=y
CONFIG_DEBUG_INFO=y

# For communication with ESP-32
CONFIG_SERIAL=y

# Network
CONFIG_NETWORKING=y
CONFIG_NET_NATIVE=y
CONFIG_NET_IPV4=y
CONFIG_NET_SOCKETS=y
CONFIG_NET_CONNECTION_MANAGER=y
CONFIG_NET_CONNECTION_MANAGER_MONITOR_STACK_SIZE=4096
# Increase this to avoid stack overflow in net_mgmt
CONFIG_NET_MGMT_EVENT_STACK_SIZE=4096

# Increase this to allow more than one IPv4 address
CONFIG_NET_IF_MAX_IPV4_COUNT=2

# Heap and stacks
CONFIG_HEAP_MEM_POOL_SIZE=1024
CONFIG_MAIN_STACK_SIZE=4096
CONFIG_SYSTEM_WORKQUEUE_STACK_SIZE=2048

# ESP specific flags
CONFIG_WIFI_LOG_LEVEL_DBG=y
CONFIG_WIFI=y
CONFIG_WIFI_ESP_AT=y
CONFIG_WIFI_ESP_AT_SCAN_RESULT_RSSI_ORDERED=y
CONFIG_WIFI_ESP_AT_PASSIVE_MODE=y
CONFIG_WIFI_ESP_AT_IP_DHCP=y
CONFIG_WIFI_ESP_AT_VERSION_2_0=y
CONFIG_WIFI_ESP_AT_DNS_USE=y
# For ESP-AT DNS
CONFIG_DNS_RESOLVER=y

# Offload and dispatcher
CONFIG_NET_SOCKETS_OFFLOAD=y
CONFIG_NET_SOCKETS_OFFLOAD_DISPATCHER=y

# Setting the priority of native sockets to less than the offloaded sockets,
# so that ESP does NOT use nrf91 socket offloading.
# Default priority is 50
CONFIG_NET_SOCKETS_PRIORITY_DEFAULT=2

# Modem LIB forces nrf91 socket offloading
CONFIG_NRF_MODEM_LIB=y

CONFIG_NET_L2_ETHERNET=y
CONFIG_NET_UDP=y
CONFIG_NET_DHCPV4=y

############################################################# LTE #############################################################
# Modem related configurations
CONFIG_LTE_CONNECTIVITY_LOG_LEVEL_DBG=y
CONFIG_NRF_MODEM_LIB_NET_IF_AUTO_START=n
CONFIG_NRF_MODEM_LIB_ON_FAULT_APPLICATION_SPECIFIC=y

# Disable Duplicate Address Detection (DAD)
# due to not being properly implemented for offloaded interfaces.
CONFIG_NET_IPV6_NBR_CACHE=n
CONFIG_NET_IPV6_MLD=n

# Zephyr NET Connection Manager and Connectivity layer.
CONFIG_LTE_CONNECTIVITY=y
CONFIG_LTE_CONNECTIVITY_AUTO_DOWN=n

# PSM
CONFIG_LTE_PSM_REQ=y
CONFIG_LTE_PSM_REQ_RPTAU="00100001"
CONFIG_LTE_PSM_REQ_RAT="00000000"

# eDRX
CONFIG_LTE_EDRX_REQ=n
CONFIG_LTE_EDRX_REQ_VALUE_LTE_M="1001"

# RAI
CONFIG_LTE_RAI_REQ=n
CONFIG_LTE_RAI_REQ_VALUE="4"

CONFIG_AT_MONITOR_HEAP_SIZE=1024

Parameters such as WiFi AP, SSID, server address, and port are defined in Kconfig.

RE: Cellular and WiFi Stack Management on a custom nRF9160 board

Simonr — Wed, 07 Aug 2024 06:38:23 GMT

I'm sorry, but we don't have much more than the networking documentation on setting up our devices. We can leave the case open if someone in the community would like to chime in though.

Best regards,

Simon

RE: Cellular and WiFi Stack Management on a custom nRF9160 board

kris_01 — Tue, 06 Aug 2024 11:30:18 GMT

Hi Simon.

Thanks for your reply, the problem is not RF co-existance but network stack co-existance.

We also looked at the cellular/modem_shell sample for the nRF9160dk where there is support for the WiFi configuration with the ESP8266 chip running AT firmware.

This setup seems to be quite similar to ours, however we can't understand how we could adapt this to our use-case.
Could your provide some details on that? Specifically, what network stack configurations are required and what APIs should we use to accomplish this?

RE: Cellular and WiFi Stack Management on a custom nRF9160 board

Simonr — Tue, 06 Aug 2024 10:43:11 GMT

On the LTE side, you can use the AT command AT%COEX, and I assume you can use the MPSL timeslots for the multiprotocol service layer. For the ESP32 co-processor we here at Nordic don't have any experience with, so I don't have much in terms of advice here.

Best regards,

Simon