Integrating memfault with NRF7002 DK

[Vigen]

Hello,

I am evaluating the use of memfault with NRF7002 DK over WiFi. I was able to connect to the memfault web APIs and send error traces. But I got stuck on sending the stack traces. In the logs I see that memfault is trying to send the stack trace with 128 bytes increments, but as 3-5kB of data is sent I start getting the following error from mflt module:

mflt: Data Send Error: len=128, errno=5

I tried debugging the issue and it led me to TLS library - sockets_tls.c::send_tls fails on calling mbedtls_ssl_write. The issue seems like a buffer overflow, but I would expect it to 'dissolve' as the buffered data is sent out. Is this a settings issue or simply 7002 is not compatible with memfault?

Thanks,
-Vigen

PS Please, find attached my prj.conf file:

# System
CONFIG_HW_ID_LIBRARY=y

# Flash access.
CONFIG_FLASH=y
CONFIG_FLASH_MAP=y
CONFIG_FCB=y
CONFIG_SETTINGS_FCB=y
CONFIG_MPU_ALLOW_FLASH_WRITE=y
CONFIG_SETTINGS=y
CONFIG_HW_STACK_PROTECTION=y

# Logging.
## Enable log module.
CONFIG_LOG=y

# Memory.
CONFIG_MAIN_STACK_SIZE=4096
CONFIG_HEAP_MEM_POOL_SIZE=150000
CONFIG_SYSTEM_WORKQUEUE_STACK_SIZE=2048

# NEWLIB C - required for Azure IoT.
CONFIG_NEWLIB_LIBC=y
CONFIG_NEWLIB_LIBC_FLOAT_PRINTF=y
CONFIG_NEWLIB_LIBC_MIN_REQUIRED_HEAP_SIZE=4096

# Network.
CONFIG_NETWORKING=y
CONFIG_NET_NATIVE=y
CONFIG_NET_SOCKETS=y
CONFIG_NET_SOCKETS_POSIX_NAMES=y
CONFIG_NET_L2_ETHERNET=y
CONFIG_NET_DHCPV4=y
CONFIG_NET_IPV4=y
CONFIG_NET_IPV6=n
CONFIG_NET_UDP=y
CONFIG_NET_TCP=y
CONFIG_NET_CONFIG_SETTINGS=y
CONFIG_NET_CONTEXT_SNDTIMEO=y
CONFIG_NET_SOCKETS_SOCKOPT_TLS=y
CONFIG_POSIX_CLOCK=y
CONFIG_POSIX_MAX_FDS=16
## Tune up.
CONFIG_NET_MAX_CONN=6
CONFIG_NET_IF_MAX_IPV4_COUNT=2
CONFIG_NET_TX_STACK_SIZE=8192
CONFIG_NET_RX_STACK_SIZE=8192
CONFIG_NET_MGMT_EVENT_STACK_SIZE=4096
CONFIG_NET_SOCKETS_POLL_MAX=10
CONFIG_NET_SOCKETS_TLS_MAX_CONTEXTS=2

# WiFi stack.
CONFIG_WIFI=y
CONFIG_WIFI_NRF700X=y
CONFIG_NET_L2_WIFI_MGMT=y
## WPA Supplicant is required to establish WiFi connection.
CONFIG_WPA_SUPP=y

# DNS Resolver.
CONFIG_DNS_RESOLVER=y
CONFIG_DNS_RESOLVER_MAX_SERVERS=2
CONFIG_DNS_NUM_CONCUR_QUERIES=5

## mbedtls
CONFIG_MBEDTLS=y
CONFIG_MBEDTLS_ENABLE_HEAP=y
# CONFIG_MBEDTLS_HEAP_SIZE=75000
CONFIG_MBEDTLS_RSA_C=y
CONFIG_MBEDTLS_KEY_EXCHANGE_RSA_ENABLED=y
CONFIG_MBEDTLS_MAC_SHA256_ENABLED=y
CONFIG_MBEDTLS_HEAP_SIZE=125000
# 2 device certs (public/private), 2 Azure CA + 4 memfault CA.
CONFIG_TLS_MAX_CREDENTIALS_NUMBER=8 
# Use hardware acceleration for crypto.
CONFIG_PSA_CRYPTO_DRIVER_CC3XX=y
CONFIG_PSA_CRYPTO_DRIVER_OBERON=n

# Azure IoT config.
CONFIG_AZURE_IOT_HUB=y
CONFIG_AZURE_IOT_HUB_DPS=y
CONFIG_AZURE_IOT_HUB_DPS_ID_SCOPE="<redacted>"

# MQTT
CONFIG_MQTT_LIB=y
CONFIG_MQTT_LIB_TLS=y
CONFIG_MQTT_CLEAN_SESSION=y
CONFIG_MQTT_HELPER_NATIVE_TLS=n
CONFIG_MQTT_HELPER_STACK_SIZE=4096
CONFIG_NET_TCP_WORKQ_STACK_SIZE=4096

# Debug info
CONFIG_ASSERT=y

## Initialize stack traces.
CONFIG_INIT_STACKS=y
CONFIG_STACK_SENTINEL=y
CONFIG_DEBUG_COREDUMP=y
CONFIG_DEBUG_COREDUMP_BACKEND_LOGGING=y
CONFIG_DEBUG_COREDUMP_MEMORY_DUMP_MIN=y

# Memfault.
CONFIG_MEMFAULT=y
CONFIG_MEMFAULT_NCS_PROJECT_KEY="<redacted>"
CONFIG_MEMFAULT_USER_CONFIG_ENABLE=y
CONFIG_MEMFAULT_NRF_CONNECT_SDK=y
CONFIG_MEMFAULT_HTTP_ENABLE=y
CONFIG_MEMFAULT_ROOT_CERT_STORAGE=y
CONFIG_MBEDTLS_SSL_SERVER_NAME_INDICATION=y

[dejans]

Hi,

How do you send stack traces compared to error traces? Could you please provide code/command which you use for sending traces?

What is the highest amount of data sent so that you do not get any error? Is it 128 bytes? What is included in these 128 bytes?

What is the data content in mentioned 3-5 KB?

Best regards,
Dejan

[Vigen]

Hi Dejan,

I am using the standard memfault APIs. The MEMFAULT_TRACE_EVENT_WITH_LOG macro is used to record traces, and a scheduled job that calls memfault_nrfconnect_port_post_data posting that data to memfault web.

The 128 bytes seems to be the memfault chunk size when data amount is big. I can see XML-ish data there, and I assume its just the stack trace of the last crash before reset happened. I am not sure what the dump size is, but I can check if you tell me how.

I deduced the 3-5kB limit by number of 128 byte chunks memfault sends before the above error is logged. It's usually between 23 and 30 chunks.

Hope this helps.

-Vigen

[dejans]

Hi,

The error might be related to the value for the buffer size specified in CONFIG_NET_BUF_DATA_SIZE which is given in boards\nrf7002dk_nrf5340_cpuapp.conf in memfault sample (nrf\samples\debug\memfault)

You could try increasing the buffer size from default value which is 128 to 1024.

CONFIG_NET_BUF_DATA_SIZE=1024

Best regards,
Dejan

[Vigen]

With that setting the wifi impl on NRF7002 DK fails to connect with error -1:

net_mgmt(NET_REQUEST_WIFI_CONNECT,...);

The 128 byte limit comes from the memfault implementation in memfault_platform_http.c:261.

I think I figured it out anyway - I set CONFIG_MEMFAULT_HTTP_MAX_POST_SIZE to 1024 and the chunking loop seem to be passing. Maybe this is something you'd like to add to memfault integration with WiFi enabled devices (your docs only list 9160 LTE and BLE memfault reporting).

[dejans]

Hi,

Thank you for sharing the solution to the problem.

I have provided your feedback internally.

Best regards,
Dejan