Issues with running FOTA and cJSON

[simon belanger]

Hi,

I recently added FOTA support on my project which runs on nRF5340 + nRF7002. I am currently working on a devboard, nRF7002DK.

Previously, I was running the project on NCS 2.7.0 and I was using the cJSON library to generate the payloads for my MQTT messages. When adding FOTA support to the project, I migrated to NCS 2.9.0 to facilitate the switch to sysbuild, which allowed be to embed MCUBoot as the bootloader. I implemented FOTA using the FOTA download library.

However, I am now having issues with the MQTT payload that I send periodically. Whenever a payload is sent, which is formatted in JSON using cJSON library, I know I will be unable to perform an FOTA update if I do not reboot first. My first guess would be memory management but I am not sure about it.

The error I get from the FOTA download is as such : 

[00:00:26.841,094] <inf> mqtt: topic received gateway/update/0011ce
[00:00:26.841,156] <inf> mqtt: Received: https://to-delete-testing.s3.ca-central-1.amazonaws.com/zephyr.signed.bin
[00:00:26.841,186] <inf> main: Update queried
[00:00:26.841,186] <inf> ota: https://bucket.s3.ca-central-1.amazonaws.com/zephyr.signed.bin
[00:00:26.841,247] <inf> ota: host : bucket.s3.ca-central-1.amazonaws.com
[00:00:26.841,918] <inf> ota: file : zephyr.signed.bin
[00:00:26.844,512] <inf> mqtt:  Deiniting MQTT
[00:00:26.844,543] <inf> mqtt: Disconnecting MQTT client
[00:00:26.846,221] <inf> mqtt: MQTT client disconnected: 0
[00:00:26.846,252] <inf> mqtt: mqtt thread suspended
[00:00:26.846,252] <inf> ota: Mqtt was closed. Starting FOTA
[00:00:26.846,313] <inf> ota: FOTA inited
[00:00:26.846,405] <inf> download_client: Downloading: zephyr.signed.bin [0]
[00:00:26.846,405] <inf> ota: FOTA started
[00:00:26.973,480] <err> download_client: DNS lookup failed bucket.s3.ca-central-1.amazonaws.com
[00:00:26.973,510] <err> fota_download: Download client error
[00:00:26.973,510] <inf> ota: Received error from fota_download, 2

However, when I do not use the cJSON library prior to starting the update, it works flawlessly.

I was wondering if there are know changes I should know about between versions 2.7.0 and 2.9.0 of NCS whcich affect the way I should configure my project. My configuration file is a such

CONFIG_GPIO=n

## To remove on production version
# Printing to UART
CONFIG_SERIAL=n
# Logging
CONFIG_LOG=n

# Wi-Fi
CONFIG_WIFI=y
CONFIG_NRF70_SR_COEX=y
CONFIG_NRF70_SR_COEX_RF_SWITCH=y

# WPA supplicant
CONFIG_WIFI_NM_WPA_SUPPLICANT=y
CONFIG_WIFI_NM_WPA_SUPPLICANT_LOG_LEVEL_OFF=y

CONFIG_NRF70_RX_NUM_BUFS=16
CONFIG_NRF70_MAX_TX_AGGREGATION=4


# Wi-Fi credentials
CONFIG_WIFI_CREDENTIALS=y
CONFIG_WIFI_CREDENTIALS_STATIC=n

# Networking Management API
CONFIG_NET_MGMT=y
CONFIG_NET_MGMT_EVENT=y
CONFIG_NET_MGMT_EVENT_INFO=y
CONFIG_NET_MGMT_EVENT_STACK_SIZE=5120
CONFIG_NET_CONNECTION_MANAGER=y
CONFIG_L2_WIFI_CONNECTIVITY=y
CONFIG_NET_CONNECTION_MANAGER_MONITOR_STACK_SIZE=5200
CONFIG_WIFI_MGMT_EXT=y
#CONFIG_WIFI_MGMT_AP_STA_INACTIVITY_TIMEOUT=0
#CONFIG_BSS_MAX_IDLE_TIME=0

# Support for shell commands
#CONFIG_SHELL=y
#CONFIG_WIFI_CREDENTIALS_SHELL=y
#CONFIG_SHELL_STACK_SIZE=4400

# DK library
CONFIG_DK_LIBRARY=y

# System settings
CONFIG_NEWLIB_LIBC=y
CONFIG_NEWLIB_LIBC_NANO=n
CONFIG_NEWLIB_LIBC_FLOAT_PRINTF=y
CONFIG_CJSON_LIB=y

# Networking
CONFIG_NETWORKING=y
CONFIG_NET_NATIVE=y
CONFIG_NET_SOCKETS=y
CONFIG_NET_SOCKETS_POSIX_NAMES=y
CONFIG_POSIX_MAX_FDS=16

CONFIG_NET_L2_ETHERNET=y
CONFIG_NET_IPV6=y
CONFIG_NET_IPV4=y
CONFIG_NET_TCP=y
CONFIG_NET_DHCPV4=y
CONFIG_DNS_RESOLVER=y

# Memory
CONFIG_SYSTEM_WORKQUEUE_STACK_SIZE=8192
CONFIG_MAIN_STACK_SIZE=5200
CONFIG_HEAP_MEM_POOL_SIZE=153600
#CONFIG_HEAP_MEM_POOL_IGNORE_MIN=y

CONFIG_NET_TCP_WORKQ_STACK_SIZE=2048
CONFIG_NET_TX_STACK_SIZE=4096
CONFIG_NET_RX_STACK_SIZE=4096
CONFIG_NET_BUF_RX_COUNT=16
CONFIG_NET_BUF_TX_COUNT=16
CONFIG_NET_BUF_DATA_SIZE=256
CONFIG_NET_TC_TX_COUNT=0

# MQTT
CONFIG_MQTT_LIB=y
CONFIG_MQTT_CLEAN_SESSION=y
CONFIG_MQTT_LIB_TLS=y  # Enable TLS for the MQTT library

# STEP 1.5 - Increase the MQTT keep alive time (seconds)
CONFIG_MQTT_KEEPALIVE=240

CONFIG_MQTT_BROKER_HOSTNAME="iot.elia.one"
CONFIG_MQTT_BROKER_PORT=8883

# TLS
# STEP 1.3 - Enable the TLS library and security relevant configurations
CONFIG_NET_SOCKETS_SOCKOPT_TLS=y
CONFIG_TLS_CREDENTIALS=y
CONFIG_MBEDTLS_RSA_C=y

# mbedTLS
CONFIG_NRF_SECURITY=y
CONFIG_MBEDTLS=y
CONFIG_MBEDTLS_ENABLE_HEAP=y
CONFIG_MBEDTLS_HEAP_SIZE=81920
CONFIG_PSA_WANT_KEY_TYPE_RSA_PUBLIC_KEY=y
CONFIG_PSA_WANT_RSA_KEY_SIZE_2048=y
#CONFIG_MBEDTLS_RSA_C=y
CONFIG_MBEDTLS_TLS_LIBRARY=y

# Certificates storage
CONFIG_FLASH=y
CONFIG_FLASH_MAP=y

CONFIG_FILE_SYSTEM=y
CONFIG_FILE_SYSTEM_LITTLEFS=y

# External Flash SPI NOR
CONFIG_GPIO=y
CONFIG_SPI=y
CONFIG_SPI_NOR=y
CONFIG_SPI_NOR_SFDP_DEVICETREE=y
CONFIG_SPI_NOR_FLASH_LAYOUT_PAGE_SIZE=4096
# QSPI drivers are enabled by default for some chips.
# Disable it explicitly to be sure QSPI is disabled.
CONFIG_NORDIC_QSPI_NOR=n 

# Misc
CONFIG_REBOOT=y
CONFIG_ADC=y #for battery monitoring
CONFIG_SYS_HEAP_RUNTIME_STATS=y

# FOTA
# FOTA library
CONFIG_FOTA_DOWNLOAD=y

# Download client
CONFIG_DOWNLOAD_CLIENT=y
CONFIG_DOWNLOAD_CLIENT_STACK_SIZE=4096

# DFU Target
CONFIG_DFU_TARGET=y

# Image manager
CONFIG_IMG_MANAGER=y
CONFIG_FLASH=y
CONFIG_FLASH_MAP=y
CONFIG_STREAM_FLASH=y
CONFIG_IMG_ERASE_PROGRESSIVELY=y

# Application Upgrade support
CONFIG_BOOTLOADER_MCUBOOT=y

# Network
CONFIG_NET_L2_WIFI_MGMT=y

CONFIG_NANOPB=y

# Memory
CONFIG_NET_MGMT_EVENT_STACK_SIZE=5120

CONFIG_NET_MGMT_EVENT_QUEUE_TIMEOUT=5000

# Bluetooth LE
CONFIG_BT=y
CONFIG_BT_PERIPHERAL=y
CONFIG_BT_PERIPHERAL_PREF_TIMEOUT=75
CONFIG_BT_WIFI_PROV=y
CONFIG_BT_BONDABLE=n

#CONFIG_BT_DEBUG_LOG=y
CONFIG_BT_SMP=y
CONFIG_BT_BUF_ACL_RX_SIZE=151
CONFIG_BT_L2CAP_TX_MTU=147
CONFIG_BT_BUF_ACL_TX_SIZE=151
CONFIG_BT_RX_STACK_SIZE=5120
CONFIG_BT_DEVICE_NAME_DYNAMIC=y

# Storing credentials
# STEP 1.2 - Enable the necessary NVS and flash configurations
CONFIG_FLASH=y
CONFIG_FLASH_PAGE_LAYOUT=y
CONFIG_FLASH_MAP=y

CONFIG_NVS=y
CONFIG_SETTINGS=y
CONFIG_SETTINGS_NVS=y

# STEP 1.3 - Enable the necessary Wi-Fi credential configurations
CONFIG_WIFI_CREDENTIALS=y
CONFIG_WIFI_CREDENTIALS_MAX_ENTRIES=1
CONFIG_WIFI_CREDENTIALS_BLE=y

# Support for shell commands
CONFIG_SHELL=n
CONFIG_WIFI_CREDENTIALS_SHELL=n
#CONFIG_SHELL_STACK_SIZE=4400

# Fixes bug where some devices would have trouble connecting 
CONFIG_BT_AUTO_PHY_UPDATE=n

and here is an excerpt of the app code that uses the cJSON library

void device_publish_status(void) {
    int err;
    char *json_payload = NULL;
    cJSON *response = NULL;

    // Root section
    if ((response = cJSON_CreateObject()) == NULL) {
        goto end;
    }
    if (cJSON_AddStringToObject(response, "device_id", device_id) == NULL) {
        goto end;
    }
    if (cJSON_AddStringToObject(response, "device_type", "room-lp") == NULL) {
        goto end;
    }
    if (cJSON_AddStringToObject(response, "topic", MQTT_TOPIC_STR[TOPIC_GATEWAY_STATUS]) == NULL) {
        goto end;
    }
    if (cJSON_AddNumberToObject(response, "uptime_mins", k_uptime_seconds()/60) == NULL) {
        goto end;
    }
    if (cJSON_AddNumberToObject(response, "mqtt_uptime_mins", mqtt_uptime()/60) == NULL) {
        goto end;
    }
    if (cJSON_AddNumberToObject(response, "net_uptime_mins", net_uptime_seconds()/60) == NULL) {
        goto end;
    }
    int battery_voltage_mv = 0;
    if ((err = battery_monitor_read_battery_voltage(&battery_voltage_mv)) != 0) {
        //LOG_ERR("Failed to read battery voltage : %d", err);
    }
    if (cJSON_AddNumberToObject(response, "battery_voltage_mv", battery_voltage_mv) == NULL) {
        //goto end;
    }

    // App section
    cJSON *app = NULL;
    if ((app = cJSON_CreateObject()) == NULL) {
        goto end;
    }
    if (cJSON_AddStringToObject(app, "version", TOSTRING(APP_BUILD_VERSION)) == NULL) {
        goto end;
    }
    if (cJSON_AddStringToObject(app, "time", __TIME__) == NULL) {
        goto end;
    }
    if (cJSON_AddStringToObject(app, "date", __DATE__) == NULL) {
        goto end;
    }
    if (cJSON_AddItemToObject(response, "app", app) == NULL) {
        goto end;
    }

    // Network status
    cJSON *net;
    if ((net = network_serialize_wifi_iface_status()) == NULL) {
        //goto end;
    }
    if (cJSON_AddItemToObject(response, "wifi_status", net) == NULL) {
        goto end;
    }

    //PS config
    cJSON* ps_status;
    if ((ps_status = network_serialize_wifi_ps_status()) == NULL) {
        goto end;
    }
    if (cJSON_AddItemToObject(response, "ps_status", ps_status) == NULL) {
        //goto end;
    }
   
    if ((json_payload = cJSON_Print(response)) == NULL) {
        LOG_ERR("Failed to format response to JSON.");
        goto end;
    }

    // send payload to response topic
    if ((err = publish_to_topic(TOPIC_GATEWAY_STATUS_RESPONSE, json_payload, strlen(json_payload))) != 0) {
        LOG_ERR("Failed to send message, %d", err);
    }
    cJSON_FreeString(json_payload);
end:
    cJSON_Delete(response);
    return;
}

[simon belanger]

As a supplement, here are my latest findings. 

It seems like the issue can be traced back to cJSON numbers. I can reproduce the issue by creating a cJSON object and adding a single numeral field using cJSON_AddNumberToObject(), then printing this cJSON as a string using cJSON_Print and then freeing everything. If I were to create a string field using cJSON_AddStringToObject(), the bug would not occur.

void device_publish_status(void) {
	char *json_payload = NULL;
	cJSON *response = NULL;

	if ((response = cJSON_CreateObject()) == NULL) {
		goto end;
	}
	if (cJSON_AddNumberToObject(response, "battery_voltage_mv", 42) == NULL) {
		goto end;
	}
	
	if ((json_payload = cJSON_Print(response)) == NULL) {
		LOG_ERR("Failed to format response to JSON.");
		goto end;
	}
	
	cJSON_FreeString(json_payload);
	LOG_INF("String has been freed");
end:
    cJSON_Delete(response);
	LOG_INF("Object has been freed");
	return;
}

So there seems to be an issue with cJSON_Number fields from my perspective that cause issues with memory which prevents the download client to perform OTA successfully. I did not include the OTA code but it is very similar to Cellular: HTTP application update.

I should also mention that I am using Zephyr's allocators (k_malloc and k_free) on system heap. This is done by running cJSON_Init() from cjson_os.h at the beginning of the main application, before doing anything else. I am also freeing the string using cJSON_FreeString() in order to free memory for the output string using the same allocator I used to allocate it.

[Simonr]

Hi

I can't find anything specific related to JSON when moving from NCS v2.7.0 to 2.9.0, but there are a few migration steps you should be wary of when moving over. First off, the steps in "migration guide for NCS v2.8.0" has a lot of steps on how to move from 2.7.0 to 2.8.0. You should also go through the "Migrating from multi-image builds to sysbuild" to make sure you've included all necessary steps. Additionally, the error that seems to return is a "No such file or directory" error from what I assume is the fota_dl_handler() function apparently due to not being able to connect to the client. How exactly are you linking this to the cJSON being the issue? Does the error only occur if you've done cJSON operations, or could it be something else like the Wi-Fi being active and thus you're not able to do a new connection?

Best regards,

Simon

[simon belanger]

Thanks for the migration guides, I will make sure I did not forget important steps.

Simonr said:

How exactly are you linking this to the cJSON being the issue? Does the error only occur if you've done cJSON operations, or could it be something else like the Wi-Fi being active and thus you're not able to do a new connection?

I am not entirely sure that cJSON is the cause, but I observed that the fota error only occurs when I use the method cJSON_AddNumberToObject() from the cJSON library, as mentionned in my last post. This led me to think that I was doing something wrong with that library. If I do not use this function from cJSON, I get no issues with FOTA. As a workaround, I formatted all my number values to strings so I could use cJSON_AddStringToObject and the FOTA works fine.

Best regards,

Simon

[Simonr]

Hi again

So, the fact that converting the number to strings makes it work it seems to be because a function somewhere expects string(s) and not numbers, and I think it might be this one from the download_client() https://docs.nordicsemi.com/bundle/nrf-apis-latest/page/group_fota_download_ga613aa199d71f84a9531ea42c56776a1b.html#ga613aa199d71f84a9531ea42c56776a1b 

So I'm guessing the download_client tried pulling one or more of your number values when a string was expected causing this issue.

Best regards,

Simon