Memfault OTA for multiple nRF boards (NCS 3.1.1)

[Yun531]

Hello,

Im currently in the process of integrating Memfault OTA on the nRF9151 DK. My project has multiple boards: nRF9151 DK connects with nRF52840 DK via UART for communication. For OTA, I figured that I can use memfault OTA to update the nRF9151 DK because it supports out of the box. However, I'm not sure how to approach for nRF52840 DK. I know that since 2 boards connect using UART, I can put it into DFU mode and just do a serial recovery. From Memfault doc:

Or the doc recommends to bundle the update into 1 OTA:

Since the nRF9151 DK has external flash, it has enough storage for this approach as well. But memfault guide http://mflt.io/nrf-fota-setup said to just call memfault_fota_start to start the OTA process. That works great for the nRF9151 DK, but what about the nRF52840 board I have connected? The document doesn't mention this very clear on how I could do the "get" and "push" for downstream board, or how I can do the "bundle" approach.

Even if I don't consider the nRF52840 DK, I still need to update both the app and the modem fw on the nRF9151 DK. Which function/API should I override? Or is there already an implementation? The only function I can override is the memfault_fota_download_callback, but this is called after everything is done. I just want to check if the payload is supposed to be for the nRF52840 DK board and save it in a different partition instead?

I'd appreciate the help. Thanks so much.

[Øyvind]

Hello, apologies for the late reply. 

I have forwarded this question to our Memfault experts and hope to have an answer by end of tomorrow. Sorry for any inconvenience

Kind regards,
Øyvind

[Yun531]

Thanks Øyvind,

Hope to hear updates from you soon as I'm currently being feature-blocked by this.

[NoahPendleton]

Thanks for the question Yun531 ! This is a somewhat complicated feature, so bear with me-

1. We recommend sending a custom update bundle (as you saw in the docs), however we don't currently have a built-in library for handling this scenario, so you'll have to generate your update bundle yourself, as part of the build process
2. You'll need to enable `CONFIG_DFU_TARGET_CUSTOM=y`, and implement the custom dfu target handler functions (notably, make sure `dfu_target_custom_identify()` can identify the header in your bundle, i.e. put a magic string at the front of it):
   1. https://docs.nordicsemi.com/bundle/ncs-latest/page/nrf/libraries/dfu/dfu_target.html
   2. https://github.com/nrfconnect/sdk-nrf/blob/902e710b42aca9fc06a2a9d69a43cbb0e8465e80/include/dfu/dfu_target_custom.h
3. To apply the downstream device update image, you'll want to use SMP over the serial port connection
   1. You'll need to enable serial SMP on the downstream device
   2. You'll want to include calls to `dfu_target_smp_*` functions inside your `dfu_target_custom` handlers, to properly execute the SMP update when processing the custom update: github.com/.../dfu_target_smp.h

I'd recommend adding a lot of `LOG_INF` and `LOG_ERR` messages in your implementation, to make it easier to debug as you go.

For a first implementation, you might just want to update the nrf91 only, by issuing the correct `dfu_target_mcuboot` calls from your custom DFU target implementation (accounting for the bundle header). Once that works, you can move on to implementing the SMP target.

Let me know if you have any questions as you work on the implementation 

Thanks!
Noah

[Yun531]

Hi Noah,

Thank you for your response. I got the nrf91 only part working. But now I got stuck at the smp part.

Currently I'm using 1 uart instance. nrf91 uses UART2 and nrf52 uses uart1. All are async uart.

Before enable download, I have in my custom dfu target function to put nrf52 into serial recovery, and disable uart2 on the nrf91 before calling smp init with the hope that it'll release the uart for smp to use.

However, my smp client init failed with err 3.

err = dfu_target_smp_client_init();
if (err) {
LOG_ERR("Failed to init smp dfu target: %d", err);
}

<err> nRF9151: Failed to init smp dfu target: 3

Error code 3 for mcumgr is just a generic invalid value error.

Here is my prj.conf for your reference, I only included relevant config here because my config is quite long:

CONFIG_MEMFAULT_FOTA=y

# Enable testing FOTA via a CLI command
# CONFIG_MEMFAULT_FOTA_CLI_CMD=y

# The subsystems we need so OTA payloads can be written to
# flash and updated by MCUBoot
CONFIG_MCUBOOT_IMG_MANAGER=y
CONFIG_DFU_TARGET=y
CONFIG_DFU_TARGET_MCUBOOT=y
CONFIG_DFU_TARGET_SMP=y
CONFIG_DFU_TARGET_CUSTOM=y
CONFIG_IMG_MANAGER=y
CONFIG_FLASH=y
CONFIG_FLASH_MAP=y
CONFIG_STREAM_FLASH=y
CONFIG_IMG_ERASE_PROGRESSIVELY=y

# For Memfault FOTA, we will use the FOTA_DOWNLOAD API's
# from the nRF Connect SDK which depends on the DOWNLOADER
CONFIG_FOTA_DOWNLOAD=y
CONFIG_DOWNLOADER=y
CONFIG_DOWNLOADER_MAX_FILENAME_SIZE=400
CONFIG_DOWNLOADER_STACK_SIZE=1600

# Enable printing of file download progress to console
CONFIG_FOTA_DOWNLOAD_PROGRESS_EVT=y

CONFIG_FOTA_PERIODIC_CHECK_INTERVAL_SECS=120

# SMP Client dependencies
CONFIG_ZCBOR=y
CONFIG_MCUMGR=y
CONFIG_SMP_CLIENT=y
CONFIG_MCUMGR_GRP_IMG_CLIENT=y
CONFIG_MCUMGR_GRP_OS_CLIENT=y
CONFIG_MCUBOOT_BOOTUTIL_LIB=y

# CONFIG_TFM_SECURE_UART=n
# CONFIG_TFM_LOG_LEVEL_SILENCE=y

CONFIG_BASE64=y
CONFIG_CRC=y

# UART
CONFIG_SERIAL=y
CONFIG_UART_ASYNC_API=y

# UART use hardware counter instead
CONFIG_UART_2_ASYNC=y
CONFIG_UART_2_NRF_HW_ASYNC=y
CONFIG_UART_2_NRF_HW_ASYNC_TIMER=2
CONFIG_NRFX_TIMER2=y

And here is the snippet I tried to init dfu_target_smp

/* Helper function to open smp for the aux update */
static int open_aux(uint32_t size) {
	int err;

	/* Activate it here because it deactivates the same uart as smp is using */
	(void)activate_recovery_cb();
	/* Enable smp but in recovery mode */
	err = dfu_target_smp_recovery_mode_enable(dummy_recovery_cb);
	if (err) {
		LOG_ERR("Failed to enable smp recovery mode: %d", err);
		return err;
	}

	err = dfu_target_smp_client_init();
	if (err) {
		LOG_ERR("Failed to init smp dfu target: %d", err);
	}
	return err;
}

In addition, this warning keeps showing up:

warning: UART_2_ASYNC (defined at drivers/serial/Kconfig.nrfx_uart_instance:14) was assigned the
value 'y' but got the value 'n'. Check these unsatisfied dependencies: (!UART_2_INTERRUPT_DRIVEN)
(=n). See docs.zephyrproject.org/.../kconfig.html and/or look up
UART_2_ASYNC in the menuconfig/guiconfig interface. The Application Development Primer, Setting
Configuration Values, and Kconfig - Tips and Best Practices sections of the manual might be helpful
too.

I thought maybe because I use the same UART instance. So I tried to an overlay for uart0 and still got the same error every time I tried to build.

My questions:

1. Is it not possible to use 1 UART instance for both normal communication and smp?

2. If it is possible, how would I do it (release uart device somehow?). If not, I think I need to use a different uart line right?

[NoahPendleton]

Hi Yun531 -

You might find this sample a good reference, it implements an SMP DFU target on the nrf9160dk (which has an on-board nrf52 connected over UART):
https://docs.nordicsemi.com/bundle/ncs-latest/page/nrf/samples/cellular/nrf_cloud_rest_fota/README.html#building_and_running

You can find the relevant Kconfig settings referenced there and in the `smp_svr` sample reference.

It's a little different, since it accepts 1 image at a time (i.e. APP, MODEM, SMP (nrf52 downstream mcuboot payload), but generally it will work similarly to your setup.

[Yun531]

Thank you. I took a look at those. I was able to move past the warning.

But when init, I encountered a SECURE FAULT because I'm sharing 1 serial line.

I tried to use 2 UART lines this time.

UART2 from nRF91 -> UART1 on nRF52 for normal communication.

UART1 from nRF91 -> UART0 on nRF52 for SMP, respectively.

When the check for OTA is available, and the dfu_target_smp tried to send an echo command to the nRF52, it failed. With hardware control lines connected, it just stuck waiting for the flow control lines to be asserted. I disabled it to check, and I was right, the echo command time out and failed.

I verified that the nRF52 is actually in serial recovery because I have the LED1 enabled when it's in serial recovery.

To verify that this is a uart issue, I also try to swap the UART1 and UART0 connection, so now UART2 -> UART0 on the nRF52 DK, and indeed the commands that normally works for UART1 now receives no response on UART0.

Here is the overlay (I put these in both the boards and in mcuboot sysbuild):

/ {
    /* Tell Zephyr the retention region is used for boot-mode */
    chosen {
        /delete-property/ zephyr,shell-uart;
        /delete-property/ zephyr,bt-mon-uart;
        /delete-property/ zephyr,bt-c2h-uart;

        zephyr,boot-mode = &retention0;
        zephyr,uart-mcumgr = &uart0;
    };

    /* Retention region for boot mode */
    sram@2003FC00 {
		compatible = "zephyr,memory-region", "mmio-sram";
		reg = <0x2003FC00 DT_SIZE_K(1)>;
		zephyr,memory-region = "RetainedMem";
		status = "okay";

		retainedmem {
			compatible = "zephyr,retained-ram";
			status = "okay";
			#address-cells = <1>;
			#size-cells = <1>;

			retention0: boot_mode@0 {
				compatible = "zephyr,retention";
				status = "okay";
				reg = <0x0 0x100>;
				prefix = [08 04];
				checksum = <1>;
			};
		};
    };
};

&gpio0 {
        status = "okay";
};

&gpio1 {
        status = "okay";
};

&gpiote0 {
        status = "okay";
};

&nrf_radio_fem_spi {
    status = "disabled";
};

&pinctrl {
    uart0_default_custom: uart0_default_custom {
        group1 {
            psels = <NRF_PSEL(UART_TX, 1, 2)>,
                <NRF_PSEL(UART_RTS, 1, 1)>;
        };
        group2 {
            psels = <NRF_PSEL(UART_RX, 1, 4)>,
                <NRF_PSEL(UART_CTS, 1, 3)>;
            bias-pull-up;
        };
    };

    uart0_sleep_custom: uart0_sleep_custom {
        group1 {
            psels = <NRF_PSEL(UART_TX, 1, 2)>,
                <NRF_PSEL(UART_RX, 1, 4)>,
                <NRF_PSEL(UART_RTS, 1, 1)>,
                <NRF_PSEL(UART_CTS, 1, 3)>;
            low-power-enable;
        };
    };

    uart1_default_custom: uart1_default_custom {
        group1 {
            psels = <NRF_PSEL(UART_TX, 0, 29)>,
                <NRF_PSEL(UART_RTS, 0, 28)>;
        };
        group2 {
            psels = <NRF_PSEL(UART_RX, 0, 31)>,
                <NRF_PSEL(UART_CTS, 0, 30)>;
            bias-pull-up;
        };
    };

    uart1_sleep_custom: uart1_sleep_custom {
        group1 {
            psels = <NRF_PSEL(UART_TX, 0, 29)>,
                <NRF_PSEL(UART_RX, 0, 31)>,
                <NRF_PSEL(UART_RTS, 0, 28)>,
                <NRF_PSEL(UART_CTS, 0, 30)>;
            low-power-enable;
        };
    };
};

&uart0 {
    compatible = "nordic,nrf-uarte";
    status = "okay";
    current-speed = <115200>;
    pinctrl-0 = <&uart0_default_custom>;
    pinctrl-1 = <&uart0_sleep_custom>;
    pinctrl-names = "default", "sleep";
};

&uart1 {
    compatible = "nordic,nrf-uarte";
    status = "okay";
    hw-flow-control;
    current-speed = <115200>;
    pinctrl-0 = <&uart1_default_custom>;
    pinctrl-1 = <&uart1_sleep_custom>;
    pinctrl-names = "default", "sleep";
};

/* Reduce SRAM0 usage to account for non-init area */
&sram0 {
    reg = <0x20000000 DT_SIZE_K(255)>;
};

I don't really know what I should do to make it work. I also attempted to cut solder bridge 50, 51, 52, 53 that connects UART0 to VCOM0, and it also doesn't work...

I even change between ASYNC using DMA and Interrupt and none of it work. UART1 is working fine though. But not UART0 no matter what I did.

CONFIG_UART_INTERRUPT_DRIVEN=y
CONFIG_UART_0_ASYNC=n
CONFIG_UART_0_INTERRUPT_DRIVEN=y

I ran out of option and don't really know what to do.