Is USB ACM serial DFU possible with ncs v2.6.1?

[khett-ptd]

I am building a BLE audio product that is based on the nRF5340. The application is largely based on the nrf5340_audio ncs example application, but I am modifying the the application so it can be re-configured at run-time instead of using the KConfig #defines.

I started the development of this application using ncs v1.9.x and with each ncs release I have been merging the changes to the ncs nrf5340_audio example into my application so I can keep up with bug fixes and general improvements in the build system.

For this product I have disabled DFU over Bluetooth and enabled DFU over USB serial (ACM com port).  I used to be able to enable the DFU mode by providing the "-m internal" flag to the python buildprog.py script, but in ncs v2.6.1 it looks like that is no longer an option because the new system has switched to using the softdevice Bluetooth controller.

To work around this, I added "CONFIG_AUDIO_DFU=1" to my proj.conf file.  When I compile the system with this option enabled I now get the build error shown in the attached .zip file.  For some reason if I insert the console output into this support ticket, the system will not allow me to post the message.

At first I thought this was because I have added code to my application and it would no longer fit into flash, but after looking into this some more, I now think that the image that does not fit in flash is the "new" softdevice BT controller.  Is this correct?

Is there a way to use the new softdevice controller and enable USB serial DFU in ncs v2.6.1?

Best Regards,
Kristopher

ConsoleOutputErrorMessage.zip

[AHaug]

Hi Kristopher,

Is there a way to use the new softdevice controller and enable USB serial DFU in ncs v2.6.1?

Yes, you can use both BLE for audio and FOTA and Serial DFU simultaneous that version of NCS, and they should be simple prj.config additions to get into your project, but I will have to look some closer into the buildprog script to see how that works w.r.t enablign this.

Could you share your relevant prj.conf, child_image/mcuboot.conf and similar?

Kind regards,
Andreas

[khett-ptd]

Hi Andreas,

I did some more investigation and discovered that the hci_ipc.conf file in the "child_image" directory was calling for the BT controller to be built with support for CENTRAL, PERIPHERAL, and BROADCASTER enabled.  I tried re-configuring my project to build using only the features required to be a BLE Audio BROADCASTER.  This greatly reduces the required flash memory, but my project still overflows the flash by about 3.5k.

My hardware has an external QSPI flash connected to the nRF5340.  I'm wondering if I could program the bootloader into the on-chip flash, have the bootloader write the application images into the external flash and then execute the program directly from the external flash using QSPI XIP.  Do you think this is possible using the zephyr system?  If so, are there any examples I can work from?

Best Regards,
Kristopher

[AHaug]

Hi Kristopher,

Could you clarify if the 3.5k flash overflow is with or without a secondary app bank on the internal flash? I'll continue the answer below assuming that it is on the internal flash.

The short answer is "no, not the way you describe it", the longer answer is that you can add some parts of the application on the external flash and run it from the external flash over QSPI, but executing code from external flash will both have an impact on the complexity of the project w.r.t you need to manage all entrypoints to your external flash. Why, you ask? Executing code over QSPI from the external flash is slower than running code on the internal flash, meaning that you can't place anything timing sensitive on the external flash or you can't place dependencies of timing critical code on the external flash. This includes for instance almost everything that has to do with BLE. Executing code from external flash is also more taxing on current consumption, so it will impact the battery life on your device (given that it is a battery operated device). This is just a quick summary, but it should cover the major pitfalls w.r.t executing code from an exeternal flash. If you however end up with this approach, then you can have a look at https://docs.nordicsemi.com/bundle/ncs-latest/page/nrf/app_dev/device_guides/nrf53/qspi_xip_guide_nrf5340.html 

What I instead would recommend that you do is to place your mcuboot secondary_application partition (the partition where you temporarily store the new update image) on the external flash and only do firmware updates over QSPI instead of running parts of your from external flash. Combine this exercise https://academy.nordicsemi.com/courses/nrf-connect-sdk-intermediate/lessons/lesson-8-bootloaders-and-dfu-fota/topic/exercise-2-dfu-over-usb-adding-external-flash/ with exercise 3 in the same course for instance.

Let me know if this answers your questions or if you have any additional follow ups

Kind regards,
Andreas

[khett-ptd]

Hi Andreas,

Thanks for the information about some of the issues with trying to run the entire application out of external flash.  I have been trying to configure my application to work in the way you suggest where the bootloader uses the external flash for the secondary "update" images, but I am still running into a linker issue.

Here's a screen capture of the build output that shows the error.  This error seems strange to me because the "Memory usage" indicates that the FLASH memory region is only 63% full, but then a few lines later there is an error message stating that the FLASH memory region has overflowed by 18688 bytes.

I am not sure where the FLASH memory region is defined.  In the "build/dev_gateway/build_release" directory I found a file named "regions.yml", but this file has no region named "FLASH".  Do you know which part of the system defines this "FLASH" memory region?

I added a status message to the top-level CMakeLists.txt file to verify that the CONFIG_AUDIO_DFU value is set to "2", which then causes the build system to include the mcuboot config and the pm_dfu_external_flash.yml partition files.

Here are my project configurations.

In dfu/conf/overlay-mcuboot.conf:

CONFIG_SIZE_OPTIMIZATIONS=y
CONFIG_MAIN_STACK_SIZE=20480
CONFIG_MBEDTLS_CFG_FILE="mcuboot-mbedtls-cfg.h"
CONFIG_PCD_APP=y
# Enable mcuboot serial over USB CDC port
CONFIG_CBPRINTF_NANO=y
CONFIG_LOG=y
CONFIG_UART_CONSOLE=y
CONFIG_MCUBOOT_SERIAL=y
CONFIG_BOOT_SERIAL_CDC_ACM=y

in child_image/hci_ipc_release.conf:

#####
# Features for unicast client (Cellphone type device)
CONFIG_BT_ISO_CENTRAL=y

#####
# Features for unicast server (earbud type device)
CONFIG_BT_ISO_PERIPHERAL=y

#####
# Features for broadcast source
CONFIG_BT_ISO_BROADCASTER=y

#####
# Features for broadcast receiver
CONFIG_BT_ISO_SYNC_RECEIVER=y


CONFIG_BT_EXT_ADV=y
CONFIG_BT_PER_ADV_SYNC_TRANSFER_RECEIVER=y
CONFIG_BT_PER_ADV_SYNC_TRANSFER_SENDER=y

CONFIG_BT_CTLR_CONN_ISO_GROUPS=1
CONFIG_BT_CTLR_CONN_ISO_STREAMS=2
CONFIG_BT_CTLR_SYNC_ISO_STREAM_COUNT=2
CONFIG_BT_CTLR_ADV_ISO_SET=1
CONFIG_BT_CTLR_ADV_ISO_STREAM_COUNT=2

# Support two headsets, each with one source and one sink stream
CONFIG_BT_MAX_CONN=4
CONFIG_BT_CTLR_SDC_PERIPHERAL_COUNT=1

# Allow using more than default advertising event length
CONFIG_BT_CTLR_ADV_DATA_LEN_MAX=251

# To present the audio at the right point in time, we need the controller and
# audio clock to be synchronized
CONFIG_MPSL_TRIGGER_IPC_TASK_ON_RTC_START=y
CONFIG_MPSL_TRIGGER_IPC_TASK_ON_RTC_START_CHANNEL=4

In the top level prj_release.conf:

#####
# Items added for UPF
CONFIG_BT_AUDIO_CODEC_CAP_MAX_METADATA_SIZE=30
CONFIG_BT_AUDIO_CODEC_CAP_MAX_DATA_SIZE=120
CONFIG_BT_AUDIO_CODEC_CFG_MAX_METADATA_SIZE=80
CONFIG_BT_BAP_UNICAST_CLIENT_ASE_SNK_COUNT=2
CONFIG_NRF_APPROTECT_USE_UICR=n
CONFIG_BT_L2CAP_TX_MTU=128

#####
# nRF5340 Audio
CONFIG_NRF5340_AUDIO=y
CONFIG_AUDIO_SOURCE_USB=y

# HEADSET = 1, GATEWAY = 2
CONFIG_AUDIO_DEV=2

# The device to scan for is now set in the PtdSettings struct
#CONFIG_BT_DEVICE_NAME="INZONE Buds"
#CONFIG_BT_DEVICE_NAME="Galaxy Buds2 Pro"

#CONFIG_BT_MAX_CONN=2
#CONFIG_BT_BAP_UNICAST_CLIENT_ASE_SNK_COUNT=2

# The Sony WF-1000XM5 earbuds require more than the default 5 codec capabilities
CONFIG_CODEC_CAP_COUNT_MAX=12

#####
# We need to enable the STREAM_BIDIRECTIONAL option so the zephyr code will
# compile with this feature enabled.  The g_settings.biDirAudio flag will
# be used in the PTD app to decide what to do at run time.
CONFIG_STREAM_BIDIRECTIONAL=y

#####
# Sample Rate Converter items
# Requied for bi-dir audio with USB because the system 
# uses 24 kHz for both source and sink streams.
CONFIG_SAMPLE_RATE_CONVERTER=y
CONFIG_SAMPLE_RATE_CONVERTER_FILTER_SIMPLE=y


#####
# Debug items

#####
# General
CONFIG_DEBUG=n
CONFIG_DEBUG_INFO=n
CONFIG_WDT_CTLR=y
CONFIG_ASSERT=n
CONFIG_STACK_USAGE=n
CONFIG_THREAD_MONITOR=n
CONFIG_THREAD_NAME=y
CONFIG_THREAD_RUNTIME_STATS=n
CONFIG_THREAD_RUNTIME_STATS_USE_TIMING_FUNCTIONS=n
CONFIG_STACK_SENTINEL=y
CONFIG_INIT_STACKS=y
CONFIG_STATS=n
CONFIG_STATS_SHELL=n

#####
# I2C Eeprom support
CONFIG_I2C=y
CONFIG_EEPROM=y
CONFIG_EEPROM_AT24=y
CONFIG_EEPROM_SHELL=y

#####
# Settings
CONFIG_SETTINGS=y
CONFIG_BT_SETTINGS=y
CONFIG_SETTINGS_RUNTIME=y
CONFIG_SETTINGS_SHELL=y

#####
# Enable USB features
CONFIG_USB_DEVICE_STACK=y
CONFIG_USB_COMPOSITE_DEVICE=y
CONFIG_USB_DEVICE_INITIALIZE_AT_BOOT=n
CONFIG_USB_DEVICE_MANUFACTURER="Portland Tool And Die"
CONFIG_USB_DEVICE_PRODUCT="PTD BQC-4150"
CONFIG_USB_DEVICE_VID=0x1915
CONFIG_USB_DEVICE_PID=0xEA9C

#####
# Enable USB CDC uart support
CONFIG_USB_CDC_ACM=y
CONFIG_UART_LINE_CTRL=y
CONFIG_UART_INTERRUPT_DRIVEN=y

# Uart driver
#CONFIG_SERIAL=y

#####
# Logging
CONFIG_LOG=y
CONFIG_NEWLIB_LIBC_FLOAT_PRINTF=y
CONFIG_LOG_TAG_MAX_LEN=2
CONFIG_LOG_TAG_DEFAULT="--"
CONFIG_PRINTK=y
# Don't mix the log output with the shell uart IO
CONFIG_LOG_BACKEND_UART=y
CONFIG_SHELL_LOG_BACKEND=n
#CONFIG_LOG_DEFAULT_LEVEL=4

# Console related defines
CONFIG_CONSOLE=n
CONFIG_UART_CONSOLE=n

# Shell related defines
CONFIG_BOOT_BANNER=n
CONFIG_SHELL=y
CONFIG_KERNEL_SHELL=y
CONFIG_BT_SHELL=n
CONFIG_BT_TESTING=n
CONFIG_USE_SEGGER_RTT=n
## Disable logs on RTT
CONFIG_SHELL_RTT_INIT_LOG_LEVEL_NONE=n
CONFIG_SHELL_BACKEND_RTT=n
CONFIG_SHELL_BACKEND_SERIAL=y
CONFIG_SHELL_BACKEND_SERIAL_CHECK_DTR=y
#CONFIG_SHELL_BACKEND_SERIAL_INIT_PRIORITY=51
CONFIG_SHELL_VT100_COLORS=n
CONFIG_SHELL_STACK_SIZE=2048
CONFIG_SHELL_CMD_BUFF_SIZE=1024
## Reduce shell memory usage
CONFIG_SHELL_WILDCARD=n
CONFIG_SHELL_HELP_ON_WRONG_ARGUMENT_COUNT=n
CONFIG_SHELL_STATS=n
CONFIG_SHELL_CMDS=n
CONFIG_SHELL_HISTORY=n
CONFIG_SHELL_PROMPT_UART="BQC-4150>"

# Turn off default shell commands
CONFIG_I2C_SHELL=n
#CONFIG_HWINFO_SHELL=n
CONFIG_CLOCK_CONTROL_NRF_SHELL=n
CONFIG_FLASH_SHELL=n
CONFIG_DEVICE_SHELL=n

# Suppress err msg from sd_check_card_type. Because SPI_SDHC has no card presence method,
# assume card is in slot. Thus error message is always shown if card is not inserted
CONFIG_SD_LOG_LEVEL_OFF=y

#####
# Serial boot loader items
# Enable bootloader framework
# 1 = internal flash, 2 = external flash
CONFIG_AUDIO_DFU=2
#CONFIG_B0N_MINIMAL=y
#CONFIG_BOOTLOADER_MCUBOOT=y
#CONFIG_PM_SINGLE_IMAGE=y

####
# BT security features
# SMP = Security Management Protocol
CONFIG_BT_SMP=y
# If CONFIG_BT_PRIVACY is no, then the system should use public addressees, otherwise RPA.
CONFIG_BT_PRIVACY=n
CONFIG_BT_BONDABLE=y
# we do not want to enforce MITM as this requires authenticated pairing
CONFIG_BT_SMP_ENFORCE_MITM=n
# we want to overwrite old bonds from the same device (optional, security risk)
CONFIG_BT_SMP_ALLOW_UNAUTH_OVERWRITE=n
CONFIG_BT_ID_UNPAIR_MATCHING_BONDS=n
# Enable CONFIG_BT_LOG_SNIFFER_INFO to have the shell print the link keys on pairing events
CONFIG_BT_LOG_SNIFFER_INFO=n

#####
#####
# BLE audio features for unicast client
#####
#####
# MPL = Media Player
CONFIG_BT_MPL=y
# MCS = Media Control Service
CONFIG_BT_MCS=y
# VCP = Volume Control Profile
CONFIG_BT_VCP_VOL_CTLR=y
CONFIG_BT_BAP_UNICAST_CONFIGURABLE=y

#####
#####
# BLE audio features for broadcast source (cellphone type device)
#####
#####
CONFIG_AURACAST=y
# Need to use the CONFIG_BT_AUDIO_BROADCAST_CONFIGURABLE to support 48kHz audio from USB
CONFIG_BT_AUDIO_BROADCAST_CONFIGURABLE=y
CONFIG_BT_AUDIO_BITRATE_BROADCAST_SRC=96000
#CONFIG_BT_BAP_BROADCAST_16_2_2=n
#CONFIG_BT_BAP_BROADCAST_24_2_1=n
CONFIG_BT_DEVICE_NAME="PTD SOURCE"
CONFIG_BT_AUDIO_BROADCAST_NAME="PTD SOURCE"
#CONFIG_BT_AUDIO_BROADCAST_NAME_ALT="PTD 2"
#CONFIG_BT_AUDIO_BROADCAST_ENCRYPTED=n
#CONFIG_BT_AUDIO_BROADCAST_ENCRYPTION_KEY="PTD_KEY"

In my custom board file nrf5340_cpuapp_common.dts:

/ {
	chosen {
		zephyr,shell-uart = &cdc_acm_uart0;	
		zephyr,uart-mcumgr = &cdc_acm_uart0;
		zephyr,console = &cdc_acm_uart1;		
		zephyr,bt-mon-uart = &cdc_acm_uart1;
		zephyr,bt-c2h-uart = &cdc_acm_uart1;
		zephyr,bt-hci-ipc = &ipc0;
		watchdog0 = &wdt0;
		nordic,pm-ext-flash = &mx25r64;
	};
	......
	&spi4 {
	compatible = "nordic,nrf-spim";
	status = "okay";
	cs-gpios = <&gpio0 11 GPIO_ACTIVE_LOW>, <&gpio0 17 GPIO_ACTIVE_LOW>;
	pinctrl-0 = <&spi4_default>;
	pinctrl-1 = <&spi4_sleep>;
	pinctrl-names = "default", "sleep";
	sdhc0: sdhc@0 {
		compatible = "zephyr,sdhc-spi-slot";
		reg = <0>;
		status = "okay";
		sdmmc {
			compatible = "zephyr,sdmmc-disk";
			status = "okay";
		};

		spi-max-frequency = <8000000>;
	};

	mx25r64: mx25r6435f@2 {
		compatible = "jedec,spi-nor";
		reg = <2>;
		spi-max-frequency = <8000000>;

		jedec-id = [c2 28 17];
		sfdp-bfp = [
			e5 20 f1 ff  ff ff ff 03  44 eb 08 6b  08 3b 04 bb
			ee ff ff ff  ff ff 00 ff  ff ff 00 ff  0c 20 0f 52
			10 d8 00 ff  23 72 f5 00  82 ed 04 cc  44 83 68 44
			30 b0 30 b0  f7 c4 d5 5c  00 be 29 ff  f0 d0 ff ff
		];
		size = <67108864>;
		has-dpd;
		t-enter-dpd = <10000>;
		t-exit-dpd = <5000>;
		dpd-wakeup-sequence = <30000 20 45000>;
	};
};

The strange thing about this error message is that the FLASH overflows by the same amount if I set CONFIG_AUDIO_DFU to "1", which I would think would overflow the internal flash by a much larger amount than if the secondary images are being re-directed to the external flash.

Any ideas of what to check next?

Best Regards,
Kristopher

[AHaug]

Hi,

Were you able to perform DFU from external flash with a minimal sample, for instance the one in the lessons I sent on your custom hardware? I strongly recommend doing this with a minimal sample before porting to a full scale application. Or try to apply the lesson 3 steps to the peripheral UART sample found in the SDK.

Lets assume you've succeeded in that and go on in the meanwhile:

khett-ptd said:

I am not sure where the FLASH memory region is defined.  In the "build/dev_gateway/build_release" directory I found a file named "regions.yml", but this file has no region named "FLASH".  Do you know which part of the system defines this "FLASH" memory region?

Within the build folder I would expect that you find a "partitions.yml" within the first level of the build folder, i.e build/partitions.yml This is the generated partitioning map for that build. It should look something like this (The Matter Weather station project which showcases DFU over external flash):

app:
  address: 0x10200
  end_address: 0xf0000
  region: flash_primary
  size: 0xdfe00
external_flash:
  address: 0x120000
  device: MX25R64
  end_address: 0x800000
  region: external_flash
  size: 0x6e0000
mcuboot:
  address: 0x0
  end_address: 0x10000
  region: flash_primary
  size: 0x10000
mcuboot_pad:
  address: 0x10000
  end_address: 0x10200
  region: flash_primary
  size: 0x200
mcuboot_primary:
  address: 0x10000
  end_address: 0xf0000
  orig_span: &id001
  - mcuboot_pad
  - app
  region: flash_primary
  size: 0xe0000
  span: *id001
mcuboot_primary_1:
  address: 0x0
  device: flash_ctrl
  end_address: 0x40000
  region: ram_flash
  size: 0x40000
mcuboot_primary_app:
  address: 0x10200
  end_address: 0xf0000
  orig_span: &id002
  - app
  region: flash_primary
  size: 0xdfe00
  span: *id002
mcuboot_secondary:
  address: 0x0
  device: MX25R64
  end_address: 0xe0000
  region: external_flash
  size: 0xe0000
mcuboot_secondary_1:
  address: 0xe0000
  device: MX25R64
  end_address: 0x120000
  region: external_flash
  size: 0x40000
otp:
  address: 0xff8100
  end_address: 0xff83fc
  region: otp
  size: 0x2fc
pcd_sram:
  address: 0x20000000
  end_address: 0x20002000
  region: sram_primary
  size: 0x2000
ram_flash:
  address: 0x40000
  end_address: 0x40000
  region: ram_flash
  size: 0x0
rpmsg_nrf53_sram:
  address: 0x20070000
  end_address: 0x20080000
  placement:
    before:
    - end
  region: sram_primary
  size: 0x10000
settings_storage:
  address: 0xf0000
  end_address: 0x100000
  region: flash_primary
  size: 0x10000
sram_primary:
  address: 0x20002000
  end_address: 0x20070000
  region: sram_primary
  size: 0x6e000

khett-ptd said:

I added a status message to the top-level CMakeLists.txt file to verify that the CONFIG_AUDIO_DFU value is set to "2", which then causes the build system to include the mcuboot config and the pm_dfu_external_flash.yml partition files.

To clarify, I've only seen the CONFIG_MCUBOOT_SERIAL=y present (which is what they use in lesson 2 I sent, but not in lesson 3). Is your end goal to set up only serial DFU, BLE FOTA or both of them?

Is CONFIG_PM_SINGLE_IMAGE=y from a sample, course or something you've set up your self? I recommend that you do simultaneous DFU instead of one and one image so that you don't risk breaking the HCI interface between the app-core and the net-core due to versioning incompatibility. 

khett-ptd said:

I am not sure where the FLASH memory region is defined. 

In NCS 2.6.x (or anything below 2.7.0), Flash memory region is set either dynamically or statically. If you don't have a pm_static.yml within your project folder, then the build system will build everything dynamically and try to place things roughly where it should be. For complex applications using multiple subsystems, for instance BLE which requires a settings partition, the SD card config which requires a NVS partition, external flash DFU which requires a secondary app partition and MCUboot (and b0 if you have updatable bootloader), we recommend you statically set up your own partition using the pm_static.yml so that you have full control over the placements both now and in the future if you need to do an update.

khett-ptd said:

The strange thing about this error message is that the FLASH overflows by the same amount if I set CONFIG_AUDIO_DFU to "1", which I would think would overflow the internal flash by a much larger amount than if the secondary images are being re-directed to the external flash.

I agree. My guess is that it has not been set up to use external flash, but I can't be sure since I don't know what CONFIG_AUDIO_DFU=2 does. Is this something you could share? If so, I'll have a look

Kind regards,
Andreas

[khett-ptd]

Hi Andreas,

Apologies for my slow response.  I have been able to get the system to fit into the on-chip flash with mcuboot by disabling some of the Bluetooth controller features listed in the child_image/hci_image/*.conf files.  

Now that the code compiles and links, it does not boot correctly, but that is a different issue than the question posed in this support ticket.  I think we should consider this issue closed and I will submit a new issue about debugging the mcuboot process.

Thanks for your help,
-- Kristopher

[AHaug]

Hi,

khett-ptd said:

Apologies for my slow response.

No worries

khett-ptd said:

have been able to get the system to fit into the on-chip flash with mcuboot by disabling some of the Bluetooth controller features listed in the child_image/hci_image/*.conf files.  

Glad to hear that!

khett-ptd said:

Now that the code compiles and links, it does not boot correctly, but that is a different issue than the question posed in this support ticket.  I think we should consider this issue closed and I will submit a new issue about debugging the mcuboot process.

Noted, I'll mark this case as closed for now, but you can always reopen it if you have any additional questions related to what we've discussed in this ticket already.

Kind regards,
Andreas