MCUBoot DFU Interference with ExFat Filesystem

RE: MCUBoot DFU Interference with ExFat Filesystem

FinnBiggs — Tue, 16 Jul 2024 01:11:33 GMT

Hi Sigurd,

Thank you so much, great find on that solution. You're right that that solves the USB sample.

In a bizarre twist, I sunk my entire workday into trying to get my GitHub repo to build and compile and ended up in a strange state where I was getting very opaque assembler errors.

[55/180] Building C object zephyr/CMakeFiles/zephyr.dir/misc/generated/configs.c.obj
FAILED: zephyr/CMakeFiles/zephyr.dir/misc/generated/configs.c.obj 
C:\ncs\toolchains\c57af46cb7\opt\zephyr-sdk\arm-zephyr-eabi\bin\arm-zephyr-eabi-gcc.exe -DKERNEL -DNRF52840_XXAA -DPICOLIBC_INTEGER_PRINTF_SCANF -DUSE_PARTITION_MANAGER=1 -D_FORTIFY_SOURCE=1 -D_POSIX_C_SOURCE=200809 -D__LINUX_ERRNO_EXTENSIONS__ -D__PROGRAM_START -D__ZEPHYR__=1 -IC:/ncs/v2.5.0/zephyr/kernel/include -IC:/ncs/v2.5.0/zephyr/arch/arm/include -IC:/ncs/v2.5.0/zephyr/include -Izephyr/include/generated -IC:/ncs/v2.5.0/zephyr/soc/arm/nordic_nrf/nrf52 -IC:/ncs/v2.5.0/zephyr/soc/arm/nordic_nrf/common/. -IC:/ncs/v2.5.0/zephyr/subsys/usb/device -IC:/ncs/v2.5.0/nrf/include -IC:/ncs/v2.5.0/nrf/tests/include -IC:/ncs/v2.5.0/modules/hal/cmsis/CMSIS/Core/Include -IC:/ncs/v2.5.0/zephyr/modules/cmsis/. -IC:/ncs/v2.5.0/modules/hal/nordic/nrfx -IC:/ncs/v2.5.0/modules/hal/nordic/nrfx/drivers/include -IC:/ncs/v2.5.0/modules/hal/nordic/nrfx/mdk -IC:/ncs/v2.5.0/zephyr/modules/hal_nordic/nrfx/. -IC:/ncs/v2.5.0/modules/debug/segger/SEGGER -IC:/ncs/v2.5.0/modules/debug/segger/Config -isystem C:/ncs/v2.5.0/nrfxlib/crypto/nrf_cc310_platform/include -fno-strict-aliasing -Os -imacros C:/Users/finnb/Documents/auli/MCUBoot_ExFat/build/zephyr/include/generated/autoconf.h -fno-printf-return-value -fno-common -g -gdwarf-4 -fdiagnostics-color=always -mcpu=cortex-m4 -mthumb -mabi=aapcs -mfp16-format=ieee --sysroot=C:/ncs/toolchains/c57af46cb7/opt/zephyr-sdk/arm-zephyr-eabi/arm-zephyr-eabi -imacros C:/ncs/v2.5.0/zephyr/include/zephyr/toolchain/zephyr_stdint.h -Wall -Wformat -Wformat-security -Wno-format-zero-length -Wno-pointer-sign -Wpointer-arith -Wexpansion-to-defined -Wno-unused-but-set-variable -Werror=implicit-int -fno-pic -fno-pie -fno-asynchronous-unwind-tables -ftls-model=local-exec -fno-reorder-functions --param=min-pagesize=0 -fno-defer-pop -fmacro-prefix-map=C:/Users/finnb/Documents/auli/MCUBoot_ExFat=CMAKE_SOURCE_DIR -fmacro-prefix-map=C:/ncs/v2.5.0/zephyr=ZEPHYR_BASE -fmacro-prefix-map=C:/ncs/v2.5.0=WEST_TOPDIR -ffunction-sections -fdata-sections --specs=picolibc.specs -std=c99 -MD -MT zephyr/CMakeFiles/zephyr.dir/misc/generated/configs.c.obj -MF zephyr\CMakeFiles\zephyr.dir\misc\generated\configs.c.obj.d -o zephyr/CMakeFiles/zephyr.dir/misc/generated/configs.c.obj -c zephyr/misc/generated/configs.c
C:\Users\finnb\AppData\Local\Temp\ccu25E1Q.s: Assembler messages:
C:\Users\finnb\AppData\Local\Temp\ccu25E1Q.s:43: Error: missing expression
[76/180] Building C object zephyr/CMakeFiles/zephyr.dir/lib/os/cbprintf_complete.c.obj
ninja: build stopped: subcommand failed

Luckily, I've been saved by a coworker, who tried this out by starting from a clean mass sample and made it simply work using your suggestions.

Here is a link to a solution. (https://github.com/FinnWBiggs/mass_pm). To any and all future readers: please feel free to do anything you like with it. .

I've added a warning to the malfunctioning ReadMe of that repo to let future viewers know, and will leave both repos up so that future viewers of this post still have a reference.

If you look at what's in my included error or in the repo (https://github.com/FinnWBiggs/MCUBoot_ExFat) and see an obvious problem, I'd be happy to know why I'm getting assembler errors.

If it looks complicated to ID a problem, then let's just close this ticket -- I'm able to continue on my project.

Regardless, I'm very glad to have a working filesystem, and I think we can close this ticket. Thank you Sigurd, you've been amazingly helpful. I hope you had a lovely and relaxing vacation.

Best regards,

- Finn

RE: MCUBoot DFU Interference with ExFat Filesystem

Sigurd Hellesvik — Mon, 15 Jul 2024 10:54:40 GMT

I think this should solve the USB mass sample (with a workaround. There seems to be a bug in partititon manager)
Bug in partition_manager_output.py leading to incorrect PM_FOREACH_AFFILIATED_TO_disk

Can you give it a go and let me know what you think?

RE: MCUBoot DFU Interference with ExFat Filesystem

Sigurd Hellesvik — Mon, 15 Jul 2024 09:15:01 GMT

I have been on vacation the last two weeks, but I am back now.
Seems like this ticket has not been picket up in the meantime in the summer rush.

[quote user="FinnBiggs"]

In the process of trying to debug and come up with a minimal example that narrows the scope of this problem, I went back to the mass sample.

I believe it may be a valid minimal example add CONFIG_PM_SINGLE_IMAGE to zephyr\samples\subsys\usb\mass sample.

How can I enable the partition manager in this sample?

[/quote]

I think this is a good plan. I will try to make this example work with the partition manager on my side.

By the way, another customer recently also asked about this in USB MASS sample and partition manager build issues

RE: MCUBoot DFU Interference with ExFat Filesystem

FinnBiggs — Thu, 04 Jul 2024 00:26:00 GMT

Hi Sigurd,

Thanks for the kind words. I've made a little bit of progress here, but mostly I've been on other issues. After devoting my whole workday to this issue I've decided to come back for more help.

[quote userid="106736" url="~/f/nordic-q-a/112280/mcuboot-dfu-interference-with-exfat-filesystem/490757"]Or are you considering using another file-system than the one you try to handle now altogether?
I do not think I understand the question fully.[/quote]

At this point I'm open to any solution that meets my two needs -- DFU OTA that works alongside a USB File System. If there's a way to those features that uses some different Zephyr tool set, I'd be happy to swap.

In this thread, we've boiled down the problem to some conflict between partition manager and flash.

In the ReadMe here: https://github.com/FinnWBiggs/MCUBoot_ExFat, I lay out build instructions, and the config options which lead to the error.

----------------------------------------------------------

In the process of trying to debug and come up with a minimal example that narrows the scope of this problem, I went back to the mass sample.

I believe it may be a valid minimal example add CONFIG_PM_SINGLE_IMAGE to zephyr\samples\subsys\usb\mass sample.

How can I enable the partition manager in this sample?

Thanks again for the help.

Best regards,

- Finn

RE: MCUBoot DFU Interference with ExFat Filesystem

Sigurd Hellesvik — Wed, 26 Jun 2024 06:50:51 GMT

[quote user="FinnBiggs"]I added pm.yml[/quote]

No wait.
pm.yml should not be parsed by user applications. This file is specifically only used for the build system, and not to be used by applications. Applications use pm_static.yml.

[quote user="FinnBiggs"](The root application does not need to define a pm.yml file, because its partition size and placement is implied by the size and placement of the child image partitions. If a root application defines a pm.yml file, it is silently ignored.)[/quote]

Ah yes I see you already learned what I said above.

[quote user="FinnBiggs"]What's the correct way to implement a partition manager while PM_SINGLE_IMAGE is enabled?[/quote]

Just configure the already enabled partition manager with pm_static.yml.

[quote user="FinnBiggs"]What do you recommend for a DFU-compatible Filesystem?[/quote]

Let us fix the file system alone first and then add DFU?
Or are you considering using another file-system than the one you try to handle now altogether?
I do not think I understand the question fully.

[quote user="FinnBiggs"]What would be the right resource to go learn about these tools so I don't eat up time on DevZone?[/quote]

All I got are these:
https://academy.nordicsemi.com/courses/nrf-connect-sdk-intermediate/lessons/lesson-8-bootloaders-and-dfu-fota/
https://developer.nordicsemi.com/nRF_Connect_SDK/doc/latest/nrf/scripts/partition_manager/partition_manager.html
https://developer.nordicsemi.com/nRF_Connect_SDK/doc/latest/nrf/config_and_build/multi_image.html

However, this mostly just requires experience, and this stuff is complicated.
I think that you can still continue to ask here on DevZone.
That being said, summer vacations will start here from next week, so we will be a bit low on staff in the following weeks.

[quote user="FinnBiggs"]Apologies for not being as helpful as I'd like to be, I'm new to this and feeling lost.[/quote]

No worries at all, this is not easy, and we know it.
We should be making it easier, and I hope that we will in the future, but for now we use tech support as a way to make it easier for you guys to use. (At least that is how I see it).
I think that you are doing a great job, both in trying to learn this and in being friendly!

Alright, now let me try to explain how I would do this:

Create the following pm_static.yml:

fatfs_storage:
  address: 0xf0000
  end_address: 0x100000
  region: flash_primary
  size: 0x10000
  affiliation: disk
  extra_params: {
      disk_name: "SD",
      disk_cache_size: 4096,
      disk_sector_size: 512,
      disk_read_only: 0
  }

You may want to change the region to your external storage, and the address range to either the start or end of that region.

I also used an arbitrary size, so maybe tweak that a bit.

When you build with this, you should be able to verify that it was added to partitioning by looking at build/partitioning.yml, which contains the final partitioning for the project.

Now that the partition does have the affiliation "disk", you should hopefully no longer get the error PM_FOREACH_SOMETHING_SOMETHING_disk that I was getting when building the project.

Let me know how this goes.

RE: MCUBoot DFU Interference with ExFat Filesystem

FinnBiggs — Wed, 26 Jun 2024 00:03:23 GMT

Hi Sigurd,

I appreciate the pointers on partition management, and the simplification of CONFIG_PM_SINGLE_IMAGE.

Just to keep things up to date, here's how it's going.

I transitioned this project from static to dynamic partitioning.

I removed pm_static.yml

I added pm.yml

Result: memory report no longer contains a partitions tab

I believe this is due to the note here (https://docs.nordicsemi.com/bundle/ncs-latest/page/nrf/scripts/partition_manager/partition_manager.html#configuration)

(The root application does not need to define a pm.yml file, because its partition size and placement is implied by the size and placement of the child image partitions. If a root application defines a pm.yml file, it is silently ignored.)

Next, I define a child image using the multi-image build tooling.

add configuration/nrf52840dk_nrf52840/child_image.

I'm concerned about this on the grounds that PM_CONFIG_SINGLE_IMAGE seems to conflict with adding child images.

I think maybe the best way forward would be for me to take a step back and ask for more general guidance.

What's the correct way to implement a partition manager while PM_SINGLE_IMAGE is enabled?

What do you recommend for a DFU-compatible Filesystem?

What would be the right resource to go learn about these tools so I don't eat up time on DevZone?

Apologies for not being as helpful as I'd like to be, I'm new to this and feeling lost.

Thank you very much for any help you can provide.

All the best,

- Finn

RE: MCUBoot DFU Interference with ExFat Filesystem

Sigurd Hellesvik — Mon, 24 Jun 2024 14:09:36 GMT

I have not solved it yet, but I were able to build your project, and i have found some things that could help.

First some basic partitioning:
You do not need to define your whole partitioning scheme. Just place your partition somewhere, and the partition manager will try to partition around it. Easiest if you put yours in an end.

And, try to build with CONFIG_PM_SINGLE_IMAGE=y instead of MCUboot. This way, you get the same error and do not use MCUboot. This is good because it proves that the issue is specifically partition manager and not MCUboot, which narrows it down.

Then for this specific error:

Just under where one of the errors point to, I see this:
https://github.com/nrfconnect/sdk-zephyr/blob/2e2523efe52a7ac89f0567b8798fd857b1e71ae3/drivers/disk/flashdisk.c#L486-L503

Looking into this, i found something interresting in Partition Manager docs:

fatfs_storage:
    affiliation: disk
    extra_params: {
        disk_name: "SD",
        disk_cache_size: 4096,
        disk_sector_size: 512,
        disk_read_only: 0
    }
    placement:
        before: [end]
        align: {start: 4096}
    inside: [nonsecure_storage]
    size: 65536

I suspect that your partition may need such an affiliation and extra parameters.
I tried to do this for your example, and it did not work first try, so I suggest that you do some more testing with this info as well, and let me know what you find

RE: MCUBoot DFU Interference with ExFat Filesystem

FinnBiggs — Fri, 21 Jun 2024 18:53:25 GMT

Hi Sigurd,

My mistake on the build instructions! I wrote up build instructions for the first version, but didn't add any to this version.

Thanks for sticking with me.

I have added build instructions to the ReadMe.

github.com/.../MCUBoot_ExFat

------------------

I've been working on this for a few hours now. Good news and bad news.

Good news!

I've added a pm_static.yml to configure the partition management. This solves the problem where the PM_some_storage_partition_ID was undefined. I can set up a file system using this partition.

Bad news

The new system, which contains a pm_static.yml recreates my initial problem -- flashdisk driver problems.

------------------

How do you advise I make these compatible?

Best,

- Finn

RE: MCUBoot DFU Interference with ExFat Filesystem

Sigurd Hellesvik — Fri, 21 Jun 2024 11:48:33 GMT

Sweet, having a smaller sample to look at is always good!

I have to be a bit difficult out of principle:
Please give me SDK version and build instructions (command/board config is enough, if nothing else is needed). Then, I can test the project.

[quote user="FinnBiggs"]When I enable MCUBoot, I get errors regarding the definition of PM_some_storage_partition_ID, which implies that MCUBoot is somehow affecting my devicetree's storage partitioning IDs. [/quote]

MCUboot will enable the Partition Manager, which will override DTS partitioning. See this lesson I wrote.

If you set this, which you have not, MCUboot will automatically use the external flash for its partitions. Other file systems can also be directed to external flash in this way by Kconfig, for example CONFIG_PM_PARTITION_REGION_NVS_STORAGE_EXTERNAL. See pm.yml.nvs for how our build system uses it to define partitions.

For file systems specifically:

These lines can be frustrating when working with file systems. As you can see they map different stuff to "storage", which may cause conflicts. So be aware of that

RE: MCUBoot DFU Interference with ExFat Filesystem

FinnBiggs — Thu, 20 Jun 2024 20:05:33 GMT

Hi Sigurd,

I'll set up a minimized version and update here.

Update:

Here is a minimized sample causing a collision between ExFat and MCUBoot.

https://github.com/FinnWBiggs/MCUBoot_ExFat

I do get a different error than I did in mini_desktop, but that's okay. I believe that if we can find where the two tools conflict, it will tell me enough about the system that I can follow the same logic to get my project working.

When MCUBoot is disabled, my application finds my devicetree's "some_storage_partition" (whose name I changed to avoid potential naming and defines, such as in flash_map_pm).The sample runs and sets up a USB storage system as intended.

When I enable MCUBoot, I get errors regarding the definition of PM_some_storage_partition_ID, which implies that MCUBoot is somehow affecting my devicetree's storage partitioning IDs.

How do you recommend making MCUBoot and Mass storage compatible based on this?

Thank you very much for your time,

- Finn

RE: MCUBoot DFU Interference with ExFat Filesystem

Sigurd Hellesvik — Thu, 20 Jun 2024 12:15:13 GMT

Hi,

[quote user=""]I've got a rather bulky project that's having a rather bulky project. In short, my project tries to enable both MCUBoot DFU and an ExFat Filesystem. I've narrowed the problem down to some kind of conflict happening between KConfig option for enabling each of these features.[/quote]

You probably have thought about this, but:

Can I suggest that you try a simpler sample with only MCUboot and ExFat and see if that works?
I find that having a lot of other things in a project as well tent to complicate debugging

Regards,
Sigurd Hellesvik