Using Partition Manager created partitions for FatFS

RE: Using Partition Manager created partitions for FatFS

Simonr — Fri, 14 Jul 2023 05:28:52 GMT

Hello

I just wanted to let you know that a potential fix for this issue is currently in review. See this pull request for details: https://github.com/nrfconnect/sdk-nrf/pull/11802

Best regards,

Simon

RE: Using Partition Manager created partitions for FatFS

Simonr — Tue, 23 May 2023 12:42:32 GMT

Thanks for the update Tiit, and for your feedback/suggestions.

Got an update just now: We do not have a fix yet I'm afraid. However, as long as you leave space on the device that can be used for storage, that space can be repurposed. The bootloader will only be interested in the areas defined for it (primary and secondary application slot for example). Once you build the bootloader it will only know of the defined areas and not try to access anything else.

So for example,

+---------------------+------------------------------------+------------------------------------------+---------------------------------------------+
| Bootloader (size A) | App primary slot (offset A, size B)| App secondary slot (Offset A + B, size C | Unallocated space (offset A + B + C, size D |
+---------------------+------------------------------------+------------------------------------------+---------------------------------------------+

Total flash size A+B+C+D.

In the above flash map only the applications will have ability to later allocate the "Unallocated" space, if they want to. The bootloader and app will be bound to slot sizes and the bootloader will only know of the position of the bootloader, and app slots. So it should be fine to add a patch for this later.

Best regards,

Simon

RE: Using Partition Manager created partitions for FatFS

Tiit — Tue, 23 May 2023 12:13:59 GMT

Hi Simon,

Thank you for checking with the devs.

What I'm currently doing is patching each new version of the SDK by just hardcoding the "area_id" value in /zephyr/drivers/disk/flashdisk.c to 6 (6 being specific to my partition manager created partition for the flashdisk).

The macro that defines all the flashdisks uses DT_FIXED_PARTITION_ID(PARTITION_PHANDLE(n)) and that seems to only work for partitions defined in the devicetree. I tried my best to figure out how to change this macro to read the Partition Manager output but due to time constraints I can't put more time into this. Since we only really need one flashdisk this hardcoded partition id works for us for now. We are not likely to change our memory layout either so it should be safe for now.

What your devs would need to do is replace the macro with one that can actually read Partition Manager outputs when that is used instead of the dts file.

Best regards,

Tiit

RE: Using Partition Manager created partitions for FatFS

Simonr — Tue, 23 May 2023 12:06:53 GMT

There haven't been any updates in the internal ticket I'm afraid. I pinged the devs for an update now, but I can't guarantee anything, and I do not have an estimate of when this can be fixed. If you need/want to escalate this, you will need to go via the Nordic regional sales manager of your area. Let me know if you don't have their contact information and I'll help you get in touch.

I've asked the developers on your thoughts to ship with a broken MSD/MSC part. We can't guarantee that this will be a patchable issue on shipped products, so if this is a necessary feature we can't recommend that you ship it with a broken USB MSD/C...

Very sorry about the inconvenience and frustration this issue has caused.

Best regards,

Simon

RE: Using Partition Manager created partitions for FatFS

Tiit — Fri, 19 May 2023 09:47:29 GMT

Hi Simon,

Any updates on this?

I looked at the solution you linked but unfortunately I could not get that to work for me. My goal is to get the MX25R64 device partitioned for slot2 for mcyboot AND also work as the storage area for FAT used by USB MSD. I can make this all work in the internal memory. Actually I can even get it kind of working when the slot2 is in the external flash. But I have not been able to get USB MSD to work from external flash when the bootloader is enabled. The solution you linked was to put settings/nvs partitions into external flash and while I did try that that is not what I want to do. I want to keep the settings partition in internal memory and only move the FAT partition used by the MSD USB to the external memory.

We now have something working that we should start shipping next week. But we need to be able to upgrade these shipped devices to Matter later. This means we need to have slot2 of the bootloader in external flash. I think that is working now but doing that broke the USB MSD part. Now we could ship with that part broken and later update things to a working state. But I'm worried that something in the partitions may need to change and since the same partitions are also used to compile the bootloader (and we can't update that) we might be stuck with whatever partition thing we ship with now. This has been causing so much frustration already that I've been seriously considering just switching to a Silabs solution.

Can someone please give and estimate for when this issue might get some attention from your engineers? I see I'm not the only one struggling with this. And from reading all the replies this seems like it is cause by the Nordic Partition Manager. That seems to be your custom solution and the documentation for that has been of very little help.

[00:00:00.026,580] <dbg> blindy_files: fa_cb: 0 0 flash-controller@4001e000 0x0 0x7000
[00:00:00.027,252] <dbg> blindy_files: fa_cb: 1 0 mx25r6435f@0 0x0 0xf4000
[00:00:00.027,893] <dbg> blindy_files: fa_cb: 2 0 flash-controller@4001e000 0x7000 0x200
[00:00:00.028,533] <dbg> blindy_files: fa_cb: 3 0 flash-controller@4001e000 0x7000 0xf4000
[00:00:00.029,205] <dbg> blindy_files: fa_cb: 4 0 flash-controller@4001e000 0x7200 0xf3e00
[00:00:00.029,876] <dbg> blindy_files: fa_cb: 5 0 flash-controller@4001e000 0x7200 0xf3e00
[00:00:00.030,517] <dbg> blindy_files: fa_cb: 6 0 mx25r6435f@0 0xf4000 0x70c000
[00:00:00.036,163] <dbg> blindy_files: fa_cb: 7 0 flash-controller@4001e000 0xfb000 0x1000
[00:00:00.036,834] <dbg> blindy_files: fa_cb: 8 0 flash-controller@4001e000 0xfc000 0x4000
[00:00:00.037,506] <inf> blindy_files: setup_flash: Area 6 at 0xf4000 on mx25r6435f@0 for 7389184 bytes

[00:00:00.038,116] <dbg> disk: disk_access_init: Disk access init NAND, dev:flash-controller@4001e000
[00:00:00.038,665] <dbg> flashdisk: disk_flash_access_init: Init flashdisk with device: flash-controller@4001e000
[00:00:00.039,245] <inf> flashdisk: flashdisk_init_runtime: Initialize device NAND
[00:00:00.039,733] <inf> flashdisk: flashdisk_init_runtime: offset f4000, sector size 512, page size 4096, volume size 7389184
[00:00:00.040,435] <err> flashdisk: flashdisk_init_runtime: Error -22 while getting page info
[00:00:00.040,954] <err> fs: fs_mount: fs mount error (-5)

I have tried to find out where things go wrong but so far I can't figure out where the wrong backend gets selected. It looks ot me that flash_area gets set up correctly. The partition table printed out from the actual running system also seems to match the one that Partition Manager should create. And even the device behind each partition seems to be correct. But still for some reason flashdisk wants to use internal flash memory api to access the partition. It actually seems to know the offset and size of the partition - only the underlying device driver seems to be selected incorrectly. Everythign then fails because using internal memory api functions to read addresses beyond the 1 MB capacity fails.

I will keep trying to figure out where the flash api for flashdisk gets selected but it is a very tedious process. Please provide any help possible. Our whole project is really hurting because of this issue.

Tiit

RE: Using Partition Manager created partitions for FatFS

Simonr — Wed, 03 May 2023 06:35:11 GMT

Hi Tiit

I've got an update for you, although it doesn't come from the developers. It seems like another DevZone user was able to do FatFS on an external flash while keeping the partition manager active here. This has not been confirmed to work by tech support or the developers yet, but you're welcome to try it out: RE: Mass Example with External Flash not working with CONFIG_BOOTLOADER_MCUBOOT=y

Best regards,

Simon

RE: Using Partition Manager created partitions for FatFS

Tiit — Mon, 20 Mar 2023 13:03:52 GMT

I would like to be kept updated on this, thank you. For the time being I will just try to keep using the weird solution I have and make sure to keep everything in sync manually.

Thank you for your help.

Tiit

RE: Using Partition Manager created partitions for FatFS

Simonr — Mon, 20 Mar 2023 12:55:40 GMT

UPDATE: Got a reply quicker than anticipated. For the time being, it's not possible to define the partition in the overlay/DTS file while partitions are managed by the PM, but our developers are aware of this, and are trying to address it. However, it's not yet resolved as it takes some time to get to the bottom of and find a sufficient solution. In short what's happening here is that the disk access uses PM (replacing the Flash API) macros to access the fixed partition areas by DTS ID, ending up with an API <-> ID mismatch.

If you'd like I can keep you updated on any progress or resolution of this issue.

Best regards,

Simon

RE: Using Partition Manager created partitions for FatFS

Simonr — Mon, 20 Mar 2023 12:39:53 GMT

I have forwarded your question internally, as I'm not sure whether this is possible or not myself. One of the developers working specifically with the partition manager will take a look and I'll let you know what their input is. Thank you for your patience!

Best regards,

Simon

RE: Using Partition Manager created partitions for FatFS

Tiit — Fri, 17 Mar 2023 11:39:14 GMT

Hi,

I'm not sure what you are asking me to do now. I have a working state for now. It is just a very cumbersome way I had to set this up to work. I will try to explain my problem again.

I want to have a device that enumerates as a USB MSD. It should use a partition in external memory for storage. I can set that up using devicetree overlays like this.

&mx25r64 {
	partitions {
		compatible = "fixed-partitions";
		#address-cells = <1>;
		#size-cells = <1>;

		external_flash: partition@0 {
			label = "external_flash";
			reg = <0x0 0x800000>;
		};
	};
};

/ {
	msc_disk0 {
		compatible = "zephyr,flash-disk";
		partition = <&external_flash>;
		disk-name = "NAND";
		cache-size = <4096>;
	};
};

And that works exactly like in the USB MSD example. But if I now add DFU to my project that will force the use of Partition Manager and the partition defined in the DTS file is no longer used (correct me if I'm wrong here though). I can see in the partitions.yml file that partitions can still be created and used and using the pm_static.yml file I can still manually configure the partitions I want to use. While that was not easy to set up I think that also works now. Since the &mx25r64 part defining a partition is no longer used I would like to now remove this from the DTS file so that the only place I have to define the partitions would be in the pm_static.yml file. But when I do that msc_disk0 can't be defined because external_flash no longer defines a partition as far as the DTS file is concerned. What I have working now is I keep the partition definitions in bot files and that works so far but it doesn't seem like a good solution as any changes in partitioning have to be manually kept in sync in multiple files.

My question is - how can I still define msc_disk0 in my overlay/DTS file while the partitions are managed by the PM?

Best regards,

Tiit

RE: Using Partition Manager created partitions for FatFS

Simonr — Fri, 17 Mar 2023 11:23:27 GMT

Hi again

I may have been preemptive in my previous reply, and would like to reproduce and test this on my end if possible. What SDK version is this reproducible in for example, and are there any steps/changes necessary to do so on my end? If not, can you share some more information on what the error(s) you're seeing on your end, for example the full error log so we can get a better look of what exactly is going wrong.

Best regards,

Simon

RE: Using Partition Manager created partitions for FatFS

Tiit — Fri, 17 Mar 2023 10:47:03 GMT

I will look at the ticket you referenced. But as far as I understand that means that much of the actual work done by the compiler based on DTS files will have to be manually done in code by hand. Like described here:

If you look into partitions.yml file under build folder, cou can see that the nvs_storage start address is at 0x3f000. If you search that address in all files under build folder, you can find that the PM_NVS_STORAGE_ADDRESS macro defined in build\zephyr\include\generated\pm_config.h file is the start address of the NVS storage. With that, you can populate the fs.offset variable.

To get the flash device binding, again found in the build\zephyr\include\generated\pm_config.h file, the PM_NVS_STORAGE_DEV_NAME macro includes NVS storage device name. With it, you can populate the fs.flash_device variable by calling device_get_binding(PM_NVS_STORAGE_DEV_NAME) function.

What I have right now is basically the &flash0 part in the overlay and then the PM generated layout that I have to keep in sync manually . That is also not a good solution I think.

I will try to see if I can understand what user IvanV did and see if I can recreate that. It would make life so much simpler if you could include an example for something like this in the future. I think using DFU is pretty much standard with all BLE devices today and having all your examples fail to build with DFU doesn't seem like a good idea.

RE: Using Partition Manager created partitions for FatFS

Simonr — Fri, 17 Mar 2023 10:33:48 GMT

Yes, this should be possible. pm_config and the available defines are available in ".../build/zephyr/include/generated/pm_config.h". (PM_MCUBOOT_SECONDARY_ADDRESS Please check out this ticket where a solution to a similar issue is suggested both by a user IvanV and my colleague Sigurd.

Is the second snippet (starting with &flash0) from your overlays or is it the build output? If it's the prior, the way to manually make partitions for the partition manager should be through pm_static.yml. Usually NCS makes these "automagically" so you won't have to do it manually though.

Best regards,

Simon