Generate storage partition during build

Hi All,

HW:
   nrf52840dk
SW:
   ncs-v2.6.1

Issue:
I'd like to be able to create a directory in my project say `/storage_part` place some files in, and during the build process have it generate a pre-built storage partition that contains my files in it that I can flash into the board. Unfortunately I've not had much luck yet.

If you've ever worked on espressif, similar to there spiffs_create_partition_image() procedure. (esp-spiffs-generate docs)

Attempt:

I'm not sure how to do this, however, I know mklittlefs (github: mklittlefs) can generate littlefs images so I thought I'd take a stab at attempting to automate that.
I created a littlefs storage partition in flash0

dts:

/delete-node/ &storage_partition;
/delete-node/ &slot0_partition;
/delete-node/ &slot1_partition;

&flash0 {
    partitions {
        compatible = "fixed-partitions";
        #address-cells = <1>;
        #size-cells = <1>;
        slot0_partition: partition@0 {
            label = "image-0";
            reg = < 0x0 0xb8000 >;
        };
        lfs1_part: partition@b8000 {
            label = "settings_storage";
            reg = <0xb8000 0x48000>;
        };
    };
};


configs:
# fs_dirent structures are big.
CONFIG_MAIN_STACK_SIZE=4096

# Let __ASSERT do its job
CONFIG_DEBUG=y

CONFIG_LOG=y
CONFIG_LOG_MODE_MINIMAL=y

CONFIG_FLASH=y
CONFIG_FLASH_MAP=y
CONFIG_FLASH_PAGE_LAYOUT=y

CONFIG_FILE_SYSTEM=y
CONFIG_FILE_SYSTEM_LITTLEFS=y

CONFIG_FS_LITTLEFS_LOOKAHEAD_SIZE=32
CONFIG_FS_LITTLEFS_CACHE_SIZE=64

# Need this when storage is on flash
CONFIG_MPU_ALLOW_FLASH_WRITE=y


Mounting code:
FS_LITTLEFS_DECLARE_DEFAULT_CONFIG(storage);

static struct fs_mount_t mountpoint = {
    .type      = FS_LITTLEFS,
    .fs_data   = &storage,
    .mnt_point = "/lfs1",
    // .storage_dev = 0xb8000,
    .storage_dev = (void *)FIXED_PARTITION_ID(lfs1_part),
};

static int littlefs_mount(struct fs_mount_t *mp)
{
    int rc;

    /* Do not mount if auto-mount has been enabled */
    rc = fs_mount(mp);
    if (rc < 0) {
        LOG_PRINTK("FAIL: mount id at %s: %d\n", mp->mnt_point, rc);
        return rc;
    }
    LOG_PRINTK("%s mount: %d\n", mp->mnt_point, rc);

    return 0;
}


Originally I wanted to as part of build process automate so I started trying to get it to work in the CMakeLists.txt, I'm not a CMake guru though I couldn't get the merge part working well.
# Generate the LittleFS image
add_custom_command(
    OUTPUT ${CMAKE_BINARY_DIR}/lfs_part.bin
    COMMAND /path/to/mklittlefs -c ${CMAKE_SOURCE_DIR}/lfs1 -b 4096 -s 0x48000 -p 16 ${CMAKE_BINARY_DIR}/lfs_part.bin
    DEPENDS ${CMAKE_SOURCE_DIR}/lfs1
)

add_custom_target(build_littlefs ALL DEPENDS ${CMAKE_BINARY_DIR}/lfs_part.bin)

# Step 1: Convert LittleFS binary to hex format
add_custom_command(
    OUTPUT ${CMAKE_BINARY_DIR}/lfs_part.hex
    COMMAND ${CMAKE_OBJCOPY} -I binary -O ihex --change-addresses 0xb8000 ${CMAKE_BINARY_DIR}/lfs_part.bin ${CMAKE_BINARY_DIR}/lfs_part.hex
    DEPENDS ${CMAKE_BINARY_DIR}/lfs_part.bin
)

# Step 2: Merge zephyr.hex and lfs_part.hex into merged.hex
add_custom_command(
    OUTPUT ${CMAKE_BINARY_DIR}/merged.hex
    COMMAND mergehex -m ${CMAKE_BINARY_DIR}/zephyr/zephyr.hex ${CMAKE_BINARY_DIR}/lfs_part.hex -o ${CMAKE_BINARY_DIR}/merged.hex
    DEPENDS ${CMAKE_BINARY_DIR}/zephyr/zephyr.hex ${CMAKE_BINARY_DIR}/lfs_part.hex
)

# Step 3: Replace zephyr.hex with merged.hex
add_custom_command(
    OUTPUT ${CMAKE_BINARY_DIR}/zephyr/zephyr.hex
    COMMAND ${CMAKE_COMMAND} -E copy ${CMAKE_BINARY_DIR}/merged.hex ${CMAKE_BINARY_DIR}/zephyr/zephyr.hex
    DEPENDS ${CMAKE_BINARY_DIR}/merged.hex
)

# Ensure lfs_part.hex is built before the final app build
add_custom_target(build_lfs_hex ALL DEPENDS ${CMAKE_BINARY_DIR}/lfs_part.hex)

# Ensure the merging happens after the application build
add_dependencies(app build_lfs_hex)


So I ended up taking the built lfs partition just doing the merge part manually like such

python3 /opt/nordic/ncs/v2.6.1/zephyr/scripts/build/mergehex.py -o merged.hex lfs_part.hex zephyr/zephyr.hex


This flashes and runs fine however, the littlefs mounting runs into an issue where it says corrupted pair:

E: WEST_TOPDIR/modules/fs/littlefs/lfs.c:1234: Corrupted dir pair at {0x1, 0x0} W: can't mount (LFS -84); formatting

It then reformats it and mounts fine but obviously wipes out any data I'm attempting to prebuild into it.


So few questions:
1) Is there an already existing framework for doing what I'm attempting that I haven't found
2) Am I on the right track or is there some special sauce in the nordic implementation for littlefs and mk can't generate a compatible image.
3) I'm not sure I'm merging them correctly, if I inspect the hex file output, I don't see the start of the littlefs partition at 0xb8000 which I'd expect to see, it's instead much earlier in the file. Is merging them not the right course to try?

Best Regards,
Wade

Parents
  • Hello Wade,

    Unfortunately, I'm not aware of any existing samples that does this in the SDK. Our sample projects are mostly using the Settings module to manage persistent storage. That said, I think I was able to make it work. I built mklittlefs from the 4.0.0 tag (did not change override any of the default build configurations) and used the littlefs sample from the SDK where the storage partition starts at 0xfa000 and is 0x6000 bytes.

    Commands I used to generate and program the filesystem image:

    $ mklittlefs -c textfiles/ -p 4096  -s 24576 littlefs.bin
    $ arm-zephyr-eabi-objcopy -I binary -O ihex --change-addresses 0xfa000 littlefs.bin littlefs.hex
    $ nrfjprog --program littlefs.hex --sectorerase --verify -r

    This was mostly based on the CMake file you posted, but instead of setting the block size, I set the page size to 4096 to match the internal page size of the chip. I'm not sure if the block size paramater applies here since we are using flash. Could this be the reason it didn't work on your end?

    Best regards,

    Vidar

  • Hi Vidar,

    Thanks for the reply,

    I tried making the changes you suggested but still unfortunately littlefs doesn't mount cleanly onto it.

    For clarity I'm using the littlefs sample from the zephyr side, v3.5.99, slightly modified to put the storage partition into interal.

    The changes I made from my post before is to define the littlefs partition in a pm_static.yml

    app:
      address: 0x0
      end_address: 0xfa000
      region: flash_primary
      size: 0xfa000
    littlefs_storage:
      address: 0xfa000
      end_address: 0x100000
      placement:
        align:
          start: 0x1000
        before:
        - end
      region: flash_primary
      size: 0x6000
    sram_primary:
      address: 0x20000000
      end_address: 0x20040000
      region: sram_primary
      size: 0x40000
    


    And I followed your steps of manually creating the partition and adjusting it's address and flashing it separately. But I still get the error when attempting to mount, (the mounting procedure I havn't changed from my post earlier)

    Error:
    *** Booting nRF Connect SDK v3.5.99-ncs1-1 ***
    Sample program to r/w files on littlefs
    I: LittleFS version 2.5, disk version 2.0
    I: FS at flash-controller@4001e000:0xfa000 is 6 0x1000-byte blocks with 512 cycle
    I: sizes: rd 16 ; pr 16 ; ca 64 ; la 32
    E: WEST_TOPDIR/modules/fs/littlefs/lfs.c:1234: Corrupted dir pair at {0x1, 0x0}
    W: can't mount (LFS -84); formatting
    I: /lfs1 mounted
    /lfs1 mount: 0
    /lfs1: bsize = 16 ; frsize = 4096 ; blocks = 6 ; bfree = 4
    
    Listing dir /lfs1 ...
    I: /lfs1 unmounted
    /lfs1 unmount: 0


    How are you mounting the filesystem and validating it?

    Best Regards,

    Wade

  • Hi Wade,

    I only verified that I was able to mount the partition and that it found the file I added (based on log output). I did not make any changes to the sample code. Attached below is the project I used. Could you please try that just to see if we get the same result?

    6675.littlefs.zip

    Best regards,

    Vidar

  • Hrm yeah I'm seeing the same thing still on it's initial boot

    Area 3 at 0xf8000 on flash-controller@4001e000 for 32768 bytes
    I: LittleFS version 2.5, disk version 2.0
    I: FS at flash-controller@4001e000:0xf8000 is 8 0x1000-byte blocks with 512 cycle
    I: sizes: rd 16 ; pr 16 ; ca 64 ; la 32
    E: WEST_TOPDIR/modules/fs/littlefs/lfs.c:1234: Corrupted dir pair at {0x1, 0x0}
    W: can't mount (LFS -84); formatting



    I just built it plain board target: nrf52840dk_52840 and prj.conf, Are you just using the standard default storage partition that's in internal?

    Then flashed it, and then flashed the littlefs.hex, 

    I tried merging them as well but no luck.

    Best Regards,
    Wade

  • Strange. I'm testing with the nRF52840 DK as the build target and using the default storage partition. This is the log I get:

    *** Using Zephyr OS v3.6.99-100befc70c74 ***
    Sample program to r/w files on littlefs
    Area 1 at 0xfa000 on flash-controller@4001e000 for 24576 bytes
    I: LittleFS version 2.8, disk version 2.1
    I: FS at flash-controller@4001e000:0xfa000 is 6 0x1000-byte blocks with 512 cycle
    I: sizes: rd 16 ; pr 16 ; ca 64 ; la 32
    /lfs mount: 0
    /lfs: bsize = 16 ; frsize = 4096 ; blocks = 6 ; bfree = 3
    
    Listing dir /lfs ...
    [FILE] boot_count (size = 1)
    [FILE] hello.txt (size = 0)
    [FILE] pattern.bin (size = 547)
    /lfs/boot_count read count:1 (bytes: 1)
    /lfs/boot_count write new boot count 2: [wr:1]
    ------ FILE: /lfs/pattern.bin ------
    

    And here is the hex file:

    merged_ltfs.hex

    Please try programming the hex file above as well.

    Best regards,

    Vidar

Reply
  • Strange. I'm testing with the nRF52840 DK as the build target and using the default storage partition. This is the log I get:

    *** Using Zephyr OS v3.6.99-100befc70c74 ***
    Sample program to r/w files on littlefs
    Area 1 at 0xfa000 on flash-controller@4001e000 for 24576 bytes
    I: LittleFS version 2.8, disk version 2.1
    I: FS at flash-controller@4001e000:0xfa000 is 6 0x1000-byte blocks with 512 cycle
    I: sizes: rd 16 ; pr 16 ; ca 64 ; la 32
    /lfs mount: 0
    /lfs: bsize = 16 ; frsize = 4096 ; blocks = 6 ; bfree = 3
    
    Listing dir /lfs ...
    [FILE] boot_count (size = 1)
    [FILE] hello.txt (size = 0)
    [FILE] pattern.bin (size = 547)
    /lfs/boot_count read count:1 (bytes: 1)
    /lfs/boot_count write new boot count 2: [wr:1]
    ------ FILE: /lfs/pattern.bin ------
    

    And here is the hex file:

    merged_ltfs.hex

    Please try programming the hex file above as well.

    Best regards,

    Vidar

Children
  • Yeah that hex works, I see the hello.txt print out.

    One thing I see different is when I build for nrf52840dk_52840 with the default prf.conf

    My default storage partition is 0xf8000 -> 0x100000 instead of 0xfa000.

    I tried building it for that

    mklittlefs -c textfiles -p 4096 -s 32768 lfs_part.bin
    /opt/nordic/ncs/toolchains/20d68df7e5/opt/zephyr-sdk/arm-zephyr-eabi/bin/arm-zephyr-eabi-objcopy -I binary -O ihex --change-addresses 0xf8000 lfs_part.bin lfs_part.hex
    python3 /opt/nordic/ncs/v2.6.1/zephyr/scripts/build/mergehex.py -o merged.hex lfs_part.hex build/zephyr/zephyr.hex


    But no luck,

    I also tried building it for 0xfa000

    But same as well.

    Are you building on v2.6.1?

    Side note as well I don't know if it matters but I'm building with the VSCode extension

    Best Regards,
    Wade

  • Ah I missed it but there's another slight difference here:

    Mine:

    I: LittleFS version 2.5, disk version 2.0

    Yours:

    I: LittleFS version 2.8, disk version 2.1

  • Edit: Sorry never mind, accidently used the wrong hex file got to excited, v2.7.0 not working for me either

    To be as explicit as possible I removed the VSCode extension from the equation.


    I took your zip file, and on nvs-v2.7.0 ( As this will setup littlefs version to 2.8, disk version 2.1 )

    I'm running the following

    west build --build-dir build/ --pristine --board nrf52840dk_nrf52840

    If I flash this on my board ignoring attempting to flash the littlefs partition on it yet. I get this output:

    Area 3 at 0xf8000 on flash-controller@4001e000 for 32768 bytes

    This from as far as I can tell is the only difference I can tell, tho even so it should be fine long as I account for it I would expect.


    My full steps:

    1) west build --build-dir build/ --pristine --board nrf52840dk_nrf52840
    2) mklittlefs -c textfiles -p 4096 -s 32768 lfs_part.bin
    3) /opt/nordic/ncs/toolchains/20d68df7e5/opt/zephyr-sdk/arm-zephyr-eabi/bin/arm-zephyr-eabi-objcopy -I binary -O ihex --change-addresses 0xf8000 lfs_part.bin lfs_part.hex

    Here I've tried two ways
    4a) west flash
    4a.1) nrfjprog --program lfs_part.hex --sectorerase --verify -r
    -----
    4b) python3 /opt/nordic/ncs/v2.6.1/zephyr/scripts/build/mergehex.py -o merged.hex lfs_part.hex build/zephyr/merged.hex
    4b.1) nrfjprog --program merged.hex --sectorerase --verify -r

  • I'm sorry, I was certain that I built the project with SDK v2.6.1, but I see now that I actually built it with SDK v2.7.0. When building with SDK v2.7.0, please make sure you use the pre-release version of the VS code extension:

    This is needed to work with sysbuild (Migrating from multi-image builds to sysbuild) and the new HW model (igrating to the current hardware model). From the release notes:

    Build configuration:

    Note: Sysbuild is enabled by default if you use West to install a new SDK, but not if you update your existing SDK with git checkout and west update. Therefore, you may want to explicitly enable it in the project configuration as shown above. (Sysbuild enabled by default)

    The partition manager gets enabled by default, which is why the storage partition got assigned a different starting address (0xfa000 vs 0xf8000).

    Here's an updated version of the project:

    littlefs_2.7.0.zip

    This version includes the custom cmake commands to create the littlefs filesystem image. The generated littlefs_image.hex can be found in build/littlefs_2.7.0. I did not manage to find a way to automatically merge it with the zephyr.hex, so it still needs to be programmed separately.

Related