Generate storage partition during build

Hi All,

HW:
   nrf52840dk
SW:
   ncs-v2.6.1

Issue:
I'd like to be able to create a directory in my project say `/storage_part` place some files in, and during the build process have it generate a pre-built storage partition that contains my files in it that I can flash into the board. Unfortunately I've not had much luck yet.

If you've ever worked on espressif, similar to there spiffs_create_partition_image() procedure. (esp-spiffs-generate docs)

Attempt:

I'm not sure how to do this, however, I know mklittlefs (github: mklittlefs) can generate littlefs images so I thought I'd take a stab at attempting to automate that.
I created a littlefs storage partition in flash0

dts:

/delete-node/ &storage_partition;
/delete-node/ &slot0_partition;
/delete-node/ &slot1_partition;

&flash0 {
    partitions {
        compatible = "fixed-partitions";
        #address-cells = <1>;
        #size-cells = <1>;
        slot0_partition: partition@0 {
            label = "image-0";
            reg = < 0x0 0xb8000 >;
        };
        lfs1_part: partition@b8000 {
            label = "settings_storage";
            reg = <0xb8000 0x48000>;
        };
    };
};


configs:
# fs_dirent structures are big.
CONFIG_MAIN_STACK_SIZE=4096

# Let __ASSERT do its job
CONFIG_DEBUG=y

CONFIG_LOG=y
CONFIG_LOG_MODE_MINIMAL=y

CONFIG_FLASH=y
CONFIG_FLASH_MAP=y
CONFIG_FLASH_PAGE_LAYOUT=y

CONFIG_FILE_SYSTEM=y
CONFIG_FILE_SYSTEM_LITTLEFS=y

CONFIG_FS_LITTLEFS_LOOKAHEAD_SIZE=32
CONFIG_FS_LITTLEFS_CACHE_SIZE=64

# Need this when storage is on flash
CONFIG_MPU_ALLOW_FLASH_WRITE=y


Mounting code:
FS_LITTLEFS_DECLARE_DEFAULT_CONFIG(storage);

static struct fs_mount_t mountpoint = {
    .type      = FS_LITTLEFS,
    .fs_data   = &storage,
    .mnt_point = "/lfs1",
    // .storage_dev = 0xb8000,
    .storage_dev = (void *)FIXED_PARTITION_ID(lfs1_part),
};

static int littlefs_mount(struct fs_mount_t *mp)
{
    int rc;

    /* Do not mount if auto-mount has been enabled */
    rc = fs_mount(mp);
    if (rc < 0) {
        LOG_PRINTK("FAIL: mount id at %s: %d\n", mp->mnt_point, rc);
        return rc;
    }
    LOG_PRINTK("%s mount: %d\n", mp->mnt_point, rc);

    return 0;
}


Originally I wanted to as part of build process automate so I started trying to get it to work in the CMakeLists.txt, I'm not a CMake guru though I couldn't get the merge part working well.
# Generate the LittleFS image
add_custom_command(
    OUTPUT ${CMAKE_BINARY_DIR}/lfs_part.bin
    COMMAND /path/to/mklittlefs -c ${CMAKE_SOURCE_DIR}/lfs1 -b 4096 -s 0x48000 -p 16 ${CMAKE_BINARY_DIR}/lfs_part.bin
    DEPENDS ${CMAKE_SOURCE_DIR}/lfs1
)

add_custom_target(build_littlefs ALL DEPENDS ${CMAKE_BINARY_DIR}/lfs_part.bin)

# Step 1: Convert LittleFS binary to hex format
add_custom_command(
    OUTPUT ${CMAKE_BINARY_DIR}/lfs_part.hex
    COMMAND ${CMAKE_OBJCOPY} -I binary -O ihex --change-addresses 0xb8000 ${CMAKE_BINARY_DIR}/lfs_part.bin ${CMAKE_BINARY_DIR}/lfs_part.hex
    DEPENDS ${CMAKE_BINARY_DIR}/lfs_part.bin
)

# Step 2: Merge zephyr.hex and lfs_part.hex into merged.hex
add_custom_command(
    OUTPUT ${CMAKE_BINARY_DIR}/merged.hex
    COMMAND mergehex -m ${CMAKE_BINARY_DIR}/zephyr/zephyr.hex ${CMAKE_BINARY_DIR}/lfs_part.hex -o ${CMAKE_BINARY_DIR}/merged.hex
    DEPENDS ${CMAKE_BINARY_DIR}/zephyr/zephyr.hex ${CMAKE_BINARY_DIR}/lfs_part.hex
)

# Step 3: Replace zephyr.hex with merged.hex
add_custom_command(
    OUTPUT ${CMAKE_BINARY_DIR}/zephyr/zephyr.hex
    COMMAND ${CMAKE_COMMAND} -E copy ${CMAKE_BINARY_DIR}/merged.hex ${CMAKE_BINARY_DIR}/zephyr/zephyr.hex
    DEPENDS ${CMAKE_BINARY_DIR}/merged.hex
)

# Ensure lfs_part.hex is built before the final app build
add_custom_target(build_lfs_hex ALL DEPENDS ${CMAKE_BINARY_DIR}/lfs_part.hex)

# Ensure the merging happens after the application build
add_dependencies(app build_lfs_hex)


So I ended up taking the built lfs partition just doing the merge part manually like such

python3 /opt/nordic/ncs/v2.6.1/zephyr/scripts/build/mergehex.py -o merged.hex lfs_part.hex zephyr/zephyr.hex


This flashes and runs fine however, the littlefs mounting runs into an issue where it says corrupted pair:

E: WEST_TOPDIR/modules/fs/littlefs/lfs.c:1234: Corrupted dir pair at {0x1, 0x0} W: can't mount (LFS -84); formatting

It then reformats it and mounts fine but obviously wipes out any data I'm attempting to prebuild into it.


So few questions:
1) Is there an already existing framework for doing what I'm attempting that I haven't found
2) Am I on the right track or is there some special sauce in the nordic implementation for littlefs and mk can't generate a compatible image.
3) I'm not sure I'm merging them correctly, if I inspect the hex file output, I don't see the start of the littlefs partition at 0xb8000 which I'd expect to see, it's instead much earlier in the file. Is merging them not the right course to try?

Best Regards,
Wade

  • Yeah that hex works, I see the hello.txt print out.

    One thing I see different is when I build for nrf52840dk_52840 with the default prf.conf

    My default storage partition is 0xf8000 -> 0x100000 instead of 0xfa000.

    I tried building it for that

    mklittlefs -c textfiles -p 4096 -s 32768 lfs_part.bin
    /opt/nordic/ncs/toolchains/20d68df7e5/opt/zephyr-sdk/arm-zephyr-eabi/bin/arm-zephyr-eabi-objcopy -I binary -O ihex --change-addresses 0xf8000 lfs_part.bin lfs_part.hex
    python3 /opt/nordic/ncs/v2.6.1/zephyr/scripts/build/mergehex.py -o merged.hex lfs_part.hex build/zephyr/zephyr.hex


    But no luck,

    I also tried building it for 0xfa000

    But same as well.

    Are you building on v2.6.1?

    Side note as well I don't know if it matters but I'm building with the VSCode extension

    Best Regards,
    Wade

  • Ah I missed it but there's another slight difference here:

    Mine:

    I: LittleFS version 2.5, disk version 2.0

    Yours:

    I: LittleFS version 2.8, disk version 2.1

  • Edit: Sorry never mind, accidently used the wrong hex file got to excited, v2.7.0 not working for me either

    To be as explicit as possible I removed the VSCode extension from the equation.


    I took your zip file, and on nvs-v2.7.0 ( As this will setup littlefs version to 2.8, disk version 2.1 )

    I'm running the following

    west build --build-dir build/ --pristine --board nrf52840dk_nrf52840

    If I flash this on my board ignoring attempting to flash the littlefs partition on it yet. I get this output:

    Area 3 at 0xf8000 on flash-controller@4001e000 for 32768 bytes

    This from as far as I can tell is the only difference I can tell, tho even so it should be fine long as I account for it I would expect.


    My full steps:

    1) west build --build-dir build/ --pristine --board nrf52840dk_nrf52840
    2) mklittlefs -c textfiles -p 4096 -s 32768 lfs_part.bin
    3) /opt/nordic/ncs/toolchains/20d68df7e5/opt/zephyr-sdk/arm-zephyr-eabi/bin/arm-zephyr-eabi-objcopy -I binary -O ihex --change-addresses 0xf8000 lfs_part.bin lfs_part.hex

    Here I've tried two ways
    4a) west flash
    4a.1) nrfjprog --program lfs_part.hex --sectorerase --verify -r
    -----
    4b) python3 /opt/nordic/ncs/v2.6.1/zephyr/scripts/build/mergehex.py -o merged.hex lfs_part.hex build/zephyr/merged.hex
    4b.1) nrfjprog --program merged.hex --sectorerase --verify -r

  • I'm sorry, I was certain that I built the project with SDK v2.6.1, but I see now that I actually built it with SDK v2.7.0. When building with SDK v2.7.0, please make sure you use the pre-release version of the VS code extension:

    This is needed to work with sysbuild (Migrating from multi-image builds to sysbuild) and the new HW model (igrating to the current hardware model). From the release notes:

    Build configuration:

    Note: Sysbuild is enabled by default if you use West to install a new SDK, but not if you update your existing SDK with git checkout and west update. Therefore, you may want to explicitly enable it in the project configuration as shown above. (Sysbuild enabled by default)

    The partition manager gets enabled by default, which is why the storage partition got assigned a different starting address (0xfa000 vs 0xf8000).

    Here's an updated version of the project:

    littlefs_2.7.0.zip

    This version includes the custom cmake commands to create the littlefs filesystem image. The generated littlefs_image.hex can be found in build/littlefs_2.7.0. I did not manage to find a way to automatically merge it with the zephyr.hex, so it still needs to be programmed separately.

Related