Generate storage partition during build

Hi All,

HW:
   nrf52840dk
SW:
   ncs-v2.6.1

Issue:
I'd like to be able to create a directory in my project say `/storage_part` place some files in, and during the build process have it generate a pre-built storage partition that contains my files in it that I can flash into the board. Unfortunately I've not had much luck yet.

If you've ever worked on espressif, similar to there spiffs_create_partition_image() procedure. (esp-spiffs-generate docs)

Attempt:

I'm not sure how to do this, however, I know mklittlefs (github: mklittlefs) can generate littlefs images so I thought I'd take a stab at attempting to automate that.
I created a littlefs storage partition in flash0

dts:

/delete-node/ &storage_partition;
/delete-node/ &slot0_partition;
/delete-node/ &slot1_partition;

&flash0 {
    partitions {
        compatible = "fixed-partitions";
        #address-cells = <1>;
        #size-cells = <1>;
        slot0_partition: partition@0 {
            label = "image-0";
            reg = < 0x0 0xb8000 >;
        };
        lfs1_part: partition@b8000 {
            label = "settings_storage";
            reg = <0xb8000 0x48000>;
        };
    };
};


configs:
# fs_dirent structures are big.
CONFIG_MAIN_STACK_SIZE=4096

# Let __ASSERT do its job
CONFIG_DEBUG=y

CONFIG_LOG=y
CONFIG_LOG_MODE_MINIMAL=y

CONFIG_FLASH=y
CONFIG_FLASH_MAP=y
CONFIG_FLASH_PAGE_LAYOUT=y

CONFIG_FILE_SYSTEM=y
CONFIG_FILE_SYSTEM_LITTLEFS=y

CONFIG_FS_LITTLEFS_LOOKAHEAD_SIZE=32
CONFIG_FS_LITTLEFS_CACHE_SIZE=64

# Need this when storage is on flash
CONFIG_MPU_ALLOW_FLASH_WRITE=y


Mounting code:
FS_LITTLEFS_DECLARE_DEFAULT_CONFIG(storage);

static struct fs_mount_t mountpoint = {
    .type      = FS_LITTLEFS,
    .fs_data   = &storage,
    .mnt_point = "/lfs1",
    // .storage_dev = 0xb8000,
    .storage_dev = (void *)FIXED_PARTITION_ID(lfs1_part),
};

static int littlefs_mount(struct fs_mount_t *mp)
{
    int rc;

    /* Do not mount if auto-mount has been enabled */
    rc = fs_mount(mp);
    if (rc < 0) {
        LOG_PRINTK("FAIL: mount id at %s: %d\n", mp->mnt_point, rc);
        return rc;
    }
    LOG_PRINTK("%s mount: %d\n", mp->mnt_point, rc);

    return 0;
}


Originally I wanted to as part of build process automate so I started trying to get it to work in the CMakeLists.txt, I'm not a CMake guru though I couldn't get the merge part working well.
# Generate the LittleFS image
add_custom_command(
    OUTPUT ${CMAKE_BINARY_DIR}/lfs_part.bin
    COMMAND /path/to/mklittlefs -c ${CMAKE_SOURCE_DIR}/lfs1 -b 4096 -s 0x48000 -p 16 ${CMAKE_BINARY_DIR}/lfs_part.bin
    DEPENDS ${CMAKE_SOURCE_DIR}/lfs1
)

add_custom_target(build_littlefs ALL DEPENDS ${CMAKE_BINARY_DIR}/lfs_part.bin)

# Step 1: Convert LittleFS binary to hex format
add_custom_command(
    OUTPUT ${CMAKE_BINARY_DIR}/lfs_part.hex
    COMMAND ${CMAKE_OBJCOPY} -I binary -O ihex --change-addresses 0xb8000 ${CMAKE_BINARY_DIR}/lfs_part.bin ${CMAKE_BINARY_DIR}/lfs_part.hex
    DEPENDS ${CMAKE_BINARY_DIR}/lfs_part.bin
)

# Step 2: Merge zephyr.hex and lfs_part.hex into merged.hex
add_custom_command(
    OUTPUT ${CMAKE_BINARY_DIR}/merged.hex
    COMMAND mergehex -m ${CMAKE_BINARY_DIR}/zephyr/zephyr.hex ${CMAKE_BINARY_DIR}/lfs_part.hex -o ${CMAKE_BINARY_DIR}/merged.hex
    DEPENDS ${CMAKE_BINARY_DIR}/zephyr/zephyr.hex ${CMAKE_BINARY_DIR}/lfs_part.hex
)

# Step 3: Replace zephyr.hex with merged.hex
add_custom_command(
    OUTPUT ${CMAKE_BINARY_DIR}/zephyr/zephyr.hex
    COMMAND ${CMAKE_COMMAND} -E copy ${CMAKE_BINARY_DIR}/merged.hex ${CMAKE_BINARY_DIR}/zephyr/zephyr.hex
    DEPENDS ${CMAKE_BINARY_DIR}/merged.hex
)

# Ensure lfs_part.hex is built before the final app build
add_custom_target(build_lfs_hex ALL DEPENDS ${CMAKE_BINARY_DIR}/lfs_part.hex)

# Ensure the merging happens after the application build
add_dependencies(app build_lfs_hex)


So I ended up taking the built lfs partition just doing the merge part manually like such

python3 /opt/nordic/ncs/v2.6.1/zephyr/scripts/build/mergehex.py -o merged.hex lfs_part.hex zephyr/zephyr.hex


This flashes and runs fine however, the littlefs mounting runs into an issue where it says corrupted pair:

E: WEST_TOPDIR/modules/fs/littlefs/lfs.c:1234: Corrupted dir pair at {0x1, 0x0} W: can't mount (LFS -84); formatting

It then reformats it and mounts fine but obviously wipes out any data I'm attempting to prebuild into it.


So few questions:
1) Is there an already existing framework for doing what I'm attempting that I haven't found
2) Am I on the right track or is there some special sauce in the nordic implementation for littlefs and mk can't generate a compatible image.
3) I'm not sure I'm merging them correctly, if I inspect the hex file output, I don't see the start of the littlefs partition at 0xb8000 which I'd expect to see, it's instead much earlier in the file. Is merging them not the right course to try?

Best Regards,
Wade

  • Hello Wade,

    Unfortunately, I'm not aware of any existing samples that does this in the SDK. Our sample projects are mostly using the Settings module to manage persistent storage. That said, I think I was able to make it work. I built mklittlefs from the 4.0.0 tag (did not change override any of the default build configurations) and used the littlefs sample from the SDK where the storage partition starts at 0xfa000 and is 0x6000 bytes.

    Commands I used to generate and program the filesystem image:

    $ mklittlefs -c textfiles/ -p 4096  -s 24576 littlefs.bin
    $ arm-zephyr-eabi-objcopy -I binary -O ihex --change-addresses 0xfa000 littlefs.bin littlefs.hex
    $ nrfjprog --program littlefs.hex --sectorerase --verify -r

    This was mostly based on the CMake file you posted, but instead of setting the block size, I set the page size to 4096 to match the internal page size of the chip. I'm not sure if the block size paramater applies here since we are using flash. Could this be the reason it didn't work on your end?

    Best regards,

    Vidar

  • Hi Vidar,

    Thanks for the reply,

    I tried making the changes you suggested but still unfortunately littlefs doesn't mount cleanly onto it.

    For clarity I'm using the littlefs sample from the zephyr side, v3.5.99, slightly modified to put the storage partition into interal.

    The changes I made from my post before is to define the littlefs partition in a pm_static.yml

    app:
      address: 0x0
      end_address: 0xfa000
      region: flash_primary
      size: 0xfa000
    littlefs_storage:
      address: 0xfa000
      end_address: 0x100000
      placement:
        align:
          start: 0x1000
        before:
        - end
      region: flash_primary
      size: 0x6000
    sram_primary:
      address: 0x20000000
      end_address: 0x20040000
      region: sram_primary
      size: 0x40000
    


    And I followed your steps of manually creating the partition and adjusting it's address and flashing it separately. But I still get the error when attempting to mount, (the mounting procedure I havn't changed from my post earlier)

    Error:
    *** Booting nRF Connect SDK v3.5.99-ncs1-1 ***
    Sample program to r/w files on littlefs
    I: LittleFS version 2.5, disk version 2.0
    I: FS at flash-controller@4001e000:0xfa000 is 6 0x1000-byte blocks with 512 cycle
    I: sizes: rd 16 ; pr 16 ; ca 64 ; la 32
    E: WEST_TOPDIR/modules/fs/littlefs/lfs.c:1234: Corrupted dir pair at {0x1, 0x0}
    W: can't mount (LFS -84); formatting
    I: /lfs1 mounted
    /lfs1 mount: 0
    /lfs1: bsize = 16 ; frsize = 4096 ; blocks = 6 ; bfree = 4
    
    Listing dir /lfs1 ...
    I: /lfs1 unmounted
    /lfs1 unmount: 0


    How are you mounting the filesystem and validating it?

    Best Regards,

    Wade

  • Hi Wade,

    I only verified that I was able to mount the partition and that it found the file I added (based on log output). I did not make any changes to the sample code. Attached below is the project I used. Could you please try that just to see if we get the same result?

    6675.littlefs.zip

    Best regards,

    Vidar

  • Hrm yeah I'm seeing the same thing still on it's initial boot

    Area 3 at 0xf8000 on flash-controller@4001e000 for 32768 bytes
    I: LittleFS version 2.5, disk version 2.0
    I: FS at flash-controller@4001e000:0xf8000 is 8 0x1000-byte blocks with 512 cycle
    I: sizes: rd 16 ; pr 16 ; ca 64 ; la 32
    E: WEST_TOPDIR/modules/fs/littlefs/lfs.c:1234: Corrupted dir pair at {0x1, 0x0}
    W: can't mount (LFS -84); formatting



    I just built it plain board target: nrf52840dk_52840 and prj.conf, Are you just using the standard default storage partition that's in internal?

    Then flashed it, and then flashed the littlefs.hex, 

    I tried merging them as well but no luck.

    Best Regards,
    Wade

  • Strange. I'm testing with the nRF52840 DK as the build target and using the default storage partition. This is the log I get:

    *** Using Zephyr OS v3.6.99-100befc70c74 ***
    Sample program to r/w files on littlefs
    Area 1 at 0xfa000 on flash-controller@4001e000 for 24576 bytes
    I: LittleFS version 2.8, disk version 2.1
    I: FS at flash-controller@4001e000:0xfa000 is 6 0x1000-byte blocks with 512 cycle
    I: sizes: rd 16 ; pr 16 ; ca 64 ; la 32
    /lfs mount: 0
    /lfs: bsize = 16 ; frsize = 4096 ; blocks = 6 ; bfree = 3
    
    Listing dir /lfs ...
    [FILE] boot_count (size = 1)
    [FILE] hello.txt (size = 0)
    [FILE] pattern.bin (size = 547)
    /lfs/boot_count read count:1 (bytes: 1)
    /lfs/boot_count write new boot count 2: [wr:1]
    ------ FILE: /lfs/pattern.bin ------
    

    And here is the hex file:

    merged_ltfs.hex

    Please try programming the hex file above as well.

    Best regards,

    Vidar

Related