fs_write freezes app for 100+ ms

[IulianI]

Hi,
    On nRF52840, on example https://github.com/zephyrproject-rtos/zephyr/tree/main/samples/subsys/mgmt/mcumgr/smp_svr bluetooth stack configurations

    I am calling fs_write( less than 512bytes) and I have the main thread frozen for ~130ms.

    I tried calling fs_write from a different thread and I have now the main thread frozen for ~90ms:

static void saveFlashThread(void* p1, void* p2, void* p3)
{
    while(true)
    {
        k_msleep(10);

        if(flag)
        {
            flag = false;

            debugPinHighLow();
            fs_open(&file, path, FS_O_WRITE); 
            fs_write(&file, array, size);
            fs_close(&file);
            debugPinHighLow();
        }
    }
}

K_THREAD_DEFINE(saveFlashThreadId, 1024, saveFlashThread, NULL, NULL, NULL, 7, 0, 0);



static void mainThread()
{
...
    flag=true;
...

    while(true)
    {
        k_msleep(5);
        debugPinToggle();
        
        ...
    }
}

   Please kindly advise, you know of an asynchronous flash write function? Or is other way to write the thread so that fs_write() will not cause freeze in other threads?

   





  The curious thing is that only part of fs_write call is blocking: may you know if the low level function code is public? If yes, where can be found for board nRF52840?

ssize_t fs_write(struct fs_file_t *zfp, const void *ptr, size_t size)
{
...
    rc = zfp->mp->fs->write(zfp, ptr, size);



fs_file_system_t  /* File operations */
{
...
    ssize_t (*write)(struct fs_file_t *filp, const void *src, size_t nbytes);

Kind Regards,
Iulian

MSc.MEd. Software Engineer and Trainer
"But God demonstrates His own love for us in this: While we were still sinners, Christ died for us." (Romans 5:8)

[Sigurd Hellesvik]

Hi,

For background on persistent storage, see  Persistent storage of keys and data using the nRF Connect SDK .

fs_write() depends on the backend used. Which backend do you use for your file system?

Which overlays do you use to build the smp_srv sample?

Regards,
Sigurd Hellesvik

[IulianI]

Hi Sigurd,
   Thank you: let me look into the link you sent

   I use the zephyr/fs/littlefs one:

 .dts file
&flash {
	partitions {
		compatible = "fixed-partitions";
		#address-cells = <1>;
		#size-cells = <1>;

        ...
	
		storage_partition: partition@XXXX {
			label = "storage";
			reg = <0xXXXX 0xXXXX>;
		};
	};
};



 main.c file

#ifdef CONFIG_MCUMGR_CMD_FS_MGMT
FS_LITTLEFS_DECLARE_DEFAULT_CONFIG(storage);
static struct fs_mount_t littlefs_mnt = {
    .type        = FS_LITTLEFS,
    .fs_data     = &storage,
    .storage_dev = (void*)FIXED_PARTITION_ID(storage_partition),
    .mnt_point   = "/lfs"
};
#endif

static struct fs_file_t file;

void main(void)
{
    /* Register the built-in mcumgr command handlers. */
#ifdef CONFIG_MCUMGR_CMD_FS_MGMT
    fs_mgmt_register_group();

    int rc = fs_mount(&littlefs_mnt);
    if(rc < 0)
    {
        LOG_ERR("Error mounting littlefs [%d]", rc);
    }
#endif

    fs_file_t_init(&file);
    fs_open(&file, path, FS_O_CREATE);
    fs_close(&file);
    
    ...
}

  I use the nrf52840dk_nrf52840.overlay


Kind Regards,
 Iulian

[Sigurd Hellesvik]

In that case, the fs_write function is declared here: https://github.com/nrfconnect/sdk-zephyr/blob/434be7b575ce338645801140fbd2e64738f8994e/subsys/fs/littlefs_fs.c#L382

Have you seen the overlay-fs.conf as well?

[IulianI]

Hi Sigurd,
  Did not manage to reach far as had to jump on another task and when get back have to decide if to continue this path or look for alternative.

  The question is, on nrf52840, is possible to save 500 bytes to flash in a separate thread without freezing other threads at all (or no more than 2 milliseconds)?

Kind Regards,
Iulian

[Sigurd Hellesvik]

IulianI said:

  The question is, on nrf52840, is possible to save 500 bytes to flash in a separate thread without freezing other threads at all (or no more than 2 milliseconds)?

I looked more into it, and I see that the CPU pauses when writing to flash.

So if you have timing requirements, you could write less data at a time when writing to flash.
Here you can see how much time it takes to write a 32 bit word to flash:

Now how the data is written depends on how your file system does the writing: Does it write all with one command or with multiple?
So maybe you have to just reduce the size you write at once.
Or if you give the fs write thread low priority, and the file system writes in multiple blocks, the scheduler will fix the rest?

An alternative could always be to use the NVMC drivers to do this manually in a low-prio thread to make sure you do not spend too much time.

[IulianI]

Hi Sigurd,
  41us x my ~500 bytes = ~5ms: so then the rest ~100ms freeze might be caused just by erase?
  Is the lowest erase amount one page 85ms as per datasheet? So even with advised low priority threads, if the CPU pauses, my whole app will be frozen at least 85ms inevitably?

  If you have at hand a tiny example with ~500bytes erase + write (even in multiple calls if needed) for nRF52840 using bluetooth stack, please kindly advise the smallest app freeze possible in milliseconds you get on your side (kindly share that example too)

Kind Regards,
Iulian

[Sigurd Hellesvik]

I have been digging and learning, and this is what I got.

Most of the nRF Connect SDK internal storage APIs use flash_write to write to flash.
For nRF devices, flash_write points to flash_nrf_write.
If Bluetooth Low energy is enabled, this function will use write_synchronously() if needed.
And this calls nrf_flash_sync_exe() for writing.
nrf_flash_sync_exe() is documented in soc_flash_nrf.h.

If you use the nRF SoftDecvice Controller for Bluetooth, I think nrf_flash_sync_exe() is declared in MPSL code.

You can follow the same path as I explained below for erase.

IulianI said:

  41us x my ~500 bytes = ~5ms: so then the rest ~100ms freeze might be caused just by erase?

The nRF52840 NVMC — Non-volatile memory controller can not erase less than a page(4kB) at a time, either a full page at once or by Partial erase of a page in flash. The last mentioned also erases the full page, but gives you time to do other stuff in the meantime.
Our driver can use partial erase as well, depending on CONFIG_SOC_FLASH_NRF_PARTIAL_ERASE. If we look at MPSL code, this also depends on CONFIG_SOC_FLASH_NRF_PARTIAL_ERASE. Try to set that option to see if it speeds things up for you.

If this does not help, can you try to see if you are still blocked in main if you use it in a project without Bluetooth Low Energy.

If disabling Bluetooth speeds things up, you can try the to leverage MPSL. See MPSL docs. And more specifically Timeslot:
"
The Timeslot interface allows developers to implement their own 2.4 GHz proprietary protocol that can be run concurrently with other protocols using the MPSL. The Timeslot feature gives the application access to the radio and other restricted peripherals during defined time intervals, denoted as timeslots. It can also be used to suppress radio activity and to reserve guaranteed time for application activities with hard timing requirements.
"