usbd_msc FATFS read/write pointer question (also littlefs)

[MichielSch]

Hi!

I am currently looking into using a file system to store logged data as files in my QSPI flash memory. To get started I looked around on the devzone and found the usbd_msc example. This works quite well so far. My question however is: I would like to add some sort of metadata to the top of my file. In this metadata I would like to have some fields to indicate how many the files has been opened or if it has been synced with my BLE app. To achieve this I want to read this data first, modify it and write it again. I have noticed that if I read the data and try to overwrite the metadata with modified data, the new data gets written a bit lower in the file. I did some debugging and I noticed the file pointer contains a read/write pointer (fptr) that is modified after the read which probably explains why I am not starting to write at the top of the file.

/* File object structure (FIL) */

typedef struct {
	_FDID	obj;			/* Object identifier (must be the 1st member to detect invalid object pointer) */
	BYTE	flag;			/* File status flags */
	BYTE	err;			/* Abort flag (error code) */
	FSIZE_t	fptr;			/* File read/write pointer (Zeroed on file open) */
	DWORD	clust;			/* Current cluster of fpter (invalid when fprt is 0) */
	DWORD	sect;			/* Sector number appearing in buf[] (0:invalid) */
#if !_FS_READONLY
	DWORD	dir_sect;		/* Sector number containing the directory entry */
	BYTE*	dir_ptr;		/* Pointer to the directory entry in the win[] */
#endif
#if _USE_FASTSEEK
	DWORD*	cltbl;			/* Pointer to the cluster link map table (nulled on open, set by application) */
#endif
#if !_FS_TINY
	BYTE	buf[_MAX_SS];	/* File private data read/write window */
#endif
} FIL;

So to avoid this issue should I modify the value of this pointer manually. As in set this pointer to zero before my write operation? Or should I close and open the file again to reset this value (seems like it would be slow). I wonder what the rationale is in having the read and the write pointer as the same thing, but  I'm not by any means a file system expert.

As a side question I was wondering if anybody has had good experiences using littlefs as an alternative to FAT on the nRF52840? I understand that it is more resilient to power drops and that it has basic wear levering. Certainly the wear leveling might be interesting for my application. How is the performance compared to the FAT example? I am less worried about the space it taking up in my external flash. The code footprint, RAM usage and speed are more critical to me.

Cheers!

Michiel

[Simonr]

Hi Michiel

I don't have any experience with the littlefs file system, so I'll leave this question open for any potential experts amongst other DevZone users.

As for the pointer predicament, let me double check this with an expert internally on this, and I'll get back to you tomorrow with a decisive answer. I have a hunch that both of the methods you suggest should be fine, and that it's up to you what you do.

Best regards,

Simon

[MichielSch]

Hi Simon,

Thanks for your reply! For now I will manually adjust it and i look forward to the decisive answer later on. Setting it manually to zero by

file.fptr = 0;

actually works quite well it seems.

Thanks again,

Best regards,

Michiel

[Simonr]

Hi Michiel

Indeed, how you implement this is up to you, if it's working well for you I see no reason this should be an issue. No one here at Nordic are really "experts" in the FatFS field, but find it a bit strange that you want to do this by editing the file system itself instead of storing this metadata in its own file or part of a file?

In order to maximize the flash lifetime it is generally not a good idea to edit a file every time you open it, as that will cause you to reach the maximum number of flash writes/reads that the flash is specced for much faster.

Best regards,

Simon

[MichielSch]

Hi Simon,

Thanks for the update.

In my case I (think) will only have to update the metadata once or twice before it will get deleted. Basically the changes will be to mark it as synchronized and ready for removal. I also considered having this information in file with sort of a table that contains this information for all the files since in that case I would be able to only read one file in my "clean up" task and remove the old files. The issue in my head with that approach would be that the place in memory containing that file (with the table) would be worn pretty hard compared to the rest of the flash.

Simonr said:

but find it a bit strange that you want to do this by editing the file system itself instead of storing this metadata in its own file or part of a file?

This I don't quite get, why would I change the file system? The metadata would be in the first few lines of the files themselves.

Thanks for the input, it's really helpful getting your point of view.

Best regards,

Michiel

[Simonr]

Hi again

Sounds like a good approach to me, but yes, this part of the flash will likely be worn out before the rest. 

Our expert found it interesting/concerning that you modified the FatFS files/libraries instead of adding a pointer like this yourself to the file you're using. 

Either way, a process like this will take a toll on the flash memory's lifetime I'd imagine.

Best regards,

Simon