using SMP service with file system (lfs)

[ziv123]

hello Nordic

i am using nrf52832 and nrf52840, working with ncs v2.3.0

currently i am using the SMP service for DFU but i also want to use the service's property to work directly with file system.

1. i saw a mention about security but if i have a secure connection (by second stage pairing), i am guessing the risk is not there right ?

2. in the following link and other places i looked i saw only read and wright commands, no delete option so my question is, what if i want to update/rewrite a file or, delete a file from file system, with the SMP service/ mcumgr, how can it be achieved ?

https://developer.nordicsemi.com/nRF_Connect_SDK/doc/latest/zephyr/services/device_mgmt/smp_groups/smp_group_8.html

https://developer.nordicsemi.com/nRF_Connect_SDK/doc/latest/zephyr/samples/subsys/mgmt/mcumgr/smp_svr/README.html

3. is there a minimal block/frame size to read/write to lfs, no only in limitation but in what size of frames is the data written to lfs and is it configurable ? 

 hope to read from you soon

best regards

Ziv

[Hung Bui]

Hi Ziv, 

1.There is no security feature implemented in SMP. You would  need to implement your own code on top of the SMP service for that. However as you already pointed out you can use general Bluetooth pairing to prevent unauthorized access. As long as you can perform the pairing in a secured way (level 3 or 4) the link is encrypted (AES 128) and the access is authorized. 

2. You would need to use file upload to overwrite the existing file. https://developer.nordicsemi.com/nRF_Connect_SDK/doc/latest/zephyr/services/device_mgmt/smp_groups/smp_group_8.html#file-upload

3. I'm not so sure. You may need to do your own search. But as far as I know it should be the multiple of 64. 

[ziv123]

hi Hung 

Hung Bui said:

You would need to use file upload to overwrite the existing

ok, just to better understand, in file system, isn't it deleting an area in flash before writing on it ? is it happening under the hood ? and do i have some why to only delete a file using the mcumgr/SMP service API ?

Hung Bui said:

3. I'm not so sure. You may need to do your own search. But as far as I know it should be the multiple of 64. 

i actually did not find so this is why i asked. what stands behind this question is to know how to better manage, divide or connect data to a file to be efficient, i would consider to have structs of data close to the size of a file frame in the lfs. for that i need to understand what is the minimal size block for a file 

hope to read from you soon

best regards

Ziv 

[Hung Bui]

Hi Ziv, 
I have little knowledge of littlefs but quite often a file system will not erase the area of flash when you delete a file or when you overwrite a file. Usually a file system will just invalid the file marking as it's deleted and then write a new record and mark it as the new location for the file. 

This way the flash will be erased less often and the whole block/page only get erased when needed (garbage collection). 

Most likely the minimum size is 128: https://github.com/littlefs-project/littlefs/issues/264

[ziv123]

hi Hung

though this thread is closed and i opened a new one, i think it is also related so this is the new thread

 using smp to download/upload files from fs is extremely slow in compare to custom service 

there is something really strange in the way the lfs behaves with all the extra reads, every time a file is open and not just then.. hope you can take a look and explain why it behaves like it does and how can i fix it 

best regards

Ziv

[Hung Bui]

Hi Ziv, 
I would suggest to continue on the ticket with AHaug. 
As file system it's normal that it has to do more reading than a simple one single read if you want to get the data out. 
It's because the system need to find where the file is located. Quite often that it need to read the whole database's records to find the file ID and then read it. 

[ziv123]

hi Hung

Hung Bui said:

It's because the system need to find where the file is located

is it not saved on some map for file name and address or ptr in a small area in flash, seems very wastful other wise (also considering that up until ncs v2.5.0 fs upload/download using smp requires to open and close the file on each fs_mgmt event

Hung Bui said:

As file system it's normal that it has to do more reading than a simple one single read if you want to get the data out. 

some overhead is reasonable but i am talking about a serious over head and the reading of all the file happen also at the write operation, after the file was already opened which does not make any sense, first because it is upload and not download and second cause it will get only few bytes to write in each fs_mgmt_upload event so how come it reads all the file again (after opening) ??

(i added some prints in ncs "fs_mgmt_write" and  as can be seen in the code here , and added part of the rtt log as a file)

/cfs-file/__key/communityserver-discussions-components-files/4/4456.canary_5F00_upload.txt

static int fs_mgmt_write(const char *path, size_t offset,
			 const void *data, size_t len)
{
	struct fs_file_t file;
	int rc;
	size_t file_size = 0;
	LOG_INF("in fs_mgmt_write");
	if (offset == 0) {
		rc = fs_mgmt_filelen(path, &file_size);
	}
	LOG_INF("in fs_mgmt_write call fs_file_t_init");
	fs_file_t_init(&file);
	LOG_INF("in fs_mgmt_write call fs_open");
	rc = fs_open(&file, path, FS_O_CREATE | FS_O_WRITE);
	if (rc != 0) {
		return MGMT_ERR_EUNKNOWN;
	}

	if (offset == 0 && file_size > 0) {
		/* Offset is 0 and existing file exists with data, attempt to truncate the file
		 * size to 0
		 */
		rc = fs_truncate(&file, 0);

		if (rc == -ENOTSUP) {
			/* Truncation not supported by filesystem, therefore close file, delete
			 * it then re-open it
			 */
			fs_close(&file);

			rc = fs_unlink(path);
			if (rc < 0 && rc != -ENOENT) {
				return rc;
			}
			LOG_INF("in fs_mgmt_write second call fs_open");
			rc = fs_open(&file, path, FS_O_CREATE | FS_O_WRITE);
		}

		if (rc < 0) {
			/* Failed to truncate file */
			return MGMT_ERR_EUNKNOWN;
		}
	} else if (offset > 0) {
		LOG_INF("in fs_mgmt_write call fs_seek");
		rc = fs_seek(&file, offset, FS_SEEK_SET);
		if (rc != 0) {
			goto done;
		}
	}
	LOG_INF("in fs_mgmt_write call fs_write");

	rc = fs_write(&file, data, len);
	if (rc < 0) {
		goto done;
	}

* File operations */
int fs_open(struct fs_file_t *zfp, const char *file_name, fs_mode_t flags)
{
	struct fs_mount_t *mp;
	int rc = -EINVAL;

	if ((file_name == NULL) ||
			(strlen(file_name) <= 1) || (file_name[0] != '/')) {
		LOG_ERR("invalid file name!!");
		return -EINVAL;
	}

	if (zfp->mp != NULL) {
		return -EBUSY;
	}
	LOG_INF("in fs_open call fs_get_mnt_point");

	rc = fs_get_mnt_point(&mp, file_name, NULL);
	if (rc < 0) {
		LOG_ERR("mount point not found!!");
		return rc;
	}

	if (((mp->flags & FS_MOUNT_FLAG_READ_ONLY) != 0) &&
	    (flags & FS_O_CREATE || flags & FS_O_WRITE)) {
		return -EROFS;
	}

	CHECKIF(mp->fs->open == NULL) {
		return -ENOTSUP;
	}
	LOG_INF("in fs_open call mp->fs->open");
	zfp->mp = mp;
	rc = mp->fs->open(zfp, file_name, flags);
	if (rc < 0) {
		LOG_ERR("file open error (%d)", rc);
		zfp->mp = NULL;
		return rc;
	}

	/* Copy flags to zfp for use with other fs_ API calls */
	zfp->flags = flags;

	return rc;
}

best regards

Ziv

[Hung Bui]

Hi Ziv, 
I don't have the insight on how exactly the file system in Zephyr works. But keeping a map file that need to be updated whenever you have a new file may wear the flash area that the map file is located. 

Please be aware that we don't make the file system in Zephyr. I would suggest to check on Zephyr support channel as well. 
Please continue in the ticket with Andreas.