using SMP service with file system (lfs)

RE: using SMP service with file system (lfs)

ziv123 — Mon, 22 Jul 2024 12:19:26 GMT

hi Icj

we have added a separate follow up command from the central after it finishes the upload it will send apply command for the peripheral to read the file that was uploaded

RE: using SMP service with file system (lfs)

lcj — Fri, 19 Jul 2024 10:54:20 GMT

ziv123 did you figure this out? I want to use reuse the SMP from MCUBoot and was thinking how I would approach this. I am using littlefs as well.

RE: using SMP service with file system (lfs)

Hung Bui — Wed, 13 Dec 2023 08:12:20 GMT

Hi Ziv,

Sorry that I couldn't do much help here. But doesn't the client send a close file command after it's done uploading the file ?

The server should receive a File Close request from the client. Maybe you can inspect how the phone does a DFU update via BLE to check if the phone send such File Close request or not.

RE: using SMP service with file system (lfs)

ziv123 — Tue, 12 Dec 2023 14:13:54 GMT

hi Hung

[quote userid="2121" url="~/f/nordic-q-a/105261/using-smp-service-with-file-system-lfs/459943"]You can take a look at this guidelines to send a request or contribution to[/quote]

i'll look to it

[quote userid="2121" url="~/f/nordic-q-a/105261/using-smp-service-with-file-system-lfs/459943"]There is a Zephyr's Discord channel that you can post your question[/quote]

i know, this is actually how i know there is no such event of upload done in smp

this is why i asked :

[quote userid="90084" url="~/f/nordic-q-a/105261/using-smp-service-with-file-system-lfs/459860"]maybe you know if there is an option to register for an event of fs_close() for specific file in lfs ?[/quote]

without any regards to SMP

[quote userid="2121" url="~/f/nordic-q-a/105261/using-smp-service-with-file-system-lfs/459943"]For example file status request is Opcode 0, Group ID 8, Command ID 1[/quote]

i am familiare with that but i am looking for the server how receives the file to know when it has finished and not to poll from the client to know when it is done and send another commend for the server to read the file

if you know about the lfs that would be nice, if not then i will close this thread once i have a PR ready in zephyr with the addition of the event

best regards

Ziv

RE: using SMP service with file system (lfs)

Hung Bui — Tue, 12 Dec 2023 13:57:16 GMT

Hi Ziv,
You can take a look at this guidelines to send a request or contribution to Zephyr: https://docs.zephyrproject.org/latest/contribute/index.html#general-guidelines

There is a Zephyr's Discord channel that you can post your question : https://discord.com/invite/Ck7jw53nU2

As I mentioned I haven't worked with the SMP protocol, but I believe that what I pointed you to doesn't require you to modify the SMP protocol. What I pointed you to is part of the SMP protocol:
https://developer.nordicsemi.com/nRF_Connect_SDK/doc/latest/zephyr/services/device_mgmt/smp_protocol.html
For example file status request is Opcode 0, Group ID 8, Command ID 1

If you write your own Bluetooth SMP client you should be able to send such command and wait for the file status response back.

RE: using SMP service with file system (lfs)

ziv123 — Tue, 12 Dec 2023 08:55:20 GMT

hi Hung

it seems there is no event for upload done on the receiving side with the ncs

(i wonder how to suggest to add one to zephyr)

[quote userid="2121" url="~/f/nordic-q-a/105261/using-smp-service-with-file-system-lfs/457538"]When you upload a chunk you will receive a file upload response that tell at which offset it has received the file chunk[/quote]

not sure i understand, this part of code is in the ncs and i don't think its a good practice to add my code there.

maybe you know if there is an option to register for an event of fs_close() for specific file in lfs ?

if not, maybe i can create an event and publish it with zbus ?

what do you think ?

hope to read from you soon

best regards

Ziv

RE: using SMP service with file system (lfs)

Hung Bui — Mon, 27 Nov 2023 12:04:52 GMT

Hi Ziv,
I dont have much experience with the SMP protocol, but have you looked at the specification ?

https://developer.nordicsemi.com/nRF_Connect_SDK/doc/latest/zephyr/services/device_mgmt/smp_groups/smp_group_8.html#mcumgr-smp-group-8

When you upload a chunk you will receive a file upload response that tell at which offset it has received the file chunk.

RE: using SMP service with file system (lfs)

ziv123 — Mon, 27 Nov 2023 09:24:17 GMT

hi Hung

OK, figured out the issue with the throughput and closed the mentioned thread.

one last question, hope you can help, one i finish the transfer of the file to my file system i need the received device to read it so it has to be in formed some how, i did not see a place for give some handler or a way to register for file upload done event or something like that.

any idea how i can get notified with in the device on the upload finish ?

hope to read from you soon

best regards

Ziv

RE: using SMP service with file system (lfs)

Hung Bui — Tue, 14 Nov 2023 09:13:44 GMT

Hi Ziv,
I don't have the insight on how exactly the file system in Zephyr works. But keeping a map file that need to be updated whenever you have a new file may wear the flash area that the map file is located.

Please be aware that we don't make the file system in Zephyr. I would suggest to check on Zephyr support channel as well.
Please continue in the ticket with Andreas.

RE: using SMP service with file system (lfs)

ziv123 — Mon, 13 Nov 2023 15:50:13 GMT

hi Hung

[quote userid="2121" url="~/f/nordic-q-a/105261/using-smp-service-with-file-system-lfs/455369"]It's because the system need to find where the file is located[/quote]

is it not saved on some map for file name and address or ptr in a small area in flash, seems very wastful other wise (also considering that up until ncs v2.5.0 fs upload/download using smp requires to open and close the file on each fs_mgmt event

[quote userid="2121" url="~/f/nordic-q-a/105261/using-smp-service-with-file-system-lfs/455369"]As file system it's normal that it has to do more reading than a simple one single read if you want to get the data out. [/quote]

some overhead is reasonable but i am talking about a serious over head and the reading of all the file happen also at the write operation, after the file was already opened which does not make any sense, first because it is upload and not download and second cause it will get only few bytes to write in each fs_mgmt_upload event so how come it reads all the file again (after opening) ??

(i added some prints in ncs "fs_mgmt_write" and as can be seen in the code here , and added part of the rtt log as a file)

/cfs-file/__key/communityserver-discussions-components-files/4/4456.canary_5F00_upload.txt

static int fs_mgmt_write(const char *path, size_t offset,
			 const void *data, size_t len)
{
	struct fs_file_t file;
	int rc;
	size_t file_size = 0;
	LOG_INF("in fs_mgmt_write");
	if (offset == 0) {
		rc = fs_mgmt_filelen(path, &file_size);
	}
	LOG_INF("in fs_mgmt_write call fs_file_t_init");
	fs_file_t_init(&file);
	LOG_INF("in fs_mgmt_write call fs_open");
	rc = fs_open(&file, path, FS_O_CREATE | FS_O_WRITE);
	if (rc != 0) {
		return MGMT_ERR_EUNKNOWN;
	}

	if (offset == 0 && file_size > 0) {
		/* Offset is 0 and existing file exists with data, attempt to truncate the file
		 * size to 0
		 */
		rc = fs_truncate(&file, 0);

		if (rc == -ENOTSUP) {
			/* Truncation not supported by filesystem, therefore close file, delete
			 * it then re-open it
			 */
			fs_close(&file);

			rc = fs_unlink(path);
			if (rc < 0 && rc != -ENOENT) {
				return rc;
			}
			LOG_INF("in fs_mgmt_write second call fs_open");
			rc = fs_open(&file, path, FS_O_CREATE | FS_O_WRITE);
		}

		if (rc < 0) {
			/* Failed to truncate file */
			return MGMT_ERR_EUNKNOWN;
		}
	} else if (offset > 0) {
		LOG_INF("in fs_mgmt_write call fs_seek");
		rc = fs_seek(&file, offset, FS_SEEK_SET);
		if (rc != 0) {
			goto done;
		}
	}
	LOG_INF("in fs_mgmt_write call fs_write");

	rc = fs_write(&file, data, len);
	if (rc < 0) {
		goto done;
	}

* File operations */
int fs_open(struct fs_file_t *zfp, const char *file_name, fs_mode_t flags)
{
	struct fs_mount_t *mp;
	int rc = -EINVAL;

	if ((file_name == NULL) ||
			(strlen(file_name) <= 1) || (file_name[0] != '/')) {
		LOG_ERR("invalid file name!!");
		return -EINVAL;
	}

	if (zfp->mp != NULL) {
		return -EBUSY;
	}
	LOG_INF("in fs_open call fs_get_mnt_point");

	rc = fs_get_mnt_point(&mp, file_name, NULL);
	if (rc < 0) {
		LOG_ERR("mount point not found!!");
		return rc;
	}

	if (((mp->flags & FS_MOUNT_FLAG_READ_ONLY) != 0) &&
	    (flags & FS_O_CREATE || flags & FS_O_WRITE)) {
		return -EROFS;
	}

	CHECKIF(mp->fs->open == NULL) {
		return -ENOTSUP;
	}
	LOG_INF("in fs_open call mp->fs->open");
	zfp->mp = mp;
	rc = mp->fs->open(zfp, file_name, flags);
	if (rc < 0) {
		LOG_ERR("file open error (%d)", rc);
		zfp->mp = NULL;
		return rc;
	}

	/* Copy flags to zfp for use with other fs_ API calls */
	zfp->flags = flags;

	return rc;
}

best regards

Ziv

RE: using SMP service with file system (lfs)

Hung Bui — Mon, 13 Nov 2023 15:20:48 GMT

Hi Ziv,
I would suggest to continue on the ticket with AHaug.
As file system it's normal that it has to do more reading than a simple one single read if you want to get the data out.
It's because the system need to find where the file is located. Quite often that it need to read the whole database's records to find the file ID and then read it.

RE: using SMP service with file system (lfs)

ziv123 — Mon, 13 Nov 2023 14:56:25 GMT

hi Hung

though this thread is closed and i opened a new one, i think it is also related so this is the new thread

using smp to download/upload files from fs is extremely slow in compare to custom service

there is something really strange in the way the lfs behaves with all the extra reads, every time a file is open and not just then.. hope you can take a look and explain why it behaves like it does and how can i fix it 🙏

best regards

Ziv

RE: using SMP service with file system (lfs)

Hung Bui — Thu, 02 Nov 2023 12:35:28 GMT

Hi Ziv,
I have little knowledge of littlefs but quite often a file system will not erase the area of flash when you delete a file or when you overwrite a file. Usually a file system will just invalid the file marking as it's deleted and then write a new record and mark it as the new location for the file.

This way the flash will be erased less often and the whole block/page only get erased when needed (garbage collection).

Most likely the minimum size is 128: https://github.com/littlefs-project/littlefs/issues/264

RE: using SMP service with file system (lfs)

ziv123 — Thu, 02 Nov 2023 07:14:26 GMT

hi Hung

[quote userid="2121" url="~/f/nordic-q-a/105261/using-smp-service-with-file-system-lfs/453550"]You would need to use file upload to overwrite the existing[/quote]

ok, just to better understand, in file system, isn't it deleting an area in flash before writing on it ? is it happening under the hood ? and do i have some why to only delete a file using the mcumgr/SMP service API ?

[quote userid="2121" url="~/f/nordic-q-a/105261/using-smp-service-with-file-system-lfs/453550"]3. I'm not so sure. You may need to do your own search. But as far as I know it should be the multiple of 64. [/quote]

i actually did not find so this is why i asked. what stands behind this question is to know how to better manage, divide or connect data to a file to be efficient, i would consider to have structs of data close to the size of a file frame in the lfs. for that i need to understand what is the minimal size block for a file

hope to read from you soon

best regards

Ziv

RE: using SMP service with file system (lfs)

Hung Bui — Wed, 01 Nov 2023 15:07:28 GMT

Hi Ziv,

1.There is no security feature implemented in SMP. You would need to implement your own code on top of the SMP service for that. However as you already pointed out you can use general Bluetooth pairing to prevent unauthorized access. As long as you can perform the pairing in a secured way (level 3 or 4) the link is encrypted (AES 128) and the access is authorized.

2. You would need to use file upload to overwrite the existing file. https://developer.nordicsemi.com/nRF_Connect_SDK/doc/latest/zephyr/services/device_mgmt/smp_groups/smp_group_8.html#file-upload

3. I'm not so sure. You may need to do your own search. But as far as I know it should be the multiple of 64.