HTTP transfer of large files

[BillH0]

We are aiming to replace our current LTE module with the nRF93M1 and want to maintain our ability to upload/download files in the region of 200-500KB. Our current module has a 600MB flash filing system which can be used as a 'staging' area for the transfers by the provided http client. This removes the need for our limited resource micro to have to deal with streaming of such large quantities of data - it simply deals instead with the module's FFS. I was hoping to replicate this strategy with the nRF93M1 but can find no data on the size of its FFS (i.e. in Data sheet, AT Command guide). However experimentation suggests that it may only be around 10KB - is this the case? If so then the FFS strategy is probably not practical and I guess I will have to attempt multipart streaming despite the limitations of our micro. Are there any application notes outlining how the FFS can be used for http file transfers and limitations involved?

[Amanda Hsieh]

Hi, 

The file system partition (littlefs) has a fixed size of 48 KB with 4KB blocks on the nRF93M1. It is shared with the system files, which take up around half of the space. In practice, customers have around ~24 KB of space to use. 

Starting from MFW 1.5.0, the AT command %FSINFO can show FS size, used and free space.

Regards,
Amanda H.

[BillH0]

Thanks Amanda, my module has 1.4.1 and doesn't respond to the %FSINFO command. But anyway 24KB would probably not be enough for our current file transfer strategy and so we will look at a different approach that doesn't require using the FFS.

Hence another question . . .

The HTTP client has a command %HTTPGETEX that seems to support getting/downloading a file in small parts using multiple requests. Presumably we could use this to download a file without needing FFS. Does the HTTP client have any similar command to support uploading a file in multiple small parts or would we have to do this ourselves with standard HTTP puts using our own headers? (Must admit I can't find one in the AT command guide but thought worth asking just in case I'm missing something).

[Amanda Hsieh]

You can check 

HTTP POST request %HTTPPOST, and see the set command. 

HTTP POST request from file %HTTPPOSTFILE, and see the set command. 

Since those commands are supported by MFW v1.5.x, you can download MFW v1.5.0 via this page to update. 

[BillH0]

Hi Amanda, I was envisaging that it would be convenient and keep my code lightweight if for uploading the http client provided a command equivalent to %HTTPGETEX, for example maybe %HTTPPOSTEX. If something goes wrong when uploading a file using multiple small part requests to my server then I would need to resume at the part that fails, i.e. specify a file start position. But %HTTPPOST doesn’t ask for this on its own. However I realise I can achieve what I describe using %HTTPPOST and my own custom headers if this is what you meant?.

Bill H

[Amanda Hsieh]

Nothing like HTTPPOSTEX in the SDK / mFW. The current way of doing that would be something like:

AT%HTTPCFG="requestheader",1
AT%HTTPPOST=<total_len>,60
CONNECT
POST /upload HTTP/1.1\r\n
Host: example.com\r\n
Content-Type: application/octet-stream\r\n
Content-Range: bytes 8192-12287/50000\r\n
Content-Length: 4096\r\n
\r\n
<part bytes>
OK
AT%HTTPREAD

[BillH0]

Thanks Amanda. Yes that's the route I'm currently going. What isn't clear from the documentation is whether <part bytes> can be binary bytes like 0x00,0x11,0x22 or if they must be converted to hexadecimal ascii 0x30,0x30,0x31,0x31,0x32,0x32 as they are for socket commands. This would almost double the size of the request. But I guess I'll discover the answer to this with my own experimentation