FOTA download breaking up

RE: FOTA download breaking up

Michal — Thu, 05 Sep 2024 10:17:51 GMT

Hello,

Please make a new ticket and please also provide us with your Modem FW version and a modem trace.

Best regards,

Michal

RE: FOTA download breaking up

Rakshita — Wed, 04 Sep 2024 08:48:55 GMT

Hey Michal
I am on sdk v2.6.1, after making changes in the download_client library also the downloading stops after every 204800 bytes (my DOWNLOAD_CLIENT_BUF_SIZE=2048), and while trying to reconnect also fails

[00:01:43.639,343] <wrn> download_client: Peer closed connection, will re-connect
[00:01:43.639,373] <dbg> download_client: download_thread: Receiving up to 2048 bytes at 0x2000095c...
[00:01:44.354,797] <dbg> download_client: handle_received: Read 2048 bytes from socket
[00:01:44.354,827] <inf> download_client: Downloaded 204800/555880 bytes (36%)

[ 36% ] |================== | (204800/555880 bytes)[00:01:44.355,529] <inf> download_client: Reconnecting...
[00:01:44.370,086] <dbg> download_client: client_connect: Port not specified, using default: 443
[00:01:44.370,208] <dbg> download_client: host_lookup: Failed to resolve hostname s3.amazonaws.com on IPv6
[00:01:44.393,005] <dbg> download_client: client_socket_connect: family: 1, type: 1, proto: 258
11122[00:01:44.393,707] <inf> download_client: Setting up TLS credentials, sec tag count 1
[00:01:44.393,798] <inf> download_client: Connecting to 52.217.125.16
[00:01:44.393,829] <dbg> download_client: client_socket_connect: fd 9, addrlen 8, fam IPv4, port 443
[00:01:44.394,287] <err> download_client: Unable to connect, errno 2
[00:01:44.394,622] <err> fota: Error -2 during download

[00:01:44.399,963] <inf> fota: Network disconnected
[00:01:44.418,762] <dbg> download_client: set_state: state = 0
[00:01:44.418,792] <inf> fota: Socket closed

[00:01:44.418,792] <err> fota: Error -117 during download

RE: FOTA download breaking up

Achim Kraus — Tue, 13 Aug 2024 09:03:25 GMT

Just in the case:

Do you use a SIM card with "rate limit"?

See NCS 2.4.1- cgev apnratectrl stat - not reported with pdn_event_handler_t

RE: FOTA download breaking up

Michal — Mon, 12 Aug 2024 23:36:34 GMT

[quote user="fastfox"]This turned out to be a wild goose chase. Why I see this breaking up is most probably because of a bug in download client here: https://github.com/nrfconnect/sdk-nrf/blob/main/subsys/net/lib/download_client/src/http.c#L223[/quote]

Thank you very much for the information, I will report that and hopefully it will get fixed in the next release.

[quote user="fastfox"]But the goose chase was because while debugging this I might have bumped into the real problem that I was hunting at. It looks like sometimes the lower layers just die and nothing goes through anymore. Signal quality is fine so this will be a bit trickier to hunt down. I don't think I can even get the modem traces out from that custom HW. But this is another story[/quote]

That does sound like something that may be hard for us to debug without a modem trace unfortunately...

Best regards,

Michal

RE: FOTA download breaking up

fastfox — Fri, 09 Aug 2024 11:45:33 GMT

This turned out to be a wild goose chase. Why I see this breaking up is most probably because of a bug in download client here: https://github.com/nrfconnect/sdk-nrf/blob/main/subsys/net/lib/download_client/src/http.c#L223

p = strnstr(client->buf, "\r\nconnection: close", sizeof(client->buf));

should probably say

p = strnstr(client->buf, "\r\nconnection: close", *hdr_len);

Now it examines if garbage or binary content happens to have that text there and it usually does if you use un-encrypted binary that has exactly that code line in it somewhere. Feel free to create a bug Michal

But the goose chase was because while debugging this I might have bumped into the real problem that I was hunting at. It looks like sometimes the lower layers just die and nothing goes through anymore. Signal quality is fine so this will be a bit trickier to hunt down. I don't think I can even get the modem traces out from that custom HW. But this is another story

RE: FOTA download breaking up

fastfox — Fri, 09 Aug 2024 09:06:27 GMT

One funny observation from this is that the download seems to break at the same binary offset. Like below with 200 and 300kB files:

[00:00:56.926,757] <inf> download_client: Downloaded 101376/227227 bytes (44%)
[00:00:57.152,160] <wrn> download_client: Peer closed connection, will re-connect
[00:00:57.152,252] <inf> download_client: Downloaded 102400/227227 bytes (45%)
[00:00:57.163,726] <inf> download_client: Reconnecting...

...

[00:01:45.487,915] <inf> download_client: Downloaded 203776/227227 bytes (89%)
[00:01:45.727,203] <wrn> download_client: Peer closed connection, will re-connect
[00:01:45.727,416] <inf> download_client: Downloaded 204800/227227 bytes (90%)
[00:01:45.738,800] <inf> download_client: Reconnecting...

[00:01:00.961,517] <inf> download_client: Downloaded 101376/310339 bytes (32%)
[00:01:01.291,076] <wrn> download_client: Peer closed connection, will re-connect
[00:01:01.291,168] <inf> download_client: Downloaded 102400/310339 bytes (32%)
[00:01:01.302,551] <inf> download_client: Reconnecting...

...

[00:01:38.770,080] <inf> download_client: Downloaded 203776/310339 bytes (65%)
[00:01:38.996,246] <wrn> download_client: Peer closed connection, will re-connect
[00:01:38.996,368] <inf> download_client: Downloaded 204800/310339 bytes (65%)
[00:01:39.007,781] <inf> download_client: Reconnecting...

Maybe I'll try to sniff the flashwriting a bit. Maybe it causes some unwanted side effects somehow

RE: FOTA download breaking up

Michal — Wed, 07 Aug 2024 21:51:30 GMT

Thank you for the clarification. I will check internally about any possible fixes.

Best regards,

Michal

RE: FOTA download breaking up

fastfox — Wed, 07 Aug 2024 05:14:24 GMT

Should have written NRF9160. Sorry. Fixed to the ticket

RE: FOTA download breaking up

Michal — Tue, 06 Aug 2024 14:53:27 GMT

Hello,

[quote user=""]

Product: NRF9161

SDK: 2.6.1

Modem: 1.3.6

[/quote]

Are you sure this is correct? nRF9161 should be using 2.x.x MFW version:

https://www.nordicsemi.com/Products/nRF9161/Download?lang=en#infotabs

Best regards,

Michal

RE: FOTA download breaking up

Achim Kraus — Tue, 06 Aug 2024 10:37:14 GMT

That depends ...

If you just want to check, if it really works as I promise, you may just use my Zephyr - CoAP/DTLS 1.2 CID demo client and the Californium - CoAP-S3-proxy . I'm currently updating the client, hope that's done until the end of week. For the server you may download it from

https://repo.eclipse.org/content/repositories/californium-releases/org/eclipse/californium/cf-s3-proxy-server/3.12.1/cf-s3-proxy-server-3.12.1.jar

The first development version with FOTA is now about a year available. As I wrote, the download itself works pretty reliable, only some mistakes from my side caused recently mcuboot issues.

If you want to use that approach for your app, then I guess it will be more work ... Maybe the open sources of my above demo client helps. Or you consider to test it with your next project or product.

RE: FOTA download breaking up

fastfox — Tue, 06 Aug 2024 10:27:54 GMT

That might very well be an option. We have californium running there already anyway. At least that would help in the ridiculous header/payload ratio that we now have. Something like 400B/1024B.

Only thing preventing to do that is me being lazy as I suspect that it would take weeks to get that done and tested properly.

RE: FOTA download breaking up

Achim Kraus — Tue, 06 Aug 2024 10:22:04 GMT

I guess, that's no option for you, but using CoAP/DTLS 1.2 CID and the Californium CoAP-S3-proxy my FOTA success rate is about 99% (370 kB) :-).

(At least, if I don't accidentally change the flash partitions and kill mcuboot. Then only the download works reliable, but the update itself fails ;-) ):