FOTA download breaking up

Product: NRF9160

SDK: 2.6.1

Modem: 1.3.6

My problem is that FOTA download is breaking up more or less every time our device is doing a SW upgrade. We download SW directly from AWS S3 bucket using download client. But that S3 is a pain to debug so instead I installed nginx to a test server and made a small script to offer HTTP file download using nodejs and express. It looks like I can reproduce similar problems there. My binary ATM is huge(300kB) and connection seems to break at least 3 times while it is being downloaded. Here is what the device logs look like

[00:00:53.951,507] <inf> download_client: Downloaded 100352/310339 bytes (32%)
[00:00:53.964,050] <dbg> download_client: download_thread: Receiving up to 2048 bytes at 0x2000f4d0...
[00:00:54.280,944] <dbg> download_client: handle_received: Read 1391 bytes from socket
[00:00:54.281,158] <dbg> download_client: http_header_parse: GET header size: 367
[00:00:54.282,226] <dbg> download_client: http_parse: Copying 1024 payload bytes
[00:00:54.282,318] <inf> download_client: Downloaded 101376/310339 bytes (32%)
[00:00:54.297,698] <dbg> download_client: download_thread: Receiving up to 2048 bytes at 0x2000f4d0...
[00:00:54.600,982] <dbg> download_client: handle_received: Read 1386 bytes from socket
[00:00:54.601,165] <dbg> download_client: http_header_parse: GET header size: 362
[00:00:54.601,318] <wrn> download_client: Peer closed connection, will re-connect
[00:00:54.601,348] <dbg> download_client: http_parse: Copying 1024 payload bytes
[00:00:54.601,440] <inf> download_client: Downloaded 102400/310339 bytes (32%)
[00:00:54.612,854] <inf> download_client: Reconnecting...
[00:00:54.614,135] <dbg> download_client: client_connect: Protocol not specified, defaulting to HTTP(S)
[00:00:54.614,166] <dbg> download_client: client_connect: Port not specified, using default: 443
[00:00:54.735,382] <wrn> download_client: Failed to resolve hostname our.test.server.com on IPv6
[00:00:54.739,288] <dbg> download_client: client_socket_connect: family: 1, type: 1, proto: 258
[00:00:54.739,685] <inf> download_client: Setting up TLS credentials, sec tag count 1
[00:00:54.739,807] <inf> download_client: Connecting to 123.456.789.123
[00:00:54.739,837] <dbg> download_client: client_socket_connect: fd 1, addrlen 8, fam IPv4, port 443
[00:00:56.220,733] <dbg> download_client: download_thread: Receiving up to 2048 bytes at 0x2000f4d0...
[00:00:56.510,192] <dbg> download_client: handle_received: Read 1391 bytes from socket
[00:00:56.510,406] <dbg> download_client: http_header_parse: GET header size: 367
[00:00:56.511,474] <dbg> download_client: http_parse: Copying 1024 payload bytes
[00:00:56.511,566] <inf> download_client: Downloaded 103424/310339 bytes (33%)

The funny(not really) thing is that now when I am actually able to get pcaps from this server setup I would like to accuse the server(nginx) of breaking up the connection with no reason.

I have tried many ways to simulate a slow connection and mimic the partial download that nrf9161 does against AWS S3 and that test server, but so far no luck, everything just always works. So I thought that maybe you guys have some more ideas what to try. Maybe there are some known incompatibility issues that I would need to workaround?

And I would still emphase that from the nrf91 point of view the problem looks identical to what we see in real environment in AWS S3, but in reality I have no idea whether the problem I see in this test server is the same problem or not.

I can see that   had pretty similar issues lately with another product, but since I have a TLS encryption used here using 4kB fragments is not really an option.
devzone.nordicsemi.com/.../download-client-via-aws-fota-stops-at-50-and-can-t-reconnect-to-aws

  • Hey Michal
    I am on sdk v2.6.1, after making changes in the download_client library also the downloading stops after every 204800 bytes (my DOWNLOAD_CLIENT_BUF_SIZE=2048), and while trying to reconnect also fails

    [00:01:43.639,343] <wrn> download_client: Peer closed connection, will re-connect
    [00:01:43.639,373] <dbg> download_client: download_thread: Receiving up to 2048 bytes at 0x2000095c...
    [00:01:44.354,797] <dbg> download_client: handle_received: Read 2048 bytes from socket
    [00:01:44.354,827] <inf> download_client: Downloaded 204800/555880 bytes (36%)
    
    [ 36% ] |================== | (204800/555880 bytes)[00:01:44.355,529] <inf> download_client: Reconnecting...
    [00:01:44.370,086] <dbg> download_client: client_connect: Port not specified, using default: 443
    [00:01:44.370,208] <dbg> download_client: host_lookup: Failed to resolve hostname s3.amazonaws.com on IPv6
    [00:01:44.393,005] <dbg> download_client: client_socket_connect: family: 1, type: 1, proto: 258
    11122[00:01:44.393,707] <inf> download_client: Setting up TLS credentials, sec tag count 1
    [00:01:44.393,798] <inf> download_client: Connecting to 52.217.125.16
    [00:01:44.393,829] <dbg> download_client: client_socket_connect: fd 9, addrlen 8, fam IPv4, port 443
    [00:01:44.394,287] <err> download_client: Unable to connect, errno 2
    [00:01:44.394,622] <err> fota: Error -2 during download
    
    [00:01:44.399,963] <inf> fota: Network disconnected
    [00:01:44.418,762] <dbg> download_client: set_state: state = 0
    [00:01:44.418,792] <inf> fota: Socket closed
    
    [00:01:44.418,792] <err> fota: Error -117 during download

  • Hello,

    Please make a new ticket and please also provide us with your Modem FW version and a modem trace.

    Best regards,

    Michal

Related