AWS FOTA using presigned URL from private S3 bucket

[Joe Boettcher]

Hello,

I've gotten the sample AWS FOTA to work on my nrf9160 using a very permissive public S3 bucket, but for production I need FOTAs to use a private S3 bucket.

According to https://devzone.nordicsemi.com/f/nordic-q-a/70753/aws-fota-download-from-s3-authentication/309614#309614, using presigned URLs is advised to allow for using a private S3 bucket. This makes sense; however, there is no documentation how to do this and I have not been able to get it to work. Would you simply append the query params to the path as shown below?

Example (failed) attempt at IoT Job using presigned URL:

{
  "operation": "app_fw_update",
  "fwversion": "v1.0.2",
  "size": 181124,
  "location": {
    "protocol": "http:",
    "host": "myfota-secure.s3.amazonaws.com",
    "path": "file_path?X-Amz-Algorithm=_____&X-Amz-Credential=____..."
  }
}

Thanks for the help

[Didrik Rokhaug]

Hi,

Yes, appending the query parameters to the file path should work, though depending on the length of the URL, you might have to increase the size of the buffer used to hold the path.

You can do this by setting CONFIG_AWS_FOTA_FILE_PATH_MAX_LEN and CONFIG_DOWNLOAD_CLIENT_MAX_FILENAME_SIZE to the required size.

The default value is 192 bytes.

Best regards,

Didrik

[Joe Boettcher]

Hi Didrik,

Thanks for the assistance.

The defaults were:

- CONFIG_AWS_FOTA_FILE_PATH_MAX_LEN = 255

- CONFIG_DOWNLOAD_CLIENT_MAX_FILENAME_SIZE = 192.

I raised those both to 500. I measured my full file path length to be about 380 characters.

Example:

https://my-secure-fota-bin.s3.us-east-1.amazonaws.com/app_update.bin?X-Amz-Algorithm=_____&X-Amz-Credential=______________________&X-Amz-Date=_____________&X-Amz-Expires=____&X-Amz-SignedHeaders=____&X-Amz-Signature=__________________________

Unfortunately, this did not fix the problem and I still get the same error as before:

00> RRC mode: Connected
00> I: Connecting to my-secure-fota-bin.s3.amazonaws.com
00> I: Downloading: app_update.bin?X-Amz-Algorithm=... [0]
00> AWS_IOT_EVT_FOTA_START
00> E: Server did not honor partial content request
00> E: Download client error
00> E: FOTA download failed, report back
00> E: AWS_FOTA_EVT_ERROR
00> AWS_IOT_EVT_ERROR, 0

Do you know what might be causing this error?

[Joe Boettcher]

These are the resulting logs after enabling the config options.

[00:06:37.776,245] <dbg> download_client.client_connect: Protocol not specified, defaulting to HTTP(S)
[00:06:37.776,275] <dbg> download_client.client_connect: Port not specified, using default: 80
[00:06:37.776,275] <dbg> download_client.client_connect: family: 1, type: 1, proto: 6
[00:06:37.776,916] <inf> download_client: Connecting to my-secure-fota.s3.amazonaws.com
[00:06:37.776,916] <dbg> download_client.client_connect: fd 2, addrlen 8, fam IPv4, port 80
[00:06:37.867,797] <err> aws_fota: Error (-7) when trying to start firmware download
AWS_IOT_EVT_FOTA_START

[00:06:38.246,826] <dbg> download_client.client_connect: Protocol not specified, defaulting to HTTP(S)
[00:06:38.246,856] <dbg> download_client.client_connect: Port not specified, using default: 80
[00:06:38.246,887] <dbg> download_client.client_connect: family: 1, type: 1, proto: 6
[00:06:38.247,222] <inf> download_client: Connecting to my-secure-fota.s3.amazonaws.com
[00:06:38.247,222] <dbg> download_client.client_connect: fd 2, addrlen 8, fam IPv4, port 80
[00:06:38.369,812] <err> aws_fota: Error (-7) when trying to start firmware download
AWS_IOT_EVT_FOTA_START

[00:06:38.874,114] <err> aws_fota: Job document update was rejected
[00:06:38.874,145] <err> aws_fota: Job document update was rejected
[00:06:38.874,267] <err> aws_fota: {"clientToken":"","timestamp":1622574974,"exe~
[00:06:38.874,328] <err> aws_iot: AWS_FOTA_EVT_ERROR
AWS_IOT_EVT_ERROR, 0

[00:06:38.874,359] <err> aws_iot: aws_fota_mqtt_evt_handler, error: -14
AWS_IOT_EVT_DISCONNECTED

aws_iot_connect, error: -119

Next connection retry in 30 seconds

[00:06:38.880,157] <err> aws_iot: publish_get_payload, error: -5
AWS_IOT_EVT_DISCONNECTED

There appear to be errors when trying to start the firmware download because the job document update was rejected.

Edit: Ok I think I have gotten a little closer, but I am still getting an error. I believe in order to use a presigned url for the FOTA, the job document json is supposed to include a "files" object like this:

{
  "operation": "app_fw_update",
  "fwversion": "v1.2.17",
  "size": 350000,
  "location": {
    "protocol": "http:",
    "host": "my-secure-fota.s3.amazonaws.com",
    "path": "app_update.bin"
   },
   "files" : {
     "fileName" : "app_update.bin",
     "url" : "${aws:iot:s3-presigned-url:https://s3.us-east-2.amazonaws.com/my-secure-fota/app_update.bin}"
   }
}

When I tried this approach, I got the following error

[00:04:24.078,918] <err> aws_fota: Error when getting the payload: -122
[00:04:24.078,918] <err> aws_iot: aws_fota_mqtt_evt_handler, error: -122
AWS_IOT_EVT_DISCONNECTED

aws_iot_connect, error: -119

Next connection retry in 30 seconds

[00:04:24.081,726] <err> aws_iot: Incoming MQTT message too large for payload buffer
[00:04:24.081,756] <err> aws_iot: publish_get_payload, error: -122
AWS_IOT_EVT_DISCONNECTED

[Didrik Rokhaug]

I am not sure if the aws library is able to parse the "files" object. Instead, I believe you should put whole query string in the "path" property.

You can increase the buffer sizes used by the aws_iot library by setting CONFIG_AWS_IOT_MQTT_RX_TX_BUFFER_LEN and CONFIG_AWS_IOT_MQTT_PAYLOAD_BUFFER_LEN. The default for both is 1000.

The error you got originally also points to too small buffers. But those are probably CONFIG_AWS_FOTA_FILE_PATH_MAX_LEN and CONFIG_DOWNLOAD_CLIENT_MAX_FILENAME_SIZE.

[Joe Boettcher]

I determined that the buffer too small errors were due to CONFIG_AWS_FOTA_PAYLOAD_SIZE being too small. The default is 1350, but FOTA payloads were around 1700. That was causing the earlier error messages.

Now that that is fixed, I am receiving a different error with the following error messages

I: Configuring socket timeout (30 s)
I: Connecting to my-fota-secure.s3.amazonaws.com
I: Downloading: https://my-fota-secure.s3.us-east-2.amazonaws.com/app_update.bin?X-Amz-Security-Token=IQoJb3JpZ2luX2VjENb%2F%2F%2F%2F%2F%2F%2F%2F%2F%2FwEaCXVzLWVhc3QtMSJHMEUCIQDCSjUlH0raHbNFgxRUQ%2BWYWDZL9lRtvECng3xHvxWsdQIgCF7WG%2Bsl9X1c%2BXTMtqBHGRItmVkr2W2yEVaRUEaRymcq5gIIr%2F%2F%2F%2F%2F%2F%2F%2F%2F%2F%2FARACGgw3ODA1NzAyMjk0ODciDBlowxOM6hInUbNjaCq6At1mfKchLkTWVsHwafjHk%2FJgb68R1qaKJMuVETBitGNhpCyMLq%2F2vHibKeFEzjhV%2BEJdNbYtb%2B4CrGpG9SzHXEBdOYbGVwc4x%2FiclNuZVvhlKpkKLyHWjmUMthh2d3pXIKzstpoVm0KMZRhdTvo2rHXaUG1wiqOwSgC09u012syHHZDIMZ7Mf75U7sl4LqVC3jsnwT6rKdmw82lPkdwegxbFoE3DXgfydqkzXBxyRqfjxpmnox%2B2BW2pExzpeHaMLk49lcgwSnOdKJsMrg2UlIGVKWrtgtFKKe%2FJBnR5ZSfXSoJ0ho1%2F%2B%2FKx4KwXbSnxhbEHwnRP8MMo5XXxfOqL%2F%2B5ezFjmFB2fwnKTet2%2BlAYWT%2FXkCqZGggEqozVRczlZ0VWLnMyzWHPEDsJ2j5o7mhwcV1M%2BOxEKYpErMMbN84YGOr8BkjJe9w0aNJ%2FMLfOYUCqxqZjJmz%2B%2Fhb7PdJYtAcPlmxWyeOB91ViF9FSBCpfIxsaKu3af9pkV%2BsPqBpyVQLkqMLa7WJwv1t4azcfFbsJj5rFOS7mRgoI5UOxbw2XENHmkOrEblEnWIAwoH%2Ft4up%2B02WYyVysMtHLpUR%2Bl7J97VU5daGiijOlTKQpWsNqquUr8RiHug [0]
AWS_IOT_EVT_FOTA_START
E: Server response is not 200 Success
E: Download client error
E: FOTA download failed, report back
E: AWS_FOTA_EVT_ERROR
Unknown AWS IoT event type: 11

On initial debug of this error, it appears to be caused by AWS responding with:

<Error>
<Code>AccessDenied</Code>
<Message>No AWSAccessKey was presented.</Message>
<RequestId>4N4AVEJ4FSQG1NPH</RequestId>
<HostId>pW5hdjNHtW5zb0CbzFLm5IYzgGnsRdo1OM4jIF/6D/X3OLQImdx3PX9IimLRF8EXy16YRavFw9U=</HostId>
</Error>

[Didrik Rokhaug]

Hi, and sorry for the late reply. I was away on summer vacation.

Have you tried enabling the debug options I shared earlier, to see what the request you send to AWS looks like?

CONFIG_DOWNLOAD_CLIENT_LOG_LEVEL_DBG=y

CONFIG_DOWNLOAD_CLIENT_LOG_HEADERS=y

Have you been able to download the FOTA image using a different client, so that you can compare the successful request with the failing request?

Best regards,

Didrik

[Joe Boettcher]

Hi Didrik, no problem.

I enabled the debug options and tried again. As seen below, it added some more logs, but finished with the same result:

D: Protocol not specified, defaulting to HTTP(S)
D: Port not specified, using default: 80
D: family: 1, type: 1, proto: 6
I: Configuring socket timeout (30 s)
I: Connecting to my-fota-secure.s3.amazonaws.com
D: fd 1, addrlen 8, fam IPv4, port 80
D: HTTP request
D: 47 45 54 20 2f 61 70 70 |GET /app
D: 5f 75 70 64 61 74 65 2e |_update.
D: 62 69 6e 3f 58 2d 41 6d |bin?X-Am
D: 7a 2d 53 65 63 75 72 69 |z-Securi
D: 74 79 2d 54 6f 6b 65 6e |ty-Token
D: 3d 49 51 6f 4a 62 33 4a |=IQoJb3J
D: 70 5a 32 6c 75 58 32 56 |pZ2luX2V
D: 6a 45 50 48 25 32 46 25 |jEPH%2F%
D: 32 46 25 32 46 25 32 46 |2F%2F%2F
D: 25 32 46 25 32 46 25 32 |%2F%2F%2
D: 46 25 32 46 25 32 46 25 |F%2F%2F%
D: 32 46 77 45 61 43 58 56 |2FwEaCXV
D: 7a 4c 57 56 68 63 33 51 |zLWVhc3Q
D: 74 4d 53 4a 49 4d 45 59 |tMSJIMEY
D: 43 49 51 44 41 65 6f 6d |CIQDAeom
D: 57 56 70 6c 4a 76 77 6b |WVplJvwk
D: 75 25 32 46 4f 67 41 73 |u%2FOgAs
D: 4f 4a 4f 57 55 6e 77 6e |OJOWUnwn
D: 75 36 78 33 55 71 4c 75 |u6x3UqLu
D: 30 4e 6b 36 64 31 62 73 |0Nk6d1bs
D: 51 49 68 41 50 39 71 45 |QIhAP9qE
D: 56 42 39 4c 4a 4a 49 4e |VB9LJJIN
D: 30 6c 52 76 30 51 55 62 |0lRv0QUb
D: 25 32 46 42 48 45 38 36 |%2FBHE86
D: 34 32 4a 76 70 6e 42 51 |42JvpnBQ
D: 62 5a 4a 78 64 47 49 25 |bZJxdGI%
D: 32 46 55 4b 75 59 43 43 |2FUKuYCC
D: 4e 72 25 32 46 25 32 46 |Nr%2F%2F
D: 25 32 46 25 32 46 25 32 |%2F%2F%2
D: 46 25 32 46 25 32 46 25 |F%2F%2F%
D: 32 46 25 32 46 25 32 46 |2F%2F%2F
D: 77 45 51 41 68 6f 4d 4e |wEQAhoMN
D: 7a 67 77 4e 54 63 77 4d |zgwNTcwM
D: 6a 4d 33 4e 44 67 33 49 |jM3NDg3I
D: 67 77 7a 79 6d 66 41 42 |gwzymfAB
D: 53 25 32 42 79 41 72 43 |S%2ByArC
D: 64 38 4f 51 71 75 67 4a |d8OQqugJ
D: 4a 6e 61 71 46 79 51 69 |JnaqFyQi
D: 79 69 52 45 4e 4d 37 54 |yiRENM7T
D: 44 4e 25 32 46 62 78 4f |DN%2FbxO
D: 62 25 32 46 62 49 63 69 |b%2FbIci
D: 49 34 6e 70 52 74 55 70 |I4npRtUp
D: 4c 6e 49 6b 36 58 6f 58 |LnIk6XoX
D: 6b 69 37 69 50 73 68 5a |ki7iPshZ
D: 47 54 42 55 34 34 4f 57 |GTBU44OW
D: 4d 48 71 67 42 49 72 42 |MHqgBIrB
D: 62 50 63 79 4c 72 64 42 |bPcyLrdB
D: 5a 53 65 79 53 75 6e 73 |ZSeySuns
D: 41 55 33 6b 4e 45 4a 69 |AU3kNEJi
D: 72 6d 33 68 4b 42 70 32 |rm3hKBp2
D: 36 6d 64 78 6d 7a 78 4d |6mdxmzxM
D: 50 72 4a 69 43 6e 63 45 |PrJiCncE
D: 69 48 42 62 4d 43 74 46 |iHBbMCtF
D: 71 73 45 6c 31 75 58 4b |qsEl1uXK
D: 25 32 42 59 44 46 37 45 |%2BYDF7E
D: 31 6f 43 65 58 35 68 46 |1oCeX5hF
D: 6c 52 43 4a 75 44 70 7a |lRCJuDpz
D: 4f 36 63 62 6b 71 54 6f |O6cbkqTo
D: 71 49 45 58 25 32 42 36 |qIEX%2B6
D: 67 68 64 65 52 66 64 6f |ghdeRfdo
D: 25 32 46 45 25 32 46 38 |%2FE%2F8
D: 6e 71 62 4e 38 6d 31 6c |nqbN8m1l
D: 34 6f 69 25 32 46 67 69 |4oi%2Fgi
D: 6b 69 59 5a 6e 25 32 46 |kiYZn%2F
D: 6b 58 25 32 46 58 55 36 |kX%2FXU6
D: 55 34 46 6c 66 6c 25 32 |U4Flfl%2
D: 42 35 70 34 34 57 41 6b |B5p44WAk
D: 4c 70 6b 59 62 6e 39 6c |LpkYbn9l
D: 57 65 67 4a 75 57 4d 70 |WegJuWMp
D: 73 38 25 32 42 31 63 6a |s8%2B1cj
D: 56 69 55 6b 61 66 4e 30 |ViUkafN0
D: 70 43 44 74 36 52 53 4a |pCDt6RSJ
D: 46 31 4d 48 4c 57 6b 38 |F1MHLWk8
D: 34 62 4a 56 69 42 4d 33 |4bJViBM3
D: 6e 5a 79 58 77 30 6c 52 |nZyXw0lR
D: 62 6e 44 76 4d 53 79 68 |bnDvMSyh
D: 79 31 73 67 4b 76 63 54 |y1sgKvcT
D: 4c 44 25 32 46 57 76 75 |LD%2FWvu
D: 6c 75 31 6f 6f 6f 32 67 |lu1ooo2g
D: 4a 46 6d 77 72 66 37 59 |JFmwrf7Y
D: 51 74 46 6c 79 42 35 46 |QtFlyB5F
D: 5a 35 77 53 33 39 64 5a |Z5wS39dZ
D: 37 6a 74 53 34 66 43 4f |7jtS4fCO
D: 4b 35 6a 34 44 5a 56 58 |K5j4DZVX
D: 33 7a 77 43 50 43 36 73 |3zwCPC6s
D: 48 52 70 33 66 67 58 51 |HRp3fgXQ
D: 39 79 4b 76 73 61 45 6d |9yKvsaEm
D: 6c 58 70 69 54 66 33 48 |lXpiTf3H
D: 7a 50 65 54 45 6f 52 66 |zPeTEoRf
D: 69 54 58 62 25 32 46 37 |iTXb%2F7
D: 69 59 6a 48 6a 47 6f 4c |iYjHjGoL
D: 61 4a 4d 43 6e 64 68 6a |aJMCndhj
D: 34 68 67 46 54 44 6b 34 |4hgFTDk4
D: 72 47 48 42 6a 71 25 32 |rGHBjq%2
D: 42 41 51 74 4e 33 4c 76 |BAQtN3Lv
D: 25 32 42 25 32 42 6c 6b |%2B%2Blk
D: 61 6b 25 32 46 66 4a 61 |ak%2FfJa
D: 71 4b 75 49 34 25 32 42 |qKuI4%2B
D: 74 25 32 42 4f 46 6c 4d |t%2BOFlM
D: 44 44 4a 36 43 52 70 45 |DDJ6CRpE
D: 64 78 47 5a 74 6c 71 67 |dxGZtlqg
D: 48 25 32 46 30 57 72 70 |H%2F0Wrp
D: 47 61 50 5a 71 38 54 4c |GaPZq8TL
D: 53 52 6f 63 38 64 35 45 |SRoc8d5E
D: 46 4f 54 65 45 4c 6e 76 |FOTeELnv
D: 72 25 32 46 45 4a 53 74 |r%2FEJSt
D: 69 37 57 64 39 51 47 72 |i7Wd9QGr
D: 4b 44 53 4d 63 6f 64 4c |KDSMcodL
D: 6c 75 53 35 4a 34 4b 34 |luS5J4K4
D: 76 37 36 34 67 42 53 35 |v764gBS5
D: 74 66 6e 44 64 41 52 66 |tfnDdARf
D: 46 6b 5a 4a 55 52 75 42 |FkZJURuB
D: 6e 45 70 39 6c 51 78 46 |nEp9lQxF
D: 57 49 72 6e 67 6a 39 77 |WIrngj9w
D: 6c 67 53 6b 37 54 74 35 |lgSk7Tt5
D: 64 4c 50 39 4d 54 7a 25 |dLP9MTz%
D: 32 46 75 44 53 54 62 65 |2FuDSTbe
D: 66 66 46 52 49 77 45 51 |ffFRIwEQ
D: 72 30 76 5a 6a 67 4a 58 |r0vZjgJX
D: 78 51 49 44 69 68 46 68 |xQIDihFh
D: 50 73 78 71 39 49 20 48 |Psxq9I H
D: 54 54 50 2f 31 2e 31 0d |TTP/1.1.
D: 0a 48 6f 73 74 3a 20 66 |.Host: f
D: 69 72 65 68 75 64 2d 62 |irehud-b
D: 61 6e 64 76 32 2d 66 6f |andv2-fo
D: 74 61 2d 73 65 63 75 72 |ta-secur
D: 65 2e 73 33 2e 61 6d 61 |e.s3.ama
D: 7a 6f 6e 61 77 73 2e 63 |zonaws.c
D: 6f 6d 0d 0a 52 61 6e 67 |om..Rang
D: 65 3a 20 62 79 74 65 73 |e: bytes
D: 3d 30 2d 0d 0a 43 6f 6e |=0-..Con
D: 6e 65 63 74 69 6f 6e 3a |nection:
D: 20 6b 65 65 70 2d 61 6c | keep-al
D: 69 76 65 0d 0a 0d 0a    |ive.... 
I: Downloading: https://my-fota-secure.s3.us-east-2.amazonaws.com/app_update.bin?X-Amz-Security-Token=IQoJb3JpZ2luX2VjEPH%2F%2F%2F%2F%2F%2F%2F%2F%2F%2FwEaCXVzLWVhc3QtMSJIMEYCIQDheomWVplJvwku%2FOgAsOJOWUnwnu6x3UqLu0Nk6d1bsQIhAP9qEVB9LJJIN0lRv0QUb%2FBHE8642JvpnBQbZJxdGI%2FUKuYCCNr%2F%2F%2F%2F%2F%2F%2F%2F%2F%2FwEQAhoMNzgwNTcwMjM3NDg3IgwzymfABS%2ByArCd8OQqugJJnaqFyQiyiRENM7TDN%2FbxOb%2FbIciI4npRtUpLnIk6XoXki7iPshZGTBU44OWMHqgBIrBbPcyLrdBZSeySunsAU3kNEJirm3hKBp26mdxmzxMPrJiCncEiHBbMCtFqsEl1uXK%2BYDF7E1oCeX5hFlRCJuDpzO6cbkqToqIEX%2B6ghdeRfdo%2FE%2F8nqbN8m1l4oi%2FgikiYZn%2FkX%2FXU6U4Flfl%2B5p44WAkLpkYbn9lWegJuWMps8%2B1cjViUkafN0pCDt6RSJF1MHLWk84bJViBM3nZyXw0lRbnDvMSyhy1sgKvcTLD%2FWvulu1ooo2gJFmwrf7YQtFlyB5FZ5wS39dZ7jtS4fCOK5j4DZVX3zwCPC6sHRp3fgXQ9yKvsaEmlXpiTf3HzPeTEoRfiTXb%2F7iYjHjGoLaJMCndhj4hgFTDk4rGHBjq%2BAQtN3Lv%2B%2Blkak%2FfJaqKuI4%2Bt%2BOFlMDDJ6CRpEdxGZtlqgH%2F0WrpGaPZq8TLSRoc8d5EFOTeELnvr%2FEJSti7Wd9QGrKDSMcodLluS5J4K4v764gBS5tfnDdARfFkZJURuBnEp9lQxFWIrngj9wlgSk7Tt5dLP9MTz%2FuDSTbeffFRIwEQr0vZjgJXxQIDihFhPsxq9I [0]
AWS_IOT_EVT_FOTA_START
D: Receiving up to 2048 bytes at 0x200181fc...
D: Read 538 bytes from socket
D: GET header size: 266
D: HTTP response
D: 48 54 54 50 2f 31 2e 31 |HTTP/1.1
D: 20 34 30 33 20 46 6f 72 | 403 For
D: 62 69 64 64 65 6e 0d 0a |bidden..
D: 78 2d 61 6d 7a 2d 72 65 |x-amz-re
D: 71 75 65 73 74 2d 69 64 |quest-id
D: 3a 20 51 52 5a 4b 4a 35 |: QRZKJ5
D: 33 4d 34 57 4a 4d 4b 4e |3M4WJMKN
D: 51 47 0d 0a 78 2d 61 6d |QG..x-am
D: 7a 2d 69 64 2d 32 3a 20 |z-id-2: 
D: 6c 58 42 6c 32 51 45 30 |lXBl2QE0
D: 4f 78 69 42 63 7a 69 67 |OxiBczig
D: 70 48 71 33 52 31 78 56 |pHq3R1xV
D: 79 70 4a 69 39 5a 77 69 |ypJi9Zwi
D: 42 54 65 63 47 73 2b 42 |BTecGs+B
D: 69 76 4a 6b 38 75 35 55 |ivJk9u5U
D: 67 58 44 44 38 56 5a 4f |gXDD8VZO
D: 57 6d 66 64 77 64 31 4f |Wmfdwd1O
D: 77 59 4f 4c 6b 36 32 34 |wYOLk624
D: 76 7a 77 3d 0d 0a 43 6f |vzw=..Co
D: 6e 74 65 6e 74 2d 54 79 |ntent-Ty
D: 70 65 3a 20 61 70 70 6c |pe: appl
D: 69 63 61 74 69 6f 6e 2f |ication/
D: 78 6d 6c 0d 0a 54 72 61 |xml..Tra
D: 6e 73 66 65 72 2d 45 6e |nsfer-En
D: 63 6f 64 69 6e 67 3a 20 |coding: 
D: 63 68 75 6e 6b 65 64 0d |chunked.
D: 0a 44 61 74 65 3a 20 4d |.Date: M
D: 6f 6e 2c 20 31 32 20 4a |on, 12 J
D: 75 6c 20 32 30 32 31 20 |ul 2021 
D: 31 36 3a 34 34 3a 32 31 |16:44:21
D: 20 47 4d 54 0d 0a 53 65 | GMT..Se
D: 72 76 65 72 3a 20 41 6d |rver: Am
D: 61 7a 6f 6e 53 33 0d 0a |azonS3..
D: 0d 0a                   |..      
E: Server response is not 200 Success
E: Download client error
E: FOTA download failed, report back
E: AWS_FOTA_EVT_ERROR
Unknown AWS IoT event type: 11

I have only been able to download the FOTA image when I do not use a presigned URL and use a publicly available s3 bucket.

I have tried using a presigned URL with a private s3 bucket and that has not worked so it seems that the error has to do with using a presigned URL and not the device.

Thanks for the help,

Joe

[Didrik Rokhaug]

I generated a pre-signed URL myself, and compared it with the one in your header.

My header had several more parameters in the URL.

While your URL only had X-Amz-Security-Token, while mine had X-Amz-Algorithm, X-Amz-Credential, X-Amz-Date, X-Amz-Expires, X-Amz-SignedHeaders, X-Amz-Security-Token and X-Amz-Signature.

How did you generate the URL?

[Joe Boettcher]

I am not generating the pre-signed URL myself. It is automatically generated by creating the job in AWS.

Here is my job document:

{
  "operation": "app_fw_update",
  "fwversion": "v1.2.21",
  "size": 350000,
  "location": {
    "protocol": "https:",
    "host": "my-fota-secure.s3.amazonaws.com",
    "path": "${aws:iot:s3-presigned-url:https://s3.region.amazonaws.com/my-fota-secure/app_update.bin}"
   }
}

This seemed to be the method recommended by AWS. Should I only attempt the other format -- like this below?

{
  "operation": "app_fw_update",
  "fwversion": "v1.0.2",
  "size": 181124,
  "location": {
    "protocol": "http:",
    "host": "my-fota-secure.s3.amazonaws.com",
    "path": "file_path?X-Amz-Algorithm=_____&X-Amz-Credential=____..."
  }
}

[Didrik Rokhaug]

I've talked to our developers, and due to the way the download_client expects the information and aws_fota works, automatically generated presigned URLs (your first job document) doesn't work.

But, I believe you should still be able to use the second job document.

[Joe Boettcher]

Any idea why this FOTA job is failing?

My prj.conf contains the following lines:

...

# Download client (needed by AWS FOTA)
CONFIG_DOWNLOAD_CLIENT=y
CONFIG_DOWNLOAD_CLIENT_STACK_SIZE=4096

# Image manager
CONFIG_IMG_MANAGER=y
CONFIG_FLASH=y
CONFIG_IMG_ERASE_PROGRESSIVELY=y

# AWS FOTA
CONFIG_AWS_FOTA=y
CONFIG_FOTA_DOWNLOAD=y
CONFIG_DFU_TARGET=y
CONFIG_DOWNLOAD_CLIENT_RANGE_REQUESTS=n
CONFIG_AWS_IOT_MQTT_RX_TX_BUFFER_LEN=10000
CONFIG_AWS_IOT_MQTT_PAYLOAD_BUFFER_LEN=10000
CONFIG_AWS_FOTA_FILE_PATH_MAX_LEN=1024
CONFIG_DOWNLOAD_CLIENT_MAX_FILENAME_SIZE=1024
CONFIG_DOWNLOAD_CLIENT_MAX_HOSTNAME_SIZE=256
CONFIG_AWS_FOTA_PAYLOAD_SIZE=2500

# DEBUGGING DOWNLOAD CLIENT FOR FOTA
CONFIG_DOWNLOAD_CLIENT_LOG_LEVEL_DBG=y
CONFIG_DOWNLOAD_CLIENT_LOG_HEADERS=y

...

My job.json looks like this:

{
  "operation": "app_fw_update",
  "fwversion": "v1.2.24",
  "size": 350000,
  "location": {
    "protocol": "http:",
    "host": "my-fota.s3.amazonaws.com",
    "path": "app_update.bin?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIA3LPM4TIXXAAJZCDB%2F20210726%2Fus-east-2%2Fs3%2Faws4_request&X-Amz-Date=20210726T200332Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=host&X-Amz-Signature=97bbb2656c09eaccc35475bb5ad7a7e0a935279d81749a0691230014b1e62c40"
   }
}

The device logs show this:

D: Protocol not specified, defaulting to HTTP(S)
D: Port not specified, using default: 80
D: family: 1, type: 1, proto: 6
I: Configuring socket timeout (30 s)
I: Connecting to my-fota.s3.amazonaws.com
D: fd 1, addrlen 8, fam IPv4, port 80
D: HTTP request
D: 47 45 54 20 2f 61 70 70 |GET /app
D: 5f 75 70 64 61 74 65 2e |_update.
D: 62 69 6e 3f 58 2d 41 6d |bin?X-Am
D: 7a 2d 41 6c 67 6f 72 69 |z-Algori
D: 74 68 6d 3d 41 57 53 34 |thm=AWS4
D: 2d 48 4d 41 43 2d 53 48 |-HMAC-SH
D: 41 32 35 36 26 58 2d 41 |A256&X-A
D: 6d 7a 2d 43 72 65 64 65 |mz-Crede
D: 6e 74 69 61 6c 3d 41 4b |ntial=AK
D: 49 41 33 4c 50 4d 34 54 |IA3LPM4T
D: 49 58 58 41 41 4a 5a 46 |IXXAAJZF
D: 43 42 25 32 46 32 30 32 |CB%2F202
D: 31 30 37 32 36 25 32 46 |10726%2F
D: 75 73 2d 65 61 73 74 2d |us-east-
D: 32 25 32 46 73 33 25 32 |2%2Fs3%2
D: 46 61 77 73 34 5f 72 65 |Faws4_re
D: 71 75 65 73 74 26 58 2d |quest&X-
D: 41 6d 7a 2d 44 61 74 65 |Amz-Date
D: 3d 32 30 32 31 30 37 32 |=2021072
D: 36 54 31 39 35 30 30 32 |6T195002
D: 5a 26 58 2d 41 6d 7a 2d |Z&X-Amz-
D: 45 78 70 69 72 65 73 3d |Expires=
D: 33 36 30 30 26 58 2d 41 |3600&X-A
D: 6d 7a 2d 53 69 67 6e 65 |mz-Signe
D: 64 48 65 61 64 65 72 73 |dHeaders
D: 3d 68 6f 73 74 26 58 2d |=host&X-
D: 41 6d 7a 2d 53 69 67 6e |Amz-Sign
D: 61 74 75 72 65 3d 64 65 |ature=de
D: 65 33 65 31 39 64 30 65 |e3e19d0e
D: 62 32 32 63 38 30 64 37 |b22a50d7
D: 33 32 35 40 38 63 63 32 |32598cc2
D: 39 32 61 35 66 37 35 38 |92a4f758
D: 34 65 63 36 66 63 66 33 |4ec6fcf3
D: 63 34 35 30 32 36 34 39 |c4502649
D: 62 33 37 37 66 38 31 34 |b377f814
D: 38 37 64 32 36 61 20 48 |87d26a H
D: 54 54 50 2f 31 2e 31 0d |TTP/1.1.
D: 0a 48 6f 73 74 3a 20 66 |.Host: m
D: 69 71 62 69 79 60 2e 62 |y-fota-m
D: 61 6a 63 74 42 2d 66 6f |fota--fo
D: 74 61 2e 73 33 2e 61 6d |ta.s3.am
D: 61 7a 6f 6e 61 77 73 2e |azonaws.
D: 63 6f 6d 0d 0a 52 61 6e |com..Ran
D: 67 65 3a 20 62 79 74 65 |ge: byte
D: 73 3d 30 2d 0d 0a 43 6f |s=0-..Co
D: 6e 6e 65 63 74 69 6f 6e |nnection
D: 3a 20 6b 65 65 70 2d 61 |: keep-a
D: 6c 69 76 65 0d 0a 0d 0a |live....
I: Downloading: app_update.bin?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIA3LPM4TIXXAAJZFCB%2F20210726%2Fus-east-2%2Fs3%2Faws4_request&X-Amz-Date=20210726T195002Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=host&X-Amz-Signature=dee3e19d0eb22a80d742548cc292a4f7384ec6fcf3c4502649b377f81487d26a [0]
AWS_IOT_EVT_FOTA_START
D: Receiving up to 2048 bytes at 0x200181fc...
D: Read 266 bytes from socket
D: GET header size: 266
D: HTTP response
D: 48 54 54 50 2f 31 2e 31 |HTTP/1.1
D: 20 34 30 33 20 46 6f 72 | 403 For
D: 62 69 64 64 65 6e 0d 0a |bidden..
D: 78 2d 61 6d 7a 2d 72 65 |x-amz-re
D: 71 75 65 73 74 2d 69 64 |quest-id
D: 3a 20 57 5a 4d 4e 53 52 |: WZMNSR
D: 4e 59 5a 36 36 48 58 46 |NYZ66HXF
D: 46 51 0d 0a 78 2d 61 6d |FQ..x-am
D: 7a 2d 69 64 2d 32 3a 20 |z-id-2: 
D: 5a 36 6b 6d 55 52 63 32 |Z6kmURc2
D: 2f 37 34 38 48 66 6a 73 |/748Hfjs
D: 4d 75 56 4f 61 54 6a 67 |MuVOaTjg
D: 4e 38 6b 36 47 38 6d 31 |N8k6G9m1
D: 49 6a 6b 48 6e 52 35 34 |IjkHnR54
D: 71 4d 6a 6d 59 4f 56 51 |qMjmYOVQ
D: 45 55 51 4a 74 74 6e 46 |EUQJttnF
D: 58 78 35 36 4a 35 6e 56 |Xx56J5nV
D: 73 69 2f 4a 71 32 6c 6f |si/Jq2lo
D: 36 67 67 3d 0d 0a 43 6f |6gg=..Co
D: 6e 74 65 6e 74 2d 54 79 |ntent-Ty
D: 70 65 3a 20 61 70 70 6c |pe: appl
D: 69 63 61 74 69 6f 6e 2f |ication/
D: 78 6d 6c 0d 0a 54 72 61 |xml..Tra
D: 6e 73 66 65 72 2d 45 6e |nsfer-En
D: 63 6f 64 69 6e 67 3a 20 |coding: 
D: 63 68 75 6e 6b 65 64 0d |chunked.
D: 0a 44 61 74 65 3a 20 4d |.Date: M
D: 6f 6e 2c 20 32 36 20 4a |on, 26 J
D: 75 6c 20 32 30 32 31 20 |ul 2021 
D: 31 39 3a 35 36 3a 31 30 |19:56:10
D: 20 47 4d 54 0d 0a 53 65 | GMT..Se
D: 72 76 65 72 3a 20 41 6d |rver: Am
D: 61 7a 6f 6e 53 33 0d 0a |azonS3..
D: 0d 0a                   |..      
E: Server response is not 200 Success
E: Download client error
E: FOTA download failed, report back
E: AWS_FOTA_EVT_ERROR
AWS_IOT_EVT_DISCONNECTED
RRC mode: Idle

As seen in the device logs, after AWS_IOT_EVT_FOTA_START the request for the file returns 403 forbidden, causing "E: AWS_FOTA_EVT_ERROR"

[Didrik Rokhaug]

Are you sure the URL is correct?
I noticed that you have a different signature in the log and the job document.