AWS FOTA using presigned URL from private S3 bucket

[Joe Boettcher]

Hello,

I've gotten the sample AWS FOTA to work on my nrf9160 using a very permissive public S3 bucket, but for production I need FOTAs to use a private S3 bucket.

According to https://devzone.nordicsemi.com/f/nordic-q-a/70753/aws-fota-download-from-s3-authentication/309614#309614, using presigned URLs is advised to allow for using a private S3 bucket. This makes sense; however, there is no documentation how to do this and I have not been able to get it to work. Would you simply append the query params to the path as shown below?

Example (failed) attempt at IoT Job using presigned URL:

{
  "operation": "app_fw_update",
  "fwversion": "v1.0.2",
  "size": 181124,
  "location": {
    "protocol": "http:",
    "host": "myfota-secure.s3.amazonaws.com",
    "path": "file_path?X-Amz-Algorithm=_____&X-Amz-Credential=____..."
  }
}

Thanks for the help

[Joe Boettcher]

I determined that the buffer too small errors were due to CONFIG_AWS_FOTA_PAYLOAD_SIZE being too small. The default is 1350, but FOTA payloads were around 1700. That was causing the earlier error messages.

Now that that is fixed, I am receiving a different error with the following error messages

I: Configuring socket timeout (30 s)
I: Connecting to my-fota-secure.s3.amazonaws.com
I: Downloading: https://my-fota-secure.s3.us-east-2.amazonaws.com/app_update.bin?X-Amz-Security-Token=IQoJb3JpZ2luX2VjENb%2F%2F%2F%2F%2F%2F%2F%2F%2F%2FwEaCXVzLWVhc3QtMSJHMEUCIQDCSjUlH0raHbNFgxRUQ%2BWYWDZL9lRtvECng3xHvxWsdQIgCF7WG%2Bsl9X1c%2BXTMtqBHGRItmVkr2W2yEVaRUEaRymcq5gIIr%2F%2F%2F%2F%2F%2F%2F%2F%2F%2F%2FARACGgw3ODA1NzAyMjk0ODciDBlowxOM6hInUbNjaCq6At1mfKchLkTWVsHwafjHk%2FJgb68R1qaKJMuVETBitGNhpCyMLq%2F2vHibKeFEzjhV%2BEJdNbYtb%2B4CrGpG9SzHXEBdOYbGVwc4x%2FiclNuZVvhlKpkKLyHWjmUMthh2d3pXIKzstpoVm0KMZRhdTvo2rHXaUG1wiqOwSgC09u012syHHZDIMZ7Mf75U7sl4LqVC3jsnwT6rKdmw82lPkdwegxbFoE3DXgfydqkzXBxyRqfjxpmnox%2B2BW2pExzpeHaMLk49lcgwSnOdKJsMrg2UlIGVKWrtgtFKKe%2FJBnR5ZSfXSoJ0ho1%2F%2B%2FKx4KwXbSnxhbEHwnRP8MMo5XXxfOqL%2F%2B5ezFjmFB2fwnKTet2%2BlAYWT%2FXkCqZGggEqozVRczlZ0VWLnMyzWHPEDsJ2j5o7mhwcV1M%2BOxEKYpErMMbN84YGOr8BkjJe9w0aNJ%2FMLfOYUCqxqZjJmz%2B%2Fhb7PdJYtAcPlmxWyeOB91ViF9FSBCpfIxsaKu3af9pkV%2BsPqBpyVQLkqMLa7WJwv1t4azcfFbsJj5rFOS7mRgoI5UOxbw2XENHmkOrEblEnWIAwoH%2Ft4up%2B02WYyVysMtHLpUR%2Bl7J97VU5daGiijOlTKQpWsNqquUr8RiHug [0]
AWS_IOT_EVT_FOTA_START
E: Server response is not 200 Success
E: Download client error
E: FOTA download failed, report back
E: AWS_FOTA_EVT_ERROR
Unknown AWS IoT event type: 11

On initial debug of this error, it appears to be caused by AWS responding with:

<Error>
<Code>AccessDenied</Code>
<Message>No AWSAccessKey was presented.</Message>
<RequestId>4N4AVEJ4FSQG1NPH</RequestId>
<HostId>pW5hdjNHtW5zb0CbzFLm5IYzgGnsRdo1OM4jIF/6D/X3OLQImdx3PX9IimLRF8EXy16YRavFw9U=</HostId>
</Error>

[Didrik Rokhaug]

Hi, and sorry for the late reply. I was away on summer vacation.

Have you tried enabling the debug options I shared earlier, to see what the request you send to AWS looks like?

CONFIG_DOWNLOAD_CLIENT_LOG_LEVEL_DBG=y

CONFIG_DOWNLOAD_CLIENT_LOG_HEADERS=y

Have you been able to download the FOTA image using a different client, so that you can compare the successful request with the failing request?

Best regards,

Didrik

[Joe Boettcher]

Hi Didrik, no problem.

I enabled the debug options and tried again. As seen below, it added some more logs, but finished with the same result:

D: Protocol not specified, defaulting to HTTP(S)
D: Port not specified, using default: 80
D: family: 1, type: 1, proto: 6
I: Configuring socket timeout (30 s)
I: Connecting to my-fota-secure.s3.amazonaws.com
D: fd 1, addrlen 8, fam IPv4, port 80
D: HTTP request
D: 47 45 54 20 2f 61 70 70 |GET /app
D: 5f 75 70 64 61 74 65 2e |_update.
D: 62 69 6e 3f 58 2d 41 6d |bin?X-Am
D: 7a 2d 53 65 63 75 72 69 |z-Securi
D: 74 79 2d 54 6f 6b 65 6e |ty-Token
D: 3d 49 51 6f 4a 62 33 4a |=IQoJb3J
D: 70 5a 32 6c 75 58 32 56 |pZ2luX2V
D: 6a 45 50 48 25 32 46 25 |jEPH%2F%
D: 32 46 25 32 46 25 32 46 |2F%2F%2F
D: 25 32 46 25 32 46 25 32 |%2F%2F%2
D: 46 25 32 46 25 32 46 25 |F%2F%2F%
D: 32 46 77 45 61 43 58 56 |2FwEaCXV
D: 7a 4c 57 56 68 63 33 51 |zLWVhc3Q
D: 74 4d 53 4a 49 4d 45 59 |tMSJIMEY
D: 43 49 51 44 41 65 6f 6d |CIQDAeom
D: 57 56 70 6c 4a 76 77 6b |WVplJvwk
D: 75 25 32 46 4f 67 41 73 |u%2FOgAs
D: 4f 4a 4f 57 55 6e 77 6e |OJOWUnwn
D: 75 36 78 33 55 71 4c 75 |u6x3UqLu
D: 30 4e 6b 36 64 31 62 73 |0Nk6d1bs
D: 51 49 68 41 50 39 71 45 |QIhAP9qE
D: 56 42 39 4c 4a 4a 49 4e |VB9LJJIN
D: 30 6c 52 76 30 51 55 62 |0lRv0QUb
D: 25 32 46 42 48 45 38 36 |%2FBHE86
D: 34 32 4a 76 70 6e 42 51 |42JvpnBQ
D: 62 5a 4a 78 64 47 49 25 |bZJxdGI%
D: 32 46 55 4b 75 59 43 43 |2FUKuYCC
D: 4e 72 25 32 46 25 32 46 |Nr%2F%2F
D: 25 32 46 25 32 46 25 32 |%2F%2F%2
D: 46 25 32 46 25 32 46 25 |F%2F%2F%
D: 32 46 25 32 46 25 32 46 |2F%2F%2F
D: 77 45 51 41 68 6f 4d 4e |wEQAhoMN
D: 7a 67 77 4e 54 63 77 4d |zgwNTcwM
D: 6a 4d 33 4e 44 67 33 49 |jM3NDg3I
D: 67 77 7a 79 6d 66 41 42 |gwzymfAB
D: 53 25 32 42 79 41 72 43 |S%2ByArC
D: 64 38 4f 51 71 75 67 4a |d8OQqugJ
D: 4a 6e 61 71 46 79 51 69 |JnaqFyQi
D: 79 69 52 45 4e 4d 37 54 |yiRENM7T
D: 44 4e 25 32 46 62 78 4f |DN%2FbxO
D: 62 25 32 46 62 49 63 69 |b%2FbIci
D: 49 34 6e 70 52 74 55 70 |I4npRtUp
D: 4c 6e 49 6b 36 58 6f 58 |LnIk6XoX
D: 6b 69 37 69 50 73 68 5a |ki7iPshZ
D: 47 54 42 55 34 34 4f 57 |GTBU44OW
D: 4d 48 71 67 42 49 72 42 |MHqgBIrB
D: 62 50 63 79 4c 72 64 42 |bPcyLrdB
D: 5a 53 65 79 53 75 6e 73 |ZSeySuns
D: 41 55 33 6b 4e 45 4a 69 |AU3kNEJi
D: 72 6d 33 68 4b 42 70 32 |rm3hKBp2
D: 36 6d 64 78 6d 7a 78 4d |6mdxmzxM
D: 50 72 4a 69 43 6e 63 45 |PrJiCncE
D: 69 48 42 62 4d 43 74 46 |iHBbMCtF
D: 71 73 45 6c 31 75 58 4b |qsEl1uXK
D: 25 32 42 59 44 46 37 45 |%2BYDF7E
D: 31 6f 43 65 58 35 68 46 |1oCeX5hF
D: 6c 52 43 4a 75 44 70 7a |lRCJuDpz
D: 4f 36 63 62 6b 71 54 6f |O6cbkqTo
D: 71 49 45 58 25 32 42 36 |qIEX%2B6
D: 67 68 64 65 52 66 64 6f |ghdeRfdo
D: 25 32 46 45 25 32 46 38 |%2FE%2F8
D: 6e 71 62 4e 38 6d 31 6c |nqbN8m1l
D: 34 6f 69 25 32 46 67 69 |4oi%2Fgi
D: 6b 69 59 5a 6e 25 32 46 |kiYZn%2F
D: 6b 58 25 32 46 58 55 36 |kX%2FXU6
D: 55 34 46 6c 66 6c 25 32 |U4Flfl%2
D: 42 35 70 34 34 57 41 6b |B5p44WAk
D: 4c 70 6b 59 62 6e 39 6c |LpkYbn9l
D: 57 65 67 4a 75 57 4d 70 |WegJuWMp
D: 73 38 25 32 42 31 63 6a |s8%2B1cj
D: 56 69 55 6b 61 66 4e 30 |ViUkafN0
D: 70 43 44 74 36 52 53 4a |pCDt6RSJ
D: 46 31 4d 48 4c 57 6b 38 |F1MHLWk8
D: 34 62 4a 56 69 42 4d 33 |4bJViBM3
D: 6e 5a 79 58 77 30 6c 52 |nZyXw0lR
D: 62 6e 44 76 4d 53 79 68 |bnDvMSyh
D: 79 31 73 67 4b 76 63 54 |y1sgKvcT
D: 4c 44 25 32 46 57 76 75 |LD%2FWvu
D: 6c 75 31 6f 6f 6f 32 67 |lu1ooo2g
D: 4a 46 6d 77 72 66 37 59 |JFmwrf7Y
D: 51 74 46 6c 79 42 35 46 |QtFlyB5F
D: 5a 35 77 53 33 39 64 5a |Z5wS39dZ
D: 37 6a 74 53 34 66 43 4f |7jtS4fCO
D: 4b 35 6a 34 44 5a 56 58 |K5j4DZVX
D: 33 7a 77 43 50 43 36 73 |3zwCPC6s
D: 48 52 70 33 66 67 58 51 |HRp3fgXQ
D: 39 79 4b 76 73 61 45 6d |9yKvsaEm
D: 6c 58 70 69 54 66 33 48 |lXpiTf3H
D: 7a 50 65 54 45 6f 52 66 |zPeTEoRf
D: 69 54 58 62 25 32 46 37 |iTXb%2F7
D: 69 59 6a 48 6a 47 6f 4c |iYjHjGoL
D: 61 4a 4d 43 6e 64 68 6a |aJMCndhj
D: 34 68 67 46 54 44 6b 34 |4hgFTDk4
D: 72 47 48 42 6a 71 25 32 |rGHBjq%2
D: 42 41 51 74 4e 33 4c 76 |BAQtN3Lv
D: 25 32 42 25 32 42 6c 6b |%2B%2Blk
D: 61 6b 25 32 46 66 4a 61 |ak%2FfJa
D: 71 4b 75 49 34 25 32 42 |qKuI4%2B
D: 74 25 32 42 4f 46 6c 4d |t%2BOFlM
D: 44 44 4a 36 43 52 70 45 |DDJ6CRpE
D: 64 78 47 5a 74 6c 71 67 |dxGZtlqg
D: 48 25 32 46 30 57 72 70 |H%2F0Wrp
D: 47 61 50 5a 71 38 54 4c |GaPZq8TL
D: 53 52 6f 63 38 64 35 45 |SRoc8d5E
D: 46 4f 54 65 45 4c 6e 76 |FOTeELnv
D: 72 25 32 46 45 4a 53 74 |r%2FEJSt
D: 69 37 57 64 39 51 47 72 |i7Wd9QGr
D: 4b 44 53 4d 63 6f 64 4c |KDSMcodL
D: 6c 75 53 35 4a 34 4b 34 |luS5J4K4
D: 76 37 36 34 67 42 53 35 |v764gBS5
D: 74 66 6e 44 64 41 52 66 |tfnDdARf
D: 46 6b 5a 4a 55 52 75 42 |FkZJURuB
D: 6e 45 70 39 6c 51 78 46 |nEp9lQxF
D: 57 49 72 6e 67 6a 39 77 |WIrngj9w
D: 6c 67 53 6b 37 54 74 35 |lgSk7Tt5
D: 64 4c 50 39 4d 54 7a 25 |dLP9MTz%
D: 32 46 75 44 53 54 62 65 |2FuDSTbe
D: 66 66 46 52 49 77 45 51 |ffFRIwEQ
D: 72 30 76 5a 6a 67 4a 58 |r0vZjgJX
D: 78 51 49 44 69 68 46 68 |xQIDihFh
D: 50 73 78 71 39 49 20 48 |Psxq9I H
D: 54 54 50 2f 31 2e 31 0d |TTP/1.1.
D: 0a 48 6f 73 74 3a 20 66 |.Host: f
D: 69 72 65 68 75 64 2d 62 |irehud-b
D: 61 6e 64 76 32 2d 66 6f |andv2-fo
D: 74 61 2d 73 65 63 75 72 |ta-secur
D: 65 2e 73 33 2e 61 6d 61 |e.s3.ama
D: 7a 6f 6e 61 77 73 2e 63 |zonaws.c
D: 6f 6d 0d 0a 52 61 6e 67 |om..Rang
D: 65 3a 20 62 79 74 65 73 |e: bytes
D: 3d 30 2d 0d 0a 43 6f 6e |=0-..Con
D: 6e 65 63 74 69 6f 6e 3a |nection:
D: 20 6b 65 65 70 2d 61 6c | keep-al
D: 69 76 65 0d 0a 0d 0a    |ive.... 
I: Downloading: https://my-fota-secure.s3.us-east-2.amazonaws.com/app_update.bin?X-Amz-Security-Token=IQoJb3JpZ2luX2VjEPH%2F%2F%2F%2F%2F%2F%2F%2F%2F%2FwEaCXVzLWVhc3QtMSJIMEYCIQDheomWVplJvwku%2FOgAsOJOWUnwnu6x3UqLu0Nk6d1bsQIhAP9qEVB9LJJIN0lRv0QUb%2FBHE8642JvpnBQbZJxdGI%2FUKuYCCNr%2F%2F%2F%2F%2F%2F%2F%2F%2F%2FwEQAhoMNzgwNTcwMjM3NDg3IgwzymfABS%2ByArCd8OQqugJJnaqFyQiyiRENM7TDN%2FbxOb%2FbIciI4npRtUpLnIk6XoXki7iPshZGTBU44OWMHqgBIrBbPcyLrdBZSeySunsAU3kNEJirm3hKBp26mdxmzxMPrJiCncEiHBbMCtFqsEl1uXK%2BYDF7E1oCeX5hFlRCJuDpzO6cbkqToqIEX%2B6ghdeRfdo%2FE%2F8nqbN8m1l4oi%2FgikiYZn%2FkX%2FXU6U4Flfl%2B5p44WAkLpkYbn9lWegJuWMps8%2B1cjViUkafN0pCDt6RSJF1MHLWk84bJViBM3nZyXw0lRbnDvMSyhy1sgKvcTLD%2FWvulu1ooo2gJFmwrf7YQtFlyB5FZ5wS39dZ7jtS4fCOK5j4DZVX3zwCPC6sHRp3fgXQ9yKvsaEmlXpiTf3HzPeTEoRfiTXb%2F7iYjHjGoLaJMCndhj4hgFTDk4rGHBjq%2BAQtN3Lv%2B%2Blkak%2FfJaqKuI4%2Bt%2BOFlMDDJ6CRpEdxGZtlqgH%2F0WrpGaPZq8TLSRoc8d5EFOTeELnvr%2FEJSti7Wd9QGrKDSMcodLluS5J4K4v764gBS5tfnDdARfFkZJURuBnEp9lQxFWIrngj9wlgSk7Tt5dLP9MTz%2FuDSTbeffFRIwEQr0vZjgJXxQIDihFhPsxq9I [0]
AWS_IOT_EVT_FOTA_START
D: Receiving up to 2048 bytes at 0x200181fc...
D: Read 538 bytes from socket
D: GET header size: 266
D: HTTP response
D: 48 54 54 50 2f 31 2e 31 |HTTP/1.1
D: 20 34 30 33 20 46 6f 72 | 403 For
D: 62 69 64 64 65 6e 0d 0a |bidden..
D: 78 2d 61 6d 7a 2d 72 65 |x-amz-re
D: 71 75 65 73 74 2d 69 64 |quest-id
D: 3a 20 51 52 5a 4b 4a 35 |: QRZKJ5
D: 33 4d 34 57 4a 4d 4b 4e |3M4WJMKN
D: 51 47 0d 0a 78 2d 61 6d |QG..x-am
D: 7a 2d 69 64 2d 32 3a 20 |z-id-2: 
D: 6c 58 42 6c 32 51 45 30 |lXBl2QE0
D: 4f 78 69 42 63 7a 69 67 |OxiBczig
D: 70 48 71 33 52 31 78 56 |pHq3R1xV
D: 79 70 4a 69 39 5a 77 69 |ypJi9Zwi
D: 42 54 65 63 47 73 2b 42 |BTecGs+B
D: 69 76 4a 6b 38 75 35 55 |ivJk9u5U
D: 67 58 44 44 38 56 5a 4f |gXDD8VZO
D: 57 6d 66 64 77 64 31 4f |Wmfdwd1O
D: 77 59 4f 4c 6b 36 32 34 |wYOLk624
D: 76 7a 77 3d 0d 0a 43 6f |vzw=..Co
D: 6e 74 65 6e 74 2d 54 79 |ntent-Ty
D: 70 65 3a 20 61 70 70 6c |pe: appl
D: 69 63 61 74 69 6f 6e 2f |ication/
D: 78 6d 6c 0d 0a 54 72 61 |xml..Tra
D: 6e 73 66 65 72 2d 45 6e |nsfer-En
D: 63 6f 64 69 6e 67 3a 20 |coding: 
D: 63 68 75 6e 6b 65 64 0d |chunked.
D: 0a 44 61 74 65 3a 20 4d |.Date: M
D: 6f 6e 2c 20 31 32 20 4a |on, 12 J
D: 75 6c 20 32 30 32 31 20 |ul 2021 
D: 31 36 3a 34 34 3a 32 31 |16:44:21
D: 20 47 4d 54 0d 0a 53 65 | GMT..Se
D: 72 76 65 72 3a 20 41 6d |rver: Am
D: 61 7a 6f 6e 53 33 0d 0a |azonS3..
D: 0d 0a                   |..      
E: Server response is not 200 Success
E: Download client error
E: FOTA download failed, report back
E: AWS_FOTA_EVT_ERROR
Unknown AWS IoT event type: 11

I have only been able to download the FOTA image when I do not use a presigned URL and use a publicly available s3 bucket.

I have tried using a presigned URL with a private s3 bucket and that has not worked so it seems that the error has to do with using a presigned URL and not the device.

Thanks for the help,

Joe

[Didrik Rokhaug]

I generated a pre-signed URL myself, and compared it with the one in your header.

My header had several more parameters in the URL.

While your URL only had X-Amz-Security-Token, while mine had X-Amz-Algorithm, X-Amz-Credential, X-Amz-Date, X-Amz-Expires, X-Amz-SignedHeaders, X-Amz-Security-Token and X-Amz-Signature.

How did you generate the URL?

[Joe Boettcher]

I am not generating the pre-signed URL myself. It is automatically generated by creating the job in AWS.

Here is my job document:

{
  "operation": "app_fw_update",
  "fwversion": "v1.2.21",
  "size": 350000,
  "location": {
    "protocol": "https:",
    "host": "my-fota-secure.s3.amazonaws.com",
    "path": "${aws:iot:s3-presigned-url:https://s3.region.amazonaws.com/my-fota-secure/app_update.bin}"
   }
}

This seemed to be the method recommended by AWS. Should I only attempt the other format -- like this below?

{
  "operation": "app_fw_update",
  "fwversion": "v1.0.2",
  "size": 181124,
  "location": {
    "protocol": "http:",
    "host": "my-fota-secure.s3.amazonaws.com",
    "path": "file_path?X-Amz-Algorithm=_____&X-Amz-Credential=____..."
  }
}