AWS FOTA using presigned URL from private S3 bucket

[Joe Boettcher]

Hello,

I've gotten the sample AWS FOTA to work on my nrf9160 using a very permissive public S3 bucket, but for production I need FOTAs to use a private S3 bucket.

According to https://devzone.nordicsemi.com/f/nordic-q-a/70753/aws-fota-download-from-s3-authentication/309614#309614, using presigned URLs is advised to allow for using a private S3 bucket. This makes sense; however, there is no documentation how to do this and I have not been able to get it to work. Would you simply append the query params to the path as shown below?

Example (failed) attempt at IoT Job using presigned URL:

{
  "operation": "app_fw_update",
  "fwversion": "v1.0.2",
  "size": 181124,
  "location": {
    "protocol": "http:",
    "host": "myfota-secure.s3.amazonaws.com",
    "path": "file_path?X-Amz-Algorithm=_____&X-Amz-Credential=____..."
  }
}

Thanks for the help

[Didrik Rokhaug]

Hi,

Yes, appending the query parameters to the file path should work, though depending on the length of the URL, you might have to increase the size of the buffer used to hold the path.

You can do this by setting CONFIG_AWS_FOTA_FILE_PATH_MAX_LEN and CONFIG_DOWNLOAD_CLIENT_MAX_FILENAME_SIZE to the required size.

The default value is 192 bytes.

Best regards,

Didrik

[Joe Boettcher]

Hi Didrik,

Thanks for the assistance.

The defaults were:

- CONFIG_AWS_FOTA_FILE_PATH_MAX_LEN = 255

- CONFIG_DOWNLOAD_CLIENT_MAX_FILENAME_SIZE = 192.

I raised those both to 500. I measured my full file path length to be about 380 characters.

Example:

https://my-secure-fota-bin.s3.us-east-1.amazonaws.com/app_update.bin?X-Amz-Algorithm=_____&X-Amz-Credential=______________________&X-Amz-Date=_____________&X-Amz-Expires=____&X-Amz-SignedHeaders=____&X-Amz-Signature=__________________________

Unfortunately, this did not fix the problem and I still get the same error as before:

00> RRC mode: Connected
00> I: Connecting to my-secure-fota-bin.s3.amazonaws.com
00> I: Downloading: app_update.bin?X-Amz-Algorithm=... [0]
00> AWS_IOT_EVT_FOTA_START
00> E: Server did not honor partial content request
00> E: Download client error
00> E: FOTA download failed, report back
00> E: AWS_FOTA_EVT_ERROR
00> AWS_IOT_EVT_ERROR, 0

Do you know what might be causing this error?

[Didrik Rokhaug]

Are you sure the URL is correct?
I noticed that you have a different signature in the log and the job document.

[Joe Boettcher]

Ah my mistake. I gave the wrong job.json document in the last response. The job.json actually looked like this:

{
  "operation": "app_fw_update",
  "fwversion": "v1.2.24",
  "size": 350000,
  "location": {
    "protocol": "http:",
    "host": "my-fota.s3.amazonaws.com",
    "path": "app_update.bin?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIA3LPM4TIXXAAJZFCB%2F20210726%2Fus-east-2%2Fs3%2Faws4_request&X-Amz-Date=20210726T195002Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=host&X-Amz-Signature=dee3e19d0eb22a80d732598cc292a4f7584ec6fcf3c4502649b377f81487d26a"
   }
}

As a double-check I performed the test again.

job.json:

{
  "operation": "app_fw_update",
  "fwversion": "v1.2.24",
  "size": 350000,
  "location": {
    "protocol": "http:",
    "host": "myiot-unsecure-fota.s3.amazonaws.com",
    "path": "app_update.bin?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIA3LPM4TIXXAAJZFCB%2F20210727%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20210727T152641Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=host&X-Amz-Signature=b6ffe1fc4fcec539e84ea7c1ac5a8ef117723e9f5476fe3dfc35f507603c9f08"
   }
}

logs:

D: Protocol not specified, defaulting to HTTP(S)
D: Port not specified, using default: 80
D: family: 1, type: 1, proto: 6
I: Configuring socket timeout (30 s)
I: Connecting to myiot-unsecure-fota.s3.amazonaws.com
D: fd 1, addrlen 8, fam IPv4, port 80
D: HTTP request
D: 47 45 54 20 2f 61 70 70 |GET /app
D: 5f 75 70 64 61 74 65 2e |_update.
D: 62 69 6e 3f 58 2d 41 6d |bin?X-Am
D: 7a 2d 41 6c 67 6f 72 69 |z-Algori
D: 74 68 6d 3d 41 57 53 34 |thm=AWS4
D: 2d 48 4d 41 43 2d 53 48 |-HMAC-SH
D: 41 32 35 36 26 58 2d 41 |A256&X-A
D: 6d 7a 2d 43 72 65 64 65 |mz-Crede
D: 6e 74 69 61 6c 3d 41 4b |ntial=AK
D: 49 41 33 4c 50 4d 34 54 |IA3LPM4T
D: 49 58 58 41 41 4a 5a 46 |IXXAAJZF
D: 43 42 25 32 46 32 30 32 |CB%2F202
D: 31 30 37 32 37 25 32 46 |10727%2F
D: 75 73 2d 65 61 73 74 2d |us-east-
D: 31 25 32 46 73 33 25 32 |1%2Fs3%2
D: 46 61 77 73 34 5f 72 65 |Faws4_re
D: 71 75 65 73 74 26 58 2d |quest&X-
D: 41 6d 7a 2d 44 61 74 65 |Amz-Date
D: 3d 32 30 32 31 30 37 32 |=2021072
D: 37 54 31 35 32 36 34 31 |7T152641
D: 5a 26 58 2d 41 6d 7a 2d |Z&X-Amz-
D: 45 78 70 69 72 65 73 3d |Expires=
D: 33 36 30 30 26 58 2d 41 |3600&X-A
D: 6d 7a 2d 53 69 67 6e 65 |mz-Signe
D: 64 48 65 61 64 65 72 73 |dHeaders
D: 3d 68 6f 73 74 26 58 2d |=host&X-
D: 41 6d 7a 2d 53 69 67 6e |Amz-Sign
D: 61 74 75 72 65 3d 62 36 |ature=b6
D: 66 66 65 31 66 63 34 66 |ffe1fc4f
D: 63 65 63 35 33 39 65 38 |cec539e8
D: 34 65 61 37 63 31 61 63 |4ea7c1ac
D: 35 61 38 65 66 31 31 37 |5a8ef117
D: 37 32 33 65 39 66 35 34 |723e9f54
D: 37 36 66 65 33 64 65 63 |76fe3dec
D: 33 35 66 35 30 37 36 30 |35f50760
D: 33 63 39 66 30 38 20 48 |3c9f08 H
D: 54 54 50 2f 31 2e 31 0d |TTP/1.1.
D: 0a 48 6f 73 74 3a 20 64 |.Host: m
D: 62 73 64 61 79 63 2f 60 |yiot-uns
D: 65 64 63 73 35 2d 66 6f |ecure-fo
D: 74 61 2e 73 33 2e 61 6d |ta.s3.am
D: 61 7a 6f 6e 61 77 73 2e |azonaws.
D: 63 6f 6d 0d 0a 52 61 6e |com..Ran
D: 67 65 3a 20 62 79 74 65 |ge: byte
D: 73 3d 30 2d 0d 0a 43 6f |s=0-..Co
D: 6e 6e 65 63 74 69 6f 6e |nnection
D: 3a 20 6b 65 65 70 2d 61 |: keep-a
D: 6c 69 76 65 0d 0a 0d 0a |live....
I: Downloading: app_update.bin?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIA3LPM4TIXXAAJZFCB%2F20210727%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20210727T152641Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=host&X-Amz-Signature=b6ffe1fc4fcec539e84ea7c1ac5a8ef117723e9f5476fe3dfc35f507603c9f08 [0]
AWS_IOT_EVT_FOTA_START
D: Receiving up to 2048 bytes at 0x200181fc...
D: Read 266 bytes from socket
D: GET header size: 266
D: HTTP response
D: 48 54 54 50 2f 31 2e 31 |HTTP/1.1
D: 20 34 30 33 20 46 6f 72 | 403 For
D: 62 69 64 64 65 6e 0d 0a |bidden..
D: 78 2d 61 6d 7a 2d 72 65 |x-amz-re
D: 71 75 65 73 74 2d 69 64 |quest-id
D: 3a 20 52 4a 35 53 59 4b |: RJ5SYK
D: 34 36 4e 35 41 54 57 31 |46N5ATW1
D: 4e 42 0d 0a 78 2d 61 6d |NB..x-am
D: 7a 2d 69 64 2d 32 3a 20 |z-id-2: 
D: 61 52 53 72 5a 4e 4e 2b |aRSrZNN+
D: 4f 36 2f 55 64 51 59 51 |O6/UdQYQ
D: 61 52 2f 65 70 4b 44 4e |aR/epKDN
D: 6b 36 46 6e 32 43 4d 6a |k6Fn2CMj
D: 77 76 64 62 42 48 49 4b |wvdbBHIK
D: 35 66 6c 7a 41 4c 76 79 |5flzALvy
D: 64 56 33 70 38 51 34 33 |dV3p8Q43
D: 6e 2f 66 34 58 4d 52 56 |n/f4XMRV
D: 55 32 7a 79 2f 58 50 56 |U2zy/XPV
D: 4b 43 45 3d 0d 0a 43 6f |KCE=..Co
D: 6e 74 65 6e 74 2d 54 79 |ntent-Ty
D: 70 65 3a 20 61 70 70 6c |pe: appl
D: 69 63 61 74 69 6f 6e 2f |ication/
D: 78 6d 6c 0d 0a 54 72 61 |xml..Tra
D: 6e 73 66 65 72 2d 45 6e |nsfer-En
D: 63 6f 64 69 6e 67 3a 20 |coding: 
D: 63 68 75 6e 6b 65 64 0d |chunked.
D: 0a 44 61 74 65 3a 20 54 |.Date: T
D: 75 65 2c 20 32 37 20 4a |ue, 27 J
D: 75 6c 20 32 30 32 31 20 |ul 2021 
D: 31 35 3a 32 39 3a 31 34 |15:29:14
D: 20 47 4d 54 0d 0a 53 65 | GMT..Se
D: 72 76 65 72 3a 20 41 6d |rver: Am
D: 61 7a 6f 6e 53 33 0d 0a |azonS3..
D: 0d 0a                   |..      
E: Server response is not 200 Success
E: Download client error
E: FOTA download failed, report back
E: AWS_FOTA_EVT_ERROR
AWS_IOT_EVT_DISCONNECTED
RRC mode: Idle

[Didrik Rokhaug]

For me, it worked with this job document (the link should be valid for another hour if you want to try it yourself, but it points to a simple text file, so the DFU update won't work):

Fullscreen manual-presigned.txt Download

{
  "operation": "app_fw_update",
  "fwversion": "v1.0.2",
  "size": 181124,
  "location": {
    "protocol": "https:",
    "host": "secret-oregon-us-west-2-bucket.s3.us-west-2.amazonaws.com",
    "path": "fota-image.txt?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=ASIAZPEUHV6QBYG4NZH4%2F20210730%2Fus-west-2%2Fs3%2Faws4_request&X-Amz-Date=20210730T144916Z&X-Amz-Expires=7200&X-Amz-SignedHeaders=host&X-Amz-Security-Token=IQoJb3JpZ2luX2VjEJ%2F%2F%2F%2F%2F%2F%2F%2F%2F%2F%2FwEaCXVzLXdlc3QtMiJHMEUCIQDliA6LVPOlO1BUbDbujGZFXRvD90%2BGBOjHknvNGJg6QgIgGYoRV0ZboN6ED6TUFamjKptGgZOZVFhoVF5DfTG5Ey0qjAMIqP%2F%2F%2F%2F%2F%2F%2F%2F%2F%2FARACGgw2NTA5OTg0Mjc1NTIiDCEfSiNhLBhe%2FuRcNCrgAqE4WDxPiN%2FVy3aCFbb37GKmNgt4QqCjb8TWvEZot2bYfIbfF7jpiUw5YtzRor7RHSP1ZEFPKxjmL%2FUb8aCQrUZAjU9RyzrGAAtYbPoAijqhcRb5vcUxHpx%2BIt9GFpZJrPKyP727z3PhDZL1CwJYGr%2BuggKOUDb5tYBigx5TQVUKWAxBZGsTUp0aozWtB26XHvaFkKhp8Q5KcqpfLg1mzHxCX3Z3oHVJQuQOWecnY%2FarWR13zwGxaHCO%2BINKyIC%2Fs%2BFtxZNTIpllDApS0Gb%2FcroQXXwzzaUigqd5r9nDteUFby1ZFMoqkVVyIejORmAzuQ%2BZOItURCUQFBQssqz2O1XuGES1IdaEU8udpgRKwulznoMKhHFZ0CzMs%2BMvcTpx37rSNaQBWcD6NGpExyTw%2BRM1qamdcfwE4g8fuwZNwTQzNnoRK6Ly0QRW7JDrEtcsUooDJPHTqVzkfQaShFVgAhkwlKGQiAY6swIjNZeQmCkeTw4F78N%2B7ibGY07irKeVwFFP9F1bIMl76frq1eXoCZsDAZVA5v8TVJxEnGTMaMC9umhbAp%2FelcscOnNLfLbqqF%2FDV71MkGPZgXSNDERmYl5KoB8p3UhqO6C14gLzZGfrcXmKH4ulrDDhzeQaUd6Exsb6TN1ufi2YO0ICHRh3AVrmQ9H4RLk4uIypP5VbEKs2hnX2BCPg10jWqFjvZVW0n9A29xlOCMB1uDgWBpyJLs28XJWFJ2WBzk16IL1A3u1asdMXnTYwTLDXUCgg5pkrUpY9Q6Khf0UUqIH2HDu%2FcA%2BVTZ0xNFEhmVJGJlBUbmzoLUKrRun295dhCv9eDBU%2FBNV8MHekB9XH%2FLoxDOjqnkKEzIr1Fp0DetXIkf81ch6rBUorjbidTG4%2FOgsh&X-Amz-Signature=8a7823eadb3804fb16790fb79e87d59654d2d05f8b6907bdecb1eb6aee67e1b4"
   }
}

The most common error I got back on my attempts that didn't work, was that the token had expired.

You can find the reject reason either in the application by reading the body of the HTTP response, or by taking a modem trace, and use the preview version of the Trace Collector application to convert it to a .pcap that can be read by Wireshark.

[Joe Boettcher]

I notice the only difference between my job document and yours is that your "location.path" contains a "X-Amz-Security-Token". Is this required? Maybe this is the reason my job request fails.

I cannot figure out how to create an aws presign url with "X-Amz-Security-Token" using the aws cli. Do you know how I could create one?

[Didrik Rokhaug]

The command I use is 'aws s3 presign <s3 uri>

[Joe Boettcher]

I use the same command and it does not provide a "X-Amz-Security-Token" for some reason. Is that necessary for FOTA's to not be "403 Forbidden"?

I think the problem might stem from IAM policies. I followed the instructions from https://developer.nordicsemi.com/nRF_Connect_SDK/doc/latest/nrf/samples/nrf9160/aws_fota/README.html

My bucket policy looks like this:

{    "Version": "2012-10-17",
     "Statement": [
         {
             "Effect": "Allow",
             "Principal": "*",
             "Action": "s3:GetObject",
             "Resource": "arn:aws:s3:::bucket_name/*"
         }
      ]
 }

And my permission bucket settings look like this:

What are the permissions for your bucket?

[Didrik Rokhaug]

I accidentally had none of them blocked.

However, it also works for me if I block all public access.

A tip for debugging this issue, as long as you don't use TLS:

We recently published a preview of the Trace Collector V2, which not only let you capture modem traces, but also decode them.

Once decoded, you can inspect the IP traffic in Wireshark, and see exactly what AWS is returning. There is often some extra information in the body of the response.

This guide explains how to capture traces with the "old" trace collector.