AWS FOTA using presigned URL from private S3 bucket

[Joe Boettcher]

Hello,

I've gotten the sample AWS FOTA to work on my nrf9160 using a very permissive public S3 bucket, but for production I need FOTAs to use a private S3 bucket.

According to https://devzone.nordicsemi.com/f/nordic-q-a/70753/aws-fota-download-from-s3-authentication/309614#309614, using presigned URLs is advised to allow for using a private S3 bucket. This makes sense; however, there is no documentation how to do this and I have not been able to get it to work. Would you simply append the query params to the path as shown below?

Example (failed) attempt at IoT Job using presigned URL:

{
  "operation": "app_fw_update",
  "fwversion": "v1.0.2",
  "size": 181124,
  "location": {
    "protocol": "http:",
    "host": "myfota-secure.s3.amazonaws.com",
    "path": "file_path?X-Amz-Algorithm=_____&X-Amz-Credential=____..."
  }
}

Thanks for the help

[Didrik Rokhaug]

I've talked to our developers, and due to the way the download_client expects the information and aws_fota works, automatically generated presigned URLs (your first job document) doesn't work.

But, I believe you should still be able to use the second job document.

[Joe Boettcher]

Any idea why this FOTA job is failing?

My prj.conf contains the following lines:

...

# Download client (needed by AWS FOTA)
CONFIG_DOWNLOAD_CLIENT=y
CONFIG_DOWNLOAD_CLIENT_STACK_SIZE=4096

# Image manager
CONFIG_IMG_MANAGER=y
CONFIG_FLASH=y
CONFIG_IMG_ERASE_PROGRESSIVELY=y

# AWS FOTA
CONFIG_AWS_FOTA=y
CONFIG_FOTA_DOWNLOAD=y
CONFIG_DFU_TARGET=y
CONFIG_DOWNLOAD_CLIENT_RANGE_REQUESTS=n
CONFIG_AWS_IOT_MQTT_RX_TX_BUFFER_LEN=10000
CONFIG_AWS_IOT_MQTT_PAYLOAD_BUFFER_LEN=10000
CONFIG_AWS_FOTA_FILE_PATH_MAX_LEN=1024
CONFIG_DOWNLOAD_CLIENT_MAX_FILENAME_SIZE=1024
CONFIG_DOWNLOAD_CLIENT_MAX_HOSTNAME_SIZE=256
CONFIG_AWS_FOTA_PAYLOAD_SIZE=2500

# DEBUGGING DOWNLOAD CLIENT FOR FOTA
CONFIG_DOWNLOAD_CLIENT_LOG_LEVEL_DBG=y
CONFIG_DOWNLOAD_CLIENT_LOG_HEADERS=y

...

My job.json looks like this:

{
  "operation": "app_fw_update",
  "fwversion": "v1.2.24",
  "size": 350000,
  "location": {
    "protocol": "http:",
    "host": "my-fota.s3.amazonaws.com",
    "path": "app_update.bin?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIA3LPM4TIXXAAJZCDB%2F20210726%2Fus-east-2%2Fs3%2Faws4_request&X-Amz-Date=20210726T200332Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=host&X-Amz-Signature=97bbb2656c09eaccc35475bb5ad7a7e0a935279d81749a0691230014b1e62c40"
   }
}

The device logs show this:

D: Protocol not specified, defaulting to HTTP(S)
D: Port not specified, using default: 80
D: family: 1, type: 1, proto: 6
I: Configuring socket timeout (30 s)
I: Connecting to my-fota.s3.amazonaws.com
D: fd 1, addrlen 8, fam IPv4, port 80
D: HTTP request
D: 47 45 54 20 2f 61 70 70 |GET /app
D: 5f 75 70 64 61 74 65 2e |_update.
D: 62 69 6e 3f 58 2d 41 6d |bin?X-Am
D: 7a 2d 41 6c 67 6f 72 69 |z-Algori
D: 74 68 6d 3d 41 57 53 34 |thm=AWS4
D: 2d 48 4d 41 43 2d 53 48 |-HMAC-SH
D: 41 32 35 36 26 58 2d 41 |A256&X-A
D: 6d 7a 2d 43 72 65 64 65 |mz-Crede
D: 6e 74 69 61 6c 3d 41 4b |ntial=AK
D: 49 41 33 4c 50 4d 34 54 |IA3LPM4T
D: 49 58 58 41 41 4a 5a 46 |IXXAAJZF
D: 43 42 25 32 46 32 30 32 |CB%2F202
D: 31 30 37 32 36 25 32 46 |10726%2F
D: 75 73 2d 65 61 73 74 2d |us-east-
D: 32 25 32 46 73 33 25 32 |2%2Fs3%2
D: 46 61 77 73 34 5f 72 65 |Faws4_re
D: 71 75 65 73 74 26 58 2d |quest&X-
D: 41 6d 7a 2d 44 61 74 65 |Amz-Date
D: 3d 32 30 32 31 30 37 32 |=2021072
D: 36 54 31 39 35 30 30 32 |6T195002
D: 5a 26 58 2d 41 6d 7a 2d |Z&X-Amz-
D: 45 78 70 69 72 65 73 3d |Expires=
D: 33 36 30 30 26 58 2d 41 |3600&X-A
D: 6d 7a 2d 53 69 67 6e 65 |mz-Signe
D: 64 48 65 61 64 65 72 73 |dHeaders
D: 3d 68 6f 73 74 26 58 2d |=host&X-
D: 41 6d 7a 2d 53 69 67 6e |Amz-Sign
D: 61 74 75 72 65 3d 64 65 |ature=de
D: 65 33 65 31 39 64 30 65 |e3e19d0e
D: 62 32 32 63 38 30 64 37 |b22a50d7
D: 33 32 35 40 38 63 63 32 |32598cc2
D: 39 32 61 35 66 37 35 38 |92a4f758
D: 34 65 63 36 66 63 66 33 |4ec6fcf3
D: 63 34 35 30 32 36 34 39 |c4502649
D: 62 33 37 37 66 38 31 34 |b377f814
D: 38 37 64 32 36 61 20 48 |87d26a H
D: 54 54 50 2f 31 2e 31 0d |TTP/1.1.
D: 0a 48 6f 73 74 3a 20 66 |.Host: m
D: 69 71 62 69 79 60 2e 62 |y-fota-m
D: 61 6a 63 74 42 2d 66 6f |fota--fo
D: 74 61 2e 73 33 2e 61 6d |ta.s3.am
D: 61 7a 6f 6e 61 77 73 2e |azonaws.
D: 63 6f 6d 0d 0a 52 61 6e |com..Ran
D: 67 65 3a 20 62 79 74 65 |ge: byte
D: 73 3d 30 2d 0d 0a 43 6f |s=0-..Co
D: 6e 6e 65 63 74 69 6f 6e |nnection
D: 3a 20 6b 65 65 70 2d 61 |: keep-a
D: 6c 69 76 65 0d 0a 0d 0a |live....
I: Downloading: app_update.bin?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIA3LPM4TIXXAAJZFCB%2F20210726%2Fus-east-2%2Fs3%2Faws4_request&X-Amz-Date=20210726T195002Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=host&X-Amz-Signature=dee3e19d0eb22a80d742548cc292a4f7384ec6fcf3c4502649b377f81487d26a [0]
AWS_IOT_EVT_FOTA_START
D: Receiving up to 2048 bytes at 0x200181fc...
D: Read 266 bytes from socket
D: GET header size: 266
D: HTTP response
D: 48 54 54 50 2f 31 2e 31 |HTTP/1.1
D: 20 34 30 33 20 46 6f 72 | 403 For
D: 62 69 64 64 65 6e 0d 0a |bidden..
D: 78 2d 61 6d 7a 2d 72 65 |x-amz-re
D: 71 75 65 73 74 2d 69 64 |quest-id
D: 3a 20 57 5a 4d 4e 53 52 |: WZMNSR
D: 4e 59 5a 36 36 48 58 46 |NYZ66HXF
D: 46 51 0d 0a 78 2d 61 6d |FQ..x-am
D: 7a 2d 69 64 2d 32 3a 20 |z-id-2: 
D: 5a 36 6b 6d 55 52 63 32 |Z6kmURc2
D: 2f 37 34 38 48 66 6a 73 |/748Hfjs
D: 4d 75 56 4f 61 54 6a 67 |MuVOaTjg
D: 4e 38 6b 36 47 38 6d 31 |N8k6G9m1
D: 49 6a 6b 48 6e 52 35 34 |IjkHnR54
D: 71 4d 6a 6d 59 4f 56 51 |qMjmYOVQ
D: 45 55 51 4a 74 74 6e 46 |EUQJttnF
D: 58 78 35 36 4a 35 6e 56 |Xx56J5nV
D: 73 69 2f 4a 71 32 6c 6f |si/Jq2lo
D: 36 67 67 3d 0d 0a 43 6f |6gg=..Co
D: 6e 74 65 6e 74 2d 54 79 |ntent-Ty
D: 70 65 3a 20 61 70 70 6c |pe: appl
D: 69 63 61 74 69 6f 6e 2f |ication/
D: 78 6d 6c 0d 0a 54 72 61 |xml..Tra
D: 6e 73 66 65 72 2d 45 6e |nsfer-En
D: 63 6f 64 69 6e 67 3a 20 |coding: 
D: 63 68 75 6e 6b 65 64 0d |chunked.
D: 0a 44 61 74 65 3a 20 4d |.Date: M
D: 6f 6e 2c 20 32 36 20 4a |on, 26 J
D: 75 6c 20 32 30 32 31 20 |ul 2021 
D: 31 39 3a 35 36 3a 31 30 |19:56:10
D: 20 47 4d 54 0d 0a 53 65 | GMT..Se
D: 72 76 65 72 3a 20 41 6d |rver: Am
D: 61 7a 6f 6e 53 33 0d 0a |azonS3..
D: 0d 0a                   |..      
E: Server response is not 200 Success
E: Download client error
E: FOTA download failed, report back
E: AWS_FOTA_EVT_ERROR
AWS_IOT_EVT_DISCONNECTED
RRC mode: Idle

As seen in the device logs, after AWS_IOT_EVT_FOTA_START the request for the file returns 403 forbidden, causing "E: AWS_FOTA_EVT_ERROR"

[Didrik Rokhaug]

Are you sure the URL is correct?
I noticed that you have a different signature in the log and the job document.

[Joe Boettcher]

Ah my mistake. I gave the wrong job.json document in the last response. The job.json actually looked like this:

{
  "operation": "app_fw_update",
  "fwversion": "v1.2.24",
  "size": 350000,
  "location": {
    "protocol": "http:",
    "host": "my-fota.s3.amazonaws.com",
    "path": "app_update.bin?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIA3LPM4TIXXAAJZFCB%2F20210726%2Fus-east-2%2Fs3%2Faws4_request&X-Amz-Date=20210726T195002Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=host&X-Amz-Signature=dee3e19d0eb22a80d732598cc292a4f7584ec6fcf3c4502649b377f81487d26a"
   }
}

As a double-check I performed the test again.

job.json:

{
  "operation": "app_fw_update",
  "fwversion": "v1.2.24",
  "size": 350000,
  "location": {
    "protocol": "http:",
    "host": "myiot-unsecure-fota.s3.amazonaws.com",
    "path": "app_update.bin?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIA3LPM4TIXXAAJZFCB%2F20210727%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20210727T152641Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=host&X-Amz-Signature=b6ffe1fc4fcec539e84ea7c1ac5a8ef117723e9f5476fe3dfc35f507603c9f08"
   }
}

logs:

D: Protocol not specified, defaulting to HTTP(S)
D: Port not specified, using default: 80
D: family: 1, type: 1, proto: 6
I: Configuring socket timeout (30 s)
I: Connecting to myiot-unsecure-fota.s3.amazonaws.com
D: fd 1, addrlen 8, fam IPv4, port 80
D: HTTP request
D: 47 45 54 20 2f 61 70 70 |GET /app
D: 5f 75 70 64 61 74 65 2e |_update.
D: 62 69 6e 3f 58 2d 41 6d |bin?X-Am
D: 7a 2d 41 6c 67 6f 72 69 |z-Algori
D: 74 68 6d 3d 41 57 53 34 |thm=AWS4
D: 2d 48 4d 41 43 2d 53 48 |-HMAC-SH
D: 41 32 35 36 26 58 2d 41 |A256&X-A
D: 6d 7a 2d 43 72 65 64 65 |mz-Crede
D: 6e 74 69 61 6c 3d 41 4b |ntial=AK
D: 49 41 33 4c 50 4d 34 54 |IA3LPM4T
D: 49 58 58 41 41 4a 5a 46 |IXXAAJZF
D: 43 42 25 32 46 32 30 32 |CB%2F202
D: 31 30 37 32 37 25 32 46 |10727%2F
D: 75 73 2d 65 61 73 74 2d |us-east-
D: 31 25 32 46 73 33 25 32 |1%2Fs3%2
D: 46 61 77 73 34 5f 72 65 |Faws4_re
D: 71 75 65 73 74 26 58 2d |quest&X-
D: 41 6d 7a 2d 44 61 74 65 |Amz-Date
D: 3d 32 30 32 31 30 37 32 |=2021072
D: 37 54 31 35 32 36 34 31 |7T152641
D: 5a 26 58 2d 41 6d 7a 2d |Z&X-Amz-
D: 45 78 70 69 72 65 73 3d |Expires=
D: 33 36 30 30 26 58 2d 41 |3600&X-A
D: 6d 7a 2d 53 69 67 6e 65 |mz-Signe
D: 64 48 65 61 64 65 72 73 |dHeaders
D: 3d 68 6f 73 74 26 58 2d |=host&X-
D: 41 6d 7a 2d 53 69 67 6e |Amz-Sign
D: 61 74 75 72 65 3d 62 36 |ature=b6
D: 66 66 65 31 66 63 34 66 |ffe1fc4f
D: 63 65 63 35 33 39 65 38 |cec539e8
D: 34 65 61 37 63 31 61 63 |4ea7c1ac
D: 35 61 38 65 66 31 31 37 |5a8ef117
D: 37 32 33 65 39 66 35 34 |723e9f54
D: 37 36 66 65 33 64 65 63 |76fe3dec
D: 33 35 66 35 30 37 36 30 |35f50760
D: 33 63 39 66 30 38 20 48 |3c9f08 H
D: 54 54 50 2f 31 2e 31 0d |TTP/1.1.
D: 0a 48 6f 73 74 3a 20 64 |.Host: m
D: 62 73 64 61 79 63 2f 60 |yiot-uns
D: 65 64 63 73 35 2d 66 6f |ecure-fo
D: 74 61 2e 73 33 2e 61 6d |ta.s3.am
D: 61 7a 6f 6e 61 77 73 2e |azonaws.
D: 63 6f 6d 0d 0a 52 61 6e |com..Ran
D: 67 65 3a 20 62 79 74 65 |ge: byte
D: 73 3d 30 2d 0d 0a 43 6f |s=0-..Co
D: 6e 6e 65 63 74 69 6f 6e |nnection
D: 3a 20 6b 65 65 70 2d 61 |: keep-a
D: 6c 69 76 65 0d 0a 0d 0a |live....
I: Downloading: app_update.bin?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIA3LPM4TIXXAAJZFCB%2F20210727%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20210727T152641Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=host&X-Amz-Signature=b6ffe1fc4fcec539e84ea7c1ac5a8ef117723e9f5476fe3dfc35f507603c9f08 [0]
AWS_IOT_EVT_FOTA_START
D: Receiving up to 2048 bytes at 0x200181fc...
D: Read 266 bytes from socket
D: GET header size: 266
D: HTTP response
D: 48 54 54 50 2f 31 2e 31 |HTTP/1.1
D: 20 34 30 33 20 46 6f 72 | 403 For
D: 62 69 64 64 65 6e 0d 0a |bidden..
D: 78 2d 61 6d 7a 2d 72 65 |x-amz-re
D: 71 75 65 73 74 2d 69 64 |quest-id
D: 3a 20 52 4a 35 53 59 4b |: RJ5SYK
D: 34 36 4e 35 41 54 57 31 |46N5ATW1
D: 4e 42 0d 0a 78 2d 61 6d |NB..x-am
D: 7a 2d 69 64 2d 32 3a 20 |z-id-2: 
D: 61 52 53 72 5a 4e 4e 2b |aRSrZNN+
D: 4f 36 2f 55 64 51 59 51 |O6/UdQYQ
D: 61 52 2f 65 70 4b 44 4e |aR/epKDN
D: 6b 36 46 6e 32 43 4d 6a |k6Fn2CMj
D: 77 76 64 62 42 48 49 4b |wvdbBHIK
D: 35 66 6c 7a 41 4c 76 79 |5flzALvy
D: 64 56 33 70 38 51 34 33 |dV3p8Q43
D: 6e 2f 66 34 58 4d 52 56 |n/f4XMRV
D: 55 32 7a 79 2f 58 50 56 |U2zy/XPV
D: 4b 43 45 3d 0d 0a 43 6f |KCE=..Co
D: 6e 74 65 6e 74 2d 54 79 |ntent-Ty
D: 70 65 3a 20 61 70 70 6c |pe: appl
D: 69 63 61 74 69 6f 6e 2f |ication/
D: 78 6d 6c 0d 0a 54 72 61 |xml..Tra
D: 6e 73 66 65 72 2d 45 6e |nsfer-En
D: 63 6f 64 69 6e 67 3a 20 |coding: 
D: 63 68 75 6e 6b 65 64 0d |chunked.
D: 0a 44 61 74 65 3a 20 54 |.Date: T
D: 75 65 2c 20 32 37 20 4a |ue, 27 J
D: 75 6c 20 32 30 32 31 20 |ul 2021 
D: 31 35 3a 32 39 3a 31 34 |15:29:14
D: 20 47 4d 54 0d 0a 53 65 | GMT..Se
D: 72 76 65 72 3a 20 41 6d |rver: Am
D: 61 7a 6f 6e 53 33 0d 0a |azonS3..
D: 0d 0a                   |..      
E: Server response is not 200 Success
E: Download client error
E: FOTA download failed, report back
E: AWS_FOTA_EVT_ERROR
AWS_IOT_EVT_DISCONNECTED
RRC mode: Idle

[Didrik Rokhaug]

For me, it worked with this job document (the link should be valid for another hour if you want to try it yourself, but it points to a simple text file, so the DFU update won't work):

Fullscreen manual-presigned.txt Download

{
  "operation": "app_fw_update",
  "fwversion": "v1.0.2",
  "size": 181124,
  "location": {
    "protocol": "https:",
    "host": "secret-oregon-us-west-2-bucket.s3.us-west-2.amazonaws.com",
    "path": "fota-image.txt?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=ASIAZPEUHV6QBYG4NZH4%2F20210730%2Fus-west-2%2Fs3%2Faws4_request&X-Amz-Date=20210730T144916Z&X-Amz-Expires=7200&X-Amz-SignedHeaders=host&X-Amz-Security-Token=IQoJb3JpZ2luX2VjEJ%2F%2F%2F%2F%2F%2F%2F%2F%2F%2F%2FwEaCXVzLXdlc3QtMiJHMEUCIQDliA6LVPOlO1BUbDbujGZFXRvD90%2BGBOjHknvNGJg6QgIgGYoRV0ZboN6ED6TUFamjKptGgZOZVFhoVF5DfTG5Ey0qjAMIqP%2F%2F%2F%2F%2F%2F%2F%2F%2F%2FARACGgw2NTA5OTg0Mjc1NTIiDCEfSiNhLBhe%2FuRcNCrgAqE4WDxPiN%2FVy3aCFbb37GKmNgt4QqCjb8TWvEZot2bYfIbfF7jpiUw5YtzRor7RHSP1ZEFPKxjmL%2FUb8aCQrUZAjU9RyzrGAAtYbPoAijqhcRb5vcUxHpx%2BIt9GFpZJrPKyP727z3PhDZL1CwJYGr%2BuggKOUDb5tYBigx5TQVUKWAxBZGsTUp0aozWtB26XHvaFkKhp8Q5KcqpfLg1mzHxCX3Z3oHVJQuQOWecnY%2FarWR13zwGxaHCO%2BINKyIC%2Fs%2BFtxZNTIpllDApS0Gb%2FcroQXXwzzaUigqd5r9nDteUFby1ZFMoqkVVyIejORmAzuQ%2BZOItURCUQFBQssqz2O1XuGES1IdaEU8udpgRKwulznoMKhHFZ0CzMs%2BMvcTpx37rSNaQBWcD6NGpExyTw%2BRM1qamdcfwE4g8fuwZNwTQzNnoRK6Ly0QRW7JDrEtcsUooDJPHTqVzkfQaShFVgAhkwlKGQiAY6swIjNZeQmCkeTw4F78N%2B7ibGY07irKeVwFFP9F1bIMl76frq1eXoCZsDAZVA5v8TVJxEnGTMaMC9umhbAp%2FelcscOnNLfLbqqF%2FDV71MkGPZgXSNDERmYl5KoB8p3UhqO6C14gLzZGfrcXmKH4ulrDDhzeQaUd6Exsb6TN1ufi2YO0ICHRh3AVrmQ9H4RLk4uIypP5VbEKs2hnX2BCPg10jWqFjvZVW0n9A29xlOCMB1uDgWBpyJLs28XJWFJ2WBzk16IL1A3u1asdMXnTYwTLDXUCgg5pkrUpY9Q6Khf0UUqIH2HDu%2FcA%2BVTZ0xNFEhmVJGJlBUbmzoLUKrRun295dhCv9eDBU%2FBNV8MHekB9XH%2FLoxDOjqnkKEzIr1Fp0DetXIkf81ch6rBUorjbidTG4%2FOgsh&X-Amz-Signature=8a7823eadb3804fb16790fb79e87d59654d2d05f8b6907bdecb1eb6aee67e1b4"
   }
}

The most common error I got back on my attempts that didn't work, was that the token had expired.

You can find the reject reason either in the application by reading the body of the HTTP response, or by taking a modem trace, and use the preview version of the Trace Collector application to convert it to a .pcap that can be read by Wireshark.