AWS FOTA using presigned URL from private S3 bucket

[Joe Boettcher]

Hello,

I've gotten the sample AWS FOTA to work on my nrf9160 using a very permissive public S3 bucket, but for production I need FOTAs to use a private S3 bucket.

According to https://devzone.nordicsemi.com/f/nordic-q-a/70753/aws-fota-download-from-s3-authentication/309614#309614, using presigned URLs is advised to allow for using a private S3 bucket. This makes sense; however, there is no documentation how to do this and I have not been able to get it to work. Would you simply append the query params to the path as shown below?

Example (failed) attempt at IoT Job using presigned URL:

{
  "operation": "app_fw_update",
  "fwversion": "v1.0.2",
  "size": 181124,
  "location": {
    "protocol": "http:",
    "host": "myfota-secure.s3.amazonaws.com",
    "path": "file_path?X-Amz-Algorithm=_____&X-Amz-Credential=____..."
  }
}

Thanks for the help

[Didrik Rokhaug]

Hi,

Yes, appending the query parameters to the file path should work, though depending on the length of the URL, you might have to increase the size of the buffer used to hold the path.

You can do this by setting CONFIG_AWS_FOTA_FILE_PATH_MAX_LEN and CONFIG_DOWNLOAD_CLIENT_MAX_FILENAME_SIZE to the required size.

The default value is 192 bytes.

Best regards,

Didrik

[Joe Boettcher]

Hi Didrik,

Thanks for the assistance.

The defaults were:

- CONFIG_AWS_FOTA_FILE_PATH_MAX_LEN = 255

- CONFIG_DOWNLOAD_CLIENT_MAX_FILENAME_SIZE = 192.

I raised those both to 500. I measured my full file path length to be about 380 characters.

Example:

https://my-secure-fota-bin.s3.us-east-1.amazonaws.com/app_update.bin?X-Amz-Algorithm=_____&X-Amz-Credential=______________________&X-Amz-Date=_____________&X-Amz-Expires=____&X-Amz-SignedHeaders=____&X-Amz-Signature=__________________________

Unfortunately, this did not fix the problem and I still get the same error as before:

00> RRC mode: Connected
00> I: Connecting to my-secure-fota-bin.s3.amazonaws.com
00> I: Downloading: app_update.bin?X-Amz-Algorithm=... [0]
00> AWS_IOT_EVT_FOTA_START
00> E: Server did not honor partial content request
00> E: Download client error
00> E: FOTA download failed, report back
00> E: AWS_FOTA_EVT_ERROR
00> AWS_IOT_EVT_ERROR, 0

Do you know what might be causing this error?

[Didrik Rokhaug]

Hi, and sorry for the late reply.

My guess is that you get a 403 error.

Could you enable some more logging, so we can see what is sent between the device and the server?

CONFIG_DOWNLOAD_CLIENT_LOG_LEVEL_DBG=y

CONFIG_DOWNLOAD_CLIENT_LOG_HEADERS=y

(The following config options are not necessary, but can be helpful in identifying where the log lines are coming from)

CONFIG_LOG_MINIMAL=n

CONFIG_LOG_BACKEND_SHOW_COLOR=n

Best regards,

Didrik

[Joe Boettcher]

These are the resulting logs after enabling the config options.

[00:06:37.776,245] <dbg> download_client.client_connect: Protocol not specified, defaulting to HTTP(S)
[00:06:37.776,275] <dbg> download_client.client_connect: Port not specified, using default: 80
[00:06:37.776,275] <dbg> download_client.client_connect: family: 1, type: 1, proto: 6
[00:06:37.776,916] <inf> download_client: Connecting to my-secure-fota.s3.amazonaws.com
[00:06:37.776,916] <dbg> download_client.client_connect: fd 2, addrlen 8, fam IPv4, port 80
[00:06:37.867,797] <err> aws_fota: Error (-7) when trying to start firmware download
AWS_IOT_EVT_FOTA_START

[00:06:38.246,826] <dbg> download_client.client_connect: Protocol not specified, defaulting to HTTP(S)
[00:06:38.246,856] <dbg> download_client.client_connect: Port not specified, using default: 80
[00:06:38.246,887] <dbg> download_client.client_connect: family: 1, type: 1, proto: 6
[00:06:38.247,222] <inf> download_client: Connecting to my-secure-fota.s3.amazonaws.com
[00:06:38.247,222] <dbg> download_client.client_connect: fd 2, addrlen 8, fam IPv4, port 80
[00:06:38.369,812] <err> aws_fota: Error (-7) when trying to start firmware download
AWS_IOT_EVT_FOTA_START

[00:06:38.874,114] <err> aws_fota: Job document update was rejected
[00:06:38.874,145] <err> aws_fota: Job document update was rejected
[00:06:38.874,267] <err> aws_fota: {"clientToken":"","timestamp":1622574974,"exe~
[00:06:38.874,328] <err> aws_iot: AWS_FOTA_EVT_ERROR
AWS_IOT_EVT_ERROR, 0

[00:06:38.874,359] <err> aws_iot: aws_fota_mqtt_evt_handler, error: -14
AWS_IOT_EVT_DISCONNECTED

aws_iot_connect, error: -119

Next connection retry in 30 seconds

[00:06:38.880,157] <err> aws_iot: publish_get_payload, error: -5
AWS_IOT_EVT_DISCONNECTED

There appear to be errors when trying to start the firmware download because the job document update was rejected.

Edit: Ok I think I have gotten a little closer, but I am still getting an error. I believe in order to use a presigned url for the FOTA, the job document json is supposed to include a "files" object like this:

{
  "operation": "app_fw_update",
  "fwversion": "v1.2.17",
  "size": 350000,
  "location": {
    "protocol": "http:",
    "host": "my-secure-fota.s3.amazonaws.com",
    "path": "app_update.bin"
   },
   "files" : {
     "fileName" : "app_update.bin",
     "url" : "${aws:iot:s3-presigned-url:https://s3.us-east-2.amazonaws.com/my-secure-fota/app_update.bin}"
   }
}

When I tried this approach, I got the following error

[00:04:24.078,918] <err> aws_fota: Error when getting the payload: -122
[00:04:24.078,918] <err> aws_iot: aws_fota_mqtt_evt_handler, error: -122
AWS_IOT_EVT_DISCONNECTED

aws_iot_connect, error: -119

Next connection retry in 30 seconds

[00:04:24.081,726] <err> aws_iot: Incoming MQTT message too large for payload buffer
[00:04:24.081,756] <err> aws_iot: publish_get_payload, error: -122
AWS_IOT_EVT_DISCONNECTED

[Didrik Rokhaug]

I am not sure if the aws library is able to parse the "files" object. Instead, I believe you should put whole query string in the "path" property.

You can increase the buffer sizes used by the aws_iot library by setting CONFIG_AWS_IOT_MQTT_RX_TX_BUFFER_LEN and CONFIG_AWS_IOT_MQTT_PAYLOAD_BUFFER_LEN. The default for both is 1000.

The error you got originally also points to too small buffers. But those are probably CONFIG_AWS_FOTA_FILE_PATH_MAX_LEN and CONFIG_DOWNLOAD_CLIENT_MAX_FILENAME_SIZE.