AWS FOTA fails when using my upload.bin file for nrf9160

[hmichel]

Previously I have been successful getting the AWS FOTA update working using the AWS FOTA example project, the example update.bin file and location provided

by Nordic for the nrf9160. I did have excellent help from Nordic to get that working.

Now I have created my own update.bin from the AWS FOTA project

1) Create update.bin by making a build (get its exact file size from Windows Explorer)

2) Create an s3 bucket for the update.bin file (it has logging turned on)

3) Create another s3 bucket for the logging of events from the bucket with update.bin

4) Upload the update.bin file to the s3 bucket

5) Create a json file to point to the update.bin file

6) Upload code to nrf9160 using Segger

7) Wait for LTE connect complete and board ready for update by observing Termite terminal 

8) Create a job to perform FOTA update using the update.bin file

9) Get MQTT disconnect error

here are related screen shots

[Øyvind]

Hello,

First of all, what version of NCS are you running and what is current modem FW?

I see that you have configured your job file for an HTTPS address. Have you configured this accordingly as described in the download client? In the original job file example shown in AWS FOTA library, the address points to a nRFCloud address using HTTP.

Error message -57 is defined as follows:

ENOTCONN 57     /* Socket is not connected */

Kind regards,

Øyvind

[hmichel]

Dear Øyvind

The NCS version is 1.0.0

The Modem Firmware is 1.0.0

[Øyvind]

hmichel said:

Thank you for the help. Especially since you are not at the office

 No problem! We are here to help 

hmichel said:

in both cases the interaction starts when I start the job but then hangs even though I wait several minutes see screen below

Getting the default: 8 would indicate that your AWS Job has triggered and download started (from what I get here as well). Not sure what causes the difference in download speed. Did you reset the board before the download was complete?

2019-09-19T09:18:02.259Z DEBUG modem << [mqtt_evt_handler:129] MQTT client connected!\x0D
2019-09-19T09:18:02.460Z DEBUG modem << [mqtt_evt_handler:194] SUBACK packet id: 2112\x0D\x0A
2019-09-19T09:18:02.588Z DEBUG modem << [mqtt_evt_handler:184] PUBACK packet id: 42170\x0D\x0A
2019-09-19T09:19:58.088Z DEBUG modem << [mqtt_evt_handler:199] default: 8\x0D
2019-09-19T09:19:58.802Z DEBUG modem << [mqtt_evt_handler:184] PUBACK packet id: 30909\x0D\x0A
2019-09-19T09:23:10.104Z DEBUG modem << [mqtt_evt_handler:184] PUBACK packet id: 59531\x0D\x0A
2019-09-19T09:23:10.650Z DEBUG modem << [mqtt_evt_handler:184] PUBACK packet id: 63819\x0D\x0A
2019-09-19T09:23:10.708Z DEBUG modem << AWS_FOTA_EVT_DONE, rebooting to apply update.\x0D\x0A
2019-09-19T09:23:10.710Z DEBUG modem << ***** Booting Zephyr OS build v1.14.99-ncs3-snapshot2-1276-g4493a423a645 *****\x0D\x0A
2019-09-19T09:23:10.712Z DEBUG modem << [00:00:00.008,056] \x1B[0m<inf> mcuboot: Starting bootloader\x1B[0m\x0D\x0A
2019-09-19T09:23:10.715Z DEBUG modem << [00:00:00.016,235] \x1B[0m<inf> mcuboot: Primary image: magic=good, swap_type=0x2, copy_done=0x1, image_ok=0x1\x1B[0m\x0D\x0A
2019-09-19T09:23:10.719Z DEBUG modem << [00:00:00.028,808] \x1B[0m<inf> mcuboot: Scratch: magic=unset, swap_type=0x1, copy_done=0x3, image_ok=0x3\x1B[0m\x0D\x0A
2019-09-19T09:23:10.764Z DEBUG modem << [00:00:00.040,802] \x1B[0m<inf> mcuboot: Boot source: none\x1B[0m\x0D\x0A
2019-09-19T09:23:10.805Z DEBUG modem << [00:00:00.047,729] \x1B[0m<inf> mcuboot: Swap type: test\x1B[0m\x0D\x0A
2019-09-19T09:23:34.230Z DEBUG modem << [00:00:23.606,658] \x1B[0m<inf> mcuboot: Bootloader chainload address offset: 0xc000\x1B[0m\x0D\x0A
2019-09-19T09:23:34.279Z DEBUG modem << [00:00:23.614,196] \x1B[0m<inf> mcuboot: Jumping to the first image slot\x1B[0m\x0D\x0A
2019-09-19T09:23:34.322Z DEBUG modem << ***** Booting Zephyr OS build v1.14.99-ncs3-snapshot2-1276-g4493a423a645 *****\x0D\x0A

hmichel said:

I then followed up with trying the Nordic provided update.bin and a json file that pointed to that file.

 Can you please elaborate, I'm not sure what you mean with "Nordic provided update.bin". The json file example in AWS FOTA library includes a path to another nRFCloud page. You need to edit the host and path to reflect your cloud. The host and path in the example point to another nRFCloud account, which is a bug.

I've been trying with AWS S3 bucket address, without luck as well.

Tests are ongoing, and will keep you posted.

Kind regards,
Øyvind

[hmichel]

I would say there is not a change in download speed. I would say that after waiting several minutes for the download for my update.bin in my s3 bucket the download is stuck and therefore failed.

The "Nordic provided update.bin" is the one referred to in the AWS FOTA library example. I use that as a reference because I can always run that update successfully.

So when my FOTA update fails i rerun the example FOTA from Nordic to make sure there is not some other problem like a lost cell signal or something else.

[Øyvind]

The issue seems to be with what a described in this answer regarding HTTPS. You need to configure the Download Client to support HTTPS, as the limitations states:

• Currently, the library only uses HTTP for downloading the firmware. It is, however, possible to have it work with HTTPS - see Download client. These changes need to be applied to Firmware over-the-air download to enable downloading firmware through HTTPS.
• The library requires content-range header to be present in the HTTP response from the server. This limitation is inherited from Download client which is used by Firmware over-the-air download.

Kind regards,
Øyvind

[hmichel]

Thanks for the update. Right now my json file is set to HTTP and the FOTA using a FW image I am hosting on AWS is still failing. As I mentioned above the FOTA starts and then hangs. See the text cut from the terminal window below: Note that I have added some printfk statements to the aws_fota.c and you will see them in the output

***** Booting Zephyr OS v1.14.99-ncs2 *****
The MQTT AWS Jobs FOTA Sample Start Image
nrf_inbuilt_key_delete(16842753, 0) => result=0
nrf_inbuilt_key_delete(16842753, 1) => result=0
nrf_inbuilt_key_delete(16842753, 2) => result=0
nrf_inbuilt_key_write => result=0
nrf_inbuilt_key_write => result=0
nrf_inbuilt_key_write => result=0
LTE Link Connecting ...
LTE Link Connected!
IPv4 Address 0x072adf12
client_id: My_nRF9160_Brd01B
aws_fota_init
construct_notify_next_topic
aws_fota_mqtt_evt_handler
update_device_shadow_version
[mqtt_evt_handler:129] MQTT client connected!
aws_fota_mqtt_evt_handler
[mqtt_evt_handler:194] SUBACK packet id: 2112
aws_fota_mqtt_evt_handler
[mqtt_evt_handler:184] PUBACK packet id: 31846
aws_fota_mqtt_evt_handler
aws_fota_on_publish_evt
publish_get_payload
publish_get_payload mqtt_read_publish_payload_blocking
construct_job_id_update_topic
construct_job_id_update_topic
aws_fota_mqtt_evt_handler
[mqtt_evt_handler:199] default: 8
aws_fota_mqtt_evt_handler
update_job_execution
aws_fota_mqtt_evt_handler
[mqtt_evt_handler:184] PUBACK packet id: 12393
aws_fota_mqtt_evt_handler
aws_fota_on_publish_evt
publish_get_payload
publish_get_payload mqtt_read_publish_payload_blocking

[hmichel]

The following snapshot shows a comparison of the printfk statements when running the Nordic example of the AWS FOTA update on the left and my example using my image stored in an AWS S3 bucket on the right.

[Øyvind]

There are two different links in use here: 

1. From https://firmware.nrfcloud.com/, the link generated for your file is in the format given in the job-file example.
   http://s3.amazonaws.com/nordic-firmware-files/fcb2394f-49cf-4d66-826c-b5f2d04828b3
   An image from my account on nRF Cloud:
   
2. From https://s3.console.aws.amazon.com, the link to your file will have this format:
   https://[USERNAME].s3.eu-north-1.amazonaws.com/update.bin
   

As our sample does not support HTTPS out-of-box, it is not sufficient to change the protocol in the job file to "HTTPS" when trying to download from AWS S3. You must follow the instructions given in this link.

Please try this solution, and let me know how this works for you.

Kind regards,
Øyvind

[hmichel]

Thank you. That is very clear. I will test later today.

[Øyvind]

Another thing to test before my last comment:
In the AWS FOTA sample documentation, under Prerequisites, you are instructed to add a permissive policy for testing only. Could you try to add a new Policy with the following:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": "s3:GetObject",
            "Resource": "arn:aws:s3:::your_S3_bucket/*"
        }
    ]
} 

I found this in the AWS documentaion.

[hmichel]

Here is my new policy

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "s3:GetObject",
      "Resource": "arn:aws:s3:::mynrf9160fwupdatev1/*"
    }
  ]
}

But now the code on the board does not proceed far

enough to indicate that it is ready for the FOTA update

But my original policy

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "iot:*",
      "Resource": "*"
    }
  ]
}

At least allows the code to get ready for the FOTA update

[hmichel]

I tried this policy instead and it got me back to where I was 2 days ago. The FOTA update will

start but then get stuck in progress

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "s3:GetObject",
      "Resource": "arn:aws:s3:::mynrf9160fwupdatev1/*"
    },
    {
      "Effect": "Allow",
      "Action": "iot:*",
      "Resource": "*"
    }
  ]
}