FOTA Hangs

RE: FOTA Hangs

Simon — Mon, 25 May 2020 22:58:46 GMT

I tried to repeat the process, and I did not see the behavior. I might have misunderstood what you meant. I did the following:

Modified the aws_fota example as described in my last answer, with CONFIG_MQTT_KEEPALIVE=930
Uploaded the sample to a bucket and programmed the same aws_fota example onto the nrf9160

Then I uploaded a job which contained the job file pointing to the uploaded aws_fota example
Saw that the aws_fota sample recognized the job and started downloading the image. After the DFU was finished it would reset the chip automatically and run the newly updated aws_fota sample
repeat step 1-3

I ended 1,2,3 step-loop above after about 5-6 iterations and did not see the behavior you explained. Have I misunderstood? Should I repeat the process more? Could you provide a detailed step-by-step method of reproducing it?

You said, "without powering off the target board please". Did that include not resetting it? However, it will do that automatically when the download is completed.

Best regards,

Simon

RE: FOTA Hangs

John — Mon, 25 May 2020 20:09:56 GMT

Thank you Simon.

Now try and repeat the process a few times, without powering off the target board please.

I would expect you to eventually see what I am seeing.

RE: FOTA Hangs

Simon — Mon, 25 May 2020 10:08:48 GMT

I am not able to reproduce your behavior with the aws_fota example in the master branch (commit 8f44fbc625f1ad).

I set CONFIG_USE_CLOUD_CLIENT_ID=y in addition to CONFIG_CLOUD_CLIENT_ID, CONFIG_CLOUD_CERT_SEC_TAG and CONFIG_MQTT_BROKER_HOSTNAME to appropriate values.

I added a job and the sample successfully started a download immediately after:

I: No queued jobs for this device
I: Checking for an available job
I: Subscribe: $aws/things/simon_client_id/jobs/fun_and_cool_job/update/#
I: Subscribed to FOTA update topic $aws/things/simon_client_id/jobs/fun_and_cool_job/update/#
I: Publish topic: $aws/things/simon_client_id/jobs/fun_and_cool_job/update
I: Publish payload {"status":"IN_PROGRESS","statusDetails": null,"expectedVersion": "1","clientToken": ""}
[mqtt_evt_handler:228] PUBACK packet id: 9062
I: Start downloading firmware from simon-simon123.s3-us-west-2.amazonaws.com/app_update.bin
I: Attempting to connect over IPv4
I: Connected to simon-simon123.s3-us-west-2.amazonaws.com
I: Downloading: app_update.bin [0]
AWS_FOTA_EVT_START, job id = fun_and_cool_job
I: Downloaded 4096/179380 bytes (2%)
I: 2 Sectors of 4096 bytes
I: alloc wra: 0, d18
I: data wra: 0, 180
I: Erasing sector at offset 0x00087000
I: Erasing sector at offset 0x00088000
I: Downloaded 8192/179380 bytes (4%)

I also tried to set CONFIG_MQTT_KEEPALIVE=930, but the download was still successful. I'm not able to see the behavior you're explaining.

Could you provide me the exact step to reproduce your behavior?

Best regards,

Simon

RE: FOTA Hangs

Simon — Mon, 25 May 2020 09:39:14 GMT

I will test it out, I have to set CONFIG_MQTT_KEEPALIVE=930 in addition I assume?

RE: FOTA Hangs

John — Mon, 25 May 2020 08:53:18 GMT

Yes I probably agree.

However, may I ask if you run the version of fota that exists in Master branch if you have the same experience that I'm having?

I'm expecting that you will.

RE: FOTA Hangs

Simon — Mon, 25 May 2020 08:47:49 GMT

I talked to one of the developers and got some tips.

First, could you change the way you are logging in order to get more information (it seems like you're using minimal)? If you use deferred logging with larger buffers you may get some more information.

In addition, MQTT_KEEPALIVE is determined by the server as well, so you might want to look at that side if the session is shut down when it exceeds 60 seconds.

Best regards,

Simon

RE: FOTA Hangs

Simon — Mon, 18 May 2020 19:05:53 GMT

I notice that your device doesn't receive the publish event "Job Id update accepted topic", and I am not sure why this is. I will ask internally about this and get back to you.

Best regards,

Simon

RE: FOTA Hangs

John — Fri, 15 May 2020 08:31:13 GMT

Thank you very much Simon, very much appreciated.

RE: FOTA Hangs

Simon — Fri, 15 May 2020 08:01:27 GMT

Hello, I am sorry for the delay in this ticket, I've been quite busy lately. I will prioritize it today and the upcoming working days.

Best regards,

Simon

RE: FOTA Hangs

John — Mon, 11 May 2020 20:20:48 GMT

Sure, but it in no way helps fix the problem.

I may be missing something but in order to implement that as a solution we would have to utilise persistent storage to notice that we needed to perform a reboot, after a reboot.

RE: FOTA Hangs

Simon — Mon, 11 May 2020 10:19:11 GMT

Any progress on this?

Another way of resuming the FOTA update, besides resetting the DK, is to run lte_lc_offline() followed by reconnecting again.

RE: FOTA Hangs

John — Wed, 06 May 2020 07:53:14 GMT

Hi Simon, no stress, but very keen for a hand with this issue.

RE: FOTA Hangs

John — Wed, 06 May 2020 07:23:46 GMT

I can now confirm that if I program a DK with the latest Asset Tracker (Built from nRF Connect SDK master branch commit bc7ade8b) and link it to NRF Cloud and do a FOTA update.

The same thing occurs whereby the device will "notice" that there is an update and begin to perform the update and then hang.

If I preset reset on the DK it will reboot and perform the update correctly.

RE: FOTA Hangs

Tiago Costa — Thu, 30 Apr 2020 10:06:03 GMT

I have changed CONFIG_MQTT_KEEPALIVE=60 and now the FOTA download request is received earlier when the system is searching for satellites. The FOTA still hangs here, ------STRIKETHROUGH----- but it does not show "Enabling PSM" as before. Can't tell what's going on here, this hang seems to be different than before.------STRIKETHROUGH----- After receiving the FOTA download request, there was a long pause that seemed like an hang, but then the console printed (and then it really hang):

[00:06:17.860,290] <dbg> nrf9160_gps.stop: Stopping GPS
[00:06:17.866,363] <inf> gps_control: GPS operation was stopped
[00:06:17.872,833] <inf> gps_control: The device will try to get fix again in 30 seconds
[00:06:47.881,530] <inf> gps_control: Enabling PSM

Before changing the keepalive, I have experimented with enabling the Watchdog in prj.conf as it was not enabled by default. After "Enabling PSM", the WD rebooted the system so it is definitely hanging when trying to enable PSM after GPS runs for a while and a FOTA update request is received as soon as the GPS searching stops.

I also wonder if the position of switch 2 of the DK has any influence over this as it messes with EDRX.

EDIT: Never mind my previous sentence, as I have CONFIG_POWER_OPTIMIZATION_ENABLE=n which disables switch 2.

RE: FOTA Hangs

John — Thu, 30 Apr 2020 09:04:39 GMT

Would you mind checking something for me please?

Change CONFIG_MQTT_KEEPALIVE=60

Perhaps through prj.conf

GPS won't work probably, but I'm very interested to see if you notice that FOTA works.

RE: FOTA Hangs

Tiago Costa — Thu, 30 Apr 2020 08:59:10 GMT

Yes, that makes perfect sense and matches what I'm also observing here. The FOTA download only works simultaneously with GPS searching mode if the download started before the device begins to search for satellites.

It's just weird that, occasionally, when the device stops searching for satellites and receives the update request from the socket. that it hangs before starting to download the file and while "Enabling PSM". Can't tell if its a problem with the asset_tracker example, an SDK library or modem firmware / carrier issue.

RE: FOTA Hangs

John — Thu, 30 Apr 2020 07:42:03 GMT

I'm confident about one aspect.

When the GPS is searching, the mode is in PSM, the device has no ability to notice socket communication until either forced communication over cellular which will reset the PSM cycle or until RX is turned on at the end of the TAU period specified when PSM was initiated.

So this is why you wouldn't notice any FOTA update initiation immdeiately, during the GPS cycle.

I think...

RE: FOTA Hangs

Tiago Costa — Wed, 29 Apr 2020 21:05:14 GMT

Hi guys,

Don't know if my issue is related to this, as my CONFIG_MQTT_KEEPALIVE=1200 by default and I haven't changed it. But I'm also on SDK 1.2.0 and have seen the example asset_tracker hang when GPS is active or after GPS has been active (with modem firmware v1.1.1 and now also with v1.2.0). I'm connected to a NB-IOT network.

I've noticed that:

While the modem is searching for satellites for about 360s, no firmware update requests are received (shouldn't LTE have priority here)?
Sometimes, upgrade requests are received and handled correctly, but only after the modem stops searching for sats
Other times, the app simply hangs after searching for sats, right after displaying "Enabling PSM" and never recovers. Shouldn't the watchdog reboot the app after some time? A manual reboot using the reset button does cause the app to receive and handle the firmware update later as requested.
Also, when the app does not hang, I can only upgrade the app image and not mcuboot even though I also have the immutable bootloader. The MCUBOOT images (s0 or s1) are downloaded by the app and the system reboots, then MCUBOOT tests the image, but upon a manual reset the MCUBOOT image is reverted and the MCUBOOT is not upgraded. (don't know if this is related to the app or MCUBOOT)

RE: FOTA Hangs

Simon — Wed, 29 Apr 2020 02:23:15 GMT

I will be gone until monday (4th of may) and I'm not able to look at
your issue until then

Best regards,
Simon

RE: FOTA Hangs

Simon — Mon, 27 Apr 2020 10:20:04 GMT

I'll take a look at your issue tomorrow (5/27)

RE: FOTA Hangs

John — Mon, 27 Apr 2020 07:39:00 GMT

After spending several days on this now, I’m a lot closer to understanding what may be happening with FOTA, the CONFIG_MQTT_KEEPALIVE default value is 60 seconds.

In order for us to use PSM effectively we must be able to extend this value for as long as possible.

During the testing of our GPS software I set it to 930 seconds (prj.conf) , anything longer than that seemed to cause socket reliability to deteriorate.

However when I have implemented FOTA and left CONFIG_MQTT_KEEPALIVE=930 I experience the issue whereby FOTA will start after waking from PSM if there is a job available and then hang, I can tell by looking at the current draw that PSM is still functioning, but nothing more is seen on the log output.

A subsequent reboot will allow FOTA to complete.

If I reprogram the device and allow either the default for CONFIG_MQTT_KEEPALIVE to be set or manually set CONFIG_MQTT_KEEPALIVE=60 in prj.conf

FOTA seems to function properly.

I would be very grateful for any advice you may be able to offer please.