AWS FOTA Update inconsistent

RE: AWS FOTA Update inconsistent

Marte Myrvold — Wed, 26 May 2021 13:56:48 GMT

Hi Adeel,

Good to hear that you figured out what was causing the issue, and that you were able to create a workaround for now.

Have you tested how this works with the AWS FOTA sample, is the behavior the same there when you have a job in queue before your device connects? It might be related to how AWS IoT notifies your device about a job. As I have mentioned before, in the sample the device receives notifications about available jobs through MQTT, by subscribing to the app_fw_update job topic. So it might be that AWS IoT only sends a notification on this topic when the job is created. If so, you should see the same behavior in the AWS FOTA sample as well, and you will have to implement something so that it will notify your device later if it is not connected when the job is created. The other possibility is that there is something on the application side that causes you to not receive this notification.

Modem trace is used by us to look at the communication with the modem, to try and figure out what is wrong if for example a disconnect happens. Since it seems like you have figured out what the problem it, I am not sure a modem trace is necessary, but you can still get one if you are still seeing errors from the modem in your log. You must use the Trace Collector in nRF Connect for Desktop to get a modem trace. The guide for doing this with Thingy:91 can be found here. Make sure that you have updated the board controller firmware and enabled tracing in the application.

Best regards,

Marte

RE: AWS FOTA Update inconsistent

Adeel — Tue, 25 May 2021 12:05:45 GMT

Hi Marte,

Sorry for the late reply. Actually the issue that I have noticed is that if I keep a job in queue before connecting to the AWS IoT, then the job is not picked up by the Thingy.

However, if I create the job when the Thingy is subscribed to the AWS , then it works perfectly. So, as a work around for now, I am creating the job when I am subscribing to the FOTA and that works for now.

What I am curious to see is why does the issue occur when the job is already created and queued in the Job section of AWS IoT:

Also, could you let me know how to get the modem trace from the Link monitor ?

Regards,

Adeel.

RE: AWS FOTA Update inconsistent

Marte Myrvold — Wed, 19 May 2021 14:42:36 GMT

Hi Adeel,

I see you are getting errors -5 and -128. -5 is EIO (input/output error), and -128 is ENOTCONN (socket is not connected). In cellular examples, EIO can be returned if you are sending an AT command to the modem and the function failed to send the command, but I am not certain why you are getting that error in your case.

Could you get a modem trace of when this happens and attach it here?

Best regards,

Marte

RE: AWS FOTA Update inconsistent

Adeel — Tue, 18 May 2021 15:37:00 GMT

Hi Marte,

Thanks for clarifying this for me. I actually am no longer getting the Version Mismatch error.

I am getting a different error which i will try to briefly explain to get some inputs from you:

I am actually connecting to a different server and subscribing to it via MQTT and getting a command from the server. The command is received by the device when it subscribes to the server. I try to initiate my FOTA update through this command (FOTA). When I do it, the FOTA works perfectly once and updates. When I try to do it the next time, I get a Job rejected error inspite of having a job available in the AWS IoT.

I get an error that I think is mostly due a conflict with the MQTT payload buf.

devzone.nordicsemi.com/.../Error-in-2nd-FOTA-update.txt

Could you help me understand where the issue could be ? I have integrated the AWS_FOTA sample (for my FOTA updates) in my application along with the mqtt_simple application (for my MQTT data) as well.

Regards,

Adeel

RE: AWS FOTA Update inconsistent

Marte Myrvold — Wed, 12 May 2021 14:56:38 GMT

Hi,

The job version that version mismatch reacts to is not the same as fwversion, but the version of the job execution. This will be the field "versionNumber" in a JobExecution object, which can for example look like:

{
"execution" : {
    "jobId" : "022",
    "thingName" : "MyThing",
    "jobDocument" : "< contents of job document >",
    "status" : "IN_PROGRESS",
    "queuedAt" : 1489096123309,
    "lastUpdatedAt" : 1489096123309,
    "versionNumber" : 1,
    "executionNumber" : 1234567890
},
"clientToken" : "client-1",
"timestamp" : 1489088524284,
}

You can read more about job executions here and here. I have not been able to figure out what might cause this error, as there is not much information about it unfortunately.

The device will subscribe to the topic, and if it receives a message it will go into the case MQTT_EVT_PUBLISH in mqtt_evt_handler(). It does not check the topic for messages and see if it is empty or if there is a message there. When it subscribes to a topic it will only receive a message when one is sent, and that is when it will react and handle that message. If you want to disconnect if it does not receive any messages on that topic for some time you can create some sort of timer or timeout, and then disconnect if it times out without any new messages being received on the topic.

Best regards,

Marte

RE: AWS FOTA Update inconsistent

Adeel — Wed, 12 May 2021 09:26:40 GMT

Hi Marte,

[quote userid="92402" url="~/f/nordic-q-a/74821/aws-fota-update-inconsistent/309151#309151"]Are you experiencing the VersionMismatch issue if you use a clean version of the AWS FOTA sample, or only with your own application?[/quote]

I only experience it with my application. Could it be something related to my version number in my Job document ?

[quote userid="92402" url="~/f/nordic-q-a/74821/aws-fota-update-inconsistent/309151#309151"]In this case you can for example disconnect if you do not receive any messages on this topic.[/quote]

Where could I call the mqtt_disconnect() function to do this exactly in the AWS_FOTA sample ? I would like to it in such a way that I disconnect if there is no message on the topic. Can you please help me with this ?

Regards,

Adeel.

RE: AWS FOTA Update inconsistent

Marte Myrvold — Tue, 11 May 2021 08:53:44 GMT

Hi,

Are you experiencing the VersionMismatch issue if you use a clean version of the AWS FOTA sample, or only with your own application?

You can disconnect from the MQTT client with the function mqtt_disconnect(). What you want the trigger that causes your device to disconnect to be is up to you. In the AWS FOTA sample the device subscribes to the app_fw_update job topic, so it will be notified when a update job is created. In this case you can for example disconnect if you do not receive any messages on this topic.

Best regards,

Marte

RE: AWS FOTA Update inconsistent

Adeel — Fri, 07 May 2021 14:01:53 GMT

Hi Marte,

Thanks, that would be great. Also, I sometimes face an issue where I have no job available and the AWS MQTT stays connected. I want it to disconnect gracefully if there is no job in the server.

How can I have a do an implementation wherein the MQTT can gracefully disconnect if it cannot find any job in the AWS IoT jobs ? Currently, it stays connected all the time and does not disconnect.

Regards,

Adeel.

RE: AWS FOTA Update inconsistent

Marte Myrvold — Fri, 07 May 2021 13:18:29 GMT

Hi,

From your log I can see that is says VersionMismatch. You get this error if the version of the job executed stored in Jobs does not match expected version. Each time you update the job execution, its version is incremented. So it seems like the version has incremented for some reason when it should not have, or that the expected version is incorrect. I am not certain why there is a mismatch, as I have not seen this error happening in NCS before. There could be a problem if you are running multiple instances of the same job simultaneously. I did find someone experiencing the same issue on the AWS forum, so you can try the suggestion there. I will keep looking into this and see if I can find any reasons for why you are getting VersionMismatch.

Best regards,

Marte

RE: AWS FOTA Update inconsistent

Adeel — Thu, 06 May 2021 13:35:38 GMT

Hi Marte,

[quote userid="92402" url="~/f/nordic-q-a/74821/aws-fota-update-inconsistent/308554#308554"]Here the application goes into mqtt_evt_handler() and the case MQTT_EVT_PUBLISH twice, so you are receiving two publish events on your device.[/quote]

This is a command that I send through MQTT to start the FOTA, so it should not interfere with the issue here.

However, I see an error that is logged after the job is rejected.

devzone.nordicsemi.com/.../Rejected-with-error-message.txt

It says a mismatch error but I could not understand what that is. This could maybe be related to my Job file as it says a mismatch.

I am attaching my job file for reference.

devzone.nordicsemi.com/.../Job.txt

Could you let me know what this error means ?

One thing that is peculiar here is that : FOTA update is successful once, but is rejected the next time.

Its happening in an alternate way (once successful, once rejected) which is really weird.

Regards,

Adeel.

RE: AWS FOTA Update inconsistent

Marte Myrvold — Thu, 06 May 2021 12:47:08 GMT

Hi Adeel,

From the log it seems like you are receiving the job twice the times it is rejected:

[mqtt_evt_handler:162] MQTT client connected!
Subscribing to: sensorstest1/nrf352656100980047/commands2
[mqtt_evt_handler:190] MQTT PUBLISH result=0 len=4
Received: FOTA
Payload buf is : FOTA
[mqtt_evt_handler:190] MQTT PUBLISH result=0 len=4
Received: FOTA
Payload buf is : FOTA

Here the application goes into mqtt_evt_handler() and the case MQTT_EVT_PUBLISH twice, so you are receiving two publish events on your device.

There was a known issue where the device received multiple notify-next messages in AWS FOTA before, but this was fixed before NCS v1.5.0, so you should not get this error when you are using v1.5.1. The changes that were done to fix this issue can be found here, but this part in nrf/subsys/net/lib/aws_fota/src/aws_fota.c has been changed since then, so it does not look the same in v1.5.1 as in that change. However, the for when get_job_execution() is called is the same in v1.5.1:

	if (is_notify_next_topic) {
		if (fota_state == NONE) {
			LOG_DBG("Checking for an available job");

			return get_job_execution(client, payload_len);
		}
		
...

	} else if (is_get_next_topic || is_get_accepted) {
		LOG_DBG("Checking for an available job");
		return get_job_execution(client, payload_len);
	}

You can also try to check your device's shadow in AWS IoT, to get more information about why the job is rejected. The device should be publishing to the /update topic, which will create and update the shadow of your device. When the job is rejected, AWS IoT will publish a message to the $aws/things/thingName/shadow/name/shadowName/update/rejected. This message contains an error response document that describes the error. For more information about this you can check out Interacting with shadows, Device Shadow service documents, and Device Shadow error messages.

Please check that your application has the correct behavior in regards to multiple notify-next messages. If this issue persists, could you please get a modem trace of this behavior?

Best regards,

Marte