nrF9160 sample aws_fota

RE: nrF9160 sample aws_fota

Marte Myrvold — Fri, 09 Apr 2021 12:51:59 GMT

Hi,

It seems like the trace doesn't show the whole exchange, but it looks like the server closes the connection for some reason. I haven't figured out why yet, but it might be that the server times out. If so, requesting smaller fragments might help.

You could also try to see if AWS IoT has reported the reason for the disconnect. AWS IoT publishes connect and disconnect events to some MQTT topics, and you can also see this in your device's shadow, as I mentioned in an earlier reply. Go to AWS IoT, Manage, Things, <your thing>, Shadows. A disconnect message will contain the element "disconnectReason", which will have one of the following valid values as the reason for the disconnect:

Disconnect reason	Description
`AUTH_ERROR`	The client failed to authenticate or authorization failed.
`CLIENT_INITIATED_DISCONNECT`	The client indicates that it will disconnect. The client can do this by sending either a MQTT `DISCONNECT` control packet or a `Close frame` if the client is using a WebSocket connection.
`CLIENT_ERROR`	The client did something wrong that causes it to disconnect. For example, a client will be disconnected for sending more than 1 MQTT `CONNECT` packet on the same connection or if the client attempts to publish with a payload that exceeds the payload limit.
`CONNECTION_LOST`	The client-server connection is cut off. This can happen during a period of high network latency or when the internet connection is lost.
`DUPLICATE_CLIENTID`	The client is using a client ID that is already in use. In this case, the client that is already connected will be disconnected with this disconnect reason.
`FORBIDDEN_ACCESS`	The client is not allowed to be connected. For example, a client with a denied IP address will fail to connect.
`MQTT_KEEP_ALIVE_TIMEOUT`	If there is no client-server communication for 1.5x of the client's keep-alive time, the client is disconnected.
`SERVER_ERROR`	Disconnected due to unexpected server issues.
`SERVER_INITIATED_DISCONNECT`	Server intentionally disconnects a client for operational reasons.
`THROTTLED`	The client is disconnected for exceeding a throttling limit.
`WEBSOCKET_TTL_EXPIRATION`	The client is disconnected because a WebSocket has been connected longer than its time-to-live value.

This can also be found here.

Best regards,

Marte

RE: nrF9160 sample aws_fota

Marte Myrvold — Wed, 31 Mar 2021 13:38:09 GMT

Hi,

I have started looking at your log and modem trace, but I have not been able to figure out what the issue is yet. Due to Easter we are short staffed, so response time might be longer than usual, but I will get back to you.

Best regards,

Marte

RE: nrF9160 sample aws_fota

dermot geraghty — Mon, 29 Mar 2021 14:33:48 GMT

Hi Marte,

I reset the system and queued a job which downloaded and rebooted the system.

Then I queued a new job.

There was a long delay and then it seemed to start the new job - but then failed.

Attaching the modem trace.

Thanks

Ddevzone.nordicsemi.com/.../trace_2D00_2021_2D00_03_2D00_29T14_2D00_06_2D00_40.238Z.bin

RE: nrF9160 sample aws_fota

Marte Myrvold — Mon, 29 Mar 2021 11:36:09 GMT

Hi,

Good to hear that updating to v1.5.0 fixed your previous issue!

How often does the disconnect happen?

The only difference I can see between when it succeeded and when it failed is that right after the transport write is complete, the modem sends the AT command +CSCON: 0 to the MCU, which indicates that the modem is in idle mode. This happens on line line 795 in your log (compared to line 157 and after, where the download succeeded). I also see that the modem starts switching between being in idle and connected (+CSCON: 1) mode after some time, from line 708 and on. I am not sure why this is happening.

Could you get a modem trace of this happening?

Best regards,

Marte

RE: nrF9160 sample aws_fota

dermot geraghty — Sun, 28 Mar 2021 12:36:46 GMT

Hi Marte,

Thanks you for that.

I updated to V1.5.0.

The FOTA has now worked.

I have tested it multiple times and it downloads and resets correctly.

However, sometimes the MQTT client disconnects and the download doesn't complete as can be seen in the attached log.

can you advise why this might be?

The AT command at the end i triggered myself to see if the system was still alive

Thanks

Dermotdevzone.nordicsemi.com/.../log_5F00_28_5F00_03.txt

RE: nrF9160 sample aws_fota

Marte Myrvold — Fri, 26 Mar 2021 13:55:36 GMT

Hi,

The developers said that it looks like you are receiving the job twice. This was an issue that was fixed before the 1.5.0 release of NCS. The changes that were done in regards to this can be found here, where you can also find a description of the issue and what was done to fix this, which I have copied below:

When entering PSM the network will cache TCP packets. This lead to
multiple notify-next messages being sent when the device was in PSM as
the MQTT ack was never recived so the server would do retrasmission.

To combat this we check what state we are in when we receive a
notify-next topic so that we won't accept it and break the download.

Addition to this we add a timeout on accepted so that things running in
the same context won't be blocked forever.

To solve this issue you can either update to NCS v1.5.0, or add the changes to your SDK. All the necessary changes can be found in the link above, but I have also added them myself in v1.4.2 and attached them here, so you can just replace the relevant files. The zip file contains the files Kconfig and aws_fota.c. Replace them with the corresponding files found in <ncs_folder>/nrf/subsys/net/lib/aws_fota and <ncs_folder>/nrf/subsys/net/lib/aws_fota/src

devzone.nordicsemi.com/.../aws_5F00_fota.zip

Best regards,

Marte

RE: nrF9160 sample aws_fota

Marte Myrvold — Fri, 26 Mar 2021 09:56:26 GMT

Hi,

Thank you for clarifying. I have asked our developers for help with this, so I will get back to you when I have more information.

Best regards,

Marte

RE: nrF9160 sample aws_fota

dermot geraghty — Wed, 24 Mar 2021 13:53:10 GMT

Hi Marte,

Sorry I meant to send you just the log from the most recent rest - attached now.

The previous resets I did myself.

For this log I deleted all job files on AWS and started a new one.

It looks like it is starting to do the down load but gets stuck.

On AWS its says the download is 'in progress'

devzone.nordicsemi.com/.../log_5F00_24_5F00_03_5F00_nbr2.txt

RE: nrF9160 sample aws_fota

Marte Myrvold — Wed, 24 Mar 2021 13:13:09 GMT

Hi,

Good to hear that you got a bit further. I have started looking at your log, but I will need some more time before I can say what is wrong. I also have some questions that might help me figuring this out.

I see that your device restarted several times during the log. Did you restart it yourself, or did it restart on its own? Does the download get stuck every time, or only sometimes? Can you point me to when in the log it gets stuck, is it when you get "Job document update was rejected", or is it somewhere else?

Best regards,

Marte

RE: nrF9160 sample aws_fota

dermot geraghty — Wed, 24 Mar 2021 12:45:24 GMT

Hi Marte

Got a bit further with this.

But now the download gets stuck as can be seen in the attached log file.

Any idea why this might be?

Tks

devzone.nordicsemi.com/.../log_5F00_24_5F00_03_5F00_nbr1.txt

RE: nrF9160 sample aws_fota

dermot geraghty — Fri, 19 Mar 2021 14:10:49 GMT

Hi Marte,

Here is the terminal log.

Thanks,

Dermotdevzone.nordicsemi.com/.../2021_2D00_03_2D00_19T12_5F00_07_5F00_08.380Z_2D00_log.txt

RE: nrF9160 sample aws_fota

Marte Myrvold — Fri, 19 Mar 2021 13:58:27 GMT

Hi,

Could you also try going into AWS IoT, Manage, Tings, <your thing>, Shadows and look at the device's shadow there? You might be able to find a reason for the disconnect there, under "disconnectReason". The description of the different disconnect reasons can be found here.

Best regards,

Marte

RE: nrF9160 sample aws_fota

Marte Myrvold — Fri, 19 Mar 2021 13:43:59 GMT

Hi,

Could you share the output log from the terminal? If you are using LTE Link Monitor you can get this by clicking on the "Open log file" button.

Best regards,

Marte

RE: nrF9160 sample aws_fota

dermot geraghty — Fri, 19 Mar 2021 13:10:53 GMT

Hi Marte,

I have done that and attached the new log file.

Thanks

Dermotdevzone.nordicsemi.com/.../trace_2D00_2021_2D00_03_2D00_19T12_2D00_08_2D00_38.774Z.bin

RE: nrF9160 sample aws_fota

Marte Myrvold — Fri, 19 Mar 2021 12:42:39 GMT

Hi,

The server is sending an encrypted alert that closes the TLS session. This happens right after something is sent, so the problem might be what is being sent. Could you enable more logging so we can see what is being sent right before the connection is closed? You can enable it by adding the following to prj.conf:

CONFIG_NET_LOG=y
CONFIG_MQTT_LOG_LEVEL_DBG=y
CONFIG_LTE_LINK_CONTROL_LOG_LEVEL_DBG=y
CONFIG_AWS_FOTA_LOG_LEVEL_DBG=y

Best regards,

Marte

RE: nrF9160 sample aws_fota

Marte Myrvold — Thu, 18 Mar 2021 14:19:33 GMT

Hi,

I will look at the modem trace and come back to you when I find anything.

Best regards,

Marte

RE: nrF9160 sample aws_fota

dermot geraghty — Wed, 17 Mar 2021 17:26:32 GMT

Hi Marte,

We did follow the guide for setting up the S2 bucket and creating a thing in AWS IoT - and correctly I think.

I'm attaching the modem trace.

I am using nrF9160DK.

devzone.nordicsemi.com/.../trace_2D00_2021_2D00_03_2D00_17T16_2D00_20_2D00_13.110Z.bin

,Thanks for your help,

Dermot

RE: nrF9160 sample aws_fota

Marte Myrvold — Wed, 17 Mar 2021 14:03:47 GMT

Hi,

Error -128 is ENOTCONN, socket is not connected. The device is being disconnected, but I do not know why. Is the cloud side configured correctly? Did you follow the guide for the sample when creating a thing in AWS IoT and setting up an AWS S3 bucket?

Could you please take a modem trace? This will help me figure out why you are being disconnected.

Best regards,

Marte

RE: nrF9160 sample aws_fota

dermot geraghty — Tue, 16 Mar 2021 14:35:39 GMT

Hi Marte,

Thanks for your help.

I have changed the prj.conf file as per your directions and moved things on a bit.

It does successfully connect to the MQTT broker but I'm not seeing that it is susbcribing to the topic (as per Nordic documentation)

[mqtt_evt_handler:129] MQTT client connected!
[00:00:14.106,140] <inf> aws_jobs: Subscribe: $aws/things/nrf-aws-fota/jobs/notify-next

My topic is:

$aws/things/TDG_logger_thing/jobs/notify-next

Here is what I am seeing:

client_id: TDG_logger_thing

hostname: a3vecosqszzrnz-ats.iot.eu-west-1.amazonaws.com

[mqtt_evt_handler:182] MQTT client connected!

[mqtt_evt_handler:235] PUBACK packet id: 50475

[mqtt_evt_handler:245] SUBACK packet id: 2114

[mqtt_evt_handler:193] MQTT client disconnected -128

ERROR: mqtt_input -128

Disconnecting MQTT client...

Could not disconnect MQTT client. Error: -128



Can you tell me what is wrong here?

Many thanks

RE: nrF9160 sample aws_fota

Marte Myrvold — Tue, 16 Mar 2021 09:12:53 GMT

Hi,

mqtt_connect -22 is the error EINVAL, which means invalid argument. This is most likely caused because the MQTT configuration is incorrect.

From your prj.conf I see that you have copied configurations from the AWS IoT sample. The AWS FOTA sample does not use the AWS IoT library, but the AWS FOTA library, which you can see under dependencies in the documentation of the sample. Therefore, you should not use CONFIG_AWS_IOT to configure your sample. Instead, you should configure it as described in the sample, under sample configuration. This description is for how to configure it using menuconfig in SES (SEGGER Embedded Studio), but if you instead want to configure it in prj.conf, which I recommend using instead of SES and menuconfig, you should add the following configs, and switch out the values so they correspond with your values:

# AWS FOTA
CONFIG_CLOUD_CERT_SEC_TAG=12345678
CONFIG_USE_CLOUD_CLIENT_ID=y
CONFIG_CLOUD_CLIENT_ID="your_client_id"
CONFIG_MQTT_BROKER_HOSTNAME="your_aws_mqtt_broker_hostname.amazonaws.com"
CONFIG_MQTT_BROKER_PORT=8883

The reason I recommend changing configurations in prj.conf and not in menuconfig in SES is because the changes done in menuconfig are only done in your current project, so if you reload the project or close it and open it again, the changes will not be there anymore, as they will be overwritten by prj.conf when you run CMake, which is run when you open a project in SES.

For later projects, if it is unclear what configs you have to set in prj.conf and the documentation only explains how to configure the project using menuconfig, you can find the configs by doing the following:

Open the project in SES
Go to Project > Configure nRF Connect SDK Project > menuconfig
Press the + next to the project settings to open up the settings (in AWS FOTA this is MQTT FOTA Sample Settings)
Select and change the configurations you want to use in your project
When you are done select Configure and wait for the configuration to finish
There will now be a file in your build folder with the generated configs for your project, in <build_folder>/zephyr/.config, so in the case of the AWS FOTA sample this will be <ncs_folder>/nrf/samples/nrf9160/aws_fota/build/zephyr/.config.
You can then copy the relevant configs from the .config file to your prj.conf. The relevant MQTT configs in the case of AWS FOTA can be found under # MQTT FOTA Sample Settings

Best regards,

Marte