Download client (via AWS FOTA) stops at 50% and can't reconnect to AWS

RE: Download client (via AWS FOTA) stops at 50% and can't reconnect to AWS

Håkon Alseth — Mon, 24 Jun 2024 08:58:19 GMT

Hi,

As previously mentioned, the download library is developed and tested with the IP-capable nRF devices in mind, so it highly likely needs to be adjusted/configured differently for your chosen ethernet bridge.

What I can recommend, in addition to what you've already discovered, is to try to disable automatic ranges and see if this happens to have an impact: CONFIG_DOWNLOAD_CLIENT_RANGE_REQUESTS=n

Kind regards,

Håkon

RE: Download client (via AWS FOTA) stops at 50% and can't reconnect to AWS

Marco Russi — Fri, 21 Jun 2024 13:48:28 GMT

Hi,

problem solved.

Apparently the http fragmentation size must be 4096. I have added:

CONFIG_DOWNLOAD_CLIENT_BUF_SIZE=4096

CONFIG_DOWNLOAD_CLIENT_HTTP_FRAG_SIZE_4096=y

and it now completes successfully.

If I define the fragmentation size to 1024 then I fails at 25%. The fact that it was originally failing at 50% is because the size was apparently 2048 by default.

What do you think ?

Thanks.

RE: Download client (via AWS FOTA) stops at 50% and can't reconnect to AWS

Marco Russi — Fri, 21 Jun 2024 10:59:58 GMT

HI, I have tried that but same result.

Anyway I have just found something interesting: at 50% the AWS server (peer) sends an HTTP header with "connection: close" so the download client reconnects as expected. Indeed the header length is 19 bytes longer than previous headers.

This causes a reconnection attempt that fails due to some reason still to find out but the main problem is that the peer sends "connection: close" in the http header.

Any idea why that happens?

NOTE: Ingore the BUBU part in the "Peer closed connection, ..." LOG line as I put that do better identify the line in the code. Indeed the piece of code that detects the disconnection request is function http_header_parse in http.c in the nrf download client library :

p = strnstr(client->buf, "\r\nconnection: close", sizeof(client->buf));

if (p) {

LOG_HEXDUMP_DBG(client->buf, *hdr_len, "HTTP response");

LOG_WRN("BUBU Peer closed connection, will re-connect");

client->http.connection_close = true;

}

I have also added the LOG line to print the header when that happens.

Thanks.

RE: Download client (via AWS FOTA) stops at 50% and can't reconnect to AWS

Håkon Alseth — Wed, 19 Jun 2024 08:32:19 GMT

When experiencing a fatal error / fault, please share the output.

[quote user="Marco Russi"]Anyway I am playing with all those config macros and I guess I'll need to spend days trying to find the right combination with little clue of what I am doing. [/quote]

The default setting for CONFIG_NET_BUF_DATA_SIZE is 128, try to atleast adjust it to 256, and see if the failure point moves from 50% to something else.

Kind regards,

Håkon

RE: Download client (via AWS FOTA) stops at 50% and can't reconnect to AWS

Marco Russi — Tue, 18 Jun 2024 14:59:43 GMT

Adding CONFIG_NET_BUF_DATA_POOL_SIZE implies enabling CONFIG_NET_BUF_VARIABLE_DATA_SIZE which is experimental. I have tried and a task goes in stak overflow. I would not expect that.

Anyway I am playing with all those config macros and I guess I'll need to spend days trying to find the right combination with little clue of what I am doing.

Thank you for your help so far.

RE: Download client (via AWS FOTA) stops at 50% and can't reconnect to AWS

Håkon Alseth — Tue, 18 Jun 2024 08:19:56 GMT

Hi,

As a generic note:

If you have faults occurring, please report the full log.

In this case, revert to your original NET_BUF_* configuration, and rather adjust the overall buffer size via CONFIG_NET_BUF_DATA_POOL_SIZE and see if this changes the behavior.

We have not tested the download sample with other devices than the IP-capable nRF's (nRF91-series, nRF70-series), so I would expect that there could be some adjustments that you need to do to make it fit for the W5500, in terms of generic configuration.

Kind regards,

Håkon

RE: Download client (via AWS FOTA) stops at 50% and can't reconnect to AWS

Marco Russi — Mon, 17 Jun 2024 14:48:18 GMT

I tried to set CONFIG_NET_BUF_DATA_SIZE=1024 and not set CONFIG_NET_BUF_FIXED_DATA_SIZE=y and it compiles fine. Stack overflow happen though.

RE: Download client (via AWS FOTA) stops at 50% and can't reconnect to AWS

Marco Russi — Mon, 17 Jun 2024 14:16:00 GMT

Hi, I have fixed the log as you suggested but the 50% download issue remains. I the tried to set CONFIG_NET_BUF_DATA_SIZE=1024 but that meant also setting CONFIG_NET_BUF_FIXED_DATA_SIZE=y. Unfortunately this increases RAM footprint a lot and linker fails. I anyway managed to make it but the fw crashes by stack overflow when DHCP starts (or when it gets response from the DHCP server). I then increased both my task and the system work queue stack but I can't solve the stack overflow issue. Note that I can't make stacks too big as RAM would not fit.

Any suggestions?

RE: Download client (via AWS FOTA) stops at 50% and can't reconnect to AWS

Håkon Alseth — Mon, 17 Jun 2024 11:33:37 GMT

Hi,

[quote user="Marco Russi"]I did not set CONFIG_LOG_MODE_IMMEDIATE=y anywhere so I guess that I am using the default mode. [/quote]

The default is normally deferred logging, but just to be sure; please verify by checking your build-folder/zephyr/.config file.

[quote user="Marco Russi"]When you say that I am having issues with logging do you mean that the log is related to the download issue I have? How can CONFIG_LOG_BLOCK_IN_THREAD allow the download client to reconnect successfully? And could also the log cause the disconnection at exactly 50% ?[/quote]

If your logger does in-place logging, you are effectively printing and blocking all threads while printing. This will cause problems, especially if you log in interrupt context (SPI for instance)

If you play around with the NET buffer sizes, do you see a difference in how the device behaves?

For instance by setting "CONFIG_NET_BUF_DATA_SIZE=1024" or adjusting the amount of buffers available?

Kind regards,

Håkon

RE: Download client (via AWS FOTA) stops at 50% and can't reconnect to AWS

Marco Russi — Mon, 17 Jun 2024 11:05:36 GMT

Hi thank you.

I did not set CONFIG_LOG_MODE_IMMEDIATE=y anywhere so I guess that I am using the default mode.

When you say that I am having issues with logging do you mean that the log is related to the download issue I have? How can CONFIG_LOG_BLOCK_IN_THREAD allow the download client to reconnect successfully? And could also the log cause the disconnection at exactly 50% ?

Thanks.

RE: Download client (via AWS FOTA) stops at 50% and can't reconnect to AWS

Håkon Alseth — Mon, 17 Jun 2024 11:01:19 GMT

Hi,

What log mode are you using?

If you are using log mode immediate (CONFIG_LOG_MODE_IMMEDIATE=y), then you'll do in-place logging; which will cause a timing skew in your firmware.

What I would recommend is that you use "CONFIG_LOG_MODE_DEFERRED", to ensure that logging takes place in a dedicated thread.

[quote user="Marco Russi"]Important: CONFIG_LOG_BLOCK_IN_THREAD=y is definitely crucial.[/quote]

This also implies that you're having issues with logging. Instead; try increasing the dedicated log buffer size:

CONFIG_LOG_BUFFER_SIZE=10240

Kind regards,

Håkon

RE: Download client (via AWS FOTA) stops at 50% and can't reconnect to AWS

Marco Russi — Sat, 15 Jun 2024 15:43:33 GMT

Finally, after some investigation, I confirm that disabling one of the enabled DBG LOG re-creates the issue.

I have also noticed that even during the full successful download, the disconnection happens but the download client is able to reconnect and keep downloading. Here is the related piece of RTT log:

To recap, at 50% of download the download client allegedly experiences a disconnection and can't reconnect (errno -2). If then I add:

CONFIG_LOG_BLOCK_IN_THREAD=y

CONFIG_NET_SOCKETS_LOG_LEVEL_DBG=y

CONFIG_MQTT_HELPER_LOG_LEVEL_DBG=y

CONFIG_MQTT_LOG_LEVEL_DBG=y

CONFIG_DOWNLOAD_CLIENT_LOG_LEVEL_DBG=y

the download client is able to reconnect and the full FOTA process succeeds.

Important: CONFIG_LOG_BLOCK_IN_THREAD=y is definitely crucial.

RE: Download client (via AWS FOTA) stops at 50% and can't reconnect to AWS

Marco Russi — Sat, 15 Jun 2024 10:56:08 GMT

Update:

I have managed to get a full download working. I have simply enabled the download client DBG log and it magically worked. Although if I disable the DBG level log then the issue appears again. I guess the issue has something to do with timing as the DBG level log should slow everything down due to the more verbose log being printed. Indeed I have also added these configs to not miss any LOG message:

CONFIG_SEGGER_RTT_BUFFER_SIZE_UP=4096

CONFIG_SEGGER_RTT_MODE_BLOCK_IF_FIFO_FULL=y

CONFIG_LOG_BLOCK_IN_THREAD=y

CONFIG_LOG_BLOCK_IN_THREAD_TIMEOUT_MS=-1

Any thoughts?

Here is the new prj.conf file devzone.nordicsemi.com/.../04737.prj.conf

and the full RTT log that shows that the download completed successfully. I also see the new application running as I made a specific application that prints " THIS IS THE NEW IMAGE FROM AWS S3!"

devzone.nordicsemi.com/.../download_5F00_full.log

devzone.nordicsemi.com/.../new_5F00_image_5F00_running.log