Download Client Redirect Querry

RE: Download Client Redirect Querry

dejans — Thu, 19 Dec 2024 14:06:14 GMT

Hi,

[quote user="danielboe"]I solved the Redirect now manually via socket.[/quote]

This is great to hear. Thank you for the update.

Best regards,
Dejan

RE: Download Client Redirect Querry

danielboe — Wed, 18 Dec 2024 20:12:34 GMT

I solved the Redirect now manually via socket. I read the complete http response and get my URL. If anybody else ever needs this here is my function:

#include <zephyr/net/socket.h>

char *redirectHttp(const char *endpoint, const char *path) {
    int fd = 0;
    int err = 0;
    int bytes;
    char send_packet[200];
    char recv_buf[1024] = {};
    struct addrinfo *res;
    struct addrinfo hints = {
        .ai_family = AF_INET,
        .ai_socktype = SOCK_STREAM,
    };

    err = getaddrinfo(endpoint, "80", &hints, &res);
    if (err) {
        LOG_ERR("[SOCKET] getaddrinfo err %d", errno);
        return 0;
    }
    ((struct sockaddr_in *)res->ai_addr)->sin_port = htons(80);

    fd = socket(AF_INET, SOCK_STREAM, IPPROTO_TCP);  // Set socket to tcp without tls
    if (fd == -1) {
        LOG_ERR("[SOCKET] failed to open");
        return 0;
    }
    char ipv4_addr[NET_IPV4_ADDR_LEN];
    inet_ntop(AF_INET, &((struct sockaddr_in *)res->ai_addr)->sin_port, ipv4_addr, sizeof(ipv4_addr));
    LOG_INF("[SOCKET] Connect to Addr IP: %s", ipv4_addr);
    err = connect(fd, res->ai_addr, sizeof(struct sockaddr_in));
    if (err) {
        LOG_ERR("[SOCKET] connect() err: %d", errno);
        return 0;
    }

    sprintf(send_packet,
            "GET %s HTTP/1.1\r\n"
            "Host: %s\r\n"
            "Accept: txt/plain\r\n\r\n",
            path, endpoint);
    bytes = send(fd, &send_packet, strlen(send_packet), 0);
    LOG_INF("[SOCKET] Sent %d bytes", bytes);
    bytes = blocking_recv(fd, recv_buf, sizeof(recv_buf), 0);
    LOG_INF("[SOCKET] Received %d bytes", bytes);

    const char *location_header = "Location: ";
    const char *start = strstr(recv_buf, location_header);
    if (start) {
        start += strlen(location_header);

        const char *end = strstr(start, "\r\n");
        if (!end) {
            end = start + strlen(start);  // End of string if no CRLF
        }

        size_t length = end - start;

        if (length < 1024) {
            char *url = (char *)malloc(length + 1);  // Allocate memory for the URL
            strncpy(url, start, length);
            url[length] = '\0';  // Null-terminate the string
            return url;
        } else {
            LOG_ERR("[SOCKET] URL is too long to fit in the buffer");
            return 0;
        }
    } else {
        LOG_ERR("[SOCKET] Location header not found in the response");
    }

    return 0;
}

Example Usage:

const char *endpoint = "ul1.epaperframe.de";
const char *path = "/tjoLta9.dl";
char *url = redirectHttp(endpoint, path);
printf("Redirected URL: %s\n", url);
free(url);  // Free the allocated memory

This ticket is solved for me now.

RE: Download Client Redirect Querry

dejans — Fri, 13 Dec 2024 15:47:09 GMT

Hi,

This is one option and decision is yours. It might be the easiest since there is support for HTTP transport in this new downloader library.

Best regards,
Dejan

RE: Download Client Redirect Querry

danielboe — Thu, 12 Dec 2024 13:47:26 GMT

So do you think waiting for this merged is the best option?

RE: Download Client Redirect Querry

dejans — Thu, 12 Dec 2024 12:16:01 GMT

Hi,

There is an update to my previous reply. Support for HTTP transport is now added in the downloader library. Please have a look at this pull request which should also show how to read headers from request.

Best regards,
Dejan

RE: Download Client Redirect Querry

danielboe — Thu, 12 Dec 2024 11:34:15 GMT

This sounds like what I want. Is there any sample to read headers from a request?

RE: Download Client Redirect Querry

dejans — Thu, 12 Dec 2024 09:52:04 GMT

Hi,

Unfortunately, there is no support for URL shorteners in download client library. Resolving short URL before calling download client library with expanded URL is currently the solution. In your case, you could consider doing an HTTP HEAD request using the Zephyr HTTP library, then read out the Location header with the expanded URL and use that subsequently with the download client. It is also recommended that you use HTTPS instead of HTTP due to security concern.

Best regards,
Dejan

RE: Download Client Redirect Querry

danielboe — Wed, 11 Dec 2024 16:00:09 GMT

NCS 2.7.0, Firmware is 2.0.1 (nrf9151)

RE: Download Client Redirect Querry

dejans — Wed, 11 Dec 2024 15:18:07 GMT

Hi,

[quote user="dejans"]Which NCS version do you use?
[/quote]

Which NCS version and modem firmware version do you use?

Best regards,
Dejan

RE: Download Client Redirect Querry

danielboe — Fri, 06 Dec 2024 14:20:19 GMT

exactly, this was my starting point. also downloading the sample works in my code. the config itself can indeed work. here is the full config I use:

CONFIG_REBOOT=y
CONFIG_CRC=y

# GPIO
CONFIG_GPIO=y
CONFIG_PWM=y

#RTT Log
CONFIG_LOG=y
CONFIG_LOG_MODE_MINIMAL=y
CONFIG_UART_CONSOLE=n
CONFIG_RTT_CONSOLE=y
CONFIG_LOG_BACKEND_UART=n
CONFIG_USE_SEGGER_RTT=y
CONFIG_LOG_DEFAULT_LEVEL=3

# Heap and stacks
CONFIG_HEAP_MEM_POOL_SIZE=8096
CONFIG_MAIN_STACK_SIZE=4096
CONFIG_SYSTEM_WORKQUEUE_STACK_SIZE=2048


#WDT
CONFIG_WDT_LOG_LEVEL_DBG=y
CONFIG_WATCHDOG=y
CONFIG_WDT_DISABLE_AT_BOOT=n

# SPI
CONFIG_SPI=y

# LTE link control
CONFIG_LTE_LINK_CONTROL=y
CONFIG_LTE_NETWORK_MODE_NBIOT=y
CONFIG_MODEM_INFO=y
CONFIG_MODEM_KEY_MGMT=y

#Networking
CONFIG_NET_SOCKETS=y
CONFIG_NETWORKING=y
CONFIG_NET_NATIVE=n #TODO check this one
CONFIG_NRF_MODEM_LIB=y

# Date Time library
CONFIG_DATE_TIME=y
CONFIG_DATE_TIME_MODEM=y

# cJSON
CONFIG_CJSON_LIB=y

# Image manager
CONFIG_FLASH_MAP=y
CONFIG_STREAM_FLASH=y
CONFIG_MCUBOOT_IMG_MANAGER=y
CONFIG_IMG_MANAGER=y

#OTA
CONFIG_FLASH=y
CONFIG_FLASH_PAGE_LAYOUT=y
CONFIG_IMG_ERASE_PROGRESSIVELY=y
CONFIG_BOOTLOADER_MCUBOOT=y

# AWS FOTA
CONFIG_AWS_FOTA=y
CONFIG_FOTA_DOWNLOAD=y
CONFIG_FOTA_SOCKET_RETRIES=200
CONFIG_FOTA_DOWNLOAD_PROGRESS_EVT=y
CONFIG_DFU_TARGET=y

# Download Client
CONFIG_DOWNLOAD_CLIENT=y
CONFIG_DOWNLOAD_CLIENT_STACK_SIZE=4096
CONFIG_DOWNLOAD_CLIENT_LOG_LEVEL_DBG=y

#AWS IOT Settings
CONFIG_AWS_IOT=y
CONFIG_AWS_IOT_BROKER_HOST_NAME="XXX-ats.iot.eu-central-1.amazonaws.com"
CONFIG_MQTT_HELPER_SEC_TAG=6
CONFIG_AWS_IOT_TOPIC_UPDATE_DELTA_SUBSCRIBE=y
CONFIG_AWS_IOT_AUTO_DEVICE_SHADOW_REQUEST=n
CONFIG_MQTT_CLEAN_SESSION=y
CONFIG_MQTT_KEEPALIVE=200

# Non-volatile Storage
CONFIG_NVS=y
CONFIG_MPU_ALLOW_FLASH_WRITE=y
CONFIG_PM_PARTITION_SIZE_NVS_STORAGE=0xE000
CONFIG_LZ4=y

RE: Download Client Redirect Querry

dejans — Thu, 05 Dec 2024 18:00:08 GMT

Hi,

Can you show your full project configuration?

Do you use this Download Client sample?

Which NCS version do you use?

Best regards,
Dejan

RE: Download Client Redirect Querry

danielboe — Wed, 04 Dec 2024 15:19:38 GMT

The ShortUrl is generated on a Serverside and send to the device. I can give you some snippet that shows how it works.

prj.conf:

# Download Client
CONFIG_DOWNLOAD_CLIENT=y
CONFIG_DOWNLOAD_CLIENT_STACK_SIZE=4096
CONFIG_DOWNLOAD_CLIENT_LOG_LEVEL_DBG=y

there is Downloader:

#define URL "http://ul1.epaperframe.de/NDUmX2z.dl"
static struct download_client downloader;
static struct download_client_cfg config = {};      
      
        if (awsGetCloudConnected() && !doOnce) {
            doOnce = true;
            LOG_INF("[MAIN] DL INIT");
            int err = download_client_init(&downloader, callbackDl);
            if (err) {
                printk("[MAIN] Failed to initialize the client, err %d", err);
            }

            err = download_client_get(&downloader, URL, &config, URL, 0);
            if (err) {
                printk("[MAIN] Failed to start the downloader, err %d", err);
                return 0;
            }

            printk("[MAIN] Downloading %s\n", URL);
        }

        if (dlClientDone) {
            dlClientDone = false;
            char recoverDataStr[2050] = {};
            strncpy(recoverDataStr, downloader.buf, sizeof(recoverDataStr));
            printk("[DL] Payload: \"%s\"\n", recoverDataStr);
        }

And there is the callback:

static int callbackDl(const struct download_client_evt *event) {
    static size_t downloaded;
    static size_t file_size;

    if (downloaded == 0) {
        download_client_file_size_get(&downloader, &file_size);
        downloaded += 0;
    }

    switch (event->id) {
        case DOWNLOAD_CLIENT_EVT_FRAGMENT:
            downloaded += event->fragment.len;
            if (file_size) {
                progress_print(downloaded, file_size);
                char recoverDataStr[2200] = {};
                strncpy(recoverDataStr, event->fragment.buf, event->fragment.len);
                printk("%s \n", recoverDataStr);
            } else {
                printk("\r[ %d bytes ] ", downloaded);
            }
            return 0;
        case DOWNLOAD_CLIENT_EVT_CLOSED:
            printk("[DL] CLOSED\n");
            dlClientDone = true;
            return 0;
        case DOWNLOAD_CLIENT_EVT_DONE:
            printk("[DL] DONE\n");
            return 0;
        case DOWNLOAD_CLIENT_EVT_ERROR:
            printk("[DL] Error %d during download\n", event->error);
            return -1;
    }
    return 0;
}

I am not sure what you mean by more log, but here is a sample output with LOG DEBUG (DOWNLOAD_CLIENT_LOG_LEVEL_DBG):

00> *** Booting nRF Connect SDK v2.7.0-5cb85570ca43 ***
00> *** Using Zephyr OS v3.6.99-100befc70c74 ***
00> I: [MAIN] Starting application 2.0.1
00> I: 16 Sectors of 4096 bytes
00> I: alloc wra: 2, fc0
00> I: data wra: 2, a20
00> I: [LED] led sample
00> I: [SPI] init
00> I: [DISPLAY] sleep start
00> I: [DISPLAY] WAIT finished
00> I: [DISPLAY] sleep done
00> I: [MAIN] Chip is NEW - set LTE-M
00> I: [MODEM] Modem Band Lock Set
00> I: [MODEM] Version: 2.0.1
00> I: [MODEM] Network registration status: Connected - roaming
00> I: [MODEM] PSM parameter update: TAU: 3240, Active time: -1
00> I: [MODEM] client_id: nrf-35940*
00> I: [AWS] iot_connect - init done
00> W: [AWS] iot_reconnect - wait 30 s connect try: 1/30
00> W: [AWS] IOT_EVT_CONNECTED
00> I: [MODEM] BOOT Image Confirmed
00> I: [MAIN] DL INIT
00> D: state = 1
00> I: Downloading: http://ul1.epaperframe.de/NDUmX2z.dl [0]
00> [MAIN] Downloading http://ul1.epaperframe.de/NDUmX2z.dl
00> D: Port not specified, using default: 80
00> D: Failed to resolve hostname ul1.epaperframe.de on IPv6
00> D: family: 1, type: 1, proto: 6
00> I: Connecting to 3.5.134.246
00> D: fd 1, addrlen 8, fam IPv4, port 80
00> D: state = 2
00> D: Receiving up to 2048 bytes at 0x2000cbd8...
00> D: Read 620 bytes from socket
00> D: GET header size: 620
00> E: Unexpected HTTP response: 301 moved permanently
00> [DL] Error -77 during download
00> D: state = 4
00> D: state = 0
00> [DL] CLOSED
00> D: Connection closed
00> [DL] Payload: "http/1.1 301 moved permanently"

The links are save to stay public But they wont work anymore after some time

The Download Client Sample worked fine btw. so thats not the issue.

RE: Download Client Redirect Querry

dejans — Wed, 04 Dec 2024 13:13:41 GMT

Hi Daniel,

[quote user=""]The Files are located in a bucket and I use a shortURL to get the signedDownload URL.[/quote]

Can you show how (and where in code) you use this shortURL?

[quote user=""]If I use this short URL in Download Client I get the Error "http/1.1 301 moved permanently"[/quote]

Can you provide full log which contains the error?

Best regards,
Dejan