Download Client Redirect Querry

Hi,

I am using download_client to get a bigger file (100KB) to my memory via LTE-M. The Files are located in a bucket and I use a shortURL to get the signedDownload URL. If I use this short URL in Download Client I get the Error "http/1.1 301 moved permanently". This seems to be correct, the URL is initialy ofcourse a redirect. But how can I handle this in my code? I couldnt find a way to read the header files with download client nor do I see any option for redirect hops or something like this. Is there even a solution? Or is it the limits of the low level integration?

Thanks for helping and best regards

daniel

Parents
  • Hi Daniel,

    The Files are located in a bucket and I use a shortURL to get the signedDownload URL.

    Can you show how (and where in code) you use this shortURL?

    If I use this short URL in Download Client I get the Error "http/1.1 301 moved permanently"

    Can you provide full log which contains the error?

    Best regards,
    Dejan

  • The ShortUrl is generated on a Serverside and send to the device. I can give you some snippet that shows how it works.

    prj.conf:

    # Download Client
    CONFIG_DOWNLOAD_CLIENT=y
    CONFIG_DOWNLOAD_CLIENT_STACK_SIZE=4096
    CONFIG_DOWNLOAD_CLIENT_LOG_LEVEL_DBG=y


    there is Downloader:
    #define URL "http://ul1.epaperframe.de/NDUmX2z.dl"
    static struct download_client downloader;
    static struct download_client_cfg config = {};      
          
            if (awsGetCloudConnected() && !doOnce) {
                doOnce = true;
                LOG_INF("[MAIN] DL INIT");
                int err = download_client_init(&downloader, callbackDl);
                if (err) {
                    printk("[MAIN] Failed to initialize the client, err %d", err);
                }
    
                err = download_client_get(&downloader, URL, &config, URL, 0);
                if (err) {
                    printk("[MAIN] Failed to start the downloader, err %d", err);
                    return 0;
                }
    
                printk("[MAIN] Downloading %s\n", URL);
            }
    
            if (dlClientDone) {
                dlClientDone = false;
                char recoverDataStr[2050] = {};
                strncpy(recoverDataStr, downloader.buf, sizeof(recoverDataStr));
                printk("[DL] Payload: \"%s\"\n", recoverDataStr);
            }

    And there is the callback:

    static int callbackDl(const struct download_client_evt *event) {
        static size_t downloaded;
        static size_t file_size;
    
        if (downloaded == 0) {
            download_client_file_size_get(&downloader, &file_size);
            downloaded += 0;
        }
    
        switch (event->id) {
            case DOWNLOAD_CLIENT_EVT_FRAGMENT:
                downloaded += event->fragment.len;
                if (file_size) {
                    progress_print(downloaded, file_size);
                    char recoverDataStr[2200] = {};
                    strncpy(recoverDataStr, event->fragment.buf, event->fragment.len);
                    printk("%s \n", recoverDataStr);
                } else {
                    printk("\r[ %d bytes ] ", downloaded);
                }
                return 0;
            case DOWNLOAD_CLIENT_EVT_CLOSED:
                printk("[DL] CLOSED\n");
                dlClientDone = true;
                return 0;
            case DOWNLOAD_CLIENT_EVT_DONE:
                printk("[DL] DONE\n");
                return 0;
            case DOWNLOAD_CLIENT_EVT_ERROR:
                printk("[DL] Error %d during download\n", event->error);
                return -1;
        }
        return 0;
    }

    I am not sure what you mean by more log, but here is a sample output with LOG DEBUG (DOWNLOAD_CLIENT_LOG_LEVEL_DBG):

    00> *** Booting nRF Connect SDK v2.7.0-5cb85570ca43 ***
    00> *** Using Zephyr OS v3.6.99-100befc70c74 ***
    00> I: [MAIN] Starting application 2.0.1
    00> I: 16 Sectors of 4096 bytes
    00> I: alloc wra: 2, fc0
    00> I: data wra: 2, a20
    00> I: [LED] led sample
    00> I: [SPI] init
    00> I: [DISPLAY] sleep start
    00> I: [DISPLAY] WAIT finished
    00> I: [DISPLAY] sleep done
    00> I: [MAIN] Chip is NEW - set LTE-M
    00> I: [MODEM] Modem Band Lock Set
    00> I: [MODEM] Version: 2.0.1
    00> I: [MODEM] Network registration status: Connected - roaming
    00> I: [MODEM] PSM parameter update: TAU: 3240, Active time: -1
    00> I: [MODEM] client_id: nrf-35940*
    00> I: [AWS] iot_connect - init done
    00> W: [AWS] iot_reconnect - wait 30 s connect try: 1/30
    00> W: [AWS] IOT_EVT_CONNECTED
    00> I: [MODEM] BOOT Image Confirmed
    00> I: [MAIN] DL INIT
    00> D: state = 1
    00> I: Downloading: http://ul1.epaperframe.de/NDUmX2z.dl [0]
    00> [MAIN] Downloading http://ul1.epaperframe.de/NDUmX2z.dl
    00> D: Port not specified, using default: 80
    00> D: Failed to resolve hostname ul1.epaperframe.de on IPv6
    00> D: family: 1, type: 1, proto: 6
    00> I: Connecting to 3.5.134.246
    00> D: fd 1, addrlen 8, fam IPv4, port 80
    00> D: state = 2
    00> D: Receiving up to 2048 bytes at 0x2000cbd8...
    00> D: Read 620 bytes from socket
    00> D: GET header size: 620
    00> E: Unexpected HTTP response: 301 moved permanently
    00> [DL] Error -77 during download
    00> D: state = 4
    00> D: state = 0
    00> [DL] CLOSED
    00> D: Connection closed
    00> [DL] Payload: "http/1.1 301 moved permanently"

    The links are save to stay public But they wont work anymore after some time

    The Download Client Sample worked fine btw. so thats not the issue.

  • This sounds like what I want. Is there any sample to read headers from a request?

  • Hi,

    There is an update to my previous reply. Support for HTTP transport is now added in the downloader library. Please have a look at this pull request which should also show how to read headers from request.

    Best regards,
    Dejan

  • So do you think waiting for this merged is the best option?

  • Hi,

    This is one option and decision is yours. It might be the easiest since there is support for HTTP transport in this new downloader library.

    Best regards,
    Dejan

  • I solved the Redirect now manually via socket. I read the complete http response and get my URL. If anybody else ever needs this here is my function:

    #include <zephyr/net/socket.h>
    
    char *redirectHttp(const char *endpoint, const char *path) {
        int fd = 0;
        int err = 0;
        int bytes;
        char send_packet[200];
        char recv_buf[1024] = {};
        struct addrinfo *res;
        struct addrinfo hints = {
            .ai_family = AF_INET,
            .ai_socktype = SOCK_STREAM,
        };
    
        err = getaddrinfo(endpoint, "80", &hints, &res);
        if (err) {
            LOG_ERR("[SOCKET] getaddrinfo err %d", errno);
            return 0;
        }
        ((struct sockaddr_in *)res->ai_addr)->sin_port = htons(80);
    
        fd = socket(AF_INET, SOCK_STREAM, IPPROTO_TCP);  // Set socket to tcp without tls
        if (fd == -1) {
            LOG_ERR("[SOCKET] failed to open");
            return 0;
        }
        char ipv4_addr[NET_IPV4_ADDR_LEN];
        inet_ntop(AF_INET, &((struct sockaddr_in *)res->ai_addr)->sin_port, ipv4_addr, sizeof(ipv4_addr));
        LOG_INF("[SOCKET] Connect to Addr IP: %s", ipv4_addr);
        err = connect(fd, res->ai_addr, sizeof(struct sockaddr_in));
        if (err) {
            LOG_ERR("[SOCKET] connect() err: %d", errno);
            return 0;
        }
    
        sprintf(send_packet,
                "GET %s HTTP/1.1\r\n"
                "Host: %s\r\n"
                "Accept: txt/plain\r\n\r\n",
                path, endpoint);
        bytes = send(fd, &send_packet, strlen(send_packet), 0);
        LOG_INF("[SOCKET] Sent %d bytes", bytes);
        bytes = blocking_recv(fd, recv_buf, sizeof(recv_buf), 0);
        LOG_INF("[SOCKET] Received %d bytes", bytes);
    
        const char *location_header = "Location: ";
        const char *start = strstr(recv_buf, location_header);
        if (start) {
            start += strlen(location_header);
    
            const char *end = strstr(start, "\r\n");
            if (!end) {
                end = start + strlen(start);  // End of string if no CRLF
            }
    
            size_t length = end - start;
    
            if (length < 1024) {
                char *url = (char *)malloc(length + 1);  // Allocate memory for the URL
                strncpy(url, start, length);
                url[length] = '\0';  // Null-terminate the string
                return url;
            } else {
                LOG_ERR("[SOCKET] URL is too long to fit in the buffer");
                return 0;
            }
        } else {
            LOG_ERR("[SOCKET] Location header not found in the response");
        }
    
        return 0;
    }

    Example Usage:

    const char *endpoint = "ul1.epaperframe.de";
    const char *path = "/tjoLta9.dl";
    char *url = redirectHttp(endpoint, path);
    printf("Redirected URL: %s\n", url);
    free(url);  // Free the allocated memory


    This ticket is solved for me now.

Reply
  • I solved the Redirect now manually via socket. I read the complete http response and get my URL. If anybody else ever needs this here is my function:

    #include <zephyr/net/socket.h>
    
    char *redirectHttp(const char *endpoint, const char *path) {
        int fd = 0;
        int err = 0;
        int bytes;
        char send_packet[200];
        char recv_buf[1024] = {};
        struct addrinfo *res;
        struct addrinfo hints = {
            .ai_family = AF_INET,
            .ai_socktype = SOCK_STREAM,
        };
    
        err = getaddrinfo(endpoint, "80", &hints, &res);
        if (err) {
            LOG_ERR("[SOCKET] getaddrinfo err %d", errno);
            return 0;
        }
        ((struct sockaddr_in *)res->ai_addr)->sin_port = htons(80);
    
        fd = socket(AF_INET, SOCK_STREAM, IPPROTO_TCP);  // Set socket to tcp without tls
        if (fd == -1) {
            LOG_ERR("[SOCKET] failed to open");
            return 0;
        }
        char ipv4_addr[NET_IPV4_ADDR_LEN];
        inet_ntop(AF_INET, &((struct sockaddr_in *)res->ai_addr)->sin_port, ipv4_addr, sizeof(ipv4_addr));
        LOG_INF("[SOCKET] Connect to Addr IP: %s", ipv4_addr);
        err = connect(fd, res->ai_addr, sizeof(struct sockaddr_in));
        if (err) {
            LOG_ERR("[SOCKET] connect() err: %d", errno);
            return 0;
        }
    
        sprintf(send_packet,
                "GET %s HTTP/1.1\r\n"
                "Host: %s\r\n"
                "Accept: txt/plain\r\n\r\n",
                path, endpoint);
        bytes = send(fd, &send_packet, strlen(send_packet), 0);
        LOG_INF("[SOCKET] Sent %d bytes", bytes);
        bytes = blocking_recv(fd, recv_buf, sizeof(recv_buf), 0);
        LOG_INF("[SOCKET] Received %d bytes", bytes);
    
        const char *location_header = "Location: ";
        const char *start = strstr(recv_buf, location_header);
        if (start) {
            start += strlen(location_header);
    
            const char *end = strstr(start, "\r\n");
            if (!end) {
                end = start + strlen(start);  // End of string if no CRLF
            }
    
            size_t length = end - start;
    
            if (length < 1024) {
                char *url = (char *)malloc(length + 1);  // Allocate memory for the URL
                strncpy(url, start, length);
                url[length] = '\0';  // Null-terminate the string
                return url;
            } else {
                LOG_ERR("[SOCKET] URL is too long to fit in the buffer");
                return 0;
            }
        } else {
            LOG_ERR("[SOCKET] Location header not found in the response");
        }
    
        return 0;
    }

    Example Usage:

    const char *endpoint = "ul1.epaperframe.de";
    const char *path = "/tjoLta9.dl";
    char *url = redirectHttp(endpoint, path);
    printf("Redirected URL: %s\n", url);
    free(url);  // Free the allocated memory


    This ticket is solved for me now.

Children
Related