Download Client Redirect Querry

Hi,

I am using download_client to get a bigger file (100KB) to my memory via LTE-M. The Files are located in a bucket and I use a shortURL to get the signedDownload URL. If I use this short URL in Download Client I get the Error "http/1.1 301 moved permanently". This seems to be correct, the URL is initialy ofcourse a redirect. But how can I handle this in my code? I couldnt find a way to read the header files with download client nor do I see any option for redirect hops or something like this. Is there even a solution? Or is it the limits of the low level integration?

Thanks for helping and best regards

daniel

Parents
  • Hi Daniel,

    The Files are located in a bucket and I use a shortURL to get the signedDownload URL.

    Can you show how (and where in code) you use this shortURL?

    If I use this short URL in Download Client I get the Error "http/1.1 301 moved permanently"

    Can you provide full log which contains the error?

    Best regards,
    Dejan

  • The ShortUrl is generated on a Serverside and send to the device. I can give you some snippet that shows how it works.

    prj.conf:

    # Download Client
    CONFIG_DOWNLOAD_CLIENT=y
    CONFIG_DOWNLOAD_CLIENT_STACK_SIZE=4096
    CONFIG_DOWNLOAD_CLIENT_LOG_LEVEL_DBG=y


    there is Downloader:
    #define URL "http://ul1.epaperframe.de/NDUmX2z.dl"
    static struct download_client downloader;
    static struct download_client_cfg config = {};      
          
            if (awsGetCloudConnected() && !doOnce) {
                doOnce = true;
                LOG_INF("[MAIN] DL INIT");
                int err = download_client_init(&downloader, callbackDl);
                if (err) {
                    printk("[MAIN] Failed to initialize the client, err %d", err);
                }
    
                err = download_client_get(&downloader, URL, &config, URL, 0);
                if (err) {
                    printk("[MAIN] Failed to start the downloader, err %d", err);
                    return 0;
                }
    
                printk("[MAIN] Downloading %s\n", URL);
            }
    
            if (dlClientDone) {
                dlClientDone = false;
                char recoverDataStr[2050] = {};
                strncpy(recoverDataStr, downloader.buf, sizeof(recoverDataStr));
                printk("[DL] Payload: \"%s\"\n", recoverDataStr);
            }

    And there is the callback:

    static int callbackDl(const struct download_client_evt *event) {
        static size_t downloaded;
        static size_t file_size;
    
        if (downloaded == 0) {
            download_client_file_size_get(&downloader, &file_size);
            downloaded += 0;
        }
    
        switch (event->id) {
            case DOWNLOAD_CLIENT_EVT_FRAGMENT:
                downloaded += event->fragment.len;
                if (file_size) {
                    progress_print(downloaded, file_size);
                    char recoverDataStr[2200] = {};
                    strncpy(recoverDataStr, event->fragment.buf, event->fragment.len);
                    printk("%s \n", recoverDataStr);
                } else {
                    printk("\r[ %d bytes ] ", downloaded);
                }
                return 0;
            case DOWNLOAD_CLIENT_EVT_CLOSED:
                printk("[DL] CLOSED\n");
                dlClientDone = true;
                return 0;
            case DOWNLOAD_CLIENT_EVT_DONE:
                printk("[DL] DONE\n");
                return 0;
            case DOWNLOAD_CLIENT_EVT_ERROR:
                printk("[DL] Error %d during download\n", event->error);
                return -1;
        }
        return 0;
    }

    I am not sure what you mean by more log, but here is a sample output with LOG DEBUG (DOWNLOAD_CLIENT_LOG_LEVEL_DBG):

    00> *** Booting nRF Connect SDK v2.7.0-5cb85570ca43 ***
    00> *** Using Zephyr OS v3.6.99-100befc70c74 ***
    00> I: [MAIN] Starting application 2.0.1
    00> I: 16 Sectors of 4096 bytes
    00> I: alloc wra: 2, fc0
    00> I: data wra: 2, a20
    00> I: [LED] led sample
    00> I: [SPI] init
    00> I: [DISPLAY] sleep start
    00> I: [DISPLAY] WAIT finished
    00> I: [DISPLAY] sleep done
    00> I: [MAIN] Chip is NEW - set LTE-M
    00> I: [MODEM] Modem Band Lock Set
    00> I: [MODEM] Version: 2.0.1
    00> I: [MODEM] Network registration status: Connected - roaming
    00> I: [MODEM] PSM parameter update: TAU: 3240, Active time: -1
    00> I: [MODEM] client_id: nrf-35940*
    00> I: [AWS] iot_connect - init done
    00> W: [AWS] iot_reconnect - wait 30 s connect try: 1/30
    00> W: [AWS] IOT_EVT_CONNECTED
    00> I: [MODEM] BOOT Image Confirmed
    00> I: [MAIN] DL INIT
    00> D: state = 1
    00> I: Downloading: http://ul1.epaperframe.de/NDUmX2z.dl [0]
    00> [MAIN] Downloading http://ul1.epaperframe.de/NDUmX2z.dl
    00> D: Port not specified, using default: 80
    00> D: Failed to resolve hostname ul1.epaperframe.de on IPv6
    00> D: family: 1, type: 1, proto: 6
    00> I: Connecting to 3.5.134.246
    00> D: fd 1, addrlen 8, fam IPv4, port 80
    00> D: state = 2
    00> D: Receiving up to 2048 bytes at 0x2000cbd8...
    00> D: Read 620 bytes from socket
    00> D: GET header size: 620
    00> E: Unexpected HTTP response: 301 moved permanently
    00> [DL] Error -77 during download
    00> D: state = 4
    00> D: state = 0
    00> [DL] CLOSED
    00> D: Connection closed
    00> [DL] Payload: "http/1.1 301 moved permanently"

    The links are save to stay public But they wont work anymore after some time

    The Download Client Sample worked fine btw. so thats not the issue.

  • exactly, this was my starting point. also downloading the sample works in my code. the config itself can indeed work. here is the full config I use:

    CONFIG_REBOOT=y
    CONFIG_CRC=y
    
    # GPIO
    CONFIG_GPIO=y
    CONFIG_PWM=y
    
    #RTT Log
    CONFIG_LOG=y
    CONFIG_LOG_MODE_MINIMAL=y
    CONFIG_UART_CONSOLE=n
    CONFIG_RTT_CONSOLE=y
    CONFIG_LOG_BACKEND_UART=n
    CONFIG_USE_SEGGER_RTT=y
    CONFIG_LOG_DEFAULT_LEVEL=3
    
    # Heap and stacks
    CONFIG_HEAP_MEM_POOL_SIZE=8096
    CONFIG_MAIN_STACK_SIZE=4096
    CONFIG_SYSTEM_WORKQUEUE_STACK_SIZE=2048
    
    
    #WDT
    CONFIG_WDT_LOG_LEVEL_DBG=y
    CONFIG_WATCHDOG=y
    CONFIG_WDT_DISABLE_AT_BOOT=n
    
    # SPI
    CONFIG_SPI=y
    
    # LTE link control
    CONFIG_LTE_LINK_CONTROL=y
    CONFIG_LTE_NETWORK_MODE_NBIOT=y
    CONFIG_MODEM_INFO=y
    CONFIG_MODEM_KEY_MGMT=y
    
    #Networking
    CONFIG_NET_SOCKETS=y
    CONFIG_NETWORKING=y
    CONFIG_NET_NATIVE=n #TODO check this one
    CONFIG_NRF_MODEM_LIB=y
    
    # Date Time library
    CONFIG_DATE_TIME=y
    CONFIG_DATE_TIME_MODEM=y
    
    # cJSON
    CONFIG_CJSON_LIB=y
    
    # Image manager
    CONFIG_FLASH_MAP=y
    CONFIG_STREAM_FLASH=y
    CONFIG_MCUBOOT_IMG_MANAGER=y
    CONFIG_IMG_MANAGER=y
    
    #OTA
    CONFIG_FLASH=y
    CONFIG_FLASH_PAGE_LAYOUT=y
    CONFIG_IMG_ERASE_PROGRESSIVELY=y
    CONFIG_BOOTLOADER_MCUBOOT=y
    
    # AWS FOTA
    CONFIG_AWS_FOTA=y
    CONFIG_FOTA_DOWNLOAD=y
    CONFIG_FOTA_SOCKET_RETRIES=200
    CONFIG_FOTA_DOWNLOAD_PROGRESS_EVT=y
    CONFIG_DFU_TARGET=y
    
    # Download Client
    CONFIG_DOWNLOAD_CLIENT=y
    CONFIG_DOWNLOAD_CLIENT_STACK_SIZE=4096
    CONFIG_DOWNLOAD_CLIENT_LOG_LEVEL_DBG=y
    
    #AWS IOT Settings
    CONFIG_AWS_IOT=y
    CONFIG_AWS_IOT_BROKER_HOST_NAME="XXX-ats.iot.eu-central-1.amazonaws.com"
    CONFIG_MQTT_HELPER_SEC_TAG=6
    CONFIG_AWS_IOT_TOPIC_UPDATE_DELTA_SUBSCRIBE=y
    CONFIG_AWS_IOT_AUTO_DEVICE_SHADOW_REQUEST=n
    CONFIG_MQTT_CLEAN_SESSION=y
    CONFIG_MQTT_KEEPALIVE=200
    
    # Non-volatile Storage
    CONFIG_NVS=y
    CONFIG_MPU_ALLOW_FLASH_WRITE=y
    CONFIG_PM_PARTITION_SIZE_NVS_STORAGE=0xE000
    CONFIG_LZ4=y

  • Hi,

    dejans said:
    Which NCS version do you use?

    Which NCS version and modem firmware version do you use?

    Best regards,
    Dejan

  • Hi,

    Unfortunately, there is no support for URL shorteners in download client library. Resolving short URL before calling download client library with expanded URL is currently the solution. In your case, you could consider doing an HTTP HEAD request using the Zephyr HTTP library, then read out the Location header with the expanded URL and use that subsequently with the download client. It is also recommended that you use HTTPS instead of HTTP due to security concern.

    Best regards,
    Dejan

  • This sounds like what I want. Is there any sample to read headers from a request?

Reply Children
  • Hi,

    There is an update to my previous reply. Support for HTTP transport is now added in the downloader library. Please have a look at this pull request which should also show how to read headers from request.

    Best regards,
    Dejan

  • So do you think waiting for this merged is the best option?

  • Hi,

    This is one option and decision is yours. It might be the easiest since there is support for HTTP transport in this new downloader library.

    Best regards,
    Dejan

  • I solved the Redirect now manually via socket. I read the complete http response and get my URL. If anybody else ever needs this here is my function:

    #include <zephyr/net/socket.h>
    
    char *redirectHttp(const char *endpoint, const char *path) {
        int fd = 0;
        int err = 0;
        int bytes;
        char send_packet[200];
        char recv_buf[1024] = {};
        struct addrinfo *res;
        struct addrinfo hints = {
            .ai_family = AF_INET,
            .ai_socktype = SOCK_STREAM,
        };
    
        err = getaddrinfo(endpoint, "80", &hints, &res);
        if (err) {
            LOG_ERR("[SOCKET] getaddrinfo err %d", errno);
            return 0;
        }
        ((struct sockaddr_in *)res->ai_addr)->sin_port = htons(80);
    
        fd = socket(AF_INET, SOCK_STREAM, IPPROTO_TCP);  // Set socket to tcp without tls
        if (fd == -1) {
            LOG_ERR("[SOCKET] failed to open");
            return 0;
        }
        char ipv4_addr[NET_IPV4_ADDR_LEN];
        inet_ntop(AF_INET, &((struct sockaddr_in *)res->ai_addr)->sin_port, ipv4_addr, sizeof(ipv4_addr));
        LOG_INF("[SOCKET] Connect to Addr IP: %s", ipv4_addr);
        err = connect(fd, res->ai_addr, sizeof(struct sockaddr_in));
        if (err) {
            LOG_ERR("[SOCKET] connect() err: %d", errno);
            return 0;
        }
    
        sprintf(send_packet,
                "GET %s HTTP/1.1\r\n"
                "Host: %s\r\n"
                "Accept: txt/plain\r\n\r\n",
                path, endpoint);
        bytes = send(fd, &send_packet, strlen(send_packet), 0);
        LOG_INF("[SOCKET] Sent %d bytes", bytes);
        bytes = blocking_recv(fd, recv_buf, sizeof(recv_buf), 0);
        LOG_INF("[SOCKET] Received %d bytes", bytes);
    
        const char *location_header = "Location: ";
        const char *start = strstr(recv_buf, location_header);
        if (start) {
            start += strlen(location_header);
    
            const char *end = strstr(start, "\r\n");
            if (!end) {
                end = start + strlen(start);  // End of string if no CRLF
            }
    
            size_t length = end - start;
    
            if (length < 1024) {
                char *url = (char *)malloc(length + 1);  // Allocate memory for the URL
                strncpy(url, start, length);
                url[length] = '\0';  // Null-terminate the string
                return url;
            } else {
                LOG_ERR("[SOCKET] URL is too long to fit in the buffer");
                return 0;
            }
        } else {
            LOG_ERR("[SOCKET] Location header not found in the response");
        }
    
        return 0;
    }

    Example Usage:

    const char *endpoint = "ul1.epaperframe.de";
    const char *path = "/tjoLta9.dl";
    char *url = redirectHttp(endpoint, path);
    printf("Redirected URL: %s\n", url);
    free(url);  // Free the allocated memory


    This ticket is solved for me now.

  • Hi,

    danielboe said:
    I solved the Redirect now manually via socket.

    This is great to hear. Thank you for the update.

    Best regards,
    Dejan

Related