Facing issues in creating multipart http request for uploading files

Hello,
I was trying to execute multipart request for uploading log files. For testing the multipart that how does it works first I made a server through python script which I linked with ngrok so that other people can also upload files. Than I tried sending the request through postman and it was executing perfectly. The below request is what I sent through postman.

POST /upload HTTP/1.1
Host: 148a-106-51-226-188.ngrok-free.app
Content-Length: 203
Content-Type: multipart/form-data; boundary=----WebKitFormBoundary7MA4YWxkTrZu0gW

------WebKitFormBoundary7MA4YWxkTrZu0gW
Content-Disposition: form-data; name="file"; filename="01.00.01.52.bin"
Content-Type: <Content-Type header here>

(data)
------WebKitFormBoundary7MA4YWxkTrZu0gW--


After this I wanted to do the same thing via firmware, I was able to retrieve the ip address of the ngrok server and was able to connect with it too but while sending the request I receive:

[00:01:36.048,339] <err> home_demo_https: Failed to send HTTP multipart headers, err: -104
[00:01:36.048,706] <err> home_demo_sdcard: Failed to send request

this error. We are already saving binary files in sd card in the application so I was trying to use that binary file for upload
I am attaching the function too for reference


static void logsUpload_cb(struct http_response *rsp, enum http_final_call final_data, void *user_data)
{
	//Define the callback function to print the body
	LOG_INF("Response status: %s", rsp->http_status);

	if (rsp->body_frag_len > 0) {
			char body_buf[rsp->body_frag_len];
			strncpy(body_buf, rsp->body_frag_start, rsp->body_frag_len);
			body_buf[rsp->body_frag_len]='\0';
			LOG_INF("Received: %s", body_buf);
	} 
}

int logsUpload_req(char *filename)
{
	int err = 0;	
	
	///Define the structure http_request and fill the block of memory 
	struct http_request req;
	memset(&req, 0, sizeof(req));

	char multipart_header[256];
	snprintf(multipart_header, sizeof(multipart_header),
    "--%s\r\n"
    "Content-Disposition: form-data; name=\"file\"; filename=\"%s\"\r\n"
    "Content-Type: application/octet-stream\r\n",
    BOUNDARY, filename);

	char multipart_footer[64];
	snprintf(multipart_footer, sizeof(multipart_footer),
		"\r\n--%s--\r\n",
		BOUNDARY);
	
	size_t content_length = strlen(multipart_header) + bytes_read + strlen(multipart_footer);

	char content_length_header[64];
	snprintf(content_length_header, sizeof(content_length_header),
         "Content-Length: %zu\r\n", content_length);
	//Populate the http_request structure 
	// Include Content-Length in headers
	const char *headers[] = {
        "Connection: close\r\n",
        "Host: 148a-106-51-226-188.ngrok-free.app\r\n",
        "Content-Type: multipart/form-data; boundary=" BOUNDARY "\r\n",
		content_length_header,
        NULL
    };
	req.header_fields = headers;
	req.method = HTTP_POST;
	req.url = "/upload";
	req.host = LOG_HOSTNAME;
	req.protocol = "HTTP/1.1";
	req.payload = multipart_header;
	req.payload_len = strlen(multipart_header);
	req.response = logsUpload_cb;
	req.recv_buf = recv_buf;
	req.recv_buf_len = sizeof(recv_buf);
	LOG_INF("Sending multipart headers...");
    err = http_client_req(sock, &req, 7000, NULL);
    if (err < 0) {
        LOG_ERR("Failed to send HTTP multipart headers, err: %d", err);
        return err;
    }

	req.payload = filename;  // Let the HTTP client handle streaming
    req.payload_len = bytes_read;

    LOG_INF("Sending file data...");
    err = http_client_req(sock, &req, 7000, NULL);
    if (err < 0) {
        LOG_ERR("Failed to send file data, err: %d", err);
        return err;
    }

	req.payload = multipart_footer;
    req.payload_len = strlen(multipart_footer);

    LOG_INF("Sending multipart footer...");
    err = http_client_req(sock, &req, 7000, NULL);
    if (err < 0) {
        LOG_ERR("Failed to send multipart footer, err: %d", err);
        return err;
    }

    LOG_INF("File uploaded successfully!");

	//Send the request to the HTTP server
	// LOG_INF("HTTP POST request: %s",req.payload);
		
	return err;
}

  • I tried removing the multipart_header and multipart_footer and just kept a simple header with the filename being in the payload

    int logsUpload_req(char *filename)
    {
    	int err = 0;	
    	
    	///Define the structure http_request and fill the block of memory 
    	struct http_request req;
    	memset(&req, 0, sizeof(req));
    	const char *headers[] = {
            "Connection: keep-alive\r\n",
            "Host: 148a-106-51-226-188.ngrok-free.app\r\n",
            "Content-Type: form-data\r\n",
            NULL
        };
    	req.header_fields = headers;
    	req.method = HTTP_POST;
    	req.url = "/upload";
    	req.host = LOG_HOSTNAME;
    	req.protocol = "HTTP/1.1";
    	req.payload = filename;
    	req.payload_len = bytes_read;
    	req.response = logsUpload_cb;
    	req.recv_buf = recv_buf;
    	req.recv_buf_len = sizeof(recv_buf);
        err = http_client_req(sock, &req, 7000, NULL);
        if (err < 0) {
            LOG_ERR("Failed to send upload file HTTP req, err: %d", err);
            return err;
        }		
    	return err;
    }

    Also after enabling NET_LOG and NET_HTTP_LOG I came to know that I was getting data allocation failed, so I increased the CONFIG_NET_BUF_DATA_SIZE=1024 from 256

    I am getting 
    [00:01:46.232,757] <dbg> net_http_client: http_flush_data: Data to send
    50 4f 53 54 20 2f 75 70 6c 6f 61 64 20 48 54 54 |POST /up load HTT
    50 2f 31 2e 31 0d 0a 48 6f 73 74 3a 20 31 34 38 |P/1.1..H ost: 148
    61 2d 31 30 36 2d 35 31 2d 32 32 36 2d 31 38 38 |a-106-51 -226-188
    2e 6e 67 72 6f 6b 2d 66 72 65 65 2e 61 70 70 0d |.ngrok-f ree.app.
    0a 43 6f 6e 6e 65 63 74 69 6f 6e 3a 20 6b 65 65 |.Connect ion: kee
    70 2d 61 6c 69 76 65 0d 0a 48 6f 73 74 3a 20 31 |p-alive. .Host: 1
    34 38 61 2d 31 30 36 2d 35 31 2d 32 32 36 2d 31 |48a-106- 51-226-1
    38 38 2e 6e 67 72 6f 6b 2d 66 72 65 65 2e 61 70 |88.ngrok -free.ap
    70 0d 0a 43 6f 6e 74 65 6e 74 2d 54 79 70 65 3a |p..Conte nt-Type:
    20 66 6f 72 6d 2d 64 61 74 61 0d 0a 43 6f 6e 74 | form-da ta..Cont
    65 6e 74 2d 4c 65 6e 67 74 68 3a 20 32 32 32 30 |ent-Leng th: 2220
    0d 0a 0d 0a |....
    [00:01:46.242,095] <dbg> net_http_client: http_client_req: (0x20004090): Sent 2400 bytes
    [00:01:46.286,285] <dbg> net_http_client: on_message_begin: (0x20004090): -- HTTP POST response (headers) --
    [00:01:46.286,773] <dbg> net_http_client: on_status: (0x20004090): HTTP response status 400 Bad Request
    [00:01:46.289,093] <dbg> net_http_client: print_header_field: (0x20004090): [14] Content-Length
    [00:01:46.290,954] <dbg> net_http_client: print_header_field: (0x20004090): [2] 11
    [00:01:46.291,961] <dbg> net_http_client: print_header_field: (0x20004090): [10] Connection
    [00:01:46.292,877] <dbg> net_http_client: print_header_field: (0x20004090): [5] close
    [00:01:46.293,762] <dbg> net_http_client: print_header_field: (0x20004090): [12] Content-Type
    [00:01:46.294,647] <dbg> net_http_client: print_header_field: (0x20004090): [10] text/plain
    [00:01:46.295,074] <dbg> net_http_client: on_headers_complete: (0x20004090): Headers complete
    [00:01:46.317,626] <dbg> net_http_client: http_wait_data: Connection error (0)
    [00:01:46.317,993] <dbg> net_http_client: http_client_req: (0x20004090): Received 0 bytes

    this currently
    I observed that it doesn't adds the BOUNDARY in the request automatically
    So does it needs to be done manually

  • Hi,

     

    Rakshita said:
    Also after enabling NET_LOG and NET_HTTP_LOG I came to know that I was getting data allocation failed, so I increased the CONFIG_NET_BUF_DATA_SIZE=1024 from 256

    Good to hear that you have gotten the buffering issue resolved.

    Rakshita said:
    I observed that it doesn't adds the BOUNDARY in the request automatically
    So does it needs to be done manually

    That looks to be a functionality that the http_client_req() does not add for you. Have you tried adding this to your req.headers?

     

    Kind regards,

    Håkon

  • Yes I tried adding it but it throws -104 error 
    I also came across https://hacod.tech/blog/sending-files-to-aws-s3-bucket-using-zephyr-rtos-http-api
    which doesn't mention any particular content-type but when I added this in my code I was getting 

    [00:01:36.222,381] <inf> home_demo_sdcard: Wrote 2220 bytes to file /SD:/home2.bin
    
    [00:01:36.229,705] <inf> home_demo_sdcard: 
    Listing dir /SD: ...
    
    [00:01:36.230,102] <inf> home_demo_sdcard: Full filename: /SD:/home2.bin
    [00:01:36.231,781] <inf> home_demo_sdcard: File size: 2220 bytes
    
    [00:01:36.232,116] <inf> home_demo_sdcard: File check 2 2
    [00:01:36.232,513] <dbg> net_http_client: http_flush_data: Data to send
                                              50 55 54 20 2f 75 70 6c  6f 61 64 20 48 54 54 50 |PUT /upl oad HTTP
                                              2f 31 2e 31 0d 0a 48 6f  73 74 3a 20 65 66 63 66 |/1.1..Ho st: efcf
                                              2d 31 30 36 2d 35 31 2d  36 38 2d 31 38 37 2e 6e |-106-51- 68-187.n
                                              67 72 6f 6b 2d 66 72 65  65 2e 61 70 70 0d 0a 43 |grok-fre e.app..C
                                              6f 6e 74 65 6e 74 2d 4c  65 6e 67 74 68 3a 20 7a |ontent-L ength: z
                                              75 0d 0a 0d 0a                                   |u....            
    [00:01:36.240,203] <inf> home_demo_https: Sent 1024 bytes
    [00:01:36.245,697] <inf> home_demo_https: Sent 1024 bytes
    [00:01:36.319,030] <err> home_demo_https: Send error, abort
    [00:01:36.319,335] <err> home_demo_https: Failed to send upload file HTTP req, err: -1
    [00:01:36.623,229] <inf> home_demo_https: CSemCount 0
    [00:01:38.745,208] <inf> home_demo_uart: Sensor index 0
    [00:01:41.319,732] <err> home_demo_sdcard: Failed to send request


    the attached error

  • Hi,

     

    I see that it returns -1 here:

    [00:01:36.319,030] <err> home_demo_https: Send error, abort
    [00:01:36.319,335] <err> home_demo_https: Failed to send upload file HTTP req, err: -1
    ..

    Is this still due to your buffers being filled up?

      

    Can you try to adjust your net buffers?

    By setting CONFIG_NET_BUF_VARIABLE_DATA_SIZE=y, you can set a combined buffer via:

    CONFIG_NET_BUF_DATA_POOL_SIZE=8192

     

    Or with never SDKs, where its split into RX and TX:

    CONFIG_NET_PKT_BUF_RX_DATA_POOL_SIZE=4096

    CONFIG_NET_PKT_BUF_TX_DATA_POOL_SIZE=6144 

     

    (feel free to change the sizes)

     

    Kind regards,

    Håkon

Related