This post is older than 2 years and might not be relevant anymore
More Info: Consider searching for newer posts

AT#XHTTPCREQ serial_lte_modem does not handle chunked HTTP response

if a response to the http request is chunked (https://en.wikipedia.org/wiki/Chunked_transfer_encoding), it is not handled correctly by the serial_lte_modem.

Sometimes - e.g. if cloudflare is used, it is not possible to disable chunk encoding, so the serial_lte_modem app is unusable in these scenarios.

00> [00:03:22.707,031] <inf> httpc: Partial data received (743 bytes)
00> [00:03:22.707,061] <inf> httpc: All the data received (743 bytes)
00> [00:03:22.707,489] <inf> at_host: TX
00> 23 58 48 54 54 50 43 52  53 50 3a 37 34 33 2c 30 |#XHTTPCR SP:743,0
00> 0d 0a                                            |..               
00> [00:03:22.709,625] <inf> at_host: TX
00> 22 7d 5d 2c 22 67 72 6f  75 70 22 3a 22 63 66 2d |"}],"gro up":"cf-
00> 6e 65 6c 22 2c 22 6d 61  78 5f 61 67 65 22 3a 36 |nel","ma x_age":6
00> 30 34 38 30 30 7d 0d 0a  4e 45 4c 3a 20 7b 22 72 |04800}.. NEL: {"r
00> 65 70 6f 72 74 5f 74 6f  22 3a 22 63 66 2d 6e 65 |eport_to ":"cf-ne
00> 6c 22 2c 22 6d 61 78 5f  61 67 65 22 3a 36 30 34 |l","max_ age":604
00> 38 30 30 7d 0d 0a 53 65  72 76 65 72 3a 20 63 6c |800}..Se rver: cl
00> 6f 75 64 66 6c 61 72 65  0d 0a 43 46 2d 52 41 59 |oudflare ..CF-RAY
00> 3a 20 35 66 31 62 30 35  63 34 39 66 63 35 64 34 |: 5f1b05 c49fc5d4
00> 33 66 2d 48 41 4d 0d 0a  0d 0a 44 65 76 69 63 65 |3f-HAM.. ..Device
00> 20 31 38 32 30 65 66 66  64 20 77 61 73 20 6e 6f | 1820eff d was no
00> 74 20 66 6f 75 6e 64 64  34 39 37 36 63 30 32 39 |t foundd 4976c029
00> 37 37 37 30 38 64 39 66  62 35 36 33 31 30 34 65 |77708d9f b563104e
00> 36 62 30 61 31 31 31 34  31 36 30 35 32 39 37 30 |6b0a1114 16052970
00> 33 34 3b 20 65 78 70 69  72 65 73 3d 53 75 6e 2c |34; expi res=Sun,
00> 20 31 33 2d 44 65 63 2d  32 30 20 31 39 3a 35 30 | 13-Dec- 20 19:50
00> 3a 33 34 20 47 4d 54 3b  20 70 61 74 68 3d 2f 3b |:34 GMT;  path=/;
00> 20 64 6f 6d 61 69 6e 3d  2e 67 77 78 63 6c 6f 75 | domain= .gwxclou
00> 64 2e 63 6f 6d 3b 20 48  74 74 70 4f 6e 6c 79 3b |d.com; H ttpOnly;
00> 20 53 61 6d 65 53 69 74  65 3d 4c 61 78 0d 0a 43 | SameSit e=Lax..C
00> 46 2d 43 61 63 68 65 2d  53 74 61 74 75 73 3a 20 |F-Cache- Status: 
00> 44 59 4e 41 4d 49 43 0d  0a 63 66 2d 72 65 71 75 |DYNAMIC. .cf-requ
00> 65 73 74 2d 69 64 3a 20  30 36 36 34 63 31 65 65 |est-id:  0664c1ee
00> 64 63 30 30 30 30 64 34  33 66 36 38 38 37 30 30 |dc0000d4 3f688700
00> 30 30 30 30 30 30 30 31  0d 0a 52 65 70 6f 72 74 |00000001 ..Report
00> 2d 54 6f 3a 20 7b 22 65  6e 64 70 6f 69 6e 74 73 |-To: {"e ndpoints
00> 22 3a 5b 7b 22 75 72 6c  22 3a 22 68 74 74 70 73 |":[{"url ":"https
00> 3a 5c 2f 5c 2f 61 2e 6e  65 6c 2e 63 6c 6f 75 64 |:\/\/a.n el.cloud
00> 66 6c 61 72 65 2e 63 6f  6d 5c 2f 72 65 70 6f 72 |flare.co m\/repor
00> 74 3f 73 3d 4e 65 75 57  66 46 50 6b 75 47 25 32 |t?s=NeuW fFPkuG%2
00> 46 4e 61 61 4f 4f 43 65  69 70 41 51 4d 59 75 53 |FNaaOOCe ipAQMYuS
00> 75 61 69 39 36 25 32 46  7a 4e 6c 61 49 6d 50 49 |uai96%2F zNlaImPI
00> 74 72 52 44 67 7a 45 4f  55 61 4a 41 6e 78 66 51 |trRDgzEO UaJAnxfQ
00> 75 47 4f 74 34 75 34 73  52 48 46 5a 65 74 49 67 |uGOt4u4s RHFZetIg
00> 34 34 7a 34 74 6c 54 53  45 32 57 62 49 70 42 38 |44z4tlTS E2WbIpB8
00> 46 58 43 36 67 71 57 52  5a 4c 43 38 4f 69 41 41 |FXC6gqWR ZLC8OiAA
00> 44 43 7a 70 59 77 58 49  34 77 25 33 44 25 33 44 |DCzpYwXI 4w%3D%3D
00> 00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00 |........ ........
00> 00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00 |........ ........
00> 00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00 |........ ........
00> 00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00 |........ ........
00> 00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00 |........ ........
00> 00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00 |........ ........
00> 00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00 |........ ........
00> 00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00 |........ ........
00> 00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00 |........ ........
00> 00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00 |........ ........
00> 00 00 00 00 00 00 00                             |.......      

Parents
  • Hi, Michal!

    Thank you for reporting this. I will bring the info to the developers. 

    Best regards,
    Carl Richard

  • Hi Carl,

    do you have any info or ETA when this might be fixed? It is a big problem for us, basically making the serial_lte_modem unsusable in production and therefore delaying production of nrf91 based devices.

    Best regards, Michal V.

  • OK, so I've tried that, due to various limitations (such as domain whitelisting on our SIM, DNS config etc..) I've only been able to test it with gwxcloud.com, that is not set up and returns an error. But this error goes through cloudflare, so it is returned with chunked encoding.

    "Log" from LTE link monitor:

    AT#XHTTPCCON=1,"gwxcloud.com",80#XHTTPCCON:1OKAT#XHTTPCREQ="GET","/",""OK#XHTTPCREQ:0#XHTTPCRSP:3489,0</div></body></html>0f="https://docs.microsoft.com/en-us/azure/app-service-web/app-service-web-tutorial-custom-domain">how to map an existing domain</a> to resolve this.</li>                <li>Client cache is still pointing the domain to old IP address. Clear the cache by running the command <i>ipconfig/flushdns.</i></li>            </ul>            <p>Checkout <a href="https://blogs.msdn.microsoft.com/appserviceteam/2017/08/08/faq-app-service-domain-preview-and-custom-domains/">App Service Domain FAQ</a> for more questions.</p>        </div>     5b8): Calling callback for partitioned 33 len data[0m[00:14:11.286,865] [0m<dbg> net_http.on_message_complete: (0x200215b8): -- HTTP GET response (complete) --[0m,723] [0m<dbg> net_http.on_body: (0x200215b8): Calling callback for partitioned 180 len data[0m[00:14:11.017,303] [0m<dbg> net_http.on_body: (0x200215b8): Processed 2128 length 528[0m[00:14:11.017,700] [0m<dbg> net_http.on_body: (0x200215b8): Calling callback for partitioned 528 len data[0m[00:14:11.284,210] [0m<dbg> net_http.on_body: (0x200215b8): Processed 2176 length 48[0m[00:14:11.284,606] [0m<dbg> net_http.on_body: (0x200215b8): Calling callback for partitioned 48 len data[0m[00:14:11.285,125] [0m<dbg> net_http.on_body: (0x200215b8): Processed 2752 length 576[0m[00:14:11.285,491] [0m<dbg> net_http.on_body: (0x200215b8): Calling callback for partitioned 576 len data[0m[00:14:11.286,010] [0m<dbg> net_http.on_body: (0x200215b8): Processed 2778 length 26[0m[00:14:11.286,407] [0m<dbg> net_http.on_body: (0x20021901288004468357KKK

    Log collected over J-Link:

    00> [00:13:56.049,255] <inf> at_cmd: Enqueueing response for sync call
    00> [00:13:56.049,621] <inf> at_cmd: Writing any pending command
    00> [00:13:56.049,896] <inf> at_cmd: Listening on socket
    00> [00:14:05.527,923] <inf> httpc: Attempting to connect over IPv6
    00> [00:14:05.528,625] <err> httpc: Unable to connect, errno 114
    00> [00:14:05.529,449] <inf> httpc: Attempting to connect over IPv4
    00> [00:14:06.104,309] <inf> httpc: Connected to gwxcloud.com
    00> [00:14:09.014,923] <dbg> net_http.Data to send
    00> 47 45 54 20 2f 20 48 54  54 50 2f 31 2e 31 0d 0a |GET / HT TP/1.1..
    00> 48 6f 73 74 3a 20 67 77  78 63 6c 6f 75 64 2e 63 |Host: gw xcloud.c
    00> 6f 6d 0d 0a                                      |om..             
    00> [00:14:09.016,693] <dbg> net_http.Data to send
    00> 0d 0a                                            |..               
    00> [00:14:09.017,578] <dbg> net_http.http_client_req: (0x200215b8): Sent 54 bytes
    00> [00:14:10.118,286] <dbg> net_http.on_message_begin: (0x200215b8): -- HTTP GET response (headers) --
    00> [00:14:10.118,743] <dbg> net_http.on_status: (0x200215b8): HTTP response status 404 Site Not Found
    00> [00:14:10.119,415] <dbg> net_http.print_header_field: (0x200215b8): [4] Date
    00> [00:14:10.120,056] <dbg> net_http.print_header_field: (0x200215b8): [29] Thu, 26 Nov 2020 10:39:44 GMT
    00> [00:14:10.120,758] <dbg> net_http.print_header_field: (0x200215b8): [12] Content-Type
    00> [00:14:10.121,398] <dbg> net_http.print_header_field: (0x200215b8): [9] text/html
    00> [00:14:10.122,039] <dbg> net_http.print_header_field: (0x200215b8): [17] Transfer-Encoding
    00> [00:14:10.122,680] <dbg> net_http.print_header_field: (0x200215b8): [7] chunked
    00> [00:14:10.123,291] <dbg> net_http.print_header_field: (0x200215b8): [10] Connection
    00> [00:14:10.123,931] <dbg> net_http.print_header_field: (0x200215b8): [10] keep-alive
    00> [00:14:10.124,511] <dbg> net_http.print_header_field: (0x200215b8): [10] Set-Cookie
    00> [00:14:10.125,183] <dbg> net_http.print_header_field: (0x200215b8): [127] __cfduid=d4df0e26ea5b2e8ad1dadbe95da3bc9aa1606387183; expires=Sat, 26-Dec-20 10:39:43 GMT; path=/; domain=.gwxcloud.com; HttpOn
    00> [00:14:10.126,068] <dbg> net_http.print_header_field: (0x200215b8): [15] CF-Cache-Status
    00> [00:14:10.131,835] <dbg> net_http.print_header_field: (0x200215b8): [7] DYNAMIC
    00> [00:14:10.132,324] <dbg> net_http.print_header_field: (0x200215b8): [13] cf-request-id
    00> [00:14:10.132,873] <dbg> net_http.print_header_field: (0x200215b8): [32] 06a5bc48ce0000fa18d4292000000001
    00> [00:14:10.133,392] <dbg> net_http.print_header_field: (0x200215b8): [9] Report-To
    00> [00:14:10.133,972] <dbg> net_http.print_header_field: (0x200215b8): [127] {"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report?s=WxfmvBeE3cOvi%2BNieDfBJUAiKLX1LXjjJy3Gv%2FAt4nx8hR2W5ko69d0fKvmC
    00> [00:14:10.135,070] <dbg> net_http.print_header_field: (0x200215b8): [23] -nel","max_age":604800}
    00> [00:14:10.135,772] <dbg> net_http.print_header_field: (0x200215b8): [3] NEL
    00> [00:14:10.136,413] <dbg> net_http.print_header_field: (0x200215b8): [39] {"report_to":"cf-nel","max_age":604800}
    00> [00:14:10.137,145] <dbg> net_http.print_header_field: (0x200215b8): [6] Server
    00> [00:14:10.137,786] <dbg> net_http.print_header_field: (0x200215b8): [10] cloudflare
    00> [00:14:10.138,397] <dbg> net_http.print_header_field: (0x200215b8): [6] CF-RAY
    00> [00:14:10.139,007] <dbg> net_http.print_header_field: (0x200215b8): [20] 5f82fcbae8d9fa18-AMS
    00> [00:14:10.139,404] <dbg> net_http.on_headers_complete: (0x200215b8): Headers complete
    00> [00:14:10.145,050] <dbg> net_http.on_body: (0x200215b8): Processed 4 length 4
    00> [00:14:10.145,385] <dbg> net_http.on_body: (0x200215b8): Calling callback for partitioned 708 len data
    00> [00:14:10.413,330] <dbg> net_http.on_body: (0x200215b8): Processed 448 length 444
    00> [00:14:10.413,757] <dbg> net_http.on_body: (0x200215b8): Calling callback for partitioned 444 len data
    00> [00:14:10.414,276] <dbg> net_http.on_body: (0x200215b8): Processed 712 length 264
    00> [00:14:10.414,642] <dbg> net_http.on_body: (0x200215b8): Calling callback for partitioned 264 len data
    00> [00:14:10.712,280] <dbg> net_http.on_body: (0x200215b8): Processed 1024 length 312
    00> [00:14:10.712,677] <dbg> net_http.on_body: (0x200215b8): Calling callback for partitioned 312 len data
    00> [00:14:10.713,226] <dbg> net_http.on_body: (0x200215b8): Processed 1420 length 396
    00> [00:14:10.713,623] <dbg> net_http.on_body: (0x200215b8): Calling callback for partitioned 396 len data
    00> [00:14:11.016,357] <dbg> net_http.on_body: (0x200215b8): Processed 1600 length 180
    00> [00:14:11.016,723] <dbg> net_http.on_body: (0x200215b8): Calling callback for partitioned 180 len data
    00> [00:14:11.017,303] <dbg> net_http.on_body: (0x200215b8): Processed 2128 length 528
    00> [00:14:11.017,700] <dbg> net_http.on_body: (0x200215b8): Calling callback for partitioned 528 len data
    00> [00:14:11.284,210] <dbg> net_http.on_body: (0x200215b8): Processed 2176 length 48
    00> [00:14:11.284,606] <dbg> net_http.on_body: (0x200215b8): Calling callback for partitioned 48 len data
    00> [00:14:11.285,125] <dbg> net_http.on_body: (0x200215b8): Processed 2752 length 576
    00> [00:14:11.285,491] <dbg> net_http.on_body: (0x200215b8): Calling callback for partitioned 576 len data
    00> [00:14:11.286,010] <dbg> net_http.on_body: (0x200215b8): Processed 2778 length 26
    00> [00:14:11.286,407] <dbg> net_http.on_body: (0x200215b8): Calling callback for partitioned 33 len data
    00> [00:14:11.286,865] <dbg> net_http.on_message_complete: (0x200215b8): -- HTTP GET response (complete) --
    00> [00:14:11.289,337] <dbg> net_http.http_client_req: (0x200215b8): Received 3489 bytes

  • Thank you, Michal! I'll forward this.

    From the logs it seems like the logging suddenly begins to write to the LTE Link Monitor (rather than serial). Can you confirm that this it what's happening? 

    Best regards,
    Carl Richard

  • Thank you for a fast response.

    Yes, but this has only happened to me when running the app with the debug logging enabled. Without it this "side-issue" was not present. But you can still see from th LTE Link Monitor "log" that the http response is broken, as it begins with `0</div></body></html>`.

    BR, Michal V.

  • I understand. The developer suspects that this has to do with issues with the thread memory or some buffer overflowing, but I saw that you have pointed to a specific line in the slm_at_host.c, so I'm looking into what trouble that may cause.

    Have a good weekend!

    Best regards,
    Carl Richard

  • Thank you for the info. The specific line in slm_at_host.c (https://devzone.nordicsemi.com/f/nordic-q-a/68346/problems-passing-payload-to-at-xhttpcreq-in-serial_lte_modem) has to do with passing payload to the request, it is unrelated to this issue.

    Wish you a nice weekend as well, BR Michal V.

Reply Children
  • I see! My misunderstanding.

    The developer has purposed a fix in this pull request. Could you test it by replacing the attached and updated slm_at_httpc.c in your project?
    From his tests it should solve the problems with the cloudflair cloud.

    /*
     * Copyright (c) 2020 Nordic Semiconductor ASA
     *
     * SPDX-License-Identifier: LicenseRef-BSD-5-Clause-Nordic
     */
    
    #include <logging/log.h>
    #include <zephyr.h>
    #include <stdio.h>
    #include <net/socket.h>
    #include <net/tls_credentials.h>
    #include <net/http_client.h>
    #include <nrf_socket.h>
    #include "slm_at_httpc.h"
    #include "slm_util.h"
    
    LOG_MODULE_REGISTER(httpc, CONFIG_SLM_LOG_LEVEL);
    
    #define HTTPC_HOST_LEN		64
    #define HTTPC_METHOD_LEN	20
    #define HTTPC_RES_LEN		256
    #define HTTPC_HEADER_LEN	512
    #define HTTPC_REQ_LEN		(HTTPC_METHOD_LEN + HTTPC_RES_LEN \
    				+ HTTPC_HEADER_LEN + 3)
    #define HTTPC_FRAG_SIZE		NET_IPV4_MTU
    #define HTTPC_BUF_LEN		1024
    #if HTTPC_REQ_LEN > HTTPC_BUF_LEN
    # error "Please specify larger HTTPC_BUF_LEN"
    #endif
    #define HTTPC_REQ_TO_S		10
    
    /* Buffers for HTTP client. */
    static uint8_t data_buf[HTTPC_BUF_LEN];
    
    /**@brief List of supported AT commands. */
    enum slm_httpc_at_cmd_type {
    	AT_HTTPC_CONNECT,
    	AT_HTTPC_REQUEST,
    	AT_HTTPC_MAX
    };
    
    /**@brief HTTP connect operations. */
    enum slm_httpccon_operation {
    	AT_HTTPCCON_DISCONNECT,
    	AT_HTTPCCON_CONNECT
    };
    
    /** forward declaration of cmd handlers **/
    static int handle_AT_HTTPC_CONNECT(enum at_cmd_type cmd_type);
    static int handle_AT_HTTPC_REQUEST(enum at_cmd_type cmd_type);
    
    /**@brief SLM AT Command list type. */
    static slm_at_cmd_list_t http_at_list[AT_HTTPC_MAX] = {
    	{AT_HTTPC_CONNECT, "AT#XHTTPCCON", handle_AT_HTTPC_CONNECT},
    	{AT_HTTPC_REQUEST, "AT#XHTTPCREQ", handle_AT_HTTPC_REQUEST},
    };
    
    static struct slm_httpc_ctx {
    	int fd;				/* HTTPC socket */
    	bool sec_transport;		/* secure session flag */
    	uint32_t sec_tag;		/* security tag to be used */
    	char host[HTTPC_HOST_LEN + 1];	/* HTTP server address */
    	uint32_t port;			/* HTTP server port */
    	char *method_str;		/* request method */
    	char *resource;			/* resource */
    	char *headers;			/* headers */
    	char *payload;			/* payload chunk */
    	size_t pl_len;			/* payload length */
    	size_t pl_to_send;		/* payload to send to server */
    	ssize_t pl_sent;		/* payload sent to server */
    	bool rsp_completed;		/* inticator of completed response */
    } httpc;
    
    /* global functions defined in different resources */
    void rsp_send(const uint8_t *str, size_t len);
    
    /* global variable defined in different resources */
    extern struct at_param_list at_param_list;
    extern char rsp_buf[CONFIG_AT_CMD_RESPONSE_MAX_LEN];
    
    #define THREAD_STACK_SIZE       KB(2)
    #define THREAD_PRIORITY         K_LOWEST_APPLICATION_THREAD_PRIO
    static K_THREAD_STACK_DEFINE(httpc_thread_stack, THREAD_STACK_SIZE);
    
    static K_SEM_DEFINE(http_req_sem, 0, 1);
    static K_SEM_DEFINE(http_data_sem, 0, 1);
    
    static int socket_sectag_set(int fd, int sec_tag)
    {
    	int err;
    	int verify;
    	sec_tag_t sec_tag_list[] = { sec_tag };
    
    	verify = TLS_PEER_VERIFY_REQUIRED;
    
    	err = setsockopt(fd, SOL_TLS, TLS_PEER_VERIFY, &verify, sizeof(verify));
    	if (err) {
    		LOG_ERR("Failed to setup peer verification, errno %d", errno);
    		return -1;
    	}
    
    	err = setsockopt(fd, SOL_TLS, TLS_SEC_TAG_LIST, sec_tag_list,
    			 sizeof(sec_tag_t) * ARRAY_SIZE(sec_tag_list));
    	if (err) {
    		LOG_ERR("Failed to set socket security tag, errno %d", errno);
    		return -1;
    	}
    
    	err = setsockopt(fd, SOL_TLS, TLS_HOSTNAME,
    			 httpc.host, sizeof(httpc.host));
    	if (err < 0) {
    		LOG_ERR("Failed to set TLS_HOSTNAME option: %d", errno);
    		return -errno;
    	}
    
    	return 0;
    }
    
    static int resolve_and_connect(int family, const char *host, int sec_tag)
    {
    	int fd;
    	int err;
    	int proto;
    	uint16_t port;
    	struct addrinfo *addr;
    	struct addrinfo *info;
    
    	__ASSERT_NO_MSG(host);
    
    	/* Set up port and protocol */
    	if (httpc.sec_transport == false) {
    		/* HTTP, port 80 */
    		proto = IPPROTO_TCP;
    	} else {
    		/* HTTPS, port 443 */
    		proto = IPPROTO_TLS_1_2;
    	}
    
    	port = htons(httpc.port);
    
    	/* Lookup host */
    	struct addrinfo hints = {
    		.ai_family = family,
    		.ai_socktype = SOCK_STREAM,
    		.ai_protocol = proto,
    	};
    
    	err = getaddrinfo(host, NULL, &hints, &info);
    	if (err) {
    		LOG_ERR("Failed to resolve hostname %s on %s",
    			log_strdup(host),
    			family == AF_INET ? "IPv4" : "IPv6");
    		return -1;
    	}
    	LOG_INF("Attempting to connect over %s",
    		family == AF_INET ? log_strdup("IPv4") : log_strdup("IPv6"));
    	fd = socket(family, SOCK_STREAM, proto);
    	if (fd < 0) {
    		LOG_ERR("Failed to create socket, errno %d", errno);
    		goto cleanup;
    	}
    
    	if (proto == IPPROTO_TLS_1_2) {
    		LOG_INF("Setting up TLS credentials");
    		err = socket_sectag_set(fd, sec_tag);
    		if (err) {
    			LOG_ERR("Fail to set up TLS credentials: %d", err);
    			goto cleanup;
    		}
    	}
    
    	/* Not connected */
    	err = -1;
    
    	for (addr = info; addr != NULL; addr = addr->ai_next) {
    		struct sockaddr *const sa = addr->ai_addr;
    
    		switch (sa->sa_family) {
    		case AF_INET6:
    			((struct sockaddr_in6 *)sa)->sin6_port = port;
    			break;
    		case AF_INET:
    			((struct sockaddr_in *)sa)->sin_port = port;
    			break;
    		}
    
    		err = connect(fd, sa, addr->ai_addrlen);
    		if (err) {
    			/* Try next address */
    			LOG_ERR("Unable to connect, errno %d", errno);
    		} else {
    			/* Connected */
    			break;
    		}
    	}
    
    cleanup:
    	freeaddrinfo(info);
    
    	if (err) {
    		/* Unable to connect, close socket */
    		close(fd);
    		fd = -1;
    	}
    
    	return fd;
    }
    
    static int socket_timeout_set(int fd)
    {
    	int err;
    	const uint32_t timeout_ms = HTTPC_REQ_TO_S * MSEC_PER_SEC;
    
    	struct timeval timeo = {
    		.tv_sec = (timeout_ms / 1000),
    		.tv_usec = (timeout_ms % 1000) * 1000,
    	};
    	LOG_DBG("Configuring socket timeout (%ld s)", timeo.tv_sec);
    	err = setsockopt(fd, SOL_SOCKET, SO_SNDTIMEO, &timeo, sizeof(timeo));
    	if (err) {
    		LOG_WRN("Failed to set socket TX timeout, errno %d", errno);
    		return -errno;
    	}
    	err = setsockopt(fd, SOL_SOCKET, SO_RCVTIMEO, &timeo, sizeof(timeo));
    	if (err) {
    		LOG_WRN("Failed to set socket RX timeout, errno %d", errno);
    		return -errno;
    	}
    
    	return 0;
    }
    
    static int server_connect(const char *host, int sec_tag)
    {
    	int fd = -1;
    	int err;
    
    	if (host == NULL) {
    		LOG_ERR("Empty remote host.");
    		return -EINVAL;
    	}
    
    	if ((httpc.sec_transport == true) && (sec_tag == -1)) {
    		LOG_ERR("Empty secure tag.");
    		return -EINVAL;
    	}
    
    	/* Attempt IPv6 connection if configured, fallback to IPv4 */
    	fd = resolve_and_connect(AF_INET6, host, sec_tag);
    	if (fd < 0) {
    		fd = resolve_and_connect(AF_INET, host, sec_tag);
    	}
    
    	if (fd < 0) {
    		LOG_ERR("Fail to resolve and connect");
    		return -EINVAL;
    	}
    	LOG_INF("Connected to %s", log_strdup(host));
    
    	/* Set socket timeout, if configured */
    	err = socket_timeout_set(fd);
    	if (err) {
    		close(httpc.fd);
    		httpc.fd = INVALID_SOCKET;
    		return err;
    	}
    
    	return fd;
    }
    
    static void response_cb(struct http_response *rsp,
    			enum http_final_call final_data,
    			void *user_data)
    {
    	static size_t data_received;
    
    	if (rsp->data_len > HTTPC_BUF_LEN) {
    		/* Increase HTTPC_BUF_LEN in case of overflow */
    		LOG_WRN("HTTP parser buffer overflow!");
    		return;
    	}
    
    	data_received += rsp->data_len;
    	if (final_data == HTTP_DATA_MORE) {
    		LOG_DBG("Partial data received (%zd bytes)", rsp->data_len);
    		if (data_received == HTTPC_BUF_LEN) {
    			sprintf(rsp_buf, "#XHTTPCRSP:%d,1\r\n",
    				HTTPC_BUF_LEN);
    			rsp_send(rsp_buf, strlen(rsp_buf));
    			rsp_send(data_buf, HTTPC_BUF_LEN);
    			data_received = 0;
    		}
    	} else if (final_data == HTTP_DATA_FINAL) {
    		LOG_DBG("All the data received (%zd bytes)", data_received);
    		sprintf(rsp_buf, "#XHTTPCRSP:%d,0\r\n", data_received);
    		rsp_send(rsp_buf, strlen(rsp_buf));
    		rsp_send(data_buf, data_received);
    		httpc.rsp_completed = true;
    		data_received = 0;
    	}
    }
    
    static int headers_cb(int sock, struct http_request *req, void *user_data)
    {
    	size_t len;
    	int ret = 0;
    
    	len = strlen(httpc.headers);
    	while (len > 0) {
    		ret = send(sock, httpc.headers + ret, len, 0);
    		if (ret < 0) {
    			LOG_ERR("send header fail: %d", ret);
    			return ret;
    		}
    		LOG_DBG("send header: %d bytes", ret);
    		len -= ret;
    	}
    
    	return len;
    }
    
    static int payload_cb(int sock, struct http_request *req, void *user_data)
    {
    	size_t total_sent = 0;
    
    	if (httpc.pl_len > 0) {
    		sprintf(rsp_buf, "#XHTTPCREQ:1\r\n");
    		rsp_send(rsp_buf, strlen(rsp_buf));
    		do {
    			/* Wait until payload is ready */
    			LOG_DBG("wait until payload is ready");
    			k_sem_take(&http_req_sem, K_FOREVER);
    			if (httpc.pl_len == 0) {
    				LOG_INF("abort sending payload");
    				httpc.pl_sent = 0;
    				return -ECONNABORTED;
    			}
    			if (total_sent + httpc.pl_to_send > httpc.pl_len) {
    				LOG_WRN("send unexpected payload");
    				httpc.pl_to_send = httpc.pl_len - total_sent;
    			}
    			while (httpc.pl_sent < httpc.pl_to_send) {
    				ssize_t ret;
    
    				ret = send(sock, httpc.payload + httpc.pl_sent,
    					   MIN(httpc.pl_to_send - httpc.pl_sent,
    					   HTTPC_FRAG_SIZE), 0);
    				if (ret < 0) {
    					LOG_ERR("send fail: %d", ret);
    					httpc.pl_len = 0;
    					httpc.pl_sent = 0;
    					k_sem_give(&http_data_sem);
    					return -errno;
    				}
    				LOG_DBG("send %d bytes payload", ret);
    				httpc.pl_sent += ret;
    				total_sent += ret;
    			}
    			if (total_sent == httpc.pl_len) {
    				httpc.pl_len = 0;
    			}
    			k_sem_give(&http_data_sem);
    		} while (total_sent < httpc.pl_len);
    		sprintf(rsp_buf, "#XHTTPCREQ:0\r\n");
    		rsp_send(rsp_buf, strlen(rsp_buf));
    	} else {
    		sprintf(rsp_buf, "#XHTTPCREQ:0\r\n");
    		rsp_send(rsp_buf, strlen(rsp_buf));
    	}
    
    	return total_sent;
    }
    
    static int do_http_connect(void)
    {
    	/* Connect to server if it is not connected yet. */
    	if (httpc.fd == INVALID_SOCKET) {
    		httpc.fd = server_connect(httpc.host, httpc.sec_tag);
    		if (httpc.fd < 0) {
    			LOG_ERR("server_connect fail.");
    			httpc.fd = INVALID_SOCKET;
    			sprintf(rsp_buf, "#XHTTPCCON:0\r\n");
    			rsp_send(rsp_buf, strlen(rsp_buf));
    		} else {
    			sprintf(rsp_buf, "#XHTTPCCON:1\r\n");
    			rsp_send(rsp_buf, strlen(rsp_buf));
    		}
    	} else {
    		LOG_ERR("Already connected to server.");
    		return -EINVAL;
    	}
    
    	return httpc.fd;
    }
    
    static int do_http_disconnect(void)
    {
    	int err = -EINVAL;
    
    	/* Close socket if it is connected. */
    	if (httpc.fd != INVALID_SOCKET) {
    		close(httpc.fd);
    		httpc.fd = INVALID_SOCKET;
    		err = 0;
    	} else {
    		return -ENOTCONN;
    	}
    	if (httpc.pl_len > 0) {
    		LOG_ERR("Exit request");
    		httpc.pl_len = 0;
    		k_sem_give(&http_req_sem);
    	}
    	sprintf(rsp_buf, "#XHTTPCCON:0\r\n");
    	rsp_send(rsp_buf, strlen(rsp_buf));
    
    	return err;
    }
    
    static int http_method_str_enum(uint8_t *method_str)
    {
    	static const char * const method_strings[] = {
    		"DELETE", "GET", "HEAD", "POST", "PUT", "CONNECT", "OPTIONS",
    		"TRACE", "COPY", "LOCK", "MKCOL", "MOVE", "PROPFIND",
    		"PROPPATCH", "SEARCH", "UNLOCK", "BIND", "REBIND", "UNBIND",
    		"ACL", "REPORT", "MKACTIVITY", "CHECKOUT", "MERGE", "M-SEARCH",
    		"NOTIFY", "SUBSCRIBE", "UNSUBSCRIBE", "PATCH", "PURGE",
    		"MKCALENDAR", "LINK", "UNLINK"};
    
    	for (int i = HTTP_DELETE; i <= HTTP_UNLINK; i++) {
    		if (!strncmp(method_str, method_strings[i],
    			HTTPC_METHOD_LEN)) {
    			return i;
    		}
    	}
    	return -1;
    }
    
    static int do_http_request(void)
    {
    	int err = -EINVAL;
    	struct http_request req;
    	int method;
    	int32_t timeout = SYS_FOREVER_MS;
    
    	if (httpc.fd == INVALID_SOCKET) {
    		LOG_ERR("Remote host is not connected.");
    		return -EINVAL;
    	}
    
    	method = http_method_str_enum(httpc.method_str);
    	if (method < 0) {
    		LOG_ERR("Request method is not allowed.");
    		return -EINVAL;
    	}
    
    	memset(&req, 0, sizeof(req));
    	req.method = (enum http_method)method;
    	req.url = httpc.resource;
    	req.host = httpc.host;
    	req.protocol = "HTTP/1.1";
    	req.response = response_cb;
    	req.recv_buf = data_buf;
    	req.recv_buf_len = HTTPC_BUF_LEN;
    	req.payload_cb =  payload_cb;
    	req.optional_headers_cb = headers_cb;
    	err = http_client_req(httpc.fd, &req, timeout, "");
    	if (err < 0) {
    		/* Socket send/recv error */
    		sprintf(rsp_buf, "#XHTTPCREQ:%d\r\n", err);
    		rsp_send(rsp_buf, strlen(rsp_buf));
    	} else if (httpc.rsp_completed == false) {
    		/* Socket was closed by remote */
    		err = -ECONNRESET;
    		sprintf(rsp_buf, "#XHTTPCRSP:0,%d\r\n", err);
    		rsp_send(rsp_buf, strlen(rsp_buf));
    	} else {
    		err = 0;
    	}
    
    	return err;
    }
    
    /**@brief handle AT#XHTTPCCON commands
     *  AT#XHTTPCCON=<op>[,<host>,<port>[,<sec_tag>]]
     *  AT#XHTTPCCON? READ command not supported
     *  AT#XHTTPCCON=?
     */
    static int handle_AT_HTTPC_CONNECT(enum at_cmd_type cmd_type)
    {
    	int err = -EINVAL;
    
    	uint16_t op;
    	size_t host_sz = HTTPC_HOST_LEN;
    
    	switch (cmd_type) {
    	case AT_CMD_TYPE_SET_COMMAND:
    		if (at_params_valid_count_get(&at_param_list) == 0) {
    			return -EINVAL;
    		}
    		err = at_params_short_get(&at_param_list, 1, &op);
    		if (err < 0) {
    			LOG_ERR("Fail to get op: %d", err);
    			return err;
    		}
    
    		if (op == AT_HTTPCCON_CONNECT) {
    			if (at_params_valid_count_get(&at_param_list) <= 3) {
    				return -EINVAL;
    			}
    			if (httpc.fd != INVALID_SOCKET) {
    				return -EINPROGRESS;
    			}
    			err = at_params_string_get(&at_param_list, 2,
    							httpc.host, &host_sz);
    			if (err < 0) {
    				LOG_ERR("Fail to get host: %d", err);
    				return err;
    			}
    
    			httpc.host[host_sz] = '\0';
    			err = at_params_int_get(&at_param_list, 3, &httpc.port);
    			if (err < 0) {
    				LOG_ERR("Fail to get port: %d", err);
    				return err;
    			}
    
    			if (at_params_valid_count_get(&at_param_list) == 5) {
    				err = at_params_int_get(&at_param_list, 4,
    							&httpc.sec_tag);
    				if (err < 0) {
    					LOG_ERR("Fail to get sec_tag: %d", err);
    					return err;
    				}
    				httpc.sec_transport = true;
    			}
    			LOG_DBG("Connect from http server");
    			if (do_http_connect() >= 0) {
    				err = 0;
    			} else {
    				err = -EINVAL;
    			}
    			break;
    		} else if (op == AT_HTTPCCON_DISCONNECT) {
    			LOG_DBG("Disconnect from http server");
    			err = do_http_disconnect();
    			if (err) {
    				LOG_ERR("Fail to disconnect. Error: %d", err);
    			}
    		} break;
    
    	case AT_CMD_TYPE_READ_COMMAND:
    		if (httpc.sec_transport) {
    			sprintf(rsp_buf, "#XHTTPCCON: %d,\"%s\",%d,%d\r\n",
    				(httpc.fd == INVALID_SOCKET)?0:1, httpc.host,
    				httpc.port, httpc.sec_tag);
    		} else {
    			sprintf(rsp_buf, "#XHTTPCCON: %d,\"%s\",%d\r\n",
    				(httpc.fd == INVALID_SOCKET)?0:1, httpc.host,
    				httpc.port);
    		}
    		rsp_send(rsp_buf, strlen(rsp_buf));
    		err = 0;
    		break;
    
    	case AT_CMD_TYPE_TEST_COMMAND:
    		sprintf(rsp_buf,
    			"#XHTTPCCON: (%d, %d),<host>,<port>,<sec_tag>\r\n",
    			AT_HTTPCCON_DISCONNECT, AT_HTTPCCON_CONNECT);
    		rsp_send(rsp_buf, strlen(rsp_buf));
    		err = 0;
    		break;
    
    	default:
    		break;
    	}
    
    	return err;
    }
    
    
    /**@brief handle AT#XHTTPCREQ commands
     *  AT#XHTTPCREQ=<method>,<resource>,<header>[,<payload_length>]
     *  AT#XHTTPCREQ? READ command not supported
     *  AT#XHTTPCREQ=?
     */
    static int handle_AT_HTTPC_REQUEST(enum at_cmd_type cmd_type)
    {
    	int err = -EINVAL;
    	int param_count;
    	size_t method_sz = HTTPC_METHOD_LEN;
    	size_t resource_sz = HTTPC_RES_LEN;
    	size_t headers_sz = HTTPC_HEADER_LEN;
    	size_t offset;
    
    	if (httpc.fd == INVALID_SOCKET) {
    		LOG_ERR("Remote host is not connected.");
    		return err;
    	}
    
    	if (httpc.pl_len > 0) {
    		LOG_ERR("Another request is not finished.");
    		return err;
    	}
    
    	switch (cmd_type) {
    	case AT_CMD_TYPE_SET_COMMAND:
    		param_count = at_params_valid_count_get(&at_param_list);
    		if (param_count < 3) {
    			return -EINVAL;
    		}
    		err = at_params_string_get(&at_param_list, 1,
    					   data_buf, &method_sz);
    		if (err < 0) {
    			LOG_ERR("Fail to get method string: %d", err);
    			return err;
    		}
    		data_buf[method_sz] = '\0';
    		httpc.method_str = data_buf;
    		offset = method_sz + 1;
    		/* Get resource path string */
    		err = at_params_string_get(&at_param_list, 2,
    					   data_buf + offset, &resource_sz);
    		if (err < 0) {
    			LOG_ERR("Fail to get resource string: %d", err);
    			return err;
    		}
    		data_buf[offset + resource_sz] = '\0';
    		httpc.resource = data_buf + offset;
    		offset = offset + resource_sz + 1;
    		/* Get header string */
    		err = at_params_string_get(&at_param_list, 3,
    					   data_buf + offset, &headers_sz);
    		if (err < 0) {
    			LOG_ERR("Fail to get option string: %d", err);
    			return err;
    		}
    		data_buf[offset + headers_sz] = '\0';
    		httpc.headers = data_buf + offset;
    		if (param_count >= 5) {
    			err = at_params_int_get(&at_param_list, 4,
    						  &httpc.pl_len);
    			if (err != 0) {
    				return err;
    			}
    		}
    		/* start sending request */
    		k_sem_give(&http_req_sem);
    		break;
    
    	case AT_CMD_TYPE_TEST_COMMAND:
    		break;
    
    	default:
    		break;
    	}
    
    	return err;
    }
    
    /**@brief API to handle HTTP AT commands
     */
    int slm_at_httpc_parse(const char *at_cmd, size_t length)
    {
    	int ret = -ENOENT;
    	enum at_cmd_type type;
    
    	for (int i = 0; i < AT_HTTPC_MAX; i++) {
    		if (slm_util_cmd_casecmp(at_cmd, http_at_list[i].string)) {
    			ret = at_parser_params_from_str(at_cmd, NULL,
    						&at_param_list);
    			if (ret) {
    				LOG_ERR("Failed to parse AT command %d", ret);
    				return -EINVAL;
    			}
    			type = at_parser_cmd_type_get(at_cmd);
    			ret = http_at_list[i].handler(type);
    			break;
    		}
    	}
    
    	/* Return if no payload to send */
    	if (ret != -ENOENT || (httpc.pl_len == 0)) {
    		return ret;
    	}
    	/* Process input data as payload */
    	httpc.payload = (char *)at_cmd;
    	httpc.pl_to_send = length;
    	httpc.pl_sent = 0;
    	/* start sending payload */
    	k_sem_give(&http_req_sem);
    	/* Don't go any further until send returns */
    	k_sem_take(&http_data_sem, K_FOREVER);
    	if (httpc.pl_sent <= 0) {
    		return -EINVAL;
    	}
    	return 0;
    }
    
    static void httpc_thread_fn(void *arg1, void *arg2, void *arg3)
    {
    	int err;
    
    	while (1) {
    		/* Don't go any further until sending HTTP request */
    		k_sem_take(&http_req_sem, K_FOREVER);
    		err = do_http_request();
    		if (err < 0) {
    			LOG_ERR("do_http_request fail:%d", err);
    			/* Disconnect from server */
    			err = do_http_disconnect();
    			if (err) {
    				LOG_ERR("Fail to disconnect. Error: %d", err);
    			}
    		}
    	}
    }
    
    int slm_at_httpc_init(void)
    {
    	httpc.fd = INVALID_SOCKET;
    
    	return 0;
    }
    
    int slm_at_httpc_uninit(void)
    {
    	int err = 0;
    
    	if (httpc.fd != INVALID_SOCKET) {
    		err = do_http_disconnect();
    		if (err != 0) {
    			LOG_ERR("Fail to disconnect. Error: %d", err);
    		}
    	}
    
    	return err;
    }
    
    /**@brief API to list HTTP AT commands
     */
    void slm_at_httpc_clac(void)
    {
    	for (int i = 0; i < AT_HTTPC_MAX; i++) {
    		sprintf(rsp_buf, "%s\r\n", http_at_list[i].string);
    		rsp_send(rsp_buf, strlen(rsp_buf));
    	}
    }
    
    K_THREAD_DEFINE(httpc_thread, K_THREAD_STACK_SIZEOF(httpc_thread_stack),
    		httpc_thread_fn, NULL, NULL, NULL,
    		THREAD_PRIORITY, 0, 0);
    


    Best regards,
    Carl Richard

  • Hi, thank you, this fix has indeed resolved the problem with chunked transport and cloudflare. However since I patched the file in our code, our subsequent code (json parsing etc..) is failing. I'm investigating it to see if it is related to this fix or it is a bug in our code. 

    BR, Michal V.

  • So I've found where the problem is - the length of the chunk is included in the response from XHTTPCREQ and that crashes the json parser. So I belive that chunked responses are still not handled correctly. But I can confirm that the issue with buffer has been fixed.

    NRF91 log:

    00> [00:07:02.802,215] <dbg> net_http.on_message_begin: (0x20021628): -- HTTP POST response (headers) --
    00> [00:07:02.802,612] <dbg> net_http.on_status: (0x20021628): HTTP response status 200 OK
    00> [00:07:02.804,779] <dbg> net_http.print_header_field: (0x20021628): [4] Date
    00> [00:07:02.805,541] <dbg> net_http.print_header_field: (0x20021628): [29] Tue, 01 Dec 2020 14:29:39 GMT
    00> [00:07:02.806,365] <dbg> net_http.print_header_field: (0x20021628): [12] Content-Type
    00> [00:07:02.807,098] <dbg> net_http.print_header_field: (0x20021628): [31] application/json; charset=utf-8
    00> [00:07:02.807,891] <dbg> net_http.print_header_field: (0x20021628): [17] Transfer-Encoding
    00> [00:07:02.808,624] <dbg> net_http.print_header_field: (0x20021628): [7] chunked
    00> [00:07:02.809,356] <dbg> net_http.print_header_field: (0x20021628): [10] Connection
    00> [00:07:02.810,028] <dbg> net_http.print_header_field: (0x20021628): [10] keep-alive
    00> [00:07:02.810,729] <dbg> net_http.print_header_field: (0x20021628): [10] Set-Cookie
    00> [00:07:02.811,492] <dbg> net_http.print_header_field: (0x20021628): [127] __cfduid=d6359a98730e1dcfa60b181709cdefba01606832978; expires=Thu, 31-Dec-20 14:29:38 GMT; path=/; domain=.gwxcloud.com; HttpOn
    00> [00:07:02.812,438] <dbg> net_http.print_header_field: (0x20021628): [4] vary
    00> [00:07:02.814,910] <dbg> net_http.print_header_field: (0x20021628): [15] Accept-Encoding
    00> [00:07:02.815,551] <dbg> net_http.print_header_field: (0x20021628): [15] CF-Cache-Status
    00> [00:07:02.816,528] <dbg> net_http.print_header_field: (0x20021628): [7] DYNAMIC
    00> [00:07:02.817,108] <dbg> net_http.print_header_field: (0x20021628): [13] cf-request-id
    00> [00:07:02.818,267] <dbg> net_http.print_header_field: (0x20021628): [32] 06c04e94010000d44788baf000000001
    00> [00:07:02.819,305] <dbg> net_http.print_header_field: (0x20021628): [9] Report-To
    00> [00:07:02.820,190] <dbg> net_http.print_header_field: (0x20021628): [127] {"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report?s=jj2GNMCYecVq5hD8ci0dB2gKd6f75hi6eoWByctGmAhyCV07FX7T8PZA76aOwnBk
    00> [00:07:02.823,120] <dbg> net_http.print_header_field: (0x20021628): [3] NEL
    00> [00:07:02.823,608] <dbg> net_http.print_header_field: (0x20021628): [39] {"report_to":"cf-nel","max_age":604800}
    00> [00:07:02.824,096] <dbg> net_http.print_header_field: (0x20021628): [6] Server
    00> [00:07:02.824,554] <dbg> net_http.print_header_field: (0x20021628): [10] cloudflare
    00> [00:07:02.850,463] <dbg> net_http.print_header_field: (0x20021628): [6] CF-RAY
    00> [00:07:02.851,165] <dbg> net_http.print_header_field: (0x20021628): [20] 5fad80666f7ad447-HAM
    00> [00:07:02.851,562] <dbg> net_http.on_headers_complete: (0x20021628): No body expected
    00> [00:07:02.851,898] <dbg> net_http.on_message_complete: (0x20021628): -- HTTP POST response (complete) --
    00> [00:07:02.852,355] <inf> at_host: TX
    00> 23 58 48 54 54 50 43 52  53 50 3a 38 33 32 2c 30 |#XHTTPCR SP:832,0
    00> 0d 0a                                            |..               
    00> [00:07:02.854,827] <inf> at_host: TX
    00> 48 54 54 50 2f 31 2e 31  20 32 30 30 20 4f 4b 0d |HTTP/1.1  200 OK.
    00> 0a 44 61 74 65 3a 20 54  75 65 2c 20 30 31 20 44 |.Date: T ue, 01 D
    00> 65 63 20 32 30 32 30 20  31 34 3a 32 39 3a 33 39 |ec 2020  14:29:39
    00> 20 47 4d 54 0d 0a 43 6f  6e 74 65 6e 74 2d 54 79 | GMT..Co ntent-Ty
    00> 70 65 3a 20 61 70 70 6c  69 63 61 74 69 6f 6e 2f |pe: appl ication/
    00> 6a 73 6f 6e 3b 20 63 68  61 72 73 65 74 3d 75 74 |json; ch arset=ut
    00> 66 2d 38 0d 0a 54 72 61  6e 73 66 65 72 2d 45 6e |f-8..Tra nsfer-En
    00> 63 6f 64 69 6e 67 3a 20  63 68 75 6e 6b 65 64 0d |coding:  chunked.
    00> 0a 43 6f 6e 6e 65 63 74  69 6f 6e 3a 20 6b 65 65 |.Connect ion: kee
    00> 70 2d 61 6c 69 76 65 0d  0a 53 65 74 2d 43 6f 6f |p-alive. .Set-Coo
    00> 6b 69 65 3a 20 5f 5f 63  66 64 75 69 64 3d 64 36 |kie: __c fduid=d6
    00> 33 35 39 61 39 38 37 33  30 65 31 64 63 66 61 36 |359a9873 0e1dcfa6
    00> 30 62 31 38 31 37 30 39  63 64 65 66 62 61 30 31 |0b181709 cdefba01
    00> 36 30 36 38 33 32 39 37  38 3b 20 65 78 70 69 72 |60683297 8; expir
    00> 65 73 3d 54 68 75 2c 20  33 31 2d 44 65 63 2d 32 |es=Thu,  31-Dec-2
    00> 30 20 31 34 3a 32 39 3a  33 38 20 47 4d 54 3b 20 |0 14:29: 38 GMT; 
    00> 70 61 74 68 3d 2f 3b 20  64 6f 6d 61 69 6e 3d 2e |path=/;  domain=.
    00> 67 77 78 63 6c 6f 75 64  2e 63 6f 6d 3b 20 48 74 |gwxcloud .com; Ht
    00> 74 70 4f 6e 6c 79 3b 20  53 61 6d 65 53 69 74 65 |tpOnly;  SameSite
    00> 3d 4c 61 78 0d 0a 76 61  72 79 3a 20 41 63 63 65 |=Lax..va ry: Acce
    00> 70 74 2d 45 6e 63 6f 64  69 6e 67 0d 0a 43 46 2d |pt-Encod ing..CF-
    00> 43 61 63 68 65 2d 53 74  61 74 75 73 3a 20 44 59 |Cache-St atus: DY
    00> 4e 41 4d 49 43 0d 0a 63  66 2d 72 65 71 75 65 73 |NAMIC..c f-reques
    00> 74 2d 69 64 3a 20 30 36  63 30 34 65 39 34 30 31 |t-id: 06 c04e9401
    00> 30 30 30 30 64 34 34 37  38 38 62 61 66 30 30 30 |0000d447 88baf000
    00> 30 30 30 30 30 31 0d 0a  52 65 70 6f 72 74 2d 54 |000001.. Report-T
    00> 6f 3a 20 7b 22 65 6e 64  70 6f 69 6e 74 73 22 3a |o: {"end points":
    00> 5b 7b 22 75 72 6c 22 3a  22 68 74 74 70 73 3a 5c |[{"url": "https:\
    00> 2f 5c 2f 61 2e 6e 65 6c  2e 63 6c 6f 75 64 66 6c |/\/a.nel .cloudfl
    00> 61 72 65 2e 63 6f 6d 5c  2f 72 65 70 6f 72 74 3f |are.com\ /report?
    00> 73 3d 6a 6a 32 47 4e 4d  43 59 65 63 56 71 35 68 |s=jj2GNM CYecVq5h
    00> 44 38 63 69 30 64 42 32  67 4b 64 36 66 37 35 68 |D8ci0dB2 gKd6f75h
    00> 69 36 65 6f 57 42 79 63  74 47 6d 41 68 79 43 56 |i6eoWByc tGmAhyCV
    00> 30 37 46 58 37 54 38 50  5a 41 37 36 61 4f 77 6e |07FX7T8P ZA76aOwn
    00> 42 6b 42 55 33 36 68 54  78 58 30 42 6f 70 48 53 |BkBU36hT xX0BopHS
    00> 37 6f 6e 63 50 69 79 75  52 6d 78 34 52 43 77 58 |7oncPiyu Rmx4RCwX
    00> 67 51 6f 6b 68 69 49 36  37 49 4c 77 30 6a 57 75 |gQokhiI6 7ILw0jWu
    00> 38 4d 4f 67 25 33 44 25  33 44 22 7d 5d 2c 22 67 |8MOg%3D% 3D"}],"g
    00> 72 6f 75 70 22 3a 22 63  66 2d 6e 65 6c 22 2c 22 |roup":"c f-nel","
    00> 6d 61 78 5f 61 67 65 22  3a 36 30 34 38 30 30 7d |max_age" :604800}
    00> 0d 0a 4e 45 4c 3a 20 7b  22 72 65 70 6f 72 74 5f |..NEL: { "report_
    00> 74 6f 22 3a 22 63 66 2d  6e 65 6c 22 2c 22 6d 61 |to":"cf- nel","ma
    00> 78 5f 61 67 65 22 3a 36  30 34 38 30 30 7d 0d 0a |x_age":6 04800}..
    00> 53 65 72 76 65 72 3a 20  63 6c 6f 75 64 66 6c 61 |Server:  cloudfla
    00> 72 65 0d 0a 43 46 2d 52  41 59 3a 20 35 66 61 64 |re..CF-R AY: 5fad
    00> 38 30 36 36 36 66 37 61  64 34 34 37 2d 48 41 4d |80666f7a d447-HAM
    00> 0d 0a 0d 0a 35 36 0d 0a  7b 22 64 61 74 61 22 3a |....56.. {"data":
    00> 22 33 35 38 32 65 32 31  39 32 39 30 30 39 39 64 |"3582e21 9290099d
    00> 38 62 34 38 62 66 61 34  38 31 34 38 65 39 64 35 |8b48bfa4 8148e9d5
    00> 34 64 31 38 34 35 64 63  64 64 36 62 30 62 66 32 |4d1845dc dd6b0bf2
    00> 37 63 30 37 64 30 30 66  30 63 36 61 32 32 31 66 |7c07d00f 0c6a221f
    00> 61 22 2c 22 63 6d 64 22  3a 6e 75 6c 6c 7d 0d 0a |a","cmd" :null}..

    On the line 79, it can be seen that there is: CR LF CR LF (separator of headers and body in HTTP) 56 CR LF (hex length of the chunk and CR LF as chunk length separator), the `56 CR LF` should not be there. 

    `00> 0d 0a 0d 0a 35 36 0d 0a  7b 22 64 61 74 61 22 3a |....56.. {"data":`

    In this simple case we can work around the issue, but I'm not sure whether the solution would hold if the body is transferred in more than one chunk.

  • Good to hear that the fix worked to a certain extent even though it revealed more issues. I'm working with the developer to sort it out!

    Best regards,
    Carl Richard

  • Hi again, Michal!

    A fix should have been pushed to the NCS master branch now. As described in the pull request the body was skipped. If you want to stay on 1.4.0 remove the following code from zephyr/subsys/net/lib/http/http_client.c 

    if ((req->method == HTTP_PUT || req->method == HTTP_POST) &&
        req->internal.response.content_length == 0) {
    	NET_DBG("No body expected");
    	return 1;
    }


    This should also fix the issue with the modem crashing, mentioned in your other ticket. Please report back there if it doesn't!

    Best regards,
    Carl Richard

Related