nRF9160DK and AWS Greengrass Discovery Sequence

[ronald-blackline]

Hi,

I am trying to implement the AWS Discovery Sequence with the nRF9160DK board as the AWS Greengrass connected device. I am using the nRF Connect SDK v1.4.0 and used the HTTPS_CLIENT sample application as the basis of my implementation. I also had the modem firmware v1.2.2 flashed into my nRF9160DK board.

Basically, the first step in AWS Greengrass discovery sequence is connecting to AWS IoT Core (with my specific AWS endpoint) securely using TLS. In that regard, I modified the sample app "cert_provision()" so instead of just setting up the Root CA, I also had to setup the Client CA and the Client Private Key. With that, my application can successfully connect to my AWS IoT endpoint securely using TLS.

The next step is to do/send the following HTTP method: "GET /greengrass/discover/thing/<name> HTTP/1.1\r\n\r\n". From the logs, the request has been sent successfully. However, I am not getting ay response of 0 bytes.

HTTPS client sample started
modem_key_mgmt_delete(42, 0) => result=0
modem_key_mgmt_delete(42, 1) => result=0
modem_key_mgmt_delete(42, 2) => result=0
Provisioning certificate
modem_key_mgmt_write => result=0
modem_key_mgmt_write => result=0
modem_key_mgmt_write => result=0
Waiting for network.. OK
Connecting to xxxxxxxxxxxxx-ats.iot.us-east-2.amazonaws.com
Sent 53 bytes
Received 0 bytes
Finished, closing socket.

I did a bit of research and the expected response packet (JSON string) is more than 2KB. I found out that the modem firmware puts a limit to the TLS secure buffer of 2KB. Next thing I did was trim down the response packet to under 2KB using the AWS IoT Console. However, in doing so, my application seems to block after sending the initial packet.

HTTPS client sample started
modem_key_mgmt_delete(42, 0) => result=0
modem_key_mgmt_delete(42, 1) => result=0
modem_key_mgmt_delete(42, 2) => result=0
Provisioning certificate
modem_key_mgmt_write => result=0
modem_key_mgmt_write => result=0
modem_key_mgmt_write => result=0
Waiting for network.. OK
Connecting to xxxxxxxxxxxxx-ats.iot.us-east-2.amazonaws.com
Sent 53 bytes

I have been experimenting with different HTTP request and one of such is sending this HTTP request: "HEAD /greengrass/discover/thing/<name> HTTP/1.1\r\n\r\n". In doing so, I can see the AWS IoT response being received.

HTTPS client sample started
modem_key_mgmt_delete(42, 0) => result=0
modem_key_mgmt_delete(42, 1) => result=0
modem_key_mgmt_delete(42, 2) => result=0
Provisioning certificate
modem_key_mgmt_write => result=0
modem_key_mgmt_write => result=0
modem_key_mgmt_write => result=0
Waiting for network.. OK
Connecting to xxxxxxxxxx-ats.iot.us-east-2.amazonaws.com
Sent 54 bytes
Received 389 bytes

>        HTTP/1.1 404 Not Found
content-type: application/json
content-length: 147
date: Wed, 09 Dec 2020 14:48:40 GMT
x-amzn-RequestId: 3ecdee7e-50d3-47db-535e-1cd205057028
connection: keep-alive
x-amzn-ErrorType: ResourceNotFoundException:

{"message":"Greengrass URI:/greengrass/discover/thing/xxxxxxx by HTTP method:HEAD is not found","traceId":"3ecdee7e-50d3-47db-535e-1cd205057028"}

Finished, closing socket.

This led me to believe that there is something wrong with the way the response is being received specially if the response is close to 2KB. I am attaching the modem trace I captured if that would be useful.

3808.trace-2020-12-09T14-18-33.181Z.bin

The rest is the source code that I used.

/*
 * Copyright (c) 2020 Nordic Semiconductor ASA
 *
 * SPDX-License-Identifier: LicenseRef-BSD-5-Clause-Nordic
 */

#include <string.h>
#include <zephyr.h>
#include <stdlib.h>
#include <net/socket.h>
#include <modem/bsdlib.h>
#include <net/tls_credentials.h>
#include <modem/lte_lc.h>
#include <modem/at_cmd.h>
#include <modem/at_notif.h>
#include <modem/modem_key_mgmt.h>

#define HTTPS_PORT 8443

#define HTTP_GET                                                              \
	"GET /greengrass/discover/thing/xxxxxx HTTP/1.1\r\n\r\n"

#define HTTP_GET_LEN (sizeof(HTTP_GET) - 1)

#define HTTP_HDR_END "\r\n\r\n"

#define RECV_BUF_SIZE 4096
#define TLS_SEC_TAG 42

static const char send_buf[] = HTTP_GET;
static char recv_buf[RECV_BUF_SIZE];

#include "../cert/certificates.h"

#define MAX_OF_2 MAX(sizeof(CLOUD_CA_CERTIFICATE),\
		     sizeof(CLOUD_CLIENT_PRIVATE_KEY))
#define MAX_LEN MAX(MAX_OF_2, sizeof(CLOUD_CLIENT_PUBLIC_CERTIFICATE))

static uint8_t certificates[][MAX_LEN] = {{CLOUD_CA_CERTIFICATE},
				       {CLOUD_CLIENT_PRIVATE_KEY},
				       {CLOUD_CLIENT_PUBLIC_CERTIFICATE} };
static const size_t cert_len[] = {
	sizeof(CLOUD_CA_CERTIFICATE) - 1, sizeof(CLOUD_CLIENT_PRIVATE_KEY) - 1,
	sizeof(CLOUD_CLIENT_PUBLIC_CERTIFICATE) - 1
};

/* Initialize AT communications */
int at_comms_init(void)
{
	int err;

	err = at_cmd_init();
	if (err) {
		printk("Failed to initialize AT commands, err %d\n", err);
		return err;
	}

	err = at_notif_init();
	if (err) {
		printk("Failed to initialize AT notifications, err %d\n", err);
		return err;
	}

	return 0;
}

/* Provision certificate to modem */
int cert_provision(void)
{
	int err;

	enum modem_key_mgnt_cred_type cred[] = {
		MODEM_KEY_MGMT_CRED_TYPE_CA_CHAIN,
		MODEM_KEY_MGMT_CRED_TYPE_PRIVATE_CERT,
		MODEM_KEY_MGMT_CRED_TYPE_PUBLIC_CERT,
	};

	/* Delete certificates */
	for (enum modem_key_mgnt_cred_type type = 0; type < 3; type++) {
		err = modem_key_mgmt_delete(TLS_SEC_TAG, type);
		printk("modem_key_mgmt_delete(%u, %d) => result=%d\n",
				TLS_SEC_TAG, type, err);
	}

	printk("Provisioning certificate\n");

	/* Write certificates */
	for (enum modem_key_mgnt_cred_type type = 0; type < 3; type++) {
		err = modem_key_mgmt_write(TLS_SEC_TAG, cred[type],
				certificates[type], cert_len[type]);
		printk("modem_key_mgmt_write => result=%d\n", err);
	}

	return 0;
}

/* Setup TLS options on a given socket */
int tls_setup(int fd)
{
	int err;
	int verify;

	/* Security tag that we have provisioned the certificate with */
	const sec_tag_t tls_sec_tag[] = {
		TLS_SEC_TAG,
	};

	/* Set up TLS peer verification */
	enum {
		NONE = 0,
		OPTIONAL = 1,
		REQUIRED = 2,
	};

	verify = REQUIRED;

	err = setsockopt(fd, SOL_TLS, TLS_PEER_VERIFY, &verify, sizeof(verify));
	if (err) {
		printk("Failed to setup peer verification, err %d\n", errno);
		return err;
	}

	/* Associate the socket with the security tag
	 * we have provisioned the certificate with.
	 */
	err = setsockopt(fd, SOL_TLS, TLS_SEC_TAG_LIST, tls_sec_tag,
			 sizeof(tls_sec_tag));
	if (err) {
		printk("Failed to setup TLS sec tag, err %d\n", errno);
		return err;
	}

	return 0;
}

void main(void)
{
	int err;
	int fd;
	char *p;
	int bytes;
	size_t off;
	struct addrinfo *res;
	struct addrinfo hints = {
		.ai_family = AF_INET,
		.ai_socktype = SOCK_STREAM,
	};

	printk("HTTPS client sample started\n\r");

	err = bsdlib_init();
	if (err) {
		printk("Failed to initialize bsdlib!");
		return;
	}

	/* Initialize AT comms in order to provision the certificate */
	err = at_comms_init();
	if (err) {
		return;
	}

	/* Provision certificates before connecting to the LTE network */
	err = cert_provision();
	if (err) {
		return;
	}

	printk("Waiting for network.. ");
	err = lte_lc_init_and_connect();
	if (err) {
		printk("Failed to connect to the LTE network, err %d\n", err);
		return;
	}
	printk("OK\n");

	err = getaddrinfo("xxxxxxxxx-ats.iot.us-east-2.amazonaws.com", NULL, &hints, &res);
	if (err) {
		printk("getaddrinfo() failed, err %d\n", errno);
		return;
	}

	((struct sockaddr_in *)res->ai_addr)->sin_port = htons(HTTPS_PORT);

	fd = socket(AF_INET, SOCK_STREAM, IPPROTO_TLS_1_2);
	if (fd == -1) {
		printk("Failed to open socket!\n");
		goto clean_up;
	}

	/* Setup TLS socket options */
	err = tls_setup(fd);
	if (err) {
		goto clean_up;
	}

	printk("Connecting to %s\n", "xxxxxxxxx-ats.iot.us-east-2.amazonaws.com");
	err = connect(fd, res->ai_addr, sizeof(struct sockaddr_in));
	if (err) {
		printk("connect() failed, err: %d\n", errno);
		goto clean_up;
	}

	off = 0;
	do {
		bytes = send(fd, &send_buf[off], HTTP_GET_LEN - off, 0);
		if (bytes < 0) {
			printk("send() failed, err %d\n", errno);
			goto clean_up;
		}
		off += bytes;
	} while (off < HTTP_GET_LEN);

	printk("Sent %d bytes\n", off);

	off = 0;
	do {
		bytes = recv(fd, &recv_buf[off], RECV_BUF_SIZE - off, 0);
		if (bytes < 0) {
			printk("recv() failed, err %d\n", errno);
			goto clean_up;
		}
		off += bytes;

	} while (bytes != 0 /* peer closed connection */);

	printk("Received %d bytes\n", off);

	/* Print HTTP response */
	//p = strstr(recv_buf, "\r\n");
	//if (p) {
	//	off = p - recv_buf;
	//	recv_buf[off + 1] = '\0';
		printk("\n>\t %s\n\n", recv_buf);
	//}

	printk("Finished, closing socket.\n");

clean_up:
	freeaddrinfo(res);
	(void)close(fd);
}

[Simon]

I'm sorry for the delay on this. I'm not too familiar with AWS/AWS Greengrass, and encountered some problems when trying set it up to reproduce your issue. I will prioritize this ticket forward.

Best regards,

Simon

[ronald-blackline]

Thank you for looking. Can you take a quick look at the modem trace and see if there is something that might be wrong when recieving the HTTP response?

[Simon]

You could try to do some debugging at the layer closest to the modem, try setting a breakpoint after nrf_recvfrom() and see if that function is blocking as well. It would be useful to be able to reproduce this myself. Are you able to upload a sample, and a step-by-step guide that helps me reproduce it?

[ronald-blackline]

With my latest code, it is not blocking anymore but Im receiving 0 bytes. Im still trying to capture logs on AWS if it is actually returning something. For the meantime, is providing you the merged HEX file of my firmware will help in reproducing the issue in your side?

[ronald-blackline]

Here is a little bit of information I just found out ... if i configure the response packet in such a way the HTTP response (including header) size is 1960 bytes, my app can received the HTTP response in entirety. However, if the response size is about 2700 bytes, the nrf_recvfrom() API returns 0 bytes.

[Simon]

ronald-blackline said:

Here is a little bit of information I just found out ... if i configure the response packet in such a way the HTTP response (including header) size is 1960 bytes, my app can received the HTTP response in entirety. However, if the response size is about 2700 bytes, the nrf_recvfrom() API returns 0 bytes.

These findings seems to align with this:

" I found out that the modem firmware puts a limit to the TLS secure buffer of 2KB." 

Do you agree?

Do you have any more question/issues regarding this that you want me to investigate/forward internally?

[ronald-blackline]

I just need 2 more information:

1) Confirmation that the modem (nrf_recvfrom) returns 0 bytes if it actually receives more than 2700 bytes

2) Is there a plan to increase TLS secure buffer in the future?

[Simon]

I talked to a colleague with more knowledge about this and gained more insight

ronald-blackline said:

1) Confirmation that the modem (nrf_recvfrom) returns 0 bytes if it actually receives more than 2700 bytes

I can confirm this. However, this applies only for TLS (maybe DTLS). If you don't encrypt your data it will be fine. Or you can run the TLS stack on the application side, and configure as large buffers as you want.

ronald-blackline said:

2) Is there a plan to increase TLS secure buffer in the future?

The answer here is unfortunately no.

Best regards,

Simon