HTTPS without T-FM and PSA

Question

Hello,

We are currently developing a WiFi library and just started on implementing HTTPS and ran into some issues. Since we don't have enough flash and memory for using the T-FM image, we decided to work on the secure build (without *ns).

As such we tried to get the exercise 'wififund_less5_exer2' running without PSA and T-FM, and using the CONFIG_WIFI_CREDENTIALS_BACKEND_SETTINGS as backend.

Unfortunately we always get the same error when calling connect():

[00:00:35.491,699] <inf> Lesson5_Exercise2: Network connected
[00:00:36.076,354] <inf> Lesson5_Exercise2: IPv4 address of HTTP server found 18.238.243.52
[00:00:36.100,402] <err> Lesson5_Exercise2: Connecting to server failed, err: 22, Invalid argument
[00:00:36.100,402] <err> Lesson5_Exercise2: Failed to initialize client

Which is exactly the same problem we're getting in our library.

Is it somehow possible to get HTTPS working without T-FM and PSA, using a different crypto? Or is it limited by the hardware/kernel?

Best regards,
Daniel Figueira

Daniel Figueira · Accepted Answer

After a lot of trial and error I finally got HTTPS working: 
 
 Over WiFi: using the nRF7002DK (both with and without TFM - meaning the secure and the non-secure builds). 
 Over LTE: using the nRF9161DK. 
 Over Ethernet: using the nRF9161DK with an ethernet port extension. 
 
 Apart from these I also got it working on our custom WiFi and LTE/Ethernet boards, which are based on the 2 development kits just mentioned. To wrap things up I'll post a description of what I had to do: 1) In the first place I had to make sure all the certificates I imported are NULL terminated, otherwise I would get an error when importing them (already mentioned in a previous post). 
 
 2) In the second place I had to make sure the code which imports the certificates (which is different between LTE and Wifi/Ethernet) was correct and was being called before the network connection was established. #if defined(USE_LTE)
 bool present_in_modem = false;

// Check if a certificate with the same tag is already present in the modem
 // (since the certificates added to the modem are persistent).
 if (!present_in_modem) {
 ret = modem_key_mgmt_exists(tag, MODEM_KEY_MGMT_CRED_TYPE_CA_CHAIN, &present_in_modem);
 if (ret != 0) {
 LOG_ERR("Failed to check if TLS certificate with tag %d is already present "
 "on the modem (err: %d)",
 tag, ret);
 return ret;
 }
 }

// Delete the old certificate (if different from the new)
 if (present_in_modem) {
 ret = modem_key_mgmt_cmp(tag, MODEM_KEY_MGMT_CRED_TYPE_CA_CHAIN, certificate, certificate_size);
 if (ret < 0) {
 LOG_ERR("Failed to compare new TLS certificate with the one already"
 "present on the modem (err: %d)",
 ret);
 return ret;
 }
 // If the certificates do not match
 if (ret == 1) {
 ret = modem_key_mgmt_delete(tag, MODEM_KEY_MGMT_CRED_TYPE_CA_CHAIN);
 if (ret < 0) {
 LOG_ERR("Failed to delete existing TLS certificate (err: %d)", ret);
 return ret;
 }
 present_in_modem = false;
 }
 }

// Add the new certificate (if not already present)
 if (!present_in_modem) {
 ret = modem_key_mgmt_write(tag, MODEM_KEY_MGMT_CRED_TYPE_CA_CHAIN, certificate, certificate_size);
 if (ret < 0) {
 LOG_ERR("Failed to add TLS certificate (err: %d)", ret);
 return ret;
 }
 }
#endif // USE_LTE

#if defined(USE_ETHERNET) || defined(USE_WIFI)
 // Try to add the new certificate
 ret = tls_credential_add(tag, TLS_CREDENTIAL_CA_CERTIFICATE, certificate, certificate_size);
 if (ret < 0 && ret != -EEXIST) {
 LOG_ERR("Failed to add TLS certificate (err: %d)", ret);
 return ret;
 }

// If a certificate with the same tag already was present delete it and try again
 // (since we don't have a way to verify if the certificate is the same)
 if (ret == -EEXIST) {
 ret = tls_credential_delete(tag, TLS_CREDENTIAL_CA_CERTIFICATE);
 if (ret < 0) {
 LOG_ERR("Failed to delete existing TLS certificate (err: %d)", ret);
 return ret;
 }
 ret = tls_credential_add(tag, TLS_CREDENTIAL_CA_CERTIFICATE, certificate, certificate_size);
 if (ret < 0) {
 LOG_ERR("Failed to add TLS certificate (err: %d)", ret);
 return ret;
 }
 }
#endif // USE_ETHERNET || USE_WIFI 
 I took me a while to understand that the certificates stored in the modem are persistent and stay on the modem even when I flash my board (for the LTE version at least...). This, grouped with some nasty bug I introduced when comparing the certificates, gave me a lot of headaches and resulted in very weird behaviors when trying to import different sets of certificates at the same time. Also good to know is that the value 0 is not a valid TAG and will result in a very hard-to-relate error (like all certificate and socket connection errors...). 
 The tags used will then need to be added using setsockopt() when creating a TLS socket: // Open a new HTTPS socket
 connection.socket = socket(AF_INET, SOCK_STREAM, IPPROTO_TLS_1_2);
 if (connection.socket < 0) {
 LOG_ERR("Unable to create local HTTPS socket (err: %d, %s)", errno, strerror(errno));
 return -errno;
 }

// Set TLS credentials to be used
 ret = setsockopt(connection.socket, SOL_TLS, TLS_SEC_TAG_LIST, certificate_tags, certificate_count * sizeof(sec_tag_t));
 if (ret < 0) {
 LOG_ERR("Failed to set TLS certificate list (err: %d, %s)", errno, strerror(errno));
 disconnect();
 return -errno;
 } 
 One would expect connect() to now work, however at this point you will most likely run into one or more somewhat obscure error codes returned by the connect() method - leading us to point 3... 
 3) Make sure you're using the right zephyr configurations, based on the network type, zephyr build (secure vs. non-secure) and the type of certificates used. 
 For me this was the hardest step, since I had to rely mostly on trial and error to come up with the minimal set of configs I seem to need: config USE_HTTPS
 bool "Enable HTTPS requests"
 default n
 select USE_HTTP
 # LTE dependencies
 select MBEDTLS_CCM_C if USE_LTE
 select MBEDTLS_ECP_C if USE_LTE
 select MBEDTLS_TLS_LIBRARY if USE_LTE
 # WiFi dependencies
 select MBEDTLS_RSA_C if USE_WIFI
 select PSA_WANT_KEY_TYPE_RSA_PUBLIC_KEY if USE_WIFI && !BUILD_WITH_TFM
 select PSA_WANT_RSA_KEY_SIZE_4096 if USE_WIFI && !BUILD_WITH_TFM
 # Ethernet dependencies
 select MBEDTLS_RSA_C if USE_ETHERNET
 select PSA_WANT_KEY_TYPE_RSA_PUBLIC_KEY if USE_ETHERNET
 select PSA_WANT_RSA_KEY_SIZE_4096 if USE_ETHERNET

help
 Enable the use of HTTPS requests

# Configure the TLS heap size
# Note: needs to be adjusted depending on the number of certificates being 
# used at the same time. Seems to cause problems if smaller than 60000.
configdefault MBEDTLS_HEAP_SIZE
 default 81920 if USE_WIFI && !BUILD_WITH_TFM
 default 81920 if USE_ETHERNET

# Configure the MPI component buffers sizes (used for public key operations)
# Note: needs to be adjusted depending on the max RSA key size supported
configdefault MBEDTLS_MPI_MAX_SIZE
 default 512 if PSA_WANT_RSA_KEY_SIZE_4096
 default 256 if PSA_WANT_RSA_KEY_SIZE_2048

# Enable TLS level error logs for HTTPS operations
# TLS error code list: https://gist.github.com/erikcorry/b25bdcacf3e0086f8a2afb688420678e
config ENABLE_HTTPS_DEBUG_MODE
 bool "Enable TLS error logs"
 default n
 depends on USE_HTTPS
 select MBEDTLS_DEBUG
 select MBEDTLS_DEBUG_C
 select MBEDTLS_SSL_DEBUG_ALL
 help
 Enable TLS error logs during HTTPS operations

# Increase the MBEDTLS_DEBUG_LEVEL further if more details are needed
configdefault MBEDTLS_DEBUG_LEVEL
 default 1 if ENABLE_HTTPS_DEBUG_MODE 
 (note that these only contain the configs I had to add on top of the configs already present for normal HTTP...) 
 For these I had to take in consideration the types of certificates I was trying to use (namely the algorithm and the key size) and adjust my settings accordingly. 
 For google.com for example, I needed to enable the RSA algorithm and add support for 2048 byte sized keys (this is a picture of the Firefox web browser which I used to download the certificates I needed)... 
 At this point it really helped to enable the MBEDTLS error logs as suggested by Stefan Schmidt . 
 Then I needed to make sure I had enough space for the heap and the buffers that are used by the TLS libs - since I'm not using only a specific certificate but a list of certificates that might vary in number/type. 
 I was then able to successfully connect to different servers using the certificates I had, however there was still one step left I had to do... 4) Understanding how the Zephyr http_client layer works and the specifics of the servers I'm connecting to (which applies to both HTTP and HTTPS requests). At this point I was able to successfully send HTTP requests over TLS to the servers, however this didn't meant that the servers would accept the requests I was sending, or maybe some servers did and some others did not... So, after doing some debugging and comparing the requests being sent using Zephyr's http_client versus the raw HTTP requests sent by the HTTPS sample I ended up with some not-so-obvious rules when it comes to building HTTP requests using Zephyr's http_client, which seem to make the requests I'm sending compatible across different servers: 
 
 The http_request 'url' field should not be left as NULL. When sending a request to the root path of the server the url needs to be set to '\' instead. 
 The http_request 'port' field is better left unset. Passing a value to it will cause the port to be appended to the "Host:" header of the request that is automatically generated by the http_client - which will cause some servers to reject your requests. 
 The hostname should either be added to the http_request 'host' field or present in the 'Host:' header set through the http_request 'headers' field, but not both - since it will cause the "Host:" header to be duplicated which might cause problems on the server side. 
 The "Content-Length:" header should not be used since it is already automatically generated based on the http_request 'body_len' field (even if left unset/zeroed). 
 All headers manually added to the http_request 'headers' field should to be 
 terminated - otherwise the request will be malformed. 
 
 I was then finally able to send my HTTPS request to multiple servers, over different network types, with and without T-FM. 
 Hopefully this will be of use for those who face similar problems in the future... I'm open to any further comments you might have, otherwise feel free to close this topic. Best regards, Daniel

HTTPS without T-FM and PSA

Top Replies