nRF Connect SDK v2.6.1 - https_client works for example.com but not other hosts such as google.com

[jchen2024]

With nRF7002DK and TF-M (nrf7002dk_nrf5340_cpuapp_ns.conf) configuration, the sample project works for example.com:

*** Booting nRF Connect SDK v3.5.99-ncs1-1 ***
HTTPS client sample started
Bringing network interface up
Provisioning certificate
Connecting to the network
[00:00:04.949,981] <inf> wifi_mgmt_ext: Connection requested
[00:00:09.183,044] <inf> net_dhcpv4: Received: 192.168.101.16
Network connectivity established and IP address assigned
Looking up example.com
Resolved 93.184.215.14 (AF_INET)
Connecting to example.com:443
Sent 61 bytes
Received 377 bytes

> HTTP/1.1 200 OK

Finished, closing socket.
Network connectivity lost
Disconnected from the network

However, replace the host with google.com and its associated CA:

[00:00:45.647,186] <inf> wifi_mgmt_ext: Connection requested
[00:00:49.878,845] <inf> net_dhcpv4: Received: 192.168.101.16
Network connectivity established and IP address assigned
Looking up google.com
Resolved 142.250.191.206 (AF_INET)
Connecting to google.com:443
connect() failed, err (22): Invalid argument
Network connectivity lost
Disconnected from the network

[MJJ]

From Ji Chen:

I set up a wifi hotspot on another PC and captured the traffic between the devkit and the hotspot to get more ideas on the tls handshake message exchanges. It seems that everything works till the point devkit had received the certificate. The devkit failed to process the certificate and reset the connection.

 

We also set up another endpoint or url with Azure. That failed at almost the same spot after receiving the certificate from the Azure host.

 

Is there a way to log more errors/debug messages in mbedtls?

 

Here are the error messages for Azure host:

 

[00:00:07.855,133] <inf> net_dhcpv4: Received: 192.168.137.214

[00:00:07.855,346] <inf> net_config: IPv4 address: 192.168.137.214

[00:00:07.855,346] <inf> net_config: Lease time: 604800 seconds

[00:00:07.855,377] <inf> net_config: Subnet: 255.255.255.0

[00:00:07.855,438] <inf> net_config: Router: 192.168.137.1

[00:00:07.855,651] <inf> https_client_v261: Network connectivity established and IP address assigned

[00:00:07.856,384] <inf> https_client_v261: Looking up <EDIT>

[00:00:08.274,078] <inf> https_client_v261: Resolved <EDIT> (AF_INET) and port: 443 and protocol 6

[00:00:08.274,139] <inf> https_client_v261: SAMPLE_TFM_MBEDTLS not enabled

[00:00:08.274,566] <inf> https_client_v261: Connecting to <EDIT>

[00:00:08.602,630] <err> net_sock_tls: TLS handshake error: -0x7080

[00:00:08.602,966] <err> https_client_v261: connect() failed, err: 113

[00:00:08.624,969] <err> net_tcp: conn: 0x20057984, new bytes 441 during TIME-WAIT state send

 

Or with the terminal:

[jchen2024]

Additional information on the TLS network traffic capture, example.com and www.google.com have smaller sizes (<4k) certificates while auth0 or azure have certificates larger than 4k bytes. Is there a way to find out if the buffer is too small in mbedtls to process the larger certificates. 

[Einar Thorsrud]

Hi,

I see. Can you try to increase these configs (starting from the top):

• CONFIG_MBEDTLS_SSL_MAX_FRAGMENT_LENGTH
• CONFIG_NET_SOCKETS_TLS_SET_MAX_FRAGMENT_LENGTH
• CONFIG_MBEDTLS_SSL_IN_CONTENT_LEN
• CONFIG_MBEDTLS_HEAP_SIZE

If that does not help, can you share a wireshark log?

[jchen2024]

I have already set those to max and very large. They didn't help. Is there an upper limit on the size of the socket receive buffer? I tried setsockopt for SOL_SOCKET with SO_RCVBUF and it didn't work for me. Does the kernel limit the size? I will set up an hotspot later today to capture the traffic.

[Einar Thorsrud]

Hi,

I see. The maximum value of NET_SOCKETPAIR_BUFFER_SIZE is 4096. This is not related to or limited by the kernel. I do not immediately see that it should matter, but if you want to, you could increase NET_SOCKETPAIR_BUFFER_SIZE beyound 4096 but then you need to modify the range in zephyr/subsys/net/lib/sockets/Kconfig.

[jchen2024]

I couldn't get the size of socket recv buffer with getsockopt either. By the way, the last Wireshark capture definitely showed the the host send a packet that is larger than 4096. After the the Nordic board just closed the socket.

[jchen2024]

I enabled the MbedTLS logging and force the lenght log as error. The log messages showed that the received message length was less than what the handshake header indicated 4642 vs 4096:

[00:00:07.373,260] <err> mbedtls: WEST_TOPDIR/modules/crypto/mbedtls/library/ssl_msg.c:3869: input record: msgtype = 22, version = [0x303], msglen = 98
[00:00:07.373,413] <err> mbedtls: WEST_TOPDIR/modules/crypto/mbedtls/library/ssl_msg.c:3242: handshake message: msglen = 98, type = 2, hslen = 98
[00:00:07.373,718] <err> mbedtls: WEST_TOPDIR/modules/crypto/mbedtls/library/ssl_msg.c:3869: input record: msgtype = 22, version = [0x303], msglen = 4096
[00:00:07.376,678] <err> mbedtls: WEST_TOPDIR/modules/crypto/mbedtls/library/ssl_msg.c:3869: input record: msgtype = 22, version = [0x303], msglen = 4096
[00:00:07.379,730] <err> mbedtls: WEST_TOPDIR/modules/crypto/mbedtls/library/ssl_msg.c:3869: input record: msgtype = 22, version = [0x303], msglen = 4096
[00:00:07.382,019] <err> mbedtls: WEST_TOPDIR/modules/crypto/mbedtls/library/ssl_msg.c:3869: input record: msgtype = 22, version = [0x303], msglen = 4096
[00:00:07.382,202] <err> mbedtls: WEST_TOPDIR/modules/crypto/mbedtls/library/ssl_msg.c:3242: handshake message: msglen = 4096, type = 11, hslen = 4642
[00:00:07.382,263] <err> mbedtls: WEST_TOPDIR/modules/crypto/mbedtls/library/ssl_msg.c:3308: TLS handshake fragmentation not supported
[00:00:07.382,354] <err> mbedtls: WEST_TOPDIR/modules/crypto/mbedtls/library/ssl_msg.c:4177: mbedtls_ssl_handle_message_type() returned -28800 (-0x7080)
[00:00:07.382,446] <err> mbedtls: WEST_TOPDIR/modules/crypto/mbedtls/library/ssl_tls.c:7536: mbedtls_ssl_read_record() returned -28800 (-0x7080)
connect() failed, err (113): Software caused connection abort
Network connectivity lost
Disconnected from the network

Google.com happened to have the largest hslen as 4010.

[Einar Thorsrud]

Hi,

Thank you for the log. We are continuin to look into this and will get back to you by the end of the week.

[Einar Thorsrud]

Hi,

Google is using a single certificate for all their domains, which means that they have to squeeze in over 450 DNS entries into the SAN (Subject Alternate Name) extension in the certificate. This is why their certificate is so big (over 4096 bytes).

The main issue is that zephyr/mbedtls restricts the maximum TLS fragment length to 4096 here and mbed TLS is not able to handle fragmented handshake messages, unfortunately (see this comment). So every message over 4096 bytes will be truncated.

If we simply skip the check and allow for any fragment size the TLS-handshake can succeed. This is done by adding the config:
CONFIG_NET_SOCKETS_TLS_SET_MAX_FRAGMENT_LENGTH=n

However, there may be unforseen onsequences of skippingthe check. So, it is probably better to create your own fragment length of 8192, which should be okay by looking at the recommendations here.

You also need to incrase a number of configs to make it work (you may not need to incrase the top three as much as here, though):

CONFIG_MBEDTLS_HEAP_SIZE=80000
CONFIG_MBEDTLS_SSL_IN_CONTENT_LEN=8192
CONFIG_MBEDTLS_SSL_OUT_CONTENT_LEN=8192
CONFIG_NET_SOCKETS_TLS_SET_MAX_FRAGMENT_LENGTH=n
CONFIG_PSA_WANT_RSA_KEY_SIZE_4096=y
CONFIG_MBEDTLS_MPI_MAX_SIZE=512

Br,

Einar

[jchen2024]

The existing configurations worked for www.google.com before because it seems that the fragment size was less than 4096. However, even with the suggested configuration, it still have troubles with the certificate fragment that is bigger than 4096, as in the case of auth0 or azure. When the size is not an issue for the link: www.auth0.com. I have this error from Mbedtls:
Looking up www.auth0.com
Resolved 104.17.254.182 (AF_INET)
Connecting to www.auth0.com:443
[00:00:06.605,346] <err> mbedtls: WEST_TOPDIR/modules/crypto/mbedtls/library/ssl_msg.c:3242: handshake message: msglen = 91, type = 2, hslen = 91
[00:00:06.610,656] <err> mbedtls: WEST_TOPDIR/modules/crypto/mbedtls/library/ssl_msg.c:3242: handshake message: msglen = 3479, type = 11, hslen = 3479
[00:00:06.614,685] <err> mbedtls: WEST_TOPDIR/modules/crypto/mbedtls/library/ssl_tls.c:7192: mbedtls_x509_crt_parse_der() returned -14848 (-0x3a00)

Do I need pick a particular cipher?

[Einar Thorsrud]

Hi,

Regardign long certificates, did you tern off max frament length negotiations using CONFIG_NET_SOCKETS_TLS_SET_MAX_FRAGMENT_LENGTH=n? If max-fragment-length negotiation is enabled every TLS handshake message over 4k will be fragmented, and re-assembly of TLS handshake message fragments is not supported in mbed TLS (see here).

Regardign the -0x3a00 error you get with auth0, that is MBEDTLS_ERR_PK_UNKNOWN_NAMED_CURVE. It seems auth0 requiers p256r1 (so you need CONFIG_PSA_WANT_ECC_SECP_R1_256=y) and also x25519 (CONFIG_PSA_WANT_ECC_MONTGOMERY_255=y) and one of aes-128 gcm, aes-256 gcm or chachapoly (see Driver configurations and supported features for enabling support for that, for instance
CONFIG_PSA_WANT_ALG_CHACHA20_POLY1305=y).

[jchen2024]

Added the following configuration items:

CONFIG_NET_SOCKETS_TLS_SET_MAX_FRAGMENT_LENGTH=n

CONFIG_PSA_WANT_ECC_SECP_R1_256=y

CONFIG_PSA_WANT_ECC_MONTGOMERY_255=y

The end results are the same:

Resolved 104.17.254.182 (AF_INET)
Connecting to www.auth0.com:443
[00:00:06.689,453] <err> mbedtls: WEST_TOPDIR/modules/crypto/mbedtls/library/ssl_msg.c:3242: handshake message: msglen = 91, type = 2, hslen = 91
[00:00:06.693,695] <err> mbedtls: WEST_TOPDIR/modules/crypto/mbedtls/library/ssl_msg.c:3242: handshake message: msglen = 3479, type = 11, hslen = 3479
[00:00:06.697,784] <err> mbedtls: WEST_TOPDIR/modules/crypto/mbedtls/library/ssl_tls.c:7192: mbedtls_x509_crt_parse_der() returned -14848 (-0x3a00)
connect() failed, err (113): Software caused connection abort
Network connectivity lost
Disconnected from the network

Do you have different responses?