Key Exchange issue

[Randall]

I'm communicating with my customers website through a certificate and key provided by an OAUTH process.  I'm posting data to the website successfully by adding those certs to my TLS Credentials

   err = tls_credential_add( TLS_SEC_TAG,

                             TLS_CREDENTIAL_SERVER_CERTIFICATE,

                             certPtr,

                             strlen(certPtr) + 1);

   if ( err ) LOG_ERR( "Error adding GET_CLOUD_CERT" );

   param_read( PARAM_REPO_CLOUD_KEY, keyPtr, CLOUD_KEY_LEN );

   err = tls_credential_add( TLS_SEC_TAG,

                             TLS_CREDENTIAL_PRIVATE_KEY,

                             keyPtr,

                             strlen(keyPtr) + 1);

   if ( err ) LOG_ERR( "Error adding GET_CLOUD_KEY" );

   err = tls_credential_add( TLS_SEC_TAG,

                             TLS_CREDENTIAL_CA_CERTIFICATE,

                             oauth_root_ca,

                             strlen(oauth_root_ca) + 1);

I'm posting this to their site device-dev.get-cloud.com

Another process that I need to implement is to renew the keys when they are close to expiration.  To do this I need to send a GET request to auth.get-cloud.com using the same cert and key pair.

This is failing on my device.  It does work if I use curl;

curl -vv --key key.pem --cert cert.pem https://auth.get-cloud.com/renew 

So I'm thinking that authentication with the cert/key is working but the key exchange is not.  I've tried enabling every exchange option that I can find;

CONFIG_NET_SOCKETS_SOCKOPT_TLS=y

# TLS configuration

CONFIG_MBEDTLS=y

CONFIG_MBEDTLS_BUILTIN=y

CONFIG_MBEDTLS_ENABLE_HEAP=y

CONFIG_MBEDTLS_HEAP_SIZE=60000

CONFIG_MBEDTLS_SSL_MAX_CONTENT_LEN=8096

CONFIG_MBEDTLS_PEM_CERTIFICATE_FORMAT=y

#CONFIG_MBEDTLS_USER_CONFIG_FILE="mbedtls_config.h"

CONFIG_MBEDTLS_ECP_ALL_ENABLED=y

CONFIG_MBEDTLS_ECP_C=y

CONFIG_MBEDTLS_ECDSA_C=y

CONFIG_MBEDTLS_ECP_DP_SECP384R1_ENABLED=y

CONFIG_MBEDTLS_ECP_DP_SECP521R1_ENABLED=y

CONFIG_MBEDTLS_ECP_DP_BP384R1_ENABLED=y

CONFIG_MBEDTLS_ECP_DP_BP512R1_ENABLED=y

CONFIG_MBEDTLS_HASH_SHA512_ENABLED=y

CONFIG_MBEDTLS_SERVER_NAME_INDICATION=y

CONFIG_MBEDTLS_DEBUG=n

CONFIG_MBEDTLS_ECDH_C=y

CONFIG_MBEDTLS_SHA1_C=y

CONFIG_MBEDTLS_KEY_EXCHANGE_ECDHE_ECDSA_ENABLED=y

CONFIG_MBEDTLS_KEY_EXCHANGE_ECDHE_RSA_ENABLED=y

CONFIG_MBEDTLS_KEY_EXCHANGE_DHE_PSK_ENABLED=y

CONFIG_MBEDTLS_KEY_EXCHANGE_DHE_RSA_ENABLED=y

CONFIG_MBEDTLS_KEY_EXCHANGE_ECDH_ECDSA_ENABLED=y

CONFIG_MBEDTLS_KEY_EXCHANGE_ECDH_RSA_ENABLED=y

CONFIG_MBEDTLS_KEY_EXCHANGE_ECDHE_PSK_ENABLED=y

CONFIG_MBEDTLS_KEY_EXCHANGE_PSK_ENABLED=y

CONFIG_MBEDTLS_KEY_EXCHANGE_RSA_ENABLED=y

CONFIG_MBEDTLS_KEY_EXCHANGE_RSA_PSK_ENABLED=y

But nothing works.  Some things seem to be out of my range because I'm using MBEDTLS_BUILTIN.  Some options seem to call for NRF_SECURITY or L2_OPENTHREAD.  I'm concerned that trying to move to any of that will cause problems with my existing setup.  I was hoping that you might be able to think of some things I can try.

[Sigurd Hellesvik]

Hi,

For us to help you with this in the best possible way, please list the following information:

• SDK Version
• Development Kit Version
• Which sample do you use?
• Do you use custom hardware or code?

Regards,
Sigurd Hellesvik

[Randall]

It's SDK 2.5.2 on custom hardware.  But we are using the Laird Pinnacle 100 module that has a 52840 paired with the Sierra Wireless modem.  But we are using the Zephyr builtin mbedTLS library.

The cloud provider dug into this some and has the following comments;

-------------------------------------------------------------------------------------------------------

For some reason Azure don't send a certificate request when using a App Service hosted in docker in Azure. Our other endpoints is hosted using Azure function and the Microsoft Identity Provider intercept the call and do a rewrite of the header. That's why it works for device-dev.get-cloud.com and not auth.get-cloud.com.

 

The are two way of do a mTLS init connection.

1) The server ask for a certificate from the client (that's what Zephyr supports out of the box)

2) The client send the cert inside the handshake (that's what curl does).

------------------------------------------------------------------------------------------------

It seems to be related to an Azure issue;

https://github.com/MicrosoftDocs/azure-docs/issues/111230

Hopefully that sheds more light

[Randall]

The person that posted the issue on the Azure github seems to think it's a bug.  I guess Microsoft doesn't agree as they haven't done anything to address it.  So if there is a way to get Zephyr to offer up the certificate on it's own, that would be great.  If not, I may be stuck unless the cloud provider offers me an alternate way to renew keys.

Thanks,

[Sigurd Hellesvik]

I think first, it would be good to have a closer look into this, maybe we can get some new info.

Can you get a wireshark sniffer log from the error you get?

Do you get UART logs from the device when it fails?
Try to turn on mbedlts logging and see if you get more info then.

CONFIG_MBEDTLS_KEY_EXCHANGE_ECDHE_ECDSA_ENABLED=y

CONFIG_MBEDTLS_KEY_EXCHANGE_ECDHE_RSA_ENABLED=y

CONFIG_MBEDTLS_KEY_EXCHANGE_DHE_PSK_ENABLED=y

CONFIG_MBEDTLS_KEY_EXCHANGE_DHE_RSA_ENABLED=y

CONFIG_MBEDTLS_KEY_EXCHANGE_ECDH_ECDSA_ENABLED=y

CONFIG_MBEDTLS_KEY_EXCHANGE_ECDH_RSA_ENABLED=y

CONFIG_MBEDTLS_KEY_EXCHANGE_ECDHE_PSK_ENABLED=y

CONFIG_MBEDTLS_KEY_EXCHANGE_PSK_ENABLED=y

CONFIG_MBEDTLS_KEY_EXCHANGE_RSA_ENABLED=y

CONFIG_MBEDTLS_KEY_EXCHANGE_RSA_PSK_ENABLED=y

I see that you have enabled 10 options while CONFIG_MBEDTLS_KEY_EXCHANGE_ALL_ENABLED selects 11. Off-chance, but maybe you are missing the last one?

[Randall]

The only one missing is ECJPAKE.  Doubted that key juggling is something that Microsoft is doing but I tried it anyway.  And it still doesn't work.  I can't really do wireshark since I'm using a cell modem.  The cloud team did with a Linux-based Zephyr setup and it basically shows that the certificate isn't being sent to the authorization server.  I tried to enable MBEDTLS debugging, but the level keeps getting set to 0.  I tried to hard wire it with CONFIG_MBEDTLS_DEBUG_LEVEL but I didn't get any debug and looking it autoconf.h it's still set to 0.  So something in the build is hard wiring it to 0 and I don't know what it is.

[Sigurd Hellesvik]

Try CONFIG_MBEDTLS_DEBUG?

[Randall]

One has to set CONFIG_MBEDTLS_DEBUG to y before setting the debug level will not trigger an error.  So yes, I've already set that config to y.

Do you know anything about TLS 1.3 support in Zephyr?

[Sigurd Hellesvik]

Randall said:

Do you know anything about TLS 1.3 support in Zephyr?

You have triggered an automatic quote from my part:
I am not able to answer questions about our timeline. You can try to ask your local sales representative from Nordic Semiconductor for information about our timeline.

For adding the cert manually, here is some info that might help you do that:

"
I looked what happens when the TLS state machine enters the state MBEDTLS_SSL_CLIENT_CERTIFICATE
https://github.com/nrfconnect/sdk-mbedtls/blob/main/library/ssl_tls12_client.c#L3552

The mbedtls_ssl_write_certificate() function will skip sending client certificate if ssl->handshake->client_auth == 0

ssl->handshake->client_auth is only set if ServerHello contains a Certificate request here:
https://github.com/nrfconnect/sdk-mbedtls/blob/main/library/ssl_tls12_client.c#L2523

If the customer wants to hack a solution that always sends the client certificate without receiving a certificate request they can probably just set ssl->handshake->client_auth somewhere in the state machine.
"

[Randall]

Interesting suggestion.  I tried several variants of overriding the client_auth flag.  Unfortunately, it all resulted in a TLS handshake error. 

Looking at the above it also appears to be using NRF_SECURITY with MBEDTLS_LIBRARY.  I'm using MBEDTLS_BUILTIN.  I don't know if that makes any difference.

Randall

[Sigurd Hellesvik]

I think that TLS code is the same for MBEDTLS NRF_SECURITY and MBEDTLS_BUILTIN. It is the underlying crypto drivers that are changed.

However, I am not 100% on that, so I suggest that you print or step through code or something like that to verify that the file is really used

[Randall]

I did, and the changes did attempt to write the certificate despite not receiving a request.  But for some reason it still produced a TLS handshake error.

On the positive side, the cloud team has agreed to give me an alternate route to get certificate and key renewals.  So I guess this ticket can be closed out.

[Sigurd Hellesvik]

Randall said:

I did, and the changes did attempt to write the certificate despite not receiving a request.  But for some reason it still produced a TLS handshake error.

To quote myself from talking to our developers:

"At the very least if the hack work that would be a good indication to them being correct about what goes wrong"

And especially when you are not able to make it work like that, I start to suspect that the handshake request really is not the cause here after all.

However, if you got a workaround for key renewal, that fixes your issue, and that is that.
Good luck onwards!