Cannot connect to LwM2M server with X509 DTLS. Wireshark shows empty client certificate.

RE: Cannot connect to LwM2M server with X509 DTLS. Wireshark shows empty client certificate.

Achim Kraus — Tue, 19 Nov 2024 13:19:54 GMT

Thanks a lot for let us know!

> hope that it gets fixed in future releases

cross the fingers.

RE: Cannot connect to LwM2M server with X509 DTLS. Wireshark shows empty client certificate.

SeppoTakalo — Tue, 19 Nov 2024 13:14:23 GMT

No, I don't think there is any such documentation.

This is something I just discovered recently, and just send a PR to nRF SDK https://github.com/nrfconnect/sdk-nrf/pull/18892

For nRF91 modem, the release notes only lists limitations of the TLS stack. The AT command manual only gives instructions regarding the use of AT, but does not explain any details of TLS stack. However, this has now been reported internally to modem team, so I hope that it gets fixed in future releases and for current releases it needs documentation update.

RE: Cannot connect to LwM2M server with X509 DTLS. Wireshark shows empty client certificate.

Achim Kraus — Tue, 19 Nov 2024 10:57:09 GMT

That's a really nasty pitfall.

Is there some documentation about that preference of PSK credentials even when other cipher suites are negotiated?

I would like to link a own source snipped removing the PSK credentials in that case with such a source.

Otherwise I would refer to this ticket.

RE: Cannot connect to LwM2M server with X509 DTLS. Wireshark shows empty client certificate.

Achim Kraus — Tue, 19 Nov 2024 10:47:26 GMT

Thanks! Surprising effect.

RE: Cannot connect to LwM2M server with X509 DTLS. Wireshark shows empty client certificate.

Scytmo — Tue, 19 Nov 2024 10:33:53 GMT

Aha! Thank you very much! This also explains why using a different security tag appeared to solve the problem - as I only have X509 credentials in that security tag.

RE: Cannot connect to LwM2M server with X509 DTLS. Wireshark shows empty client certificate.

SeppoTakalo — Tue, 19 Nov 2024 10:29:17 GMT

Hi,

I would like to comment here that problems described here are not caused by LwM2M engine itself, nor does it mean that X509 would not work.

What happens is as follows:

nRF91 modem stores security credential in placeholders called "security tags". You can store PSK credentials, private keys or X509 client or CA certificates on security tags. See AT command manual for CMNG command for details. These credential are stored before connection starts.
When given security tag contains both client certificate and PSK credentials the TLS stack inside modem tries to use PSK.
If you set a ciphersuite for TLS using `zsock_setsockopt(fd, SOL_TLS, TLS_CIPHERSUITE_LIST, ...)` this information is fed into TLS handshake, but does not affect the selection of PSK/X509 from given security tag. This ciphersuite socket option is used inside LwM2M engine to switch between X509 of PSK mode when connecting.
When LWM2M engine tries to connect in X509 mode, but sectag has both, modem runs into conflict as selected ciphersuites don't allow PSK, so it send empty certificate.

To fix the issue:

Remove all credentials from security tag before you write new ones, if new ones are different type.

RE: Cannot connect to LwM2M server with X509 DTLS. Wireshark shows empty client certificate.

Achim Kraus — Thu, 14 Nov 2024 09:19:21 GMT

You're welcome.

As developer of Eclipse/Californium I usually try to check, if there is some issue with that. So far my experience with the nRF9160/DTLS/x509 is OK.

> why there's a discrepancy

I guess, that's more a job for Nordic to sync their nrf samples with the zephyr samples ;-).

RE: Cannot connect to LwM2M server with X509 DTLS. Wireshark shows empty client certificate.

Scytmo — Thu, 14 Nov 2024 09:12:30 GMT

Yes, that works! Thank you so much, Achim - I really appreciate your help :-)

I'll also try to work out why there's a discrepancy in what tag is used, and I'll raise a separate ticket if/when I find the root cause for that.

RE: Cannot connect to LwM2M server with X509 DTLS. Wireshark shows empty client certificate.

Achim Kraus — Wed, 13 Nov 2024 08:53:42 GMT

short:

CONFIG_LWM2M_SECURITY_KEY_SIZE=1500
CONFIG_NRF_MODEM_LIB_TRACE=y
CONFIG_APP_LWM2M_PSK=""
CONFIG_LWM2M_CLIENT_UTILS_SERVER_TLS_TAG=16842753

does it on "my machine".

long:

With your detailed description it's fast and easy to reproduce the failure. To exclude the credentials, I've tested them with my own coap-client against leshan, works. Exchanging the CA causes my client to abort the handshake without client certificate. Changing the CA in the lwm2m sample didn't change the handshake. enable logging showed, that sec_tag 35724861 is used to write the credentials and to configure to the socket ....

However, the cellular monitor uses 16842753, and so I guess, that somewhere the socket config of lwm2m_engine is overwritten. That's it. Use 16842753 as sec_tag, and it starts working.

RE: Cannot connect to LwM2M server with X509 DTLS. Wireshark shows empty client certificate.

Achim Kraus — Tue, 12 Nov 2024 18:53:50 GMT

> I can't get X509 to work with LwM2M using the example in the link of my first comment,

Thanks! I hope the details you provided helps Nordic to check and see the issue.

In my experience (no lwm2m) using "AT%CMNG" via "modem_key_mgmt_write" works well with x509.

Do you have a log with CONFIG_LWM2M_CLIENT_UTILS_LOG_LEVEL on DEBUG?

RE: Cannot connect to LwM2M server with X509 DTLS. Wireshark shows empty client certificate.

Scytmo — Tue, 12 Nov 2024 16:24:34 GMT

Hi Achim,

I can't get X509 to work with LwM2M using the example in the link of my first comment, so I do not have a working configuration to experiment with - otherwise I would have already done the step-by-step investigation that you suggest (it's very frustrating that I don't have a working example).

I can reproduce my issue from first principles with the following precise steps:

Generate CA and client keys with the following:

openssl ecparam -genkey -name prime256v1 -out ca.key
openssl req -x509 -new -SHA256 -nodes -key ca.key -days 3650 -subj '/O=My Company/CN=My root CA/' -out ca.crt
openssl ecparam -genkey -name prime256v1 -out client.key
openssl req -new -key client.key -subj '/CN=urn:imei:351901930699792/' -out client.csr
openssl x509 -req -SHA256 -CA ca.crt -CAkey ca.key -CAcreateserial -in client.csr -out client.crt

Note that this isn't *exactly* as per the instructions in the article I linked to, because I get an error in this suggested line, which appears to be an attempt to create a certificate in a single line without going via a CSR. So you'll see my commands above create a CSR then create the client certificate from that. I'm not sure if these are equivalent?

(the following openssl line from the article I linked to does not work for me, as '-CA' and '-CAkey' are not recognised as options for this command)

openssl req -x509 -new -SHA256 -nodes -key client.key -CA ca.crt -CAkey ca.key -subj '/CN=urn:imei:<your_15_digit_imei_goes_here>/' -out client.crt

I can then build the Nordic `lwm2m_client` sample application from the 2.6.1 SDK, with the following changes/additions to the prj.conf:

CONFIG_LWM2M_SECURITY_KEY_SIZE=1500
CONFIG_NRF_MODEM_LIB_TRACE=y
CONFIG_APP_LWM2M_PSK=""

I get the server CA for the leshan public server using the following

openssl s_client -dtls -connect leshan.eclipseprojects.io:5684 -showcerts

And I modified main.c from line 275 to set X509 mode with these certs/keys, as follows:

	if (sizeof(CONFIG_APP_LWM2M_PSK) > 1) {
		/* Write hard-coded PSK key to engine */
		/* First security instance is the right one, because in bootstrap mode, */
		/* it is the bootstrap PSK. In normal mode, it is the server key */
		lwm2m_security_set_psk(0, CONFIG_APP_LWM2M_PSK, sizeof(CONFIG_APP_LWM2M_PSK), true,
				       endpoint_name);
	}
	else {
		const char cert[] =
			"-----BEGIN CERTIFICATE-----\n"
			"MIIBRzCB7gIUR/mT2EVlsjzqzxsSWSWyGVRRfV4wCgYIKoZIzj0EAwIwKjETMBEG\n"
			"A1UECgwKTXkgQ29tcGFueTETMBEGA1UEAwwKTXkgcm9vdCBDQTAeFw0yNDExMTIx\n"
			"NTEwMzJaFw0yNDEyMTIxNTEwMzJaMCMxITAfBgNVBAMMGHVybjppbWVpOjM1MTkw\n"
			"MTkzMDY5OTc5MjBZMBMGByqGSM49AgEGCCqGSM49AwEHA0IABEtcrseTtAeUv8lX\n"
			"krsZaJ9jyyArugFYxPiZD6FZX8kQBgIfmsJb4OWI0da9oEyLkCBPR94UrBWu1WVI\n"
			"WBro1i8wCgYIKoZIzj0EAwIDSAAwRQIhAN20dtXr8JVaQcYgbiecrUqjMx1ADPN/\n"
			"Ppwg7u6YSQGlAiBdwlO9+lMRYcTZ9rT8kACpvQjobe63Zjmwh9wPOtLnIQ==\n"
			"-----END CERTIFICATE-----\n";
		const char key[] =
			"-----BEGIN EC PRIVATE KEY-----\n"
			"MHcCAQEEIKAl7+dggEz+YswzwfutuuGisLhta/qGtqY7Gb4tJO1boAoGCCqGSM49\n"
			"AwEHoUQDQgAES1yux5O0B5S/yVeSuxlon2PLICu6AVjE+JkPoVlfyRAGAh+awlvg\n"
			"5YjR1r2gTIuQIE9H3hSsFa7VZUhYGujWLw==\n"
			"-----END EC PRIVATE KEY-----\n";
		const char CA[] =
			"-----BEGIN CERTIFICATE-----\n"
			"MIICKDCCAc+gAwIBAgIURnwIw3M0mK4G0+tMmUj1PhlQkVMwCgYIKoZIzj0EAwIw\n"
			"aTELMAkGA1UEBhMCQVUxEzARBgNVBAgMClNvbWUtU3RhdGUxITAfBgNVBAoMGElu\n"
			"dGVybmV0IFdpZGdpdHMgUHR5IEx0ZDEiMCAGA1UEAwwZbGVzaGFuLmVjbGlwc2Vw\n"
			"cm9qZWN0cy5pbzAgFw0yMTA3MDkxNDQ0MTRaGA8yMTIxMDYxNTE0NDQxNFowaTEL\n"
			"MAkGA1UEBhMCQVUxEzARBgNVBAgMClNvbWUtU3RhdGUxITAfBgNVBAoMGEludGVy\n"
			"bmV0IFdpZGdpdHMgUHR5IEx0ZDEiMCAGA1UEAwwZbGVzaGFuLmVjbGlwc2Vwcm9q\n"
			"ZWN0cy5pbzBZMBMGByqGSM49AgEGCCqGSM49AwEHA0IABErD49ZejGF2dVpP8i1x\n"
			"s85aUHJ2PE3XU6nSOtkQ6vZzD1AxBmCiIiGdfsdcgBdVKzGzipIBmn6P2FJf0q5a\n"
			"HtCjUzBRMB0GA1UdDgQWBBTssKYTMXjSRAk/MPTZZal1nxA6fjAfBgNVHSMEGDAW\n"
			"gBTssKYTMXjSRAk/MPTZZal1nxA6fjAPBgNVHRMBAf8EBTADAQH/MAoGCCqGSM49\n"
			"BAMCA0cAMEQCIG7iKVckdsMu9nQQXcvblwOqGXRmO9xxpZiZnGb0oICLAiAWKrzr\n"
			"mac0tYvCDlAfARHDC7mFfKiLbUSIBgroovsH3g==\n"
			"-----END CERTIFICATE-----\n";


			lwm2m_security_set_certificate(0, cert, sizeof(cert), key, sizeof(key), CA, sizeof(CA));
	}

If I use the Cellular Monitor to capture a trace a load it into Wireshark, I get the same symptoms that I described earlier (empty certificate sent from the client).

RE: Cannot connect to LwM2M server with X509 DTLS. Wireshark shows empty client certificate.

Achim Kraus — Tue, 12 Nov 2024 13:37:07 GMT

Unfortunately, if you work with encryption, any small change may end up in a completely different behavior.

I don't know, if the procedure in the link of your first comment using openssl works, but I would recommend, that you really exactly test that, step by step.

If that doesn't work, you may post your PEMs and how you actually provide them to "lwm2m_security_set_certificate".

But I guess, it works. And that means, that any change in your process may cause the error. To locate the cause you will need to apply the changes one by one to see, where it fails. A lot of time, unfortunately.

But, if the procedure in the link works, no one else will have a chance to see, what you changed to make it failing.

RE: Cannot connect to LwM2M server with X509 DTLS. Wireshark shows empty client certificate.

Scytmo — Tue, 12 Nov 2024 09:30:40 GMT

Hi Michal,

Thanks very much for looking at this. I am passing the certificates as char arrays into lwm2m_security_set_certificate(). I create the char arrays using the `xxd` utility (`xxd -i`) from the PEM format files (the `cert.txt` attached to this ticket). I have also checked in the output from AT%CMNG=1 that the sha256 matches that of my PEM file.

RE: Cannot connect to LwM2M server with X509 DTLS. Wireshark shows empty client certificate.

Michal — Mon, 11 Nov 2024 19:33:55 GMT

I had to extend my leave until the end of last week unfortunately.

How exactly is the certificate passed to the functions?

RE: Cannot connect to LwM2M server with X509 DTLS. Wireshark shows empty client certificate.

Michal — Mon, 04 Nov 2024 16:09:21 GMT

Sorry for the delay, I was out of office.

I will try to do more debugging on it tomorrow.

RE: Cannot connect to LwM2M server with X509 DTLS. Wireshark shows empty client certificate.

Achim Kraus — Mon, 28 Oct 2024 13:22:40 GMT

OK. I guess, then you need to wait for Michal and Nordic's analysis.

RE: Cannot connect to LwM2M server with X509 DTLS. Wireshark shows empty client certificate.

Scytmo — Mon, 28 Oct 2024 13:20:27 GMT

My apologies, I misunderstood previously.

OK, I've now extracted the private key as you've described (and checked it with the ASN.1 decoder), and load that as the device key (and I've double checked that the modem reports the matching sha256 result for that file) but unfortunately I still get the same result of an empty client certificate.

Once more, thanks for your reply and suggestion - it's much appreciated.

RE: Cannot connect to LwM2M server with X509 DTLS. Wireshark shows empty client certificate.

Achim Kraus — Mon, 28 Oct 2024 13:00:45 GMT

It's not about the "EC PARAMS", it's about the content of "EC PRIVATE KEY". Copy the base 64 part to the ASN.1 decoder (address above) and you will see, if it contains the public key as well. And so you may test, if using only the private key works, using the "openssl ec -no_public -in client.key" to extract that. for my test key-pair the output will be:

-----BEGIN EC PRIVATE KEY-----
MDECAQEEIBh0Zh5RqJf49jVJhYRL6MJsIb2+RlmGGUSobg1jg2iIoAoGCCqGSM49
AwEH
-----END EC PRIVATE KEY-----

RE: Cannot connect to LwM2M server with X509 DTLS. Wireshark shows empty client certificate.

Scytmo — Mon, 28 Oct 2024 12:49:02 GMT

Thanks very much for your reply and your suggestion Achim. As it happens, my certifcate PEM didn't include the EC PARAMS. However, I've now tried it with the EC PARAMS included and I get the same behaviour.

Also I note that the message from SeppoTakalo in the post I referenced includes the following code snippet, which doesn't appear to include the EC PARAMS in the key.

	static const char key[] =
		"-----BEGIN EC PRIVATE KEY-----\n"
		"...clip..."
		"-----END EC PRIVATE KEY-----";

Thanks again for your reply and your suggestion, though.

RE: Cannot connect to LwM2M server with X509 DTLS. Wireshark shows empty client certificate.

Achim Kraus — Sun, 27 Oct 2024 09:13:46 GMT

Such an empty client certificate is commonly used, if the client is not able to use a certificate with private key matching the parameters in the Certificate Request. But from a first check, the certificate should match, so there may be something else prevent the client from using it, maybe an issue with the private key.

> I am calling lwm2m_security_set_certificate() where I previously called lwm2m_security_set_psk() and have client certificate, client key, and server root CA certificate in PEM form

I'm not sure, if the "client key" in PEM is really the intended one for the lwm2m_security_set_certificate. If you follow dejans openssl instructions, the PEM is rather a key pair than a "private key", I get:

and if you pass the to ASN.1 decoder you see a a Octet String with 32 bytes (that's the private key), and a BIT String with 520 bit (that's the corresponding public key).

Dejan refer also to use the Cellular Monitor, maybe that handle this. If lwm2m_security_set_certificate is able to handle this, isn't known by me.

If you want to check, if that key-pair instead of a pure private key is your issue, you may use


openssl ec -no_public -in client.key

to extract the private key only (but still with algorithm identifier). Maybe someone from Nordic points you to the documentation, what's precisely expected there as parameter.

RE: Cannot connect to LwM2M server with X509 DTLS. Wireshark shows empty client certificate.

Michal — Fri, 25 Oct 2024 19:22:44 GMT

Thank you for the information,

I will look more into it next week.

Best regards,

Michal

RE: Cannot connect to LwM2M server with X509 DTLS. Wireshark shows empty client certificate.

Scytmo — Thu, 24 Oct 2024 15:13:50 GMT

Just to clarify, this is with SDK 2.6.1 and modem firmware 1.3.6