Cannot connect to LwM2M server with X509 DTLS. Wireshark shows empty client certificate.

[Scytmo]

Hi,

I am developing a product based on the nRF9160 using the LwM2M client.  I successfully have this working using PSK, but want to use X509.

I am calling lwm2m_security_set_certificate() where I previously called lwm2m_security_set_psk() and have client certificate, client key, and server root CA certificate in PEM form created by following the guidance from dejans and SeppoTakalo in this other post:  LWM2M Client With X.509 Certificate 

The LwM2M library reports the following

[00:01:19.463,531] <err> net_lwm2m_engine: Cannot connect UDP (-111)
[00:01:19.474,121] <err> net_lwm2m_engine: lwm2m_engine_start lwm2m_socket_start() returned -111

...which is 'connection refused'

I have taken a modem lib trace of an attempt to connect to the leshan public server, and looking at this in Wireshark, the client DTLS handshake response to the server's 'Certificate Request' contains a zero length certificate, followed by the server returning a 'Bad Certificate' fatal error (which I'm assuming results in the -111 in the modem DTLS handling).

I have attached the device certificate that I'm using, and the modem lib trace.

I have confirmed with AT%CMNG=1 that the the sha256 checksum of the stored credentials match the sha256 sum of this certificate (the file I've attached).

I'd be grateful for someone to confirm my analysis of what's going on, and suggest why the modem is not including the content of the client certificate in the DTLS handshake.

Thanks

Ian

Fullscreen cert.txt Download

-----BEGIN CERTIFICATE-----
MIIB1jCCAXwCFE4B5ovn7jwh+YJpWqx6ToAcJu9DMAoGCCqGSM49BAMCMGkxCzAJ
BgNVBAYTAkdCMRIwEAYDVQQIDAlIYW1wc2hpcmUxFDASBgNVBAcMC1NvdXRoYW1w
dG9uMRQwEgYDVQQKDAtVdG9ub215IEx0ZDEaMBgGA1UEAwwRVXRvbm9teSBMdGQg
RUMgQ0EwIBcNMjQxMDI0MDgyMzAxWhgPMjEyNDA5MzAwODIzMDFaMHAxCzAJBgNV
BAYTAkdCMRIwEAYDVQQIDAlIYW1wc2hpcmUxFDASBgNVBAcMC1NvdXRoYW1wdG9u
MRQwEgYDVQQKDAtVdG9ub215IEx0ZDEhMB8GA1UEAwwYdXJuOmltZWk6MzUxOTAx
OTMwNjk5NzkyMFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEqej1or6Ukoqjyfj1
3rDTYg9aqyz6qaxTJkI8esY3zXnRDH+xqsn/7j33Y3pNN1QuIPj+3Ni9bpRxji1x
MbkfEjAKBggqhkjOPQQDAgNIADBFAiEA1Gf+wRIXsI2+1+rOHdyHup0OEHPgMUnz
GxBpnS/nNzgCIBlitcVFBkYFB/dLQlMWARJvIT7+CUFNfooTyW9fj6ZA
-----END CERTIFICATE-----

trace-2024-10-24T13-59-55.879Z.mtrace

[Achim Kraus]

Unfortunately, if you work with encryption, any small change may end up in a completely different behavior.

I don't know, if the procedure in the link of your first comment using openssl works, but I would recommend, that you really exactly test that, step by step.

If that doesn't work, you may post your PEMs and how you actually provide them to "lwm2m_security_set_certificate".

But I guess, it works. And that means, that any change in your process may cause the error. To locate the cause you will need to apply the changes one by one to see, where it fails. A lot of time, unfortunately.

But, if the procedure in the link works, no one else will have a chance to see, what you changed to make it failing.

[Scytmo]

Hi Achim,

I can't get X509 to work with LwM2M using the example in the link of my first comment, so I do not have a working configuration to experiment with - otherwise I would have already done the step-by-step investigation that you suggest (it's very frustrating that I don't have a working example).

I can reproduce my issue from first principles with the following precise steps:

Generate CA and client keys with the following:

openssl ecparam -genkey -name prime256v1 -out ca.key
openssl req -x509 -new -SHA256 -nodes -key ca.key -days 3650 -subj '/O=My Company/CN=My root CA/' -out ca.crt
openssl ecparam -genkey -name prime256v1 -out client.key
openssl req -new -key client.key -subj '/CN=urn:imei:351901930699792/' -out client.csr
openssl x509 -req -SHA256 -CA ca.crt -CAkey ca.key -CAcreateserial -in client.csr -out client.crt

Note that this isn't *exactly* as per the instructions in the article I linked to, because I get an error in this suggested line, which appears to be an attempt to create a certificate in a single line without going via a CSR.  So you'll see my commands above create a CSR then create the client certificate from that.  I'm not sure if these are equivalent?

(the following openssl line from the article I linked to does not work for me, as '-CA' and '-CAkey' are not recognised as options for this command)

openssl req -x509 -new -SHA256 -nodes -key client.key -CA ca.crt -CAkey ca.key -subj '/CN=urn:imei:<your_15_digit_imei_goes_here>/' -out client.crt

I can then build the Nordic `lwm2m_client` sample application from the 2.6.1 SDK, with the following changes/additions to the prj.conf:

CONFIG_LWM2M_SECURITY_KEY_SIZE=1500
CONFIG_NRF_MODEM_LIB_TRACE=y
CONFIG_APP_LWM2M_PSK=""

I get the server CA for the leshan public server using the following

openssl s_client -dtls -connect leshan.eclipseprojects.io:5684 -showcerts

And I modified main.c from line 275 to set X509 mode with these certs/keys, as follows:

	if (sizeof(CONFIG_APP_LWM2M_PSK) > 1) {
		/* Write hard-coded PSK key to engine */
		/* First security instance is the right one, because in bootstrap mode, */
		/* it is the bootstrap PSK. In normal mode, it is the server key */
		lwm2m_security_set_psk(0, CONFIG_APP_LWM2M_PSK, sizeof(CONFIG_APP_LWM2M_PSK), true,
				       endpoint_name);
	}
	else {
		const char cert[] =
			"-----BEGIN CERTIFICATE-----\n"
			"MIIBRzCB7gIUR/mT2EVlsjzqzxsSWSWyGVRRfV4wCgYIKoZIzj0EAwIwKjETMBEG\n"
			"A1UECgwKTXkgQ29tcGFueTETMBEGA1UEAwwKTXkgcm9vdCBDQTAeFw0yNDExMTIx\n"
			"NTEwMzJaFw0yNDEyMTIxNTEwMzJaMCMxITAfBgNVBAMMGHVybjppbWVpOjM1MTkw\n"
			"MTkzMDY5OTc5MjBZMBMGByqGSM49AgEGCCqGSM49AwEHA0IABEtcrseTtAeUv8lX\n"
			"krsZaJ9jyyArugFYxPiZD6FZX8kQBgIfmsJb4OWI0da9oEyLkCBPR94UrBWu1WVI\n"
			"WBro1i8wCgYIKoZIzj0EAwIDSAAwRQIhAN20dtXr8JVaQcYgbiecrUqjMx1ADPN/\n"
			"Ppwg7u6YSQGlAiBdwlO9+lMRYcTZ9rT8kACpvQjobe63Zjmwh9wPOtLnIQ==\n"
			"-----END CERTIFICATE-----\n";
		const char key[] =
			"-----BEGIN EC PRIVATE KEY-----\n"
			"MHcCAQEEIKAl7+dggEz+YswzwfutuuGisLhta/qGtqY7Gb4tJO1boAoGCCqGSM49\n"
			"AwEHoUQDQgAES1yux5O0B5S/yVeSuxlon2PLICu6AVjE+JkPoVlfyRAGAh+awlvg\n"
			"5YjR1r2gTIuQIE9H3hSsFa7VZUhYGujWLw==\n"
			"-----END EC PRIVATE KEY-----\n";
		const char CA[] =
			"-----BEGIN CERTIFICATE-----\n"
			"MIICKDCCAc+gAwIBAgIURnwIw3M0mK4G0+tMmUj1PhlQkVMwCgYIKoZIzj0EAwIw\n"
			"aTELMAkGA1UEBhMCQVUxEzARBgNVBAgMClNvbWUtU3RhdGUxITAfBgNVBAoMGElu\n"
			"dGVybmV0IFdpZGdpdHMgUHR5IEx0ZDEiMCAGA1UEAwwZbGVzaGFuLmVjbGlwc2Vw\n"
			"cm9qZWN0cy5pbzAgFw0yMTA3MDkxNDQ0MTRaGA8yMTIxMDYxNTE0NDQxNFowaTEL\n"
			"MAkGA1UEBhMCQVUxEzARBgNVBAgMClNvbWUtU3RhdGUxITAfBgNVBAoMGEludGVy\n"
			"bmV0IFdpZGdpdHMgUHR5IEx0ZDEiMCAGA1UEAwwZbGVzaGFuLmVjbGlwc2Vwcm9q\n"
			"ZWN0cy5pbzBZMBMGByqGSM49AgEGCCqGSM49AwEHA0IABErD49ZejGF2dVpP8i1x\n"
			"s85aUHJ2PE3XU6nSOtkQ6vZzD1AxBmCiIiGdfsdcgBdVKzGzipIBmn6P2FJf0q5a\n"
			"HtCjUzBRMB0GA1UdDgQWBBTssKYTMXjSRAk/MPTZZal1nxA6fjAfBgNVHSMEGDAW\n"
			"gBTssKYTMXjSRAk/MPTZZal1nxA6fjAPBgNVHRMBAf8EBTADAQH/MAoGCCqGSM49\n"
			"BAMCA0cAMEQCIG7iKVckdsMu9nQQXcvblwOqGXRmO9xxpZiZnGb0oICLAiAWKrzr\n"
			"mac0tYvCDlAfARHDC7mFfKiLbUSIBgroovsH3g==\n"
			"-----END CERTIFICATE-----\n";


			lwm2m_security_set_certificate(0, cert, sizeof(cert), key, sizeof(key), CA, sizeof(CA));
	}

If I use the Cellular Monitor to capture a trace a load it into Wireshark, I get the same symptoms that I described earlier (empty certificate sent from the client).

[Achim Kraus]

> I can't get X509 to work with LwM2M using the example in the link of my first comment,

Thanks! I hope the details you provided helps Nordic to check and see the issue.

In my experience (no lwm2m) using "AT%CMNG" via "modem_key_mgmt_write" works well with x509.

Do you have a log with CONFIG_LWM2M_CLIENT_UTILS_LOG_LEVEL on DEBUG?

[Achim Kraus]

short:

CONFIG_LWM2M_SECURITY_KEY_SIZE=1500
CONFIG_NRF_MODEM_LIB_TRACE=y
CONFIG_APP_LWM2M_PSK=""
CONFIG_LWM2M_CLIENT_UTILS_SERVER_TLS_TAG=16842753

does it on "my machine".

long:

With your detailed description it's fast and easy to reproduce the failure. To exclude the credentials, I've tested them with my own coap-client against leshan, works. Exchanging the CA causes my client to abort the handshake without client certificate. Changing the CA in the lwm2m sample didn't change the handshake. enable logging showed, that sec_tag 35724861 is used to write the credentials and to configure to the socket ....

However, the cellular monitor uses 16842753, and so I guess, that somewhere the socket config of lwm2m_engine is overwritten. That's it. Use 16842753 as sec_tag, and it starts working.

[Scytmo]

Yes, that works!  Thank you so much, Achim - I really appreciate your help :-)

I'll also try to work out why there's a discrepancy in what tag is used, and I'll raise a separate ticket if/when I find the root cause for that.