ISRG Root X1 certificate fails to parse

RE: ISRG Root X1 certificate fails to parse

MarkoSagadin — Wed, 05 Feb 2025 13:48:28 GMT

Hello Jeremy!

Thank you very much for providing this answer. Adding CONFIG_MBEDTLS_MPI_MAX_SIZE=512 to my prj.conf made it possible to connect to a different public broker (broker.emqx.io).

Provide some context for anybody coming from the internet:

I had a similar issue on NCS v2.7.0 with the mqtt sample. I could connect via TLS to the sample's default mqtt broker (mosquitto) without a problem, however I couldn't connect to the broker.emqx.io.

The annoying thing about this was that I made a successfully connection with the emqx broker, a year ago, with the NCS v.2.5.2.

The logs themselves were pretty much not useful:

*** Booting nRF Connect SDK v2.7.0-5cb85570ca43 ***
*** Using Zephyr OS v3.6.99-100befc70c74 ***
[00:00:00.539,855] <inf> network: Bringing network interface up and connecting to the network
[00:00:00.540,374] <dbg> mqtt_helper: mqtt_state_set: State transition: MQTT_STATE_UNINIT --> MQTT_STATE_DISCONNECTED
[00:00:00.542,419] <dbg> mqtt_helper: mqtt_helper_poll_loop: Waiting for connection_poll_sem
[00:00:02.109,527] <inf> wifi_mgmt_ext: Connection requested
[00:00:06.183,624] <inf> net_dhcpv4: Received: 192.168.76.247
[00:00:06.183,898] <inf> network: Network connectivity established
[00:00:11.185,821] <dbg> mqtt_helper: broker_init: Resolving IP address for broker.emqx.io
[00:00:11.202,972] <err> net_dns_resolve: DNS recv error (-103)
[00:00:11.730,163] <dbg> mqtt_helper: broker_init: IPv4 Address found 34.243.217.54 (AF_INET)
[00:00:11.730,194] <dbg> mqtt_helper: certificates_provision: CA certificate already exists, sec tag: 955
[00:00:11.730,255] <dbg> mqtt_helper: mqtt_state_set: State transition: MQTT_STATE_DISCONNECTED --> MQTT_STATE_TRANSPORT_CONNECTING
[00:00:11.730,957] <dbg> net_mqtt_sock_tls: mqtt_client_tls_connect: (): Created socket 14
[00:00:11.732,666] <err> net_dns_resolve: DNS recv error (-4)
[00:00:12.072,814] <err> net_sock_tls: TLS handshake error: -0x3b00
[00:00:12.080,200] <err> mqtt_helper: mqtt_connect, error: -113
[00:00:12.080,261] <dbg> mqtt_helper: mqtt_state_set: State transition: MQTT_STATE_TRANSPORT_CONNECTING --> MQTT_STATE_DISCONNECTED
[00:00:12.080,261] <err> transport: Failed connecting to MQTT, error code: -113
uart:~$

Even when I enabled mbedtls logs (check this Devzone post on how to do that) I wouldn't get any hint to what was happening:

*** Booting nRF Connect SDK v2.7.0-5cb85570ca43 ***
*** Using Zephyr OS v3.6.99-100befc70c74 ***
[00:00:00.585,388] <inf> network: Bringing network interface up and connecting to the network
[00:00:00.585,876] <dbg> mqtt_helper: mqtt_state_set: State transition: MQTT_STATE_UNINIT --> MQTT_STATE_DISCONNECTED
[00:00:00.587,921] <dbg> mqtt_helper: mqtt_helper_poll_loop: Waiting for connection_poll_sem
[00:00:02.150,543] <inf> wifi_mgmt_ext: Connection requested
[00:00:06.226,562] <inf> net_dhcpv4: Received: 192.168.76.247
[00:00:06.226,867] <inf> network: Network connectivity established
--- 144 messages dropped ---
[00:00:11.963,592] <dbg> mbedtls: zephyr_mbedtls_debug: WEST_TOPDIR/modules/crypto/mbedtls/library/ssl_msg.c:3037: 0110:  97 f9 b2 62 3c ce aa 31 42 6f bc 00 0b 7a cd da  ...b<..1Bo...z..
[00:00:11.963,836] <dbg> mbedtls: zephyr_mbedtls_debug: WEST_TOPDIR/modules/crypto/mbedtls/library/ssl_msg.c:3037: 0120:  c2 72 ed 00 ba 25 c0 51 37 9b 62 5e ed 23 b2 b9  .r...%.Q7.b^.#..
[00:00:11.964,080] <dbg> mbedtls: zephyr_mbedtls_debug: WEST_TOPDIR/modules/crypto/mbedtls/library/ssl_msg.c:3037: 0130:  9f 09 c0 25 24 18 ef d2 52 41 ec 60 59 eb e4 d0  ...%$...RA.`Y...
--- 72 messages dropped ---
[00:00:11.964,324] <dbg> mbedtls: zephyr_mbedtls_debug: WEST_TOPDIR/modules/crypto/mbedtls/library/ssl_msg.c:3037: 0140:  ed 3d af a9 20 0d 9e ec 61 37 ce d1 a1 b1 4d 09  .=.. ...a7....M.
[00:00:11.964,569] <dbg> mbedtls: zephyr_mbedtls_debug: WEST_TOPDIR/modules/crypto/mbedtls/library/ssl_msg.c:3037: 0150:  d8 2f 30 9c c1 eb 31 fc ff 54 f6 d8 b0 66 ff 2f  ./0...1..T...f./
--- 33 messages dropped ---
[00:00:11.964,813] <dbg> mbedtls: zephyr_mbedtls_debug: WEST_TOPDIR/modules/crypto/mbedtls/library/ssl_msg.c:3037: 0160:  2c 11 aa 25 e1 41 04 98 7f e2 29 10 e8 32 aa 2f  ,..%.A....)..2./
[00:00:11.965,026] <dbg> mbedtls: zephyr_mbedtls_debug: WEST_TOPDIR/modules/crypto/mbedtls/library/ssl_msg.c:3037: 0170:  fc e7 94 54 62 fe bf dc cf bb 79 bc 66 fa b6 8d  ...Tb.....y.f...
--- 257 messages dropped ---
[00:00:12.033,752] <wrn> mbedtls: WEST_TOPDIR/modules/crypto/mbedtls/library/ssl_tls12_client.c:1662: <= parse server hello
[00:00:12.033,813] <wrn> mbedtls: WEST_TOPDIR/modules/crypto/mbedtls/library/ssl_msg.c:2358: => flush output
[00:00:12.041,015] <wrn> mbedtls: WEST_TOPDIR/modules/crypto/mbedtls/library/ssl_msg.c:2323: ssl->f_recv(_timeout)() returned 768 (-0xfffffd00)
[00:00:12.041,107] <wrn> mbedtls: WEST_TOPDIR/modules/crypto/mbedtls/library/ssl_msg.c:2320: in_left: 1439, nb_want: 3437
[00:00:12.041,198] <wrn> mbedtls: WEST_TOPDIR/modules/crypto/mbedtls/library/ssl_tls.c:3925: <= handshake
[00:00:12.090,881] <wrn> mbedtls: WEST_TOPDIR/modules/crypto/mbedtls/library/ssl_tls.c:3914: => handshake
[00:00:12.191,650] <dbg> mbedtls: zephyr_mbedtls_debug: WEST_TOPDIR/modules/crypto/mbedtls/library/ssl_msg.c:3976: 0cc0:  03 02 48 a8 e5 5e 9d 1d d7 b7 24 36 55 1f 36 aa  ..H..^....$6U.6.
[00:00:12.191,894] <dbg> mbedtls: zephyr_mbedtls_debug: WEST_TOPDIR/modules/crypto/mbedtls/library/ssl_msg.c:3976: 0cd0:  10 ba c6 c9 71 b4 d7 fb 7f 63 5d c7 61 bb 31 e9  ....q....c].a.1.
[00:00:12.192,138] <dbg> mbedtls: zephyr_mbedtls_debug: WEST_TOPDIR/modules/crypto/mbedtls/library/ssl_msg.c:3976: 0ce0:  b8 c2 91 61 c8 f0 d3 d8 fe 94 27 63 27 ac 3f 85  ...a......'c'.?.
[00:00:12.192,352] <dbg> mbedtls: zephyr_mbedtls_debug: WEST_TOPDIR/modules/crypto/mbedtls/library/ssl_msg.c:3976: 0cf0:  0b ff d9 28 7e 7f 11 1a 3d ea 08 73 f1 5a 8d 96  ...(~...=..s.Z..
[00:00:12.192,596] <dbg> mbedtls: zephyr_mbedtls_debug: WEST_TOPDIR/modules/crypto/mbedtls/library/ssl_msg.c:3976: 0d00:  62 f9 45 7a 3c 2a cf 6b 32 bf c0 77 dc 70 63 88  b.Ez<*.k2..w.pc.
[00:00:12.192,840] <dbg> mbedtls: zephyr_mbedtls_debug: WEST_TOPDIR/modules/crypto/mbedtls/library/ssl_msg.c:3976: 0d10:  72 46 f0 33 e7 dd b4 9b 25 1f 7f 07 54 a9 cd 12  rF.3....%...T...
[00:00:12.193,084] <dbg> mbedtls: zephyr_mbedtls_debug: WEST_TOPDIR/modules/crypto/mbedtls/library/ssl_msg.c:3976: 0d20:  bc f9 45 9d a8 6c 66 0d 79 b9 3e 47 90 ae 3c b4  ..E..lf.y.>G..<.
[00:00:12.193,328] <dbg> mbedtls: zephyr_mbedtls_debug: WEST_TOPDIR/modules/crypto/mbedtls/library/ssl_msg.c:3976: 0d30:  74 74 4c bb 8b 1f c6 91 a7 38 78 28 9f d8 a7 4b  ttL......8x(...K
[00:00:12.193,572] <dbg> mbedtls: zephyr_mbedtls_debug: WEST_TOPDIR/modules/crypto/mbedtls/library/ssl_msg.c:3976: 0d40:  00 44 d6 fe f6 2d 51 e0 58 39 cc f3 6f 1e cd 81  .D...-Q.X9..o...
[00:00:12.193,786] <dbg> mbedtls: zephyr_mbedtls_debug: WEST_TOPDIR/modules/crypto/mbedtls/library/ssl_msg.c:3976: 0d50:  6c 8b de d2 f9 30 c4 0c be 47 8e f6 ee a6 33 97  l....0...G....3.
[00:00:12.194,030] <dbg> mbedtls: zephyr_mbedtls_debug: WEST_TOPDIR/modules/crypto/mbedtls/library/ssl_msg.c:3976: 0d60:  7d 36 ef 0f 63 10 50 ba 1c c5 d1 68 37           }6..c.P....h7
[00:00:12.194,152] <inf> mbedtls: WEST_TOPDIR/modules/crypto/mbedtls/library/ssl_msg.c:3242: handshake message: msglen = 3432, type = 11, hslen = 3432
[00:00:12.201,965] <wrn> mbedtls: WEST_TOPDIR/modules/crypto/mbedtls/library/ssl_msg.c:4194: <= read record
[00:00:12.203,765] <wrn> mbedtls: WEST_TOPDIR/modules/crypto/mbedtls/library/ssl_msg.c:5103: => send alert message
[00:00:12.203,857] <inf> mbedtls: WEST_TOPDIR/modules/crypto/mbedtls/library/ssl_msg.c:5104: send alert level=2 message=42
[00:00:12.203,918] <wrn> mbedtls: WEST_TOPDIR/modules/crypto/mbedtls/library/ssl_msg.c:2948: => write record
[00:00:12.204,040] <inf> mbedtls: WEST_TOPDIR/modules/crypto/mbedtls/library/ssl_msg.c:3032: output record: msgtype = 21, version = [3:3], msglen = 2
[00:00:12.204,132] <dbg> mbedtls: zephyr_mbedtls_debug: WEST_TOPDIR/modules/crypto/mbedtls/library/ssl_msg.c:3037: dumping 'output record sent to network' (7 bytes)
[00:00:12.204,315] <dbg> mbedtls: zephyr_mbedtls_debug: WEST_TOPDIR/modules/crypto/mbedtls/library/ssl_msg.c:3037: 0000:  15 03 03 00 02 02 2a                             ......*
[00:00:12.204,376] <wrn> mbedtls: WEST_TOPDIR/modules/crypto/mbedtls/library/ssl_msg.c:2358: => flush output
[00:00:12.204,467] <wrn> mbedtls: WEST_TOPDIR/modules/crypto/mbedtls/library/ssl_msg.c:2372: message length: 7, out_left: 7
[00:00:12.204,864] <wrn> mbedtls: WEST_TOPDIR/modules/crypto/mbedtls/library/ssl_msg.c:2379: ssl->f_send() returned 7 (-0xfffffff9)
[00:00:12.204,925] <wrn> mbedtls: WEST_TOPDIR/modules/crypto/mbedtls/library/ssl_msg.c:2406: <= flush output
[00:00:12.204,986] <wrn> mbedtls: WEST_TOPDIR/modules/crypto/mbedtls/library/ssl_msg.c:3085: <= write record
[00:00:12.205,047] <wrn> mbedtls: WEST_TOPDIR/modules/crypto/mbedtls/library/ssl_msg.c:5115: <= send alert message
[00:00:12.205,139] <err> mbedtls: WEST_TOPDIR/modules/crypto/mbedtls/library/ssl_tls.c:7192:  mbedtls_x509_crt_parse_der() returned -15104 (-0x3b00)
[00:00:12.205,505] <wrn> mbedtls: WEST_TOPDIR/modules/crypto/mbedtls/library/ssl_tls.c:3925: <= handshake
[00:00:12.205,535] <err> net_sock_tls: TLS handshake error: -0x3b00
[00:00:12.209,228] <wrn> mbedtls: WEST_TOPDIR/modules/crypto/mbedtls/library/ssl_msg.c:5974: => write close notify
[00:00:12.209,289] <wrn> mbedtls: WEST_TOPDIR/modules/crypto/mbedtls/library/ssl_msg.c:5985: <= write close notify
[00:00:12.213,043] <err> mqtt_helper: mqtt_connect, error: -113
[00:00:12.213,104] <dbg> mqtt_helper: mqtt_state_set: State transition: MQTT_STATE_TRANSPORT_CONNECTING --> MQTT_STATE_DISCONNECTED
[00:00:12.213,104] <err> transport: Failed connecting to MQTT, error code: -113
uart:~$

So the der parser failed above. Running gdb through the mbedtls code to figue why is not fun...

So yes, for me the mentioned config worked. Thank you again!

RE: ISRG Root X1 certificate fails to parse

jeremyherbert — Tue, 26 Mar 2024 07:59:48 GMT

I ended up finding the problem - by default, mbedtls is built with MPI_MAX_SIZE=256 which means that the largest RSA key size supported is 2048 bits. Instead, you need to build with CONFIG_MBEDTLS_MPI_MAX_SIZE=512 to support 4096 bit keys. It would be helpful if there was a more useful error in this case.

I would suggest at least adding to the readme here that 4096 bit keys are not supported by default: https://github.com/nrfconnect/sdk-nrf/tree/main/samples/net/https_client - otherwise it is very non-obvious what needs doing. Can I also suggest adding some documentation somewhere about what the maximum RSA/ECDSA key sizes are too? There are a lot of places that mention that the certificate size can't be over 4kB, but the certificate in this case is well under this limit.

Note that the TLS offload system also only supports a maximum of 2048 bit keys, though I haven't found a place where this is documented (and neither for the maximum ECDSA key size).

In my case, the ISRG root certificate is 4096 bits so it can't be used with TLS offloading or the default mbedtls configuration. The solution with Let's Encrypt is to use an intermediate certificate as the CA certificate which is 2048 bits: https://letsencrypt.org/certificates/ - at this point in time, the "R3" certificate is 2048 bits so that is the one that should be used. However, the expiry time is much shorter than the root certificate (intermediate is 2025 vs 2035 for the root certificate) so that becomes a problem in itself.

Also please note that this unsolved issue may be related: devzone.nordicsemi.com/.../nrf9160-native-tls-with-offload-sockets-and-mqtt-client

RE: ISRG Root X1 certificate fails to parse

jeremyherbert — Mon, 25 Mar 2024 21:58:54 GMT

Just to add to this, I have built mbedtls on my host machine (mac, x86) and it is able to parse the certificate with no problems:

$ ./ssl_client2 server_name=letsencrypt.org server_port=443 ca_file=isrgrootx1.pem debug_level=1
build version: Mbed TLS 3.5.2 (build 50659840)

  . Seeding the random number generator... ok
  . Loading the CA root certificate ... ok (0 skipped)
  . Loading the client cert. and key... ok (key type: RSA)
  . Setting up the SSL/TLS structure... ok
  . Connecting to tcp/letsencrypt.org/443... ok
  . Performing the SSL/TLS handshake...
Verify requested for (Depth 2):
cert. version     : 3
serial number     : 82:10:CF:B0:D2:40:E3:59:44:63:E0:BB:63:82:8B:00
issuer name       : C=US, O=Internet Security Research Group, CN=ISRG Root X1
subject name      : C=US, O=Internet Security Research Group, CN=ISRG Root X1
issued  on        : 2015-06-04 11:04:38
expires on        : 2035-06-04 11:04:38
signed using      : RSA with SHA-256
RSA key size      : 4096 bits
basic constraints : CA=true
key usage         : Key Cert Sign, CRL Sign
  This certificate has no flags

Verify requested for (Depth 1):
cert. version     : 3
serial number     : 91:2B:08:4A:CF:0C:18:A7:53:F6:D6:2E:25:A7:5F:5A
issuer name       : C=US, O=Internet Security Research Group, CN=ISRG Root X1
subject name      : C=US, O=Let's Encrypt, CN=R3
issued  on        : 2020-09-04 00:00:00
expires on        : 2025-09-15 16:00:00
signed using      : RSA with SHA-256
RSA key size      : 2048 bits
basic constraints : CA=true, max_pathlen=0
key usage         : Digital Signature, Key Cert Sign, CRL Sign
ext key usage     : TLS Web Client Authentication, TLS Web Server Authentication
certificate policies : ???, ???
  This certificate has no flags

Verify requested for (Depth 0):
cert. version     : 3
serial number     : 04:A7:BD:BA:BE:69:67:7A:A5:DF:42:99:AB:F6:E7:F8:DD:3B
issuer name       : C=US, O=Let's Encrypt, CN=R3
subject name      : CN=lencr.org
issued  on        : 2024-01-27 21:30:22
expires on        : 2024-04-26 21:30:21
signed using      : RSA with SHA-256
EC key size       : 256 bits
basic constraints : CA=false
subject alt name  :
    dNSName : lencr.org
    dNSName : letsencrypt.com
    dNSName : letsencrypt.org
    dNSName : www.lencr.org
    dNSName : www.letsencrypt.com
    dNSName : www.letsencrypt.org
key usage         : Digital Signature
ext key usage     : TLS Web Server Authentication, TLS Web Client Authentication
certificate policies : ???
  This certificate has no flags
 ok
    [ Protocol is TLSv1.2 ]
    [ Ciphersuite is TLS-ECDHE-ECDSA-WITH-CHACHA20-POLY1305-SHA256 ]
    [ Key size is 256 ]
    [ Record expansion is 21 ]
    [ Maximum incoming record payload length is 16384 ]
    [ Maximum outgoing record payload length is 16384 ]
  . Verifying peer X.509 certificate... ok
  . Peer certificate information    ...
cert. version     : 3
serial number     : 04:A7:BD:BA:BE:69:67:7A:A5:DF:42:99:AB:F6:E7:F8:DD:3B
issuer name       : C=US, O=Let's Encrypt, CN=R3
subject name      : CN=lencr.org
issued  on        : 2024-01-27 21:30:22
expires on        : 2024-04-26 21:30:21
signed using      : RSA with SHA-256
EC key size       : 256 bits
basic constraints : CA=false
subject alt name  :
    dNSName : lencr.org
    dNSName : letsencrypt.com
    dNSName : letsencrypt.org
    dNSName : www.lencr.org
    dNSName : www.letsencrypt.com
    dNSName : www.letsencrypt.org
key usage         : Digital Signature
ext key usage     : TLS Web Server Authentication, TLS Web Client Authentication
certificate policies : ???

  > Write to server: 34 bytes written in 1 fragments

GET / HTTP/1.0
Extra-header:


  < Read from server: 146 bytes read

HTTP/1.0 400 Bad Request
Date: Mon, 25 Mar 2024 22:25:35 GMT
Server: Netlify
X-Nf-Request-Id: 01HSVVG474RHM9C83RK7FFN5PJ
Content-Length: 0

  . Closing the connection... done

RE: ISRG Root X1 certificate fails to parse

jeremyherbert — Mon, 25 Mar 2024 20:14:09 GMT

Hi Michal,

in NCS 2.6 you do not need to manually convert the certificate to a C string - this happens via a script in the CMakeLists file. You just pass in the raw PEM file (this is how the example now works by default).

Thanks,

Jeremy

RE: ISRG Root X1 certificate fails to parse

Michal — Mon, 25 Mar 2024 12:51:41 GMT

Hello,

Did you convert the certificate properly? Remember that you have to escape the newlines in the certificate file to be able to include it directly in C.

Please take a look at the example certificate in the sample code for how it is done.

Best regards,

Michal