nRF54LM20 MbedTLS Heap Exhaustion

[ashlin4010]

Issue

On the nrf54lm20 using DTLS with MbedTLS, MbedTLS's heap can be exhausted. This will result in a TLS handshake error.
When this issue happens MbedTLS does not catch the heap exhaustion, it will just fail to complete the TLS handshake.

Errors

Failed to connect to socket: -116 or TLS handshake error: -0x6e00.

 Here are some more detailed error logs:

[00:00:35.575,555] <inf> mbedtls: WEST_TOPDIR/modules/crypto/mbedtls/library/ssl_tls.c:8300: 0x20009840: dumping 'calc finished result' (12 bytes)
[00:00:35.575,726] <inf> mbedtls: WEST_TOPDIR/modules/crypto/mbedtls/library/ssl_tls.c:8300: 0x20009840: 0000:  59 09 12 00 00 00 00 00 40 98 00 20              Y.......@.. 
[00:00:35.575,775] <wrn> mbedtls: WEST_TOPDIR/modules/crypto/mbedtls/library/ssl_tls.c:8304: 0x20009840: <= calc finished
[00:00:35.575,835] <wrn> mbedtls: WEST_TOPDIR/modules/crypto/mbedtls/library/ssl_msg.c:4299: 0x20009840: => read record
[00:00:35.575,888] <wrn> mbedtls: WEST_TOPDIR/modules/crypto/mbedtls/library/ssl_msg.c:2156: 0x20009840: => fetch input
[00:00:35.575,960] <wrn> mbedtls: WEST_TOPDIR/modules/crypto/mbedtls/library/ssl_msg.c:2191: 0x20009840: next record in same datagram, offset: 14
[00:00:35.576,021] <wrn> mbedtls: WEST_TOPDIR/modules/crypto/mbedtls/library/ssl_msg.c:2202: 0x20009840: in_left: 61, nb_want: 13
[00:00:35.576,071] <wrn> mbedtls: WEST_TOPDIR/modules/crypto/mbedtls/library/ssl_msg.c:2210: 0x20009840: <= fetch input
[00:00:35.576,351] <inf> mbedtls: WEST_TOPDIR/modules/crypto/mbedtls/library/ssl_msg.c:3966: 0x20009840: input record: msgtype = 22, version = [0x303], msglen = 48
[00:00:35.577,091] <wrn> mbedtls: WEST_TOPDIR/modules/crypto/mbedtls/library/ssl_msg.c:1511: 0x20009840: => decrypt buf
[00:00:35.578,420] <wrn> mbedtls: WEST_TOPDIR/modules/crypto/mbedtls/library/ssl_msg.c:2122: 0x20009840: <= decrypt buf
[00:00:35.578,840] <inf> mbedtls: WEST_TOPDIR/modules/crypto/mbedtls/library/ssl_msg.c:3239: 0x20009840: handshake message: msglen = 24, type = 20, hslen = 24
[00:00:35.578,964] <wrn> mbedtls: WEST_TOPDIR/modules/crypto/mbedtls/library/ssl_msg.c:4371: 0x20009840: <= read record
[00:00:35.579,023] <err> mbedtls: WEST_TOPDIR/modules/crypto/mbedtls/library/ssl_tls.c:8593: 0x20009840: bad finished message                                <---------------------------------------------- HERE
[00:00:35.579,079] <wrn> mbedtls: WEST_TOPDIR/modules/crypto/mbedtls/library/ssl_msg.c:5295: 0x20009840: => send alert message
[00:00:35.579,144] <inf> mbedtls: WEST_TOPDIR/modules/crypto/mbedtls/library/ssl_msg.c:5296: 0x20009840: send alert level=2 message=51
[00:00:35.579,197] <wrn> mbedtls: WEST_TOPDIR/modules/crypto/mbedtls/library/ssl_msg.c:2944: 0x20009840: => write record
[00:00:35.579,251] <wrn> mbedtls: WEST_TOPDIR/modules/crypto/mbedtls/library/ssl_msg.c:0936: 0x20009840: => encrypt buf
[00:00:35.580,019] <inf> mbedtls: WEST_TOPDIR/modules/crypto/mbedtls/library/ssl_msg.c:1190: 0x20009840: before encrypt: msglen = 2, including 0 bytes of padding
[00:00:35.580,539] <wrn> mbedtls: WEST_TOPDIR/modules/crypto/mbedtls/library/ssl_msg.c:1475: 0x20009840: <= encrypt buf
[00:00:35.580,639] <inf> mbedtls: WEST_TOPDIR/modules/crypto/mbedtls/library/ssl_msg.c:3028: 0x20009840: output record: msgtype = 21, version = [254:253], msglen = 26
[00:00:35.581,179] <wrn> mbedtls: WEST_TOPDIR/modules/crypto/mbedtls/library/ssl_msg.c:2354: 0x20009840: => flush output
[00:00:35.581,249] <wrn> mbedtls: WEST_TOPDIR/modules/crypto/mbedtls/library/ssl_msg.c:2368: 0x20009840: message length: 39, out_left: 39
[00:00:35.583,139] <wrn> mbedtls: WEST_TOPDIR/modules/crypto/mbedtls/library/ssl_msg.c:2375: 0x20009840: ssl->f_send() returned 39 (-0xffffffd9)
[00:00:35.583,188] <wrn> mbedtls: WEST_TOPDIR/modules/crypto/mbedtls/library/ssl_msg.c:2402: 0x20009840: <= flush output
[00:00:35.583,239] <wrn> mbedtls: WEST_TOPDIR/modules/crypto/mbedtls/library/ssl_msg.c:3081: 0x20009840: <= write record
[00:00:35.583,296] <wrn> mbedtls: WEST_TOPDIR/modules/crypto/mbedtls/library/ssl_msg.c:5307: 0x20009840: <= send alert message
[00:00:35.583,349] <wrn> mbedtls: WEST_TOPDIR/modules/crypto/mbedtls/library/ssl_tls.c:4673: 0x20009840: <= handshake
[00:00:35.583,360] <err> net_sock_tls: TLS handshake error: -0x6e00
[00:00:35.583,463] <inf> mbedtls: WEST_TOPDIR/modules/crypto/mbedtls/library/ssl_misc.h:1353: 0x20009840: handshake state: 13 (MBEDTLS_SSL_SERVER_FINISHED) -> 0 (MBEDTLS_SSL_HELLO_REQUEST)

Quick Fix

Increase MbedTLS heap: 

CONFIG_MBEDTLS_HEAP_SIZE=16384

Steps to Recreate

For testing, I have been using the golioth samples, getting them running is a little involved, so I will not give all the details.
If needed I can prepare an example project with the needed setup. But what I have done is take their hello sample and added ppp networking and some other options needed to get the nrf54lm20 running. They are here:

/cfs-file/__key/communityserver-discussions-components-files/4/golioth_5F00_sample_5F00_updated_5F00_files.zip

Debugging

From some of the logs I have had a look into some of the code, but not much.
https://github.com/nrfconnect/sdk-mbedtls/blob/main/library/ssl_tls.c

On line 8591 is a call to mbedtls_ct_memcmp, it looks like ssl->in_msg is invalid or pointing to the wrong place.
Adding more to the stack fixes this issue. I was not able to work out why/where MbedTLS went wrong.

Environment
Linux
SDK 3.2.1
Toolchain 3.2.0
golioth: v0.22.0

[ashlin4010]

I made an error. Golioth is setting CONFIG_MBEDTLS_HEAP_SIZE=10240, a value of 49152 is sufficient to avoid the TLS errors.

Regardless, should MbedTLS detect heap exhaustion?

[Maria Gilje]

Hello, 

Thank you for contacting us about this. I have forwarded this internally and I will share feedback I get with you. 

Best regards,

Maria

[Maria Gilje]

You should be able to get more information from the sample by using a console or terminal output and setting 

CONFIG_DEBUG_C=y

CONFIG_MBEDTLS_MEMORY_DEBUG=y 

In the logs you can look for i.e. allocation issues. 

Best regards,

Maria