HardFault from mbedtls_internal_aes_encrypt after porting mqttsn_client_publisher to use Open Thread 1.2

[bradastronics]

Hello! We are targeting an nrf52840. We were able to compile and successfully use the mqttsn_client_publisher from the latest nRF SDK v4.1.0 for Thread and Zigbee. We then attempted to upgrade to Open Thread 1.2 per these instructions infocenter.nordicsemi.com/index.jsp. Having successfully rebuilt, we loaded the example and now get a HardFault in mbedtls_internal_aes_encrypt shortly after running the application. Exact same call stack and issue as reported here. devzone.nordicsemi.com/.../282556 but unfortunately no resolution was given in that thread.

Any help to offer on what might be going on?

Thanks, Brad

[bradastronics]

Hi,

There were a few build errors after updating to the new header files for OT 1.2 to address the removal of otCliUartInit, the change to otUdp* params, and removal of otHeapSetCAllocFree. So the application itself didn't change but the components using the above ot functions changed slightly. Only about 10 lines of code.

Using openthread commit e66b9a3958bfdc3fb495a189737265561e9a99bd (https://github.com/openthread/openthread/commit/e66b9a3958bfdc3fb495a189737265561e9a99bd)

One thing I haven't tried yet is copying over libmbedcrypto libraries as the Noridc instructions didn't say to. However, these static libs are referenced in the project. Should I copy those over as well? Would it be possible to use the cc310 backend to use hardware acceleration instead since this is an nRF52840? Sorry, we are new to the nRF5 SDK.

Thanks for the help,

Brad

[Jørgen Holmefjord]

Hi,

bradastronics said:

Using openthread commit e66b9a3958bfdc3fb495a189737265561e9a99bd (https://github.com/openthread/openthread/commit/e66b9a3958bfdc3fb495a189737265561e9a99bd)

It is quite long time (over 1 year) between this commit and the original commit used in nRF5 SDK for Thread and Zigbee v4.1.0 (https://github.com/openthread/openthread/commit/c6a258e3a5bd90aa26d8245c435c0ae5797027f4). You may have been lucky that there was not more API changes, but there may also be some changes in usage that will not necessarily generate compile errors.

bradastronics said:

One thing I haven't tried yet is copying over libmbedcrypto libraries as the Noridc instructions didn't say to. However, these static libs are referenced in the project. Should I copy those over as well?

Yes, if this library was modified in the new OpenThread version and rebuilt, I would recommend you to try to replace this as well. It could look like this is the source of the HardFault

bradastronics said:

Would it be possible to use the cc310 backend to use hardware acceleration instead since this is an nRF52840?

If the MbedTLS library used is built with CC310 support, that is possible. By default, PCA10056/nRF52840 examples should use CC310.

Best regards,
Jørgen

[bradastronics]

I did copy over the entire nrf_security folder, replacing back some of the glue libraries that weren't included with OpenThread but were in the nRF5 SDK, but got linker errors:

1> C:/Program Files/SEGGER/SEGGER Embedded Studio for ARM 5.50b/gcc/arm-none-eabi/bin/ld: ../../../../../../external/nrf_security/lib/libmbedtls_base_vanilla.a(md.c.obj): in function `mbedtls_md_free':
1> /home/map5/repo/ncs/mbedtls/library/md.c:321: undefined reference to `mbedtls_sha512_free'
1> C:/Program Files/SEGGER/SEGGER Embedded Studio for ARM 5.50b/gcc/arm-none-eabi/bin/ld: ../../../../../../external/nrf_security/lib/libmbedtls_base_vanilla.a(md.c.obj): in function `mbedtls_md_setup':
1> /home/map5/repo/ncs/mbedtls/library/md.c:458: undefined reference to `mbedtls_sha512_init'
1> C:/Program Files/SEGGER/SEGGER Embedded Studio for ARM 5.50b/gcc/arm-none-eabi/bin/ld: ../../../../../../external/nrf_security/lib/libmbedtls_base_vanilla.a(md.c.obj): in function `mbedtls_md_starts':
1> /home/map5/repo/ncs/mbedtls/library/md.c:520: undefined reference to `mbedtls_sha512_starts_ret'
1> C:/Program Files/SEGGER/SEGGER Embedded Studio for ARM 5.50b/gcc/arm-none-eabi/bin/ld: ../../../../../../external/nrf_security/lib/libmbedtls_base_vanilla.a(md.c.obj): in function `mbedtls_md_update':
1> /home/map5/repo/ncs/mbedtls/library/md.c:564: undefined reference to `mbedtls_sha512_update_ret'
1> C:/Program Files/SEGGER/SEGGER Embedded Studio for ARM 5.50b/gcc/arm-none-eabi/bin/ld: ../../../../../../external/nrf_security/lib/libmbedtls_base_vanilla.a(md.c.obj): in function `mbedtls_md_finish':
1> /home/map5/repo/ncs/mbedtls/library/md.c:608: undefined reference to `mbedtls_sha512_finish_ret'

Pretty much out of ideas other than going back to commit e66b9a3958bfdc3fb495a189737265561e9a99bd. Can you confirm whether that commit contains OT 1.2? Our border router only supports OT 1.2.

[Jørgen Holmefjord]

Where did you copy nrf_security folder from and to? There may be different configurations in nRF5 SDK and the source where you got it from.

bradastronics said:

Pretty much out of ideas other than going back to commit e66b9a3958bfdc3fb495a189737265561e9a99bd.

But this is the commit that gave you a HardFault, right?

bradastronics said:

Can you confirm whether that commit contains OT 1.2?

It looks like the default Thread version was set to 1.2 in a pull request merged on 2021 March 15, while the commit you used is from a pull request merged on 2021 June 17, so it should support Thread 1.2 features.

bradastronics said:

Our border router only supports OT 1.2.

What exactly does this mean? Do the border router require features from Thread 1.2? As far as I know, Thread 1.2 should be fully backwards compatible with Thread 1.1 (see FAQ on bottom of this Thread group webpage).

[bradastronics]

OK sorry I copied the wrong hash, but it sounds like c6a258e3a5bd90aa26d8245c435c0ae5797027f4 is too old to include OT 1.2.

I am on vacation until Wednesday of next week, but I will post exactly where I copied nrf_security to where then.

As to my OTBR only supporting 1.2 - it is unable to communicate with a 1.1 NCP Thread device flashed from the Nordic example (nrf52840_xxaa_pca10056.hex) due to missing capabilities in that firmware. I have build my own RCP firmware based on 1.2 and have no issues. I'll have to go copy the error message next week, something about security.

[Jørgen Holmefjord]

bradastronics said:

OK sorry I copied the wrong hash, but it sounds like c6a258e3a5bd90aa26d8245c435c0ae5797027f4 is too old to include OT 1.2.

I see. Yes, nRF5 SDK for T&Z v4.1.0 does not support Thread 1.2 features.

bradastronics said:

As to my OTBR only supporting 1.2 - it is unable to communicate with a 1.1 NCP Thread device flashed from the Nordic example (nrf52840_xxaa_pca10056.hex) due to missing capabilities in that firmware. I have build my own RCP firmware based on 1.2 and have no issues. I'll have to go copy the error message next week, something about security.

I can understand that the OTBR requires some features from Thread 1.2 in the NCP/RCP firmwares used with the border router, but it sounds like you are building a firmware for another node in the network? Will the BR not be able to communicate with other nodes in the network that only supports Thread 1.1? 

[bradastronics]

The error seen running latest OTBR against Nordic NCP FW is:

Jun 16 18:03:53 c4b55ba8e651 otbr-agent[214]: [CRIT]-PLAT----: RCP is missing required capabilities: tx-security tx-timing

I will give OTBR with RCP running OT 1.2 running against the MQTTSN example running OT 1.1 as provided in the Nordic SDK and report back. You are correct the MQTTSN is a different node on the network and I assumed OT 1.1 and OT 1.2 nodes would be incompatible.