modem_key_mgmt_cmp() triggers modem firmware memory management fault

[Fisherdude]

I am running the Dev Academy Lesson 4 Exercise 2 - MQTT with TLS and getting the error:

    [00:00:00.650,085] <err> nrf_modem: Modem has crashed, reason 0x4, PC: 0xecb64

I see that is a modem firmware memory management fault. The first time I built and flashed the program, it worked fine. When I reset the system, I started seeing the error. I've erased the flash, done "west flash --recovery", and cycled power. All give the same result.

A bit of debugging shows the error gest triggered from within the call to modem_key_mgmt_cmp(). Specifically, the call to key_fetch(sec_tag, cred_type) returns -110; I suspect this underlying cause. If I comment out the code that calls modem_key_mgmt_exists() and modem_key_mgmt_cmp(), thus always writing the certificate to the modem, the application works fine.

I'm using a nRF9160 DK and SDK v2.6.1.

Thanks,

Rick

[Hakon]

Can you do modem trace?

[Fisherdude]

I can try a modem trace, but I don't think it will show anything. The memory fault occurs before I turn the modem online. The sequence that causes the error is:

1. nrf_modem_lib_init()
2. modem_key_mgmt_exists() - this returns true
3. modem_key_mgmt_cmp() - this triggers the error
4. modem_key_mgmt_write()
5. lte_lc_connect_async()

If I comment out steps 2-3, I do not get the memory fault error.

[Fisherdude]

The logged error from the library is:

<err> nrf_modem: Modem has crashed, reason 0x4, PC: 0xecb64

This simple program reproduces the problem:

/*
 * Copyright (c) 2018 Nordic Semiconductor ASA
 *
 * SPDX-License-Identifier: LicenseRef-Nordic-5-Clause
 */

#include <stdio.h>
#include <ncs_version.h>
#include <zephyr/kernel.h>
#include <zephyr/net/socket.h>
#include <zephyr/net/mqtt.h>

#include <zephyr/logging/log.h>
#include <dk_buttons_and_leds.h>
#include <modem/nrf_modem_lib.h>
#include <modem/lte_lc.h>

#include "mqtt_connection.h"

#include <modem/modem_key_mgmt.h>
#include "certificate.h"

LOG_MODULE_REGISTER(MemFaultKeyMgmt, LOG_LEVEL_INF);

static int modem_configure(void)
{
	int err;

	LOG_INF("Initializing modem library");

	err = nrf_modem_lib_init();
	if (err) {
		LOG_ERR("Failed to initialize the modem library, error: %d", err);
		return err;
	}
	LOG_INF("Modem library initialized");

	/* STEP 4.3 - Store the certificate in the modem while the modem is in offline mode  */
	bool exists;

	err = modem_key_mgmt_exists(CONFIG_MQTT_TLS_SEC_TAG,
				    MODEM_KEY_MGMT_CRED_TYPE_CA_CHAIN,
				    &exists);
	if (err) {
		LOG_ERR("Failed to check for certificates err %d\n", err);
		return err;
	}

	if (exists) {
		/* Let's compare the existing credential */
		err = modem_key_mgmt_cmp(CONFIG_MQTT_TLS_SEC_TAG,
					 MODEM_KEY_MGMT_CRED_TYPE_CA_CHAIN,
					 CA_CERTIFICATE, 
					 strlen(CA_CERTIFICATE));
		LOG_INF("%s\n", err ? "mismatch" : "match");
		if (!err) {
			return 0;
		}
	}

	return err;
}


int main(void)
{
	int err;

	err = modem_configure();
	if (err) {
		LOG_ERR("Failed to configure the modem");
		return 0;
	}

	// After some milliseconds, the modem library sends the memory management fault,
	// <err> nrf_modem: Modem has crashed, reason 0x4, PC: 0xecb64
	while (1) {
		k_msleep(5);
	}	

	/* This is never reached */
	return 0;
}