(2.6.1 update) lte_lc_connect_async crashes [Illegal use of the EPSR]

RE: (2.6.1 update) lte_lc_connect_async crashes [Illegal use of the EPSR]

CRD — Wed, 29 May 2024 14:10:08 GMT

It's possible the ble_enable was keeping that work function running or something. The code was basically this before (when the bug was occurring):

void start_ble_work_fn(struct k_work *work);
K_WORK_DELAYABLE_DEFINE(start_ble_work, start_ble_work_fn);    
    
void start_ble_work_fn(struct k_work *work){    
    ARG_UNUSED(work);
    
    err = bt_enable(NULL);
	if (err) {
		LOG_ERR("Bluetooth init failed (err %d)\n", err);
		return err;
	}

	if (IS_ENABLED(CONFIG_BT_SETTINGS)) {
		settings_load();
	}

	ble_interface_evt_handler = event_handler;
	
    if (err) {
    	LOG_ERR("ble_interface_init, error: %d", err);
    }else{
    	SEND_EVENT(ble, BLE_EVT_START_ADVERTISING);
    	ble_enabled = true;
    }
}

void main_loop(){
    if (IS_EVENT(msg, app, APP_EVT_START)){
        k_work_schedule(&start_ble_work, K_SECONDS(3));
    }
}

That code worked in SDK 2.5.2 but not after we updated to 2.6.1. It would crash as soon as LTE connected and this was called in lte_lc_helpers.c "curr->handler(evt);" as shown above in a previous response.

I was able to fix this by instead making the caller wait 3 seconds before sending an event to start ble. This new delayed event then triggered the "start_ble()" function directly rather than from a k_work_delayable function.

In the end I removed the delay all together and now send an event when all the relevant modules have been all initialized removing the need for the arbitrary 3 second delay.

RE: (2.6.1 update) lte_lc_connect_async crashes [Illegal use of the EPSR]

Charlie — Wed, 29 May 2024 13:46:02 GMT

Hi Colin,

Thanks for the update. Yes, it is very strange.

I just wonder, have you done something time-consuming or even blocking in your work task function. Have you used log to check when it is actually finished?

Best regards,

Charlie

RE: (2.6.1 update) lte_lc_connect_async crashes [Illegal use of the EPSR]

CRD — Mon, 27 May 2024 20:25:14 GMT

I have finally resolved this but I don't fully understand what was wrong. I will put my best guess here in case anyone else finds this and has a similar issue.

My device also runs BLE. When the device boots up it reads the stored BLE name from persistent memory and triggers an event to let the BLE module know it can start advertising. However, the function in my ble module that receives this event uses a k_work_schedule with a 3 second delay.

Even though the BLE starts way before (30 seconds) the LTE connection attempt and this delayable should be long gone it was somehow leading to the ESPR issue. If I remove the k_work_schedule for starting BLE everything works normally now. I have now reworked the code to make sure everything is initialized instead of using the arbitrary 3 second delay which has allowed me to remove the k_work_schedule and resolved the issues.

I find it very strange that this code would have impacted anything (especially an unrelated call happening 30 seconds later on a different thread). It's almost like there was a memory leak or corruption from the k_work_schedule. Very confusing.

RE: (2.6.1 update) lte_lc_connect_async crashes [Illegal use of the EPSR]

Charlie — Wed, 22 May 2024 13:43:59 GMT

Hi Colin,

Ok, it seems the cause is from the nRF library, but I feel not confident when you start to modify the library codes.

Have you ported the UDP sample to your custom board to have a try? Or test your minimal codes that can repeat this issue on a nRF9160DK?

These tests will help to identify if it is the hardware to make the difference.

Best regards,

Charlie

RE: (2.6.1 update) lte_lc_connect_async crashes [Illegal use of the EPSR]

CRD — Thu, 16 May 2024 20:07:24 GMT

To add to this: with those lines commented out I get good LTE connections now but I no longer get the datetime updated after connecting to LTE which was one of my original hunches as a potential cause of the fault messages. So some event sent by that macro for datetime updates is leading to the kernel crash.

RE: (2.6.1 update) lte_lc_connect_async crashes [Illegal use of the EPSR]

CRD — Thu, 16 May 2024 14:43:09 GMT

I finally made some big progress. I used the amr-zephyr-eabi-addr2line.exe program to work back the fault. I finally got this:

arm-zephyr-eabi-addr2line.exe -e ./build/debug/zephyr/zephyr.elf -a 0x00025ab7
0x00025ab7
C:/ncs/v2.6.1/nrf/lib/lte_link_control/lte_lc_helpers.c:124 (discriminator 343)

If I comment out the SYS_SLIST_FOR_EACH_CONTAINER_SAFE macro (see below) in the function "event_handler_list_dispatch" the app now connects to lte and works normally. As far as I can tell this seems to be unchanged from SDK 2.5.1 so I don't see how its now causing a problem... something deeper in fails with slist peek_next().

void event_handler_list_dispatch(const struct lte_lc_evt *const evt)
{
	struct event_handler *curr, *tmp;

	if (event_handler_list_is_empty()) {
		return;
	}

	k_mutex_lock(&list_mtx, K_FOREVER);

	/* Dispatch events to all registered handlers */
	LOG_DBG("Dispatching event: type=%d", evt->type);
	// SYS_SLIST_FOR_EACH_CONTAINER_SAFE(&handler_list, curr, tmp, node) {
	// 	LOG_DBG(" - handler=0x%08X", (uint32_t)curr->handler);
	// 	curr->handler(evt);
	// }
	LOG_DBG("Done");

	k_mutex_unlock(&list_mtx);
}

RE: (2.6.1 update) lte_lc_connect_async crashes [Illegal use of the EPSR]

Charlie — Thu, 16 May 2024 14:32:35 GMT

Hi Colin,

Ok, have you tried to run lte connection actions in a standalone thread? but I think it is not necessary.

Our Cellular: UDP — nRF Connect SDK 2.6.1 documentation (nordicsemi.com) sample shows simple usage of lte_lc_connect_async and it is not multi-threads application, could you read through its source codes to get some hint?

Best regards,

Charlie

RE: (2.6.1 update) lte_lc_connect_async crashes [Illegal use of the EPSR]

CRD — Tue, 14 May 2024 19:51:36 GMT

Okay I have found out more. (my app is based off the aws asset tracked v2 project btw).

When I was connecting previously it was after a delay. So:

main.c was calling nrf_modem_lib_init() on bootup
After 30 seconds the modem module thread would call lte_lc_connect_async
This crashes immediately after changing the CFUN to 1

During my testing I modified the code (I tried running them from the same thread, from main etc etc..) but at some point I ran the lte_lc_connect_async() at bootup time immediately after initializing the modem. This connects to the network..... but then crashes after about 30 seconds the same way.

I have tried recreating this issue in the lte_ble_gateway sample, and even copied all my modem module code into it but strangely it works without any issues in that case... This sample isn't running multiple module threads so maybe that is part of the issue?

I added the AT_HOST_LIBRARY=y so I could log out the AT commands that are being sent during connection and before the crash through the RTT logs:

LTE CONNECTING....
%CESQ: 34,1,0,0
modem_rsrp_handler: Incoming RSRP status message, RSRP value is -106
+CEREG: 0
lte_evt_handler: LTE cell changed: Cell ID: -1, Tracking area: -1
+CEREG: 2,"<hidden>","<hidden>",7
lte_evt_handler: PSM parameter update: TAU: 2, Active time: 0
lte_evt_handler: LTE cell changed: Cell ID: <hidden>, Tracking area: <hidden>
+CSCON: 1
lte_evt_handler: RRC mode: Connected
%CESQ: 41,2,4,0
modem_rsrp_handler: Incoming RSRP status message, RSRP value is -99
+CGEV: ME PDN ACT 0,0
modem: PDN LTE CONNECTED
+CNEC_ESM: 50,0
pdn_event_handler: Event: PDP context 0, PDN type IPv4 only allowed
%MDMEV: SEARCH STATUS 2
+CEREG: 5,"<hidden>","<hidden>",7,,,"00001010","11000001"
lte_evt_handler: Network registration status: Connected - roaming
lte_evt_handler: PSM parameter update: TAU: 1152000, Active time: 20
%XTIME: ,"42504191339469","01"

(About 10 seconds pass)

%CESQ: 255,0,255,0
+CSCON: 0
[00:00:27.201,293] <err> os: ***** USAGE FAULT *****
[00:00:27.201,324] <err> os:   Illegal use of the EPSR
[00:00:27.201,354] <err> os: r0/a1:  0x20020ff8  r1/a2:  0x20013aa8  r2/a3:  0x20013aa8
[00:00:27.201,354] <err> os: r3/a4:  0x0002b000 r12/ip:  0x0ccccccc r14/lr:  0x00029c07
[00:00:27.201,385] <err> os:  xpsr:  0x60000000
[00:00:27.201,385] <err> os: s[ 0]:  0x00000003  s[ 1]:  0x2001cb08  s[ 2]:  0x2001cb04  s[ 3]:  0x00029317
[00:00:27.201,416] <err> os: s[ 4]:  0x00000003  s[ 5]:  0x00000000  s[ 6]:  0x00000000  s[ 7]:  0x00000000
[00:00:27.201,446] <err> os: s[ 8]:  0x00000000  s[ 9]:  0x00000000  s[10]:  0x00000000  s[11]:  0x00000000
[00:00:27.201,477] <err> os: s[12]:  0x00000000  s[13]:  0x00000000  s[14]:  0x00000000  s[15]:  0x00000000
[00:00:27.201,477] <err> os: fpscr:  0x00000000
[00:00:27.201,507] <err> os: Faulting instruction address (r15/pc): 0x0002b000
[00:00:27.201,538] <err> os: >>> ZEPHYR FATAL ERROR 35: Unknown error on CPU 0
[00:00:27.201,599] <err> os: Current thread: 0x20013aa8 (sysworkq)
[00:00:27.733,703] <err> fatal_error: Resetting system

RE: (2.6.1 update) lte_lc_connect_async crashes [Illegal use of the EPSR]

Charlie — Tue, 14 May 2024 11:27:04 GMT

Hi Colin,

Good to see so much useful information.

[quote user="CRD"] I was able to load the lte_ble_gateway sample on the same device (built with 2.6.1 and it works fine) + some other info below.[/quote]

Do you mean the same issue works fine with NCSv2.6.1 lte_ble_gateway sample on your custom device?

The codes you shared are related to your custom codes?

[quote user="CRD"]If I leave just err = nrf_modem_at_printf(cscon); and don't check the error it seems to run stable...[/quote]

This also sounds very weird to me. Could you repeat this issue with your codes on a nRF9160DK? Just share a minimal part of your codes that can repeat this issue if you can.

Best regards,

Charlie

RE: (2.6.1 update) lte_lc_connect_async crashes [Illegal use of the EPSR]

CRD — Mon, 13 May 2024 20:11:52 GMT

Hi Charlie,

I did a bunch of testing today. We are using MFW1.3.4 but it doesn't seem to be related to this. I was able to load the lte_ble_gateway sample on the same device (built with 2.6.1 and it works fine) + some other info below.

I was able to trace the error to this line:

lte_lc.c (line 546)
SDK v2.6.1

/* +CSCON notifications */
err = nrf_modem_at_printf(cscon);
if (err) {
	LOG_WRN("Failed to enable RRC notifications (+CSCON), error %d", err);
	return -EFAULT;
}

SDK v2.5.1

/* +CSCON notifications */
err = nrf_modem_at_printf(cscon);
if (err) {
	char buf[50];

	/* AT+CSCON is supported from modem firmware v1.1.0, and will
	 * not work for older versions. If the command fails, RRC
	 * mode change notifications will not be received. This is not
	 * considered a critical error, and the error code is therefore
	 * not returned, while informative log messages are printed.
	 */
	LOG_WRN("AT+CSCON failed (%d), RRC notifications are not enabled", err);
	LOG_WRN("AT+CSCON is supported in nRF9160 modem >= v1.1.0");

	err = nrf_modem_at_cmd(buf, sizeof(buf), "AT+CGMR");
	if (err == 0) {
		char *end = strstr(buf, "\r\nOK");

		if (end) {
			*end = '\0';
		}

		LOG_WRN("Current modem firmware version: %s", buf);
	}
}

return 0;

If I comment that command it seems to connect and run stable. But there is still some strange behaviour I can't quite figure out. It almost seems like the LOG_WRN calls are leading to the crash? But as far as I know it is not returning an error so those shouldn't even be called.

If I leave just err = nrf_modem_at_printf(cscon); and don't check the error it seems to run stable...

Very confusing but hopefully on the right track.

I also tried sending AT+CSCON=1 command manually and it returned OK and didn't crash the app.

RE: (2.6.1 update) lte_lc_connect_async crashes [Illegal use of the EPSR]

Charlie — Mon, 13 May 2024 12:50:22 GMT

Hi Colin,

May I know which NCS version your previous codes are based on?

Have you also updated to the latest modem firmware(MFW1.3.6 for nRF9160 at current time)?

Best regards,

Charlie