LwM2M client sample: Astronomically high power draw when losing connection between registration and registration update

RE: LwM2M client sample: Astronomically high power draw when losing connection between registration and registration update

AlanIoT — Mon, 01 Jul 2024 06:54:14 GMT

That's true, I've neglected that point in my calculation, which means I've calculated way too much power draw for a productive device. We had UART active here for the modem trace.

RE: LwM2M client sample: Astronomically high power draw when losing connection between registration and registration update

Achim Kraus — Sun, 23 Jun 2024 09:01:03 GMT

> Most of the power draw is from having UART active, so the draw is semi-reasonable.

In my experience, that's mainly the UART receiving. Using the "ASYNC UART" approach, "uart_rx_disable()" and "uart_rx_enable()" may be used to optimize it.

RE: LwM2M client sample: Astronomically high power draw when losing connection between registration and registration update

Achim Kraus — Sun, 23 Jun 2024 08:55:02 GMT

The pain with the "periodic cell search" is mainly, that if no cell is found, it stick to idle instead of sleeping. At least last year it was like that.

%PERIODICSEARCHCONF - relation to EMM reject cause

RE: LwM2M client sample: Astronomically high power draw when losing connection between registration and registration update

SeppoTakalo — Mon, 17 Jun 2024 10:01:38 GMT

There is a short section in nRF Connect SDK documentation about power optimizing on nRF91.

Periodic cell searches are mentioned there:
* https://docs.nordicsemi.com/bundle/ncs-latest/page/nrf/test_and_optimize/optimizing/power_nrf91.html#making_periodic_cell_searches_less_frequent

* https://docs.nordicsemi.com/bundle/ref_at_commands/page/REF/at_commands/nw_service/periodicsearchconf.html

Those might give enough information to configure the network searches. Overall, I think the lte_lc_offline() is unnecessary if the network was behaving correctly and we just lost the connection due to poor RF conditions. The sample is not properly demonstrating this.

RE: LwM2M client sample: Astronomically high power draw when losing connection between registration and registration update

AlanIoT — Mon, 17 Jun 2024 07:29:52 GMT

Testing with the new logic we have now reached acceptable levels of power draw, so it's a fine solution for now. Most of the power draw is from having UART active, so the draw is semi-reasonable. Of course the 8C connection attempts hurt, but there's little we can do about that without hurting connection reliability.

Your input is interesting. Is there anyone who could assist with this?

RE: LwM2M client sample: Astronomically high power draw when losing connection between registration and registration update

SeppoTakalo — Mon, 17 Jun 2024 07:08:41 GMT

I recommend also trying to remove the whole reconnection logic.

I suspect that it exist only to speed up testing. But modem also has own timers that it regularly scans the network. And it might do it with much less power when not constantly restarted.

But I am not aware of where these are documented, or exactly how it works.

RE: LwM2M client sample: Astronomically high power draw when losing connection between registration and registration update

AlanIoT — Fri, 14 Jun 2024 12:36:23 GMT

Thanks for helping me understand.

I see your point and it makes more sense to modify the client_states than the functions, so I have reverted my previous changes made the following new ones:

1. On the 4th step (client_state connected but disconnected), NETWORK_ERROR is now triggered instead of suspending the client:

		case CONNECTED:
			if (!modem_connected_to_network) {

				// modified: trigger network error instead of suspending engine
				// makes NETWORK_ERROR trigger modem_connect()
				reconnect = true;
				// makes modem_connect() wait for 24 hours instead of 6 hours after the first failed reconnection attempt
				if (reconnection_counter == 0) {
					reconnection_counter = 1;
				}
				state_trigger_and_unlock(NETWORK_ERROR);

				/* LTE connection down suspend LwM2M engine */
				//suspend_lwm2m_engine();
				// modification end

			} else {
				k_mutex_unlock(&lte_mutex);
				LOG_INF("LwM2M is connected to server");
				reconnection_counter = 0;
				if (update_session_lifetime) {
					/* Read a current server  lifetime value */
					rd_client_update_lifetime(client.srv_obj_inst);
				}

#if defined(CONFIG_APP_LWM2M_CONFORMANCE_TESTING)
				lwm2m_register_server_send_mute_cb();
#endif
				/* Get current time and date */
				date_time_update_async(date_time_event_handler);
			}
			break;

2. NETWORK_ERROR always sleeps for 6 hours after stopping the client and setting the modem to flight mode, and before calling modem_connect():

		case NETWORK_ERROR:
			/* Stop the LwM2M engine. */
			LOG_INF("Entered NETWORK_ERROR state");
			state_trigger_and_unlock(START);
			lwm2m_rd_client_stop(&client, rd_client_event, false);

			/* Set network state to start for blocking LTE */
			if (reconnect) {
				reconnect = false;

				LOG_INF("LwM2M restart requested. The sample will try to"
					" re-establish network connection.");

				/* Try to reconnect to the network. */
				ret = lte_lc_offline();
				if (ret < 0) {
					LOG_ERR("Failed to put LTE link in offline state (%d)",
						ret);
				}
				// wait 6 hours before reconnecting
				k_sleep(K_SECONDS(60*60*6));

				modem_connect();
			}
#if defined(CONFIG_APP_LWM2M_CONFORMANCE_TESTING)
			k_work_cancel_delayable(&send_periodical_work);
			send_count = 0;
#endif
			break;

I'll start testing with these changes then.

RE: LwM2M client sample: Astronomically high power draw when losing connection between registration and registration update

SeppoTakalo — Fri, 14 Jun 2024 11:34:58 GMT

I did not do a complete analysis of your changes, but I feel like there might be a possibility of deadlock.

I think the sample application is probably trying to be a bit too clever, so it hides actually the what causes of certain events.

I don't recommend touching the "suspend_client()" as that is a result of LTE Link Controller event that tells application that modem have lost the network. So it then pauses the client.

Please see:

in main() lte_lc_register_handler(lte_notify_handler); is registered.
When network status changes lte_notify_handler() is called with type LTE_LC_EVT_NW_REG_STATUS
It calls lwm2m_lte_reg_handler_notify() which modified the modem_connected_to_network variable and triggers the main state machine.
Main state machine is in CONNECTED state and sees that it is not anymore connected, so it calls suspend_client().
suspend_client() pauses the LwM2M engine.

If you now trigger NETWORK_ERROR, it might call lwm2m_rd_client_stop(), but the client is paused, so it does not stop. It must be resumed first.

Or at least, if you modify, be aware of this. If some of your event have caused client to pause, it must also resume at some point.

Another cause of this issue is:

LwM2M client loses connection to server or fails on its registration attempts. It issues LWM2M_RD_CLIENT_EVENT_NETWORK_ERROR event.
Application reacts to this event by issuin NETWORK_ERROR to the main event loop
main even loop forces modem to rescan the network.

So if you add timeouts, add timeout to this second handling as well. So probably just modify the handling of NETWORK_ERROR so that any case that leads here, will add a long timeout before issuing "modem_connect" again.

RE: LwM2M client sample: Astronomically high power draw when losing connection between registration and registration update

AlanIoT — Fri, 14 Jun 2024 11:07:39 GMT

Thanks for your answers, Øyvind and Seppo

That makes sense and I have come to a similar conclusion.

The thing is, I feel like the described case doesn't usually trigger the network_error state, only if lwm2_engine_pause() fails. From the capture it is also apparent that the modem doesn't reconnect, else there would be a lot more action in the modem trace (modem flight mode, connect and other AT commands), or am I mistaken? Mainly, the lack of any modem activity surprised me, but this might just be normal.

I have changed the code so the suspend_lwm2m_engine() function now triggers the network_error state:

		case CONNECTED:
			if (!modem_connected_to_network) {
				/* LTE connection down suspend LwM2M engine */
				suspend_lwm2m_engine();

			} else {
				k_mutex_unlock(&lte_mutex);
				LOG_INF("LwM2M is connected to server");
				reconnection_counter = 0;
				if (update_session_lifetime) {
					/* Read a current server  lifetime value */
					rd_client_update_lifetime(client.srv_obj_inst);
				}

#if defined(CONFIG_APP_LWM2M_CONFORMANCE_TESTING)
				lwm2m_register_server_send_mute_cb();
#endif
				/* Get current time and date */
				date_time_update_async(date_time_event_handler);
			}
			break;

static void suspend_lwm2m_engine(void)
{
	int ret;

	LOG_INF("entered suspend_lwm2m_engine() function");
	// makes NETWORK_ERROR trigger modem_connect()
	reconnect = true;
	// tells modem_connect() to sleep for 6 hours before attempting first connection
	reconnect_timeout = true;
	// makes modem_connect() sleep for 24 hours instead of 6 hours after the first failed reconnection attempt
	if (reconnection_counter == 0) {
		reconnection_counter = 1;
	}
	state_trigger_and_unlock(NETWORK_ERROR);
	
	//state_trigger_and_unlock(LTE_OFFLINE);

	ret = lwm2m_engine_pause();
	if (ret) {
		LOG_ERR("LwM2M engine pause fail %d", ret);
		reconnect = true;
		k_mutex_lock(&lte_mutex, K_FOREVER);
		state_trigger_and_unlock(NETWORK_ERROR);
	}
}

Anyway, in the described case, I would say that the Problem is fixed for us, but unless I'm misunderstanding, maybe it would be worth investigating the state machine in the case of a complete loss of network coverage after a successful registration.

Hope this helps someone in the future.

RE: LwM2M client sample: Astronomically high power draw when losing connection between registration and registration update

SeppoTakalo — Fri, 14 Jun 2024 08:59:43 GMT

[quote userid="131108" url="~/f/nordic-q-a/112052/lwm2m-client-sample-astronomically-high-power-draw-when-losing-connection-between-registration-and-registration-update"]Testing our device under harsh conditions has shown that the lwm2m client sample can enter a state in which it continuously tries to connect to the network, leading to incredibly high power draw.[/quote]

Yes.

LwM2M client sample is just a sample.

When LwM2M engine signals that it cannot connect, and when we lose the network, the sample immediately restarts the network scan.

See https://github.com/nrfconnect/sdk-nrf/blob/d653bc0495a7a289df96d4e41e7d4e7131b88c48/samples/cellular/lwm2m_client/src/main.c#L771-L789

You might want to refactor that part so that there is some idle period before it re-attempts.

RE: LwM2M client sample: Astronomically high power draw when losing connection between registration and registration update

AlanIoT — Fri, 14 Jun 2024 06:39:10 GMT

whoops, I see the same kind of logic is the point of the suspend_lwm2m_engine() function:

static void suspend_lwm2m_engine(void)
{
	int ret;

	state_trigger_and_unlock(LTE_OFFLINE);
	ret = lwm2m_engine_pause();
	if (ret) {
		LOG_ERR("LwM2M engine pause fail %d", ret);
		reconnect = true;
		k_mutex_lock(&lte_mutex, K_FOREVER);
		state_trigger_and_unlock(NETWORK_ERROR);
	}
}

so that approach does not seem fruitful.

I will implement logging and record the application logs in an effort to understand the application's behaviour regarding this case.

RE: LwM2M client sample: Astronomically high power draw when losing connection between registration and registration update

Øyvind — Thu, 13 Jun 2024 13:13:22 GMT

Hello,

Thanks for sharing all the details. I see you have been in contact with SeppoTakalo in your other ticket. I have reached out to him to get more information on this issue.

Kind regards,
Øyvind

RE: LwM2M client sample: Astronomically high power draw when losing connection between registration and registration update

AlanIoT — Thu, 13 Jun 2024 09:56:14 GMT

I'm currently testing this solution:

        case CONNECTED:
            if (!modem_connected_to_network) {
                /* LTE connection down suspend LwM2M engine */
                suspend_lwm2m_engine();
                // modification START
 // custom indicator so the modem doesn't try to reconnect immediately, but after a set timeout of e.g. 6 hours
                reconnect_timeout = true;
                // modem will try to reconnect from the NETWORK_ERROR after the timeout
                reconnect = true;
                state_trigger_and_unlock(NETWORK_ERROR);
                // modification END

            } else {
                k_mutex_unlock(&lte_mutex);
                LOG_INF("LwM2M is connected to server");
                reconnection_counter = 0;
                if (update_session_lifetime) {
                    /* Read a current server  lifetime value */
                    rd_client_update_lifetime(client.srv_obj_inst);
                }

#if defined(CONFIG_APP_LWM2M_CONFORMANCE_TESTING)
                lwm2m_register_server_send_mute_cb();
#endif
                /* Get current time and date */
                date_time_update_async(date_time_event_handler);
            }
            break;

I'll keep you updated

RE: LwM2M client sample: Astronomically high power draw when losing connection between registration and registration update

AlanIoT — Thu, 13 Jun 2024 07:45:52 GMT

Since the engine was suspended but the modem stayed awake, putting the modem to sleep by adding lte_lc_offline() to the ENGINE_SUSPENDED EVENT would surely fix the issue we see here. Is there a reason not to do this? what would happen next?

Would the device wake up on expiry of the pTAU timer (t3412) and reregister? If yes, that would be fine.

Are there other cases where this would hurt?

    case LWM2M_RD_CLIENT_EVENT_ENGINE_SUSPENDED:
        #if defined(CONFIG_PRIOT_CONNECTION_DEBUGGING)
            ts_last_engine_suspended = k_uptime_get(); 
            last_rd_event = LWM2M_RD_CLIENT_EVENT_ENGINE_SUSPENDED;
        #endif
        LOG_DBG("LwM2M engine suspended");
 lte_lc_offline();
        k_mutex_unlock(&lte_mutex);
        break;

Or am I thinking too far and it's just simple configuration that's needed? maybe setting the modem to constrained mode or something...

RE: LwM2M client sample: Astronomically high power draw when losing connection between registration and registration update

AlanIoT — Thu, 13 Jun 2024 06:44:57 GMT

Here's the modem trace... There's a modem wake up, then an AT+CEREG=4, then nothing in the trace for hours while it keeps draining the battery:

devzone.nordicsemi.com/.../20240613_5F00_0804_5F00_no_5F00_connection_5F00_at_5F00_first_5F00_reg_5F00_update.pcapng

Screenshot of the same event:

Within 10 hours, roughly 5% of our battery capacity is drained, giving a theoretical lifetime of 200h, which would be catastrophic in the field.

Any indications?