Long-running connections fail due to low-battery

[dan_opito]

Our application is designed to remain connected to the network - sending data every 10 minutes or so. We've customised the asset tracker v2 application, as this provides the overall architecture we need including LTE and MQTT connection recovery. For the most part, it works well. However, we notice all our devices running our "new zephyr" codebase are less stable (i.e. they reboot more often). I set up a longevity test, and capture logs from both the cloud and modem modules - which I extended slightly to get verbose logs about the modem events.

I notice this:

2025-03-13 23:25:55.033 | INFO     | serial_logger:run:36 - [13:30:04.638,275] <inf> app_event_manager: DATA_EVT_DATA_READY
2025-03-13 23:25:55.039 | INFO     | serial_logger:run:36 - [13:30:04.643,096] <inf> app_event_manager: DATA_EVT_DATA_SEND
2025-03-13 23:25:55.050 | INFO     | serial_logger:run:36 - [13:30:04.643,920] <inf> opito_indication: Setting pattern to CLOUD_PUBLISHING
2025-03-13 23:25:55.057 | INFO     | serial_logger:run:36 - [13:30:04.644,744] <dbg> cloud_module: QOS_EVT_MESSAGE_NEW
2025-03-13 23:25:55.068 | INFO     | serial_logger:run:36 - [13:30:04.644,805] <inf> app_event_manager: CLOUD_EVT_DATA_SEND_QOS
2025-03-13 23:25:55.078 | INFO     | serial_logger:run:36 - [13:30:04.644,958] <dbg> cloud_module: QOS_EVT_MESSAGE_REMOVED_FROM_LIST
2025-03-13 23:25:55.085 | INFO     | serial_logger:run:36 - [13:30:04.644,989] <dbg> cloud_module: Freeing pointer: 0x20025910
2025-03-13 23:25:55.095 | INFO     | serial_logger:run:36 - [13:30:04.712,829] <dbg> modem_module: LTE event: MODEM_EVENT
2025-03-13 23:25:55.106 | INFO     | serial_logger:run:36 - [13:30:04.712,860] <dbg> modem_module: Modem domain event, type: CE level 0
2025-03-13 23:25:55.113 | INFO     | serial_logger:run:36 - [13:30:04.811,584] <dbg> modem_module: LTE event: MODEM_EVENT
2025-03-13 23:25:55.125 | INFO     | serial_logger:run:36 - [13:30:04.811,645] <dbg> modem_module: Modem domain event, type: Low battery
2025-03-13 23:25:55.913 | INFO     | serial_logger:run:36 - [13:30:04.911,865] <dbg> modem_module: LTE event: RRC_UPDATE
2025-03-13 23:25:55.920 | INFO     | serial_logger:run:36 - [13:30:04.911,895] <dbg> modem_module: RRC mode: Connected
2025-03-13 23:25:55.930 | INFO     | serial_logger:run:36 - [13:30:05.689,788] <err> mqtt_helper: Socket error: POLLERR
2025-03-13 23:25:55.937 | INFO     | serial_logger:run:36 - [13:30:05.689,788] <err> mqtt_helper: Connection was unexpectedly closed
2025-03-13 23:25:55.948 | INFO     | serial_logger:run:36 - [13:30:05.690,612] <dbg> modem_module: LTE event: NW_REG_STATUS
2025-03-13 23:25:55.958 | INFO     | serial_logger:run:36 - [13:30:05.690,643] <dbg> modem_module: NW_REG_STATUS: Not registered
2025-03-13 23:25:55.965 | INFO     | serial_logger:run:36 - [13:30:05.690,704] <dbg> modem_module: LTE event: CELL_UPDATE
2025-03-13 23:25:55.976 | INFO     | serial_logger:run:36 - [13:30:05.690,704] <dbg> modem_module: LTE cell: Cell ID: -1, Tracking area: -1
2025-03-13 23:25:55.983 | INFO     | serial_logger:run:36 - [13:30:05.690,765] <dbg> modem_module: LTE event: LTE_MODE_UPDATE
2025-03-13 23:25:55.993 | INFO     | serial_logger:run:36 - [13:30:05.690,856] <dbg> modem_module: PDN_EVENT_NETWORK_DETACH
2025-03-13 23:25:56.000 | INFO     | serial_logger:run:36 - [13:30:05.691,009] <dbg> cloud_module: CLOUD_WRAP_EVT_DISCONNECTED
2025-03-13 23:25:56.010 | INFO     | serial_logger:run:36 - [13:30:05.691,101] <inf> app_event_manager: CLOUD_EVT_DISCONNECTED

Once we see that low battery event (AKA `LTE_LC_MODEM_EVT_BATTERY_LOW`) we're no longer able to connect to the network.

Our product is battery powered - but with a 7.4 V lithium pack. The battery pack will cut out before the 3.3 V rail ever drops - and this longevity test is conducted with a power supply rather than a battery. All to say low battery events are not (supposed to be) valid for our use case. We have had our custom PCB design reviewed by Nordic already - and they did note some minor changes we should make. Our design was based off the reference design (in fact, the power section was a direct copy).

Are we able to just disable this check? Does this event highlight potential issues with our design? Are there additional steps I can take to get more information?

[Achim Kraus]

I'm not that sure, that the "battery low" event should be ignored.

Even if your 7.4V battery will hardly go down to the 3V, I guess, that 7.4V are not connected directly to the nRF9151 VDD, or? So there is a component in the power rail, which may cause that drop. You may check the voltage at VDD using an AT cmd "AT%XVBAT".

> However, we notice all our devices running our "new zephyr" codebase are less stable

There are many factors that pays in to this topic, e.g. varying radio signal strength, other MNO and so on. So I would not expect, that the "zephyr codebase" (running on the app mcu) has a that large effect on the power consumption of the modem on the SoC. What may have changed is the handling of the "battery low" events and how the "battery OK" is detected afterwards. But I'm not common to the asset tracker, so that's out of my knowledge.

[dan_opito]

Hi Achim. Thanks for your insights. I have been working with a Nordic engineer on this issue - and he, like yourself, suspected power supply issues. Our design, unfortunately, has a flaw that means the VDD rail of the modem can drop - and this is due to a ferrite bead we have in series. I haven't been able to capture it on an oscilloscope, but by raising the `AT%XPOFWARN` to a higher voltage, I can more readily produce the same fault.

My fix is necessarily in software. The asset tracker is designed to catch the PDN context going down (I'll be honest, I'm not familiar with what a PDN context is) - however in this fault the modem doesn't include any further notifications. I catch the low battery event, read the battery voltage with `AT%VBAT` for my logs, and then call `AT+CFUN=1` to reenable the modem. The specific low-voltage event is only caused by large current draw (such as low signal strength like you said) - so we can be confident that event has passed.

The correct fix is updating our circuit by adding a capacitor on the VDD pin (instead of through a ferrite bead) - but we also need to make what we have work and I think this fix will do so.

One other change I made was the reconnection logic. Rather than using exponential back off, I just try every minute for an hour before deciding it's never going to work and reboot the device. For our application - where devices are supposed to be online all the time - this logic will work fine.

[Achim Kraus]

> I catch the low battery event, read the battery voltage with `AT%VBAT` for my logs, and then call `AT+CFUN=1` to reenable the modem. 

So, if you have verified, that it is simply a glitch of the power rail and not the "new zephyr" codebase, `AT+CFUN=1` helps well.

Just to mention:

Using `AT+CFUN=1` will require to reenable some events or modem configs. That is usually done with "lte_lc_normal()" and the standard callbacks added by the nrf-library. For some custom stuff, a "LTE_LC_ON_CFUN" handler helps.