Modem does not connect after Modem delta upgrade

[hjetschko]

Hi, we used the modem_delta_update example as a reference in our firmware to be able to do modem delta FOTA upgrades from an S3 bucket. We are currently using NCS V2.2 The upgrading works very well, we use it to upgrade modem firmware V1.3.3 to V1.3.4. The problem is that on some units the modem stops connecting to the LTE network after the upgrade. The modem still responds correctly to AT commands, but no matter what I try, it does not want to connect to LTE. I manually programmed modem firmware V1.3.3 and V1.3.4 to see if this will fix the problem, but no luck. Are there any non-volatile settings in the modem that could have been lost?

Thank you for your help.

Hubert

[hjetschko]

I made a mistake above, we are currently using NCS V2.1.0 for this project. Sorry about this.

[Øyvind]

Hello, 

could you please provide some logs of the issue? A modem trace would be great as well.

Thanks!

Kind regards,
Øyvind 

[Øyvind]

hjetschko said:

We do not call IRQ_CONNECT or IRQ_DIRECT_CONNECT from our code. Can you please give me advice about this?

No, this is handled by the Zephyr OS interrupts kernel service. IRQ_CONNECT is used to initialize an interrupt handler. 

hjetschko said:

Has IRQ_CONNECT or IRQ_DIRECT_CONNECT accidentally been invoked on the same irq multiple times?

This explains what is happening in your project. If this is new, then you need to check what is causing this error. How did you move the lpuart to another UART? 

[hjetschko]

Hi Øyvind

I moved our lpuart from uart1 to uart 0:

In device tree:

uart0: uart@8000 {
compatible = "nordic,nrf-uarte";
reg = < 0x8000 0x1000 >;
interrupts = < 0x8 0x1 >;
status = "okay";
current-speed = < 0x1c200 >;
pinctrl-0 = < &uart0_default >;
pinctrl-1 = < &uart0_sleep >;
pinctrl-names = "default", "sleep";
lpuart: nrf-sw-lpuart {
compatible = "nordic,nrf-sw-lpuart";
status = "okay";
label = "LPUART";
req-pin = < 0x1b >;
rdy-pin = < 0x1a >;
};
};

And for KConfig:

CONFIG_UART_0_ASYNC=y
CONFIG_UART_0_NRF_HW_ASYNC=y
CONFIG_UART_0_NRF_HW_ASYNC_TIMER=2

It works correctly on uart0. I also moved the zephyr stuff from uart 0 to uart 2:

zephyr,console = &uart2;
zephyr,shell-uart = &uart2;
zephyr,uart-mcumgr = &uart2;

To me it looks like it should free uart1. I have attached the zephyr.dts and .config files. Can you please let me know if you anything that will clash with uart 1dot_config.txt0028.zephyr.dts?

[hjetschko]

Hi Øyvind

Unfortunately I could not figure out how to fix the IRQ error. I saw there is another option which uses uart_sync.c if I set CONFIG_NRF_MODEM_LIB_TRACE_BACKEND_UART_SYNC. With this set I can capture data over the uart using the nRF Connect Trace Collector. I have done two captures similar to before for both 1.3.3 and 1.3.4 modem firmware version. Can you please have another look if you can see any reason why these modems stopped connecting after doing a delta modem upgrade?

Thanks and regards

Hubert

trace-2023-05-02T01-50-21.991Z mv1.3.3.bin

trace-2023-05-02T00-32-55.201Z mv1.3.4.bin

[Øyvind]

hjetschko said:

Can you please have another look if you can see any reason why these modems stopped connecting after doing a delta modem upgrade?

I'm afraid these did not provide any information.

hjetschko said:

The only way I could get it working was over RTT. This log was captures by J-Link RTT logger.

I think we need to retry with this method. Can you please revert back to how the project was configured to capture RTT modem trace? Following the steps provided under Modem Trace Module, please add CONFIG_NRF_MODEM_LIB_TRACE_LEVEL_LTE_AND_IP=y to prj.conf.

[Øyvind]

hjetschko said:

Can you please have another look if you can see any reason why these modems stopped connecting after doing a delta modem upgrade?

I'm afraid these did not provide any information.

hjetschko said:

The only way I could get it working was over RTT. This log was captures by J-Link RTT logger.

I think we need to retry with this method. Can you please revert back to how the project was configured to capture RTT modem trace? Following the steps provided under Modem Trace Module, please add CONFIG_NRF_MODEM_LIB_TRACE_LEVEL_LTE_AND_IP=y to prj.conf.

[hjetschko]

Hi Øyvind

Thank you for looking into this. We worked through all the commands from the AT Commands PDF to see if we can find a parameter that could affect this.  When we tried AT%XFACTORYRESET=0 the device started working again. We also tried this on a bunch of other devices that stopped working and we were able to recover all of them. Can I mark this as the solution or do you think there is still something to be concerned about?

Regards

Hubert

[Øyvind]

Hi Hubert,

hjetschko said:

Can I mark this as the solution or do you think there is still something to be concerned about?

If this fixes the issue on all your devices, then I thing we can say that this is the fix. The XFACTORYRESET might have removed some data that prevented the update to be fulfilled, but I don't know what. 

If you are happy, then let's call this a success. And if you do see any issues afterwards, please let us know. 

Kind regards,
Øyvind