Modem does not connect after Modem delta upgrade

[hjetschko]

Hi, we used the modem_delta_update example as a reference in our firmware to be able to do modem delta FOTA upgrades from an S3 bucket. We are currently using NCS V2.2 The upgrading works very well, we use it to upgrade modem firmware V1.3.3 to V1.3.4. The problem is that on some units the modem stops connecting to the LTE network after the upgrade. The modem still responds correctly to AT commands, but no matter what I try, it does not want to connect to LTE. I manually programmed modem firmware V1.3.3 and V1.3.4 to see if this will fix the problem, but no luck. Are there any non-volatile settings in the modem that could have been lost?

Thank you for your help.

Hubert

[hjetschko]

I made a mistake above, we are currently using NCS V2.1.0 for this project. Sorry about this.

[Øyvind]

Hello, 

could you please provide some logs of the issue? A modem trace would be great as well.

Thanks!

Kind regards,
Øyvind 

[Øyvind]

Thanks for providing the modem fw. Could you please redo test with both mfw 1.3.3 and 1.3.4? Please start modem trace and then reset board in order to see what happens before main. Could you please leave the trace running for a minimum of 20 minutes for each? This way we can ensure that the network is found and the connection progress is done completely.

Thanks

[hjetschko]

Hi Øyvind

Thank you for your help analyzing these files. I have done both 1.3.3 and 1.3.4 traces doing the following:

1. Started the RTT logging
2. Reset the nRF9160 firmware
3. Leave it running for 20 minutes

Please let me know if you need more information.RTTLogger_Channel_modem_trace_modemfw_1.3.3.logRTTLogger_Channel_modem_trace_modemfw_1.3.4.log

[Øyvind]

Unfortunately, the modem traces do not capture required information. Are you not able to collect modem traces via UART1? By default, nRF Connect SDK's modem library uses the UART1 peripheral for trace output

Still investigating.

[hjetschko]

Hi Øyvind

Thank you for your help. Our product used UART1 (lpuart) for communication to another micro. I moved it to another UART and made sure it works. UART1 should be free but I cannot get it to compile. This is the error I get when trying to use modem trace with UART1:

gen_isr_tables.py: error: multiple registrations at table_index 9 for irq 9 (0x9)
Existing handler 0x775a9, new handler 0x7d7c1
Has IRQ_CONNECT or IRQ_DIRECT_CONNECT accidentally been invoked on the same irq multiple times?

We do not call IRQ_CONNECT or IRQ_DIRECT_CONNECT from our code. Can you please give me advice about this?

[Øyvind]

hjetschko said:

We do not call IRQ_CONNECT or IRQ_DIRECT_CONNECT from our code. Can you please give me advice about this?

No, this is handled by the Zephyr OS interrupts kernel service. IRQ_CONNECT is used to initialize an interrupt handler. 

hjetschko said:

Has IRQ_CONNECT or IRQ_DIRECT_CONNECT accidentally been invoked on the same irq multiple times?

This explains what is happening in your project. If this is new, then you need to check what is causing this error. How did you move the lpuart to another UART? 

[hjetschko]

Hi Øyvind

I moved our lpuart from uart1 to uart 0:

In device tree:

uart0: uart@8000 {
compatible = "nordic,nrf-uarte";
reg = < 0x8000 0x1000 >;
interrupts = < 0x8 0x1 >;
status = "okay";
current-speed = < 0x1c200 >;
pinctrl-0 = < &uart0_default >;
pinctrl-1 = < &uart0_sleep >;
pinctrl-names = "default", "sleep";
lpuart: nrf-sw-lpuart {
compatible = "nordic,nrf-sw-lpuart";
status = "okay";
label = "LPUART";
req-pin = < 0x1b >;
rdy-pin = < 0x1a >;
};
};

And for KConfig:

CONFIG_UART_0_ASYNC=y
CONFIG_UART_0_NRF_HW_ASYNC=y
CONFIG_UART_0_NRF_HW_ASYNC_TIMER=2

It works correctly on uart0. I also moved the zephyr stuff from uart 0 to uart 2:

zephyr,console = &uart2;
zephyr,shell-uart = &uart2;
zephyr,uart-mcumgr = &uart2;

To me it looks like it should free uart1. I have attached the zephyr.dts and .config files. Can you please let me know if you anything that will clash with uart 1dot_config.txt0028.zephyr.dts?

[hjetschko]

Hi Øyvind

Unfortunately I could not figure out how to fix the IRQ error. I saw there is another option which uses uart_sync.c if I set CONFIG_NRF_MODEM_LIB_TRACE_BACKEND_UART_SYNC. With this set I can capture data over the uart using the nRF Connect Trace Collector. I have done two captures similar to before for both 1.3.3 and 1.3.4 modem firmware version. Can you please have another look if you can see any reason why these modems stopped connecting after doing a delta modem upgrade?

Thanks and regards

Hubert

trace-2023-05-02T01-50-21.991Z mv1.3.3.bin

trace-2023-05-02T00-32-55.201Z mv1.3.4.bin

[Øyvind]

hjetschko said:

Can you please have another look if you can see any reason why these modems stopped connecting after doing a delta modem upgrade?

I'm afraid these did not provide any information.

hjetschko said:

The only way I could get it working was over RTT. This log was captures by J-Link RTT logger.

I think we need to retry with this method. Can you please revert back to how the project was configured to capture RTT modem trace? Following the steps provided under Modem Trace Module, please add CONFIG_NRF_MODEM_LIB_TRACE_LEVEL_LTE_AND_IP=y to prj.conf.

[Øyvind]

hjetschko said:

Can you please have another look if you can see any reason why these modems stopped connecting after doing a delta modem upgrade?

I'm afraid these did not provide any information.

hjetschko said:

The only way I could get it working was over RTT. This log was captures by J-Link RTT logger.

I think we need to retry with this method. Can you please revert back to how the project was configured to capture RTT modem trace? Following the steps provided under Modem Trace Module, please add CONFIG_NRF_MODEM_LIB_TRACE_LEVEL_LTE_AND_IP=y to prj.conf.

[hjetschko]

Hi Øyvind

Thank you for looking into this. We worked through all the commands from the AT Commands PDF to see if we can find a parameter that could affect this.  When we tried AT%XFACTORYRESET=0 the device started working again. We also tried this on a bunch of other devices that stopped working and we were able to recover all of them. Can I mark this as the solution or do you think there is still something to be concerned about?

Regards

Hubert