Potential Modem Failure

[John]

We have an interesting difficulty, whereby after a period of time which varies never longer than 12 hours, often around 6 hours, our application disconnects from the network and attempts to reconnect and never manages to.

We are using V1.0.0 and the latest modem firmware and Segger running on Windows 10.

We are using a modified version of lte_ble_gateway.connecting to AWS

The connection is CAT M1 and our software is running on a nRF9160-DK (Version 0.8.5)

Our log (attached) shows that the modem is unresponsive at the time or after the disconnect as evidenced by the AT command at the end of the log.

The operative two lines of code that should get executed are as follows:

It seems like the write of AT+CFUN=0 never returns due to the modem being (kind of ) dead for whatever reason.

Therefore it may follow that the bsd_shutdown(); never happens.

err = lte_lc_power_off();
bsd_shutdown();

We can imagine that everything else seems to be OK, as we continue to receive data from the UART whilst in a disconnected state.

To make matters even more interesting, if we connect to a different carrier it seems that the problem almost evaporates. I say almost as we have had our software running for many days now at two sites, one of which required a reset, whilst the other continues without failure.

I know you will ask me for a modem trace which I'm currently in the process of capturing.

If the nRF9160-DK is reset it will run for another few hours, before failure.

Another question that I have is would you please explain if there is any difficulty/danger in replacing the modem shutdown and reboot with  the following:

sys_reboot(SYS_REBOOT_COLD);

My concern is that if the modem is constantly wring to NVM and our timing was "perfect" is there the possibility of corrupting the modem flash?

2019-07-14T04_11_29.739Z-log.txt

[Håkon Alseth]

Hi,

 

Sorry for the late reply. I can confirm that there is an open issue internally on bsd_shutdown() where the current consumption is left high after it is done, which is being looked into. However; given your symptoms, it seems that this is a new issue.

You mention that this is mostly visible on a single carrier. Is there anything out-of-the-ordinary, like low signal strength or similar?

PS: Your trace file is empty (just filled with zeros). Did you remember to give the AT command "AT%XMODEMTRACE=1,2" after booting?

If you have erased the nRF52 firmware on the DK, you might need to reprogram it to get the UART pins outputted to the segger IC. You can find a .zip file here: https://devzone.nordicsemi.com/nordic/nordic-blog/b/blog/posts/how-to-get-modem-trace-using-trace-collector-in-nrf-connect

 

Kind regards,

Håkon

[John]

Yes I do have low signal strength where the test has been running.

I'll rerun the trace now that you have shown me what to do.

Kind regards

John

[John]

Hakon, you touched on something interesting regarding current draw.

In all cases when we fail to reconnect, the current draw is fixed at 33 mA

May I ask if the two lines of code.

bsd_shutdown();

and

sys_reboot(SYS_REBOOT_COLD);

Ultimately react with the modem in a similar manner, whilst trying to achieve a graceful shutdown, which demonstrably seems to not be occurring?

[John]

Replied to myself

[John]

Replied to myself

[John]

A new piece of information that is more than a little bit disturbing, is if I downgrade and use an nrf9160_DK 0.8.2 with the same application and the latest version of the modem.software.

The problem that we were experiencing with the carrier goes away.

I'd be grateful if you would explain the EXACT steps that you would perform if an erase all was completed with NRF Connect Programmer and how to load the modem firmware after that has been done.