Spontaneous MFW downgrade from 1.0.3 to 1.0.1?

[jbrzozoski]

From everything I know and understand of the system architecture, this shouldn't be possible, but one of our early production units out in the fields has twice now apparently downgraded itself from modem firmware 1.0.3 back to 1.0.1.

It was originally sent out into the field with 1.0.1.  I have upgraded it twice remotely with the 1.0.1-to-1.0.3 binary diff.

The particular application has no carrier library, so the mobile network operator can't be doing it behind our back...

Additionally, I have never seen binary diffs provided to support a 1.0.3-to-1.0.1 path.  Assuming they don't exist, does that mean the only way this could happen is if someone went up to it with a JTAG?

Or is there some way that the modem could come up functional, claiming to be running 1.0.3, but somehow not fully complete the upgrade and lose it on the next reboot?

I'm adding more logging on the server side to watch just this one device and keep historical record of which MFW it reports and when.  I'm upgrading it a THIRD time, and just going to watch.  It seems to be functioning normally besides this one oddball bit.

[Martin Lesund]

Hi Justin,

When you are doing the modem firmware upgrade for the mfw1.0.1 -> mfw1.0.3 your device will check if the new patch is correct and working, or else it will do a rollback to the previous mfw version.

So what is most likely happening in your case is that for some reason the new patch is not approved by your application which in turn causes this rollback. The reason for such rollback can be e.g. if the new mfw image cannot set up a network connection or that your application does not purge the old mfw 1.0.1 image in time which in turn could produce this unwanted rollback of the delta image.

The fact that only one unit stubbornly returns to the old version modem firmware version can be an indication of a fault in the early startup phase after the upgrade.

Are you able to access the unit to do some debugging. e.g. turn ON logging etc. to try to look into the reason for this rollback?

Best regards,
Martin L. 

[jbrzozoski]

I unfortunately don't have a way to capture any useful logging on this unit since it is out in the field.

After the more recent time I did the 1.0.1 to 1.0.3 upgrade, I observed the device reconnect back to our cloud service, validated it was working and reporting 1.0.3, and then manually asked it to reboot again.  It cycled offline/online and came back on with modem firmware 1.0.3 a second time.  It wasn't until a couple days later that it appeared to reboot on it's own and come online running 1.0.1.

Would the fallback be triggered by any reboot? Is it possible that it is triggered differently by a watchdog versus other reboot methods?  Some of our devices do have an on-going issue with watchdog reboots when the main MQTT handling thread blocks unexpectedly.  I'm theorizing that the manual reboots didn't trigger a fallback but the watchdog could have occurred a couple days later and *did* trigger a fallback.  Is that plausible?

[Martin Lesund]

Hi Justin,

Thank you very much for your feedback and help.

Thanks to you we found a bug so that the issue you are seeing should not happen with the next mfw release.

A new mfw release will happen in the very near future that will address this.

(so please look on the download page for the next release)