Serial LTE Modem Application (NRF9160) FOTA does not apply new update

[ronald-blackline]

Hi,

I have the SLM application running in a custom board with uses the NRF9160. The SLM is driven an external microprocessor within the same board. Everything seems to work but FOTA will not apply the update.

Basically, we send the following commands after we confirmed successful network registration:

AT#XFOTA=8
AT#XFOTA=1,"">xxxxxx.amazonaws.com/app_update.bin"

This sequence of commands downloaded the FOTA image successfully. Then, we apply the AT#RESET command and this is what we are seeing from RTT

00> *** Booting Zephyr OS build v2.7.0-ncs1 ***
00> I: Starting bootloader
00> I: Primary image: magic=good, swap_type=0x4, copy_done=0x1, image_ok=0x1
00> I: Secondary image: magic=good, swap_type=0x2, copy_done=0x3, image_ok=0x3
00> I: Boot source: none
00> I: Swap type: test
00> D: erasing trailer; fa_id=3
00> D: initializing status; fa_id=3
00> D: writing swap_info; fa_id=3 off=0x6ffd8 (0x7ffd8), swap_type=0x2 image_num=0x0
00> D: writing swap_size; fa_id=3 off=0x6ffd0 (0x7ffd0)
00> D: writing magic; fa_id=3 off=0x6fff0 (0x7fff0)
00> D: erasing trailer; fa_id=7
00> D: writing copy_done; fa_id=3 off=0x6ffe0 (0x7ffe0)
00> I: Bootloader chainload address offset: 0x10000
00> I: Jumping to the first image slot
00> *** Booting Zephyr OS build v2.7.0-ncs1 ***
00> I: Starting bootloader
00> I: Primary image: magic=good, swap_type=0x2, copy_done=0x1, image_ok=0x3
00> I: Secondary image: magic=unset, swap_type=0x1, copy_done=0x3, image_ok=0x3
00> I: Boot source: none
00> I: Swap type: revert
00> I: Secondary image: magic=unset, swap_type=0x1, copy_done=0x3, image_ok=0x3
00> D: erasing trailer; fa_id=7
00> D: writing image_ok; fa_id=7 off=0x6ffe8 (0xeffe8)
00> D: writing swap_size; fa_id=7 off=0x6ffd0 (0xeffd0)
00> D: writing magic; fa_id=7 off=0x6fff0 (0xefff0)
00> D: erasing trailer; fa_id=3
00> D: initializing status; fa_id=3
00> D: writing swap_info; fa_id=3 off=0x6ffd8 (0x7ffd8), swap_type=0x4 image_num=0x0
00> D: writing image_ok; fa_id=3 off=0x6ffe8 (0x7ffe8)
00> D: writing swap_size; fa_id=3 off=0x6ffd0 (0x7ffd0)
00> D: writing magic; fa_id=3 off=0x6fff0 (0x7fff0)
00> D: erasing trailer; fa_id=7
00> D: writing copy_done; fa_id=3 off=0x6ffe0 (0x7ffe0)
00> I: Bootloader chainload address offset: 0x10000
00> I: Jumping to the first image slot

After this, the SLM gets run again and we read the following response from the application:

b'\x00Ready\r\n'
b'\r\n'
b'#XFOTA: 5,1,-9\r\n'

This basically indicates failure in the FOTA process. Can someone please help.

[Håkon Alseth]

Hi,

 

Given that it downloads, resets and prints this:

00> I: Swap type: test

It means that mcuboot recognizes this image, but on the very next sequence:

00> I: Jumping to the first image slot
00> *** Booting Zephyr OS build v2.7.0-ncs1 ***
00> I: Starting bootloader
00> I: Primary image: magic=good, swap_type=0x2, copy_done=0x1, image_ok=0x3
00> I: Secondary image: magic=unset, swap_type=0x1, copy_done=0x3, image_ok=0x3
00> I: Boot source: none
00> I: Swap type: revert

You are essentially trying to boot the new image, and you're reset again, where it reverts to the original image.

 

This can happen if the configuration differs from what mcuboot expects, for instance in the flash layout of the secondary/primary slot.

Could you share details on your original projects "partitions.yml" and/or "pm_static.yml", as well as the "update project"'s partition layout?

It is very important that you follow the partition layout of the original image for all update images.

 

Kind regards,

Håkon

[ronald-blackline]

So i fixed the issue "Protect mcuboot flash failed, cancel startup" by defining "CONFIG_FPROTECT=n" in the mcuboot.conf (in "serial_lte_modem/child_image" folder). However, even with the board files ported properly, Im back seeing the original problem:

00> *** Booting Zephyr OS build v2.7.0-ncs1 ***
00> I: Starting bootloader
00> I: Primary image: magic=good, swap_type=0x4, copy_done=0x1, image_ok=0x1
00> I: Secondary image: magic=good, swap_type=0x2, copy_done=0x3, image_ok=0x3
00> I: Boot source: none
00> I: Swap type: test
00> D: erasing trailer; fa_id=2
00> D: initializing status; fa_id=2
00> D: writing swap_info; fa_id=2 off=0x68fd8 (0x74fd8), swap_type=0x2 image_num=0x0
00> D: writing swap_size; fa_id=2 off=0x68fd0 (0x74fd0)
00> D: writing magic; fa_id=2 off=0x68ff0 (0x74ff0)
00> D: erasing trailer; fa_id=6
00> D: writing copy_done; fa_id=2 off=0x68fe0 (0x74fe0)
00> I: Bootloader chainload address offset: 0xc000
00> I: Jumping to the first image slot
00> *** Booting Zephyr OS build v2.7.0-ncs1 ***
00> I: Starting bootloader
00> I: Primary image: magic=good, swap_type=0x2, copy_done=0x1, image_ok=0x3
00> I: Secondary image: magic=unset, swap_type=0x1, copy_done=0x3, image_ok=0x3
00> I: Boot source: none
00> I: Swap type: revert
00> I: Secondary image: magic=unset, swap_type=0x1, copy_done=0x3, image_ok=0x3
00> D: erasing trailer; fa_id=6
00> D: writing image_ok; fa_id=6 off=0x68fe8 (0xddfe8)
00> D: writing swap_size; fa_id=6 off=0x68fd0 (0xddfd0)
00> D: writing magic; fa_id=6 off=0x68ff0 (0xddff0)
00> D: erasing trailer; fa_id=2
00> D: initializing status; fa_id=2
00> D: writing swap_info; fa_id=2 off=0x68fd8 (0x74fd8), swap_type=0x4 image_num=0x0
00> D: writing image_ok; fa_id=2 off=0x68fe8 (0x74fe8)
00> D: writing swap_size; fa_id=2 off=0x68fd0 (0x74fd0)
00> D: writing magic; fa_id=2 off=0x68ff0 (0x74ff0)
00> D: erasing trailer; fa_id=6
00> D: writing copy_done; fa_id=2 off=0x68fe0 (0x74fe0)
00> I: Bootloader chainload address offset: 0xc000
00> I: Jumping to the first image slot

I already attached the file "pm_static.yml" in the previous reply. Below is the generated "partitions.yml" file:

EMPTY_0:
  address: 0xfc000
  end_address: 0xfe000
  region: flash_primary
  size: 0x2000
app:
  address: 0x18200
  end_address: 0x75000
  region: flash_primary
  size: 0x5ce00
mcuboot:
  address: 0x0
  end_address: 0xc000
  placement:
    before:
    - mcuboot_primary
  region: flash_primary
  size: 0xc000
mcuboot_pad:
  address: 0xc000
  end_address: 0xc200
  placement:
    align:
      start: 0x1000
    before:
    - mcuboot_primary_app
  region: flash_primary
  size: 0x200
mcuboot_primary:
  address: 0xc000
  end_address: 0x75000
  region: flash_primary
  size: 0x69000
  span:
  - spm
  - mcuboot_pad
  - app
mcuboot_primary_app:
  address: 0xc200
  end_address: 0x75000
  region: flash_primary
  size: 0x68e00
  span:
  - app
  - spm
mcuboot_scratch:
  address: 0xde000
  end_address: 0xfc000
  placement:
    after:
    - app
    align:
      start: 0x1000
  region: flash_primary
  size: 0x1e000
mcuboot_secondary:
  address: 0x75000
  end_address: 0xde000
  placement:
    after:
    - mcuboot_primary
    align:
      start: 0x1000
  region: flash_primary
  share_size:
  - mcuboot_primary
  size: 0x69000
nrf_modem_lib_ctrl:
  address: 0x20010000
  end_address: 0x200104e8
  inside:
  - sram_nonsecure
  placement:
    after:
    - spm_sram
    - start
  region: sram_primary
  size: 0x4e8
nrf_modem_lib_rx:
  address: 0x200124e8
  end_address: 0x200144e8
  inside:
  - sram_nonsecure
  placement:
    after:
    - nrf_modem_lib_tx
  region: sram_primary
  size: 0x2000
nrf_modem_lib_sram:
  address: 0x20010000
  end_address: 0x200144e8
  orig_span: &id001
  - nrf_modem_lib_ctrl
  - nrf_modem_lib_tx
  - nrf_modem_lib_rx
  region: sram_primary
  size: 0x44e8
  span: *id001
nrf_modem_lib_tx:
  address: 0x200104e8
  end_address: 0x200124e8
  inside:
  - sram_nonsecure
  placement:
    after:
    - nrf_modem_lib_ctrl
  region: sram_primary
  size: 0x2000
otp:
  address: 0xff8108
  end_address: 0xff83fc
  region: otp
  size: 0x2f4
settings_storage:
  address: 0xfe000
  end_address: 0x100000
  placement:
    after:
    - mcuboot_scratch
  region: flash_primary
  size: 0x2000
spm:
  address: 0xc200
  end_address: 0x18200
  inside:
  - mcuboot_primary_app
  placement:
    before:
    - app
  region: flash_primary
  size: 0xc000
spm_sram:
  address: 0x20000000
  end_address: 0x20010000
  inside:
  - sram_secure
  placement:
    after:
    - start
  region: sram_primary
  size: 0x10000
sram_nonsecure:
  address: 0x20010000
  end_address: 0x20040000
  orig_span: &id002
  - sram_primary
  - nrf_modem_lib_ctrl
  - nrf_modem_lib_tx
  - nrf_modem_lib_rx
  region: sram_primary
  size: 0x30000
  span: *id002
sram_primary:
  address: 0x200144e8
  end_address: 0x20040000
  region: sram_primary
  size: 0x2bb18
sram_secure:
  address: 0x20000000
  end_address: 0x20010000
  orig_span: &id003
  - spm_sram
  region: sram_primary
  size: 0x10000
  span: *id003

Let me know if you need more files. Thanks!

[ronald-blackline]

i tried programming the app_moved_test_update.hex but the result is the same, getting reverted back.

This is also how i changed the main code (snippet of start_execute):

	LOG_INF("Serial LTE Modem");

	/* Init and load settings */
	err = slm_settings_init();
	if (err) {
		LOG_ERR("Failed to init slm settings: %d", err);
		return;
	}

	LOG_DBG("fota_stage: %d", fota_stage);
	///* Post-FOTA handling */
	//if (fota_stage != FOTA_STAGE_INIT) {
	//	handle_nrf_modem_lib_init_ret();
	//	handle_mcuboot_swap_ret();
	//}

	err = boot_write_img_confirmed();
	if (err) {
		fota_status = FOTA_STATUS_ERROR;
		fota_info = err;
	} else {
		fota_status = FOTA_STATUS_OK;
		fota_info = 0;
	}

Other than the change in slm_at_host, have you changed something else?

[Håkon Alseth]

Hi,

 

ronald-blackline said:

i tried programming the app_moved_test_update.hex but the result is the same, getting reverted back.

Are you certain that the original pm_static.yml is the one that you're currently using?

Meaning, are you certain that the static addresses are indeed matching those that your mcuboot expects?

 

I do not see this by using your initial pm_static.yml, but that might also be because I use that as the "starting point" to ensure that the partition layout is equal for the pending update image.

 

What I did for testing your process:

* used your pm_static.yml, and set FPROTECT off.

* Configured ncs v1.5.0 SLM and flashed that.

* checked out ncs v1.8.0, reconfigured SLM

* Changed the printout, just to see that the image booted (from "Ready" to "Update Ready")

* flashed the ncs v1.8.0 "app_moved_test_update.hex" using nrfjprog into the mcuboot secondary slot (starting at 0x75000).

* Reset the board, and saw that the image booted and printed my modified string (ie. Update Ready).

 

* (optional step) call boot_write_img_confirmed() on the ncs v1.8.0 image to make the new image permanent. If this is not done, the image will revert to ncs v1.5.0 on the next reset.

 

Kind regards,

Håkon

[ronald-blackline]

Håkon Alseth said:

Meaning, are you certain that the static addresses are indeed matching those that your mcuboot expects?

How do I verify this?

[Håkon Alseth]

Hi,

 

ronald-blackline said:

How do I verify this?

Do you have the original build (and artifacts, such as the partitions.yml?)?

 

Could you confirm if the "update images" gets executed at all, or is it specifically only the "boot_write_img_confirmed()" that is not making the image permanent? From the logs that you have provided, it seems that the app_*.hex files does not seem to run properly.

  

Kind regards,

Håkon

[ronald-blackline]

I was able to make it work but I have to remove one of the required configuration I have to use in my project. I have this config defined:

CONFIG_SLM_START_SLEEP=y

Whenever I removed this configuration in the project, the actual image swap is successful. If I put it back in, the swap reverts back (thus failing). I investigated and whenever the following APIs are called (in main.c) ...

nrf_power_resetreas_get(NRF_POWER_NS);
nrf_power_resetreas_clear(NRF_POWER_NS, 0x70017);

... the swap fails.

As a workaround I moved the following calls before the *nrf_power_resetreas_* APIs:

handle_nrf_modem_lib_init_ret();
handle_mcuboot_swap_ret();

This actually makes swapping successful although the FOTA completion message says otherwise:

#XFOTA: 5,1,-9

This is the last issue Im looking and I think the swapping issue is now behind us.

[Håkon Alseth]

Hi,

 

Thank you very much for the detailed update.

ronald-blackline said:

I was able to make it work but I have to remove one of the required configuration I have to use in my project. I have this config defined:

CONFIG_SLM_START_SLEEP=y

Whenever I removed this configuration in the project, the actual image swap is successful. If I put it back in, the swap reverts back (thus failing). I investigated and whenever the following APIs are called (in main.c) ...

nrf_power_resetreas_get(NRF_POWER_NS);
nrf_power_resetreas_clear(NRF_POWER_NS, 0x70017);

... the swap fails.

Yes, I can see that this scenario will fail on my end as well. I have submitted a bug report internally on the SLM project to make the developers aware of this issue! Thank you for updating and helping us improve the application!

 

ronald-blackline said:

As a workaround I moved the following calls before the *nrf_power_resetreas_* APIs:

handle_nrf_modem_lib_init_ret();
handle_mcuboot_swap_ret();

This actually makes swapping successful although the FOTA completion message says otherwise:

#XFOTA: 5,1,-9

This is the last issue Im looking and I think the swapping issue is now behind us.

I believe that the XFOTA return error is a issue that was recently fixed in ncs v1.9.0:

https://github.com/nrfconnect/sdk-nrf/pull/6918

 

Kind regards,

Håkon

[Håkon Alseth]

Hi,

ronald-blackline said:

I was able to make it work but I have to remove one of the required configuration I have to use in my project. I have this config defined:

CONFIG_SLM_START_SLEEP=y

Whenever I removed this configuration in the project, the actual image swap is successful. If I put it back in, the swap reverts back (thus failing). I investigated and whenever the following APIs are called (in main.c) ...

nrf_power_resetreas_get(NRF_POWER_NS);
nrf_power_resetreas_clear(NRF_POWER_NS, 0x70017);

... the swap fails.

As a workaround I moved the following calls before the *nrf_power_resetreas_* APIs:

handle_nrf_modem_lib_init_ret();
handle_mcuboot_swap_ret();

I just wanted to notify you that we have a proposed fix merged to main branch now:

https://github.com/nrfconnect/sdk-nrf/pull/7040

 

Kind regards,

Håkon