Serial LTE Modem Application (NRF9160) FOTA does not apply new update

RE: Serial LTE Modem Application (NRF9160) FOTA does not apply new update

Håkon Alseth — Wed, 09 Mar 2022 10:01:43 GMT

Hi,

[quote user="ronald-swiftlabs"]

I was able to make it work but I have to remove one of the required configuration I have to use in my project. I have this config defined:

CONFIG_SLM_START_SLEEP=y

Whenever I removed this configuration in the project, the actual image swap is successful. If I put it back in, the swap reverts back (thus failing). I investigated and whenever the following APIs are called (in main.c) ...

nrf_power_resetreas_get(NRF_POWER_NS);
nrf_power_resetreas_clear(NRF_POWER_NS, 0x70017);

... the swap fails.

As a workaround I moved the following calls before the *nrf_power_resetreas_* APIs:

handle_nrf_modem_lib_init_ret();
handle_mcuboot_swap_ret();

[/quote]

I just wanted to notify you that we have a proposed fix merged to main branch now:

https://github.com/nrfconnect/sdk-nrf/pull/7040

Kind regards,

Håkon

RE: Serial LTE Modem Application (NRF9160) FOTA does not apply new update

Håkon Alseth — Thu, 24 Feb 2022 08:57:54 GMT

Hi,

Thank you very much for the detailed update.

[quote user="ronald-swiftlabs"]

I was able to make it work but I have to remove one of the required configuration I have to use in my project. I have this config defined:

CONFIG_SLM_START_SLEEP=y

nrf_power_resetreas_get(NRF_POWER_NS);
nrf_power_resetreas_clear(NRF_POWER_NS, 0x70017);

... the swap fails.

[/quote]

Yes, I can see that this scenario will fail on my end as well. I have submitted a bug report internally on the SLM project to make the developers aware of this issue! Thank you for updating and helping us improve the application!

[quote user="ronald-swiftlabs"]

As a workaround I moved the following calls before the *nrf_power_resetreas_* APIs:

handle_nrf_modem_lib_init_ret();
handle_mcuboot_swap_ret();

This actually makes swapping successful although the FOTA completion message says otherwise:

#XFOTA: 5,1,-9

This is the last issue Im looking and I think the swapping issue is now behind us.

[/quote]

I believe that the XFOTA return error is a issue that was recently fixed in ncs v1.9.0:

https://github.com/nrfconnect/sdk-nrf/pull/6918

Kind regards,

Håkon

RE: Serial LTE Modem Application (NRF9160) FOTA does not apply new update

ronald-blackline — Wed, 23 Feb 2022 17:53:31 GMT

I was able to make it work but I have to remove one of the required configuration I have to use in my project. I have this config defined:

CONFIG_SLM_START_SLEEP=y

nrf_power_resetreas_get(NRF_POWER_NS);
nrf_power_resetreas_clear(NRF_POWER_NS, 0x70017);

... the swap fails.

As a workaround I moved the following calls before the *nrf_power_resetreas_* APIs:

handle_nrf_modem_lib_init_ret();
handle_mcuboot_swap_ret();

This actually makes swapping successful although the FOTA completion message says otherwise:

#XFOTA: 5,1,-9

This is the last issue Im looking and I think the swapping issue is now behind us.

RE: Serial LTE Modem Application (NRF9160) FOTA does not apply new update

Håkon Alseth — Wed, 23 Feb 2022 13:24:19 GMT

Hi,

[quote user="ronald-swiftlabs"]How do I verify this?[/quote]

Do you have the original build (and artifacts, such as the partitions.yml?)?

Could you confirm if the "update images" gets executed at all, or is it specifically only the "boot_write_img_confirmed()" that is not making the image permanent? From the logs that you have provided, it seems that the app_*.hex files does not seem to run properly.

Kind regards,

Håkon

RE: Serial LTE Modem Application (NRF9160) FOTA does not apply new update

ronald-blackline — Tue, 22 Feb 2022 15:37:35 GMT

[quote userid="2115" url="~/f/nordic-q-a/84796/serial-lte-modem-application-nrf9160-fota-does-not-apply-new-update/354079#354079"]Meaning, are you certain that the static addresses are indeed matching those that your mcuboot expects?[/quote]

How do I verify this?

RE: Serial LTE Modem Application (NRF9160) FOTA does not apply new update

Håkon Alseth — Mon, 21 Feb 2022 09:57:05 GMT

Hi,

[quote user="ronald-swiftlabs"]i tried programming the app_moved_test_update.hex but the result is the same, getting reverted back.[/quote]

Are you certain that the original pm_static.yml is the one that you're currently using?

Meaning, are you certain that the static addresses are indeed matching those that your mcuboot expects?

I do not see this by using your initial pm_static.yml, but that might also be because I use that as the "starting point" to ensure that the partition layout is equal for the pending update image.

What I did for testing your process:

* used your pm_static.yml, and set FPROTECT off.

* Configured ncs v1.5.0 SLM and flashed that.

* checked out ncs v1.8.0, reconfigured SLM

* Changed the printout, just to see that the image booted (from "Ready" to "Update Ready")

* flashed the ncs v1.8.0 "app_moved_test_update.hex" using nrfjprog into the mcuboot secondary slot (starting at 0x75000).

* Reset the board, and saw that the image booted and printed my modified string (ie. Update Ready).

* (optional step) call boot_write_img_confirmed() on the ncs v1.8.0 image to make the new image permanent. If this is not done, the image will revert to ncs v1.5.0 on the next reset.

Kind regards,

Håkon

RE: Serial LTE Modem Application (NRF9160) FOTA does not apply new update

ronald-blackline — Fri, 18 Feb 2022 15:08:16 GMT

i tried programming the app_moved_test_update.hex but the result is the same, getting reverted back.

This is also how i changed the main code (snippet of start_execute):

	LOG_INF("Serial LTE Modem");

	/* Init and load settings */
	err = slm_settings_init();
	if (err) {
		LOG_ERR("Failed to init slm settings: %d", err);
		return;
	}

	LOG_DBG("fota_stage: %d", fota_stage);
	///* Post-FOTA handling */
	//if (fota_stage != FOTA_STAGE_INIT) {
	//	handle_nrf_modem_lib_init_ret();
	//	handle_mcuboot_swap_ret();
	//}

	err = boot_write_img_confirmed();
	if (err) {
		fota_status = FOTA_STATUS_ERROR;
		fota_info = err;
	} else {
		fota_status = FOTA_STATUS_OK;
		fota_info = 0;
	}

Other than the change in slm_at_host, have you changed something else?

RE: Serial LTE Modem Application (NRF9160) FOTA does not apply new update

Håkon Alseth — Fri, 18 Feb 2022 09:01:46 GMT

Hi,

[quote user="ronald-swiftlabs"]First question is, are the files (pm_static.yml and paritions.yml) looks correct?[/quote]

Yes, they match.

[quote user="ronald-swiftlabs"]

Second, after the swap, when it jumps to the image in slot0, does that mean it is calling the "update"?

[/quote]

"test" means that it is now copying the image from the secondary slot to the primary slot, and will boot it.

[quote user="ronald-swiftlabs"]As for the call to boot_write_img_confirmed(), i forced calling that function in the current firmware and in the new update application.[/quote]

In the build-folder/zephyr/ folder, there should be several .hex files. If you have hardcoded the new application to call boot_write_img_confirmed(), you can flash the file:

app_moved_test_update.hex

using nrfjprog directly:

nrfjprog --program build-folder/zephyr/app_moved_test_update.hex --sectorerase -f nrf91

Then either reset via a button, or "nrfjprog --reset".

Does this procedure work as expected on your end?

I flashed SLM to my board, saw that it booted as expected, by printing "Ready" on the UART console.

Then changed the string in slm_at_host.c::SLM_SYNC_STR (line 27 in ncs v1.8.0), and flashed the "app_moved_test_update.hex" directly, and saw that it now printed my changed string.

Kind regards,

Håkon

RE: Serial LTE Modem Application (NRF9160) FOTA does not apply new update

ronald-blackline — Thu, 17 Feb 2022 14:39:33 GMT

First question is, are the files (pm_static.yml and paritions.yml) looks correct?

Second, after the swap, when it jumps to the image in slot0, does that mean it is calling the "update"?

As for the call to boot_write_img_confirmed(), i forced calling that function in the current firmware and in the new update application.

Im also using the SLM application which sees this issue. However, the http_update application works fine. So im thinking there is something different the SLM is doing compare to the http_update.

RE: Serial LTE Modem Application (NRF9160) FOTA does not apply new update

Håkon Alseth — Thu, 17 Feb 2022 14:07:28 GMT

Hi,

It is hard to say what happens here, as there's no application logs between the "test" and the "revert" boot log.

In your new update application, do you call "boot_write_img_confirmed()" after booting?

If this function is not called, then it will revert on the next reset.

Does the application reboot itself, or do you manually trigger a reset?

Kind regards,

Håkon

RE: Serial LTE Modem Application (NRF9160) FOTA does not apply new update

ronald-blackline — Wed, 16 Feb 2022 19:51:42 GMT

So i fixed the issue "Protect mcuboot flash failed, cancel startup" by defining "CONFIG_FPROTECT=n" in the mcuboot.conf (in "serial_lte_modem/child_image" folder). However, even with the board files ported properly, Im back seeing the original problem:

00> *** Booting Zephyr OS build v2.7.0-ncs1 ***
00> I: Starting bootloader
00> I: Primary image: magic=good, swap_type=0x4, copy_done=0x1, image_ok=0x1
00> I: Secondary image: magic=good, swap_type=0x2, copy_done=0x3, image_ok=0x3
00> I: Boot source: none
00> I: Swap type: test
00> D: erasing trailer; fa_id=2
00> D: initializing status; fa_id=2
00> D: writing swap_info; fa_id=2 off=0x68fd8 (0x74fd8), swap_type=0x2 image_num=0x0
00> D: writing swap_size; fa_id=2 off=0x68fd0 (0x74fd0)
00> D: writing magic; fa_id=2 off=0x68ff0 (0x74ff0)
00> D: erasing trailer; fa_id=6
00> D: writing copy_done; fa_id=2 off=0x68fe0 (0x74fe0)
00> I: Bootloader chainload address offset: 0xc000
00> I: Jumping to the first image slot
00> *** Booting Zephyr OS build v2.7.0-ncs1 ***
00> I: Starting bootloader
00> I: Primary image: magic=good, swap_type=0x2, copy_done=0x1, image_ok=0x3
00> I: Secondary image: magic=unset, swap_type=0x1, copy_done=0x3, image_ok=0x3
00> I: Boot source: none
00> I: Swap type: revert
00> I: Secondary image: magic=unset, swap_type=0x1, copy_done=0x3, image_ok=0x3
00> D: erasing trailer; fa_id=6
00> D: writing image_ok; fa_id=6 off=0x68fe8 (0xddfe8)
00> D: writing swap_size; fa_id=6 off=0x68fd0 (0xddfd0)
00> D: writing magic; fa_id=6 off=0x68ff0 (0xddff0)
00> D: erasing trailer; fa_id=2
00> D: initializing status; fa_id=2
00> D: writing swap_info; fa_id=2 off=0x68fd8 (0x74fd8), swap_type=0x4 image_num=0x0
00> D: writing image_ok; fa_id=2 off=0x68fe8 (0x74fe8)
00> D: writing swap_size; fa_id=2 off=0x68fd0 (0x74fd0)
00> D: writing magic; fa_id=2 off=0x68ff0 (0x74ff0)
00> D: erasing trailer; fa_id=6
00> D: writing copy_done; fa_id=2 off=0x68fe0 (0x74fe0)
00> I: Bootloader chainload address offset: 0xc000
00> I: Jumping to the first image slot

I already attached the file "pm_static.yml" in the previous reply. Below is the generated "partitions.yml" file:

EMPTY_0:
  address: 0xfc000
  end_address: 0xfe000
  region: flash_primary
  size: 0x2000
app:
  address: 0x18200
  end_address: 0x75000
  region: flash_primary
  size: 0x5ce00
mcuboot:
  address: 0x0
  end_address: 0xc000
  placement:
    before:
    - mcuboot_primary
  region: flash_primary
  size: 0xc000
mcuboot_pad:
  address: 0xc000
  end_address: 0xc200
  placement:
    align:
      start: 0x1000
    before:
    - mcuboot_primary_app
  region: flash_primary
  size: 0x200
mcuboot_primary:
  address: 0xc000
  end_address: 0x75000
  region: flash_primary
  size: 0x69000
  span:
  - spm
  - mcuboot_pad
  - app
mcuboot_primary_app:
  address: 0xc200
  end_address: 0x75000
  region: flash_primary
  size: 0x68e00
  span:
  - app
  - spm
mcuboot_scratch:
  address: 0xde000
  end_address: 0xfc000
  placement:
    after:
    - app
    align:
      start: 0x1000
  region: flash_primary
  size: 0x1e000
mcuboot_secondary:
  address: 0x75000
  end_address: 0xde000
  placement:
    after:
    - mcuboot_primary
    align:
      start: 0x1000
  region: flash_primary
  share_size:
  - mcuboot_primary
  size: 0x69000
nrf_modem_lib_ctrl:
  address: 0x20010000
  end_address: 0x200104e8
  inside:
  - sram_nonsecure
  placement:
    after:
    - spm_sram
    - start
  region: sram_primary
  size: 0x4e8
nrf_modem_lib_rx:
  address: 0x200124e8
  end_address: 0x200144e8
  inside:
  - sram_nonsecure
  placement:
    after:
    - nrf_modem_lib_tx
  region: sram_primary
  size: 0x2000
nrf_modem_lib_sram:
  address: 0x20010000
  end_address: 0x200144e8
  orig_span: &id001
  - nrf_modem_lib_ctrl
  - nrf_modem_lib_tx
  - nrf_modem_lib_rx
  region: sram_primary
  size: 0x44e8
  span: *id001
nrf_modem_lib_tx:
  address: 0x200104e8
  end_address: 0x200124e8
  inside:
  - sram_nonsecure
  placement:
    after:
    - nrf_modem_lib_ctrl
  region: sram_primary
  size: 0x2000
otp:
  address: 0xff8108
  end_address: 0xff83fc
  region: otp
  size: 0x2f4
settings_storage:
  address: 0xfe000
  end_address: 0x100000
  placement:
    after:
    - mcuboot_scratch
  region: flash_primary
  size: 0x2000
spm:
  address: 0xc200
  end_address: 0x18200
  inside:
  - mcuboot_primary_app
  placement:
    before:
    - app
  region: flash_primary
  size: 0xc000
spm_sram:
  address: 0x20000000
  end_address: 0x20010000
  inside:
  - sram_secure
  placement:
    after:
    - start
  region: sram_primary
  size: 0x10000
sram_nonsecure:
  address: 0x20010000
  end_address: 0x20040000
  orig_span: &id002
  - sram_primary
  - nrf_modem_lib_ctrl
  - nrf_modem_lib_tx
  - nrf_modem_lib_rx
  region: sram_primary
  size: 0x30000
  span: *id002
sram_primary:
  address: 0x200144e8
  end_address: 0x20040000
  region: sram_primary
  size: 0x2bb18
sram_secure:
  address: 0x20000000
  end_address: 0x20010000
  orig_span: &id003
  - spm_sram
  region: sram_primary
  size: 0x10000
  span: *id003

Let me know if you need more files. Thanks!

RE: Serial LTE Modem Application (NRF9160) FOTA does not apply new update

ronald-blackline — Wed, 16 Feb 2022 17:43:20 GMT

Im getting now this error:

00> *** Booting Zephyr OS build v2.7.0-ncs1 ***
00> I: Starting bootloader
00> I: Primary image: magic=unset, swap_type=0x1, copy_done=0x3, image_ok=0x3
00> I: Secondary image: magic=unset, swap_type=0x1, copy_done=0x3, image_ok=0x3
00> I: Boot source: none
00> I: Swap type: none
00> I: Bootloader chainload address offset: 0xc000
00> I: Jumping to the first image slot
00> E: Protect mcuboot flash failed, cancel startup.

And here is the content of the pm_static.yml file:

app: {address: 0x18200, size: 0x5ae00}
mcuboot:
  address: 0x0
  placement:
    before: [mcuboot_primary]
  size: 0xc000
mcuboot_pad:
  address: 0xc000
  placement:
    align: {start: 0x1000}
    before: [mcuboot_primary_app]
  size: 0x200
mcuboot_primary:
  address: 0xc000
  size: 0x69000
  span: [spm, mcuboot_pad, app]
mcuboot_primary_app:
  address: 0xc200
  size: 0x68e00
  span: [app, spm]
mcuboot_scratch:
  address: 0xde000
  placement:
    after: [app]
    align: {start: 0x1000}
  size: 0x1e000
mcuboot_secondary:
  address: 0x75000
  placement:
    after: [mcuboot_primary]
    align: {start: 0x1000}
  share_size: [mcuboot_primary]
  size: 0x69000
EMPTY_0:
  address: 0xfc000
  size: 0x2000
settings_storage:
  address: 0xfe000
  placement:
    after: [mcuboot_scratch]
  size: 0x2000
spm:
  address: 0xc200
  inside: [mcuboot_primary_app]
  placement:
    before: [app]
  size: 0xc000

And for "board_partition_conf.dts"

&slot0_partition {
    reg = <0x00010000 0x30000>;
};

&slot0_ns_partition {
    reg = <0x00040000 0x40000>;
};

&slot1_partition {
    reg = <0x00080000 0x30000>;
};

&slot1_ns_partition {
    reg = <0x000b0000 0x40000>;
};

/* Default SRAM planning when building for nRF9160 with
 * ARM TrustZone-M support
 * - Lowest 64 kB SRAM allocated to Secure image (sram0).
 * - 64 kB SRAM reserved for and used by the BSD socket
 *   library.
 * - Upper 128 kB allocated to Non-Secure image (sram0_ns).
 */

&sram0_s {
    reg = <0x20000000 DT_SIZE_K(64)>;
};

&sram0_bsd {
    reg = <0x20010000 DT_SIZE_K(64)>;
};

&sram0_ns {
    reg = <0x20020000 DT_SIZE_K(128)>;
};

RE: Serial LTE Modem Application (NRF9160) FOTA does not apply new update

Håkon Alseth — Wed, 16 Feb 2022 15:06:49 GMT

Let me know if you run into any issues, or have questions!

Kind regards,

Håkon

RE: Serial LTE Modem Application (NRF9160) FOTA does not apply new update

ronald-blackline — Wed, 16 Feb 2022 14:27:35 GMT

Thank you for pointing that one. After looking at those files you requested, looks like I need to re-implement the board files since I had it initially implemented using NCS v1.5.0.

I will get back to you regardless if it fixed the issue or not.

RE: Serial LTE Modem Application (NRF9160) FOTA does not apply new update

Håkon Alseth — Wed, 16 Feb 2022 08:51:13 GMT

Hi,

Given that it downloads, resets and prints this:

[quote user=""]00> I: Swap type: test[/quote]

It means that mcuboot recognizes this image, but on the very next sequence:

[quote user=""]00> I: Jumping to the first image slot
00> *** Booting Zephyr OS build v2.7.0-ncs1 ***
00> I: Starting bootloader
00> I: Primary image: magic=good, swap_type=0x2, copy_done=0x1, image_ok=0x3
00> I: Secondary image: magic=unset, swap_type=0x1, copy_done=0x3, image_ok=0x3
00> I: Boot source: none
00> I: Swap type: revert[/quote]

You are essentially trying to boot the new image, and you're reset again, where it reverts to the original image.

This can happen if the configuration differs from what mcuboot expects, for instance in the flash layout of the secondary/primary slot.

Could you share details on your original projects "partitions.yml" and/or "pm_static.yml", as well as the "update project"'s partition layout?

It is very important that you follow the partition layout of the original image for all update images.

Kind regards,

Håkon

RE: Serial LTE Modem Application (NRF9160) FOTA does not apply new update

ronald-blackline — Tue, 15 Feb 2022 19:06:30 GMT

This is using NCS SDK v1.8.0