Writing image to Slot1 partition causes a hard-fault for firmware update

[Ryjan]

I'm using VS Code on a Windows 10 machine to develop a product that will use an over the air device firmware update. I have my system setup with static memory partitions. I can get the data to the device, but as soon as I go to write the data to the partition using the functions provided, the system crashes with a hard-fault.

This is my pm_static.yml file:

app:
  address: 0xc200
  end_address: 0x44000
  region: flash_primary
  size: 0x37e00
mcuboot:
  address: 0x0
  end_address: 0xc000
  placement:
    before:
    - mcuboot_primary
  region: flash_primary
  size: 0xc000
mcuboot_pad:
  address: 0xc000
  end_address: 0xc200
  placement:
    align:
      start: 0x1000
    before:
    - mcuboot_primary_app
  region: flash_primary
  size: 0x200
mcuboot_primary:
  address: 0xc000
  end_address: 0x44000
  orig_span: &id001
  - app
  - mcuboot_pad
  region: flash_primary
  sharers: 0x1
  size: 0x38000
  span: *id001
mcuboot_primary_app:
  address: 0xc200
  end_address: 0x44000
  orig_span: &id002
  - app
  region: flash_primary
  size: 0x37e00
  span: *id002
mcuboot_secondary:
  address: 0x44000
  end_address: 0x7c000
  placement:
    after:
    - mcuboot_primary
    align:
      start: 0x1000
  region: flash_primary
  share_size:
  - mcuboot_primary
  size: 0x38000
settings_storage:
  address: 0x7c000
  end_address: 0x80000
  placement:
    align:
      start: 0x1000
    before:
    - end
  region: flash_primary
  size: 0x4000
sram_primary:
  address: 0x20000000
  end_address: 0x20010000
  region: sram_primary
  size: 0x10000

The new partition should be written at 0x00044000.

Using Ozone I was able to see that in the MMFSR I have the flags MMARVALID, MSTKEPP and DACCVIOL set. It looks to me that the memory protection unit is preventing me from writing to flash. I commented out the actual flash write function "flash_img_buffered_write" and that seemed to be the source. After I dug into the code I was able to determin that the offset address for the flash area was set to 0x00000000. That is wrong. So that led me to the "flash_img_init_id" function where it find the context for the flash write. I was using SOC_FLASH_0_ID as the ID (since the definition had the comment "Provided for compatibility with MCUboot"), but I had tried some other things too. The header file does not give much help besides "flash area id of partition where the image should be written". I was not sure how to derive one of these. I tried switching to "flash_img_init" where the function assumes a location to write it to and using "UPLOAD_FLASH_AREA_ID" as the ID. That now gave me the expected address of 0x00044000, but unfortunately I still have the same error s before.

At this point it looks like the using the call "flash_img_buffered_write" should work, but it looks like there is some sort of memory protection there too. Is that assumption correct? I have not found any thing about needing to modify the MPU to do that. I'm not sure if I'm looking in the right place.

[Vidar Berg]

Hello,

Please confirm if your build includes CONFIG_MPU_ALLOW_FLASH_WRITE=y (you can check this in the .config file located in <build folder>/zephyr/). If this is not enabled, then you will get a fault exception you like you described. Although, this setting should have been selected already if you have enabled the flash APIs through the CONFIG_MCUMGR_CMD_IMG_MGMT config setting.

Best regards,

Vidar

[Ryjan]

I checked and found "CONFIG_MPU_ALLOW_FLASH_WRITE=y" in the build config, but I then checked for "CONFIG_MCUMGR_CMD_IMG_MGMT " in my prj.config and found that I had commented it out. When I try to add it, VSCode claims that the in the build it's set to "n". It is very hard to trace and find out why, but I assume it is a Kconfig rule set somewhere else that is in conflict with another. Still "CONFIG_MPU_ALLOW_FLASH_WRITE" is set in the build file.

[Vidar Berg]

Could you check the MMFAR address as well to see if it is indeed in the slot 1 memory range? Also, can you please confirm whether you are using the mcumgr protocol to transfer the image over BLE or not (i.e if you are using the same approach as covered in this tutorial  Add DFU support to your applicationn)

[Ryjan]

First, I will confirm that I am not using mcumgr protocol for BLE. I was not aware of that system, so that will prove valuable for some future projects. This project, however needs to be compatible with our old system. It is using our own proprietary protocol to be legacy compatible with older products.

I checked the MMFAR register and it shows 0x20007538 as the stored value. That looks like a location in ram rather than a location in flash. It's not that same address as the one I see calculated in code for the write. In the steam structure in the flash_ctx structure I get the correct offset of 0x00044000, but that's not the same as the MMFAR register. I has assumed that the address in the MMFAR was the location of the instruction that was causing the crash.

I've included a screenshot of the crash in Ozone:

[Vidar Berg]

Thanks for confirming. This looks more a stack overflow (MPU-assisted stack overflow detection). Do you have logging enabled so you can get the crash log from error handler? With CONFIG_THREAD_NAME=y it should tell you in which thread the stack overflow occurred.

[Ryjan]

When you say that this looks more like a stack overflow, what do you see that makes you think that? Is is because of the RAM address? I had a stack overflow last week and the log message spit our some good information so I was able to fix the thread size. With this I see no data. I turned the suggested option on, but I saw no difference in the terminal output. Right now I'm reading through the link you included about MPU-assisted stack overflow protection.

[Vidar Berg]

I suspect it's a stack overflow because the MSTKERR bit is set and that MMFAR contains a RAM address. 

Don't you get a crash log at all? Is it also the same if you run the program without Ozone (looks like ozone is halting the cpu when the exception is raised which may prevent the crash log from being printed)?

[Ryjan]

Ok, that is making sense to me. I do think that it's fixed now. I increased the same thread that I had before a little more, by about 40 bytes and the log message finally spit out a message about a stack overflow. I doubled the stack size and now the function call to "flash_img_buffered_write" long longer fails.

Since this was a project I was handed with no familiarity with Zephyr, but was tasked with getting some DFU running the main task (always_run, in this case) was not large enough to handle the receive buffer I made for the DFU data. So this makes sense, it was just difficult to find the cause since the thread was so undersized that it would not spit out a log message. It would skip the log message running Ozone or the RTT viewer just the same.

Thank you, Vidar!

[Vidar Berg]

I'm glad to hear that it works now, thanks for the update. If you encounter a similar problem in the future, I would suggest you try the Thread analyzer. It will periodically print the stack usage for all threads.

Config settings to enable thread analyzer:

CONFIG_THREAD_ANALYZER=y
CONFIG_THREAD_ANALYZER_AUTO=y
CONFIG_THREAD_ANALYZER_AUTO_INTERVAL=5
CONFIG_THREAD_NAME=y