Crash after powerdown if ram is not powered

[Seb-labs]

sdk: 2.5.1

cpu: nrf5340

I have a bootloader which have a big ram block for upgrade purpose (around 250kb) in bss section (I will move it in noinit, but it is not the point here).

I  have an app, that poweroff unused ram with VMC register.

Sometimes, the app will enter in poweroff (using REGULATORS register), and will be awakened by vbus or nfc.

When the cpu is awakened, the bss section and idle stacks are in a part of the ram that is powered off, and this is a unrecoverable crash.

I found a solution with powering ram at early startup, but I would like to check with you if it is correct.

The solution may also be about powering up ram before entering poweroff, but I would like may bootloader to be robust.

I wrapped the method z_arm_reset (with linker argument --wrap), and wrote this method:

extern void __real_z_arm_reset();

__attribute__((naked))
void __wrap_z_arm_reset(void)
{
  NRF_VMC->RAM[0].POWERSET = 0xFFFF;
  NRF_VMC->RAM[1].POWERSET = 0xFFFF;
  NRF_VMC->RAM[2].POWERSET = 0xFFFF;
  NRF_VMC->RAM[3].POWERSET = 0xFFFF;
  NRF_VMC->RAM[4].POWERSET = 0xFFFF;
  NRF_VMC->RAM[5].POWERSET = 0xFFFF;
  NRF_VMC->RAM[6].POWERSET = 0xFFFF;
  NRF_VMC->RAM[7].POWERSET = 0xFFFF;
   __asm__ volatile ("b __real_z_arm_reset");
}

I checked the generated code, and no stack seem to be used:

00002a28 <__wrap_z_arm_reset>:
    2a28:       f64f 72ff       movw    r2, #65535      ; 0xffff
    2a2c:       4b09            ldr     r3, [pc, #36]   ; (2a54 <__wrap_z_arm_reset+0x2c>)
    2a2e:       f8c3 2604       str.w   r2, [r3, #1540] ; 0x604
    2a32:       f8c3 2614       str.w   r2, [r3, #1556] ; 0x614
    2a36:       f8c3 2624       str.w   r2, [r3, #1572] ; 0x624
    2a3a:       f8c3 2634       str.w   r2, [r3, #1588] ; 0x634
    2a3e:       f8c3 2644       str.w   r2, [r3, #1604] ; 0x644
    2a42:       f8c3 2654       str.w   r2, [r3, #1620] ; 0x654
    2a46:       f8c3 2664       str.w   r2, [r3, #1636] ; 0x664
    2a4a:       f8c3 2674       str.w   r2, [r3, #1652] ; 0x674
    2a4e:       f00a b9e7       b.w     ce20 <z_arm_reset>
    2a52:       bf00            nop
    2a54:       50081000        .word   0x50081000

Waiting for your answer.

Regards,

[Bjarki Andreasen]

Hi, the following example code will configure the RAM blocks to behave as you describe:

I created the following zephyr test application (modified main.c and prj.conf of the hello_world sample)

#include <stdio.h>

#include <hal/nrf_gpio.h>
#include <zephyr/sys/poweroff.h>
#include <zephyr/kernel.h>

/*
 * NOTE when running this sample, don't debug it as it can't enter real
 * power off while a debugger is attached. It will just be spinning forever.
 *
 * NOTE also that this is the default RAM behavior, so if you don't
 * wish to retain any RAM blocks or sections of said blocks, you don't
 * need to modify NRF_VMC
 *
 * Last note, this sample requires the following additional Kconfig options:
 *     CONFIG_POWEROFF=y
 *     CONFIG_GPIO=y
 */

int main(void)
{
	printf("Hello World! %s\n", CONFIG_BOARD_TARGET);

	/* Before doing anything which disables the debug interface, sleep
	 * to provide enough time to start a debug session / erase the device.
	 */
	k_msleep(10000);

	/* Prepare for power off
	 *
	 * The nRF5340 has two RAM blocks, we want the
	 * entire RAM block to be enabled when system powers on after reset, this
	 * is done by setting the section power bits (16 sections mapped to the lowest
	 * 16 bits of the NRF_VMC->RAM registers).
	 */
	NRF_VMC->RAM[0].POWERSET = UINT16_MAX;
	NRF_VMC->RAM[1].POWERSET = UINT16_MAX;

	/* We want all RAM blocks to be powered down in power off mode, to do this, we
	 * clear the section retain bits (16 sections mapped to the upper 16 bits of
	 * the NRF_VMC->RAM registers).
	 */
	NRF_VMC->RAM[0].POWERCLR = UINT16_MAX << 16;
	NRF_VMC->RAM[1].POWERCLR = UINT16_MAX << 16;

	/* Enable some wake up source. This is Button1 on the nRF5340DK connected to
	 * pin 0.23
	 */
	nrf_gpio_cfg(
		23,
		NRF_GPIO_PIN_DIR_INPUT,
		NRF_GPIO_PIN_INPUT_CONNECT,
		NRF_GPIO_PIN_PULLUP,
		NRF_GPIO_PIN_S0S1,
		NRF_GPIO_PIN_SENSE_LOW
	);

	/* Print last message, and wait for it to print to console before power off */
	printf("Good night!\n");
	k_msleep(100);

	/* When entering power off, the RAM blocks 0 and 1 will be powered down.
	 * When the system wakes up again, the RAM blocks come back automatically.
	 * After power off, press the button
	 */
	sys_poweroff();
	return 0;
}

Hopefully we can compare solutions to figure out why it is failing for you :)

[Seb-labs]

I think I did not explained the problem correctly.

At wakeup, the ram of the main stack inside the bootloader was turned off by the app because the main stack of the bootloader is in a part of ram unused for the app. When the reset function of the bootloader is called at wakeup, there is an exception immediately as the stack is used. The main method will never been executed. The purpose of my code was to turn ram on inside the bootloader before any use of ram.

[Bjarki Andreasen]

OK, either your bootloader or application is must configure the behavior of the VMC before entering poweroff.

The VMC automatically powers on/off sections of the RAM blocks depending on the system power state. You can configure two options for each section of each RAM block:

1. Should this section be powered when the system is ON.

2. Should this section be retained (powered) when the system is OFF.

By default, after a "real" reset (using the reset pin or power reset), all RAM blocks will be powered when the system is ON, and not powered when the system is OFF.

The behavior you are describing I have only been able to recreate by configuring the VMC to disable RAM sections in system ON which are in use by your bootloader/app, by doing the following:

NRF_VMC->RAM[0].POWERCLR = 0xFFFF;
NRF_VMC->RAM[1].POWERCLR = 0xFFFF;
sys_poweroff();

This will cause a crash when you reboot unless you manage to manually power the RAM sections again before accessing them, which seems to be what you are doing with

void __wrap_z_arm_reset(void)

How are you disabling power to the RAM blocks in power off would be my next question if the above is not accurate to your case :)

[Seb-labs]

I power down ram inside app, at startup using power_up_unused_ram in nrf\include\ram_pwrdn.h.

The ram power status is then not modified anymore.

The app is a sample app with very low ram usage, so a lot of ram is powered down.

The bootloader as a big ram buffer in bss section (that I will move in uninit, but it is not the point here). This buffer is before stacks that are in noinit sections. So the stacks are in positions that are powered off by app.

[Bjarki Andreasen]

Ah, now I think I understand your setup. This is how I would solve your problem:

The bootloader and application will share a "small" section of RAM, which will be configured to always follow the system state, powered when the system is ON, not powered when the system is OFF. I would make this explicit in the devicetree like this:

/ {
	chosen {
		zephyr,sram = &common_sram;
	};

	reserved-memory {
		common_sram: image@20000000 {
			reg = <0x20000000 0x4000>;
		};

		bank_sram: image@20004000 {
			reg = <0x20004000 0x3c000>;
		};
	};
};

The common_sram section is the RAM containing .bss, .no_init etc. shared by the application and bootloader. This section will never be powered down while the system is ON. The section bank_sram will be used exclusively for the RAM bank.

Your bootloader will need to power this bank manually before using the memory, you can get the address and size of your memory section from the devicetree

uintptr_t bank_start = DT_REG_ADDR(DT_NODELABEL(bank_sram));
size_t bank_size = DT_REG_SIZE(DT_NODELABEL(bank_sram));
uintptr_t bank_end = DT_REG_ADDR(DT_NODELABEL(bank_sram)) + DT_REG_SIZE(DT_NODELABEL(bank_sram));

 

Which you can then pass to power_up_ram() and power_down_ram() before and after using the RAM bank. From you application, simply call power_down_unused_ram() if you don't intend to power down the bank_sram RAM from the bootloader before entering the application.

[Seb-labs]

I can’t make myself understood.

As the ram is powered down, the bootloader crash very very early. Far before init of the kernel. Inside the reset part, just when the MPS is needed.

[Bjarki Andreasen]

Okay, this is the complete sequence leading to successful start to crash on reset in your case:

1. After device reset (reset pin or power reset), all RAM is configured through VMC to be enabled when the system is ON, and powered down when the system is OFF.

2. Your bootloader starts successfully, as all RAM is powered.

3. You jump to the application, all works as expected.

4. At this point, before entering power off, you configure all RAM to be powered down in system ON state.

5. You immediately enter poweroff, if you tried to read from RAM here, the application would likely crash, and never enter OFF state.

6. The system is now in the OFF state, and the VMC is configured to always power down all RAM, regardless of power state.

7. You trigger wakeup

8. System powers up, you bootloader starts

9. VMC has not been configured to automatically power on any RAM, including the RAM used by your bootloader for normal operation

10. Bootloader reaches an instruction which requires reading from RAM

11. System crashes

---

To add your solution to the sequence, this happens after step 9.

9. ...

10. Manually configure VMC to power all RAM

11. Bootloader reaches an instruction which requires reading from RAM

12. Bootloader runs as expected

---

So, to answer your original question:

Seb-labs said:

I found a solution with powering ram at early startup, but I would like to check with you if it is correct.

Wrapping early C code to manually power on RAM used by the bootloader for normal operation is not robust, nor necessary. The VMC is designed to automatically power the RAM needed for system ON, and retain any RAM required by the application as well.

The solution is to configure the VMC to power RAM used by your bootloader and application automatically, and create a separate section in RAM which your bootloader/application manually powers on/off at runtime.

[Seb-labs]

4. At this point, before entering power off, you configure all unused RAM to be powered down in system ON state.

5. You immediately enter poweroff, if you tried to read from RAM here, the application would likely crash, and never enter OFF state.

6. The system is now in the OFF state, and the VMC is configured to always power down all RAM unused by app

I do not want to have separate section in ram for bootloader and app, as there are some app that will consume all the ram. It is not possible for the bootloader to reserve a part of ram.

What I want here is to ensure that bootloader will not crash, even if app is doing bad thinks, like power down the ram needed by the bootloader. It must not be a need for the bootloader that the app developper is clever.

[Bjarki Andreasen]

If you wish to protect your bootloader/trusted application from a user application, you will want to look into the SPU (System Protection Unit) https://infocenter.nordicsemi.com/topic/ps_nrf5340/spu.html?cp=4_0_0_6_31_3#peripheral_access_control and TF-M (Trusted firmware) https://trustedfirmware-m.readthedocs.io/en/latest/introduction/index.html 

The SPU allows you to configure access to peripherals from the secure/non-secure domains, you can use this to disallow the application from accessing VMC all together, preventing invalid configuration of most peripherals, VMC included :)

[Seb-labs]

I will stop there, because I cannot make myself understood. The code I mentioned in my first post works: the bootloader starts normally even if the ram has been turned off by the application (nothing to do with spu protection: I do not want to prevent the extinction of ram, since it is an important element for low-power). I just wanted to have it checked by the Nordic experts, and also shared.