Limitations of Direct-XIP on nRF5340

[Hardware CI]

My platform is the NRF5340, coupled with 32MB of external flash, so I have plenty of external storage available.
However, there's a risk that my project may eventually outgrow the internal flash.

Here’s a wild idea—what if both the MCUBoot primary and secondary slots were placed in external flash and executed directly from there?
Would this be feasible, perhaps by leveraging Direct-XIP capability?
I’ve seen some examples where parts of the code (libraries, objects) are executed from an external XIP area, but never the entire codebase.

After some research, it seems that full execution from external flash isn't possible due to certain limitations.
Summing them up (correct me if I’m wrong), functionalities like interrupt handling and Zephyr kernel functions cannot be executed from external flash. Relevant discussions:

• Direct XIP External QSPI
• Running an nRF Connect SDK App from External Flash (XIP) on nRF52840

However, I’d like to be absolutely sure. Could you provide more details on why this is the case?
Is this fundamentally impossible due to hardware limitations, or is it just challenging to achieve with custom MCUBoot modifications?

Lastly, regarding the security of the external XIP area—how does it compare to an encrypted/signed MCUBoot primary slot in internal flash?
Would it be possible to protect it using TF-M?

Thanks in advance!

[AHaug]

Hi,

However, there's a risk that my project may eventually outgrow the internal flash.

Unfortunately in modern embedded devices this is more and more becoming a risk due to large wireless protocol stacks. Do you have a current draft over what you aim to include in this project (for instance which protocols, what use case, is it powered battery operated)? If you have a brief summary, throw them out here, and I'll list up a couple of suggestions and pit falls w.r.t memory optimizing large embedded applications.

After some research, it seems that full execution from external flash isn't possible due to certain limitations.
Summing them up (correct me if I’m wrong), functionalities like interrupt handling and Zephyr kernel functions cannot be executed from external flash. Relevant discussions:

It looks to me based on what you're writing that you're already aware of this, but lets go through a couple of items nonetheless just to be 100% certain that both of us are on the same page

• External flash != an MCU

This probably says itself, but the main illustration I want to illustrate with this is that you can't write threads that runs on the external flash. What XIP allows you is to run your main application on internal flash, and use some code that is stored on external flash and execute it over QSPI. The threads themselves run on the internal flash.

• Dirext XiP != XIP

Direct XIP means that you're building your firmware to be able to be ran from the slot it is built for, meaning you can do DFU and skip the swapping procedure. This only works from internal flash due to the first item.

XIP is the part that is supported on the nRF5340 only. (It is written in the PS of the nRF52840 that XIP is supported, and it has the hardware registry concerning the XIP region in external flash, but in practice it does not work due to qspi limitations and other erratas.) This means that you build your application so that it is split into two parts - one residing on the internal flash and one residing on the external flash. The internal flash is the main application with features etc that you're supposed to use frequently, whereas the XIP-part of the application is the code, drivers and/or libraries that resides on the external flash. The external flash code is code that is not often used and has a couple of other limitations as well. 

XIP is vulnerable to timing. This means that code that ISR (interrupt service routines) either depends on or a ISR from external flash to internal flash will cause issues. Executing code from internal flash is extremely fast, whereas executing code from external flash requires you to get access to the qspi, make sure that nothing else is using the qspi and/or ensure that you're the only thing that has access to it, potentially power up the external flash if it is in deep power down and then executing the code from there with a slower frequency than the internal flash has. As an analogy; think of it as cooking and the difference between fetching an ingredient from your fridge in the kitchen vs the food larder in the basement of a house. 

• XIP memory mapping

The hardware itself has a limited memory mapping space for XIP application from the PS: 

• Power consumption

Executing code from the external flash requires you to have both internal flash, qspi peripheral (and any other relevant peripheral) as well as the external flash powered on. This drains quite a bit of power vs putting peripherals and external flash in sleep when you don't need them. XIP from external flash is not suitable for battery operated devices/or requires you to take this consideration in mind when designing the lifespan of a battery operated device.

Here’s a wild idea—what if both the MCUBoot primary and secondary slots were placed in external flash and executed directly from there?
Would this be feasible, perhaps by leveraging Direct-XIP capability?
I’ve seen some examples where parts of the code (libraries, objects) are executed from an external XIP area, but never the entire codebase.

So based on the first two items, unfortunately this will not work. Direct XIP requires a MCU and you need the MCU to be able to execute code since the external flash. Relocating certain parts of the application will work and is supported, but that is only for non-timing sensitive code (i.e no ISRs). 

However, I’d like to be absolutely sure. Could you provide more details on why this is the case?
Is this fundamentally impossible due to hardware limitations, or is it just challenging to achieve with custom MCUBoot modifications?

The short answer is thus that it is primarily fundamentally impossible and/or severly-not-supported to do this due to hardware limitations and secondarily a challenge to achieve due to bootloader limitations

Lastly, regarding the security of the external XIP area—how does it compare to an encrypted/signed MCUBoot primary slot in internal flash?
Would it be possible to protect it using TF-M?

As of now XIP-split + TF-M is not quite yet supported. Due to bootloader and build system limitations. There are internal discussion ongoing for this, but status and ETA I can't state anything w.r.t (reach out to your regional sales manager for more info if any exists).

Let me know if this sheds some light on the discussion

Kind regards,
Andreas

[Hardware CI]

Thank you for the very detailed explanation. I think you have covered the topic thoroughly. To be honest, this is what I anticipated, but thanks to the additional information from you, my understanding is more complete now.