Is it possible to access a QSPI flash from nrf5340 as 'direct' memory access?

[BrianW]

I have a custom board, with nrf5340 and a external QPSI flash (this is exactly like the nrf5340 DK).

I use the external flash for multiple partitions (mcuboot slots, FAT-FS file system, etc), and would like to use a partition as a "memory mapped file system" ie have blocks of data in the flash partition and allow code to access them as though it was data in the internal flash (ie just with a pointer in the 0x100000000 region). This lets me use this const data as though it was in my main application build but without having to use up space in the (limited) internal nrf5340 flash...

I can use the flash_open/read/write()  interface driver api to set up the partition data just fine.

However, when the code attempts to access the memory space directly (as a pointer access for example), it faults...

eg:

// the QPSI flash should be mapped at 0x10000000 and my partition starts at offset 0x780000 in my pm_static.yml

uint32_t* aptr = (uint32_t*)(0x10000000 + 0x780000); 

uint32_t val = *aptr;    // fault!

Your AI says there is no DTS or linker magic required to let the code read from this area, and I have checked that the flash initialisation priority (41 for the driver and 50 for flash) mean it should be initialised before my application starts running (priority 90).

The code also accesses the partition with flash_area_open() so that it can write to it via this api : could this cause an issue?

thanks!

Brian

[AHaug]

Hi Brian,

Could you let me know if you've enabled CONFIG_XIP for this? https://docs.nordicsemi.com/r/bundle/ps_nrf5340/page/chapters/soc_overview/doc/soc_overview.html-xip 

Kind regards,
Andreas

[BrianW]

not explicitly, but it has

#define CONFIG_XIP 1

in the autoconf.h si I guess yes!

Also, I use this external flash for the nrf70 firmware which I think requires XIP to be enabled?

[AHaug]

BrianW said:

My main issue is then that I really don't want to have to run my CPU at 64MHz all the time just in case some of my code needs to access the external flash in XIP mode. Pretty much says using external flash for general data store is only for non-performance devices... 

Very understandable. I don't disagree with the sentiment. If I remember correct from ~3-4 years ago, there was a general recommendation (I can't remember if it was "official" statement or just an engineering recommendation) w.r.t the XIP QSPI feature for the nRF5340, but to summarize: The recommendation back then was that if you're going to execute code from external flash with XIP over QSPI it was recommended to

1. Not have any timing sensitive code/interrupt based code executed from external flash due to execution speed + time to init / deinit the qspi
2. Not have the feature on a battery powered device with a small battery due to increased current consumption. The feature is best suited for MAINS powered devices due to cost with both having the external flash powered and to init/de-init the QSPI

I assume that you're already very committed to this project + BOM and socs, but just in case this is still in early evaluation from your end. Since it seems to me that memory might be a limitation. Is it an option to swap the host MCU to a nRF54LM20 to avoid having to use the XIP over QSPI, or has that ship sailed for this product/does that still not fit within your flash/RRAM requirements?

BrianW said:

Now if your LLM can generate a project where the whole of wpa_suppliant/ related mbedtls dependancies can be placed in a XIP segment and enabled when run at AP authentication time and then disabled once its done, that would be very useful! Thats around 250kB of internal flash saving right there I reckon....

There should be built in support to have the WPA supplicant + related mbetls in a XIP segment, but since supplicant keeps running after it connects we can't power of the XIP partition without any risk. There's a rekey interval that we will need ensure that we have the external flash powered on, as I'm sure you're aware of.

The WPA supplicant handles group and pairwise rekey (EAPOL frames), deauth and disconnect events, roaming and background scans, SA query, and wifi status API calls. Its event loop thread is also blocked inside hostap code. Any of these with XIP off gives a bus fault. 

But that said, it shouldn't be impossible, it's just a bit risky. I'll do an over the weekend test with a gated implementation (i.e turning on the XIP when we need to use the supplicant) to see if it works in my setup. The rekey interval is relatively high on the access point I'm using to test, so I won't get any significant results in by today.

Before those results are in, here's the results of relocating the hostap in xip on my setup. The Wifi FW patch is in the external flash using SB_CONFIG_WIFI_PATCHES_EXT_FLASH_STORE, and not SB_CONFIG_WIFI_PATCHES_EXT_FLASH_XIP (check which you have in the generated config).

Pass	Build	Internal flash	Boots OK	Ping replies	Connect time	Idle current
0	Control	609 308 B	5/5	50/50	5.21–5.26 s	1.392 mA
1	hostap code in XIP	486 800 B	5/5	50/50	5.19–5.24 s	4.058 mA
2	XIP on, nothing relocated	609 416 B	5/5	50/50	5.18–5.31 s	4.063 mA

Current is as you see relatively high, and it might be higher as I don't measure the external flash usage here, only VDD_nrf through a PPK2 and as of now the external flash is on VDD_PER.

Kind regards,
Andreas

[BrianW]

AHaug said:

Not have the feature on a battery powered device with a small battery due to increased current consumption.

Ah... thats exactly my use case

AHaug said:

Since it seems to me that memory might be a limitation. Is it an option to swap the host MCU to a nRF54LM20 to avoid having to use the XIP over QSPI, or has that ship sailed for this product/does that still not fit within your flash/RRAM requirements?

we are already in production and deployment. Currently I am trying to squeeze more functionality in based on customer projects - and trying to get the wifi enterprise stuff in there is a problem....

A bit of speed hit is managable, but a battery hit will be a problem - I'm already struggling with my sleep current on the board (up around 5mA when it should be <1mA). That said, I'm pretty happy with the current used in operation with the wifi connected running MQTT.... 

AHaug said:

The WPA supplicant handles group and pairwise rekey (EAPOL frames), deauth and disconnect events, roaming and background scans, SA query, and wifi status API calls. Its event loop thread is also blocked inside hostap code. Any of these with XIP off gives a bus fault. 

ah, true. for roaming, I understood it doesn't currently support any fast roaming (802.11r), so I've been handling this from the application by checking wifi_status and doing a reconnect as fast as I can... Since I manage this from the app, I can explicitly turn on/off XIP...

And managing the XIP on/off from the application is working for my 'memory map data files' operations (for fonts, I have't tried it for https TLS certificates yet....)

AHaug said:

Before those results are in, here's the results of relocating the hostap in xip on my setup.

Interesting, yes. I use SB_CONFIG_WIFI_PATCHES_EXT_FLASH_STORE (hence my XIP wasn't on by default...). 

I guess the current being nearly 4x higher with XIP on is just beacause it leaves the HCLK192 at DIV1 and not DIV4? Or are you running PM and getting the enter_dpd/exit_dpd benefit in non-XIP mode?

I'm not running PM - maybe this is causing some of my high 'sleep' current...

thanks for testing all this!

[AHaug]

Hi again,

Results are in. I'll start with answering commenting on your latest reply and then go through the results. I won't share the full generated report to start with because I think that might be too much information to have any meaningful conversations around it, but I can share it at a later point if you and I think thats of any use. For now lets just have a look at the compiled configurations and results: 

BrianW said:

I guess the current being nearly 4x higher with XIP on is just beacause it leaves the HCLK192 at DIV1 and not DIV4? Or are you running PM and getting the enter_dpd/exit_dpd benefit in non-XIP mode?

I'm not running PM - maybe this is causing some of my high 'sleep' current...

HFCLK192M left at /1 instead of /4 is part of the XIP cost, but it isn't everything that contributes to an increased consumption.

PM is off in all of these builds, and as you can see in the results I discuss in the following wall of text I see about an 2.9x increase when comparing it to the baseline application. 

We do also not enter deep power down for the mx25 flash chip on the nRF5340 in this test bench as far as I can see, meaning that the external flash is on standby at all time.

Errata 43 and 159 are taken into account in my test bench.

BrianW said:

we are already in production and deployment. Currently I am trying to squeeze more functionality in based on customer projects - and trying to get the wifi enterprise stuff in there is a problem....

A bit of speed hit is managable, but a battery hit will be a problem - I'm already struggling with my sleep current on the board (up around 5mA when it should be <1mA). That said, I'm pretty happy with the current used in operation with the wifi connected running MQTT.... 

There might be some gains for you here, but I think we might have to lower our expectations w.r.t gains both on memory saving side and on consumption side.

Results were somewhat as expected, but also somewhat better than expected w.r.t current consumption. 

I've used a relatively simple application in my test bench, meaning that it's easy for me to get an overview over all entry points into what requires/triggers XIP operation.

The application you're working with most likely has more entry points, so take my table with a grain of salt when you read it. If yours have more entry points, then there might be that you have less potential for code to relocate.

Starting with the test matrix: I set up (or rather prompted) for a set of libraries, files and drivers to relocate and to check if we can find anything that causes a bus fault to occur and focused on excluding anything that depends on IRQs to be relocated to the external flash. This resulted in the following matrix:

And the entry points in the gated builds:

Memory wise we don't reach the 250kB mark you estimated in your guess.

• Saved is how much internal flash the build frees: control internal flash (609 308 B) minus this build's internal flash.
• EXTFLASH is how much code the linker placed in the external flash region at 0x10100000.

At most I've managed to relocate within the specs I sat is as follows

Current consumption is as we've established still significantly increased when comparing to a non-XIP app. Note that here we see the difference between the "Full set + PSA" and "Gated + PSA". In the first, XIP is on all the time, whereas with the Gated variant we only do XIP when we enter any of the entry points.

Do also note that I've not cut SB16 and shorted SB18 per the documentation to measure the consumption of the external flash, so the numbers may differ quite a bit.

I ran the last test over the weekend to check if I hit any bus faults over longer period of time, and it seemed to run without any crashes. Again, this is a simple app.

XIP-on time is dominated by the connect sequence: about 4 s and 50 enables per boot. After that, XIP was on for about 1.1 s per hour during the weekend, mostly from the runner's own wifi status, TCP and xipg commands.

Plots to compare the memory and current consumptions:

Some comments for what to expect for a more "complete" wifi product:

As a rule of thumb, relocate code that only runs in the supplicant thread, and keep anything shared with drivers, the network stack, Bluetooth or the application internal unless every call site can be gated. For a feature-rich product, hostap only (102 KB, gate in three files) is the lower-risk choice. Adding PSA (another 48 KB) is worth it when the supplicant is the only heavy crypto user.

Let me know your thoughts when you've had the time to look through this and if you have any questions w.r.t the results feel free to ask.

Kind regards,
Andreas

[BrianW]

Hi Andreas,

Excellent info, thanks. Very useful to see the tradeoffs involved in the different options!

For my application, which uses both https and mqtts heavily, as you say putting the PSA (I assume thats PSA interfacing + mbedtls as PSA crypto provider) in the XIP is not going to be useful..

But getting a space gain in internal flash of 102KB for the hostap is very worth while, and the cost is limited (in code hacks and power/speed consumption). 

Reality check:

 - Getting this to work with the mcuboot OTA slots will be a challenge (need to add a 4th slot to get mainapp, netcpu, nrf70 and hostap as upgradeable parts). Unless it could be merged with the nrf70 block to make a single updateable slot?

 - validating there are no corner cases where it breaks other flash accesses will need testing

On the last point, I have just run into this : my dev code now uses the QSPI to store font files. All works fine (gated XIP enable/disable when the display updates) and no badness with the other major flash usage (as a flash disk). Except when I unplug the USB cable : the disk is also exposed as a USB MSC device, and something in there doesn't cooperate -> nrf5340 crashes HARD - no logs, and even the debugger cannot access it to find where it went bad. Hard reset is the only option....

thanks a lot for this work. Can you post a zip of the updated files (for the gates, and to get the linker to put the code in an external partition)? 

[AHaug]

BrianW said:

 - Getting this to work with the mcuboot OTA slots will be a challenge (need to add a 4th slot to get mainapp, netcpu, nrf70 and hostap as upgradeable parts). Unless it could be merged with the nrf70 block to make a single updateable slot?

Good question. This is a part of the application I've not touched upon at all in my bench.  

I also stumbled over this when looking at how to do OTA of an application with XIP/relocation https://github.com/nrfconnect/sdk-nrf/pull/29780. In the bench I've only used dts based partitioning and Partition Manager has been off for all builds. Which partitioning scheme are you using on your end?

Looks to me at least based on this that in NCS v3.4.0 users will have to use Partition Manager, which is off by default in 3.4.0 and the LTS in general, but there is support to enable it if users needs it. This will not be an option in NCS v3.5.0

As to how one would implement this in practice is something that I would have to spend maybe a man week on trying to figure out. I can't promise an as extensive exercise w.r.t this topic during this week at least.

BrianW said:

 - validating there are no corner cases where it breaks other flash accesses will need testing

100%

BrianW said:

On the last point, I have just run into this : my dev code now uses the QSPI to store font files. All works fine (gated XIP enable/disable when the display updates) and no badness with the other major flash usage (as a flash disk). Except when I unplug the USB cable : the disk is also exposed as a USB MSC device, and something in there doesn't cooperate -> nrf5340 crashes HARD - no logs, and even the debugger cannot access it to find where it went bad. Hard reset is the only option....

For context, I assume you store the fonts there through a file system, and this happens when you plug the board into a computer which enumerates the USB and writes to the external flash/starts the MSC.

Then it makes sense (to me) that you might have a condition where the USB writes/erases immediately when connected and if the nRF attempts to XIP on the external flash simultaneously, or if the nRF is already XIP-ing you might enter a situation where you have a bus conflict or something 

This will be an entry point that you will have to handle somehow, i.e ensure that the host and the device use the volume at the same time. Any XIP fetch during a write from the USB is unsafe as I see it.

It could also be an issue if the USB runs at 128MHz whereas qspi uses 64 due to Errata 159, there's something that negotiates the speed up 128M when USB is connected. This is just a guess from my end, I haven't looked much into the USB driver in this context.

BrianW said:

thanks a lot for this work. Can you post a zip of the updated files (for the gates, and to get the linker to put the code in an external partition)? 

Happy to help! Was a fun little experiment to get some numbers on what I've preached regarding this use case earlier.

Gated application, both variants with and without PSA relocated attached. Readme should be sufficient to run the sample. Just as a disclaimer to cover my end, please validate and sanity check the sample up against your use case, and its provided as is for your own evaluation.wifi_sta_gated_xip.zip

Quick requirement summary for running the sample: 

HW: nRF5340dk + nRF7002EK
SW: NCS v3.4.0

Kind regards,
Andreas