Several problems arising with OTA DFU.

RE: Several problems arising with OTA DFU.

lengthnull — Tue, 10 Jan 2023 03:08:39 GMT

Same issue here with SDK 14.2.0

RE: Several problems arising with OTA DFU.

Einar Thorsrud — Tue, 13 Dec 2022 10:13:59 GMT

That depends on what you want and how you use it? As mentioned, FDS is a simple file system that handles things like updating data etc in flash, which is a bit of work (due to the nature of flash), and is also easy to mess up (there are many corner cases, for instance if you get a reset while updating data). So I would consider using FDS for all flash storage, but there are special cases where it might not be the best solution. You can read about FDS in the documentation to get a better understanding of it.

RE: Several problems arising with OTA DFU.

Juliusc — Tue, 13 Dec 2022 08:57:19 GMT

very nice, I am using a peer manager indeed but would you say that its generally better to use fds module as opposed to the storage module I am using right now?

RE: Several problems arising with OTA DFU.

Einar Thorsrud — Tue, 13 Dec 2022 08:47:03 GMT

FDS virtual page size is normally the same as a physical page size, so 0x1000 (4 kB). Regarding the value of FDS_VIRTUAL_PAGES_RESERVED, the number of pages here depends on how much data you store in FDS. If you don't use FDS at all, you can disable FDS by setting FDS_ENABLED to 0 and it will not matter. However, remember that for instance the peer manager library that handles bonding in BLE use FDS, so if that is used you will need FDS. If FDS is enabled, the lowest possible number of pages is 2, as that leaves 1 for swap and 1 for data.

The FDS pages are always right below the bootloader. So if your bootloader starts at 0xec000 (you can just check the bootloader project to verify) and you have 3 FDS pages, that means that FDS starts at 0xEC000 - 0x1000 = 0xE9000. So I would suggest that you use the page right below there for your data that is outside of FDS, so starting 0xE8000. If you do, that means NRF_DFU_APP_DATA_AREA_SIZE should be 0x4000, as the bootloader needs to know that it should never touch the 4 pages right below it.

A last point is that FDS can be used for multiple things, so you could also consider to use FDS for your other data. That is a common ting to do, and FDS is a simple file system that allows easy updating of data etc, and it handles the complexities of flash memory for you (things like only being able to flip bits from 1 to 0 except after a page erase, etc).

RE: Several problems arising with OTA DFU.

Juliusc — Tue, 13 Dec 2022 08:23:52 GMT

thanks for your help, I have one more question because I am confused with this:

NRF_DFU_APP_DATA_AREA_SIZE i 0x3000 in my bootloader (which actually shouldn't be enough with my calculations because if my storage starts at 0xec000 that's far off the 0xf8000 bootloader start address and would actually require more than 0x9000)

how does NRF_DFU_APP_DATA_AREA_SIZE connect with FDS_RESERVED_VIRTUAL_PAGES and FDS_PAGES?

If I set FDS_RESERVED_VIRTUAL_PAGES to 3, this should have an impact on my bootloader application right? these 3 defines don't really make sense to me as in I don't know which one to change and if all 3 need to be changed at once

RE: Several problems arising with OTA DFU.

Einar Thorsrud — Mon, 12 Dec 2022 11:06:15 GMT

It is not so much a matter of viewing it, but more drawing it up on a piece of paper or similar, with the addresses you have for the various parts of your project. Some things are given though.

First, look at the memory layout from the bootloader documentation. That has numbers for various ICs so you can use that as a starting point, also the figure you can re-use.

Then add the following:

first page (0x0): MBR
starting at the second page (0x1000). SoftDevice. This will end at the size of the SoftDevice, which you can see form the SoftDevice specification (0x26000 for S140 6.1.1 if that is what you are using)
The application starts right after the SoftDevice, so at (0x26000 if using S140 6.1.1). Where it ends depends on how large it is. Look at the build output or check the application hex in nRF Connect Programmer or some other tool
The bootloader starts wherever the bootloader project you are using is defined to start in the linker configuration. Let's say the bootloader start at 0xf8000 for now as an example (this is a typical value SDK 15.3 based BLE bootloaders in release mode).
There must be two available flash pages at the end though, one form MBR params (second to last) and one for bootloader settings (the last).

These are all the things that are easy to locate. And based on this, you can say more. If you use FDS, they are right below (lower address) than the bootloader. The number of pages are configured in the application sdk_config.h.

If you don't use FDS at all in your application, you can use a page right below the bootloader for stuff you store directly to flash. If you do, pick a page right below this again. Put all this in a figure so that you see your full flash memory layout.

And lastly, remember to update NRF_DFU_APP_DATA_AREA_SIZE in the bootloader's sdk_config.h so that it stays away of any flash you use right below the bootloader (this needs to cover both FDS and anything else). This needs to be a full multiple of 0x1000 which is a flash page, and counts downwards from the bootloader. See the figure linked to earlier in this post.

RE: Several problems arising with OTA DFU.

Juliusc — Mon, 12 Dec 2022 10:00:50 GMT

can you give me a hint on how to view my memory layout? That would probably help us both in the assessment.

RE: Several problems arising with OTA DFU.

Einar Thorsrud — Mon, 12 Dec 2022 07:37:57 GMT

Hi,

This makes sense and is good to hear. I don't know your exa t memory layout(?), but if you have gotten an overview of that and see that the page starting at 0xec000 does not overlap with anything ese and is not used by FDS, and is within the reserved area of the bootloader (that it will never touch), then this is OK.

RE: Several problems arising with OTA DFU.

Juliusc — Sat, 10 Dec 2022 10:58:00 GMT

Hey Einar, so I changed the flash start and end addresses to 0xec000 to 0xeffff after finding out that the dfu always worked if I didn't use the storage module.

This now seems to work, I did some tests and until now it didn't fail. My question is now if you think that this is tangible or just a fluke and if I should take a close look to something in my storage usage?

RE: Several problems arising with OTA DFU.

Juliusc — Fri, 09 Dec 2022 16:04:25 GMT

yeah but thats where I am right now, in the hardfault handler and it faults at this nrf_log_final_flush function

my hardfault handler is enabled tho

RE: Several problems arising with OTA DFU.

Einar Thorsrud — Fri, 09 Dec 2022 15:16:30 GMT

I see, here it is in a hardfault handler. There are several approaches you could use, but I would start by using the hardfault handler library. With that and logging you will get a printout that can be very helpful. Most example projects have the files as part of the project already, so you typically only need to set HARDFAULT_HANDLER_ENABLED to 1 in sdk_config.h.

RE: Several problems arising with OTA DFU.

Juliusc — Fri, 09 Dec 2022 14:49:12 GMT

I understand what youre saying, I cant tell you more than that here:

that's one of the halts I get (p_stack_address is 0 at this point)

there are some other points where the application fails, but that's now the last failure I got after doing a DFU

Do you have another pointer on what I shall look for?

> So you need to get a overview of your memory layout and place the flash region you use for persistent storage at a sensible location as explained in my previous post.

Understood, I will do that!

RE: Several problems arising with OTA DFU.

Einar Thorsrud — Fri, 09 Dec 2022 10:15:00 GMT

[quote user="Juliusc"]I mean it just doesnt work any more and if I attach a debugger I see that it hangs on in that particular state (as shown in the screenshots). [/quote]

I see. The CPU will do something though (either be in sleep waiting for an event, in an eternal loop in an error handler, or something else). It is not just "hanging". So you should debug more to find what is actually the state (and from there backtrack to find out why).

[quote user="Juliusc"]that should be enough as far as I can see?[/quote]

It should be enough yes, but the size is not the issue here. This counts from right below the bootloader. So you need to get a overview of your memory layout and place the flash region you use for persistent storage at a sensible location as explained in my previous post.

RE: Several problems arising with OTA DFU.

Juliusc — Fri, 09 Dec 2022 09:31:09 GMT

I mean it just doesnt work any more and if I attach a debugger I see that it hangs on in that particular state (as shown in the screenshots).

The DATA_AREA_SIZE was set to 12288 in the bootloader SDK, that should be enough as far as I can see?

RE: Several problems arising with OTA DFU.

Einar Thorsrud — Fri, 09 Dec 2022 09:24:14 GMT

[quote user="Juliusc"]I cant see whats going on in that advertising function, it crashes at this handler:[/quote]

When you write "crashes", what exactly do that mean? Can you elaborate? What do you find about the state of the device and what happened from debugging? And what do you see form the RTT log?

[quote user="Juliusc"]07c000 to 7ffff,[/quote]

As long as the flash is not used by anything else it should be OK. However, even if it is not used by anything else when the application is running, it could be that it gets overwritten during a DFU. Generally, you should use flash right below the bootloader (but take into account that the FDS pages are there as well if you use that, and in that case use flash right after the FDS pages again). See memory layout. This is because you can configure the bootloader to lay away of this region by adjusting NRF_DFU_APP_DATA_AREA_SIZE in the bootloader's sdk_config.h.

RE: Several problems arising with OTA DFU.

Juliusc — Thu, 08 Dec 2022 09:47:53 GMT

Ok so here's what I did:

flash initial software to MCU -> test it, works

make new software (which has basically no changes in sdk_config, mainly business logic) -> test it, works

DFU (new software replaces old software) -> sensor crashes

this is the call stack after attaching a debugger after the DFU

so I reset the sensor in debug mode, then this crash report appears:

then I reattached the debugger and this crash report appears

NRF_LOG is enabled and RTT is enabled on both.

does this help?

I cant see whats going on in that advertising function, it crashes at this handler:

RE: Several problems arising with OTA DFU.

Juliusc — Wed, 07 Dec 2022 14:01:20 GMT

okay let me try to narrow it down.

It happened while performing a DFU, I did that by using the nrf connect app and then the MCU didn't show up again (no advertising) hence I attached a debugger and noticed that the firmware crashed in some sort of logging process. But this was only one of some issues, it also happened once that advertising failed with an internal error (0x3)

> but there could be changed in the other parts of the flash in case of a dual bank update. Normally that would not cause problems, but it could potentially trigger a bug in your application.

yep that's what I though as well, I am using the internal flash storage module, writing to addresses 07c000 to 7ffff, maybe this helps? How can I see which banks are being written during the dfu process and therefore always keep those clear? Could it also be a problem if the application is too big?

I will inspect the issue in more detail and come back with more explanation, but that's it for now.

RE: Several problems arising with OTA DFU.

Einar Thorsrud — Wed, 07 Dec 2022 13:50:49 GMT

Hi,

Have you confirmed that you get an actual hardfault, or something else? What have you found out by debugging? How exactly did this happen? (There are very many things that can go wrong, so we need to narrow things down as much as possible as early as possible).

Is it so that the issue issue happened after adding buttonless DFU to your application, or does it happen after a DFU update?

The application would be identical after a DFU update as if it was flashed directly (assuming you made no changes), but there could be changed in the other parts of the flash in case of a dual bank update. Normally that would not cause problems, but it could potentially trigger a bug in your application.

In short, please explain in more detail what you did and what exactly fails and how. If you go back to when this issue did not happen, can you isolate at which point things start to fail and what exactly you do in that case? Then, what can you find from debugging (for instance logging and inspecting after en error handler is run or something else, depending on what actually happened)?