problem with UART update on nRF52840

RE: problem with UART update on nRF52840

ste2108 — Wed, 29 Mar 2023 15:20:22 GMT

Dear Hung Bui,

Thanks so much for your help throughout all this thread. In the end the best way for us to implement what you advised in your last message within our code (which does in c exactly what nrfutil does in python) was to ignore the offset and crc returned by the firmware and set both to 0. In this way when the update is restarted after an interruption everything is sent from scratch and the update is successful.

Kind regards,

Stefano

RE: problem with UART update on nRF52840

Hung Bui — Wed, 29 Mar 2023 10:30:22 GMT

Hi Stefano,

From what I can find in the code, if you set NRF_DFU_SAVE_PROGRESS_IN_FLASH to 0 the progress will only be stored in RAM:

// <q> NRF_DFU_SAVE_PROGRESS_IN_FLASH - Save DFU progress in flash.

// <i> Save DFU progress to flash so that it can be resumed if interrupted, instead of being restarted.
// <i> Keep this setting disabled to maximize transfer speed and minimize flash wear.
// <i> The init packet is always saved in flash, regardless of this setting.

#ifndef NRF_DFU_SAVE_PROGRESS_IN_FLASH
#define NRF_DFU_SAVE_PROGRESS_IN_FLASH 0
#endif

Here is what I find in nrf_dfu_req_handler.c:

So if you set the configuration to 0, the bootloader will store to flash when the whole image is stored. So if you trigger a reset on the DFU target either soft reset or pin/power reset, you should be able to clear the progress.

However, the init packet progress will always be stored:

So if you want to avoid the nrfutil to recover and continue (at least the init packet) from the previous failed DFU update, I would suggest to look at this line: https://github.com/NordicSemiconductor/pc-nrfutil/blob/master/nordicsemi/dfu/dfu_transport_serial.py#L250

If we remove the try_to_recover() or change it to always return false, I believe that nrfutil will always start sending as new image and will not try to continue from where it stopped.

RE: problem with UART update on nRF52840

ste2108 — Tue, 28 Mar 2023 07:09:44 GMT

Just an addendum: I also tried to start the update of our Nordic board with nrfutil and the serial adapter and to unplug the serial adapter instead of killing by ctrl-c. If I restart nrfutil, the update crashes as the offset is still the old one (i.e. 0x0000).
It looks like the update sistematically starts where it was interrupted by requiring the offset related to the last written data (which is of course 0x0000 when the update starts from scratch). Is there a way to always require an update from scratch, maybe by setting some sdk_config define? This would possibily solve our problem since we do not really care about saving time...
Thanks.
Regards,

Stefano

RE: problem with UART update on nRF52840

ste2108 — Tue, 28 Mar 2023 06:56:35 GMT

Dear Hung Bui,

Thank you for your answer and sorry for being late, I was away a few days.
I performed the test you asked on the dev board and indeed when the nrfutil process is killed and subsequently restarted the update restarts where it was interrupted and is finalized.
What I did subsequently was to compare my sdk_config.h with the one of the official signed serial bootloader (v. 16.0.0). I realigned the defines to the latter version as much as I could (you may remember that we also need the combined BLE bootloader, so all the related BLE defines must be there). After the realignment, an update through nrfutil with a serial adapter connected directly to our board works almost perfectly. That "almost" refer to the fact that, from time to time, I get what is shown in the picture: there I started the update through nrfutil, killed it (at 16%) and then restarted: the update continued where it was stopped. However, when I killed (at 52%) and restarted nrfutil again, it crashed because the continued update started where the first (at about 16%) and not the second (at 52%) nrfutil was stopped. If one looks at the logs, this corresponds to the firmware providing the first offset, as if the second one (i.e. the one corresponding to 52% advancement) was not written in flash when nrfutil was killed the second time.
This looks actually the same way in which the update on our full board (i.e. the Nordic connected to another MCU) fails: when we kill the update and then restart it, the offset provided by the MCU is always 0 (i.e. the beginning of the flash space reserved to the firmware image), thus ignoring what was written up to then.
Can you please tell me how/where this offset is written in flash and let me know if you have any idea on why sometimes it doesn't seem to be written (rarely with nrfutil, sistematically in our custom update) when we kill the update process?
Many thanks.
Regards,

Stefano

RE: problem with UART update on nRF52840

Hung Bui — Thu, 23 Mar 2023 12:50:45 GMT

Hi Stefano,

If your board uses different UART pins then of course the hex file I provided will not work. But I would strongly suggest to try testing on a DK to align what we have with the DK before testing with your board.
I did a quick test here without flow control and it also worked after the interruptions.

RE: problem with UART update on nRF52840

ste2108 — Wed, 22 Mar 2023 17:16:25 GMT

Hello Hung Bui,

I am using our proprietary board mounting a Nrf52840, connected to a PC with a serial adapter (FTDI FT232RL chip). This is the same configuration under which the (uninterrupted) update works. I can try with your board and I am sure that the combination of the Nordic dev board and your fw will work, but I am not sure this can help much in solving the problem I see.

RE: problem with UART update on nRF52840

Hung Bui — Wed, 22 Mar 2023 17:10:25 GMT

Hi Stefano,

To be able to debug the bootloader, you should use the debug version of the bootloader, not the project without "debug". The one without "debug" is optimized and you can't step in the code. Make sure you erase the chip before flashing the debug version because it needs to re-write the UICR.

Please let me know which hardware board you are using for testing my firmware.

RE: problem with UART update on nRF52840

ste2108 — Wed, 22 Mar 2023 17:07:30 GMT

In addition I am experiencing serious problems debugging the bootloader. I can start the debug session (breakpoint at the beginning of main) and I set a bp in the function you pointed out, but when I let the bl run and I start the update process the debug session sistematically returns a series of errors in reading registers and I have to restart it. Is there some register that I can read, e.g. using nrfjpog, whose value can shed some light on why the interrupted update is not working?
Thanks.

Regards,

Stefano

RE: problem with UART update on nRF52840

Hung Bui — Wed, 22 Mar 2023 16:58:41 GMT

Hi Stefano,

Could you let me know which hardware you were using for testing ? I was using nRF52840 DK for testing. And the COM port used was the COMPORT created by the debugger.

RE: problem with UART update on nRF52840

ste2108 — Wed, 22 Mar 2023 15:18:23 GMT

Dear Hung Bui,

I managed to test the files you sent me and here is the result. I uploaded the SD and the BL you sent me (in this order) and then I tried to update the board using your application and nrfutil version 6.1.7 (which should be equal to 6.1.6). As you see in the picture, the update does not event start. On the oscilloscope I see pings from the serial device but no response. The only difference I have wrt what you advised is the flow control, which in my case is set to 0 as I do not have the needed pins on my board.
I do not really need the update to start from the point where it was interrupted, starting from scratch after an interruption would be just fine. Also, I read the flash image just to report the issue to you, it is not something I do sistematically and the result is the same with or without it.

RE: problem with UART update on nRF52840

Hung Bui — Fri, 17 Mar 2023 14:25:40 GMT

Hi Stefano,

I did another test with SDK v16.0 and nrfutil 6.1.6 . I don't have an issue to continue update after aborting:

But I do find that if I trigger a reset in between the DFU update will start from 0 again. From what I understand, before the image is completely received, the DFU progress is only stored in RAM.

It could be the case when you try to read flash in between the update. The Bootloader will be hung and won't be able to continue. You will need to do a reset, and it will clear the progress and it will start from 0. So in your next test, please don't read flash.

I attached here the files I used. Please use them to test.

devzone.nordicsemi.com/.../6371.DFU2.zip

RE: problem with UART update on nRF52840

ste2108 — Thu, 16 Mar 2023 15:11:42 GMT

Dear Hung Bui,

Thanks again for your answer. You are right when you say that we are using our own program to update the Nordic (as it is on a mezzanine board connected via UART to another MCU. It is the latter that updates the Nordic, i.e. replaces nrfutil). Anyway I am able to detach the Nordic mezzanine, connect tx/rx to a serial adapter and use nrfutil to update it. The result is what you see attached:
1) If I erase the Nordic, upload the SD+bootloader and then update the Nordic with the signed zip packet through nrfutil, the update is successful and I see the Nordic fw starting at 0x26000 (as the flash is blank the start address of the cached image coincides with the application image I believe), see _1 images. Please note that I used -prn 200 as in your tests and that I cannot use flow control due to its absence in our board. Could this represent an issue?

2) If I now erase again the Nordic, upload the SD+bootloader and then update the Nordic with the signed zip packet through nrfutil BUT stop the procedure with ctrl+c at about half, I see what I would expect, i.e. data till about half of the application fw size (0x3FC80) and then 0xffs, see _2 images.

3) If I try the update from nrfutil again (without previously erasing the MCU this time) with the same parameters I see no change in the MCU image wrt point 2. On the oscilloscope one can see nrfutil pinging the MCU (whose bootloader should be active since the upload of the application in point 2 failed) but with no answer from the MCU. You might remember that when our proprietary update tool was used the behaviour was slightly different, with the first 1K of the applcation being sent before the CRC request was not served by the bootloader.

As for your questions:
- I haven't modified the bootloader but I enabled the relevant options in the sdk_config.h to operate DFU both through BLE and UART, which is what we need in our application.
- As mentioned before, we use our update tool, but the behaviour with nrfutil is similar (not equal, see point 3)

I still have to try with 17.1 (should check with other co-developers if this is feasible, as the project might not compile and/or produce unwanted behaviours) and debug the bootloader to see what is going on there. Will do the latter immediately after this, meanwhile any further tips based on this additional info would be great 🙂
Thanks.
Kind regards,
Stefano

Stefanodevzone.nordicsemi.com/.../SuccessfulUpdate_5F00_1.txt devzone.nordicsemi.com/.../InterruptedUpdate_5F00_2.txt devzone.nordicsemi.com/.../UpdateAfterInterruptedUpdate_5F00_3.txt

RE: problem with UART update on nRF52840

Hung Bui — Wed, 15 Mar 2023 10:53:55 GMT

Hi Stefano,

I think we need to align the test process before continue.
- I assume you haven't modified the bootloader ?
- In the test you described in your last reply, you used your own update tool, not nrfutil ?
If it's the case, have you tried to do exactly what I did of breaking nrfutil when updating a .zip test package ?
If it doesn't work please try again with SDK v17.1 (just for testing). Then we know it must be something changed in the newer SDK.

If nrfutil works, then we will need to look into what's the different between your update tool and nrfutil.

It's quite strange to see the difference between hex #1 and hex #2. From what I can see it seems when you do BLE update the application goes straight into the primary bank, without storing in secondary bank. When with your update tool it does to the secondary bank before copied to primary bank.

Regarding the change at 0XFE000. It's normal to see that. It's because the MBR_Param page, when in bootloader mode, will be used as mp_dfu_settings_backup_buffer to back up the DFU settings. You can find that that page is identical to page 0xFF000, because it's a backup. It's normal that the bootloader setting change after you start/continue DFU process. It's where the progress of the DFU is stored.

The question here is that why the bootloader couldn't continue and calculate the CRC. I think at this point you would need to step into the bootloader and try to debug it to see why it works with nrfutil (at least on my side) but not with your dfu update tool.
I would suggest to test with the debug version of the bootloader. With the debug version you can put a breakpoint and step into the code. You can also add log and print the log out on RTT.
I assume you have turned on PRN (packet receive notification) when you start the DFU bootloader process ?

The main handling of the DFU process is inside nrf_dfu_req_handler.c. The nrf_dfu_command_req() function is where you should start.

RE: problem with UART update on nRF52840

ste2108 — Tue, 14 Mar 2023 14:13:24 GMT

Dear Hung Bui,

Thank you for your answer. I performed the test you advised yesterday. In the attached file _1 and _2 you can see Nordic MCU flash images where a successful update is performed respectively using the BLE nordic nrf connect app (starting with just the SD+bootloader onboard) and our update tool, which reproduces the Nordic update through the slip protocol via UART, including sending the nordic in bootloader by setting the related pin to active while rebooting. As you can see the two images are the same, except that in the UART version I see the cached copy of the new fw starting at 0x57000. Only Nordic fw is uploaded this time, no other hidden data.

In file _3 you can see an interrupted update. The update started on a Nordic set up as in flash image _2. The differences between _3 and _2, as you can see by comparing the two files, are the "0xff-set" 0x1000 bytes at address 0x05E000 (the starting address different from the one in my former email because of the randomicity of my ctrl+c update interruption) and a few bytes at address 0xFE000, which according to the linker script is where the mbr_params_range is. If I then try to repeat the update after the interruption (image _4) I see that the update process stops after sendin the first 0x1000 bytes. By comparing images _3 and _4 the only difference is in the first 0x1000 bytes at 0x57000, which in _4 are set to 0xff. By looking at the oscilloscope during this update process I can confirm that only the first 0x1000 byte data is sent to the Nordic, but somehow not written, the CRC is not returned to the program updating the Nordic and thus the update process exits. As a comparison, in the successful update before the interruption I can see the whole firmware flowing on the serial link with the oscilloscope.

Could the modified lines at 0xFE000 in _3 have something to do with the failing update? Or could somehow the flash enter a state where it is not writable after I interrupt the update, so that when I try the update again (_4 file) the first 0x1000 data batch is sent to the Nordic but not written? Or maybe this used to be a problem in old Nordic libraries which is now solved (I have to use 16.0.0 for legacy reasons)?

Really running short of ideas here, any tip is very welcome!

Kind regards,

Stefano

devzone.nordicsemi.com/.../Nordic_5F00_Update_5F00_with_5F00_app_5F00_1.txt devzone.nordicsemi.com/.../Nordic_5F00_Update_5F00_from_5F00_serial_5F00_2.txt devzone.nordicsemi.com/.../Nordic_5F00_Update_5F00_from_5F00_serial_5F00_with_5F00_interruption_5F00_3.txt devzone.nordicsemi.com/.../Nordic_5F00_Update_5F00_from_5F00_serial_5F00_after_5F00_interruption_5F00_4.txt

RE: problem with UART update on nRF52840

Hung Bui — Tue, 14 Mar 2023 09:00:22 GMT

Hi Stefano,

The bank_0.image_size that you can find in the code is from the init data inside the .zip file that you use to DFU update the current firmware. If you have a look at a DFU .zip package you will find this (here is the test image in the SDK)

The .dat file contain the information about the size of the image. It will be transmitted to the bootloader and the bootloader use the information to receive the image. Please take a look here: https://infocenter.nordicsemi.com/topic/sdk_nrf5_v17.1.0/lib_bootloader_dfu_validation.html?cp=9_1_3_5_1_1_0#lib_bootloader_dfu_init
If you generate the .zip file after you manipulate with the hex I think the app_size should reflect the new size of the hex. I am not aware of anywhere in the hex we store the original size of the image.

RE: problem with UART update on nRF52840

ste2108 — Mon, 13 Mar 2023 16:34:12 GMT

Dear Hung Bui,

I will perform the test you advised tomorrow morning, could not do it today. Just one question in the meanwhile: I create the manipulated image by adding the additional firmware to the standard Nordic firmware with srecord and then I sign the resulting hex (with "nrf package generate") to be able to upload it to a secure bootloader. I guess what you said means that somewhere in the new hex there is still the value of the standalone nordic image (i.e. bank_0.image_size) in place of the Nordic+custom data size, and this somehow causes problems when the update is interrupted (but does not seem to prevent a "regular" update). Can you tell me exactly where this value is in the hex?
Thank you.

Kind regards,

Stefano

RE: problem with UART update on nRF52840

Hung Bui — Mon, 13 Mar 2023 09:51:11 GMT

Hi Stefano,

Thanks for the explanation.
Your understanding is correct. The bootloader will allocate the new image into bank 1 starting from the first page right after the current application.

If you have a look at update_data_addr_get() you can find how it's calculated.

Basically nrf_dfu_bank1_start_addr() = nrf_dfu_bank0_start_addr() + bank_0.image_size

nrf_dfu_bank0_start_addr() is either MBR or softdevice size, depends if the softdevice is flashed or not.

bank_0.image_size is the image size you provide in your init packet of the previous DFU update.

So what I suspect is that the image size may not match with the actual image size you have including the extra data pages.

I would suggest to test first with a "normal" image , for example our example project. And make sure you can continue after you break the DFU process. After that you can test with your customized image. Please double check which bank_0.image_size you have.

RE: problem with UART update on nRF52840

ste2108 — Fri, 10 Mar 2023 16:27:03 GMT

Dear Hung Bui,

Thank you for your quick answer. The 0x57000 address is not where the application starts (instead it correctly starts at the flash address in the linker script, 0x26000) but where the fw I am sending through the slip protocol is cached before being moved to 0x26000. My understanding is that in the dual bank framework the bootloader starts caching the fw I send through the slip protocol from the first useful address after the "old" application ends, in blocks of 0x1000: indeed my old application starts at 0x26000 and ends at 0x56EE0 and, in the case of successful update (_1 file) I see a copy of the application fw from 0x57000 on. My settings in the sdk_config is not to force dual bank update, but I believe there is enough space to proceed with dual bank (and I assume that in a single bank update an abrupt interruption of the update process would corrupt the application fw).
If I interrupt the update process (_2 file) I see 4k (sorry if in my previous message I said 1k, it is 0x1000=4096) set to 0xFF from the address 0x64000 to 0x65000, which should correspond to the 0x1000 being transmitted when I hit the ctrl-c keys. If I then retry the process (_3 file), the initial 0x1000 (i.e. from 0x57000 to 0x58000) is set to 0xff, which means that the update failed when trying to upload the first 4k data.
I should also mention that the first image present before the first successful update is larger than the one uploaded through the serial protocol, as it contains some additional, extra-fw data we need for our application, which you can see in the _1 dump starting at 0x58000 and finishing at about 0x71000. Even if those data are never reached by the application fw, the packet has been generated in such a way that those bytes are part of the fw itself, i.e. the computed crc is coherent (and indeed the Nordic correctly boots after the application has been loaded). Can you clarify to me how exactly the bootloader sets the initial address where the CACHED fw is stored during the update process before being moved to the application address? In our case I would expect the bootloader to cache the new image at about 0x72000, instead it starts at 0x57000, thus erasing our custom data (which are not relevant to us anymore at that moment).
The dump was taken with J-flash v. 7.84.

Thank you.

Kind regards,

Stefano

RE: problem with UART update on nRF52840

Hung Bui — Fri, 10 Mar 2023 12:47:34 GMT

Hi Stefano,

Did you do any modification to the bootloader ?

Why you application starts at 0x57000 ? Is there any reason for that? It's not a familiar start address to me. How did you configure the bootloader to jump to this address ?

How large is your application ? I would assume the size of the application should be around 0x64000 - 0x57000 = 52 kB ?

I assume it's dual bank DFU you are doing.

Normally the bootloader would receive the new image to the secondary bank (slot 1, the application is in primary bank-slot 0). Only after the image is received properly it will start erasing the original application and swapping the new image to the primary bank. So it's very strange that 0x57000 being erase even before the DFU update process is finished.

Which software did you use to get the flash dump out ? I just want to double check if the address is correct.

RE: problem with UART update on nRF52840

ste2108 — Thu, 09 Mar 2023 17:15:28 GMT

Dear Hung Bui,

thank you for your answer and sorry for being late, I was stuck between multiple deadlines...

In order to do the test I actually interrupted nrfutil with a Ctrl+C, as you do in your example. I am also sure that the device is in DFU mode, as it answers to initial SLIP packets (whose protocol is only implemented in the bootloader, and not in my application). Consider that in my case I do not have flow control, I do not know if it makes any difference.

I also collected memory readouts of the entire chip which might shed some light on what is going on. The first one shows a readout where a successful (i.e. non-interrupted) update has taken place. The second shows a readout of an interrupted update. By comparing with the first readout one sees that 1k starting from 0x64000 is set to 0xff (my firmware starts at 0x57000, which is probably where the bootloader caches the received data before moving them to the address specified by the linker), most probably the one being transferred when I hit ctrl+c. One can also see that some bits in the end are also different, probably those related to the CRC. The third readout refers to an update attempt after the interruption has taken place. One sees that, with respect to the second readout, the initial 1k (i.e. the one starting at 0x57000) is set to 0xff, which means that the first block was erased by the update process but the update could not proceed. From my log I can see indeed that the first 1k was transferred (see slip_log.txt), but the CRC requested after it was never received.

I believe that the interrupted fw update triggers somehow a wrong CRC when a subsequent update attempt is performed, but I cannot clearly see how to cure it. Do you have more insight on the issue?

Thanks.
Regards,

Stefano

PS: the "Nordic" txt files are originally .mot files which I renamed. For some reason Nordic page does not allow to upload .mot files

Stefanodevzone.nordicsemi.com/.../slip_5F00_log.txt devzone.nordicsemi.com/.../Nordic_5F00_after_5F00_successful_5F00_update_5F00_1.txt devzone.nordicsemi.com/.../nordic_5F00_after_5F00_interrupted_5F00_update_5F00_2.txt devzone.nordicsemi.com/.../nordic_5F00_after_5F00_interrupted_5F00_update_5F00_and_5F00_failed_5F00_second_5F00_attempt_5F00_3.txt

RE: problem with UART update on nRF52840

Hung Bui — Fri, 17 Feb 2023 15:11:52 GMT

Hi Stefano,

Could you let me know how exactly you did the test ? How you interrupt the DFU process , by killing nrfutil or by resetting the nRF5 ?

I did a quick test here and found that the nrfutil actually can continue a DFU update that was interrupted, as long as you turn on the PRN.

You can see the screen shot here , I break (Ctrl+C) nrfutil at 17% and then it continue from 17% next time I started it. Then I break again at 51% and then next time it will start at 51% and will eventually finish DFU.

( I use nrfutil v7.x , it requires extra parameter nrf5sdk-tools to use the "legacy" nrfutil feature)

If I do a reset of the chip, the next time it will start from 0 and I don't see the issue that the device don't response.

Have you made sure after you reset the device enter DFU mode? By default you need to hold Button 4 on the DK to make the device enter DFU mode.

RE: problem with UART update on nRF52840

ste2108 — Fri, 17 Feb 2023 14:06:21 GMT

Dear Hung Bui,

Thank you for your quick answer. Trying to understand which is the last unsent packet and restart the update from there can be tricky and we don't want to do that. However, the problem I described in my message takes place not only if I perform an update attempt (through nrfutil) immediately after I cancelled the ongoing one, but also if I stop the update, switch the board off/on and then perform the second DFU attempt. In our custom board we perform a board reset each time we attempt a DFU (because we need to send the board in DFU mode from the application fw) and the result is the same as the one observed on the Nordic updated through nrfutil and connected straight to a serial adapter
Do you have some other ideas on why this happens? Could you do a quick test on your side (it takes justa couple of minutes) to confirm if you see the same behaviour?

Thanks.

Kind regards,

Stefano

RE: problem with UART update on nRF52840

Hung Bui — Fri, 17 Feb 2023 13:04:30 GMT

Hi Stefano,

As far as I know nrfutil doesn't have the feature to continue DFU update after interrupted. The documentation here mentioned about that the DFU master can start by select the last object was sent and can determine that it should re-send the object or continue with the new object.
But this feature is not implemented in nrfutil.

From the log you provided nrfutil simply send a new init packet (initialize packet ) for a new DFU update. The problem here is that the bootloader is still in the process of the interrupted DFU update. I would assume that it wouldn't expect the init packet and didn't response to the init packet send request.

So in your case if you are planning to make your own DFU master then you can implement this feature to continue DFU update after interrupted (you can refer to our BLE DFU library on the phone that you can find inside nRF Toolbox or nRF Device Firmware Update. As far as I know the feature to continue after interrupted is implemented on the apps.

Another option is to send an abort command (0x0C) to reset the bootloader and start a new DFU update. I don't think we have the feature implemented on nrfutil neither. The solution is do a power/pin reset on the board after you have an interrupted DFU update.