Problem with mesh DFU

RE: Problem with mesh DFU

Gerry — Fri, 12 Apr 2019 08:12:16 GMT

Hi Hung,

yes this really helped. Thanks a lot for your support. I think, this could also be interesting for others (especially the optimal_bank_address()), so feel free to make this ticket public if you want.

Regards Gerry

RE: Problem with mesh DFU

Hung Bui — Wed, 10 Apr 2019 16:01:18 GMT

Hi Gerry,

I'm testing here and seeing the same problem. For some reason with your client.hex file the bootloader crashed when receiving the init packet. I tested with even bigger image but it didn't crash.

I noticed that when doing DFU with your image, one of the init packet is transmitted twice (didn't get ack):

Same can be seen in your log. This doesn't happens with other image.

I increased the ACK_WAIT_TIME to 5 and SEND_START_DFU_WAIT_TIME to 5 then it worked now.

I guess it's something to do with the timing. We will need to improve this to avoid dependence on the image size.

But for now I would suggest to just increase those timing numbers. It won't affect the overall update time, only affect the init packets sending time at the beginning.

RE: Problem with mesh DFU

Gerry — Tue, 09 Apr 2019 06:50:34 GMT

Good idea, you can download the project from here. It includes the SDKs and it's good to compile.

To reproduce the issue:

- Open the the project connectivity_client under \source\connectivity_client and compile it (using SES)

- Change the values for Comport and client_version in Version.ini under \DFU

- Connect a PCA10056 and run Prog_Client.bat under \DFU

- Run DFU_Client.bat under \DFU

Here DFU_Client.bat updates the client successful sometimes, but most of the time it ends with ERROR_INVALID_DATA (135) on a DFU data packet.

RE: Problem with mesh DFU

Hung Bui — Mon, 08 Apr 2019 15:22:36 GMT

I'm not sure what could be wrong now.

Is it possible to provide us your source code that we can test on the NRF52 DK and can reproduce the issue here . We need to add some debug log to see at which point it crashed. Please give us the detailed instruction how you test that firmware as well.

Also please make sure you have increased the application version when you want to update the same firmware.

RE: Problem with mesh DFU

Gerry — Fri, 05 Apr 2019 13:50:16 GMT

Hi,

I changed these values in dfu_transport_mesh.py
ACK_WAIT_TIME = 2
SEND_START_DFU_WAIT_TIME = 5

also changed the nrfutil parameter -i to 1000

Now it works approx. every 5th time I start DFU.

When it fails, it outputs:

 0> <t:     141603>, nrf_mesh_dfu.c,  390, 	New firmware!
 0> <t:     141606>, mesh.c,  783, DFU event:13
 0> <t:     141608>, mesh.c,  744, DFU update application 
 0> <t:     141611>, mesh.c,  795, DFU me 
 0> <t:     141612>, mesh.c,  796, optimal_bank_address: 573440 
 0> <t:     141615>, nrf_mesh_dfu.c,  528, 	RADIO TX! SLOT 0, count 255, interval: periodic, handle: FFFD
 0> <t:     141619>, nrf_mesh_dfu.c,  534, Killing a TX slot prematurely (repeats done: 1).
 0> <t:     141623>, nrf_mesh_dfu.c,  561, 	SERIAL TX!
 0> <t:     306203>, nrf_mesh_dfu.c,  528, 	RADIO TX! SLOT 0, count 255, interval: periodic, handle: FFFD
 0> <t:     306207>, nrf_mesh_dfu.c,  534, Killing a TX slot prematurely (repeats done: 0).
 0> <t:     306211>, nrf_mesh_dfu.c,  561, 	SERIAL TX!
 0> <t:     471073>, nrf_mesh_dfu.c,  430, 	DFU start
 0> <t:     471075>, mesh.c,  783, DFU event:9
 0> <t:     471077>, mesh.c,  810, DFU start 
 0> <t:     471080>, nrf_mesh_dfu.c,  528, 	RADIO TX! SLOT 3, count 6, interval: exponential, handle: FFFC
 0> <t:     553710>, nrf_mesh_dfu.c,  329, Erase complete (0x8C000)
 0> <t:     553713>, nrf_mesh_dfu.c,  333, Flash idle.

and RTT:

Flushing com-port...
Opened com-port
Starting DFU upgrade of type 4, SoftDevice size: 0, bootloader size: 0, application size: 122412
Sending DFU start packet, afterwards we wait for the flash on target to be initialized before continuing.
PC -> target: 0502aabbccdd
target -> PC: 0582aabbccdd
Got echo response
Sending DFU init packet
PC -> target: 1378fdff040f587ed7dc59000000010003000000
target -> PC: 16a6045900000001000300000059000000010002000000
target -> PC: 03847800
PC -> target: 1378fdff040f587ed7dc59000000010003000000
target -> PC: 03847800
PC -> target: 1478fcff0000587ed7dcffffffff8b77000000000c
PC -> target: 1478fcff0000587ed7dcffffffff8b77000000000c
target -> PC: 0da2010459000000010003000000
target -> PC: 03847800
Sending firmware file
  [------------------------------------]    1%  00:30:54PC -> target: 1978fcff0100587ed7dc00f0032059620200896202008b620200
target -> PC: 03847887
PC -> target: 1978fcff0100587ed7dc00f0032059620200896202008b620200
PC -> target: 1978fcff0100587ed7dc00f0032059620200896202008b620200
PC -> target: 1978fcff0100587ed7dc00f0032059620200896202008b620200
PC -> target: 1978fcff0100587ed7dc00f0032059620200896202008b620200
target -> PC: 628200896202008b6202001978fcff0100587ed7dc00f0032059620200896202008b6202001978fcff0100587ed7dc00f0032059620200896202008b6202001978fcff0100587ed7dc00f0032059620200896202008b6202001978fcff0100587ed7dc
target -> PC: 03840084


Failed to upgrade target. Error is: Device returned status code ERROR_INVALID_DATA (135) on a DFU data packet.

I agree, the issue changed, but there still seems to be a problem.

Do you have an Idea?

EDIT: Updating the servers over the air, using this client as relay works fine now (this didn't work, when I opened this ticket)

RE: Problem with mesh DFU

Hung Bui — Fri, 05 Apr 2019 12:07:24 GMT

Hi Gerry,

Seems to be a timing issue to me as well, but was the nrftutil still report ERROR_INVALID_DATA (135) ? or it's "crashed on start packet" ?

It doesn't seems to me it's the same issue like before, the bank size issue is solved.

You can try to follow this https://devzone.nordicsemi.com/f/nordic-q-a/43335/mesh-dfu-nrf52840-with-other-binary-file to check if that helps ?

RE: Problem with mesh DFU

Gerry — Fri, 05 Apr 2019 09:30:06 GMT

Yes, the fix is implemented. I now also made an extended Log-output to see all events inside mesh_evt_handler()

static void mesh_evt_handler(const nrf_mesh_evt_t* p_evt)
{
    __LOG(LOG_SRC_APP, LOG_LEVEL_INFO, "DFU event:%u\n",p_evt->type);
    switch (p_evt->type)
    {
        case NRF_MESH_EVT_TX_COMPLETE:
            sending_message = false;
            break;
        case NRF_MESH_EVT_FLASH_STABLE:
            break;
        case NRF_MESH_EVT_DFU_FIRMWARE_OUTDATED:
        case NRF_MESH_EVT_DFU_FIRMWARE_OUTDATED_NO_AUTH:
            if (fw_updated_event_is_for_me(&p_evt->params.dfu))
            {
                __LOG(LOG_SRC_APP, LOG_LEVEL_INFO, "DFU me \n");
                __LOG(LOG_SRC_APP, LOG_LEVEL_INFO, "optimal_bank_address: %u \n", optimal_bank_address());
                ERROR_CHECK(nrf_mesh_dfu_request(p_evt->params.dfu.fw_outdated.transfer.dfu_type,
                                                 &p_evt->params.dfu.fw_outdated.transfer.id,
                                                 optimal_bank_address()));
            }
            else
            {
                __LOG(LOG_SRC_APP, LOG_LEVEL_INFO, "DFU relay \n");
                ERROR_CHECK(nrf_mesh_dfu_relay(p_evt->params.dfu.fw_outdated.transfer.dfu_type,
                                               &p_evt->params.dfu.fw_outdated.transfer.id));
            }
            break;

The RTT-output is:

 0> <t:     215501>, nrf_mesh_dfu.c,  390, 	New firmware!
 0> <t:     215504>, mesh.c,  783, DFU event:13
 0> <t:     215506>, mesh.c,  744, DFU update application 
 0> <t:     215508>, mesh.c,  795, DFU me 
 0> <t:     215510>, mesh.c,  796, optimal_bank_address: 573440 
 0> <t:     215513>, nrf_mesh_dfu.c,  528, 	RADIO TX! SLOT 0, count 255, interval: periodic, handle: FFFD
 0> <t:     215517>, nrf_mesh_dfu.c,  534, Killing a TX slot prematurely (repeats done: 1).
 0> <t:     215521>, nrf_mesh_dfu.c,  561, 	SERIAL TX!
 0> <t:     314224>, nrf_mesh_dfu.c,  528, 	RADIO TX! SLOT 0, count 255, interval: periodic, handle: FFFD
 0> <t:     314228>, nrf_mesh_dfu.c,  534, Killing a TX slot prematurely (repeats done: 0).
 0> <t:     314231>, nrf_mesh_dfu.c,  561, 	SERIAL TX!
 0> <t:     413564>, nrf_mesh_dfu.c,  430, 	DFU start
 0> <t:     413567>, mesh.c,  783, DFU event:9
 0> <t:     413569>, mesh.c,  810, DFU start 
 0> <t:     413572>, nrf_mesh_dfu.c,  528, 	RADIO TX! SLOT 5, count 6, interval: exponential, handle: FFFC
 0> <t:     496202>, nrf_mesh_dfu.c,  329, Erase complete (0x8C000)
 0> <t:     496205>, nrf_mesh_dfu.c,  333, Flash idle.

So optimal_bank_addr is o573440 and the last event is: 9/start

RE: Problem with mesh DFU

Gerry — Fri, 05 Apr 2019 08:12:19 GMT

Since this now seems to be a problem with my client project, I uploaded my project here.

It includes the SDKs and should be good to compile. The client project can be found under source/connectivity_client.

The scripts I use for programming and DFU are in the folder DFU (Prog_Client.bat and DFU_Client.bat)

It seems to me like a timing issue, since DFU worked one time now. (1 time out of maybe 20-30 times I tried)

RE: Problem with mesh DFU

Hung Bui — Thu, 04 Apr 2019 14:41:35 GMT

Have you implemented the fix in your original DFU application ?
Could you printout the optimal_bank_addr calculation when it's calculated inside mesh_evt_handler() ?

Getting RTT logging inside dfu_evt_handler() also important to know what wrong in your case ("Abort event. Reason:")

RE: Problem with mesh DFU

Gerry — Thu, 04 Apr 2019 14:36:14 GMT

Ok, I programmed your hexfile (DFU-example) and then run my DFU script for the client.

This works well, the RTT-output looks as follows:

 0> <t:          1>, main.c,  223, ----- Bluetooth Mesh DFU Example -----
 0> <t:          3>, main.c,  232, rom_base   26201
 0> <t:          5>, main.c,  233, rom_end    42BE8
 0> <t:          7>, main.c,  234, rom_length 1C9E7
 0> <t:          9>, main.c,  236, bank_addr   43000
 0> <t:      12921>, serial_handler_prov.c,  267, Generating encryption keypair...
 0> <t:      17768>, main.c,  241, optimal bank addr   8C000
 0> <t:      17770>, main.c,  242, Initialization complete!
 0> <t:      22540>, main.c,  263, Enabling serial interface...
 0> <t:      22543>, mesh_app_utils.c,   65, Device UUID (raw): D617B16F3FEC8643A3C33E33908A3BED
 0> <t:      22546>, mesh_app_utils.c,   70, Device UUID : 6FB117D6-EC3F-4386-A3C3-3E33908A3BED
 0> <t:      22554>, nrf_mesh_dfu.c,  528, 	RADIO TX! SLOT 0, count 255, interval: periodic, handle: FFFE
 0> <t:      22559>, nrf_mesh_dfu.c,  561, 	SERIAL TX!
 0> <t:      22565>, main.c,  271, DFU example started!
 0> <t:    2101608>, serial_handler_openmesh.c,   51, 	DFU Data!
 0> <t:    2101611>, nrf_mesh_dfu.c,  390, 	New firmware!
 0> <t:    2101614>, nrf_mesh_dfu.c,  528, 	RADIO TX! SLOT 0, count 255, interval: periodic, handle: FFFD
 0> <t:    2101618>, nrf_mesh_dfu.c,  534, Killing a TX slot prematurely (repeats done: 7).
 0> <t:    2101622>, nrf_mesh_dfu.c,  561, 	SERIAL TX!
 0> <t:    2200674>, serial_handler_openmesh.c,   51, 	DFU Data!
 0> <t:    2200677>, nrf_mesh_dfu.c,  528, 	RADIO TX! SLOT 0, count 255, interval: periodic, handle: FFFD
 0> <t:    2200681>, nrf_mesh_dfu.c,  534, Killing a TX slot prematurely (repeats done: 30).
 0> <t:    2200684>, nrf_mesh_dfu.c,  561, 	SERIAL TX!
 0> <t:    2299386>, serial_handler_openmesh.c,   51, 	DFU Data!
 0> <t:    2299389>, nrf_mesh_dfu.c,  430, 	DFU start
 0> <t:    2299392>, nrf_mesh_dfu.c,  528, 	RADIO TX! SLOT 3, count 6, interval: exponential, handle: FFFC
 0> <t:    2401288>, serial_handler_openmesh.c,   51, 	DFU Data!
 0> <t:    2401291>, nrf_mesh_dfu.c,  528, 	RADIO TX! SLOT 4, count 3, interval: exponential, handle: FFFC
 0> <t:    2424970>, serial_handler_openmesh.c,   51, 	DFU Data!
 0> <t:    2424973>, nrf_mesh_dfu.c,  528, 	RADIO TX! SLOT 5, count 3, interval: exponential, handle: FFFC
...

my client-application uses the same UART-pins like the DFU-example with UART0

Seems like the problem only occurs, when my client is running at DFU start.

RE: Problem with mesh DFU

Hung Bui — Thu, 04 Apr 2019 13:17:01 GMT

Hi Gerry,

Your client.zip worked fine for me here.

I haven't tested to flash your client.hex file but only use our SDK's dfu example to do a DFU. (I'm not sure if you use the same UART pins in your client.hex,)

Could you try testing using our DFU example ? I attached our main.c file with the fixplease replace with the original main.c and send us the RTT log file after you test. The .hex of the DFU application is also attached.

devzone.nordicsemi.com/.../dfu_5F00_nrf52840_5F00_xxAA_5F00_s140_5F00_6.1.0.hex

devzone.nordicsemi.com/.../main.c

RE: Problem with mesh DFU

Gerry — Thu, 04 Apr 2019 08:47:19 GMT

Hi,

thanks for you investigation on this. This changed a lot. "Crashed on start package" disapeared now.

ERROR_INVALID_DATA (135) persists. I made a log output to RTT for the bank_address:

        case NRF_MESH_EVT_DFU_FIRMWARE_OUTDATED_NO_AUTH:
            if (fw_updated_event_is_for_me(&p_evt->params.dfu))
            {
                __LOG(LOG_SRC_APP, LOG_LEVEL_INFO, "DFU me \n");
                __LOG(LOG_SRC_APP, LOG_LEVEL_INFO, "optimal_bank_address: %u \n", optimal_bank_address());
                ERROR_CHECK(nrf_mesh_dfu_request(p_evt->params.dfu.fw_outdated.transfer.dfu_type,
                                                 &p_evt->params.dfu.fw_outdated.transfer.id,
                                                 optimal_bank_address()));
            }

The output on RTT is: optimal_bank_address: 573440

During the DFU I get this output now:

Upgrading target on COM6 with DFU package C:\Users\se\Desktop\Oblamatik\Connectivityair_15.2_3.10\DFU\pc-nrfutil-mesh_dfu\client.zip. Flow control is enabled.
Flushing com-port...
Opened com-port
Starting DFU upgrade of type 4, SoftDevice size: 0, bootloader size: 0, application size: 122268
Sending DFU start packet, afterwards we wait for the flash on target to be initialized before continuing.
PC -> target: 0502aabbccdd
target -> PC: 0582aabbccdd
Got echo response
Sending DFU init packet
PC -> target: 1378fdff040f46d4dcf159000000010002000000
target -> PC: 16a6045900000001000200000059000000010001000000
target -> PC: 03847800
PC -> target: 1378fdff040f46d4dcf159000000010002000000
target -> PC: 03847800
PC -> target: 1478fcff000046d4dcf1ffffffff6777000000000c
PC -> target: 1478fcff000046d4dcf1ffffffff6777000000000c
target -> PC: 0da2010459000000010002000000
target -> PC: 03847800
Sending firmware file
  [------------------------------------]    1%  00:20:58PC -> target: 1978fcff010046d4dcf100f0032059620200896202008b620200
target -> PC: 03847887
PC -> target: 1978fcff010046d4dcf100f0032059620200896202008b620200
PC -> target: 1978fcff010046d4dcf100f0032059620200896202008b620200
PC -> target: 1978fcff010046d4dcf100f0032059620200896202008b620200
PC -> target: 1978fcff010046d4dcf100f0032059620200896202008b620200
target -> PC: 628200896202008b6202001978fcff010046d4dcf100f0032059620200896202008b6202001978fcff010046d4dcf100f0032059620200896202008b6202001978fcff010046d4dcf100f0032059620200896202008b6202001978fcff010046d4dcf1
target -> PC: 03840084


Failed to upgrade target. Error is: Device returned status code ERROR_INVALID_DATA (135) on a DFU data packet.

devzone.nordicsemi.com/.../client.hex

Regards

Gerry

RE: Problem with mesh DFU

Hung Bui — Wed, 03 Apr 2019 14:03:26 GMT

Hi Gerry,

Good news, I think we have found the bug in our dfu application code that limit the size of the new image to be not bigger than the size of the original application.

Please apply this fix on the application:

Add this code to main.c

static uint32_t * optimal_bank_address(void)
{
    /* The incoming transfer has to fit on both sides of the bank address: First it needs to fit
     * above the bank address when we receive it, then it needs to fit below the bank address when
     * we install it. We want to put the bank address in the middle of the available application
     * code area, to maximize the potential transfer size we can accept. */
    const uint32_t * p_start;
    uint32_t dummy;
    ERROR_CHECK(mesh_stack_persistence_flash_usage(&p_start, &dummy));

    uint32_t middle_of_app_area = (CODE_START + (intptr_t) p_start) / 2;

    /* The bank can't start in the middle of the application code, and should be page aligned: */
    return (uint32_t *) ALIGN_VAL(MAX(middle_of_app_area, CODE_END), PAGE_SIZE);
}

And replace (uint32_t*) bank_addr with optimal_bank_address() in mesh_evt_handler()

In addition, to solve the issue "Crashed on start packet" you can increase ACK_WAIT_TIME to 2 and rebuild nrfutil for dfu mesh and test again.

Please let us know if the issues persist.

RE: Problem with mesh DFU

Hung Bui — Tue, 02 Apr 2019 12:14:45 GMT

Hi Gerry,

I'm sorry that the case has been dragging for so long. I have reproduced the issue with your client.zip file here. Starting the investigation, will try our best to find the root cause.

RE: Problem with mesh DFU

Gerry — Tue, 02 Apr 2019 09:22:47 GMT

Additional info:

Normally I get the second output (Crashed on start packet). The first one (ERROR_INVALID_DATA (135)) I only got 2 or 3 times.

Here are my Hex- and generated zip-files. Maybe you can reproduce the issue.

devzone.nordicsemi.com/.../connectivity_5F00_client.hex

devzone.nordicsemi.com/.../client.zip