Mesh DFU problem

[Igor]

We have a Mesh app that should be updated via DFU.
The problem is that if a device (for whatever reason) restarts in the middle of DFU or it is powered up later (i.e. not working during DFU), it newer receives update.
Here is a app log (device started after DFU):

<t:         23>, ble_softdevice_support.c,  162, sd_ble_enable: app_ram_base should be adjusted to 0x20002DA0
<t:        489>, main.c,   68, Initializing and adding models
<t:        496>, main.c,  111, rom_base   26201
<t:        498>, main.c,  112, rom_end    424D4
<t:        500>, main.c,  113, rom_length 1C2D3
<t:        502>, main.c,  114, bank_addr   43000
<t:        509>, bizlogic.c,  213, Bizlogic init
<t:        511>, gap_listener.c,   85, GAP scanner started.
<t:        514>, gap_advertiser.c,   74, GAP advertiser init
<t:       5389>, nrf_mesh_dfu.c,  529, 	RADIO TX! SLOT 0, count 255, interval: periodic, handle: FFFE
<t:       5398>, main.c,  141, Started.
<t:       5510>, nrf_mesh_dfu.c,  391, 	New firmware!
<t:       5512>, dfu.c,   48, NRF_MESH_EVT_DFU_FIRMWARE_OUTDATED_NO_AUTH
<t:       5515>, nrf_mesh_dfu.c,  529, 	RADIO TX! SLOT 0, count 255, interval: periodic, handle: FFFD
<t:       5519>, nrf_mesh_dfu.c,  535, Killing a TX slot prematurely (repeats done: 0).
<t:       8167>, nrf_mesh_dfu.c,  529, 	RADIO TX! SLOT 0, count 255, interval: periodic, handle: FFFD
<t:       8171>, nrf_mesh_dfu.c,  535, Killing a TX slot prematurely (repeats done: 0).

,

Device obviously knows that is should be updated (NRF_MESH_EVT_DFU_FIRMWARE_OUTDATED_NO_AUTH) but it fails to transfer firmware (Killing a TX slot prematurely).
How to fix this?

And also when the update is transmission over the Mesh is stopped?
If we have Client with ID = 1 and Server with ID = 2, and if Server devices are now updated, they still broadcast this new firmware to all devices and if you want to update Client device, before you can start it is already receiving update from Server devices and this update is transmitted all the time.
How to stop firmware re-transmission over the Mesh, so that you can update Client without Server firmware being relayed, because if you send init packet via serial to Client you get *84 78 87*. ?
And how to update Server devices then to newer version after like a day or so if some device is still broadcasting older firmware?

[Mesh SDK 3.1, nRF SDK 15.2, SD 6.1, nRF52840]

[Edvin]

Hello,

Device obviously knows that is should be updated (NRF_MESH_EVT_DFU_FIRMWARE_OUTDATED_NO_AUTH) but it fails to transfer firmware (Killing a TX slot prematurely).

Did you also restart the transmission vie serial to the first device, or does the device enter in the middle of the same transmission that it was reset? 

And also when the update is transmission over the Mesh is stopped?

Then the DFU will eventually time out. Check out TIMER_START_TIMEOUT_US and TIMER_DATA_TIMEOUT_US in nrf_mesh_dfu.c on line 74 and 75.

 

If we have Client with ID = 1 and Server with ID = 2, and if Server devices are now updated, they still broadcast this new firmware to all devices

 yes. 

 

and if you want to update Client device, before you can start it is already receiving update from Server devices and this update is transmitted all the time.

 I don't understand what you ask for here.

  

 

How to stop firmware re-transmission over the Mesh, so that you can update Client without Server firmware being relayed, because if you send init packet via serial to Client you get *84 78 87*. ?
And how to update Server devices then to newer version after like a day or so if some device is still broadcasting older firmware?

 They will not retransmit for that long. You can't disable retransmits. But again, check out the timeout variables. They decide how long to stay on the same update until it times out.

BR,

Edvin

[Tomi]

It fails in all cases

Examples
1. -Run DFU for Server on 4 devices - 1 Client(serial) and 3 Servers
    -In the middle of DFU update, disconnect 1 server
    -Reconnect it again and only "NRF_MESH_EVT_DFU_FIRMWARE_OUTDATED_NO_AUTH" happens and that is it. It goes into adding event to timer but nothing else happens.
    -After some time, 2 servers get updated and that is it. The on disconnected stays outdated and never gets update

2. -Run DFU for Server on 4 devices - 1 Client(serial) and 3 Servers
    -End DFU successfully
    -Connect new Server into network, which has older version
    -Same thing as in example 1, shows New firmware but nothing happens and it never updates

So if there is some case when it looses connection/restarts or you connect new one, they will never update to current mesh firmware

[Edvin]

Tomi said:

So if there is some case when it looses connection/restarts or you connect new one, they will never update to current mesh firmware

 If the device joins in in the middle of a DFU, it will have missed some packets, and therefore, it will not continue to write the rest of the packets. What you need to do is to start the update with the same DFU image again when the device that was turned off is turned back on. Then it will accept the new packet, and the rest of the nodes will reject it (because it is already up to date with an application with application-version with the same number). However, the other nodes will still retransmit/relay the packets for the DFU update, to help the remaining nodes get up to date.

BR,

Edvin

[Tomi]

Thank you for response!

So if we have
Server 1 - 50% Updated
Server 2 - 50% Updated
Server 3 - disconnected, just came back online - 0%

1. We can detect Server 3 power on/off and without sending Init packet to Client again, we can just start DFU from 0% and Server 1 and Server 2 will wait, till Server 3 gets to 50% and then all Server 1, 2 and 3 will be on 50% and then DFU will resume on all devices till 100%? So basically they will wait till this device gets up to date with them?

2. If this is true, and timeout for DFU is 1h (currently) and DFU is 95% done (~50min) on Server 1 and 2, and if we start DFU again because Server 3 is on 0%, will Server 1 and 2 timeout after 10 minutes, or does this timeout reset in this case and if Server 3 is updating this timeout will not occur on Server 1 and 2?

Thanks in advance!

[Edvin]

1: Have you tested? I haven't studied the state machine flow for the Mesh bootloader, but it should be quite easy to test.

2: Yes. If the overall timeout is 1h, then the DFU will time out after 1 hour, regardless, as long as the DFU update is not done. However, this is for the TIMER_START_TIMEOUT_US. The TIMER_DATA_TIMEOUT_US will not time out as long as it receives DFU packets, regardless whether it is for itself or for someone else.

Best regards,

Edvin

[Tomi]

1. I have tested this and this works but I have to make TID value, when sending DFU same as before, because this value is random generated when you start DFU, and on next DFU only new devices / devices already updated will run, not devices waiting on like 50%

2. Ok thank you!

3. With our tests done this DFU is not really usable in working environments and it looks like it works only if everything is perfect and we don't know what would be the best thing to do...


-If devices are far apart with like 3+ jumps between mesh devices, it usually just aborts DFU on device farthest from device running DFU, even when sending really slow(each packet on 2+second - tested with all devices provisioned, sending something each 10sec)
-You never know when some device stops on 1% and this nrfutil update is pretty useless because you only know that device on Serial got update and for others is a mystery until you update it 100% and request their version.

So only thing I see that can be done is 
-run DFU on other device (not for example Client which has provisioned devices)
-Send to client all DFU_STARTS / DFU_ENDS from all nodes
-If you get DFU_END from some Server, redo whole DFU procedure
-Pray it works, or you can update DFU all day

Is there something else that we can do here to make this DFU more reliable and not so time consuming?

Thank you in advance!

[Edvin]

DFU on Mesh is time consuming. This is because Mesh is a low power network, with low throughput. When you say it isn't working with 3+ jumps. Are they far apart? Do you experience a lot of loss on a regular basis between these nodes? Have you tried turning up the relay count on the nodes? 

Look in nrf_mesh_config_core.h. What is your MESH_FEATURE_RELAY_ENABLED, and what is your CORE_TX_REPEAT_RELAY_DEFAULT? If they are both 1, can you try to increase CORE_TX_REPEAT_RELAY_DEFAULT to  2 or 3?

How many nodes are in the network that you are trying to perform the DFU on?

BR,

Edvin

[Tomi]

Hi!

MESH_FEATURE_RELAY_ENABLED is set to 1

I tried setting CORE_TX_REPEAT_RELAY_DEFAULT to 2 or 3 but there is no difference.


1. A week ago I tried
-compiling Release and creating .zip file
-Updating with 500ms
-I forgot to change server app version from 1 -> 4 so it updated thinking it was version 4(I set 4 when making zip file), but then it flashed back to 1 and client devices, are still sending update for server all the time and set other servers into New firmware so it doesn't look this abort timeout is working here because I left devices untouched for atleast 4 days and when you reboot 1 device you always get "New firmware".
Shouldn't client devices timeout after 10minutes? Why do I get update is in progress after 4 days?

2. 1 week ago it worked, but when I tried it again today it doesn't work and I get timeout reason after some time like packets where lost in transition

2. We have our own dfu.c file 

bool dfu_mode = true;
static nrf_mesh_evt_handler_t dfu_event_handler;

bool fw_updated_event_is_for_me(const nrf_mesh_evt_dfu_t *p_evt) {
    switch (p_evt->fw_outdated.transfer.dfu_type) {
    case NRF_MESH_DFU_TYPE_APPLICATION:
        return (p_evt->fw_outdated.current.application.app_id == p_evt->fw_outdated.transfer.id.application.app_id &&
                p_evt->fw_outdated.current.application.company_id == p_evt->fw_outdated.transfer.id.application.company_id &&
                p_evt->fw_outdated.current.application.app_version < p_evt->fw_outdated.transfer.id.application.app_version);

    case NRF_MESH_DFU_TYPE_BOOTLOADER:
        return (p_evt->fw_outdated.current.bootloader.bl_id == p_evt->fw_outdated.transfer.id.bootloader.bl_id &&
                p_evt->fw_outdated.current.bootloader.bl_version < p_evt->fw_outdated.transfer.id.bootloader.bl_version);

    case NRF_MESH_DFU_TYPE_SOFTDEVICE:
        return false;

    default:
        return false;
    }
}

void dfu_event_cb(const nrf_mesh_evt_t *p_evt) {
    switch (p_evt->type) {
    case NRF_MESH_EVT_DFU_FIRMWARE_OUTDATED:
    case NRF_MESH_EVT_DFU_FIRMWARE_OUTDATED_NO_AUTH:
        __LOG(LOG_SRC_APP, LOG_LEVEL_INFO, "NRF_MESH_EVT_DFU_FIRMWARE_OUTDATED_NO_AUTH\n");
        if (fw_updated_event_is_for_me(&p_evt->params.dfu))
            ERROR_CHECK(nrf_mesh_dfu_request(p_evt->params.dfu.fw_outdated.transfer.dfu_type, &p_evt->params.dfu.fw_outdated.transfer.id, (uint32_t *)bank_addr));
        else
            ERROR_CHECK(nrf_mesh_dfu_relay(p_evt->params.dfu.fw_outdated.transfer.dfu_type, &p_evt->params.dfu.fw_outdated.transfer.id));
        break;

    case NRF_MESH_EVT_DFU_START:
        __LOG(LOG_SRC_APP, LOG_LEVEL_INFO, "NRF_MESH_EVT_DFU_START\n");
        dfu_mode = true;
        break;

    case NRF_MESH_EVT_DFU_END:
        __LOG(LOG_SRC_APP, LOG_LEVEL_INFO, "NRF_MESH_EVT_DFU_END\n");
        dfu_mode = false;
        send_dfu(p_evt->type);
        break;

    case NRF_MESH_EVT_DFU_BANK_AVAILABLE:
        __LOG(LOG_SRC_APP, LOG_LEVEL_INFO, "NRF_MESH_EVT_DFU_BANK_AVAILABLE\n");
        ERROR_CHECK(nrf_mesh_dfu_bank_flash(p_evt->params.dfu.bank.transfer.dfu_type));
        break;
    default:
        //__LOG(LOG_SRC_APP, LOG_LEVEL_INFO, "Unhandled Mesh Event: %d \n", p_evt->type);
        break;
    }
}

void dfu_init() {
    dfu_mode = false;
    rom_length = (uint32_t)rom_end - rom_base;
    bank_addr = (uint32_t)(rom_end & FLASH_PAGE_MASK) + FLASH_PAGE_SIZE;
    __LOG(LOG_SRC_APP, LOG_LEVEL_INFO, "rom_base   %X\n", rom_base);
    __LOG(LOG_SRC_APP, LOG_LEVEL_INFO, "rom_end    %X\n", rom_end);
    __LOG(LOG_SRC_APP, LOG_LEVEL_INFO, "rom_length %X\n", rom_length);
    __LOG(LOG_SRC_APP, LOG_LEVEL_INFO, "bank_addr  %X\n", bank_addr);
    dfu_event_handler.evt_cb = dfu_event_cb;
    nrf_mesh_evt_handler_add(&dfu_event_handler);
}

Should there be some other event handler here added or is that it?

3. When trying DFU there is so many options it goes wrong
-It can just stop when starting *RADIO TX! SLOT 1* and it says "count 99" so like it was trying to send something but couldn't and then it doesn't go on
-It can Abort randomly with timeout
-It can say NRF_MESH_DFU_END_ERROR_PACKET_LOSS
-or NRF_MESH_DFU_END_ERROR_BANK_IN_BOOTLOADER_AREA

And if I want to stop ALL sending via mesh when making DFU I cannot know any of this until DFU is complete (after 1h) and I can find out that 3/4 of devices got some error and don't work.


4. What if 
Client 1 and Server 1 currently have firmware 1 and they are broadcasting it
Then I run DFU on Client 2 with firmware 2 for Servers.
4.1 Will Server 1 and others be updated properly?
4.2 Can this somehow screw DFU if some old DFU is still in the air?
4.3 After Server update, should I delete DFU on Client devices, so they don't broadcast it anymore?

5. Is there anything else I can try? 

In python script I changed random TID to static, so I can send same firmware with same ID if something went wrong, but I don't see this changed have any influence on this problems.

Thank you in advance!

[Edvin]

Hello,

 

Tomi said:

Shouldn't client devices timeout after 10minutes?

 + 

Tomi said:

We have our own dfu.c file 

It is difficult to say what's going on without knowing the extent of your changes. 

 

Tomi said:

Should there be some other event handler here added or is that it?

 You still have the dfu_evt_handler() in nrf_mesh_dfu.c, right?

In this function, do you still use the TIMER_START_TIMEOUT_US and the TIMER_DATA_TIMEOUT_US? But you never see "Timeout fired @..." anywhere in the log? Or if you set a breakpoint there? (in nrf_mesh_dfu.c in timer_timeout?

I know it isn't straight forward, but by default, it should work like this:

One device initiates the DFU (the one being connected to the computer via serial), and starts transmitting the DFU packets. When the first DFU packet is received, each device will get the BLE_EVT_TYPE_DFU_START, and start a timer with timeout TIMER_START_TIMEOUT_US. Then, after that, each incoming DFU packet will trigger the BL_EVT_TYPE_DFU_DATA_SEGMENT, and start/restart the timer with TIMER_DATA_TIMEOUT_US. If any of these timeouts hit, the timer will abort the DFU session. 

In addition to starting these timers, each node will check whether the packet is intended for itself or not in the mesh_evt_handler() in main.c. It uses fw_updated_event_is_for_me() to check this. As you can see, it will, based on this, either store the data from the packet, then relay it, or just relay it. 

Can you check whether any of the timeouts fire? You can test this easier by reducing TIMER_DATA_TIMEOUT_US and TIMER_START_TIMEOUT_US to one minute or so, instead of 10 minutes.