Zigbee OTA is aborted when network is congested and Coordinator is far from the End Device

[KKoovalsky]

Hello!

Lately we are testing FOTA functionality in our products. FOTA works well when the distance, between the ED (our device under test) and the coordinator, is short, but when the distance increases to 3-4 walls (2 rooms in between), and routing goes through routers, then FOTA is aborted.

What's weird is that this is related to hardware as well, because on some boards FOTA works on such a distance, but on other it doesn't. That makes me think that it's related to the antenna circuitry, whose quality differs across PCBs, which are made by different manufacturers.

According to the previous paragraph, the question is: is that possible that ZBOSS stack will abort DFU when e.g. the signal strength is low, or SNR is low?

We use nRF52840 chip, a custom board, nRF SDK for Thread and Zigbee in the version 4.2.1

I attach the logs from our device:

<info> app: DRV_ZIGBEE: Starting.
<info> app: State changed: not paired
<info> app: Loading application production config
<info> app:     IEEE address: f4ce36000000000e
<info> app:     Channel mask 07FFF800
<info> app: Production configuration successfully loaded
<info> app: Zigbee stack initialized
<info> app: Device started for the first time
<info> app: Start network steering
<info> app: Started network rejoin procedure.
<info> app: DRV_ZIGBEE: App: updating local temperature attribute to 30.62 (ZCL compliant: 0x0BF7)
<info> app: DRV_ZIGBEE: Joined network successfully
<info> app: State changed: paired, network ON
<info> app: Joined network successfully (Extended PAN ID: 0015bc002c801165, PAN ID: 0x796A)
<info> app: Network rejoin procedure stopped as NOT scheduled.
<info> app: DRV_ZIGBEE: Starting DFU procedure
<info> app: DRV_ZIGBEE: Zigbee DFU Aborted
<info> app: ABORT Zigbee DFU
<info> background_dfu: Current DFU Diag version: Jul 11 2020 21:09:46, 0xB138C66C
<debug> nrf_dfu_settings: Destination settings are identical to source, write not needed. Skipping.
<warning> app: Wanted to resume download but it was already resumed

I attach sniffed traffic as well. The dump contains all the packets which are exchanged between the OTA client and the coordinator, so since the "ZCL OTA: Query Next Image Request" to "ZCL OTA: Upgrade End Request".
FailedOTA_dump.pcapng

Could you help mi in finding the reasons of such a behavior?

[Edvin]

Hello,

Are all the nodes nRF52840 devices? You mention different PCBs, but are they using the same version of the nRF52840? Do you happen to know what revision you are running?

BR,
Edvin

[KKoovalsky]

Regarding the revision, I have also thought about that might be related to nRF52840 revision. All of them are:
Q1AACD
1835SEL
so this is not the cause of the problem.

The nodes are not nRF52840. The End Devices we test are, obviously, nRF52840-based devices. The routers are Smart Plugs from a third party, so is the Zigbee Coordinator.

[Edvin]

Are you sure about "QIAACD"? Did you mean QIAAC0 or QIAAD0? However, it is not QIAACA0 or QIAADA0? (engineering versions).

I just wanted to check, because the release notes of the T&Z SDK v4.1.0 (which I assume you meant to write), it says that only pca10056v1.0.0 or later is supported, meaning the preview DKs (PDK) is not supported, and in fact has a bug that will trigger in DFU. pca10056 v1.0.0 uses the QIAAC0, so if that is what you are using, then that shouldn't be the problem. I have to run this by our Zigbee team. I will keep you posted. 

In case this is needed later. Are you able to reproduce this using only DKs? In other words, so that we can reproduce it in the office if needed? Also, can you share the command that you use to initialize the OTA client, and what command you use to generate the Zigbee image. I assume it is correct, if you say that it works in some cases.

Best regards,

Edvin

[KKoovalsky]

Sorry, it's "QIAAC0" - the last character looks like D in bad light.

It would be hard for me to reproduce that on the DKs. I could try, but the thing is that we use I2C driver to communicate with external devices, and also other drivers, so that means that some hardware related work should be done and some resolderding. This makes the environment not simulated well enough.

And could you elaborate about what reproducing scenario did you mean? Shall the Routers and the Roordinator be also on the DKs? Or did you mean to have only the End Device run on the DK (to replace our custom board with the DK running as the End Device? The former could be reproduced with some effort, but the latter would be really hard, because of lack of the DKs we have in our office.

One more thing which I need to add: in our lab/office, in which we run the tests, there is quite a lot of various 2.4 GHz based networks. We have a few Zigbee Networks, more than 10 BT devices and also a congested WiFi network.

Edvin, you wrote: "can you share the command that you use to initialize the OTA client" - what command and what initialization do you mean?

This is how we generate the images:

nrfutil settings generate --family NRF52840 --application firmware.hex  \                              
                 --application-version 0x00034201                       \                        
                 --bootloader-version 1 --bl-settings-version 2         \
                 --app-boot-validation VALIDATE_ECDSA_P256_SHA256       \
                 --key-file dfu_keys/priv.pem settings.hex
                 
mergehex -m firmware.hex settings.hex -o firmware_with_settings.hex                                         

nrfutil pkg generate --hw-version 52 --sd-req 0x00                                  \                        
                 --application-version 0x00034201                                   \           
                 --application firmware.hex                                         \
                 --key-file dfu_keys/priv.pem app_dfu_package.zip --zigbee True     \        
                 --zigbee-manufacturer-id 50001 --zigbee-image-type 0               \
                 --zigbee-comment LertaBB                                           \
                 --zigbee-ota-hw-version 52 --zigbee-ota-fw-version 0x00034201                  

[Edvin]

Hello,

Can you try to enable some logging on the node that receives the update?

Try to set:

#define BACKGROUND_DFU_CONFIG_LOG_LEVEL 4

and 

#define NRF_LOG_DEFAULT_LEVEL 4

in sdk_config.h on the OTA client. What does the log say?

BR,,

Edvin