[Zigbee] Unable to subscribe to attributes and/or receive reports on device with multiple endpoints

[happy_nordic_user]

Hello, I'm using Zigbee SDK 4.0.0 with Segger Embedded Studio 4.30 and custom pca10059-compatible debug board.

I ran into difficulties when trying to subscribe to attributes when the device has 4 identical endpoints (endpoint numbers are 10, 11, 12 and 13).

Each endpoint provides on/off and level control clusters with two attributes that I can subscribe to:

#define ZB_HA_DECLARE_LIGHT_EP(ep_name, ep_id, cluster_list)                                                                               \
    ZB_ZCL_DECLARE_HA_DIMMABLE_LIGHT_SIMPLE_DESC_VA(ep_name,                                                                               \
                                                    ep_id,                                                                                 \
                                                    ZB_HA_DIMMABLE_LIGHT_IN_CLUSTER_NUM,                                                   \
                                                    ZB_HA_DIMMABLE_LIGHT_OUT_CLUSTER_NUM);                                                 \
    ZBOSS_DEVICE_DECLARE_REPORTING_CTX(reporting_info##ep_name, ZB_HA_DIMMABLE_LIGHT_REPORT_ATTR_COUNT);                                   \
    ZBOSS_DEVICE_DECLARE_LEVEL_CONTROL_CTX(cvc_alarm_info##ep_name, ZB_HA_DIMMABLE_LIGHT_CVC_ATTR_COUNT);                                  \
    ZB_AF_DECLARE_ENDPOINT_DESC(ep_name,                                                                                                   \
                                ep_id,                                                                                                     \
                                ZB_AF_HA_PROFILE_ID,                                                                                       \
                                0,                                                                                                         \
                                NULL,                                                                                                      \
                                ZB_ZCL_ARRAY_SIZE(cluster_list, zb_zcl_cluster_desc_t),                                                    \
                                cluster_list,                                                                                              \
                                (zb_af_simple_desc_1_1_t *)&simple_desc_##ep_name,                                                         \
                                ZB_HA_DIMMABLE_LIGHT_REPORT_ATTR_COUNT,                                                                    \
                                reporting_info##ep_name,                                                                                   \
                                ZB_HA_DIMMABLE_LIGHT_CVC_ATTR_COUNT,                                                                       \
                                cvc_alarm_info##ep_name)

ZB_HA_DIMMABLE_LIGHT_REPORT_ATTR_COUNT is defined as 2, so each endpoint contains array of two zb_zcl_reporting_info_t structs.

At start ZB_AF_REGISTER_DEVICE_CTX macro registers all endpoints and fills fields of zb_zcl_reporting_info_t structs.

zb_zcl_reporting_info_t of first 3 endpoints (10, 11 and 12) receive 0x11 at flags field, but last endpoint (13) receive 0x13 at flags field.

After the device starts and connects to the network I try to bind and subscribe to attributes from the cli tool. Subscribe command fails for endpoints 10, 11 and 12 but succeeds for the last endpoint 13. Fail response looks like this:

ZigBee Application Support Layer Data, Dst Endpt: 64, Src Endpt: 10
    Frame Control Field: Data (0x40)
    Destination Endpoint: 64
    Cluster: Level Control (0x0008)
    Profile: Home Automation (0x0104)
    Source Endpoint: 10
    Counter: 103
ZigBee Cluster Library Frame, Command: Configure Reporting Response, Seq: 46
    Frame Control Field: Profile-wide (0x18)
        .... ..00 = Frame Type: Profile-wide (0x0)
        .... .0.. = Manufacturer Specific: False
        .... 1... = Direction: Server to Client
        ...1 .... = Disable Default Response: True
    Sequence Number: 46
    Command: Configure Reporting Response (0x07)
    Attribute Status Record
        Status: Failure (0x01)
        Direction: Reported (0x00)
        Attribute: Current Level (0x0000)

At this time I do not see any meaningful information in the log that could help.

For endpoint 13 I'm able to subscribe and receive attribute change reports.

Exactly the same behavior with default unmodified example ble_zigbee_dynamic_color_light_bulb_thingy from Zigbee SDK 4.0.0: I'm able to subscribe to the last endpoint's attributes, but subscribe command fails for first and second endpoint.

Then I change ZB_HA_DIMMABLE_LIGHT_REPORT_ATTR_COUNT definition in my code from 2 to 4 (so each endpoint contains 4 zb_zcl_reporting_info_t structs) I'm able to subscribe to all 8 attributes (2 per endpoint) without any errors, but in this case I receive attribute change reports from the last (13) endpoint only. No reports received from endpoints 10, 11 and 12.

In this case ZB_AF_REGISTER_DEVICE_CTX fills first two (of 4 total in the array) zb_zcl_reporting_info_t structs at each endpoint, but after successful subscription to all 8 attributes (2 per endpoint) I see that for the last endpoint (13) first two zb_zcl_reporting_info_t structs are overwritten by my subscription data, but for first three endpoints (10, 11, 12) first two zb_zcl_reporting_info_t structs are remains unchanged and subscription data is written to the third and fourth position of zb_zcl_reporting_info_t array.

For some reason unknown to me, first two reporting info structures remains unchanged for first three endpoints and at subscription Zigbee stack uses next two available structures - that's why subscription fails when the only two reporting info structs are available.

But for the last endpoint this reporting info structures are overwritten at subscription (as expected by me). That's why the subscription was successful on the last endpoint when the only two reporting info structs are available.

It seems like I'm doing something wrong, but I just can't figure it out. Could you help me with this?

Update 1, sniffer logs for ble_zigbee_dynamic_color_light_bulb_thingy + cli:

Bind:

Request for endpoint 10:

ZigBee Application Support Layer Data, Dst Endpt: 0, Src Endpt: 0
    Frame Control Field: Data (0x40)
    Destination Endpoint: 0
    Bind Request (Cluster ID: 0x0021)
    Profile: ZigBee Device Profile (0x0000)
    Source Endpoint: 0
    Counter: 69
ZigBee Device Profile, Bind Request, Level Control (Cluster ID: 0x0008) Src: NordicSe_63:c6:13:27:6d, Dst: NordicSe_b9:fb:32:38:09
    Sequence Number: 2
    Source: NordicSe_63:c6:13:27:6d (f4:ce:36:63:c6:13:27:6d)
    Source Endpoint: 10
    Cluster: 0x0008 (Level Control)
    Address Mode: Unicast (3)
    Destination: NordicSe_b9:fb:32:38:09 (f4:ce:36:b9:fb:32:38:09)
    Destination Endpoint: 64

Response for endpoint 10:

ZigBee Application Support Layer Data, Dst Endpt: 0, Src Endpt: 0
    Frame Control Field: Data (0x40)
    Destination Endpoint: 0
    Bind Response (Cluster ID: 0x8021)
    Profile: ZigBee Device Profile (0x0000)
    Source Endpoint: 0
    Counter: 54
ZigBee Device Profile, Bind Response, Status: Success
    Sequence Number: 2
    Status: Success (0)

Request for endpoint 11:

ZigBee Application Support Layer Data, Dst Endpt: 0, Src Endpt: 0
    Frame Control Field: Data (0x40)
    Destination Endpoint: 0
    Bind Request (Cluster ID: 0x0021)
    Profile: ZigBee Device Profile (0x0000)
    Source Endpoint: 0
    Counter: 70
ZigBee Device Profile, Bind Request, Level Control (Cluster ID: 0x0008) Src: NordicSe_63:c6:13:27:6d, Dst: NordicSe_b9:fb:32:38:09
    Sequence Number: 3
    Source: NordicSe_63:c6:13:27:6d (f4:ce:36:63:c6:13:27:6d)
    Source Endpoint: 11
    Cluster: 0x0008 (Level Control)
    Address Mode: Unicast (3)
    Destination: NordicSe_b9:fb:32:38:09 (f4:ce:36:b9:fb:32:38:09)
    Destination Endpoint: 64

Response for endpoint 11:

ZigBee Application Support Layer Data, Dst Endpt: 0, Src Endpt: 0
    Frame Control Field: Data (0x40)
    Destination Endpoint: 0
    Bind Response (Cluster ID: 0x8021)
    Profile: ZigBee Device Profile (0x0000)
    Source Endpoint: 0
    Counter: 55
ZigBee Device Profile, Bind Response, Status: Success
    Sequence Number: 3
    Status: Success (0)


Request for endpont 12:

ZigBee Application Support Layer Data, Dst Endpt: 0, Src Endpt: 0
    Frame Control Field: Data (0x40)
    Destination Endpoint: 0
    Bind Request (Cluster ID: 0x0021)
    Profile: ZigBee Device Profile (0x0000)
    Source Endpoint: 0
    Counter: 71
ZigBee Device Profile, Bind Request, Level Control (Cluster ID: 0x0008) Src: NordicSe_63:c6:13:27:6d, Dst: NordicSe_b9:fb:32:38:09
    Sequence Number: 4
    Source: NordicSe_63:c6:13:27:6d (f4:ce:36:63:c6:13:27:6d)
    Source Endpoint: 12
    Cluster: 0x0008 (Level Control)
    Address Mode: Unicast (3)
    Destination: NordicSe_b9:fb:32:38:09 (f4:ce:36:b9:fb:32:38:09)
    Destination Endpoint: 64

Response for endpont 12:

ZigBee Application Support Layer Data, Dst Endpt: 0, Src Endpt: 0
    Frame Control Field: Data (0x40)
    Destination Endpoint: 0
    Bind Response (Cluster ID: 0x8021)
    Profile: ZigBee Device Profile (0x0000)
    Source Endpoint: 0
    Counter: 56
ZigBee Device Profile, Bind Response, Status: Success
    Sequence Number: 4
    Status: Success (0)

Subscribe:

Request for endpoint 10:

ZigBee Application Support Layer Data, Dst Endpt: 10, Src Endpt: 64
    Frame Control Field: Data (0x40)
    Destination Endpoint: 10
    Cluster: Level Control (0x0008)
    Profile: Home Automation (0x0104)
    Source Endpoint: 64
    Counter: 72
ZigBee Cluster Library Frame, Command: Configure Reporting, Seq: 0
    Frame Control Field: Profile-wide (0x00)
    Sequence Number: 0
    Command: Configure Reporting (0x06)
    Reporting Configuration Record
        Direction: Reported (0x00)
        Attribute: Current Level (0x0000)
        Data Type: 8-Bit Unsigned Integer (0x20)
        Current Level: 0
    Minimum Interval: 1
    Maximum Interval: 2

Response for endpoint 10:

ZigBee Application Support Layer Data, Dst Endpt: 64, Src Endpt: 10
    Frame Control Field: Data (0x40)
    Destination Endpoint: 64
    Cluster: Level Control (0x0008)
    Profile: Home Automation (0x0104)
    Source Endpoint: 10
    Counter: 57
ZigBee Cluster Library Frame, Command: Configure Reporting Response, Seq: 0
    Frame Control Field: Profile-wide (0x18)
    Sequence Number: 0
    Command: Configure Reporting Response (0x07)
    Attribute Status Record
        Status: Failure (0x01)
        Direction: Reported (0x00)
        Attribute: Current Level (0x0000)


Request for endpoint 11:

ZigBee Application Support Layer Data, Dst Endpt: 11, Src Endpt: 64
    Frame Control Field: Data (0x40)
    Destination Endpoint: 11
    Cluster: Level Control (0x0008)
    Profile: Home Automation (0x0104)
    Source Endpoint: 64
    Counter: 73
ZigBee Cluster Library Frame, Command: Configure Reporting, Seq: 1
    Frame Control Field: Profile-wide (0x00)
    Sequence Number: 1
    Command: Configure Reporting (0x06)
    Reporting Configuration Record
        Direction: Reported (0x00)
        Attribute: Current Level (0x0000)
        Data Type: 8-Bit Unsigned Integer (0x20)
        Current Level: 0
    Minimum Interval: 1
    Maximum Interval: 2


Response for endpoint 11:

ZigBee Application Support Layer Data, Dst Endpt: 64, Src Endpt: 11
    Frame Control Field: Data (0x40)
    Destination Endpoint: 64
    Cluster: Level Control (0x0008)
    Profile: Home Automation (0x0104)
    Source Endpoint: 11
    Counter: 58
ZigBee Cluster Library Frame, Command: Configure Reporting Response, Seq: 1
    Frame Control Field: Profile-wide (0x18)
    Sequence Number: 1
    Command: Configure Reporting Response (0x07)
    Attribute Status Record
        Status: Failure (0x01)
        Direction: Reported (0x00)
        Attribute: Current Level (0x0000)

Request for endpoint 12:

ZigBee Application Support Layer Data, Dst Endpt: 12, Src Endpt: 64
    Frame Control Field: Data (0x40)
    Destination Endpoint: 12
    Cluster: Level Control (0x0008)
    Profile: Home Automation (0x0104)
    Source Endpoint: 64
    Counter: 74
ZigBee Cluster Library Frame, Command: Configure Reporting, Seq: 2
    Frame Control Field: Profile-wide (0x00)
    Sequence Number: 2
    Command: Configure Reporting (0x06)
    Reporting Configuration Record
        Direction: Reported (0x00)
        Attribute: Current Level (0x0000)
        Data Type: 8-Bit Unsigned Integer (0x20)
        Current Level: 0
    Minimum Interval: 1
    Maximum Interval: 2


Response for endpoint 12:

ZigBee Application Support Layer Data, Dst Endpt: 64, Src Endpt: 12
    Frame Control Field: Data (0x40)
    Destination Endpoint: 64
    Cluster: Level Control (0x0008)
    Profile: Home Automation (0x0104)
    Source Endpoint: 12
    Counter: 59
ZigBee Cluster Library Frame, Command: Configure Reporting Response, Seq: 2
    Frame Control Field: Profile-wide (0x18)
    Sequence Number: 2
    Command: Configure Reporting Response (0x07)
    Status: Success (0x00)

Cli commands used to subscribe:

> bdb channel 16
Done

> bdb role zc
Coordinator set
Done

> bdb start
Started coordinator
Done

> zdo bind on f4ce3663c613276d 10 f4ce36b9fb323809 64 0x0008 0x1ffa
Done

> zdo bind on f4ce3663c613276d 11 f4ce36b9fb323809 64 0x0008 0x1ffa
Done

> zdo bind on f4ce3663c613276d 12 f4ce36b9fb323809 64 0x0008 0x1ffa
Done

> zcl subscribe on f4ce3663c613276d 10 0x0008 0x0104 0x0000 32 1 2
Error: Unable to configure attribute 0 reporting. Status: 1
Error: One or more attributes reporting were not configured successfully

> zcl subscribe on f4ce3663c613276d 11 0x0008 0x0104 0x0000 32 1 2
Error: Unable to configure attribute 0 reporting. Status: 1
Error: One or more attributes reporting were not configured successfully

> zcl subscribe on f4ce3663c613276d 12 0x0008 0x0104 0x0000 32 1 2
Done

Update 2: added raw Wireshark dump:

zigbee_nrf52840_subscription_fail.pcapng

[Edvin]

Hello,

Is it possible to get a sniffer trace from the network while you are doing this? You can use the nRF Sniffer for 802.15.4 to capture this sniffer trace.

[happy_nordic_user]

Hello. I uploaded the pcapng file. Sorry, I'm replying in the top comment because of I do not see reply button in nested comments:

[happy_nordic_user]

Hello.

I'm sorry to distract you. I just wanted to remind you that I still could not solve this issue on my own.

[Edvin]

Hello,

From my internal ticket it looks like our Zigbee team has reproduced this on their master branch. They are investigating further. I will let you know when I know more.

Sorry for the delay.

Best regards,

Edvin

[happy_nordic_user]

Hello, I apologize for bothering, I just wanted to remind that I still can not resolve this itself and I still need a solution.

[Edvin]

Hello,

No worries. When researching this behavior, they discovered a bug. They are working on workarounds and bugfices as we speak, but I don't have anything to show for at this point, unfortunately. I'll keep you updated.

BR,

Edvin

[happy_nordic_user]

Hello. I received an email that this ticket is closed. Why did this happen?

[Edvin]

Hello,

I am sorry. I misread the internal ticket. I thought they wrote that they had forwarded the fix directly to the customer, but it said: fix for the customer to apply directly". I am sorry. Thank you for bringing up the question!

But the good news is that they have found the issue, and a workaround that will work until the bug is fixed in the zboss stack. Check out the attached files:

Fullscreen multi_ep_fixer.c Download

#include "multi_ep_fixer.h"

/* Store previous write attr hook for the Identify cluster. */
zb_zcl_cluster_write_attr_hook_t m_p_identify_write_hook;

static zb_void_t sort_ep_desc_list(zb_uint8_t dst_ep)
{
    zb_uint8_t          dst_ep_desc_index   = 0;
    zb_af_device_ctx_t  *p_dev_ctx          = zb_zcl_get_ctx()->device_ctx;

    /* Find index of dest_ep desc in ep_desc list. */
    while ((p_dev_ctx->ep_desc_list)[dst_ep_desc_index]->ep_id != dst_ep)
    {
        dst_ep_desc_index++;
    }

    /* Check if desc of dest_ep is stored as last element of the ep desc list. */
    if (dst_ep_desc_index < (p_dev_ctx->ep_count - 1))
    {
        /* Store last ep_desc as temp ep_desc  */
        zb_af_endpoint_desc_t *p_temp_ep_desc = (p_dev_ctx->ep_desc_list)[(p_dev_ctx->ep_count - 1)];
        /* Put desired ep_desc as last */
        (p_dev_ctx->ep_desc_list)[(p_dev_ctx->ep_count - 1)] = (p_dev_ctx->ep_desc_list)[dst_ep_desc_index];
        /* Restore temp ep_desc in new place */
        (p_dev_ctx->ep_desc_list)[dst_ep_desc_index] = p_temp_ep_desc;
    }
}

static zb_uint8_t multi_ep_handler(zb_bufid_t bufid)
{
    zb_zcl_parsed_hdr_t *cmd_info = ZB_BUF_GET_PARAM(bufid, zb_zcl_parsed_hdr_t);
    /* Nbr of endpoint to which frame is to be delivered. */
    zb_uint8_t          dst_ep    = cmd_info->addr_data.common_data.dst_endpoint;

    if ((cmd_info->cmd_id == ZB_ZCL_CMD_CONFIG_REPORT)
        || (cmd_info->cmd_id == ZB_ZCL_CMD_REPORT_ATTRIB)
        || (cmd_info->cmd_id == ZB_ZCL_CMD_READ_REPORT_CFG))
    {
        sort_ep_desc_list(dst_ep);
    }
    return ZB_FALSE;
}

static void multi_ep_identify_cluster_write_attr_hook(zb_uint8_t endpoint, zb_uint16_t attr_id, zb_uint8_t *new_value)
{
    sort_ep_desc_list(endpoint);
    m_p_identify_write_hook(endpoint, attr_id, new_value);
}

static void multi_ep_generic_cluster_write_attr_hook(zb_uint8_t endpoint, zb_uint16_t attr_id, zb_uint8_t *new_value)
{
    ZVUNUSED(attr_id);
    ZVUNUSED(new_value);

    sort_ep_desc_list(endpoint);
}

/* Function which initializes fixer methods and hooks. */
void init_multi_ep_fixers(void)
{
    zb_zcl_globals_t * p_zcl_ctx = zb_zcl_get_ctx();
    zb_uint8_t endpoint = 0;
    zb_uint8_t index = 0;

    ZB_ASSERT(p_zcl_ctx);

    /* Find Identify attr hook, store previous value if exist and assing new identify cluster wrtie attr hook. */
    while (index < p_zcl_ctx->zcl_handlers_cnt)
    {
        if ((p_zcl_ctx->zcl_handlers[index].cluster_id == ZB_ZCL_CLUSTER_ID_IDENTIFY)
            && (p_zcl_ctx->zcl_handlers[index].cluster_write_attr_hook != NULL))
        {
            m_p_identify_write_hook = p_zcl_ctx->zcl_handlers[index].cluster_write_attr_hook;
            p_zcl_ctx->zcl_handlers[index].cluster_write_attr_hook = multi_ep_identify_cluster_write_attr_hook;
        }
        /* For clusters with ID >= 0x03, if there is no attr hook assigned, assign generic one. If already exists, assert. */
        else if (p_zcl_ctx->zcl_handlers[index].cluster_id >= ZB_ZCL_CLUSTER_ID_IDENTIFY)
        {
            ZB_ASSERT(p_zcl_ctx->zcl_handlers[index].cluster_write_attr_hook == NULL);

            p_zcl_ctx->zcl_handlers[index].cluster_write_attr_hook = multi_ep_generic_cluster_write_attr_hook;
        }
        index++;
    }

    /* For each endpoint register device callback. Make sure that there is no endpoint handler set already. */
    for (index = 0; index < p_zcl_ctx->device_ctx->ep_count; index++)
    {
        endpoint = p_zcl_ctx->device_ctx->ep_desc_list[index]->ep_id;

        ZB_ASSERT(zb_af_get_endpoint_desc((endpoint))->device_handler == NULL);
        ZB_AF_SET_ENDPOINT_HANDLER(endpoint, multi_ep_handler);
    }
}

multi_ep_fixer.h

Can you try to call init_multi_ep_fixers() in your initialization steps, just after the CTX and ZCL device callbacks are registered. They also said that:

"In case in which application uses write attribute hooks or endpoints handlers, small modification in workaround is required. If this is true for customer's application I will explain what should be changed."

So let me know if that is required.

Best regards,

Edvin

[happy_nordic_user]

Hello.

It seems that this fix works for me, but now I have problems with saving the state of subscriptions and bindings in non-volatile memory. After a reboot, the firmware forgets about all subscriptions. I think that this new problem is not related to the current one, as I collect the necessary information - I will try to publish a separate ticket in the support service.

[happy_nordic_user]

Ticket regarding subscriptions and nvram: devzone.nordicsemi.com/.../zigbee-attribute-subscription-to-multi-endpoint-device-breaks-nvram