NRF54L15DK vs NRF54L15PDK: Different behaviour of OpenThread exposed to WiFi Interference

[begi_]

We want to measure latency and PER between two OpenThread Nodes exposed to WiFi interference. 
Multiple Raspberry PIs, which are configured as AP and/or client, simulate flooded WiFi Traffic with iperf3.

Two NRF54L15 are used to build an OpenThread link (CONFIG_OPENTHREAD=y, CONFIG_OPENTHREAD_NORDIC_LIBRARY_MASTER=n) in the same frequency band as the WiFi.
A simple ID is exchanged between the nodes (otIcmp6SendEchoRequest) to calculate the roundtrip time and the PER. A delay of 250ms between two messages is added. 
The two nodes are synced with CONFIG_OPENTHREAD_TIME_SYNC.
The Rapsberry PIs are physically arranged as a rectangle, with the two NRF54L15 in the middle.

Measurements with the NRF54L15PDK yield the expected results (More interference results in higher Roundtriptime/PER).
However, if we use the same code/same build config with the NRF54L15DK, the ping sender crashes (Secure Fault) under heavy WiFi Interference.
Resistance to interference also appears to be lower, as indicated by a much higher PER for the DK compared to the PDK.

We checked with 4 different NRF54L15DK and tried different message sizes (64/127 bytes) for the ping.
Nothing is listed in the errata for the two different DKs and the specific SoC Version respectively.

How can the same code, yield two completely different results?
Are there some changes, which are not listed in the errata or in other words: What could be the issue for this behaviour?

Any help is appreciated.
Kind regards


Used Versions:
nRF Connect SDK v3.1.1 (Zephyr 4.1.99, west 1.4.0), Build config: nrf54l15dk/nrf54l15/cpuapp/ns
OpenThread:       v 1.4
NRF54L15DK:     v1.0.0 (SoC Ref 2)
NRF54L15PDK:   v0.8.1 (SoC Ref 1)

[Kenneth]

Hello,

As far as I know, there is no board files for nRF54L15PDK in v3.1.1, so I am not entirely sure how you were able to test with the PDK? Can you elaborate how you solved this?

In general though the old nRF54L15PDK did have a workaround that prevented low power mode, this was set by this kconfig option:
CONFIG_SOC_NRF_FORCE_CONSTLAT=y 
(see: https://github.com/nrfconnect/sdk-zephyr/blob/v3.6.99-ncs2/boards/nordic/nrf54l15pdk/nrf54l15pdk_nrf54l15_cpuapp_defconfig#L28)

It could be interesting to see if this kconfig option have any effect on the problem you see.

Also, do you see the problem if you don't build for nrf54l15dk/nrf54l15/cpuapp/ns, but build for nrf54l15dk/nrf54l15/cpuapp/

Kenneth

[begi_]

Thank you for the fast reply.

Kenneth said:

As far as I know, there is no board files for nRF54L15PDK in v3.1.1, so I am not entirely sure how you were able to test with the PDK? Can you elaborate how you solved this?

As there were no explicit board files for the PDK, we assumed one can use the same as the DK. We used nrf54l15dk/nrf54l15/cpuapp/ns for both PDK and DK. Altough secure/nonsecure doesn't make a difference, see below

Kenneth said:

It could be interesting to see if this kconfig option have any effect on the problem you see.

The issue still persists and the DK crashes regardless of CONFIG_SOC_NRF_FORCE_CONSTLAT with a Secure Fault if heavy interference is present.

Kenneth said:

Also, do you see the problem if you don't build for nrf54l15dk/nrf54l15/cpuapp/ns, but build for nrf54l15dk/nrf54l15/cpuapp/

The problem persists if I use the secure build (I also tested both options for CONFIG_SOC_NRF_FORCE_CONSTLAT)

 

begi_ said:

Resistance to interference also appears to be lower, as indicated by a much higher PER for the DK compared to the PDK.

Here is a comparison between the PDK and DK:
~470 Pings are sent with different Levels of WiFi Interference and the round trip time and PER are calculated. 
As the observation period is very short, no "statistics" can be established, but it does show a trend of a worse performing PK.


begi

[Marte Myrvold]

Hi,

Is the code based on one of the SDK samples, and if so, which one?

Do you see similar behavior with the IEEE 802.15.4 PHY test tool or Radio test (short-range) samples?

Can you debug the secure fault to gather more information about it? We have a lesson on debugging in our nRF Connect SDK Intermediate course on DevAcademy: Lesson 2 – Debugging and troubleshooting.

Best regards,
Marte

[begi_]

Marte Myrvold said:

Is the code based on one of the SDK samples, and if so, which one?

The code is based on the ot-cli example, extended with time sync and the pings. I attached the used code+prj.conf.

Marte Myrvold said:

Can you debug the secure fault to gather more information about it? We have a lesson on debugging in our nRF Connect SDK Intermediate course on DevAcademy: Lesson 2 – Debugging and troubleshooting.

Addr2Line reports:
addr2line -p -e build/thila2_test_node/zephyr/zephyr.elf 0x000955ae
/home/userxy/ncs/v3.1.1/modules/lib/openthread/src/core/common/message.hpp:461

message.hpp Line 461:

/**
* Returns the number of bytes in the message.
*
* @returns The number of bytes in the message.
*
*/
uint16_t GetLength(void) const { return GetMetadata().mLength; }

Stacktrace shows that the call comes from main -> otMessageAppend -> Messages::AppendBytes.
where it crashes on the first call of GetLength() on Line 406.
message.cpp:

Error Message::AppendBytes(const void *aBuf, uint16_t aLength)
{
    Error    error     = kErrorNone;
    uint16_t oldLength = GetLength();
 
    SuccessOrExit(error = SetLength(GetLength() + aLength));
    WriteBytes(oldLength, aBuf, aLength);
 
exit:
    return error;
}

In the disassembly (see red arrow) r0 (which is 0) is loaded with offset 34 (0x22) and therefore tries to read from 0x22 which lies in the secure partition:

Marte Myrvold said:

Do you see similar behavior with the IEEE 802.15.4 PHY test tool or Radio test (short-range) samples?

I will get back to this once I figure out a meaningful test 

begi

Fullscreen 6874.main.c Download

// 
#include <SEGGER_RTT.h>
#include <stdio.h>

#include <zephyr/kernel.h>
#include <zephyr/logging/log.h>
#include <zephyr/net/openthread.h>

#include <openthread.h>
#include <openthread/thread.h>
#include <openthread/network_time.h>
#include <openthread/icmp6.h>
#include <openthread/thread_ftd.h>

LOG_MODULE_REGISTER(lat_test, CONFIG_OT_COMMAND_LINE_INTERFACE_LOG_LEVEL);

#define TIME_SYNC_PERIOD_SECONDS    (10)
// NOTE: this can be found out by running "ot ipaddr" on the destination
#define PING_DESTINATION_IP         "fdde:ad00:beef:0:0:ff:fe00:e800"
// 1: if this node is the sender, 0: for destination
#define IS_PING_SENDER              (1)
// Sizeof of icmp message including the header
#define MESSAGE_SIZE                (127)
// Sleep betweeen sending icmp messages
#define SLEEP_DURATION              K_MSEC(250)

typedef enum {
    MESSAGE_SEND,
    MESSAGE_RECEIVE,
} Message_Direction;

typedef struct {
    otInstance*     instance;
    struct k_sem    is_network_synced;
} Context;

static
void thread_sync_event(void* user_data) {
    Context* context = user_data;

    uint64_t network_time = 0;
    otNetworkTimeStatus time_status = otNetworkTimeGet(context->instance, &network_time);
    if (time_status == OT_NETWORK_TIME_SYNCHRONIZED) {
        k_sem_give(&context->is_network_synced);
    }
}

static
void thread_state_changed(otChangedFlags flags, void* user_data) {
    LOG_INF("State changed");
}

static 
struct openthread_state_changed_callback ot_state_chaged_cb = {
    .otCallback = thread_state_changed,
};

static
void log_icmp(Context* context, const otIcmp6Header* icmp_header, Message_Direction direction) {
    char* message_type = "unknown";
    switch (icmp_header->mType) {
        case OT_ICMP6_TYPE_ECHO_REQUEST:
            message_type = "icmp_echo";
        break;
        case OT_ICMP6_TYPE_ECHO_REPLY:
            message_type = "icmp_reply";
        break;
    }

    char* message_direction = "sent";
    if (direction == MESSAGE_RECEIVE) {
        message_direction = "received";
    }

    uint64_t message_time = 0;
    otNetworkTimeStatus time_status = otNetworkTimeGet(context->instance, &message_time);
    if (time_status != OT_NETWORK_TIME_SYNCHRONIZED) {
        SEGGER_RTT_WriteString(0, "Tried to log but network is not synchronized");
    }

    uint16_t identifier = BSWAP_16(icmp_header->mData.m16[0]);

    char print_buffer[128] = {0};
    snprintf(print_buffer, sizeof(print_buffer), "{%u;%s;%s;%u}\n", (uint32_t)message_time, message_direction, message_type, identifier);
    SEGGER_RTT_WriteString(0, print_buffer);
}

static
void icmp_message_handler(void* user_data, otMessage* message, const otMessageInfo* message_info, const otIcmp6Header* icmp_header) {
    Context* context = user_data;
    log_icmp(context, icmp_header, MESSAGE_RECEIVE);
}

static Context thread_context;

int main(void) {
    k_sem_init(&thread_context.is_network_synced, 0, 1);

#if IS_PING_SENDER
    LOG_INF("Sender mode");
#else
    LOG_INF("Receiver mode");
#endif

    thread_context.instance = openthread_get_default_instance();

    openthread_state_changed_callback_register(&ot_state_chaged_cb);

    static otIcmp6Handler icmp_handler = {
        .mContext = &thread_context,
        .mReceiveCallback = icmp_message_handler,
    };
    otError icmp_handler_status = otIcmp6RegisterHandler(thread_context.instance, &icmp_handler);
    if (icmp_handler_status != OT_ERROR_NONE) {
        LOG_ERR("Could not reguster icmp handler");
    }

    otError period_status = otNetworkTimeSetSyncPeriod(thread_context.instance, TIME_SYNC_PERIOD_SECONDS);
    if (period_status != OT_ERROR_NONE) {
        LOG_ERR("Could not set sync period");
        return 0;
    }

    otNetworkTimeSyncSetCallback(thread_context.instance, thread_sync_event, &thread_context);

    openthread_run();

#if IS_PING_SENDER
    uint32_t icmp_identifier = 1;

    LOG_INF("Waiting for network to be synced...");
    k_sem_take(&thread_context.is_network_synced, K_FOREVER);
    LOG_INF("Network synced");

    otIp6Address ping_destination = {0};
    if (otIp6AddressFromString(PING_DESTINATION_IP, &ping_destination) != OT_ERROR_NONE) {
        LOG_ERR("Could not parse ip addr");
    }

    while (true) {
        otMessage* message = otIp6NewMessage(thread_context.instance, NULL);
        const otMessageInfo message_info = {
            .mPeerAddr = ping_destination,
        };

        otIcmp6Header header = {
            .mType = OT_ICMP6_TYPE_ECHO_REQUEST,
            // We alrady set this with the otIcmp6SendEchoRequest api but because logging looks at this value
            // before we call the api we fill it out anyway
            .mData.m16[0] = BSWAP_16(icmp_identifier),
        };
        otMessageAppend(message, &header, sizeof(header));

        static uint8_t fill_payload[MESSAGE_SIZE - sizeof(header)] = {0};
        otMessageAppend(message, fill_payload, sizeof(fill_payload));

        log_icmp(&thread_context, &header, MESSAGE_SEND);
        otIcmp6SendEchoRequest(thread_context.instance, message, &message_info, icmp_identifier);

        k_sleep(SLEEP_DURATION);

        icmp_identifier += 1;   
    }
#endif

    return 0;
}

43847.prj.conf

[begi_]

begi_ said:

Marte Myrvold said:

Do you see similar behavior with the IEEE 802.15.4 PHY test tool or Radio test (short-range) samples?

Marte Myrvold said:

Do you see similar behavior with the IEEE 802.15.4 PHY test tool or Radio test (short-range) samples?

I used the 802.15.4 PHY test tool and sent 500 rpings with different number of WiFi Interference Streams.
The PER seems to be roughly the same between the DK and PDK:

Although I'm not entirely sure how this test compares to the OpenThread measurement, as I am not quite aware of how the PHY/MAC differs (eg, number of retries)

We built the provided code for a nrf52840dk as well and repeated the test with different number of WiFi interference streams once more. This time we used a message length of 64. 470 Messages are sent, although the nrf54l15DK crashed after 337 messages:

The nrf52840 (green) performs as expected and similar to the nrf54l15pdk (blue). The nrf54l15dk (red) performs worse.

begi

[Marte Myrvold]

Hi,

begi_ said:

Messages are sent, although the nrf54l15DK crashed after 337 messages:

Is this the same crash as you mentioned earlier in the ticket? I.e. this:

However, if we use the same code/same build config with the NRF54L15DK, the ping sender crashes (Secure Fault) under heavy WiFi Interference.

Does the nRF54L15 DK crash every time, and is it both the sender and receiver that crash, or only the sender?

Do you see any difference when testing the 802.15.4 PHY test tool with the nRF52840?

Best regards,
Marte