nrf_wifi fails : memory allocation failure at boot and crash at config

[BrianW]

Using an nrf5340 with nrf7002 wifi, connected via SPI4. Using a shared (chip) antenna, and the COEX signals (pretty much exactly as per the dev board schema).

DTS snippets:

    // Wifi nrf7002 : SPI4 using SCK P0.08, MOSI P0.09, MISO P0.10, CS P0.11

    // BUCKEN P0.12, IRQ P0.23, COEX_GRANT P0.24, COEX_REQ P0.28, STATUS0 P0.30, STATUS1 P0.29, NRF7002_IOVDD_EN ioexp0/GPB0

    nrf_radio_coex: nrf7001-coex {

        status = "okay";

        compatible = "nordic,nrf700x-coex";

        req-gpios = <&gpio0 28 GPIO_ACTIVE_HIGH>;

        status0-gpios = <&gpio0 30 GPIO_ACTIVE_HIGH>;

        grant-gpios = <&gpio0 24 (GPIO_PULL_DOWN | GPIO_ACTIVE_LOW)>;

        swctrl1-gpios = <&gpio0 29 GPIO_ACTIVE_HIGH>;

        // btrf-switch-gpios = <&gpio1 10 GPIO_ACTIVE_HIGH>; No, controlled by nrf7002 directly???

    };

    nordic_wlan0: nordic_wlan0 {

        compatible = "nordic,wlan0";

        status = "okay";

    };

&spi4 {                     // Wifi nrf7002

    status = "okay";

    compatible = "nordic,nrf-spim";

    pinctrl-0 = <&spi4_default>;

    pinctrl-1 = <&spi4_sleep>;

    cs-gpios = <&gpio0 11 GPIO_ACTIVE_LOW>; /* P0.11 */

    pinctrl-names = "default", "sleep";

    nrf700x: nrf7002@0 {

        status = "okay";

        compatible = "nordic,nrf700x-spi";

        reg = <0>;

        spi-max-frequency = <33000000>;

//      sck-frequency = <24000000>;

        /* Wi-Fi Pins used */

        iovdd-ctrl-gpios = <&ioexp0 8 GPIO_ACTIVE_HIGH>;            // GPB00

        bucken-gpios = <&gpio0 12 GPIO_ACTIVE_HIGH>;

        host-irq-gpios = <&gpio0 23 GPIO_ACTIVE_HIGH>;

    };

};

I have configured prj.conf to have the wifi STA 'iot' operation, as per the docs:

# WIFI config
CONFIG_WIFI=y
CONFIG_WIFI_NRF700X=y

# WPA supplicant
CONFIG_WPA_SUPP=y
CONFIG_NRF700X_AP_MODE=n
CONFIG_NRF700X_P2P_MODE=n

# Networking layers
CONFIG_NETWORKING=y
CONFIG_NET_SOCKETS=y
CONFIG_NET_LOG=y
CONFIG_NET_IPV4=y
CONFIG_NET_UDP=y
CONFIG_NET_TCP=y
CONFIG_NET_DHCPV4=y

CONFIG_NET_IF_UNICAST_IPV4_ADDR_COUNT=2
CONFIG_NET_MAX_CONTEXTS=1
CONFIG_NET_CONTEXT_SYNC_RECV=y

CONFIG_NET_L2_ETHERNET=y

CONFIG_NET_CONFIG_SETTINGS=y
CONFIG_NET_CONFIG_INIT_TIMEOUT=0

CONFIG_NET_SOCKETS_POLL_MAX=1

# Memory usage
CONFIG_NET_TX_STACK_SIZE=4096
CONFIG_NET_RX_STACK_SIZE=4096

# Optimized networking settings for memory usage (STA MODE IOT DEVICES profile)
# https://docs.nordicsemi.com/bundle/ncs-latest/page/nrf/protocols/wifi/stack_configuration.html
CONFIG_NRF700X_MAX_TX_TOKENS=5
CONFIG_NRF700X_MAX_TX_AGGREGATION=1
CONFIG_NRF700X_RX_NUM_BUFS=6

CONFIG_NET_TC_TX_COUNT=1
CONFIG_NET_PKT_RX_COUNT=6
CONFIG_NET_PKT_TX_COUNT=6

CONFIG_NET_BUF_RX_COUNT=6
CONFIG_NET_BUF_TX_COUNT=12
# can take BUF_DATAT_SIZE down to 500
CONFIG_NET_BUF_DATA_SIZE=800

However, at boot time I see these logs:

[00:00:00.015,686] <inf> wifi_nrf_bus: SPIM spi@a000: freq = 33 MHz
[00:00:00.022,369] <inf> wifi_nrf_bus: SPIM spi@a000: latency = 1
[00:00:01.074,981] <err> wifi_nrf: hal_fw_patch_load: Failed to allocate memory for patch UMAC-bin: chunk 1/6, size: 8192
[00:00:01.086,639] <err> wifi_nrf: nrf_wifi_fmac_fw_load: UMAC patch load failed

[00:00:01.094,818] <err> wifi_nrf: nrf_wifi_fw_load: nrf_wifi_fmac_fw_load failed
[00:00:01.103,027] <err> wifi_nrf: nrf_wifi_fmac_dev_add_zep: nrf_wifi_fw_load failed
[00:00:01.113,891] <err> wifi_nrf: nrf_wifi_if_start_zep: nrf_wifi_fmac_dev_add_zep failed

Note I also have configured malloc to use the system heap, and for it to occupy all remaining RAM:

CONFIG_COMMON_LIBC_MALLOC=y

CONFIG_COMMON_LIBC_MALLOC_ARENA_SIZE=-1

The heap size at boot time is 233576 bytes so no reason for a malloc to fail?

Then when I attempt to initialise the wifi I get this crash

[00:00:13.322,784] <err> os: ***** USAGE FAULT *****
[00:00:13.328,399] <err> os: Unaligned memory access
[00:00:13.334,197] <err> os: r0/a1: 0xa07eeaff r1/a2: 0x00000068 r2/a3: 0x0000005c
[00:00:13.342,834] <err> os: r3/a4: 0x20047130 r12/ip: 0x00000000 r14/lr: 0x00063dc9
[00:00:13.351,440] <err> os: xpsr: 0x61000000
[00:00:13.356,628] <err> os: Faulting instruction address (r15/pc): 0x0009bafe
[00:00:13.364,501] <err> os: >>> ZEPHYR FATAL ERROR 31: Unknown error on CPU 0
[00:00:13.372,375] <err> os: Current thread: 0x20007340 (sysworkq)
[00:00:13.379,180] <err> os: Halting system

in  nrf_wifi_osal_mem_zalloc() according to zephyr.map.

This is called somewherr from nrf_wifi_coex_config_non_pta():

    /* Configure Coexistence Hardware */

    log_info("netwifi: Configuring non-PTA registers. seperate antennas:%s  is_sr_protocol_ble: %s",separate_antennas?"YES":"NO", is_sr_protocol_ble?"YES":"NO");

    ret = nrf_wifi_coex_config_non_pta(separate_antennas, is_sr_protocol_ble);

    if (ret != 0) {

        log_warn("netwifi: Configuring non-PTA registers of CoexHardware FAIL");

        return false;

    }

Note the wifi sample projects (wifi_radio_test, wifi_shell)all run just FINE (with no error logs at boot) so the hardware is apparently good, but I haven't managed to get from them to my full app without this problem occuring...

Any ideas?

[BrianW]

Both the sta sample and the wifi-shell samples work fine.

I took my prj.conf keys from these samples anyway...

However, I note that internally the wpa_suppliant code uses sockets and poll(), so the fact that I trimmed 

CONFIG_NET_MAX_CONTEXTS=5

CONFIG_NET_SOCKETS_POLL_MAX=10

down to just 1 each (as that is all my app code will need) might be the cause of some of the pain... although I can't see the impact on the wifi_nm and the ops callbacks...

 

Do you see any other bad things in my prj.conf? 

[BrianW]

Well, putting back the number of contexts/polls didn't help.

It _seems_ like its related to having the libc malloc heap size as -1 (use rest of RAM). I have the k_malloc heap set to 50000 (which should be fine), but if I set 

CONFIG_COMMON_LIBC_MALLOC_ARENA_SIZE=-1

CONFIG_HEAP_MEM_POOL_SIZE=50000

than I immediately get this error on boot, and when wpa_suppliant fails....

[00:00:01.401,947] <err> net_sock: invalid access on sock 1 by thread 0x20007580

If I change to (no change to anything else)

CONFIG_COMMON_LIBC_MALLOC_ARENA_SIZE=150000

then no errors at boot... why would allocating the rest of the RAM to the libc malloc break the networking code????????

[BrianW]

To be sure about this, I made the following changes to prj.conf for the wifi 'sta' sample:

# use malloc that maps to k_malloc(), and allocate all remaing RAM to heap (-1)

CONFIG_COMMON_LIBC_MALLOC=y

CONFIG_COMMON_LIBC_MALLOC_ARENA_SIZE=-1

CONFIG_RESET_ON_FATAL_ERROR=n

Boom! as well as the same socket error log, it then core dumps!

[00:00:00.395,874] <inf> wifi_nrf_bus: SPIM spi@a000: freq = 33 MHz
[00:00:00.395,904] <inf> wifi_nrf_bus: SPIM spi@a000: latency = 1
*** Booting nRF Connect SDK 3758bcbfa5cd ***
[00:00:01.642,303] <inf> net_config: Initializing network
[00:00:01.642,303] <inf> net_config: Waiting interface 1 (0x20001588) to be up...
[00:00:01.642,425] <inf> net_config: IPv4 address: 192.168.1.99
[00:00:01.642,486] <inf> net_config: Running dhcpv4 client...
[00:00:01.642,761] <inf> sta: Starting cc1medv1_nrf5340_cpuapp with CPU frequency: 64 MHz
[00:00:01.643,463] <err> net_sock: invalid access on sock 1 by thread 0x200025e8
[00:00:01.643,524] <err> os: ***** MPU FAULT *****
[00:00:01.643,554] <err> os:   Data Access Violation
[00:00:01.643,554] <err> os:   MMFAR Address: 0xe4a
[00:00:01.643,585] <err> os: r0/a1:  0x20038128  r1/a2:  0x00000e48  r2/a3:  0x00000000
[00:00:01.643,585] <err> os: r3/a4:  0x0000ea84 r12/ip:  0x20001880 r14/lr:  0x0004d74b
[00:00:01.643,585] <err> os:  xpsr:  0x81000000
[00:00:01.643,585] <err> os: Faulting instruction address (r15/pc): 0x0004d460
[00:00:01.643,615] <err> os: >>> ZEPHYR FATAL ERROR 19: Unknown error on CPU 0
[00:00:01.643,646] <err> os: Current thread: 0x200025e8 (unknown)
[00:00:01.751,708] <err> coredump: #CD:BEGIN#
[00:00:01.756,744] <err> coredump: #CD:5a4501000300050013000000
[00:00:01.763,305] <err> coredump: #CD:4102004400
[00:00:01.768,676] <err> coredump: #CD:28810320480e00000000000084ea0000801800204bd7040060d4040000000081
[00:00:01.778,686] <err> coredump: #CD:60cf002000000000000000000000000000000000000000000000000000000000
[00:00:01.788,665] <err> coredump: #CD:00000000
[00:00:01.793,853] <err> coredump: #CD:4d0100e825002080260020
[00:00:01.800,262] <err> coredump: #CD:38270020684e0020000000000080000000000000000000000000000000000000
[00:00:01.810,302] <err> coredump: #CD:0000000000000000000000000000000000000000e02000203c37002000000000
[00:00:01.820,312] <err> coredump: #CD:1c14002088150020000000000000000038cf0020000000004026002040260020
[00:00:01.830,322] <err> coredump: #CD:000000000900000010b000200020000000000000841200200000000000000000
[00:00:01.840,332] <err> coredump: #CD:00000000000000000000000000000000f5ffffff00000000
[00:00:01.848,968] <err> coredump: #CD:4d010010b0002010d00020
[00:00:01.855,377] <err> coredump: #CD:f0f0f0f0aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
[00:00:01.865,386] <err> coredump: #CD:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
[00:00:01.875,396] <err> coredump: #CD:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
[00:00:01.885,406] <err> coredump: #CD:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
[00:00:01.895,446] <err> coredump: #CD:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
[00:00:01.905,456] <err> coredump: #CD:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
[00:00:01.915,466] <err> coredump: #CD:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
[00:00:01.925,476] <err> coredump: #CD:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
[00:00:01.935,485] <err> coredump: #CD:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
[00:00:01.945,495] <err> coredump: #CD:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
[00:00:01.955,535] <err> coredump: #CD:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
[00:00:01.965,545] <err> coredump: #CD:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
[00:00:01.975,555] <err> coredump: #CD:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
[00:00:01.985,565] <err> coredump: #CD:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
[00:00:01.995,574] <err> coredump: #CD:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
[00:00:02.005,584] <err> coredump: #CD:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
[00:00:02.015,594] <err> coredump: #CD:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
[00:00:02.025,604] <err> coredump: #CD:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
[00:00:02.035,644] <err> coredump: #CD:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
[00:00:02.045,654] <err> coredump: #CD:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
[00:00:02.055,664] <err> coredump: #CD:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
[00:00:02.065,673] <err> coredump: #CD:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
[00:00:02.075,683] <err> coredump: #CD:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
[00:00:02.085,723] <err> coredump: #CD:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
[00:00:02.095,733] <err> coredump: #CD:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
[00:00:02.105,743] <err> coredump: #CD:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
[00:00:02.115,753] <err> coredump: #CD:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
[00:00:02.125,762] <err> coredump: #CD:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
[00:00:02.135,772] <err> coredump: #CD:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
[00:00:02.145,782] <err> coredump: #CD:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
[00:00:02.155,792] <err> coredump: #CD:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
[00:00:02.165,832] <err> coredump: #CD:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
[00:00:02.175,842] <err> coredump: #CD:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
[00:00:02.185,852] <err> coredump: #CD:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
[00:00:02.195,861] <err> coredump: #CD:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
[00:00:02.205,871] <err> coredump: #CD:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
[00:00:02.215,881] <err> coredump: #CD:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
[00:00:02.225,891] <err> coredump: #CD:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
[00:00:02.235,931] <err> coredump: #CD:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
[00:00:02.245,941] <err> coredump: #CD:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
[00:00:02.255,950] <err> coredump: #CD:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
[00:00:02.265,960] <err> coredump: #CD:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
[00:00:02.275,970] <err> coredump: #CD:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
[00:00:02.285,980] <err> coredump: #CD:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
[00:00:02.295,989] <err> coredump: #CD:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
[00:00:02.305,999] <err> coredump: #CD:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
[00:00:02.316,009] <err> coredump: #CD:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
[00:00:02.326,049] <err> coredump: #CD:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
[00:00:02.336,059] <err> coredump: #CD:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
[00:00:02.346,069] <err> coredump: #CD:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
[00:00:02.356,079] <err> coredump: #CD:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
[00:00:02.366,088] <err> coredump: #CD:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
[00:00:02.376,098] <err> coredump: #CD:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
[00:00:02.386,108] <err> coredump: #CD:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
[00:00:02.396,118] <err> coredump: #CD:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
[00:00:02.406,127] <err> coredump: #CD:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
[00:00:02.416,137] <err> coredump: #CD:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
[00:00:02.426,147] <err> coredump: #CD:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
[00:00:02.436,157] <err> coredump: #CD:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
[00:00:02.446,197] <err> coredump: #CD:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
[00:00:02.456,207] <err> coredump: #CD:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
[00:00:02.466,217] <err> coredump: #CD:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
[00:00:02.476,226] <err> coredump: #CD:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
[00:00:02.486,236] <err> coredump: #CD:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
[00:00:02.496,246] <err> coredump: #CD:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
[00:00:02.506,256] <err> coredump: #CD:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
[00:00:02.516,265] <err> coredump: #CD:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
[00:00:02.526,275] <err> coredump: #CD:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
[00:00:02.536,285] <err> coredump: #CD:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
[00:00:02.546,325] <err> coredump: #CD:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
[00:00:02.556,335] <err> coredump: #CD:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
[00:00:02.566,345] <err> coredump: #CD:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
[00:00:02.576,354] <err> coredump: #CD:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
[00:00:02.586,364] <err> coredump: #CD:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
[00:00:02.596,374] <err> coredump: #CD:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
[00:00:02.606,384] <err> coredump: #CD:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
[00:00:02.616,394] <err> coredump: #CD:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
[00:00:02.626,434] <err> coredump: #CD:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
[00:00:02.636,444] <err> coredump: #CD:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
[00:00:02.646,453] <err> coredump: #CD:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
[00:00:02.656,463] <err> coredump: #CD:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
[00:00:02.666,473] <err> coredump: #CD:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
[00:00:02.676,483] <err> coredump: #CD:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
[00:00:02.686,523] <err> coredump: #CD:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
[00:00:02.696,533] <err> coredump: #CD:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
[00:00:02.706,542] <err> coredump: #CD:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
[00:00:02.716,552] <err> coredump: #CD:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
[00:00:02.726,562] <err> coredump: #CD:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
[00:00:02.736,572] <err> coredump: #CD:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
[00:00:02.746,582] <err> coredump: #CD:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
[00:00:02.756,591] <err> coredump: #CD:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
[00:00:02.766,632] <err> coredump: #CD:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
[00:00:02.776,641] <err> coredump: #CD:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
[00:00:02.786,651] <err> coredump: #CD:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
[00:00:02.796,661] <err> coredump: #CD:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
[00:00:02.806,671] <err> coredump: #CD:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
[00:00:02.816,680] <err> coredump: #CD:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
[00:00:02.826,690] <err> coredump: #CD:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
[00:00:02.836,730] <err> coredump: #CD:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
[00:00:02.846,740] <err> coredump: #CD:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
[00:00:02.856,750] <err> coredump: #CD:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
[00:00:02.866,760] <err> coredump: #CD:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
[00:00:02.876,800] <err> coredump: #CD:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
[00:00:02.886,810] <err> coredump: #CD:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
[00:00:02.896,820] <err> coredump: #CD:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
[00:00:02.906,829] <err> coredump: #CD:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
[00:00:02.916,870] <err> coredump: #CD:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
[00:00:02.926,879] <err> coredump: #CD:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
[00:00:02.936,889] <err> coredump: #CD:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
[00:00:02.946,899] <err> coredump: #CD:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
[00:00:02.956,909] <err> coredump: #CD:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
[00:00:02.966,918] <err> coredump: #CD:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
[00:00:02.976,928] <err> coredump: #CD:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
[00:00:02.986,938] <err> coredump: #CD:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
[00:00:02.996,978] <err> coredump: #CD:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
[00:00:03.006,988] <err> coredump: #CD:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
[00:00:03.016,998] <err> coredump: #CD:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
[00:00:03.027,008] <err> coredump: #CD:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
[00:00:03.037,017] <err> coredump: #CD:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
[00:00:03.047,027] <err> coredump: #CD:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
[00:00:03.057,037] <err> coredump: #CD:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
[00:00:03.067,047] <err> coredump: #CD:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
[00:00:03.077,056] <err> coredump: #CD:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
[00:00:03.087,066] <err> coredump: #CD:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
[00:00:03.097,076] <err> coredump: #CD:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
[00:00:03.107,116] <err> coredump: #CD:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
[00:00:03.117,126] <err> coredump: #CD:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
[00:00:03.127,136] <err> coredump: #CD:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
[00:00:03.137,145] <err> coredump: #CD:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
[00:00:03.147,155] <err> coredump: #CD:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
[00:00:03.157,196] <err> coredump: #CD:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
[00:00:03.167,205] <err> coredump: #CD:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
[00:00:03.177,215] <err> coredump: #CD:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
[00:00:03.187,225] <err> coredump: #CD:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
[00:00:03.197,235] <err> coredump: #CD:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
[00:00:03.207,244] <err> coredump: #CD:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
[00:00:03.217,285] <err> coredump: #CD:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
[00:00:03.227,294] <err> coredump: #CD:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
[00:00:03.237,304] <err> coredump: #CD:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
[00:00:03.247,344] <err> coredump: #CD:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
[00:00:03.257,354] <err> coredump: #CD:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
[00:00:03.267,364] <err> coredump: #CD:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
[00:00:03.277,374] <err> coredump: #CD:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
[00:00:03.287,384] <err> coredump: #CD:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
[00:00:03.297,424] <err> coredump: #CD:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
[00:00:03.307,434] <err> coredump: #CD:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
[00:00:03.317,443] <err> coredump: #CD:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
[00:00:03.327,453] <err> coredump: #CD:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
[00:00:03.337,463] <err> coredump: #CD:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
[00:00:03.347,503] <err> coredump: #CD:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
[00:00:03.357,513] <err> coredump: #CD:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
[00:00:03.367,523] <err> coredump: #CD:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
[00:00:03.377,532] <err> coredump: #CD:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
[00:00:03.387,542] <err> coredump: #CD:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
[00:00:03.397,552] <err> coredump: #CD:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
[00:00:03.407,592] <err> coredump: #CD:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
[00:00:03.417,602] <err> coredump: #CD:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
[00:00:03.427,612] <err> coredump: #CD:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
[00:00:03.437,622] <err> coredump: #CD:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
[00:00:03.447,631] <err> coredump: #CD:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
[00:00:03.457,672] <err> coredump: #CD:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
[00:00:03.467,681] <err> coredump: #CD:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
[00:00:03.477,691] <err> coredump: #CD:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
[00:00:03.487,701] <err> coredump: #CD:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
[00:00:03.497,741] <err> coredump: #CD:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
[00:00:03.507,751] <err> coredump: #CD:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
[00:00:03.517,761] <err> coredump: #CD:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
[00:00:03.527,770] <err> coredump: #CD:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
[00:00:03.537,780] <err> coredump: #CD:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
[00:00:03.547,821] <err> coredump: #CD:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
[00:00:03.557,830] <err> coredump: #CD:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
[00:00:03.567,840] <err> coredump: #CD:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
[00:00:03.577,850] <err> coredump: #CD:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
[00:00:03.587,890] <err> coredump: #CD:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
[00:00:03.597,900] <err> coredump: #CD:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
[00:00:03.607,910] <err> coredump: #CD:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
[00:00:03.617,919] <err> coredump: #CD:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
[00:00:03.627,929] <err> coredump: #CD:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
[00:00:03.637,939] <err> coredump: #CD:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
[00:00:03.647,949] <err> coredump: #CD:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
[00:00:03.657,989] <err> coredump: #CD:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
[00:00:03.667,999] <err> coredump: #CD:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
[00:00:03.678,009] <err> coredump: #CD:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
[00:00:03.688,018] <err> coredump: #CD:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
[00:00:03.698,028] <err> coredump: #CD:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
[00:00:03.708,038] <err> coredump: #CD:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
[00:00:03.718,048] <err> coredump: #CD:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
[00:00:03.728,057] <err> coredump: #CD:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
[00:00:03.738,067] <err> coredump: #CD:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
[00:00:03.748,077] <err> coredump: #CD:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
[00:00:03.758,087] <err> coredump: #CD:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
[00:00:03.768,096] <err> coredump: #CD:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
[00:00:03.778,106] <err> coredump: #CD:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
[00:00:03.788,116] <err> coredump: #CD:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
[00:00:03.798,156] <err> coredump: #CD:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
[00:00:03.808,166] <err> coredump: #CD:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
[00:00:03.818,176] <err> coredump: #CD:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
[00:00:03.828,186] <err> coredump: #CD:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
[00:00:03.838,195] <err> coredump: #CD:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
[00:00:03.848,205] <err> coredump: #CD:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
[00:00:03.858,215] <err> coredump: #CD:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
[00:00:03.868,255] <err> coredump: #CD:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
[00:00:03.878,265] <err> coredump: #CD:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
[00:00:03.888,275] <err> coredump: #CD:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
[00:00:03.898,284] <err> coredump: #CD:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
[00:00:03.908,294] <err> coredump: #CD:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
[00:00:03.918,304] <err> coredump: #CD:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
[00:00:03.928,314] <err> coredump: #CD:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
[00:00:03.938,354] <err> coredump: #CD:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
[00:00:03.948,364] <err> coredump: #CD:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
[00:00:03.958,374] <err> coredump: #CD:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
[00:00:03.968,383] <err> coredump: #CD:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
[00:00:03.978,393] <err> coredump: #CD:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
[00:00:03.988,403] <err> coredump: #CD:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
[00:00:03.998,413] <err> coredump: #CD:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
[00:00:04.008,422] <err> coredump: #CD:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
[00:00:04.018,432] <err> coredump: #CD:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
[00:00:04.028,472] <err> coredump: #CD:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
[00:00:04.038,482] <err> coredump: #CD:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
[00:00:04.048,492] <err> coredump: #CD:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
[00:00:04.058,502] <err> coredump: #CD:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
[00:00:04.068,511] <err> coredump: #CD:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
[00:00:04.078,552] <err> coredump: #CD:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
[00:00:04.088,562] <err> coredump: #CD:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
[00:00:04.098,571] <err> coredump: #CD:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
[00:00:04.108,581] <err> coredump: #CD:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
[00:00:04.118,591] <err> coredump: #CD:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
[00:00:04.128,631] <err> coredump: #CD:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
[00:00:04.138,641] <err> coredump: #CD:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
[00:00:04.148,651] <err> coredump: #CD:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
[00:00:04.158,660] <err> coredump: #CD:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
[00:00:04.168,670] <err> coredump: #CD:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
[00:00:04.178,680] <err> coredump: #CD:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
[00:00:04.188,720] <err> coredump: #CD:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
[00:00:04.198,730] <err> coredump: #CD:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
[00:00:04.208,740] <err> coredump: #CD:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
[00:00:04.218,750] <err> coredump: #CD:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
[00:00:04.228,759] <err> coredump: #CD:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
[00:00:04.238,769] <err> coredump: #CD:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
[00:00:04.248,779] <err> coredump: #CD:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
[00:00:04.258,789] <err> coredump: #CD:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
[00:00:04.268,798] <err> coredump: #CD:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
[00:00:04.278,778] <err> coredump: #CD:aaaaaaaaaaaaaaaa50cf00204020000000000000714d0000aaaaaaaaaaaaaaaa
[00:00:04.288,787] <err> coredump: #CD:aaaaaaaaaaaaaaaa01000000000000000ccf0020aaaaaaaaaaaaaaaacbe10400
[00:00:04.298,797] <err> coredump: #CD:aaaaaaaaaaaaaaaaec11002063df040000000000aaaaaaaaaaaaaaaa00000000
[00:00:04.308,807] <err> coredump: #CD:c001000000000000ff010000ec1100207018002029e004007018002040200000
[00:00:04.318,817] <err> coredump: #CD:ec1100204000000070180020b76200000a00000004cf00200400000008000000
[00:00:04.328,826] <err> coredump: #CD:1000000000000000a0620600100000003d000000f08203202b00000080180020
[00:00:04.338,836] <err> coredump: #CD:100000001000000000070020000000000100000000000000000000008fc00300
[00:00:04.348,846] <err> coredump: #CD:5edf00001bd604002b000000288103202e000000e03f0020e825002039d60400
[00:00:04.358,856] <err> coredump: #CD:00000000000700200000000041b0030028810320480e00000000000084ea0000
[00:00:04.368,865] <err> coredump: #CD:801800204bd7040060d4040000000081e8250020510e00000400000001780000
[00:00:04.378,875] <err> coredump: #CD:80810320efdd0100484000205040002000000000df92020000000000aaaaaaaa
[00:00:04.388,885] <err> coredump: #CD:aaaaaaaa00000000000000000000000006000000000000000000000000000000
[00:00:04.398,895] <err> coredump: #CD:0000000000000000000000000000000000000000000000000000000000000000
[00:00:04.408,905] <err> coredump: #CD:5192020051920200000000000000000000000000fbd3040000000000aaaaaaaa
[00:00:04.418,884] <err> coredump: #CD:END#
[00:00:04.423,706] <err> os: Halting system

Setting 

CONFIG_COMMON_LIBC_MALLOC_ARENA_SIZE=50000

and it runs fine...

My conclusion : something is broken in the libc/malloc/newlib code, and in fact there are 2 pieces of code using the space from end of used RAM (&_end) up to end of ram, and they are overwriting each other.

And given that I have been using the 'heap==all of remaining RAM' config for a while in my app dev, and only have started having this problem since I integrated the wifi stuff, I'm assuming its somewhere in the mountains of code that the networking pulls in....

 Charlie  

 - can you confirm that making just these changes to the sta sample also breaks it for you

 - and see with the developers about what is causing it? 

Needless to say, this is 100% blocking for my product development.... 

[Charlie]

Hi Brian，

Sorry for the late reply. I have been travling last week.

I did a try with the following configurations added to original NCS v2.7.0 https://docs.nordicsemi.com/bundle/ncs-2.7.0/page/nrf/samples/wifi/sta/README.html sample on nRF7002DK.

CONFIG_COMMON_LIBC_MALLOC=y

CONFIG_COMMON_LIBC_MALLOC_ARENA_SIZE=-1

CONFIG_RESET_ON_FATAL_ERROR=n

Here is the log:

*** Booting nRF Connect SDK v2.7.0-6a541e852d54 ***
*** Using Zephyr OS v3.6.99-100befc70c74 ***
[00:00:00.609,130] <inf> net_config: Initializing network
[00:00:00.609,161] <inf> net_config: Waiting interface 1 (0x20000ed8) to be up...
[00:00:00.609,313] <inf> net_config: IPv4 address: 192.168.1.99
[00:00:00.609,344] <inf> net_config: Running dhcpv4 client...
[00:00:00.610,198] <inf> sta: Starting nrf7002dk with CPU frequency: 64 MHz
[00:00:01.610,382] <inf> sta: QSPI Encryption disabled
[00:00:01.610,443] <inf> sta: Static IP address (overridable): 192.168.1.99/255.255.255.0 -> 192.168.1.1
[00:00:01.610,443] <inf> sta: Waiting for Wi-Fi to be ready
[00:00:03.150,756] <inf> wifi_mgmt_ext: Connection requested
[00:00:03.150,787] <inf> sta: Connection requested
[00:00:03.150,848] <inf> sta: ==================
[00:00:03.150,878] <inf> sta: State: SCANNING
[00:00:03.451,019] <inf> sta: ==================
[00:00:03.451,049] <inf> sta: State: SCANNING
[00:00:03.751,190] <inf> sta: ==================
[00:00:03.751,220] <inf> sta: State: SCANNING
[00:00:04.051,361] <inf> sta: ==================
[00:00:04.051,361] <inf> sta: State: SCANNING
[00:00:04.351,531] <inf> sta: ==================
[00:00:04.351,562] <inf> sta: State: SCANNING
[00:00:04.651,702] <inf> sta: ==================
[00:00:04.651,733] <inf> sta: State: SCANNING
[00:00:04.951,873] <inf> sta: ==================
[00:00:04.951,904] <inf> sta: State: SCANNING
[00:00:05.252,044] <inf> sta: ==================
[00:00:05.252,075] <inf> sta: State: SCANNING
[00:00:05.552,215] <inf> sta: ==================
[00:00:05.552,215] <inf> sta: State: SCANNING
[00:00:05.852,386] <inf> sta: ==================
[00:00:05.852,416] <inf> sta: State: SCANNING
[00:00:06.152,557] <inf> sta: ==================
[00:00:06.152,587] <inf> sta: State: SCANNING
[00:00:06.452,728] <inf> sta: ==================
[00:00:06.452,758] <inf> sta: State: SCANNING
[00:00:06.752,899] <inf> sta: ==================
[00:00:06.752,899] <inf> sta: State: SCANNING
[00:00:07.053,070] <inf> sta: ==================
[00:00:07.053,100] <inf> sta: State: AUTHENTICATING
[00:00:07.236,053] <inf> sta: Connected
[00:00:07.251,525] <inf> net_dhcpv4: Received: 192.168.50.23
[00:00:07.251,678] <inf> net_config: IPv4 address: 192.168.50.23
[00:00:07.251,678] <inf> net_config: Lease time: 86400 seconds
[00:00:07.251,739] <inf> net_config: Subnet: 255.255.255.0
[00:00:07.251,770] <inf> net_config: Router: 192.168.50.1
[00:00:07.251,892] <inf> sta: DHCP IP address: 192.168.50.23
[00:00:07.363,464] <inf> sta: ==================
[00:00:07.363,494] <inf> sta: State: COMPLETED
[00:00:07.363,525] <inf> sta: Interface Mode: STATION
[00:00:07.363,555] <inf> sta: Link Mode: WIFI 6 (802.11ax/HE)
[00:00:07.363,555] <inf> sta: SSID: WiFi6
[00:00:07.363,586] <inf> sta: BSSID: C8:7F:54:91:94:50
[00:00:07.363,616] <inf> sta: Band: 2.4GHz
[00:00:07.363,616] <inf> sta: Channel: 10
[00:00:07.363,647] <inf> sta: Security: WPA2-PSK
[00:00:07.363,677] <inf> sta: MFP: Optional
[00:00:07.363,677] <inf> sta: RSSI: -26

I see you are using different NCS version, could you have a try with NCS v2.7.0?

Best regards,

Charlie

[BrianW]

Charlie said:

I see you are using different NCS version, could you have a try with NCS v2.7.0?

Well, I'm quite wary about doing an update to 2.7.0, as I understand there are some quite major changes to the build system (use of sysbuild), also maybe changes to the hardware DTS models?

I don't want to get even further behind in the project...

Can you try the change with NCS 2.6.2?