nRF7002dk nrf5340 bus fault after shutting down interface

[bcornell]

Hello,

I am working on a low power sensor logging project using the nRF7002dk. So far, I have updated the main loop in the wifi/sta sample and in place of the:

k_sleep(K_FOREVER);

I have added:

status = net_mgmt(NET_REQUEST_WIFI_DISCONNECT, iface, NULL, 0);
status = net_if_down(iface); 
k_sleep(K_SECONDS(2)); // will be longer in production, shortened for testing
status = net_if_up(iface);
k_sleep(K_SECONDS(2)); // allow the interface time to come up


I also have code verifying the statuses are returned as 0, but haven't posted that here to simplify my post.

I find that after a few connections, there is a kernel panic when trying to bring the interface back up.

[00:04:56.560,913] <inf> sta: State: SCANNING
[00:04:56.861,053] <inf> sta: ==================
[00:04:56.861,083] <inf> sta: State: SCANNING
[00:04:57.150,177] <err> os: ***** BUS FAULT *****
[00:04:57.150,177] <err> os: Precise data bus error
[00:04:57.150,207] <err> os: BFAR Address: 0x40000b08
[00:04:57.150,207] <err> os: r0/a1: 0x20000200 r1/a2: 0x20000580 r2/a3: 0x20000289
[00:04:57.150,207] <err> os: r3/a4: 0x20000588 r12/ip: 0x2003cca0 r14/lr: 0x200006a8
[00:04:57.150,238] <err> os: xpsr: 0x01000000
[00:04:57.150,238] <err> os: Faulting instruction address (r15/pc): 0x0003e97a
[00:04:57.150,268] <err> os: >>> ZEPHYR FATAL ERROR 25: Unknown error on CPU 0
[00:04:57.150,299] <err> os: Current thread: 0x20003470 (unknown)

The kernel panics stop if I add a 1 second delay between the net_mgmt and net_if_up calls. 

It appears that I can not shut down the interface immediately after disconnecting or something is not cleaned up that causes issues when bringing the interface back up. Obviously in production code, I would like to replace the delay with a check to ensure the disconnect has completed in case it takes longer than 1 second. How can I determine when it is safe to shut down the interface?

[dejans]

Hi,

I have added your modifications to the wifi sta sample from NCS v2.4.2. I tested this modified sample on nrf7002-dk board, but I could not reproduce your issue. In my case, the sample behaved as expected.

If you experience some congestion in your environment, you could try to increase the value of CONFIG_NET_MGMT_EVENT_QUEUE_TIMEOUT in the prj.conf file.

Best regards,
Dejan

[bcornell]

Hello,

Thank you for looking into this so quickly. I apologize, I forgot to mention that I am using the main branch. I have not tested this issue specifically in the NCS v2.4.2 release.

Idle power consumption is extremely high in the NCS v2.4.2 release, so it is not suitable for our application.

[bcornell]

Hello,

The error we have been getting when testing with our main development board and new WiFi 6 access point occurs after 2 or 3 hours and we see the message I posted previously. We have tested this 5 or 6 times with the same result every time.

Today, as suggested, we tested on a different development board and with a different (older WiFi 4) access point.

After 3 hours, it stopped operating with this message:

[2023-10-25 14:29:12] addrinfo @0x2006c038: ai_family=1, ai_socktype=1, ai_protocol=6, sa_family=1, sin_port=5000
[2023-10-25 14:29:12] sock = 9
[2023-10-25 14:29:12] OK
[2023-10-25 14:29:12] FAIL

We restarted the development board and after 3 hours got the same message.

Neither board has run for 4 continuous hours without crashing, but they always run longer than an hour and a half.

Switching routers/development boards does not fix the problem.

[dejans]

Hi,

Thank you for this update and additional information.

Which access points did you use (manufacturer and version) and which band did you use for your testing?

Best regards,
Dejan

[bcornell]

We have an old D-Link DIR-601B1 which was tested at 2.4GHz and a mesh network using multiple Cisco Business 150AX Access Points which support both 2.4GHz and 5GHz. The development board connects to that network using the 5Ghz band.

[dejans]

Hi,

Could you please provide a full crash log in both cases (old and new access point)?
In addition, could you provide elf files from your project's zephyr folder in both cases?

Best regards,
Dejan

[bcornell]

Hello,

Today, I tested both cases on the newer board and logged each time.

I noticed that in each instance of the first error case (WiFi 6 MPU Fault) it happens immediately after failing to obtain an IP address. I do not remember seeing that when testing previously on the other board.

The WiFi 4 case behaved exactly as I saw last week It worked great for a period of time before stopping without any type of indication as to what the problem is.

I have uploaded my logs and ELF files here: dl.defelsko.com/.../nordic_logs.zip

[dejans]

Hi,

Thank you for additional files and information.

We will look into your issue. I will get back to you when I get new information or if I need anything else from you.

Thank you for your patience.

Best regards,
Dejan

[dejans]

Hi,

In Wi-Fi 4 case, we would need to get more information about the cause of failure, and we could get this from your code. Could you share relevant piece of code from which we could get more information what causes error to appear?
Please note that in case your code contains any private or sensitive information, I can make this case private.

Best regards,
Dejan

[bcornell]

Hello,

I have already provided my code.

bcornell said:

Here is our project with Bluetooth commented out: https://dl.defelsko.com/downloads/nordic_test_no_ble.zip 

The only differences between that and what I am running are that my code obviously has an SSID and password, and in http_get.c I've specified the address of an internal test server which posts a PHP endpoint which just returns the posted data.

[dejans]

Hi,

bcornell said:

I have already provided my code.

Yes, you are right. We will look into it.

bcornell said:

The only differences between that and what I am running are that my code obviously has an SSID and password, and in http_get.c I've specified the address of an internal test server which posts a PHP endpoint which just returns the posted data.

Thank you for this information.

Best regards,
Dejan

[dejans]

Hi,

Could you try to use possible fix given below?

---
 overlay-debug.conf | 14 ++++++++++++++
 prj.conf           |  4 ++++
 src/main.c         |  7 +++++++
 3 files changed, 25 insertions(+)
 create mode 100644 overlay-debug.conf

diff --git a/overlay-debug.conf b/overlay-debug.conf
new file mode 100644
index 0000000..dd83de0
--- /dev/null
+++ b/overlay-debug.conf
@@ -0,0 +1,14 @@
+CONFIG_SHELL=y
+CONFIG_SHELL_BACKEND_SERIAL=y
+CONFIG_SHELL_STACK_SIZE=4096
+CONFIG_NET_SHELL=y
+CONFIG_SHELL_GETOPT=y
+CONFIG_SHELL_CMDS_RESIZE=n
+CONFIG_NRF700X_UTIL=y
+CONFIG_NET_L2_WIFI_SHELL=y
+CONFIG_NET_STATISTICS=y
+CONFIG_NET_STATISTICS_WIFI=y
+CONFIG_NET_STATISTICS_USER_API=y
+CONFIG_SYS_HEAP_RUNTIME_STATS=y
+# Enable for debugging connection issues
+# CONFIG_WPA_SUPP_LOG_LEVEL_DBG=y
diff --git a/prj.conf b/prj.conf
index 9b91578..32695f2 100644
--- a/prj.conf
+++ b/prj.conf
@@ -107,3 +107,7 @@ CONFIG_NET_HTTP_LOG_LEVEL_DBG=y
 CONFIG_THREAD_NAME=y
 
 CONFIG_RESET_ON_FATAL_ERROR=n
+
+
+# debugging
+CONFIG_SHELL_STACK_SIZE=4096
diff --git a/src/main.c b/src/main.c
index d9bcfe7..a3a76aa 100644
--- a/src/main.c
+++ b/src/main.c
@@ -416,6 +416,12 @@ int wifi_poweron(struct net_if *iface)
 
 //~ #define BT_LE_ADV_CONN_DEF BT_LE_ADV_PARAM(BT_LE_ADV_OPT_CONNECTABLE, 0x0640, 0x0680, NULL)
 
+void dump_rpu_stats(void)
+{
+	shell_execute_cmd(shell_backend_uart_get_ptr(), "wifi_util tx_stats 0");
+	shell_execute_cmd(shell_backend_uart_get_ptr(), "wifi_util rpu_stats all");
+}
+
 int main(void)
 {
 	//volatile unsigned int *myPointer = (volatile unsigned int *) 0x5002B500;
@@ -505,6 +511,7 @@ int main(void)
 			while (!have_ip) {
 				attempt++;		
 				printf("%d\n",attempt);
+				dump_rpu_stats();
 				if (attempt==200) {
 					printf("No ip.\n");
 					err=-99;
-- 

Please make sure that you build with overlay-debug.conf by providing extra argument    -DOVERLAY_CONFIG=overlay-debug.conf. If the issue is still present, please provide full logs, and elf files.

Best regards,
Dejan