nRF7002dk nrf5340 bus fault after shutting down interface

RE: nRF7002dk nrf5340 bus fault after shutting down interface

dejans — Tue, 12 Dec 2023 11:19:10 GMT

Hi,

Could you add CONFIG_LOG_MODE_IMMEDIATE=y to your configuration and provide obtained log file?

Best regards,
Dejan

RE: nRF7002dk nrf5340 bus fault after shutting down interface

dejans — Thu, 07 Dec 2023 09:59:19 GMT

Hi,

Thank you for providing the latest version of your project. I have forwarded your file to our developers who will continue looking into it. I will get back to you when I get new information.

Best regards,
Dejan

RE: nRF7002dk nrf5340 bus fault after shutting down interface

bcornell — Wed, 06 Dec 2023 13:49:56 GMT

Hello,

I have uploaded my latest test code here: https://dl.defelsko.com/downloads/nordic_test_dec_6.zip

It is the same code as I provided in October, but with the additional debug output you have requested since then.

Thank You

RE: nRF7002dk nrf5340 bus fault after shutting down interface

dejans — Wed, 06 Dec 2023 10:18:07 GMT

Hi,

Could you share your latest code with us? I can make this ticket private if your code contains any private or sensitive information you would not want to share publicly. Please let me know first if you want that I make this ticket private.

Best regards,
Dejan

RE: nRF7002dk nrf5340 bus fault after shutting down interface

dejans — Thu, 30 Nov 2023 20:02:50 GMT

Hi,

This is just a short update. Our developers are still looking into the issue. I will get back to you when I get new information.

Best regards,
Dejan

RE: nRF7002dk nrf5340 bus fault after shutting down interface

dejans — Tue, 28 Nov 2023 09:33:04 GMT

Hi,

Thank you for the log. We will look into it. I will get back to you as soon as I have some update, hopefully by the end of the week.

Best regards,
Dejan

RE: nRF7002dk nrf5340 bus fault after shutting down interface

bcornell — Mon, 27 Nov 2023 20:41:51 GMT

Hello,

We rebased again. The behavior is the same: https://dl.defelsko.com/downloads/nordic_log_nov27.txt

RE: nRF7002dk nrf5340 bus fault after shutting down interface

dejans — Fri, 24 Nov 2023 09:00:45 GMT

Hi,

For addressing scanning failures, there is this pull request available. If you still observe any issues, please collect the log and send it to us. For the purpose of debugging, you can enable debug features.

Best regards,
Dejan

RE: nRF7002dk nrf5340 bus fault after shutting down interface

dejans — Wed, 22 Nov 2023 16:09:25 GMT

Hi,

Your log shows scan timeout

<err> wpa_supp: wpa_drv_zep_get_scan_results2: Timed out waiting for scan results

followed by the scan failures.

Our developers will look into these issues. I will get back to you when I get some relevant information.

Best regards,
Dejan

RE: nRF7002dk nrf5340 bus fault after shutting down interface

bcornell — Wed, 22 Nov 2023 13:01:38 GMT

[quote userid="111786" url="~/f/nordic-q-a/104499/nrf7002dk-nrf5340-bus-fault-after-shutting-down-interface/456876"]Which error did you get? Can you show the error? How long did it take for error to appear?[/quote]

The complete logs are here:

[quote userid="126110" url="~/f/nordic-q-a/104499/nrf7002dk-nrf5340-bus-fault-after-shutting-down-interface/456762"]

We have uploaded this mornings log here: https://dl.defelsko.com/downloads/nordic_logs_nov21.zip

[/quote]

You will see that after about 2 hours, there is an endless stream of State: SCANNING messages as indicated on Nov 20th.

RE: nRF7002dk nrf5340 bus fault after shutting down interface

dejans — Wed, 22 Nov 2023 10:51:14 GMT

Hi,

[quote user="bcornell"]It appears that our issues may be related to timing. We tested yesterday without the very verbose dump_rpu_stats function you previously provided. When we added the logging you requested and re-enabled the dump_rpu_stats function, the error took significantly longer to occur.[/quote]

Thank you for the update.

Which error did you get? Can you show the error? How long did it take for error to appear?

Best regards,
Dejan

RE: nRF7002dk nrf5340 bus fault after shutting down interface

bcornell — Tue, 21 Nov 2023 16:44:50 GMT

Hello,

It appears that our issues may be related to timing. We tested yesterday without the very verbose dump_rpu_stats function you previously provided. When we added the logging you requested and re-enabled the dump_rpu_stats function, the error took significantly longer to occur.

We have uploaded this mornings log here: https://dl.defelsko.com/downloads/nordic_logs_nov21.zip

This is the result of running the same project that I have been running, but with CONFIG_WPA_SUPP_LOG_LEVEL_DBG=y and the latest version of NCS from github.

RE: nRF7002dk nrf5340 bus fault after shutting down interface

dejans — Tue, 21 Nov 2023 12:53:22 GMT

Hi,

Could you please add CONFIG_WPA_SUPP_LOG_LEVEL_DBG=y to your configuration and provide full log (from very beginning) when using Wi-Fi 6?

Best regards,
Dejan

RE: nRF7002dk nrf5340 bus fault after shutting down interface

dejans — Tue, 21 Nov 2023 12:15:38 GMT

Hi,

[quote user="bcornell"]I have rebased using the latest development branch and the behavior has changed. After 15 minutes, on wifi 6, I get an endless stream of State:Scanning messages[/quote]

This is unexpected. I have asked internally about this.

[quote user="bcornell"]Could you elaborate on how you tested? Our use case seems like an extremely basic test case:
* connect to wifi
* get http end point
* disconnect from wifi
* power off wifi
* wait for period of time
* power on wifi
* repeat[/quote]

For wi-fi shell test, we used the following repeating sequence:

net iface up
wifi scan
wifi connect
wifi status
net ping
net stats
wifi disconnect
net iface down

Best regards,
Dejan

RE: nRF7002dk nrf5340 bus fault after shutting down interface

bcornell — Mon, 20 Nov 2023 21:36:21 GMT

Hello,

Thank you for getting back to me.

I have rebased using the latest development branch and the behavior has changed. After 15 minutes, on wifi 6, I get an endless stream of State:Scanning messages
[2023-11-20 14:12:16] [02:34:24.108,917] <inf> sta: State: SCANNING
[2023-11-20 14:12:16] [02:34:24.409,057] <inf> sta: ==================
[2023-11-20 14:12:16] [02:34:24.409,088] <inf> sta: State: SCANNING
[2023-11-20 14:12:16] [02:34:24.709,197] <inf> sta: ==================
[2023-11-20 14:12:16] [02:34:24.709,228] <inf> sta: State: SCANNING
[2023-11-20 14:12:16] [02:34:25.009,338] <inf> sta: ==================
[2023-11-20 14:12:16] [02:34:25.009,368] <inf> sta: State: SCANNING
[2023-11-20 14:12:17] [02:34:25.309,478] <inf> sta: ==================
[2023-11-20 14:12:17] [02:34:25.309,509] <inf> sta: State: SCANNING
[2023-11-20 14:12:17] [02:34:25.610,168] <inf> sta: ==================
[2023-11-20 14:12:17] [02:34:25.610,198] <inf> sta: State: SCANNING
[2023-11-20 14:12:17] [02:34:25.910,339] <inf> sta: ==================
[2023-11-20 14:12:17] [02:34:25.910,369] <inf> sta: State: SCANNING
[2023-11-20 14:12:17] [02:34:26.210,479] <inf> sta: ==================
[2023-11-20 14:12:17] [02:34:26.210,510] <inf> sta: State: SCANNING
[2023-11-20 14:12:18] [02:34:26.510,620] <inf> sta: ==================
[2023-11-20 14:12:18] [02:34:26.510,650] <inf> sta: State: SCANNING
[2023-11-20 14:12:18] [02:34:26.810,791] <inf> sta: ==================
[2023-11-20 14:12:18] [02:34:26.810,821] <inf> sta: State: SCANNING

I tested multiple times today and the issue was very repeatable.

What are we doing wrong? Other customers must be using the nrf7002. In our testing, the drivers are completely unstable.

[quote userid="111786" url="~/f/nordic-q-a/104499/nrf7002dk-nrf5340-bus-fault-after-shutting-down-interface/456386"]With regard to your Wi-Fi 4 issue, we have done some testing using wi-fi shell sample and there were no issues on our side.[/quote]

Could you elaborate on how you tested? Our use case seems like an extremely basic test case:
* connect to wifi
* get http end point
* disconnect from wifi
* power off wifi
* wait for period of time
* power on wifi
* repeat
Do you have any suggestions on how to prove that this mode of operation is possible for extended periods of time?

RE: nRF7002dk nrf5340 bus fault after shutting down interface

dejans — Mon, 20 Nov 2023 12:17:36 GMT

Hi,

I am sorry for delayed reply. I was out of the office.

With regard to your Wi-Fi 4 issue, we have done some testing using wi-fi shell sample and there were no issues on our side.

Regarding Wi-Fi 6 problems, you could try the following pull requests - data vs control path races (which should resolve the Wi-Fi crash) and nrf70 patch update (for resolving data stall). As Wi-Fi 6 data stall (no IP) fix has been merged into the newest NCS, you could also try to checkout the latest NCS.

Best regards,
Dejan

RE: nRF7002dk nrf5340 bus fault after shutting down interface

dejans — Fri, 03 Nov 2023 08:45:15 GMT

Hi,

Thank you for providing additional files.
We will continue looking into your reported issues. I expect to get back to you with new information during next week.

Best regards,
Dejan

RE: nRF7002dk nrf5340 bus fault after shutting down interface

bcornell — Thu, 02 Nov 2023 20:41:31 GMT

[quote userid="111786" url="~/f/nordic-q-a/104499/nrf7002dk-nrf5340-bus-fault-after-shutting-down-interface/453796"]For further debugging we would need sniffer trace of wi-fi 4 communication. Could you provide it?
[/quote]

Here ( dl.defelsko.com/.../nordic_logs_nov2.zip ) is the Wireshark capture and log output from a test I ran today. The board main loop stopped about 5 minutes before I noticed and stopped the Wireshark capture. The code I ran is the same as yesterday so the ELF file is available in that download if needed.

The only device on the "dlink" network during testing was the Nordic development board.

RE: nRF7002dk nrf5340 bus fault after shutting down interface

bcornell — Thu, 02 Nov 2023 15:14:35 GMT

[quote userid="111786" url="~/f/nordic-q-a/104499/nrf7002dk-nrf5340-bus-fault-after-shutting-down-interface/453796"]

Based on your comment, is it correct that OK/FAIL is not coming from your internal server?

[/quote]

That is correct.

I will work on gathering that today and will provide the sniffer trace and console logs once they are available.

RE: nRF7002dk nrf5340 bus fault after shutting down interface

dejans — Thu, 02 Nov 2023 15:09:36 GMT

Hi,

[quote user="bcornell"]I do not know what the criterion for OK/FAIL are. That is why I posted in the DevZone asking for help. I assume they come from somewhere in your driver code or is a message being passed through from the modem?[/quote]

Based on your comment, is it correct that OK/FAIL is not coming from your internal server?

[quote user="bcornell"]Has any progress been made on the initial WiFi 6 issue where an MPU fault occurs when you disconnect & bring the interface down while to obtain an IP address? Based on the logs, it appears that the problem there is that once I give up waiting for an IP and try to disconnect and bring the interface down, the MPU fault occurs. I assume that is related to background processing not being cleaned up? [/quote]

This is still work in progress.

[quote user="dejans"]Could you provide sniffer trace for Wi-Fi 4 communication?[/quote]

For further debugging we would need sniffer trace of wi-fi 4 communication. Could you provide it?

Best regards,
Dejan

RE: nRF7002dk nrf5340 bus fault after shutting down interface

bcornell — Thu, 02 Nov 2023 14:52:53 GMT

[quote userid="111786" url="~/f/nordic-q-a/104499/nrf7002dk-nrf5340-bus-fault-after-shutting-down-interface/453753"]In wi-fi 4 case, what is the criterion for OK/FAIL? Are OK and FAIL status codes from httpbin.org?
[/quote]

We are not using httpbin.org as I mentioned previously:

[quote userid="126110" url="~/f/nordic-q-a/104499/nrf7002dk-nrf5340-bus-fault-after-shutting-down-interface/453323"]I've specified the address of an internal test server which posts a PHP endpoint which just returns the posted data.[/quote]

I added httpbin.org to the code provided so that you could see how the http post code functions. You will likely need a PHP endpoint on your own server to test this, I found that httpbin.org appears to have DoS protections that prevent this test from working continuously.

I do not know what the criterion for OK/FAIL are. That is why I posted in the DevZone asking for help. I assume they come from somewhere in your driver code or is a message being passed through from the modem?

Has any progress been made on the initial WiFi 6 issue where an MPU fault occurs when you disconnect & bring the interface down while to obtain an IP address? Based on the logs, it appears that the problem there is that once I give up waiting for an IP and try to disconnect and bring the interface down, the MPU fault occurs. I assume that is related to background processing not being cleaned up?

RE: nRF7002dk nrf5340 bus fault after shutting down interface

dejans — Thu, 02 Nov 2023 13:50:09 GMT

Hi,

Could you provide sniffer trace for Wi-Fi 4 communication?

In wi-fi 4 case, what is the criterion for OK/FAIL? Are OK and FAIL status codes from httpbin.org?

Best regards,
Dejan

RE: nRF7002dk nrf5340 bus fault after shutting down interface

dejans — Thu, 02 Nov 2023 09:06:56 GMT

Hi,

Thank you for testing and for providing required files.
We will look into it. I will get back to you with new information as soon as possible.

Best regards,
Dejan

RE: nRF7002dk nrf5340 bus fault after shutting down interface

bcornell — Wed, 01 Nov 2023 20:35:03 GMT

Hello,

I have tested the changes you provided, they made no difference to the behavior. They all appear to be related to adding more debug output.

I tested with both access points and have uploaded my elf files and logs here: dl.defelsko.com/.../nordic_logs_nov1.zip

RE: nRF7002dk nrf5340 bus fault after shutting down interface

dejans — Wed, 01 Nov 2023 08:37:01 GMT

Hi,

Could you try to use possible fix given below?

---
 overlay-debug.conf | 14 ++++++++++++++
 prj.conf           |  4 ++++
 src/main.c         |  7 +++++++
 3 files changed, 25 insertions(+)
 create mode 100644 overlay-debug.conf

diff --git a/overlay-debug.conf b/overlay-debug.conf
new file mode 100644
index 0000000..dd83de0
--- /dev/null
+++ b/overlay-debug.conf
@@ -0,0 +1,14 @@
+CONFIG_SHELL=y
+CONFIG_SHELL_BACKEND_SERIAL=y
+CONFIG_SHELL_STACK_SIZE=4096
+CONFIG_NET_SHELL=y
+CONFIG_SHELL_GETOPT=y
+CONFIG_SHELL_CMDS_RESIZE=n
+CONFIG_NRF700X_UTIL=y
+CONFIG_NET_L2_WIFI_SHELL=y
+CONFIG_NET_STATISTICS=y
+CONFIG_NET_STATISTICS_WIFI=y
+CONFIG_NET_STATISTICS_USER_API=y
+CONFIG_SYS_HEAP_RUNTIME_STATS=y
+# Enable for debugging connection issues
+# CONFIG_WPA_SUPP_LOG_LEVEL_DBG=y
diff --git a/prj.conf b/prj.conf
index 9b91578..32695f2 100644
--- a/prj.conf
+++ b/prj.conf
@@ -107,3 +107,7 @@ CONFIG_NET_HTTP_LOG_LEVEL_DBG=y
 CONFIG_THREAD_NAME=y
 
 CONFIG_RESET_ON_FATAL_ERROR=n
+
+
+# debugging
+CONFIG_SHELL_STACK_SIZE=4096
diff --git a/src/main.c b/src/main.c
index d9bcfe7..a3a76aa 100644
--- a/src/main.c
+++ b/src/main.c
@@ -416,6 +416,12 @@ int wifi_poweron(struct net_if *iface)
 
 //~ #define BT_LE_ADV_CONN_DEF BT_LE_ADV_PARAM(BT_LE_ADV_OPT_CONNECTABLE, 0x0640, 0x0680, NULL)
 
+void dump_rpu_stats(void)
+{
+	shell_execute_cmd(shell_backend_uart_get_ptr(), "wifi_util tx_stats 0");
+	shell_execute_cmd(shell_backend_uart_get_ptr(), "wifi_util rpu_stats all");
+}
+
 int main(void)
 {
 	//volatile unsigned int *myPointer = (volatile unsigned int *) 0x5002B500;
@@ -505,6 +511,7 @@ int main(void)
 			while (!have_ip) {
 				attempt++;		
 				printf("%d\n",attempt);
+				dump_rpu_stats();
 				if (attempt==200) {
 					printf("No ip.\n");
 					err=-99;
--

Please make sure that you build with overlay-debug.conf by providing extra argument -DOVERLAY_CONFIG=overlay-debug.conf. If the issue is still present, please provide full logs, and elf files.

Best regards,
Dejan