nrf5340/nrf7002 Connecting to wifi : ends up in blocked state consuming 60mA current?

RE: nrf5340/nrf7002 Connecting to wifi : ends up in blocked state consuming 60mA current?

BrianW — Wed, 18 Dec 2024 09:06:20 GMT

devzone.nordicsemi.com/.../trouble-migrating-from-ncs-2-6-to-2-8

RE: nrf5340/nrf7002 Connecting to wifi : ends up in blocked state consuming 60mA current?

BrianW — Tue, 17 Dec 2024 17:04:44 GMT

[quote userid="2115" url="~/f/nordic-q-a/116831/nrf5340-nrf7002-connecting-to-wifi-ends-up-in-blocked-state-consuming-60ma-current/514925"]

Let me know if you run into any issues, and I will try to help out.

[/quote]

So many issues.... I will open a new ticket about the migration as its completely broken my project...

RE: nrf5340/nrf7002 Connecting to wifi : ends up in blocked state consuming 60mA current?

Håkon Alseth — Fri, 13 Dec 2024 14:42:49 GMT

Hi,

v2.8.0 has the fix that was mentioned here:

[quote user="hkn"]

My apologies for this inconvenience. We have an open PR to fix this in the v2.6.x-branch:

https://github.com/nrfconnect/sdk-nrf/pull/18790

[/quote]

So in that aspect, yes; you should see stability improvement when running v2.8.0 as compared to v2.6.2.

[quote user="BrianW"]Any advice on such a migration?[/quote]

We have a migration guide in our documentation on how to go from multi-build to sysbuild:

https://docs.nordicsemi.com/bundle/ncs-latest/page/nrf/releases_and_maturity/migration/migration_sysbuild.html

Let me know if you run into any issues, and I will try to help out.

Kind regards,

Håkon

RE: nrf5340/nrf7002 Connecting to wifi : ends up in blocked state consuming 60mA current?

BrianW — Thu, 12 Dec 2024 14:10:07 GMT

Question : would migrating my project from 2.6.x to 2.8 latest help the wifi stability?

Any advice on such a migration?

RE: nrf5340/nrf7002 Connecting to wifi : ends up in blocked state consuming 60mA current?

Håkon Alseth — Mon, 02 Dec 2024 16:23:41 GMT

Hi,

[quote user="BrianW"]The heap size is fine, the problem is that each re-try that fails loses about 120-140 bytes! So it doesn't matter how big the heap is to start with, eventually it ends up in the bad state. What I need is a fix for the memory leak! (presumably in the WPA supplient code...)[/quote]

My deepest apologies for this issue. We have a fix in-place for v2.7-branch and v2.8, but the backport of it to v2.6-branch is yet to be merged.

[quote user="BrianW"]I'm not sure I signed up as a beta tester for your wifi products.... I'll take a look...[/quote]

This was not our intention, and I am sorry for the inconvenience this has caused.

[quote user="BrianW"]

This is not a sample, this is my real application :-) Looking at the sample wifi_sta, the struct wifi_connect_req_params timeout field is set to

CONFIG_STA_CONN_TIMEOUT_SEC * MSEC_PER_SEC;

which implies it should be the timeout for the connection attempt? But the stack always seems to take 10s...

However, I see in this sample that the struct net_mgmt_event_callback *cb 'info' field should contan the connection attempt status (although there is both "status' and 'conn_status' which seems redundant? The sample just uses 'status!=0' as being 'failure' and doesnt look at conn_status.)

The possible values for 'conn_status' seem more useful: I will try using these to detect the failed connection attempt!

[/quote]

Sorry, should have been a bit clearer in my question, and given a bit more background to it. The reason why I am asking is that there are subsystems that override the values, such as setting CONFIG_WIFI_CREDENTIALS_STATIC_SSID="MySSID" and CONFIG_WIFI_CREDENTIALS_STATIC_PASSWORD="MyPassword".

My question is how do you setup your credentials? Is it using a secure storage, directly via the net_mgmt APIs or static in Kconfig?

As an example, if you use the NET_REQUEST_WIFI_CONNECT_STORED type, as wifi/sta does:

https://github.com/nrfconnect/sdk-nrf/blob/v2.6.2/samples/wifi/sta/src/main.c#L225

You will load the default values given here:

https://github.com/nrfconnect/sdk-nrf/blob/v2.6.2/subsys/net/lib/wifi_mgmt_ext/wifi_mgmt_ext.c#L74

This .timeout value is then given to the WPA supplication (hostap):

https://github.com/nrfconnect/sdk-nrf/blob/v2.6.2/modules/hostap/src/supp_api.c#L542

and checked against a timer here:

https://github.com/nrfconnect/sdk-nrf/blob/v2.6.2/modules/hostap/src/supp_api.c#L125-L135

[quote user="BrianW"]

I'll take a look at the info you reference; although this seems very complex to just get robust long-term wifi operation on the device... Why doesn't the stack deal with this internally or when I disconnect it?!

This is in general the problem with the samples, they don't help with creating a 'real' application which has to deal with long running operation with connect/disconnect/errors that occur in real-life...

[/quote]

If the scenario occurs, where there for some reason gives a locked up state, the process of taking down the interface and booting it up again is handled in the background.

However, it will require the application to manually re-connect, which is the reason why this is given back as a "wi-fi ready" event. This is then a signal to the application that the wi-fi IF has been taken down and re-initialized, thus the application will need to re-connect over wi-fi and setup any socket based communication as required.

Kind regards,

Håkon

RE: nrf5340/nrf7002 Connecting to wifi : ends up in blocked state consuming 60mA current?

BrianW — Fri, 29 Nov 2024 20:05:15 GMT

Thanks for the responses!

[quote userid="2115" url="~/f/nordic-q-a/116831/nrf5340-nrf7002-connecting-to-wifi-ends-up-in-blocked-state-consuming-60ma-current/512815"]This is a typical issue if the heap is not large enough. Can you try to increase the heap via CONFIG_HEAP_MEM_POOL_SIZE to see if this helps?[/quote]

The heap size is fine, the problem is that each re-try that fails loses about 120-140 bytes! So it doesn't matter how big the heap is to start with, eventually it ends up in the bad state. What I need is a fix for the memory leak! (presumably in the WPA supplient code...)

[quote userid="2115" url="~/f/nordic-q-a/116831/nrf5340-nrf7002-connecting-to-wifi-ends-up-in-blocked-state-consuming-60ma-current/512815"]

shouldn't it use the timeout value?

Are you using the static configuration? If yes, then it is configured via CONFIG_WIFI_MGMT_EXT_CONNECTION_TIMEOUT.

[/quote]

This is not a sample, this is my real application :-) Looking at the sample wifi_sta, the struct wifi_connect_req_params timeout field is set to

CONFIG_STA_CONN_TIMEOUT_SEC * MSEC_PER_SEC;

which implies it should be the timeout for the connection attempt? But the stack always seems to take 10s...

The possible values for 'conn_status' seem more useful: I will try using these to detect the failed connection attempt!

For the continual retries:

[quote userid="2115" url="~/f/nordic-q-a/116831/nrf5340-nrf7002-connecting-to-wifi-ends-up-in-blocked-state-consuming-60ma-current/512815"]

My apologies for this inconvenience. We have an open PR to fix this in the v2.6.x-branch:

https://github.com/nrfconnect/sdk-nrf/pull/18790

Please note that this is still in review, and is not yet merged, but feel free to test it to see if this helps plug the memleak.

[/quote]

I'm not sure I signed up as a beta tester for your wifi products.... I'll take a look...

As for the badness when I disconnect before its given up:

[quote userid="2115" url="~/f/nordic-q-a/116831/nrf5340-nrf7002-connecting-to-wifi-ends-up-in-blocked-state-consuming-60ma-current/512815"]You can use the Wifi ready library (which uses the underlying RPU recovery functionality) to detect if such a problem has occurred.[/quote]

RE: nrf5340/nrf7002 Connecting to wifi : ends up in blocked state consuming 60mA current?

Håkon Alseth — Fri, 29 Nov 2024 16:21:20 GMT

Hi,

Your log indicates that the device is going into a bad state, where it cannot send umac commands to the nRF7002.

[quote user="BrianW"]

[14:23:55.279,052] <err> wifi_nrf: umac_cmd_alloc: Failed to allocate UMAC cmd
[14:23:55.286,895] <err> wifi_nrf: umac_cmd_cfg: umac_cmd_alloc failed
[14:23:55.294,067] <err> wifi_nrf: nrf_wifi_wpa_set_supp_port: nrf_wifi_fmac_chg_sta failed
[14:23:56.303,741] <err> wifi_nrf: nrf_wifi_fmac_scan: Unable to allocate memory
[14:23:56.311,737] <err> wifi_nrf: nrf_wifi_wpa_supp_scan2: Scan trigger failed

And the memory:

[14:24:03.500,335] <inf> app: System Stats: Heap now : free 1580, used 67112, max used 68040.

Just after boot, free is around 13000, and is stable when the wifi is connected.

[/quote]

This is a typical issue if the heap is not large enough. Can you try to increase the heap via CONFIG_HEAP_MEM_POOL_SIZE to see if this helps?

[quote user=""]

When trying to connect to a wifi AP with incorrect credentials, I have several issues:

1/ the result of the (failed) connect request always returns 10s later, no matter the value of 'timeout' in the wifi_connect_req_params structure passed to the net_mgmt request.

- shouldn't it use the timeout value?

[/quote]

Are you using the static configuration? If yes, then it is configured via CONFIG_WIFI_MGMT_EXT_CONNECTION_TIMEOUT.

[quote user=""]

2/ the result, returned as event NET_EVENT_WIFI_CONNECT_RESULT in a net_mgmt callback handler, doesn't give me a 'result' I can use?

How do I find out that the connection attempt failed? I tried checking the 'state' of the connection by doing a NET_REQUEST_WIFI_IFACE_STATUS, but it shows the state as being 'WIFI_STATE_ASSOCIATED', which the wifi demo code uses to indicate the wifi is connected!

(I can clearly see that the attempt has stopped after 10s because the power consumpion drops back to 5-6mA, instead of around 60mA)

[/quote]

In this state, you will be in the 4-way handshake state until a timer has elapsed. Based on your description, it does sound like the sample in question does not handle this scenario gracefully. Which example are you using here?

[quote user=""]

3/ if I do not explicltly disconnect after the failed attempt, the stack continually retries the connection every 30s, and appears to have a memory leak of about 120 bytes at eash attempt... leading eventually to a hang because it exhausts the heap...

To avoid this, I added a 'connect timer' in my code, which does an explicit NET_REQUEST_WIFI_DISCONNECT after X seconds. This stops the automatic retry, and lets my code retry the connect at its own pace....

[/quote]

My apologies for this inconvenience. We have an open PR to fix this in the v2.6.x-branch:

https://github.com/nrfconnect/sdk-nrf/pull/18790

Please note that this is still in review, and is not yet merged, but feel free to test it to see if this helps plug the memleak.

[quote user=""]

4/ After a few attempts (<10, one every 3 minutes), the wifi stack seems to get in a bad state, and the power consumption sticks at 60mA average! (as though it was still trying to connect to the AP)

I get this log:

[00:57:24.123,870] <err> wifi_nrf: nrf_wifi_wpa_supp_scan_abort: Timedout waiting for scan abort response, ret = -11

Requesting a disconnect operation is accepted by the wifi stack but has no apparent affect on the power consumption. And it doesn't then accept a connect attempt with valid credentials after getting in this state. Could this be because I force disconnect when its still trying to connect?

Any pointers on how to get this to be more robust?

[/quote]

You can use the Wifi ready library (which uses the underlying RPU recovery functionality) to detect if such a problem has occurred.

Here's a guide on how to enable it:

To enable the RPU recovery you need to enable this kconfig: https://docs.nordicsemi.com/bundle/ncs-2.6.2/page/nrf/drivers/wifi/nrf700x/nrf700x.html#cmdoption-arg-CONFIG_NRF_WIFI_RPU_RECOVERY
And implement the Wi-Fi Ready library + handle the events in the application itself: https://docs.nordicsemi.com/bundle/ncs-2.6.2/page/nrf/libraries/networking/wifi_ready.html
An example of such an handling is implemented in the wifi/sta sample: https://docs.nordicsemi.com/bundle/ncs-2.6.2/page/nrf/samples/wifi/sta/README.html#rpu_recovery
See the implementations guarded with "CONFIG_WIFI_READY_LIB" here: https://github.com/nrfconnect/sdk-nrf/blob/v2.5-branch/samples/wifi/sta/src/main.c#L56

A reset of the nRF7002 will require that the application re-connects to the network and also handle any previous connections as such.

Kind regards,

Håkon

RE: nrf5340/nrf7002 Connecting to wifi : ends up in blocked state consuming 60mA current?

BrianW — Fri, 29 Nov 2024 08:01:39 GMT

When it runs out of memory, the logs:

And the memory:

[14:24:03.500,335] <inf> app: System Stats: Heap now : free 1580, used 67112, max used 68040.

Just after boot, free is around 13000, and is stable when the wifi is connected.