Peer manager Flash storage is full

RE: Peer manager Flash storage is full

Edvin — Mon, 04 May 2020 06:13:10 GMT

Thank you for sharing. Yes, I believe this was fixed in SDK16, so porting at least the peer manager to this SDK should fix that issue. If I recall correctly, this bug triggers when the FDS is full, but you don't have any peers stored, and then try to save some bonding data.

BR,

Edvin

RE: Peer manager Flash storage is full

Acutetech — Sun, 03 May 2020 03:48:39 GMT

It might be useful to record that I also encountered this problem (bug?) today. It happened on a PCA10040 with SDK 15.3.0. I found this thread (thanks!). I noticed that peer_manager_handler.c has been updated in SDK 16.0.0, in this area, so I moved the code to SDK 16.0.0 and the problem went away.

For the record, here are the error messages with 15.3.0:

<warning> peer_manager_handler: Flash storage is full
<info> peer_manager_handler: Attempting to clean flash.
<error> peer_manager_handler: There are no peers to delete.
<info> peer_manager_handler: Connection secured: role: Peripheral, conn_handle: 0, procedure: Bonding
<info> peer_manager_handler: Attempting to clean flash.
<error> peer_manager_handler: There are no peers to delete.

and here are the messages the first time I ran the app with 16.0.0:

<info> peer_manager_handler: Attempting to clean flash.
<info> peer_manager_handler: Connection secured: role: Peripheral, conn_handle: 0, procedure: Bonding
<info> peer_manager_handler: Peer data updated in flash: peer_id: 0, data_id: Peer rank, action: Update
<info> peer_manager_handler: Peer data updated in flash: peer_id: 0, data_id: Bonding data, action: Update
<info> peer_manager_handler: Peer data updated in flash: peer_id: 0, data_id: Local database, action: Update
<info> peer_manager_handler: Peer data updated in flash: peer_id: 0, data_id: Central address resolution, action: Update
<info> peer_manager_handler: Peer data updated in flash: peer_id: 0, data_id: Local database, action: Update

RE: Peer manager Flash storage is full

Edvin — Mon, 29 Jul 2019 06:22:53 GMT

Hello,

As mentioned, I have forwarded this corner case to the SDK team, and they will take it into consideration for the next version of the SDK. Until then, you can either patch the peer manager to do an fds_gc() in this case, or run the fds_gc() if you are not able to add any new peers because the storage is full.

Best regards,

Edvin

RE: Peer manager Flash storage is full

SamJenkins2010 — Sat, 27 Jul 2019 00:23:03 GMT

I'm having the same problem on my project. I think it is a bug in the latest SDK 15.3.0.

Inside peer_manager_handler.c in the pm_handler_flash_clean() procedure...

The problem is that when the PM_EVT_STORAGE_FULL event occurs because the storage is full and requires a garbage collection. In my case all of the peers have been deleted by the application code, but no garbage collection was done. Later, when a new peer attempts to get added it comes through this path "Attempting to clean flash." (line 342) -> "There are no peers to delete." (line 356) and the err_code is set to NRF_ERROR_NOT_FOUND because there are no peers to delete and pm_peer_ranks_get() returned a proper error code. This then prevents the fds_gc() from even getting called on line 369 because it is guarded by the check for err_code == NRF_SUCCESS.

I'm not sure it makes sense to guard the call to fds_gc() based on the return value of pm_peer_ranks_get(). I think fds_gc() should get executed regardless of the presence of peers in the system.

I still have the binary image of my FDS when this bad state occurred. I have 4 pages (16 KiB) dedicated for storage. The stat of my FDS indicates the following:

Pages Available: 4
Open Records: 0
Valid Records: 6
Dirty Records: 228
Words Reserved: 0
Words Used: 3065
Largest Contig: 3
Freeable Words: 3033
Corruption: 0

There is an entire SWAP page ready to be used in my FDS that would facilitate a garbage collection.

It would be great if the SDK code could get fixed up for the next release to properly handle this scenario.

RE: Peer manager Flash storage is full

Demetrio Magrin REEL — Tue, 16 Apr 2019 12:27:03 GMT

Hello,

Thank you very much.

Demetrio Magrin

RE: Peer manager Flash storage is full

Edvin — Tue, 16 Apr 2019 12:01:28 GMT

Hello,

Thank you for posting.

I have forwarded this internally. This is sort of an unlikely corner case. That the FDS is full due to adding a peer in the peer manager (it doesn't typically use a lot of data).

But I guess it isn't unlikely if the FDS is only used by the peer manager. However, it would require you to delete the peers outside of the peer manager.

For now, you need to change the library to handle this.

Best regards,

Edvin

RE: Peer manager Flash storage is full

Demetrio Magrin REEL — Fri, 12 Apr 2019 12:35:05 GMT

Hello,

I understood what you advised me to do.
To do this I would have to modify the libraries.
I wonder if what I reported is a bug or is normal.

Thank you

BR

Demetrio Magrin

RE: Peer manager Flash storage is full

Edvin — Fri, 12 Apr 2019 07:12:02 GMT

[quote user="Demetrio Magrin REEL"]- Is it not correct to execute the "fds_gc()" function in case "PM_EVT_STORAGE_FULL" and "There are no peers to
delete." ?[/quote]

Yes, it is. If the storage is full, and there are no peers to delete, that means that it isn't the peer manager that is full, but the flash. Running a fds_gc() should clean the flash.

The reason why I asked is that "Running flash garbage collection" will only be printed if fds_gc() returns NRF_SUCCESS, so I am not sure whether fds_gc is not called, or if it is called, but not successfully.

If it is not called, you should call it (you can do that from elsewhere). If it is called, but can't succeed, then it may be because you are in a connection with too low (fast) connection interval. If so, try extending the connection interval, or disconnect before running the fds_gc().

BR,

Edvin

RE: Peer manager Flash storage is full

Demetrio Magrin REEL — Thu, 11 Apr 2019 15:27:30 GMT

Hello,

For me it is now difficult to reproduce the problem because I deleted the FLASH with SEGGER J-FLASH program.

Analyzing the messages displayed by J-LINK RTT program the code does not execute the "fds_gc()" function because "err_code = NRF_ERROR_NOT_FOUND" -> see msg "There are no peers to delete." and the msg
"Running flash garbage collection." does not appear in the log from J-Link RTT.

----------------------

00> <info> peer_manager_handler: Attempting to clean flash.
00> <error> peer_manager_handler: There are no peers to delete.

......

void pm_handler_flash_clean(pm_evt_t const * p_pm_evt)
{

case PM_EVT_STORAGE_FULL:

            if (!flash_cleaning)
            {
                err_code = NRF_SUCCESS;
                NRF_LOG_INFO("Attempting to clean flash.");
                if (!flash_write_after_gc)
                {
                    pm_peer_id_t peer_id_to_delete;
                    err_code = pm_peer_ranks_get(NULL, NULL, &peer_id_to_delete, NULL);
                    if (err_code == NRF_SUCCESS)
                    {
                        NRF_LOG_INFO("Deleting lowest ranked peer (peer_id: %d)", peer_id_to_delete);
                        err_code = pm_peer_delete(peer_id_to_delete);
                        APP_ERROR_CHECK(err_code);
                        flash_write_after_gc = true;
                    }
                    if (err_code == NRF_ERROR_NOT_FOUND)
                    {
                        NRF_LOG_ERROR("There are no peers to delete.");
                    }
                    else if (err_code == NRF_ERROR_NOT_SUPPORTED)
                    {
                        NRF_LOG_WARNING("Peer ranks functionality is disabled, so no peers are deleted.");
                    }
                    else
                    {
                        APP_ERROR_CHECK(err_code);
                    }
                }
                if (err_code == NRF_SUCCESS)
                {
                    err_code = fds_gc();
                    if (err_code == NRF_SUCCESS)
                    {
                        NRF_LOG_DEBUG("Running flash garbage collection.");
                        flash_cleaning = true;
                    }
                    else if (err_code != FDS_ERR_NO_SPACE_IN_QUEUES)
                    {
                        APP_ERROR_CHECK(err_code);
                    }
                }
            }
            break;

---------------------

Questions:
- Is it not correct to execute the "fds_gc()" function in case "PM_EVT_STORAGE_FULL" and "There are no peers to
delete." ?

Demetrio Magrin

RE: Peer manager Flash storage is full

Edvin — Thu, 11 Apr 2019 14:59:08 GMT

Hello,

Can you check whether fds_gc() is ever called inside your pm_handler_flash_clean() (on line 369 in peer_manager_handler.c)

Best regards,

Edvin

RE: Peer manager Flash storage is full

Demetrio Magrin REEL — Thu, 11 Apr 2019 10:13:47 GMT

Hi,

I forgot to report the "void pm_evt_handler (pm_evt_t const * p_evt)" function below.

static void pm_evt_handler(pm_evt_t const * p_evt)
{
   pm_handler_on_pm_evt(p_evt);
    pm_handler_flash_clean(p_evt);

    switch (p_evt->evt_id)
    {
        case PM_EVT_PEERS_DELETE_SUCCEEDED:
        {
            advertising_start(false);
        }
        break;
        case PM_EVT_CONN_SEC_CONFIG_REQ:
       {
           //
           // Allow pairing request from an already bonded peer.
           // Note that this will enable a potential attacker to update an existing bond.
           // So there is a trade-off between security and user-friendliness.
           //
           pm_conn_sec_config_t conn_sec_config = { .allow_repairing = true };
           pm_conn_sec_config_reply(m_conn_handle, &conn_sec_config);
       }
       break;
        default:
        break;
    }
}

Thank you.

Demetrio Magrin