Peer manager Flash storage is full

[Demetrio Magrin REEL]

I modified the FW "ble_app_uart" using the example "\nRF5_SDK_15.2.0_9412b96\examples\ble_peripheral\ble_app_uart \".
I added the pairing function and "delete_bonds ()".
The "delete_bonds ()" function is activated by pressing the EVAL button "Button 2" at startup.

After many pairing and delete_bonds () tests with the same device (Client) I saw that encrypted communication was no longer active (test by sniffer and wireshark).
I have seen that in these cases JLINK RTT sends me what is shown below.

00> <info> app: Debug logging for UART over RTT started.
00> <info> app: Connected
00> <info> app: Data len is set to 0xF4(244)
00> <info> peer_manager_handler: Connection security failed: role: Peripheral, conn_handle: 0x0, procedure: Bonding, error: 133
00> <info> app: Disconnected
00> <info> app: Connected
00> <info> app: Data len is set to 0xF4(244)
00> <warning> peer_manager_handler: Flash storage is full
00> <info> peer_manager_handler: Attempting to clean flash.
00> <error> peer_manager_handler: There are no peers to delete.
00> <info> peer_manager_handler: Connection secured: role: Peripheral, conn_handle: 0, procedure: Bonding
00> <info> peer_manager_handler: Attempting to clean flash.
00> <error> peer_manager_handler: There are no peers to delete.
00> <info> app: Disconnected
00>

To solve this problem I had to perform the "Erase flash" function through the "SEGGER J-FLASH" program.

Questions:
1) Why the "delete_bonds()" function does not solve this problem ?
2) Is there a way to solve this problem without having to perform the "Erase Flash" function through the SEGGER program ?

Thank you.

BR

Demetrio Magrin

[Edvin]

Hello,

Can you check whether fds_gc() is ever called inside your pm_handler_flash_clean() (on line 369 in peer_manager_handler.c)

Best regards,

Edvin

[Demetrio Magrin REEL]

Hello,

For me it is now difficult to reproduce the problem because I deleted the FLASH with SEGGER J-FLASH program.

Analyzing the messages displayed by J-LINK RTT program the code does not execute the "fds_gc()" function because "err_code = NRF_ERROR_NOT_FOUND" -> see msg "There are no peers to delete." and the msg
 "Running flash garbage collection." does not appear in the log from J-Link RTT.

----------------------

00> <info> peer_manager_handler: Attempting to clean flash.
00> <error> peer_manager_handler: There are no peers to delete.

......

void pm_handler_flash_clean(pm_evt_t const * p_pm_evt)
{

        case PM_EVT_STORAGE_FULL:

            if (!flash_cleaning)
            {
                err_code = NRF_SUCCESS;
                NRF_LOG_INFO("Attempting to clean flash.");
                if (!flash_write_after_gc)
                {
                    pm_peer_id_t peer_id_to_delete;
                    err_code = pm_peer_ranks_get(NULL, NULL, &peer_id_to_delete, NULL);
                    if (err_code == NRF_SUCCESS)
                    {
                        NRF_LOG_INFO("Deleting lowest ranked peer (peer_id: %d)", peer_id_to_delete);
                        err_code = pm_peer_delete(peer_id_to_delete);
                        APP_ERROR_CHECK(err_code);
                        flash_write_after_gc = true;
                    }
                    if (err_code == NRF_ERROR_NOT_FOUND)
                    {
                        NRF_LOG_ERROR("There are no peers to delete.");
                    }
                    else if (err_code == NRF_ERROR_NOT_SUPPORTED)
                    {
                        NRF_LOG_WARNING("Peer ranks functionality is disabled, so no peers are deleted.");
                    }
                    else
                    {
                        APP_ERROR_CHECK(err_code);
                    }
                }
                if (err_code == NRF_SUCCESS)
                {
                    err_code = fds_gc();
                    if (err_code == NRF_SUCCESS)
                    {
                        NRF_LOG_DEBUG("Running flash garbage collection.");
                        flash_cleaning = true;
                    }
                    else if (err_code != FDS_ERR_NO_SPACE_IN_QUEUES)
                    {
                        APP_ERROR_CHECK(err_code);
                    }
                }
            }
            break;

---------------------

Questions:
- Is it not correct to execute the "fds_gc()" function in case "PM_EVT_STORAGE_FULL" and "There are no peers to  
  delete." ?

BR

Demetrio Magrin

[Edvin]

Demetrio Magrin REEL said:

- Is it not correct to execute the "fds_gc()" function in case "PM_EVT_STORAGE_FULL" and "There are no peers to  
  delete." ?

 Yes, it is. If the storage is full, and there are no peers to delete, that means that it isn't the peer manager that is full, but the flash. Running a fds_gc() should clean the flash.

The reason why I asked is that "Running flash garbage collection" will only be printed if fds_gc() returns NRF_SUCCESS, so I am not sure whether fds_gc is not called, or if it is called, but not successfully. 

If it is not called, you should call it (you can do that from elsewhere). If it is called, but can't succeed, then it may be because you are in a connection with too low (fast) connection interval. If so, try extending the connection interval, or disconnect before running the fds_gc().

BR,

Edvin

[Demetrio Magrin REEL]

Hello,

I understood what you advised me to do.
To do this I would have to modify the libraries.
I wonder if what I reported is a bug or is normal.

Thank you

BR

Demetrio Magrin

[Edvin]

Hello,

Thank you for posting.

I have forwarded this internally. This is sort of an unlikely corner case. That the FDS is full due to adding a peer in the peer manager (it doesn't typically use a lot of data).

But I guess it isn't unlikely if the FDS is only used by the peer manager. However, it would require you to delete the peers outside of the peer manager.

For now, you need to change the library to handle this.

Best regards,

Edvin

[Demetrio Magrin REEL]

Hello,

Thank you very much.

BR

Demetrio Magrin

[SamJenkins2010]

I'm having the same problem on my project. I think it is a bug in the latest SDK 15.3.0.

Inside peer_manager_handler.c in the pm_handler_flash_clean() procedure...

The problem is that when the PM_EVT_STORAGE_FULL event occurs because the storage is full and requires a garbage collection. In my case all of the peers have been deleted by the application code, but no garbage collection was done. Later, when a new peer attempts to get added it comes through this path "Attempting to clean flash." (line 342) -> "There are no peers to delete." (line 356) and the err_code is set to NRF_ERROR_NOT_FOUND because there are no peers to delete and pm_peer_ranks_get() returned a proper error code. This then prevents the fds_gc() from even getting called on line 369 because it is guarded by the check for err_code == NRF_SUCCESS.

I'm not sure it makes sense to guard the call to fds_gc() based on the return value of pm_peer_ranks_get(). I think fds_gc() should get executed regardless of the presence of peers in the system.

I still have the binary image of my FDS when this bad state occurred. I have 4 pages (16 KiB) dedicated for storage. The stat of my FDS indicates the following:

Pages Available: 4
Open Records: 0
Valid Records: 6
Dirty Records: 228
Words Reserved: 0
Words Used: 3065
Largest Contig: 3
Freeable Words: 3033
Corruption: 0

There is an entire SWAP page ready to be used in my FDS that would facilitate a garbage collection.

It would be great if the SDK code could get fixed up for the next release to properly handle this scenario.

[Edvin]

Hello,

As mentioned, I have forwarded this corner case to the SDK team, and they will take it into consideration for the next version of the SDK. Until then, you can either patch the peer manager to do an fds_gc() in this case, or run the fds_gc() if you are not able to add any new peers because the storage is full.

Best regards,

Edvin

[Acutetech]

It might be useful to record that I also encountered this problem (bug?) today. It happened on a PCA10040 with SDK 15.3.0. I found this thread (thanks!). I noticed that peer_manager_handler.c has been updated in SDK 16.0.0, in this area, so I moved the code to SDK 16.0.0 and the problem went away.

For the record, here are the error messages with 15.3.0:

<warning> peer_manager_handler: Flash storage is full
<info> peer_manager_handler: Attempting to clean flash.
<error> peer_manager_handler: There are no peers to delete.
<info> peer_manager_handler: Connection secured: role: Peripheral, conn_handle: 0, procedure: Bonding
<info> peer_manager_handler: Attempting to clean flash.
<error> peer_manager_handler: There are no peers to delete.

and here are the messages the first time I ran the app with 16.0.0:

<info> peer_manager_handler: Attempting to clean flash.
<info> peer_manager_handler: Connection secured: role: Peripheral, conn_handle: 0, procedure: Bonding
<info> peer_manager_handler: Peer data updated in flash: peer_id: 0, data_id: Peer rank, action: Update
<info> peer_manager_handler: Peer data updated in flash: peer_id: 0, data_id: Bonding data, action: Update
<info> peer_manager_handler: Peer data updated in flash: peer_id: 0, data_id: Local database, action: Update
<info> peer_manager_handler: Peer data updated in flash: peer_id: 0, data_id: Central address resolution, action: Update
<info> peer_manager_handler: Peer data updated in flash: peer_id: 0, data_id: Local database, action: Update