Bug in peer manager in SDK 14.2

RE: Bug in peer manager in SDK 14.2

dkkeller — Sat, 16 Mar 2019 11:58:30 GMT

This was fixed with SDK15.0.0 👍

RE: Bug in peer manager in SDK 14.2

Amy — Thu, 22 Feb 2018 10:04:34 GMT

Hi Petter,

I faced the same issue.

Could you also provide the SDK14.0.0 for peer_database.c?

The code base is changed a lot between SDK14.2.0 adn SDK14.0.0

Thanks.

RE: Bug in peer manager in SDK 14.2

jumpingfool — Sun, 04 Feb 2018 05:11:35 GMT

Yes, that's fixed it - thanks!

Even after reviewing your changes I still have no idea what the cause of the problem was, I can only be thankful that Nordic have such a skilled support team. :)

RE: Bug in peer manager in SDK 14.2

yacine — Thu, 01 Feb 2018 21:33:11 GMT

Forget my comment you're right,it has nothing to do with your issue. the only common points are gatt and sdk14. It's too late, I need rest !

RE: Bug in peer manager in SDK 14.2

yacine — Thu, 01 Feb 2018 20:52:29 GMT

the last lines are:

nrf_sdh_ble: BLE event: 0x12. // BLE_GAP_EVT_CONN_PARAM_UPDATE nrf_sdh_ble: BLE event: 0x12. // BLE_GAP_EVT_CONN_PARAM_UPDATE

It stays like this. I use the same client as for SDK12. with SDK12 it connects and I can write/read etc.

RE: Bug in peer manager in SDK 14.2

jumpingfool — Thu, 01 Feb 2018 20:46:09 GMT

yacine - this looks like a different issue to me. What does the stack trace look like when your app gets stuck?

RE: Bug in peer manager in SDK 14.2

yacine — Thu, 01 Feb 2018 20:08:44 GMT

hello Peter, jumpingfool,

I have a very similar problem since my migration to SDK14, but without the flash problems.

<debug> my_app: Advertising....
<debug> nrf_sdh_ble: BLE event: 0x10. //BLE_GAP_EVT_CONNECTED
<debug> nrf_sdh_ble: BLE event: 0x23. //BLE_GAP_EVT_DATA_LENGTH_UPDATE_REQUEST
<debug> ble_gatt: Peer on connection 0x0 requested a data length of 27 bytes.
<debug> ble_gatt: Updating data length to 27 bytes on connection 0x0.
<debug> nrf_sdh_ble: BLE event: 0x24. //BLE_GAP_EVT_DATA_LENGTH_UPDATE
<debug> ble_gatt: Data length updated to 27 on connection 0x0.
<debug> ble_gatt: max_rx_octets: 27
<debug> ble_gatt: max_tx_octets: 27
<debug> ble_gatt: max_rx_time: 328
<debug> ble_gatt: max_tx_time: 328
<debug> my_app: ATT MTU exchange completed. central 0x17 peripheral 0x17
<debug> nrf_sdh_ble: BLE event: 0x12. //BLE_GAP_EVT_CONN_PARAM_UPDATE
<debug> nrf_sdh_ble: BLE event: 0x12. //BLE_GAP_EVT_CONN_PARAM_UPDATE

After the GATT negotiation, I receive two or three BLE_GAP_EVT_CONN_PARAM_UPDATE and I'lm stucked here.

on SDK12 (-without the gatt), it worked like a charm

thx

yacine

RE: Bug in peer manager in SDK 14.2

Petter Myhre — Wed, 31 Jan 2018 14:47:06 GMT

I see.

The CCCD shall be persistent across connections for bonded devices. See Section 3.3.3.3, Part G, Vol. 3 in the Bluetooth core v5 spec.

Thank you for the information, I will have to look into this further. Will update here when I have any news.

Edit 02.02.2018

Could you try fix this and see if it resolves the issue?devzone.nordicsemi.com/.../fix.rar

RE: Bug in peer manager in SDK 14.2

jumpingfool — Wed, 31 Jan 2018 14:29:02 GMT

Just trying to debug a little further (which is quite hard as adding logging seems to cause thread priority issues!) it seems that the peer database updates that are initiated after garbage collection has started but before it has completed get stuck in a loop calling reattempt_previous_operations() where the call to write_space_reserve() is returning FDS_ERR_NO_SPACE_IN_FLASH which then generates PM_EVT_STORAGE_FULL and it goes around in a cycle. This is despite the fact that all indications are that garbage collection has finished by this point so there should be storage available - as evidenced by the fact that the FDS_EVT_GC has been received and the output of fds_stat() which shows the dirty pages are gone. Perhaps peer manager is getting out of sync with FDS somehow because of the GC operation moving where the record for the peer data is located in flash after the write operation has started, but the code around that is mighty complicated and I am struggling to follow the logic. At this point I think the workaround for me is simply to put a delay in between writing to each CCCD on the iOS side - that is a fudge I'd rather avoid if I can but I need to move past this issue. What I can say is this issue was not present in SDK 12.3.

RE: Bug in peer manager in SDK 14.2

jumpingfool — Wed, 31 Jan 2018 12:58:21 GMT

Ok, so this is confirmed, if I limit my app to only writing to one CCCD at connection time then everything works as expected - if the flash buffer is full GC completes fine and everything runs normally. So the issue is that the peer manager/FDS is not correctly managing the situation where multiple CCCD writes are made simultaneously and FDS needs to do garbage collection to store their values in the local cache.

I also note that if I disable the gatt module my peripheral no longer receives BLE_GATTS_EVT_WRITE events - so turning it off seems like a bad idea!

RE: Bug in peer manager in SDK 14.2

jumpingfool — Wed, 31 Jan 2018 12:27:12 GMT

Unfortunately my central is an unpublished iOS app so although I'd be happy to share the project with you I don't think that's going to be much help without the app to test with.

However, I think I may now know what is happening. I do get PM_EVT_FLASH_GARBAGE_COLLECTED and FDS_EVT_GC after the first CCCD write and fds_stat() shows the dirty records have gone at that point. However, I can see three flash writes going out before this happens (as my app requests notification on three CCCDs at connection). My guess is that we have contention here on the flash buffer - the CCCD writes are coming in while the GC process is still in flight and this is somehow screwing up the FDS state and causing problems . Is that plausible?

Out of interest why is persistent storage for the CCCD state required? Am I doing the right thing by writing to the CCCD each time the app connects to ensure I get notifications or should I be reading the state and only updating if notifications are not already enabled?

RE: Bug in peer manager in SDK 14.2

Petter Myhre — Wed, 31 Jan 2018 11:31:31 GMT

Hmm. I didn't expect the writes to go away by commenting out gatt_init(). I expected MTU exchange to go away. Which central example are using? Maybe the write is trigger by the MTU exchange or something.

If you don't want to use the GATT module you should also remove the files from the project, and/or disable the module in sdk_config.h see NRF_BLE_GATT_ENABLED. Implications will be that some events will not be handled anymore, see nrf_ble_gatt_on_ble_evt().

When you run fds_stat() immediately after calling fds_gc()? That is expected, the GC is not complete until you get PM_EVT_FLASH_GARBAGE_COLLECTED and FDS_EVT_GC. Do you get these events? What do fds_stat() return then?

I'm not sure why the logging changes anything. Would it be possible to upload your complete projects? And tell me how I can reproduce this? And which logging removes removes the issue?

RE: Bug in peer manager in SDK 14.2

jumpingfool — Wed, 31 Jan 2018 10:36:50 GMT

Hi Petter - thanks for your help. Yes I am writing to (several) CCCDs immediately when the connection is established so my central can receive notifications. A few updates.

First I tried commenting out gatt_init from main as you suggested and this makes the problem go away - the flash writes simply don't happen anymore. I'm not sure if there are other implications with doing that though?

Secondly I implemented some logging around the PM_EVT_STORAGE_FULL event. Calling fds_gc() returns zero but the GC does not actually complete - there are still over a hundred dirty pages when I run fds_stat() immediately after. Interestingly I also added some logging to PM_EVT_LOCAL_DB_CACHE_APPLIED so I could see the cache filling up and the problem went away - GC did clear out the dirty records. The delay associated with the extra logging clearly helped so I think this must be a race condition - perhaps the previous flash write is not completing before fds_gc() is executed?

RE: Bug in peer manager in SDK 14.2

Petter Myhre — Tue, 30 Jan 2018 12:45:25 GMT

You say GC is attempted? Could you elaborate?

Do you get the PM_EVT_STORAGE_FULL event in pm_evt_handler()? If you do, do fds_gc() return 0x00000000 (NRF_SUCCESS)? Do you get the PM_EVT_FLASH_GARBAGE_COLLECTED event in pm_evt_handler()? Or the FDS_EVT_GC event in fds_evt_handler()?

Could you run fds_stat() and see what it returns?

The ATT MTU request is probably done by the GATT module, I think you can just comment out gatt_init() in main() and this will not happen.

I'm guessing that the BLE_GATTS_EVT_WRITE is a write to a CCCD? Every time a CCCD is written to, the PM_PEER_DATA_ID_GATT_LOCAL_V2 record will be updated. This means that a new record will be written, and the old should be invalidated. Then when doing GC this invalidated record should be collected.

Would it be possible for you to upload the complete projects so I can test them here?