How to disable storage in flash memory all information of bonding/pairing/wititelist/passkey ? Needs for test-fixture that connect just one time

[Cedric M]

Hello,

I have products based on NRF51822, that are central and peripheral.

These products use Softdevice S310, bootloader and application code. It use peer manager, fs storage. I do developpment based on different nordic examples.

Products are connected together with bonding, whitelist, passkey, and there are compliant with standard BLE of smartphone too.

each product include normal and test program (I can access to test function by specific custom registers in application code)

Products works.

My problem is for testing on production line.

I developp a test-fixture based on same chip (and near same code, difference is about adding UART connexion on test-fixture to an external automatic PC program to control BLE chip).

For each card that I want to test, I must connect to it (on BLE) to execute program test (NRF51 under test is wire controlled to I/O that I want to test too under radio link)

To connect to each card (DUT) with the test-fixture, I must do a complete connexion (bonding...) and parameters card are saving in flash storage. To test another card, I must do a total erase to remove all data stored in flash, and this is my problem.

I do not find explicit function of saving at each step (bonding, passkey...), I test several things in code to remove step of saving, but I have a lot of errors, it seems to be more complicated that I think to not use saving and loading from flash (maybe functions included in soft device ?)...

Is there a solution to perform all steps (bonding....passkey) without saving (and loading) anything in flash storage (then running only on actual session on RAM) ?

My objective is to test quickly in series all card... (and without degrade endurance of flash cycle that would be another problem, too)

Thanks,

Cedric

[Edvin]

Hello,

The peer manager is quite tangled into the flash operations, since that is it's only job. Do you really need to test bonding on every device during production? Perhaps you should at least consider testing only pairing. 

I don't know what parts of the bonding you want to test. If it is just the bonding procedure (which is almost identical to the pairing procedure), you may be able to find out just where it writes to flash, and comment out that part (and return a fake NRF_SUCCESS). But if you intend to disconnect and connect again, you would also have to fake the part where it looks up the bonding information, which may be a bit more tedious. 

One way I can think of is to write a custom backend for FDS that doesn't use NVMC or the softdevice API, but a custom RAM file system of some sort. This way you wouldn't have to touch the peer manager at all. The alternative way is to find out where the peer manager stores the bonding information, and just write this to a custom variable in RAM instead of a flash record. Then, you would need to find out where it reads out these records later, and fetch the data from the custom variable instead.

So can you elaborate a bit on what parts of the bonding you want to test?

Best regards,

Edvin

[Cedric M]

Hello,

I add complement of #2. I made another test today :

Compared to the last remarks, I investiguated the way in which Flash memory is accessed during bonding.
In the sys_evt_dispatch () function (of BLE source code), we find the function fs_sys_event_handler (), where fs_ = FlashStorage. This function seems to be call after an event bonding.

If I inhibit this function permanently, the application does not start. Compared to if I let it run the first time and then deactivate it afterwards, I have the impression that the bonding is working and is no longer flash savings.

I noticed that the 1st time the memory is written with the data 0xC007 0x0014.

- What does this data correspond to (test condition is : boot without any connexion of peripheral) ?

- is my method consistent with you (to execute function only first boot, and after never) ?

Please see picture of data memory that is written first time :

Regards,

CM

[Cedric M]

Hello,

#1

I have examined the HEX file in link. A priori it is not an NRF51, but whatever the principle remains the same. I have the same thing when I program, the memories are well initialized without any data. I am using nrf connect by doing an "erase all".
My remarks come after the execution has taken place at least once, that is to say that it is the software which modifies the memory.

#2

Thanks for documentation link.

I understand better how the bytes are used.
I confirm to you that initially the memory is quite empty (erased 0xFF). There is no such data. It is the execution of the software that adds it.
If I understood correctly, to read (and understand) these values, there is no need to swap the bytes, we read the addresses in order in a basic way (not like DECOADDE which becomes DEADCODE).
For now, I haven't figured out what 07 C0 14 00 is yet. According to the documentation 07 C0 would be the record key, and the address range is in 0x0001-0xBFFF, so it's a data range because the peer manager uses 0xC000-0xFFFE.

For the SDK, I checked again if I was up to date, it is. As I indicated at the beginning, the chip is an NRF51, and according to the official Nordic correspondence table, I have to use the S130 softdevice (BLE in central and peripheral role).

here is the link :

https://infocenter.nordicsemi.com/index.jsp?topic=%2Fcomp_matrix_nrf51%2FCOMP%2Fnrf51%2Fnrf51_comp_matrix.html

Then, I am using SDK 12.3.0 with softdevice v2.0.1 as requested. Other SDKs do not support NRF51.

nRF5 SDK v12.3.0
----------------
Release Date: Week 19, 2017

For the link of flash storage, I use it (instead of v17). I compared each one, it seems to be identical.

https://infocenter.nordicsemi.com/index.jsp?topic=%2Fsdk_nrf5_v17.0.0%2Flib_fds_format.html

Yes, I use the peer manager.

since yesterday, I still have not managed to identify which function could write data, and what the key record 07 C0 corresponds to.

A new records (07 C0....) is saved (in central) each time I connect a new perripheral while the function fs_sys_event_handler(sys_evt); is disabled.

Regards,

[Edvin]

Cedric M said:

I have examined the HEX file in link. A priori it is not an NRF51, but whatever the principle remains the same

 Indeed. It is an nRF52840. It is what I had connected to the computer. I don't have any nRF51 DKs at the moment, but I used SDK 12.3.0, and the ble_app_hrs example for the nRF52840, which should behave equivalent. At least FDS wise.

 

Cedric M said:

My remarks come after the execution has taken place at least once, that is to say that it is the software which modifies the memory.

 My flash dump was taken after I started the application. The softdevice never writes anything to the FDS pages directly. The only link between the FDS and the softdevice is that FDS uses fstorage, which uses a softdevice backend to read and write to flash, but the softdevice never initiates a flash write without it coming from the application. Typically, the softdevice triggers events, e.g. in the peer manager, which then writes to the FDS -> fstorage -> softdevice backend.

 

Cedric M said:

If I understood correctly, to read (and understand) these values, there is no need to swap the bytes, we read the addresses in order in a basic way (not like DECOADDE which becomes DEADCODE).

 This always confuses me. I suspect that we do. I think that the record key is 0xC007. I guess this comes from PDS_FIRST_RESERVED_FILE_ID or PDS_FIRST_RESERVED_RECORD_KEY, which is used in PEER_ID_TO_FILE_ID, FILE_ID_TO_PEER_ID, DATA_ID_TO_RECORD_KEY and RECORD_KEY_TO_DATA_ID. So the list of places to check quickly becomes very large. 

What happened to not bonding, but just pairing in your production test?

If you want to see where the peer manager uses fds, I guess I would start by commenting out fds_register() in peer_manager_init() -> pm_init() -> pds_init() -> fds_register(fds_evt_handler); Perhaps also fds_init() in the same function.

And then see where it fails. 

I just noticed when checking how pm_init() is set up that there is something relevant in pds_init():

pds_init() -> peer_ids_load():

uint16_t const record_key = peer_data_id_to_record_key(PM_PEER_DATA_ID_BONDING);

PM_PEER_DATA_ID_BONDING = PM_PEER_DATA_ID_BONDING_V2 = 7, and peer_data_id_to_record_key() will add DATA_ID_TO_RECORD_KEY (0xC000) to this. 

So peer_ids_load() will try to find a record with record key 0xC007. It doesn't write it, but this is at least somewhat related. 

pm_peer_data_bonding_store() will store a record using PM_PEER_DATA_ID_BONDING together with pm_peer_data_bonding_store (which makes up 0xC007), so you can check where this is called, and possibly comment out that place. 

However, after a "deep dive" into this, I realized that I don't think any of this is necessary. Why do you need to disable the flash writes? 

1. I don't really think bonding is necessary to test during production.

2. If you want to test bonding, then test it, and call pm_peers_delete() on every disconnect. It will delete all your stored peer data. This will eventually burn out your flash, but that will take a long time. Deleting a peer doesn't delete the flash. It just marks the record as a dirty record. First when all of your flash pages (except the swap page) is full it will do a garbage collection, and actually delete the flash pages. At this point, you have used 1 flash write cycle. 

3. If you bond during testing, you will end up with all of the DUTs having a stored bond in your flash, right? (Or do you intend to erase the flash on these and reprogram them afterwards?) If they contain bonding data, then they would typically advertise with a whitelist (not accepting connections from new devices). Have you concidered this?

My suggestions:

1: Don't test bonding. If it works on one device, it probably works on all.

2: If you decide to test bonding, just delete the bond information in the tester whenever you want to test a new device. This can all be done in the application, so you don't need to do anything inside the peer manager libraries. 

Best regards,

Edvin

[Cedric M]

Thank you very much, your analysis was perfect and allowed me to find the problem. You got it right:

The "record key" was indeed 0xC007 (you have to swap the bytes). From this key, I went up, as you had noticed, to the function which called somewhere to a record. It is indeed PM_PEER_DATA_ID_BONDING, I would never have found it without you since it is the concatenation of 2 constants in the code, and it is deeply hidden in a long list of function calls.
This is indeed a function of the peer manager when there is a BLE event, and according to the path it is the result of a connection with successful authentication.

this is the "pds_peer_data_store ()" function in <peer_data_storage.c> whose call path is:

ble_evt_dispatch() --> pm_on_ble_evt() --> sm_ble_evt_handler() --> smd_ble_evt_handler() --> auth_status_process() --> auth_status_success_process() --> pdb_write_buf_store() --> pdb_write_buf_store()

I therefore copied the source code of this library into my code (exclude the link in the library so as not to corrupt the other projects in the tree structure of common libraries), and I inhibited the call to this function.

To check the operation, I started with a totally erased chip, then I programmed the softdevice and the application.
I made a series of connections with 2 blank peripherals (with the central and peripherals in full bonding / passkey ....) and I can do my tests with the connection fully operational, and without any writing in the flash.

To answer your remark (1) about pairing vs bonding, I failed to get the code implemented in bonding (with passkey, whitelist, private key) to work by degrading it to pairing, I have a systematic error settings that I cannot resolve.
I too would have liked not to do bonding for the test. I hope to be able to do it later but for the moment the solution and "skipping" the flash step is operational and allowed me to understand its operation for related uses that I implement in the test-fixture.

For your remark (2) about pm_peers_delete (), I recently had the problem to delete devices in "factory" mode for customer use, but unfortunately this does not delete the whitelist data (so in the end I am also stuck). I found some posts on the forum (with replies from Nordic developpers) who knew about this problem and offered solutions to get around it by independently managing the erasure of data from the white list, but it seemed quite complicated.

For your remark (3), this is absolutely correct, it was part of my first notes at the beginning of the initial post (post 07/09/20 12:41 # 1), and it joins the remark (1). After the test, I have to erase each device, but only the data (identified!) by the code by making a direct access (not protected) in erasure. It is less annoying because it only requires one cycle on the flash of a DUT (unlike the test-fixture).

Once again a big thank you

Cedric

[Edvin]

Hello Cedric,

Glad to hear that you found the source. It was a coincidence that I found the ID = 0x07, and I had to check what it was used for. As stated earlier in the conversation, the FDS is quite heavily weaved into the peer manager, so it would take some effort to completely disable it in a smooth way.

Just in case you decide to dig further into the pairing mode:

(1): That is correct. Whitelisting is usually applied to bonded devices, however, it is possible to add it manually. Look at how the peer manager does it by default (I don't know this by heart). 

(2): similar to (1), the whitelist needs to be "reloaded" after you delete bonding data. I think it is usually done in the advertising_init() or advertising_start() functions in the examples that uses whitelisting. The ble_app_gls example is usually the peripheral example I use to look up things with pairing/bonding. The ble_app_hrs_c is the central example I use for the same.

NB: The ble_app_hrs (peripheral) example doesn't initiate pairing nor bonding, but it supports it. I don't think it uses a whitelist.

(3): Yes. If the devices are running the final FW, which supports bonding, then it obviously will contain bonding data after the test. So you need to consider that (I just wanted to make sure you were aware, but it seems like you know this stuff quite well ). You can probably do something like only erasing the FDS pages. The next time the application boots up, it will see that it needs to re-initialize the FDS pages, since they are blank. Just like in the first ever boot up. 

Best regards,

Edvin

[Cedric M]

Hello,

Yes thanks. There is always an element of luck, but there is also knowledge!
For the peer manager, I have indeed searched everywhere and I think I have flushed out all the flash accesses, my tests show that there is no longer access to the data (in writing) on ​​the flash, therefore the data always remains initialized when read, so each test starts from the same initial conditions (without degrading the test bench).

(1) I would indeed like to be able to implement pairing (without bonding). This is a point that I have failed to resolve. What I noticed is that the management of the whitelist is not trivial and is not as well managed natively by the libraries as the other steps like bonding. I have already had several problems with solutions I found on nordic dev zone (this is a problem others have had).

(2) For the removal of the whitelist, I will look in the example project you advise me (ble_app_gls), hoping to find a management of the whitelist. It is already less serious because what I implemented has the merit of having unlocked me but if it is not the cleanest (less maintainable for futur).

-> If I ever get stuck on these two technical points (which are linked), I will open a new ticket.

(3) Yes, I have already implemented the appropriate mechanisms. I had identified the flash sections dedicated to data (with or without a bootloader which shifts the positioning), and I use the function:
ble_flash_page_erase () starting from the address relative to BLE_FLASH_PAGE_END

Best Regards,

Cedric