Zephyr bt_conn_unref() not consistently working

RE: Zephyr bt_conn_unref() not consistently working

Ryjan — Thu, 24 Aug 2023 19:45:07 GMT

I've come back enlightened. I was bouncing between a couple of projects and now had time to dive deeper into this one.

The issue is now resolved so I will explain what I have learned. Reference counting is something that I did not fully understand. I was thinking that it was something more specific to the module, but it turns out that it is a more general concept. To sum it up, an object (in this case the connection object) exists in a library layer abstracted away from me the application programmer. The connection is handled without my need to manage it, but I have to interact with it. When I need to interact with it, such as sending or receiving a message, I get a pointer to it. In the documentation under "Connection Management" it says-

"Connection objects are reference counted, and the application is expected to use the bt_conn_ref() API whenever storing a connection pointer for a longer period of time, since this ensures that the object remains valid (even if the connection would get disconnected). Similarly the bt_conn_unref() API is to be used when releasing a reference to a connection."

To me it sounded like bt_conn_ref() and bt_conn_unref() are opposites this turned out to not be true. In the code examples for a central I noticed that br_conn_ref() is not used. The function bt_conn_le_create() is where the first counting reference shows up. My initial assumption that bt_conn_ref() and bt_conn_unref() have a 1:1 relationship is incorrect. Any function that gives you the connection reference increments the reference counter for that reference. These aren't listed in the overview paragraph, but they can be found in each function's definition. So if you aren't looking for it's easy to miss. It would be nice if in the definition of bt_conn_unref() it listed the functions that will require it's use.

Ultimately the mistake in my code was after the usage of bt_conn_lookup_addr_le(). I don;t think that this was shown in any of the example code, but it is necessary for sending a message originating from the central device to a particular peripheral device. In my case, this central needs to connect to 4 peripherals and route the messages correctly. I have to look up the connection using the MAC address. That gets me the connection pointer and increments the reference counter. As soon as I write to an attribute I have to call bt_conn_unref(). Her's is what I ended up writing:

central_send_error_t central_send_tx(uint8_t * p_mac, uint8_t type, uint8_t * p_data, uint16_t len)
{
	int err = 0;
	bt_addr_le_t bt_addr_le;
	memcpy(&bt_addr_le.a.val, p_mac, BT_ADDR_SIZE);
	bt_addr_le.type = type;
	struct bt_conn * p_conn = bt_conn_lookup_addr_le(BT_ID_DEFAULT, &bt_addr_le);
	if(p_conn == NULL){
		LOG_ERR("Invalid MAC address: %02x:%02x:%02x:%02x:%02x:%02x", bt_addr_le.a.val[0], bt_addr_le.a.val[1], bt_addr_le.a.val[2], bt_addr_le.a.val[3], bt_addr_le.a.val[4], bt_addr_le.a.val[5]);
		bt_conn_unref(p_conn);
		return CEN_SND_BAD_MAC;
	}
	LOG_HEXDUMP_DBG(p_data, len, "Packet: ");
	LOG_DBG("Send on handle 0x%04x", m_tx_value_handle);
	// LOG_HEXDUMP_INF(p_mac, BT_ADDR_SIZE, "MAC: ");
	err = bt_gatt_write_without_response_cb(p_conn, m_tx_value_handle, p_data, len, false, _write_finished_cb, 0);
	if(err){
		LOG_ERR("Write error: %d", err);
		bt_conn_unref(p_conn);
		return;
	}
	bt_conn_unref(p_conn);
	return CEN_SND_NO_ERROR;
}

Before I was looking up the connection, but never calling bt_conn_unref(), so the reference count was going up and up with every communication. I had a work around that was calling it if found an existing connection, but it would only decrement it by on each time it failed. So the longer I was connected, the more messages were written. It would take longer to unref the connection.

The connection management summary says to get a connection reference "whenever storing a connection pointer for a longer period of time." This was vague, but I had assumed that could be for the duration of the connection. That combined with the acquisition of a connection reference with every write unknown to me was causing my issue.

RE: Zephyr bt_conn_unref() not consistently working

Ryjan — Thu, 27 Jul 2023 16:51:09 GMT

Thank you, I look forward to reading through that.

RE: Zephyr bt_conn_unref() not consistently working

Ryjan — Thu, 27 Jul 2023 16:50:42 GMT

Thanks, I've been slowly reading through the code. Thanks for the link too I may find more information there. I'll keep digging. It's very odd that eventually is will get rid of the connection after many bt_conn_unref() calls. I'll report what I find.

RE: Zephyr bt_conn_unref() not consistently working

Susheel Nuguru — Wed, 26 Jul 2023 21:12:06 GMT

The implementation of bt_conn_ref is vendor independent. It is pure C code that is very well documented in the https://github.com/zephyrproject-rtos/zephyr/blob/zephyr-v3.4.0/subsys/bluetooth/host/conn.c#L1289C50-L1289C50

conn is a pointer to struct and have a long atomic_t member with name ref. Instead of just returning the connection pointer, the call to bt_conn_ref increments the counter ref in a safe and atomic way.

in bt_conn_unref, the ref is decremented safely and when ref is equal to zero, then it is assumed no contexts need the conn object (normally in disconnect callback) and this unref also starts the advertiser.

I think all this is very clear from the c code in a very readable format and this part of the source code have no dependency on the vendor implementation of the stack.

RE: Zephyr bt_conn_unref() not consistently working

Simonr — Wed, 26 Jul 2023 13:27:17 GMT

I've asked the team, and the author of the mentioned article for some more specifics on what you're looking for. I'll get back to you when I hear from them.

Best regards,

Simon

RE: Zephyr bt_conn_unref() not consistently working

Ryjan — Tue, 25 Jul 2023 17:31:19 GMT

I do not expect support support for systems not implemented by Nordic but I would like to find more information on the conn.c module. The above quoted article refers to ownership and how reference counting is supposed to simplify this. I would like to find more information on that to try to become more of an expert on how it works.

Am I mistaken that the Bluetooth implementation was written by Nordic?

RE: Zephyr bt_conn_unref() not consistently working

Simonr — Mon, 24 Jul 2023 09:17:53 GMT

In the connection management API here you can read more on how the bt_conn_ref() works: https://developer.nordicsemi.com/nRF_Connect_SDK/doc/latest/zephyr/connectivity/bluetooth/api/connection_mgmt.html#c.bt_conn_ref

So are you just implementing the Zephyr RTOS on your own then? We will not be able to support you if you move outside of the SDKs we are familiar with and provide support for I'm afraid.

Best regards,

Simon

RE: Zephyr bt_conn_unref() not consistently working

Ryjan — Thu, 20 Jul 2023 16:21:26 GMT

Thank you for some explanation. Where can I go to read more on how the bt_conn_ref() works. I do not have a firm grasp on what is going on. So far I understand that I get a pointer to a connection reference, but the only explanation in the comments is about it incrementing and decrementing a counter. I would like to know where to look to learn about ownership and references particularly about their application to connection references in the Bluetooth stack.

To answer your question about my version of NCS, I am not using NCS. I'm trying to maintain greater portability across device. But my issues are with understanding the implementation of the Bluetooth stack.

RE: Zephyr bt_conn_unref() not consistently working

Simonr — Thu, 20 Jul 2023 08:06:17 GMT

One of our developers made a short article on ownership and how to use the bt_conn_ref and unref functions. Please give it a read and see if that helps you understand:

"Ownership and references are a complex topic. It can be limited a bit and made simpler by reifying ownership as a pointer value in a variable, which I recommend. After all, reference counting is supposed to make resource management easier, not harder. The following is a sketch of the technique.

Declare variables as only-for-owned-references by use of code comments. Their value shall be NULL when not owing a reference. Function parameters are indeed variables that may be declared as only-for-owned-references. A function return value is also variable-like for this purpose.

Owning references must be conserved. When moving a owned reference from a variable, immediately when able set that variable's value to NULL to reify the move. Consider using an atomic exchange operation.

When obtaining ownership trough a return value, as soon as possible reify that by assigning the returned pointer value to a variable that is only-for-owned-references. Make sure the variable was NULL before to avoid losing a reference! Consider using an atomic exchange operation.

Owned references should never be forgotten. Forgetting an owned reference is a resource leak. The "source" of that leak, we vaguely define as the missing logic that would conserve the reference appropriately. This also applies to failure-handling code!

When an owning variable is about to go out of scope, you may assert that its value is NULL. It should always be that if we follow the rules above.

Keep in mind the ownership semantics for any API you use is. For example, `bt_conn_ref` entrusts you with a reference for safekeeping, and `bt_conn_ref` moves a reference into the host where it is no longer the applications job to keep it safe. Other API may have similar, or more complicated semantics. Some API moves the ownership only sometimes (e.g. on success, which is indicated by return code).

It's interesting to note that `bt_conn_ref` seems to duplicate a reference. But this is not possible in the abstract sense I have in mind when I say "reference". The reference itself is not a number or pointer or anything, so there is nothing meaningful to copy. Instead, `bt_conn_ref` operates outside of this "reference" safety net, and the most consistent way of thinking about this is that `bt_conn_ref` has a large supply of references that point to the object you want, that it can hand out. And it want's them back! (..through `bt_conn_unref`.)"

You say you've been using nRF Connect SDK version v0.15.2, is this a typo? Since this is not a real NCS version. When building a sample in the nRF Connect SDK, you will get the Zephyr version used stated at the beginning of the build log for example:

Building peripheral_uart
C:\WINDOWS\system32\cmd.exe /d /s /c "west build --build-dir c:/ncs/v2.4.0/nrf/samples/bluetooth/peripheral_uart/build c:/ncs/v2.4.0/nrf/samples/bluetooth/peripheral_uart"

[0/1] Re-running CMake...
Loading Zephyr default modules (Zephyr base (cached)).
-- Application: C:/ncs/v2.4.0/nrf/samples/bluetooth/peripheral_uart
-- CMake version: 3.20.5
-- Cache files will be written to: C:/ncs/v2.4.0/zephyr/.cache
-- Zephyr version: 3.3.99 (C:/ncs/v2.4.0/zephyr)

Best regards,

Simon