Cant pair bluetooth peripheral with Secure connection and static passkey (Connect SDK)

I want to use the BLE Secure connection feature with the following options

CONFIG_BT_SMP_SC_PAIR_ONLY=y
CONFIG_BT_SMP_SC_ONLY=y
CONFIG_BT_SMP_APP_PAIRING_ACCEPT=y
CONFIG_BT_SMP_DISABLE_LEGACY_JW_PASSKEY=y
CONFIG_BT_FIXED_PASSKEY=y

I know that using a fixed passkey is discouraged but my peripheral will not have a display and the pairing should be done automatically. I have opted for creating a secret algorithm that generates the passkey dynamically with bt_passkey_set. The Central also knows the algorithm and can figure automatically which passkey should be sent.

Before I continue with the idea of generating dynamically the passkey with bt_passkey_set. I decided to enter a constant passkey at the beginning.

bt_passkey_set(123456);

Unfortunately when I try to pair with for example an Android Device I dont get the passkey dialog and the connection has a timeout and closes. If I do not use a static passkey with CONFIG_BT_FIXED_PASSKEY=y I get a random passkey, the BLE central gets the dialog to enter the passkey and zephyr calls the display passkey_display callback.

To Reproduce

Use any example that has a BLE peripheral and add the following Kconfig options

CONFIG_BT_SMP_SC_PAIR_ONLY=y
CONFIG_BT_SMP_SC_ONLY=y
CONFIG_BT_SMP_APP_PAIRING_ACCEPT=y
CONFIG_BT_SMP_DISABLE_LEGACY_JW_PASSKEY=y
CONFIG_BT_FIXED_PASSKEY=y

As callbacks use the following for authentication

static void auth_passkey_display(bt_conn *conn, unsigned int passkey)
{
	char addr[BT_ADDR_LE_STR_LEN];

	bt_addr_le_to_str(bt_conn_get_dst(conn), addr, sizeof(addr));

	printk("Passkey for %s: %06u\n", addr, passkey);
}

static void auth_cancel(bt_conn *conn)
{
	char addr[BT_ADDR_LE_STR_LEN];

	bt_addr_le_to_str(bt_conn_get_dst(conn), addr, sizeof(addr));

	printk("Pairing cancelled: %s\n", addr);
}

static void pairing_confirm(bt_conn *conn)
{
    bt_conn_auth_passkey_confirm(conn);
}

static bt_conn_auth_cb auth_cb_display = {
	.passkey_display = auth_passkey_display,
	.passkey_entry = nullptr,
    .cancel = auth_cancel,
    .pairing_confirm = pairing_confirm
};

In your bluetooth initialization call the following to register the callbacks and set the passkey:

bt_conn_auth_cb_register(&auth_cb_display);
bt_passkey_set(123456);


Expected behavior

Android Central device should get a passkey dialog to enter the passkey after pairing attempt with BLE Secure connection enabled on Zephyr.

If anyone could point me out what I am doing wrong API-wise or Kconfig-wise would be really helpful-
Related