Crash on connect with NCS 2.6.0

RE: Crash on connect with NCS 2.6.0

m1cha — Thu, 05 Dec 2024 05:39:09 GMT

I just tested it and it works great, thanks.

RE: Crash on connect with NCS 2.6.0

Edvin — Fri, 01 Nov 2024 08:27:58 GMT

Hello,

I have some good news! Our SoftDevice team have found the cause of this bug, and the fix is merged into our next release in v2.8.0. The issue was related to a lot of UART events being generated, which caused some issues with some internal buffers and parameters not being correctly cleared.

I believe v2.8.0 is right around the corner, so you can test the fix when it is released. (I am not certain it is part of the release candidates that are already available).

Best regards,

Edvin

RE: Crash on connect with NCS 2.6.0

Edvin — Mon, 28 Oct 2024 14:29:19 GMT

Hello,

Sorry for the late replies. Someone in our SoftDevice team is looking into this, but they have not yet figured out the source of the issue.

Just wanted to let you know that this ticket is not forgotten or ignored. I will let you know as soon as I have more details.

Best regards,

Edvin

RE: Crash on connect with NCS 2.6.0

m1cha — Tue, 22 Oct 2024 04:22:16 GMT

I did some additional testing(and have updated the repository) and have some more info to share:

- In my previous post I was testing with peripheral_hr(from my repo, not the original) but I missed that the problem disappeared from peripheral_hr_coded(from my repo, not the original). Enabling CONFIG_BT_HCI_CORE_LOG_LEVEL_DBG made the problem reappear. That further hints at a race condition.

- While changing the UART timeout from 100us to 1000us makes the problem disappear from peripheral_hr it does NOT make it disappear from peripheral_hr_coded, which is interesting to say the least. That means that the suggested workaround does not work reliably, unfortunately.

- I have never been able to reproduce this with CONFIG_BT_EXT_ADV enabled, so this might be the real workaround here. I'm a little too scared to rely on this though, given that this might just be making the race condition harder to trigger, like the other changes I made.

RE: Crash on connect with NCS 2.6.0

m1cha — Mon, 30 Sep 2024 12:24:42 GMT

All I had to do was to change the nRF SDK version in west.yml and run `west update`.

For the central, only 2.7.0 makes the bug disappear. With 2.6.0 and 2.6.2, it still happens.

RE: Crash on connect with NCS 2.6.0

Edvin — Mon, 30 Sep 2024 12:11:31 GMT

Hello,

Thanks for coming back to us. I remember this issue. I see that there is no update on my internal ticket, but I will ping it. Can you please let me know whether you needed to do any changes to the application to reproduce it using 2.7.0? And just out of curiousity - Does it matter if the central is using 2.7.0 as well? Still reproducible? I am not saying that it is a fix to make sure that both are 2.7.0, but it can be a place to start to know whether it is reproducible when the central is 2.7.0 or not.

Best regards,

Edvin

RE: Crash on connect with NCS 2.6.0

m1cha — Fri, 27 Sep 2024 08:22:17 GMT

I can still reproduce the issue (of the peripheral crashing) if I use 2.6.0(or 2.6.2) for the central and 2.7.0 for the peripheral. That means a slight difference in the behavior of the central just makes it less likely for the issue to appear, but it's not fixed yet, unfortunately.

RE: Crash on connect with NCS 2.6.0

Edvin — Mon, 13 May 2024 11:30:34 GMT

Hello,

Edit:

After talking to our SoftDevice team, they said that they are not able to reproduce the issue on the current main branch (which will turn into 2.7.0 at some point), but they are not yet sure what caused the issue. They are still looking into it. In the meantime, for further development, you can increase the uart inactivity timeout from 100µs to 1000µs, until we find the cause for the issue.

Best regards,

Edvin

RE: Crash on connect with NCS 2.6.0

Edvin — Fri, 10 May 2024 11:52:42 GMT

Sorry. I forgot to mention, but I also changed your timeout for the UART. Try to increase it from 100 to at least 1000. I would also recommend that you change the memory[100] buffer with a buffer defined outside your work function, so it is valid after this work function returns.

I am not sure why it fails when the timeout is 100µs. I have forwarded this to our SoftDevice controller team.

Best regards,

Edvin

RE: Crash on connect with NCS 2.6.0

m1cha — Fri, 10 May 2024 04:38:47 GMT

1) It's interesting that it worked for you because for me, it doesn't. See the updated GitHub repository.

2) It's not true, that you have to provide a second buffer with uart_rx_buf_rsp. This is totally optional and only needed if you want to continue receiving after the current buffer has filled up.

RE: Crash on connect with NCS 2.6.0

Edvin — Wed, 08 May 2024 07:26:44 GMT

Hello,

Simon is out of office, and asked if I could look into your ticket.

So the MPSL issue is because of an error in the UART handler. Your UART handler was empty, so you probably didn't notice it being called, but it was being called with event->type = UART_RX_BUF_REQUEST. As you can see from uart.h:

	/**
	 * @brief Driver requests next buffer for continuous reception.
	 *
	 * This event is triggered when receiving has started for a new buffer,
	 * i.e. it's time to provide a next buffer for a seamless switchover to
	 * it. For continuous reliable receiving, user should provide another RX
	 * buffer in response to this event, using uart_rx_buf_rsp function
	 *
	 * If uart_rx_buf_rsp is not called before current buffer
	 * is filled up, receiving will stop.
	 */
	UART_RX_BUF_REQUEST,

So you need to provide a buffer via uart_rx_buf_rsp().

The reason you need to provide another buffer is that the uart is double buffered.

Try something like this:

uint8_t my_uart_buffer[100];

static void uart_callback(const struct device *const uart_dev, struct uart_event *const event,
			  void *const user_data)
{
    
    switch (event->type) {
        case UART_RX_BUF_REQUEST:
            LOG_INF("uart cb type UART_RX_BUF_REQUEST");
            uart_rx_buf_rsp(uart_dev, my_uart_buffer, sizeof(my_uart_buffer));
            break;
        default:
            LOG_INF("uart cb type %d", event->type);
            break;
    }

}

static void work_handler(struct k_work *work)
{
	ARG_UNUSED(work);

	int ret;

	ret = uart_callback_set(uart_dev, uart_callback, NULL);
	if (ret) {
		LOG_ERR("Failed to set UART driver callback: %d", ret);
		return;
	}

	//static uint8_t memory[100];
    ret = uart_rx_enable(uart_dev, my_uart_buffer, sizeof(my_uart_buffer), SYS_FOREVER_US);
	if (ret) {
		LOG_ERR("Failed to enable UART RX: %d", ret);
		return;
	}

	LOG_INF("UART initialized");
}

That worked in my case.

You may notice that I am just using the same buffer again, so this means that it will use the same buffer when the buffer is full. So if you receive data too fast, it may start overwriting the start of the buffer before you get to read it out. To work around this, either see how it is done in the peripheral_uart sample, or use a double buffer, where you change what buffer is being provided in the UART_RX_BUF_REQUEST event.

Best regards,

Edvin

RE: Crash on connect with NCS 2.6.0

m1cha — Thu, 02 May 2024 04:15:54 GMT

first: We don't need a workaround, because we are willing to wait a little bit for a proper fix. We simply don't want to be stuck with 2.5.1 forever(where the bug doesn't exist).

I implemented your suggested change and pushed it to GitHub. It did not make a difference:

[00:00:00.012,939] <inf> main: BT ready
[00:00:00.037,963] <dbg> bt_hci_core: bt_recv: buf 0x200187ec len 68
[00:00:00.037,994] <dbg> bt_hci_core: rx_work_handler: Getting net_buf from queue
[00:00:00.038,024] <dbg> bt_hci_core: rx_work_handler: buf 0x200187ec type 1 len 68
[00:00:00.038,024] <dbg> bt_hci_core: hci_event: event 0x3e
[00:00:00.038,055] <dbg> bt_hci_core: hci_le_meta_event: subevent 0x08
[00:00:00.038,085] <dbg> bt_ecc: bt_hci_evt_le_pkey_complete: status: 0x00
[00:00:01.000,488] <inf> main: UART initialized
[00:00:05.054,046] <dbg> bt_hci_core: bt_recv: buf 0x200187ec len 33
[00:00:05.055,511] <err> os: ***** BUS FAULT *****
[00:00:05.055,511] <err> os:   Imprecise data bus error
[00:00:05.055,541] <err> os: r0/a1:  0x0014043e  r1/a2:  0x200125bf  r2/a3:  0x00000006
[00:00:05.055,572] <err> os: r3/a4:  0x0014043f r12/ip:  0x00000003 r14/lr:  0x00004e27
[00:00:05.055,572] <err> os:  xpsr:  0xa1000000
[00:00:05.055,572] <err> os: Faulting instruction address (r15/pc): 0x0000bb74
[00:00:05.055,633] <err> os: >>> ZEPHYR FATAL ERROR 26: Unknown error on CPU 0
[00:00:05.055,664] <err> os: Current thread: 0x20012958 (MPSL Work)

RE: Crash on connect with NCS 2.6.0

Simonr — Tue, 30 Apr 2024 13:28:27 GMT

Okay, the BUS error seems to come from the SoftDevice handler, and after some discussing with colleagues, we are back at the UART likely being the blocker here and it stops the SoftDevice controller from handling the MTU exchange and callbacks correctly. I reproduced it here as well, but have not had time to test this out yet on my end. Try moving uart_rx_enable() into a workqueue as is done in the peripheral_uart sample, with the following:

k_work_init_delayable(&uart_work, uart_work_handler);

static void uart_work_handler(struct k_work *item)
{
    struct uart_data_t *buf;
 
    buf = k_malloc(sizeof(*buf));
    if (buf) {
        buf->len = 0;
    } else {
        LOG_WRN("Not able to allocate UART receive buffer");
        k_work_reschedule(&uart_work, UART_WAIT_FOR_BUF_DELAY);
        return;
    }
 
    uart_rx_enable(uart, buf->data, sizeof(buf->data), UART_WAIT_FOR_RX);
}

Best regards,

Simon

RE: Crash on connect with NCS 2.6.0

m1cha — Tue, 30 Apr 2024 07:33:19 GMT

And some more questions I forgot to answer 🙃

> Hmm, is there a reason you're using this Zephyr advertising API on your end? I don't see it being a great fit with the SoftDevice controller.

I see it the other way around: We don't need and don't want to use extended advertising due to incompatibility with certain centrals. There is no reason for us to enable or use that API and it might save flash size(which not an important reason though).

> It could be MTU related it seems, so could it be that the MTU sizes are invalid when disabling the extended advertising config and trying to enable the UART service?

I don't know, but isn't that handled by zephyr internally? The only thing we can do is to set some limits using kconfigs and accept or reject a centrals change request using one of the connection callbacks.

RE: Crash on connect with NCS 2.6.0

m1cha — Tue, 30 Apr 2024 07:21:39 GMT

Sry, that I missed your question about addr2line. There is no line information for that closed-source function, but it's from nrfxlib/mpsl/lib/cortex-m4/soft-float/libmpsl.a and this is the relevant objdump:

0000bb5c <sym_DQONLUECJTIEYFOFJXXAPJO4POIAJKJNKBGVN5A>:
    bb5c:       ea81 0300       eor.w   r3, r1, r0
    bb60:       079b            lsls    r3, r3, #30
    bb62:       e92d 41f0       stmdb   sp!, {r4, r5, r6, r7, r8, lr}
    bb66:       d15d            bne.n   bc24 <sym_DQONLUECJTIEYFOFJXXAPJO4POIAJKJNKBGVN5A+0xc8>
    bb68:       0787            lsls    r7, r0, #30
    bb6a:       d05f            beq.n   bc2c <sym_DQONLUECJTIEYFOFJXXAPJO4POIAJKJNKBGVN5A+0xd0>
    bb6c:       2a00            cmp     r2, #0
    bb6e:       d057            beq.n   bc20 <sym_DQONLUECJTIEYFOFJXXAPJO4POIAJKJNKBGVN5A+0xc4>
    bb70:       4603            mov     r3, r0
    bb72:       e001            b.n     bb78 <sym_DQONLUECJTIEYFOFJXXAPJO4POIAJKJNKBGVN5A+0x1c>
    bb74:       2a00            cmp     r2, #0
    bb76:       d053            beq.n   bc20 <sym_DQONLUECJTIEYFOFJXXAPJO4POIAJKJNKBGVN5A+0xc4>

RE: Crash on connect with NCS 2.6.0

Simonr — Mon, 29 Apr 2024 14:02:47 GMT

Hmm, is there a reason you're using this Zephyr advertising API on your end? I don't see it being a great fit with the SoftDevice controller. I'm sorry, but I have not had time to set this up on my end yet but will try to do so tomorrow. It could be MTU related it seems, so could it be that the MTU sizes are invalid when disabling the extended advertising config and trying to enable the UART service?

I still think using the command window in your build folder and the following command to get a file name and line number for that address could be helpful, so please give it a try in the meantime: arm-none-eabi-addr2line -e _build/zephyr/zephyr.elf 0xbb74.

Best regards,

Simon

RE: Crash on connect with NCS 2.6.0

m1cha — Fri, 26 Apr 2024 11:00:19 GMT

I've updated the code on the linked repository(please read the commit messages)

- I have copied peripheral_hr_coded. This by itself doesn't crash

- I have disabled the coded phy, changed the advertising name and removed the ext flag from the advertising options(but still using the ext API). Still doesn't crash

- I have initialized UART DMA. Still doesn't crash

- I have switched to zephyrs non ext advertising API. Still doesn't crash

- I have disabled CONFIG_BT_EXT_ADV. Now it crashes.

I tested a few more permutations and it looks like it's mostly important that DMA UART is initialized and that CONFIG_BT_EXT_ADV is disabled. To me, this still looks like a softdevice_controller bug rather than a usage bug.

RE: Crash on connect with NCS 2.6.0

Simonr — Thu, 25 Apr 2024 07:35:36 GMT

I'm very sorry, for some reason I thought uart_rx_enable was enabling the radio peripheral for some reason, but I see I got way ahead of myself. I'm very sorry about the confusion here.

Taking another look at the sample you're using I see you're basing your sample on the peripheral_hr sample, which is a Zephyr project, not tested with the nRF SoftDevice controller, so that might be the issue here, although I agree it's very strange it only propagates when uart_rx_enable and the SoftDevice controller is used.

Please compare your project with the peripheral_hr_coded which is designed and tested for the nRF SoftDevice controller.

Bus faults are a bit tricky to debug, but you can try to check Faulting instruction address (r15/pc): 0x0000bb74 for what address this line is pointing to, you can use the command window in your build folder and the following command to get a file name and line number for that address: arm-none-eabi-addr2line -e _build/zephyr/zephyr.elf 0xbb74.

Best regards,

Simon

RE: Crash on connect with NCS 2.6.0

m1cha — Tue, 23 Apr 2024 11:38:55 GMT

Hi Simon,

1) Moving uart_rx_enable to after bt_enable didn't fix the issue.

2) Which sample do you mean? nrf/samples/bluetooth/peripheral_uart calls uart_rx_enable before bt_enable, too.

3) Since we are heavily relying on zephyrs driver/device model, and initialization(and deinitialization) of Bluetooth and UART happen independently and based on external factors, changing the order is not an option for us.

Thanks

Michael

RE: Crash on connect with NCS 2.6.0

Simonr — Tue, 23 Apr 2024 09:18:13 GMT

It's clear that the peripheral is what is crashing here. Based on your description I think the uart_rx_enable() is being enabled before you try to enable BLE with bt_enable(). Please see how uart_rx_enable() is used in our sample projects (as part of uart_init()).

Best regards,

Simon