nRF52811 BLE hangs or crashes after central disconnects with UART enabled

RE: nRF52811 BLE hangs or crashes after central disconnects with UART enabled

ovrebekk — Tue, 04 Jul 2023 07:42:46 GMT

Hi Harold

Good to hear you discovered the issue. Thanks for sharing your findings 🙂

I must admit I wasn't aware that the disconnected callback would be triggered separately for every handle in your attribute table.

Best regards
Torbjørn

RE: nRF52811 BLE hangs or crashes after central disconnects with UART enabled

keegretupmoc — Mon, 03 Jul 2023 09:11:44 GMT

Hi,

It's been a while since I was on. Been doing a lot of different things and thought I would go through my various tickets and see if I have any answers for them.

On this one, I have figured it out. The issue with the 52811 crashing when a BT connected peripheral disconnects is due to insufficient stack size on the work queue. It was originally set to only 256 bytes and it appears to be used to unwind the BT services and characteristics. The hang/crashes never gave me stack overflow and maybe that is because there was so much data that it whacked everything out of shape. But I would see only up to 4 of the cb's getting handled when there were at least 28 and probably more and other things to be dealt with on the disconnect. After that, the disconnected callback would be called and I would be able to do things:

00> [01:58:42.914,550] <inf> bt_gatt: disconnected_cb: passed in attr: 0x24d68, handle: 1, and ud: 0x200017e0
00> [01:58:42.915,740] <inf> bt_gatt: disconnected_cb: passed in attr: 0x24d7c, handle: 2, and ud: 0x200017e0
00> [01:58:42.916,870] <inf> bt_gatt: disconnected_cb: passed in attr: 0x24d90, handle: 3, and ud: 0x200017e0
00> [01:58:42.918,060] <inf> bt_gatt: disconnected_cb: passed in attr: 0x24da4, handle: 4, and ud: 0x200017e0
00> [01:58:42.919,189] <inf> bt_gatt: disconnected_cb: passed in attr: 0x24db8, handle: 5, and ud: 0x200017e0
00> [01:58:42.920,379] <inf> bt_gatt: disconnected_cb: passed in attr: 0x24dcc, handle: 6, and ud: 0x200017e0
00> [01:58:42.921,508] <inf> bt_gatt: disconnected_cb: passed in attr: 0x24978, handle: 7, and ud: 0x200017e0
00> [01:58:42.922,698] <inf> bt_gatt: disconnected_cb: passed in attr: 0x2498c, handle: 8, and ud: 0x200017e0
00> [01:58:42.923,828] <inf> bt_gatt: disconnected_cb: passed in attr: 0x249a0, handle: 9, and ud: 0x200017e0
00> [01:58:42.925,018] <inf> bt_gatt: disconnected_cb: passed in attr: 0x249b4, handle: 10, and ud: 0x200017e0
00> [01:58:42.926,147] <inf> bt_gatt: disconnected_cb: passed in attr: 0x249c8, handle: 11, and ud: 0x200017e0
00> [01:58:42.927,337] <inf> bt_gatt: disconnected_cb: passed in attr: 0x249dc, handle: 12, and ud: 0x200017e0
00> [01:58:42.928,527] <inf> bt_gatt: disconnected_cb: passed in attr: 0x249f0, handle: 13, and ud: 0x200017e0
00> [01:58:42.929,656] <inf> bt_gatt: disconnected_cb: passed in attr: 0x24a04, handle: 14, and ud: 0x200017e0
00> [01:58:42.930,847] <inf> bt_gatt: disconnected_cb: passed in attr: 0x24a18, handle: 15, and ud: 0x200017e0
00> [01:58:42.932,006] <inf> bt_gatt: disconnected_cb: passed in attr: 0x24a2c, handle: 16, and ud: 0x200017e0
00> [01:58:42.933,166] <inf> bt_gatt: disconnected_cb: passed in attr: 0x24a40, handle: 17, and ud: 0x200017e0
00> [01:58:42.934,326] <inf> bt_gatt: disconnected_cb: passed in attr: 0x24a54, handle: 18, and ud: 0x200017e0
00> [01:58:42.935,516] <inf> bt_gatt: disconnected_cb: passed in attr: 0x24a68, handle: 19, and ud: 0x200017e0
00> [01:58:42.936,645] <inf> bt_gatt: disconnected_cb: passed in attr: 0x24a7c, handle: 20, and ud: 0x200017e0
00> [01:58:42.937,835] <inf> bt_gatt: disconnected_cb: passed in attr: 0x24a90, handle: 21, and ud: 0x200017e0
00> [01:58:42.939,025] <inf> bt_gatt: disconnected_cb: passed in attr: 0x24aa4, handle: 22, and ud: 0x200017e0
00> [01:58:42.940,155] <inf> bt_gatt: disconnected_cb: passed in attr: 0x24ab8, handle: 23, and ud: 0x200017e0
00> [01:58:42.941,345] <inf> bt_gatt: disconnected_cb: passed in attr: 0x24acc, handle: 24, and ud: 0x200017e0
00> [01:58:42.942,504] <inf> bt_gatt: disconnected_cb: passed in attr: 0x24ae0, handle: 25, and ud: 0x200017e0
00> [01:58:42.943,695] <inf> bt_gatt: disconnected_cb: passed in attr: 0x24af4, handle: 26, and ud: 0x200017e0
00> [01:58:42.944,824] <inf> bt_gatt: disconnected_cb: passed in attr: 0x24b08, handle: 27, and ud: 0x200017e0
00> [01:58:42.946,014] <inf> bt_gatt: disconnected_cb: passed in attr: 0x24b1c, handle: 28, and ud: 0x200017e0

I had to increase the work queue to 1024 to get it to stop crashing. And running the thread analyzer showed that I was using upwards of 80% of the space when it wasn't crashing at 1k. But that's the most it will to, so that's probably ok. It's such a little device... And that sent the SRAM from 91% to 97% utilized as I had to adjust a couple other stacks that were getting close to 100%.

Anyway, thanks for your help.

You can close this ticket.

RE: nRF52811 BLE hangs or crashes after central disconnects with UART enabled

ovrebekk — Thu, 09 Mar 2023 08:39:22 GMT

Hi Harold

Fair point, the thread will eat up a bit of RAM as well. You can adjust the stack size through one of the config parameters, if you are careful about stack use in the callback function.

[quote user="keegretupmoc"]We did disable the assert (config_assert=n) in prj.conf which looks like recovered about 23k of flash space. That is just for debugging, correct? You don't want that on for production? [/quote]

You are correct. The idea is to use this during development to detect and identify runtime errors etc, and then disable it in production to improve performance and reduce the memory footprint. The documentation for the config has a bit more context.

[quote user="keegretupmoc"]I am not completely abandoning the new UART code, just putting it on hold for now until we get FOTA working on the 52811. We really have to get that working. It is a top priority.[/quote]

Sounds like a good decision, having a stable FOTA solution is pretty critical 🙂

Best regards
Torbjørn

RE: nRF52811 BLE hangs or crashes after central disconnects with UART enabled

keegretupmoc — Wed, 08 Mar 2023 19:45:30 GMT

Torbjørn,

I was more concerned about the additional thread.

We did disable the assert (config_assert=n) in prj.conf which looks like recovered about 23k of flash space. That is just for debugging, correct? You don't want that on for production? I checked on the 9160 side and it isn't enabled or even listed in the prj.conf. I didn't really see any articles on it...

I am not completely abandoning the new UART code, just putting it on hold for now until we get FOTA working on the 52811. We really have to get that working. It is a top priority. So once we get that going, we will see about implementing the code. Since you mentioned that it should be in the 2.2.0 version of the sdk, that will work well for us. We are planning at some point to upgrade to sdk 2.2.0 (currently on 2.0.0).

At some point in the future if we redesign our device, we will select a BLE chip with more flash/ram...

Thanks,

RE: nRF52811 BLE hangs or crashes after central disconnects with UART enabled

ovrebekk — Wed, 08 Mar 2023 09:04:10 GMT

Hi Harold

Keep in mind that the app_uart driver doesn't use dynamic memory allocation, which means if you don't need this anywhere else you can reduce the size of the heap, or remove it altogether. This should give you some extra RAM.

You can also configure the size of the static buffers through the configuration parameters, to reduce RAM consumption of the driver itself.

The best of luck with the memory optimization task!

Best regards
Torbjørn

RE: nRF52811 BLE hangs or crashes after central disconnects with UART enabled

keegretupmoc — Wed, 08 Mar 2023 07:03:03 GMT

Torbjørn,

I've been looking at your UART code. I am probably going to wait a while before trying it. I don't think I have enough memory at the moment. I need to open another ticket to try to reduce the size. Here's where we are currently. This is again on the 52811 BLE processor:

Memory region Used Size Region Size %age Used
FLASH: 173112 B 192 KB 88.05%
SRAM: 24014 B 24 KB 97.71%
IDT_LIST: 0 GB 2 KB 0.00%

Thanks,

RE: nRF52811 BLE hangs or crashes after central disconnects with UART enabled

keegretupmoc — Wed, 08 Mar 2023 02:05:15 GMT

Torbjørn,

I will look at the new UART code and see if it helps. If it reduces the footprint and complexity, every little bit helps.

On the setting the logging to immediate, it didn't reduce the crashing, it stopped the hanging. if you remember, originally the iOS device disconnecting from bluetooth would result in either a crash, or the BLE would hang. With the immediate display flag set to y, the hanging stopped happening. So each time we disconnected from the BLE, it crashed. So nothing was reduced, just the behavior changed. And it was more clear that it was in the sysworkq, as I was seeing that every time.

Let me know if you want anymore data on the crashing with immediate enabled, or if there is something else to check when that is running.

Thanks,

RE: nRF52811 BLE hangs or crashes after central disconnects with UART enabled

ovrebekk — Tue, 07 Mar 2023 14:36:22 GMT

Hi Harold

Thanks for sharing the TX code, it makes sense that this would fail if you tried to free a static buffer.

You don't have a similar code snippet for the disconnected case?

[quote user="keegretupmoc"]The other issue I am starting to work on is to try to reduce the footprint of the 52811 to get mcuboot in there. Again, probably another ticket?[/quote]

Yes, this is not really related to this issue and should definitely be a separate ticket. The devzone is not really designed to handle multiple different issues in the same ticket.

[quote user="keegretupmoc"]So not sure how this changes the behavior. When I set it back to n, and start doing disconnects, I see it hang again and the WDT is resetting it. So something changes with that being enabled. Hopefully that gives us a hint or something.[/quote]

It's odd that enabling immediate logging would lead to less crashes, usually it is the other way around since immediate logging will try to write the log output right away, even when you are inside an interrupt. Normal logging will buffer the log message in RAM and have it printed at a later time, outside interrupt context.

The fact that the current thread is the sysworkq is a bit interesting, this implies that the crash happens inside a work item scheduled from the application. Do you have any log output to go with this crash?

Another request I have is if you could set aside an hour or so to test my UART wrapper and see if it solves your issues.

Normally I wouldn't insist you test this driver before I have had time to integrate it in the official SDK, but if it does work it should save you significant time going forward.

The driver and accompanying samples are available here.

Included in the repo you will also find a port of the peripheral_uart sample, replacing the original UART async code with the app_uart driver. This change alone reduced the size of main.c by almost 300 lines of code thanks to the significantly simpler interface it provides.

Initialization looks like this, receiving data and forwarding it to the BLE stack looks like this, and forwarding data from the BLE stack to the UART is done like this.

The repo has been updated for v2.2.0 of the SDK, but I tested the peripheral_uart sample in V2.0.0 as well and it seems to work fine.

If you do decide to give this driver a go and have any issues getting it running just let me know, and I will do my best to help. Obviously you should be able to run this both on the nRF9160 and the nRF52811 side.

Best regards
Torbjørn

RE: nRF52811 BLE hangs or crashes after central disconnects with UART enabled

keegretupmoc — Tue, 07 Mar 2023 03:17:29 GMT

One thing I forgot to mention. When you had me set CONFIG_LOG_MODE_IMMEDIATE=y to see if any additional information might be printed when I disconnect the BT. When I had first enabled it, it seemed like the BLE started crashing. I think this was before I had even try to connect the iOS app. So I turned it off. I just turned it back on, and it's not crashing the BLE (52811) outright, but the behavior of the disconnect hang/crash seems to have changed when that parameter is enabled:

It seems to crash differently with it enabled. The disconnect still causes a crash, but it seems like it's in a different place and it doesn't hang anymore. I have tried it about 20 times and I would think I would have at least 25% of the be a hang. But there have been no hangs. Here's what the crash looks like and it seems similar:

10> [00:00:05.018,951] <err> os: ***** USAGE FAULT *****
10> [00:00:05.019,409] <err> os: Illegal use of the EPSR
10> [00:00:05.019,897] <err> os: r0/a1: 0x200022bc r1/a2: 0x00021d83 r2/a3: 0x00000000
10> [00:00:05.020,568] <err> os: r3/a4: 0x00000000 r12/ip: 0x00026aa0 r14/lr: 0x20004adc
10> [00:00:05.021,240] <err> os: xpsr: 0x20004a00
10> [00:00:05.021,697] <err> os: Faulting instruction address (r15/pc): 0x01000000
10> [00:00:05.022,308] <err> os: >>> ZEPHYR FATAL ERROR 0: CPU exception on CPU 0
10> [00:00:05.022,888] <err> os: Current thread: 0x20001df0 (sysworkq)
10> [00:00:05.028,350] <err> fatal_error: Resetting system[0m

So not sure how this changes the behavior. When I set it back to n, and start doing disconnects, I see it hang again and the WDT is resetting it. So something changes with that being enabled. Hopefully that gives us a hint or something.

But no additional info from it, though since the behavior changes, it's hard to know if there was still something we weren't seeing...

Thanks,

RE: nRF52811 BLE hangs or crashes after central disconnects with UART enabled

keegretupmoc — Tue, 07 Mar 2023 01:54:37 GMT

Torbjørn,

Yes, the callback code is a conglomeration of the initial sample peripheral_uart code as well as some examples, etc. and then some tweaking on my part as a result of a lot of testing. It works pretty good, except for that strangeness where the UART pipe from the 9160 appears to hang up for several seconds so that 4 messages get put into one message that then overflows into several full message buffers.

The code that generates the BLE_connecd message is:

#ifdef CONFIG_NRFX_UARTE0

LOG_INF("Sending BLE_connected, Ptr: %p", &theMsg);

if (uart_tx(uart, theMsg, strlen(theMsg), SYS_FOREVER_MS))

{

LOG_WRN("uart_tx_done: Failed to send data over UART");

}

#endif // CONFIG_NRFX_UARTE0

On the UART_TX_DONE section that was commented out, I was starting to see if I could communicate from the BLE (52811) back to the 9160, and I was getting hung up in UART_TX_DONE that was crashing the BLE processor. I think what was happening was that I was calling uart_tx() like in the above, and when it ended up in UART_TX_DONE, it was trying to free the pointer connected to the string I think, which was not originally malloc'd, that that was bad.

Now one of the things we need to do (not related to the above issue) is to be able to send messages form the BLE to the 9160. So if I need some of the code in UART_TX_DONE to do that, please let me know. I can also create another ticket on that specific issue. The other issue I am starting to work on is to try to reduce the footprint of the 52811 to get mcuboot in there. Again, probably another ticket?

Please advise.

Thanks for your help.

RE: nRF52811 BLE hangs or crashes after central disconnects with UART enabled

ovrebekk — Mon, 06 Mar 2023 12:57:24 GMT

Hi Harold

If you use the JLink Viewer then you are using the RTT backend, as opposed to the UART backend which is the most common.

I can't spot anything in the code, it seems you have copied most of it from the original example.

I do notice that you removed all of the code from the UART_TX_DONE callback?
Is there any particular reason for that?

Looking at your earlier log it includes the following:

peripheral_uart: Sending BLE_connected, Ptr: 0x20003e74

Do you have a similar message sent when you disconnect?

Could you send me the code you use to send these messages (BLE_connected, and similar for disconnected if you have one)?

Best regards
Torbjørn

RE: nRF52811 BLE hangs or crashes after central disconnects with UART enabled

keegretupmoc — Sat, 04 Mar 2023 04:38:19 GMT

Torbjørn,

Not sure what you mean by backend. I am using a couple of segger's to interface with the 9160 and 52811. And the JLink Viewer to see the log information. I did try setting the config log mode immediate to y. I crashed it several times, but didn't see anything more with it.

I did implement the WDT so that when it hangs it will reset the processor. That works pretty well. I have it set to 15 seconds and feed it every 10 seconds. I've tested the hang over 30 times and every time it was reset anywhere from 5 to 15 seconds after the disconnect. So that will bandaid the situation while we figure out what's going on.

As far as the uart code, here's the callback:

#ifdef CONFIG_NRFX_UARTE0
static void uart_cb(const struct device *dev, struct uart_event *evt, void *user_data)
{
/*
LOG_INF("\n\n\n%p %s %s %s %s entered uart_cb callback: type %d\n",
k_current_get(),
k_thread_name_get(k_current_get()),
k_is_in_isr() ? "ISR" : "THREAD",
k_is_pre_kernel() ? "preK" : "postK",
k_is_preempt_thread() ? "preempt" : "nopreempt",
evt->type);
*/

ARG_UNUSED(dev);

static uint8_t *current_buf;
static size_t aborted_len;
static bool buf_release, disable_req;
struct uart_data_t *buf;
static uint8_t *aborted_buf;
//static uint8_t theBuf[21];

switch (evt->type) {

case UART_TX_DONE:
LOG_INF("uart_tx_done: len: %d", evt->data.tx.len);

/*
if ((evt->data.tx.len == 0) ||
(!evt->data.tx.buf)) {
LOG_INF("uart_tx_done: no evt data buf or len 0, exiting");
return;
}

if (aborted_buf) {
buf = CONTAINER_OF(aborted_buf, struct uart_data_t, data);
aborted_buf = NULL;
aborted_len = 0;
} else {
buf = CONTAINER_OF(evt->data.tx.buf, struct uart_data_t, data);
}

LOG_INF("uart_tx_done: freeing buf: %p", buf);
k_free(buf);

buf = k_fifo_get(&fifo_uart_tx_data, K_NO_WAIT);
LOG_INF("uart_tx_done: buf %p from fifo queue", buf);
if (!buf) {
LOG_INF("uart_tx_done: no buf, exiting");
return;
}

LOG_INF("uart_tx_done: data |%*s| len %d sent to uart_tx(...)", buf->len, buf->data, buf->len);
if (uart_tx(uart, buf->data, buf->len, SYS_FOREVER_MS)) {
LOG_WRN("uart_tx_done: Failed to send data over UART");
}
*/

break;

case UART_RX_RDY:
// LOG_INF("uart_rx_rdy: len %d", evt->data.rx.len);

buf = CONTAINER_OF(evt->data.rx.buf, struct uart_data_t, data);
buf->len += evt->data.rx.len;

// LOG_INF("uart_rx_rdy: buf %p", buf);

// LOG_INF("uart_rx_rdy: buf %p, %d bytes |%s|", buf, buf->len, buf->data);

if (disable_req) {
// LOG_INF("uart_rx_rdy: disable_req set, exiting");
return;
}

if ((evt->data.rx.buf[buf->len - 1] == '\n') ||
(evt->data.rx.buf[buf->len - 1] == '\r')) {
disable_req = true;
// LOG_INF("uart_rx_rdy: set disable_req to true and call uart_rx_disable for %p", buf);
uart_rx_disable(uart);
}

break;

case UART_RX_DISABLED:
// LOG_INF("uart_rx_disabled: buf wants %d bytes, buf is %p", sizeof(*buf), buf);

disable_req = false;

buf = k_malloc(sizeof(*buf));
if (buf) {
buf->len = 0;
// LOG_INF("uart_rx_disabled: after k_malloc with buf %p", buf);
} else {
LOG_WRN("uart_rx_disabled: Not able to allocate UART receive buffer");
k_work_reschedule(&uart_work, UART_WAIT_FOR_BUF_DELAY);
return;
}

// LOG_INF("uart_rx_disabled: calling uart_rx_enable with buf %p", buf);
uart_rx_enable(uart, buf->data, sizeof(buf->data),
UART_WAIT_FOR_RX);

break;

case UART_RX_BUF_REQUEST:
// LOG_INF("uart_rx_buf_request: buf wants %d bytes, buf is %p", sizeof(*buf), buf);

buf = k_malloc(sizeof(*buf));
if (buf) {
buf->len = 0;
// LOG_INF("uart_rx_buf_request: after k_malloc, calling uart_rx_buf_rsp with buf %p", buf);
uart_rx_buf_rsp(uart, buf->data, sizeof(buf->data));
} else {
LOG_WRN("uart_rx_buf_request: Not able to allocate UART receive buffer");
}

break;

case UART_RX_BUF_RELEASED: // section that needs the debug logic to work //
// LOG_INF("uart_rx_buf_released: for evt %p, %d bytes, buf %p",
// evt, sizeof(*evt), CONTAINER_OF(evt->data.rx_buf.buf, struct uart_data_t, data));

buf = CONTAINER_OF(evt->data.rx_buf.buf, struct uart_data_t, data);

///*
if (buf->len > 0) {
// LOG_INF("uart_rx_buf_released: fifo %p put |%s| for %d bytes", buf, buf->data, buf->len);
k_fifo_put(&fifo_uart_rx_data, buf);
} else {
// LOG_INF("uart_rx_buf_released: freeing buf %p, %d bytes", buf, buf->len);
k_free(buf);
}
//*/

//if (!buf->len) k_free(buf);/

break;

case UART_TX_ABORTED:
// LOG_INF("uart_tx_aborted:");

if (!aborted_buf) {
aborted_buf = (uint8_t *)evt->data.tx.buf;
}

aborted_len += evt->data.tx.len;
buf = CONTAINER_OF(aborted_buf, struct uart_data_t,
data);

LOG_INF("uart_tx_aborted: calling uart_tx(...)");
uart_tx(uart, &buf->data[aborted_len],
buf->len - aborted_len, SYS_FOREVER_MS);

break;

case UART_RX_STOPPED:
LOG_INF("uart_rx_stopped: reason %d", evt->data.rx_stop.reason);
break;

default:
LOG_INF("evt type %d on default:", evt->type);
break;
}
}

static void uart_work_handler(struct k_work *item)
{
LOG_INF("uart_work_handler: callback:");
struct uart_data_t *buf;

LOG_INF("uart_work_handler: buf wants %d bytes, buf is %p", sizeof(*buf), buf);

buf = k_malloc(sizeof(*buf));
if (buf) {
buf->len = 0;
} else {
LOG_WRN("uart_work_handler: not able to allocate UART receive buffer");
k_work_reschedule(&uart_work, UART_WAIT_FOR_BUF_DELAY);
return;
}

LOG_INF("uart_work_handler: calling uart_rx_enable:");
uart_rx_enable(uart, buf->data, sizeof(buf->data), UART_WAIT_FOR_RX);
}

static int uart_init(void)
{
int err;
struct uart_data_t *rx;

LOG_INF("uart_init: %p %s %s %s %s calling device_get_binding",
k_current_get(),
k_thread_name_get(k_current_get()),
k_is_in_isr() ? "ISR" : "THREAD",
k_is_pre_kernel() ? "preK" : "postK",
k_is_preempt_thread() ? "preempt" : "nopreempt");

uart = device_get_binding(DT_LABEL(DT_NODELABEL(uart0)));
if (!uart) {
return -ENXIO;
}

rx = k_malloc(sizeof(*rx));
if (rx) {
rx->len = 0;
LOG_INF("uart_init: k_malloc %d bytes for intial rx uart buffer %p", sizeof(*rx), rx);
} else {
return -ENOMEM;
}

LOG_INF("uart_init: calling k_work_init_delayable");

k_work_init_delayable(&uart_work, uart_work_handler);

LOG_INF("uart_init: calling uart_callback_set to start uart_cb callback");

err = uart_callback_set(uart, uart_cb, NULL);
if (err) {
return err;
}

LOG_INF("uart_init: calling uart_rx_enable for first time as we exit uart_init");
return uart_rx_enable(uart, rx->data, sizeof(rx->data), UART_WAIT_FOR_RX);
}
#endif // CONFIG_NRFX_UARTE0

Let me know if you see anything. I have the buffer set to about 50 bytes. And I am trying not to send any messages from the 9160 over that and end all messages with \r\n which should get caught in uart_rx_rdy where we then call uart_rx_disable(). And then in uart_rx_buf_released we put the buffer that is a length greater than 0 in the fifo queue. If the length is zero we free the buffer.

Another thread sits on the fifo queue doing a get forever until there is something there and processes the message.

As I said in a. previous message, there is some kind of delay that cause 4 or 5 seconds of messages on the 9160 side to come over ran together on the BLE side and so we have a few message buffers that are full and not terminated with \r\n which causes some difficulty. I have some code that checks the format and if it is incorrect it discards the data.

Thanks,

RE: nRF52811 BLE hangs or crashes after central disconnects with UART enabled

ovrebekk — Wed, 01 Mar 2023 11:13:58 GMT

Hi Harold

Talking about logging, which backend are you using?

Typically UART is used for logging, but possibly you are using a different backend since the UART is used in the application?

Logging will be deferred by default, which means that log messages will be stored in a buffer, and printed later from a thread. Console (printk) is not, and will be printed right away, even if called from interrupt context.

In order to make logging work the same way you can add the following configuration to your project:

CONFIG_LOG_MODE_IMMEDIATE=y

Then the log calls will be blocking, and the code execution should not proceed until the log output is complete.
Could you try to make this change and see if the log output changes when the crash occurs?

Best regards
Torbjørn

RE: nRF52811 BLE hangs or crashes after central disconnects with UART enabled

keegretupmoc — Tue, 28 Feb 2023 20:19:07 GMT

Torbjørn,

The logging may very well be deferred. I have thought on a number of occasions that it may behave that way, but I am not that familiar with its various operational modes. If you have something for me to change so it is more responsive that would be great. I also see it skipping messages if a lot come through at one time. So, as you say, it may be possible that there are more statements before we hang or crash.

As far as the uart_rx_disable, it is separate from the BT disconnect. So I wouldn't think that would be a factor. We don't disable the 9160 UART messaging when a device is not connected. Also, as I had said before, I did test where no messages were being sent from the 9160 over the UART, so although everything was initialized, there was no message traffic and it didn't seem to matter.

As far as things going on with the UART before disconnect, there isn't a specific time or place with the UART, it could be sending, or not, or not having any data coming across. It doesn't seem to matter. And on the BLE side, just a simple disconnect after a simple connect (no notification, etc) will cause this.

I do think if we can get the jlink messages not deferred that would be good. I see some times where it seems like the messages are buffered up and display in spike a couple of seconds apart, etc.

I will try to provide the rest of the info from your message soon.

Thanks,

RE: nRF52811 BLE hangs or crashes after central disconnects with UART enabled

ovrebekk — Tue, 28 Feb 2023 08:38:04 GMT

HI Harold

Is the logging deferred or direct?
Have you tried to change the logging in the UART async handler to use console/printk or non deferred logging, in order to make sure that all the messages go through before the crash?

If the logging is deferred (which is default) we might very well be missing some of the last messages that occur before the crash.

For instance, your log doesn't show any indication that the uart_rx_disable request is happening before the crash, but I have earlier seen issues with this code related to the disable request. Knowing the flow of your program, does it seem reasonable that a Bluetooth disconnect would lead to a RX disable request?

Do you send anything over the UART when the disconnect happens?

Would you be able to attach the async handler code so I can have a look at it?
If you don't want to share your code in a public ticket just let me know, and I will make the ticket private.

As a side note I have been working on a more streamlined wrapper for the UART async driver that removes the need to handle the async handler directly, and makes all the temporary buffers static rather than relying on heap. With this wrapper you essentially just gave an init function, a transmit function, and a callback providing RX messages. I haven't gotten the time to get it properly integrated in the SDK yet, but if you want to have a look at it I made it available here.

Best regards
Torbjørn

RE: nRF52811 BLE hangs or crashes after central disconnects with UART enabled

keegretupmoc — Mon, 27 Feb 2023 18:51:35 GMT

Torbjørn,

There have been a lot of changes. I don't think it is related to a memory leak or something like that from allocating buffers and not releasing them. In my testing, I have tried to isolate the various routines that are processing the UART messages from the 9160 side to the BLE side including not starting the thread on the 9160 side that calls uart_tx to send the messages over to the 52811. And also the routine that processes the messages on the BLE side after the callback has taken the message and put it in the fifo queue (so we only recover the message from the queue, then release the buffer).

The message buffer size is 50 and both sides are set up async. There is only one call to the RDY callback. So I don't think it's related to the buffers. In getting the UART "working" over the months, I can tell when memory runs out and have a LOG_INF where we allocate the buffer in the DISABLE section and also print out the pointer address, which doesn't change over time, so I know we aren't leaking there because the address doesn't change except for the other buffer that is allocated.

As I said, I have also tried not transmitting any data from the 9160 side, so from the BLE side, it only starts up the UART by calling uart_rx_enable, but never receives any data from the 9160 side. But it sill hangs/crashes on disconnect from a BT device. It's only when I have the CONFIG_NRFX_UARTE0 turned off that it doesn't have any issues. It also happens right away if I disconnect right after the BLE side has reset, so no time to run out of memory. I also thought it was when you have notifications enabled, as the iOS app does automatically, but it happens even when it is not enabled.

I just used my android with a BLE scanner on it to connect and then disconnect a few seconds later (no notifications or access to any of the services) and it crashed. Here's the output:

10> [00:00:06.107,635] <inf> peripheral_uart: Starting to advertise
10>
10>
10>
10> [00:00:06.109,130] <inf> peripheral_uart: uart_init: 0x20002250 main THREAD postK preempt calling device_get_binding
10> [00:00:06.109,191] <inf> peripheral_uart: uart_init: k_malloc 56 bytes for intial rx uart buffer 0x20005490
10> [00:00:06.109,222] <inf> peripheral_uart: uart_init: calling k_work_init_delayable
10> [00:00:06.109,252] <inf> peripheral_uart: uart_init: calling uart_callback_set to start uart_cb callback
10> [00:00:06.109,252] <inf> peripheral_uart: uart_init: calling uart_rx_enable for first time as we exit uart_init
10> [00:00:06.116,363] <inf> peripheral_uart:
10>
10> Entered connected callback
10>
10> [00:00:06.116,699] <inf> peripheral_uart: Connected to 7C:45:CB:93:51:AD (random)
10> [00:00:06.116,729] <inf> peripheral_uart: Sending BLE_connected, Ptr: 0x20003e74
10> [00:00:06.118,804] <inf> peripheral_uart: uart_tx_done: len: 24
10> [00:00:10.539,428] <inf> peripheral_uart: uart_rx_disabled: after k_malloc with buf 0x200054d0
10> [00:00:11.613,555] <inf> peripheral_uart: uart_rx_disabled: after k_malloc with buf 0x20005490
10> [00:00:11.614,196] <inf> peripheral_uart: Move: 501 12900 10237 32767 32767 32767
10> [00:00:12.612,701] <inf> peripheral_uart: uart_rx_disabled: after k_malloc with buf 0x200054d0
10> [00:00:12.613,189] <inf> peripheral_uart: THP: 65086 1000 285 9762198
10> [00:00:13.611,083] <inf> peripheral_uart: uart_rx_disabled: after k_malloc with buf 0x20005490
10> [00:00:13.611,328] <inf> peripheral_uart: BIO: 0 0
10> [00:00:14.611,541] <inf> peripheral_uart: uart_rx_disabled: after k_malloc with buf 0x200054d0
10> [00:00:14.611,785] <inf> peripheral_uart: Alerts: 00001000
10> [00:00:15.611,450] <inf> peripheral_uart: uart_rx_disabled: after k_malloc with buf 0x20005490
10> [00:00:15.611,724] <inf> peripheral_uart: BATS: 399 85
10> [00:00:17.842,437] <inf> bt_gatt: disconnected_cb: passed in attr: 0x257bc, handle: 1, and ud: 0x20001cf8
10> [00:00:17.842,468] <inf> bt_gatt: disconnected_cb: passed in attr: 0x257d0, handle: 2, and ud: 0x20001cf8
10> [00:00:17.842,529] <err> os: ***** MPU FAULT *****
10> [00:00:17.842,559] <err> os: Instruction Access Violation
10> [00:00:17.842,590] <err> os: r0/a1: 0xffffffff r1/a2: 0x0001c4cb r2/a3: 0x00000002
10> [00:00:17.842,620] <err> os: r3/a4: 0x00027017 r12/ip: 0x00000004 r14/lr: 0x200052d8[0m
10> [00:00:17.842,620] <err> os: xpsr: 0x20005200
10> [00:00:17.842,651] <err> os: Faulting instruction address (r15/pc): 0x00000000
10> [00:00:17.842,681] <err> os: >>> ZEPHYR FATAL ERROR 0: CPU exception on CPU 0
10> [00:00:17.842,712] <err> os: Current thread: 0x200022e8 (sysworkq)
10> [00:00:18.141,235] <err> fatal_error: Resetting system

We were trying to send a message back to the 9160 which is what is happening at 6.116,729/118,804 (I have tried commenting that out, so nothing is trying to send to the 9160 but it still hangs/crashes). The k_malloc's are for the normal 9160 messages coming over. You can see the messages after the k_malloc (Move, THP, BIO, ALERTS, BATS). You can see the addresses on the malloc's aren't advancing which is why I don't think there is a memory leak. You can see the calls to disconnected_cb right before the fault which is where the device disconnected. There is a message in the disconnect callback like you see when we connect ("entered connected callback"), so we haven't called the associated disconnect callback, which is probably done last after all the handles to the services have been dealt with, which is where I think it is hanging/crashing..

There is one other oddity that bears mentioning in case you know what might be causing that. Sometimes the messages do not come through clean and nice as in the above. If you look at the timestamp for the messages you will see that they are being sent with a 1 second wait in-between on the 9160 side so they don't come in too fast. The oddity is sometimes I get several buffers on the BLE side with multiple messages in them. Since the messages are sent over a roughly 5 second period, and the timeout for uart_tx is 100ms, they must somehow get delayed on the BLE side and maybe prior to going into the callback until they come all at once and there are a couple or so of the 50 byte buffers instead of the normal less then 50 byte messages with the \r\n end of messages which we look for in UART_RX_RDY:

if ((evt->data.rx.buf[buf->len - 1] == '\n') ||

(evt->data.rx.buf[buf->len - 1] == '\r')) {

//// if (buf->len < UART_BUF_SIZE) {

disable_req = true;

// LOG_INF("uart_rx_rdy: set disable_req to true and call uart_rx_disable for %p", buf);

uart_rx_disable(uart);

This doesn't happen very often, and I have some code that mostly detects this and discards it as a corrupted message. But it does give me the impression that something is happening that is delaying the normal flow for at least 5 seconds. It also doesn't appear to happen at regular intervals, etc. so it's not a regularly occurring process. But the main issue is with the hanging/crashing. It wouldn't be as bad if it was only crashing, since this was from a device disconnect. It would probably not even be noticed. But when the BLE hangs, I have to reset it from the programmer. So in a production device, the power would need to be cycled on the entire device to get the BLE unstuck when it happens there.

I tried isolating out all that I could of the process on both sides of sending data across the UART but just the fact that it is active seems to cause the issue. I didn't try not initializing the UART on the BLE side, I guess I could try that...

Thanks,

RE: nRF52811 BLE hangs or crashes after central disconnects with UART enabled

ovrebekk — Mon, 27 Feb 2023 10:13:05 GMT

Hi Harold

Have you made a lot of changes to the example, or is it close to the original peripheral_uart code?

Put another way, would you be able to reproduce the issue with the standard example, or is it somehow related to the changes you made?

To be fair the way the example uses dynamic memory allocation for allocating UART buffers is a bit unreliable, and there might be some issues in the original example. Once I manage to reproduce it on my end I will look into it.

Best regards
Torbjørn