BUS FAULT (use-after-free) in att.c on disconnect - Precise data bus error

Hi, 

My central device is running into Bus fault when continuously sending data to multiple connected peripheral device and any of peripheral device got disconneced. Issue can be reproduce when only one or two device conneced with central. 

Central is sending 20 bytes data packet to peripheral at interval of same as connection interval (188ms) and it can send upto 4 packets per connection event to every connected peripheral device. 

Environment

  • nRF Connect SDK version: Found on v3.3.0, but reproduced identically on v3.3.1 as well
  • SoC: nRF52840

Connection parameter used between central and peripheral

  • Connection interval: 188ms
  • Slave latency: 0
  • Supervision timeout: 8000ms

Below is the error logs at time of fault,

[00:06:52.180,572] <err> os: ***** BUS FAULT *****
[00:06:52.180,969] <err> os: Precise data bus error
[00:06:52.181,365] <err> os: BFAR Address: 0x701315dd
[00:06:52.181,854] <err> os: r0/a1: 0x200287f4 r1/a2: 0x00000000 r2/a3: 0x00000000
[00:06:52.182,464] <err> os: r3/a4: 0x00000000 r12/ip: 0x2001a5dc r14/lr: 0x00043d4b
[00:06:52.183,074] <err> os: xpsr: 0x21000000
[00:06:52.183,502] <err> os: Faulting instruction address (r15/pc): 0x0006f022
[00:06:52.184,051] <err> os: >>> ZEPHYR FATAL ERROR 25: Unknown error on CPU 0
[00:06:52.184,600] <err> os: Current thread: 0x2000f748 (sysworkq)

I run the add2line on zephyr.elf and found following files,

arm-zephyr-eabi-addr2line -e zephyr.elf -f -C 0x0006f022
C:/ncs/v3.3.0/zephyr/include/zephyr/sys/slist.h:213

arm-zephyr-eabi-addr2line -e zephyr.elf -f -C 0x00043d4b
C:/ncs/v3.3.0/zephyr/subsys/bluetooth/host/att.c:775

To debug the issue, I have enabled the few more debug logs and find out that, host is try to send the data even if the link is disconnected. At application level, I am making sure that app is not calling bt_gatt_write_without_response_cb if connection object is NULL or invalid. Setting connection object invalid when receive disconnection event.

[00:03:20.082,092] <dbg> bt_conn: bt_conn_set_state: disconnect-complete <<-- disconnected
[00:03:20.082,122] <dbg> bt_conn: bt_conn_tx_processor: start
[00:03:20.082,153] <dbg> bt_conn: bt_conn_tx_processor: no connection wants to do stuff
[00:03:20.082,214] <dbg> bt_conn: tx_notify_process: conn 0x20007038
[00:03:20.082,275] <dbg> bt_conn: bt_conn_set_state: trigger disconnect work
[00:03:20.082,305] <dbg> bt_conn: bt_conn_unref: handle 6 ref 3 -> 2
[00:03:20.082,366] <dbg> bt_conn: deferred_work: conn 0x20007038
[00:03:20.082,427] <dbg> bt_att: bt_att_disconnected: chan 0x20028b64 cid 0x0004
[00:03:20.082,458] <dbg> bt_att: att_chan_detach: chan 0x20028b60
[00:03:20.082,580] <dbg> bt_att: bt_att_released: chan 0x20028b60 <<-- channel released here
[00:03:20.066,406] <dbg> bt_conn: bt_conn_unref: handle 21 ref 3 -> 2
[00:03:20.084,442] <inf> usbd_cdc_acm: tx_en: trigger irq_cb_work
[00:03:20.084,625] <dbg> bt_conn: bt_conn_unref: handle 6 ref 2 -> 1
[00:03:20.084,655] <dbg> bt_conn: bt_conn_unref: handle 6 ref 1 -> 0
[00:03:20.084,686] <dbg> bt_att: att_tx_destroy_work_handler: 0x20039288 
[00:03:20.084,716] <dbg> bt_att: att_on_sent_cb: opcode 0x52
[00:03:20.084,747] <dbg> bt_att: att_on_sent_cb: UATT bearer, calling bt_att_sent: conn 0x20028cc0 chan 0x20028b64
[00:03:20.084,777] <dbg> bt_att: bt_att_sent: chan 0x20028b60 <<-- Data sent on already released channel??
[00:03:20.084,808] <err> os: ***** BUS FAULT *****
[00:03:20.084,808] <err> os: Precise data bus error
[00:03:20.084,838] <err> os: BFAR Address: 0x8a1a9443
[00:03:20.084,869] <err> os: r0/a1: 0x00000000 r1/a2: 0x00000000 r2/a3: 0x00000000
[00:03:20.084,899] <err> os: r3/a4: 0x00000000 r12/ip: 0x00000010 r14/lr: 0x000715ad
[00:03:20.084,899] <err> os: xpsr: 0xa1000000
[00:03:20.084,930] <err> os: Faulting instruction address (r15/pc): 0x00047016
[00:03:20.084,960] <err> os: >>> ZEPHYR FATAL ERROR 25: Unknown error on CPU 0
[00:03:20.085,021] <err> os: Current thread: 0x2000faa0 (sysworkq)
[00:03:20.884,613] <err> os: Halting system

Would you please help me to fix this issue? 

Thanks,

Narendra

Parents
  • Hello,

    This is likely a race condition issue in the Bluetooth ATT layer where data is being sent on an already-released channel after disconnection.

    I can pass it along to the team, but I did ask AI to take a look at the race condition, and it came up with a suggested fix, can you give it a go before I ask the team?

    Try to replace: \zephyr\subsys\bluetooth\host\att.c

    (removed since it didn't work)

    Rebuild the project with west build --pristine

    Thanks,
    Kenneth

  • Hi Kenneth,

    I have replaced the att.c file with the new fix, but the issue remains unresolved. The device is still experiencing the same fault.

    Below is the log for the same,

    [00:01:49.406,768] <dbg> bt_conn: bt_conn_tx_processor: pop: cb (nil) userdata (nil)
    [00:01:49.406,799] <dbg> bt_conn: bt_conn_tx_processor: TX process: conn 0x20008400 buf 0x2003a0a8 (last)
    [00:01:49.406,829] <dbg> bt_conn: send_buf: conn 0x20008400 buf 0x2003a0a8 len 27 buf->len 27 cb (nil) ud (nil)
    [00:01:49.406,829] <dbg> bt_conn: conn_tx_alloc: 0x2001b7b8[0m
    [00:01:49.406,860] <dbg> bt_conn: get_data_frag: get-acl-frag: outside 0x2003a0a8 window 0x20035ef8 size 27
    [00:01:49.406,860] <dbg> bt_conn: send_buf: send frag: buf 0x20035ef8 len 27
    [00:01:49.406,890] <dbg> bt_conn: send_buf: conn 0x20008400 buf 0x20035ef8 len 27 flags 0x02
    [00:01:49.406,951] <dbg> bt_conn: frag_destroy: 
    [00:01:49.406,951] <dbg> bt_conn: bt_conn_unref: handle 1 ref 4 -> 3
    [00:01:49.407,012] <dbg> bt_conn: bt_conn_set_state: trigger disconnect work
    [00:01:49.407,043] <dbg> bt_conn: bt_conn_unref: handle 2 ref 4 -> 3
    [00:01:49.407,073] <dbg> bt_conn: bt_conn_tx_processor: start
    [00:01:49.407,104] <dbg> bt_conn: should_stop_tx: 0x20008400
    [00:01:49.407,104] <dbg> bt_conn: bt_conn_ref: handle 1 ref 3 -> 4
    [00:01:49.407,135] <dbg> bt_conn: bt_conn_tx_processor: processing conn 0x20008400
    [00:01:49.407,165] <dbg> bt_conn: bt_conn_data_ready: DR
    [00:01:49.407,165] <dbg> bt_conn: bt_conn_ref: handle 1 ref 4 -> 5
    [00:01:49.407,196] <dbg> bt_conn: bt_conn_unref: handle 1 ref 5 -> 4
    [00:01:49.407,226] <dbg> bt_conn: bt_conn_data_ready: Connection 0x20008400 already in conn_ready list
    [00:01:49.407,257] <dbg> bt_conn: bt_conn_tx_processor: pop: cb (nil) userdata (nil)
    [00:01:49.407,287] <dbg> bt_conn: bt_conn_tx_processor: TX process: conn 0x20008400 buf 0x2003a080 (last)
    [00:01:49.407,318] <dbg> bt_conn: send_buf: conn 0x20008400 buf 0x2003a080 len 27 buf->len 27 cb (nil) ud (nil)
    [00:01:49.407,318] <dbg> bt_conn: conn_tx_alloc: 0x2001b7c4[0m
    [00:01:49.407,348] <dbg> bt_conn: get_data_frag: get-acl-frag: outside 0x2003a080 window 0x20035ef8 size 27
    [00:01:49.407,379] <dbg> bt_conn: send_buf: send frag: buf 0x20035ef8 len 27
    [00:01:49.407,379] <dbg> bt_conn: send_buf: conn 0x20008400 buf 0x20035ef8 len 27 flags 0x02
    [00:01:49.407,440] <dbg> bt_conn: frag_destroy: 
    [00:01:49.407,440] <dbg> bt_conn: bt_conn_unref: handle 1 ref 4 -> 3
    [00:01:49.407,470] <dbg> bt_att: att_tx_destroy_work_handler: 0x2003a0a8
    [00:01:49.407,501] <dbg> bt_att: att_on_sent_cb: opcode 0x52
    [00:01:49.407,531] <dbg> bt_att: att_on_sent_cb: UATT bearer, calling bt_att_sent: conn 0x20008400 chan 0x200297f4
    [00:01:49.407,531] <dbg> bt_att: bt_att_sent: chan 0x200297f0
    [00:01:49.407,562] <dbg> bt_att: chan_req_notif_sent: chan 0x200297f0 CID 0x0004
    [00:01:49.407,592] <dbg> bt_conn: bt_conn_tx_processor: start
    [00:01:49.407,623] <dbg> bt_conn: should_stop_tx: 0x20008400
    [00:01:49.407,623] <dbg> bt_conn: bt_conn_ref: handle 1 ref 3 -> 4
    [00:01:49.407,653] <dbg> bt_conn: bt_conn_tx_processor: processing conn 0x20008400
    [00:01:49.407,684] <dbg> bt_conn: bt_conn_data_ready: DR
    [00:01:49.407,714] <dbg> bt_conn: bt_conn_ref: handle 1 ref 4 -> 5
    [00:01:49.407,745] <dbg> bt_conn: bt_conn_unref: handle 1 ref 5 -> 4
    [00:01:49.407,775] <dbg> bt_conn: bt_conn_data_ready: Connection 0x20008400 already in conn_ready list
    [00:01:49.407,836] <dbg> bt_conn: bt_conn_tx_processor: pop: cb (nil) userdata (nil)
    [00:01:49.407,867] <dbg> bt_conn: bt_conn_tx_processor: TX process: conn 0x20008400 buf 0x2003a030 (last)
    [00:01:49.407,897] <dbg> bt_conn: send_buf: conn 0x20008400 buf 0x2003a030 len 27 buf->len 27 cb (nil) ud (nil)
    [00:01:49.407,928] <dbg> bt_conn: conn_tx_alloc: 0x2001b410
    [00:01:49.407,958] <dbg> bt_conn: get_data_frag: get-acl-frag: outside 0x2003a030 window 0x20035ef8 size 27
    [00:01:49.407,989] <dbg> bt_conn: send_buf: send frag: buf 0x20035ef8 len 27
    [00:01:49.408,020] <dbg> bt_conn: send_buf: conn 0x20008400 buf 0x20035ef8 len 27 flags 0x02
    [00:01:49.408,081] <dbg> bt_conn: frag_destroy: 
    [00:01:49.408,111] <dbg> bt_conn: bt_conn_unref: handle 1 ref 4 -> 3
    [00:01:49.408,142] <dbg> bt_conn: deferred_work: conn 0x200084d8
    [00:01:49.408,203] <dbg> bt_att: bt_att_disconnected: chan 0x200298a4 cid 0x0004
    [00:01:49.408,203] <dbg> bt_att: att_chan_detach: chan 0x200298a0
    [00:01:49.408,355] <dbg> bt_att: bt_att_released: chan 0x200298a0
    [00:01:49.408,599] <dbg> bt_conn: bt_conn_tx_processor: start
    [00:01:49.408,630] <dbg> bt_conn: should_stop_tx: 0x20008400
    [00:01:49.408,660] <dbg> bt_conn: get_conn_ready: appending 0x20008400 to back of TX queue
    [00:01:49.408,660] <dbg> bt_conn: bt_conn_data_ready: DR
    [00:01:49.408,691] <dbg> bt_conn: bt_conn_ref: handle 1 ref 3 -> 4
    [00:01:49.408,752] <dbg> bt_conn: bt_conn_data_ready: Connection 0x20008400 added to conn_ready list
    [00:01:49.408,782] <dbg> bt_conn: bt_conn_tx_processor: processing conn 0x20008400
    [00:01:49.408,813] <dbg> bt_conn: bt_conn_tx_processor: pop: cb (nil) userdata (nil)
    [00:01:49.408,843] <dbg> bt_conn: bt_conn_tx_processor: TX process: conn 0x20008400 buf 0x2003a008 (last)
    [00:01:49.408,905] <dbg> bt_conn: send_buf: conn 0x20008400 buf 0x2003a008 len 27 buf->len 27 cb (nil) ud (nil)
    [00:01:49.408,905] <dbg> bt_conn: conn_tx_alloc: 0x2001b41c
    [00:01:49.408,935] <dbg> bt_conn: get_data_frag: get-acl-frag: outside 0x2003a008 window 0x20035ef8 size 27
    [00:01:49.408,966] <dbg> bt_conn: send_buf: send frag: buf 0x20035ef8 len 27
    [00:01:49.408,996] <dbg> bt_conn: send_buf: conn 0x20008400 buf 0x20035ef8 len 27 flags 0x02
    [00:01:49.409,057] <dbg> bt_conn: frag_destroy: 
    [00:01:49.409,088] <dbg> bt_conn: bt_conn_unref: handle 1 ref 4 -> 3
    [00:01:49.409,454] <dbg> bt_conn: bt_conn_tx_processor: start
    [00:01:49.409,484] <dbg> bt_conn: get_conn_ready: no LL bufs for 0x20008328
    [00:01:49.409,515] <dbg> bt_conn: get_conn_ready: no LL bufs for 0x200084d8
    [00:01:49.409,545] <dbg> bt_conn: get_conn_ready: no LL bufs for 0x20008400
    [00:01:49.409,545] <dbg> bt_conn: bt_conn_tx_processor: no connection wants to do stuff
    [00:01:49.409,790] <dbg> bt_conn: bt_conn_tx_processor: start
    [00:01:49.409,820] <dbg> bt_conn: get_conn_ready: no LL bufs for 0x20008328
    [00:01:49.409,851] <dbg> bt_conn: get_conn_ready: no LL bufs for 0x200084d8
    [00:01:49.409,881] <dbg> bt_conn: get_conn_ready: no LL bufs for 0x20008400
    [00:01:49.409,881] <dbg> bt_conn: bt_conn_tx_processor: no connection wants to do stuff
    [00:01:49.410,156] <dbg> bt_conn: bt_conn_tx_processor: start
    [00:01:49.410,186] <dbg> bt_conn: get_conn_ready: no LL bufs for 0x20008328
    [00:01:49.410,217] <dbg> bt_conn: get_conn_ready: no LL bufs for 0x200084d8
    [00:01:49.410,247] <dbg> bt_conn: get_conn_ready: no LL bufs for 0x20008400
    [00:01:49.410,247] <dbg> bt_conn: bt_conn_tx_processor: no connection wants to do stuff
    [00:01:49.411,163] <dbg> bt_conn: bt_conn_unref: handle 2 ref 3 -> 2
    [00:01:49.411,193] <dbg> bt_conn: bt_conn_unref: handle 2 ref 2 -> 1
    [00:01:49.411,224] <dbg> bt_att: att_tx_destroy_work_handler: 0x2003a080
    [00:01:49.411,254] <dbg> bt_att: att_on_sent_cb: opcode 0x52
    [00:01:49.411,285] <dbg> bt_att: att_on_sent_cb: UATT bearer, calling bt_att_sent: conn 0x20008400 chan 0x200297f4
    [00:01:49.411,315] <dbg> bt_att: bt_att_sent: chan 0x200297f0
    [00:01:49.411,346] <dbg> bt_att: chan_req_notif_sent: chan 0x200297f0 CID 0x0004
    [00:01:49.411,437] <dbg> bt_att: att_tx_destroy_work_handler: 0x2003a030
    [00:01:49.411,468] <dbg> bt_att: att_on_sent_cb: opcode 0x52
    [00:01:49.411,499] <dbg> bt_att: att_on_sent_cb: UATT bearer, calling bt_att_sent: conn 0x20008400 chan 0x200297f4
    [00:01:49.411,529] <dbg> bt_att: bt_att_sent: chan 0x200297f0
    [00:01:49.411,560] <dbg> bt_att: chan_req_notif_sent: chan 0x200297f0 CID 0x0004
    [00:01:49.411,712] <dbg> bt_att: att_tx_destroy_work_handler: 0x2003a148
    [00:01:49.411,773] <dbg> bt_att: att_on_sent_cb: opcode 0x52
    [00:01:49.411,773] <dbg> bt_att: att_on_sent_cb: UATT bearer, calling bt_att_sent: conn 0x20029a00 chan 0x200298a4
    [00:01:49.411,804] <dbg> bt_att: bt_att_sent: chan 0x200298a0
    [00:01:49.411,834] <err> os: ***** BUS FAULT *****
    [00:01:49.411,865] <err> os:   Precise data bus error
    [00:01:49.411,865] <err> os:   BFAR Address: 0x1100b2c
    [00:01:49.411,895] <err> os: r0/a1:  0x00000000  r1/a2:  0x00000000  r2/a3:  0x00000000[0m
    [00:01:49.411,926] <err> os: r3/a4:  0x00000000 r12/ip:  0x00000010 r14/lr:  0x00070da1
    [00:01:49.411,956] <err> os:  xpsr:  0x21000000
    [00:01:49.411,956] <err> os: Faulting instruction address (r15/pc): 0x00046702
    [00:01:49.412,017] <err> os: >>> ZEPHYR FATAL ERROR 25: Unknown error on CPU 0
    [00:01:49.412,048] <err> os: Current thread: 0x200112a0 (sysworkq)
    [00:01:52.747,283] <err> os: Halting system

    Thanks,

    Narendra

  • Hello,

    Okey, I will check with the team. Do you have a project so we can recreate this here?

    Kenneth

  • Looks like the fix is working. No bus fault detected during testing. 

    Thanks. 
    Narendra

Reply Children
Related