Our company's FMNA+FMDN dual system firmware, developed based on NCS v2.7.0 and Find My SDK v2.7.0, experienced an abnormal system crash. This version of firmware has been prepared for sample review in several hardware products. Recently, a customer made 10 products and completed firmware testing before handing them over to their users. After using them for a few days, we found that 4 of them could not boot up. We obtained 2 of the abnormal products and did not find any hardware design issues by replacing the components. After updating the version with UART logs, we obtained the following logs, which revealed the following error message:
[00027661] <inf> fp_fmdn: Fast Pair: advertising in the discoverable mode
[00028005] <err> mpsl_init: MPSL ASSERT: 112, 1617
[00028192] <err> os: ***** HARD FAULT *****
[00028357] <err> os: Fault escalation (see below)
[00028545] <err> os: ARCH_EXCEPT with reason 3
[00028723] <err> os: r0/a1: 0x00000003 r1/a2: 0x20008068 r2/a3: 0x00000000
[00028996] <err> os: r3/a4: 0x00000003 r12/ip: 0x00000000 r14/lr: 0x000584b7
[00029268] <err> os: xpsr: 0x01000018
[00029422] <err> os: Faulting instruction address (r15/pc): 0x0003fd60
[00029667] <err> os: >>> ZEPHYR FATAL ERROR 3: Kernel oops on CPU 0
[00029904] <err> os: Fault during interrupt handling
[00030098] <err> os: Current thread: 0x20005af8 (unknown)
[00030306] <err> os: Halting system
The file attachment contains more log information.
*** Booting nRF Connect SDK v2.7.0-5cb85570ca43 *** *** Using Zephyr OS v3.6.99-100befc70c74 *** [00000447] <inf> fs_nvs: 2 Sectors of 4096 bytes [00000619] <inf> fs_nvs: alloc wra: 1, ec0 [00000773] <inf> fs_nvs: data wra: 1, 6a8 load params: 8493465600, 707788800 buttons_init! btn_wake_on = 0 device_state = 0 device_state = DEV_IDLE batt_volts = 3372 batt_volts adjust = 3472 percentage_batt_lvl = 100% battery level: 100% [00001665] <inf> fp_fmdn: FMDN: setting (0x64) battery level twi read tx failed! 0xbae0001 twi read rx failed 0xbae0001 twi read tx failed! 0xbae0001 twi read rx failed 0xbae0001 twi read tx failed! 0xbae0001 twi read rx failed 0xbae0001 ACC Chip not found! Platform initialized 0x10 0xff 0x9f 0xcf 0xdc 0x54 0xf3 0xac [00003306] <inf> bt_sdc_hci_driver: SoftDevice Controller build revision: d6 da c7 ae 08 db 72 6f 2a a3 26 49 2a 4d a8 b3 |......ro *.&I*M.. 98 0e 07 7f |.... [00004298] <inf> bt_hci_core: HW Platform: Nordic Semiconductor (0x0002) [00004542] <inf> bt_hci_core: HW Variant: nRF52x (0x0002) [00004741] <inf> bt_hci_core: Firmware: Standard Bluetooth controller (0x00) Version 214.51162 Build 1926957230 [00005119] <inf> bt_hci_core: No ID address. App must call settings_load() [00005429] <inf> bt_hci_core: Identity[0]: D0:66:BE:04:67:9D (random) [00005667] <inf> bt_hci_core: Identity[1]: C3:81:A3:C2:D2:92 (random) [00005903] <inf> bt_hci_core: HCI: version 5.4 (0x0d) revision 0x11fb, manufacturer 0x0059 [00006200] <inf> bt_hci_core: LMP: version 5.4 (0x0d) subver 0x11fb Bluetooth initialized Starting the FMNA application [00006642] <inf> fmna: FMN pairing information is not present in the storage [00006983] <dbg> fmna: Advertising TX power set to 4 dBm [00007218] <dbg> fmna: Initializing FMN State to: Unpaired The FMN accessory transitioned to the unpaired state fmna_paired change to 0 [00007718] <inf> fmna: SW UUID: ed 56 19 8b 23 c7 49 14 ba 55 48 8c 13 86 6c 7c |.V..#.I. .UH...l| [00008194] <inf> fmna: SW Authentication Token (base64 format): [00008414] <inf> fmna: MYG/ME8CAQECAQEE (... 228 more chars ...) PkOgjHwrgyaleCE= [00008742] <inf> fmna: Serial Number: 61 30 31 31 38 32 66 30 36 35 30 34 36 37 39 64 |a01182f0 6504679d [00009168] <inf> fmna: Application firmware version: v2.2.3 [00009373] <wrn> fmna: The FMN stack is configured for qualification [00009613] <wrn> fmna: The qualification configuration should not be used for production FMNA initialized [00009963] <inf> fp_fmdn: Starting the FMDN application [00010155] <inf> fp_fmdn: FMDN initialized *** Booting nRF Connect SDK v2.7.0-5cb85570ca43 *** *** Using Zephyr OS v3.6.99-100befc70c74 *** [00000445] <inf> fs_nvs: 2 Sectors of 4096 bytes [00000617] <inf> fs_nvs: alloc wra: 1, eb0 [00000772] <inf> fs_nvs: data wra: 1, 6d8 load params: 8493465600, 707788800 buttons_init! btn_wake_on = 1 device_state = 0 batt_volts = 3366 batt_volts adjust = 3466 percentage_batt_lvl = 100% battery level: 100% [00001597] <inf> fp_fmdn: FMDN: setting (0x64) battery level twi read tx failed! 0xbae0001 twi read rx failed 0xbae0001 twi read tx failed! 0xbae0001 twi read rx failed 0xbae0001 twi read tx failed! 0xbae0001 twi read rx failed 0xbae0001 ACC Chip not found! Platform initialized 0x10 0xff 0x9f 0xcf 0xdc 0x54 0xf3 0xac [00003238] <inf> bt_sdc_hci_driver: SoftDevice Controller build revision: d6 da c7 ae 08 db 72 6f 2a a3 26 49 2a 4d a8 b3 |......ro *.&I*M.. 98 0e 07 7f |.... [00004230] <inf> bt_hci_core: HW Platform: Nordic Semiconductor (0x0002) [00004473] <inf> bt_hci_core: HW Variant: nRF52x (0x0002) [00004672] <inf> bt_hci_core: Firmware: Standard Bluetooth controller (0x00) Version 214.51162 Build 1926957230 [00005051] <inf> bt_hci_core: No ID address. App must call settings_load() [00005365] <inf> bt_hci_core: Identity[0]: D0:66:BE:04:67:9D (random) [00005603] <inf> bt_hci_core: Identity[1]: FD:5B:40:B1:3C:51 (random) [00005837] <inf> bt_hci_core: HCI: version 5.4 (0x0d) revision 0x11fb, manufacturer 0x0059 [00006136] <inf> bt_hci_core: LMP: version 5.4 (0x0d) subver 0x11fb Bluetooth initialized Starting the FMNA application [00006579] <inf> fmna: FMN pairing information is not present in the storage [00006922] <dbg> fmna: Advertising TX power set to 4 dBm [00007154] <dbg> fmna: Initializing FMN State to: Unpaired The FMN accessory transitioned to the unpaired state fmna_paired change to 0 [00007660] <inf> fmna: SW UUID: ed 56 19 8b 23 c7 49 14 ba 55 48 8c 13 86 6c 7c |.V..#.I. .UH...l| [00008141] <inf> fmna: SW Authentication Token (base64 format): [00008362] <inf> fmna: MYG/ME8CAQECAQEE (... 228 more chars ...) PkOgjHwrgyaleCE= [00008694] <inf> fmna: Serial Number: 61 30 31 31 38 32 66 30 36 35 30 34 36 37 39 64 |a01182f0 6504679d [00009122] <inf> fmna: Application firmware version: v2.2.3 [00009328] <wrn> fmna: The FMN stack is configured for qualification [00009568] <wrn> fmna: The qualification configuration should not be used for production FMNA initialized [00009918] <inf> fp_fmdn: Starting the FMDN application [00010110] <inf> fp_fmdn: FMDN initialized [00023220] <inf> fp_fmdn: FMDN: setting (0x64) battery level [00023455] <inf> fp_fmdn: Fast Pair: RPA expired [00023632] <inf> fp_fmdn: Fast Pair: setting RPA timeout to 751 [s] [00023883] <wrn> bt_hci_core: opcode 0x202e status 0x01 [00024087] <err> bt_id: Failed to send HCI RPA timeout command [00024354] <inf> fp_fmdn: Fast Pair: prepared the advertising set [00024592] <wrn> bt_hci_core: opcode 0x202e status 0x01 [00024794] <err> bt_id: Failed to send HCI RPA timeout command [00025065] <wrn> bt_hci_core: opcode 0x202e status 0x01 [00025268] <err> bt_id: Failed to send HCI RPA timeout command [00025518] <dbg> fp_fmdn_state: FMDN State: TX power set to 0 [dBm] [00025757] <dbg> fp_fmdn_state: FMDN State: enabled [00025941] <inf> dult_user: DULT enabled [00026089] <dbg> fast_pair: Fast Pair is enabled FMDN enabled [00026302] <inf> fp_fmdn: FMDN: state changed to unprovisioned has_account_key = 0 fmdn_provisioned change to 0 [00026726] <wrn> bt_hci_core: opcode 0x202e status 0x01 [00027425] <err> bt_id: Failed to send HCI RPA timeout command [00027661] <inf> fp_fmdn: Fast Pair: advertising in the discoverable mode [00028005] <err> mpsl_init: MPSL ASSERT: 112, 1617 [00028192] <err> os: ***** HARD FAULT ***** [00028357] <err> os: Fault escalation (see below) [00028545] <err> os: ARCH_EXCEPT with reason 3 [00028723] <err> os: r0/a1: 0x00000003 r1/a2: 0x20008068 r2/a3: 0x00000000 [00028996] <err> os: r3/a4: 0x00000003 r12/ip: 0x00000000 r14/lr: 0x000584b7 [00029268] <err> os: xpsr: 0x01000018 [00029422] <err> os: Faulting instruction address (r15/pc): 0x0003fd60 [00029667] <err> os: >>> ZEPHYR FATAL ERROR 3: Kernel oops on CPU 0 [00029904] <err> os: Fault during interrupt handling [00030098] <err> os: Current thread: 0x20005af8 (unknown) [00030306] <err> os: Halting system *** Booting nRF Connect SDK v2.7.0-5cb85570ca43 *** *** Using Zephyr OS v3.6.99-100befc70c74 *** [00000443] <inf> fs_nvs: 2 Sectors of 4096 bytes [00000615] <inf> fs_nvs: alloc wra: 1, ea0 [00000770] <inf> fs_nvs: data wra: 1, 708 load params: 8493465600, 707788800 buttons_init! btn_wake_on = 1 device_state = 0 batt_volts = 3366 batt_volts adjust = 3466 percentage_batt_lvl = 100% battery level: 100% [00001593] <inf> fp_fmdn: FMDN: setting (0x64) battery level twi read tx failed! 0xbae0001 twi read rx failed 0xbae0001 twi read tx failed! 0xbae0001 twi read rx failed 0xbae0001 twi read tx failed! 0xbae0001 twi read rx failed 0xbae0001 ACC Chip not found! Platform initialized 0x10 0xff 0x9f 0xcf 0xdc 0x54 0xf3 0xac [00003227] <inf> bt_sdc_hci_driver: SoftDevice Controller build revision: d6 da c7 ae 08 db 72 6f 2a a3 26 49 2a 4d a8 b3 |......ro *.&I*M.. 98 0e 07 7f |.... [00004217] <inf> bt_hci_core: HW Platform: Nordic Semiconductor (0x0002) [00004459] <inf> bt_hci_core: HW Variant: nRF52x (0x0002) [00004658] <inf> bt_hci_core: Firmware: Standard Bluetooth controller (0x00) Version 214.51162 Build 1926957230 [00005036] <inf> bt_hci_core: No ID address. App must call settings_load() [00005351] <inf> bt_hci_core: Identity[0]: D0:66:BE:04:67:9D (random) [00005589] <inf> bt_hci_core: Identity[1]: F6:40:9E:85:84:5C (random) [00005823] <inf> bt_hci_core: HCI: version 5.4 (0x0d) revision 0x11fb, manufacturer 0x0059 [00006120] <inf> bt_hci_core: LMP: version 5.4 (0x0d) subver 0x11fb Bluetooth initialized Starting the FMNA application [00006563] <inf> fmna: FMN pairing information is not present in the storage [00006906] <dbg> fmna: Advertising TX power set to 4 dBm [00007137] <dbg> fmna: Initializing FMN State to: Unpaired The FMN accessory transitioned to the unpaired state fmna_paired change to 0 [00007643] <inf> fmna: SW UUID: ed 56 19 8b 23 c7 49 14 ba 55 48 8c 13 86 6c 7c |.V..#.I. .UH...l| [00008125] <inf> fmna: SW Authentication Token (base64 format): [00008345] <inf> fmna: MYG/ME8CAQECAQEE (... 228 more chars ...) PkOgjHwrgyaleCE= [00008678] <inf> fmna: Serial Number: 61 30 31 31 38 32 66 30 36 35 30 34 36 37 39 64 |a01182f0 6504679d [00009105] <inf> fmna: Application firmware version: v2.2.3 [00009309] <wrn> fmna: The FMN stack is configured for qualification [00009548] <wrn> fmna: The qualification configuration should not be used for production FMNA initialized [00009898] <inf> fp_fmdn: Starting the FMDN application [00010090] <inf> fp_fmdn: FMDN initialized [00023200] <inf> fp_fmdn: FMDN: setting (0x64) battery level [00023434] <inf> fp_fmdn: Fast Pair: RPA expired [00023610] <inf> fp_fmdn: Fast Pair: setting RPA timeout to 787 [s] [00023862] <wrn> bt_hci_core: opcode 0x202e status 0x01 [00024064] <err> bt_id: Failed to send HCI RPA timeout command [00024329] <inf> fp_fmdn: Fast Pair: prepared the advertising set [00024566] <wrn> bt_hci_core: opcode 0x202e status 0x01 [00024768] <err> bt_id: Failed to send HCI RPA timeout command [00025037] <wrn> bt_hci_core: opcode 0x202e status 0x01 [00025239] <err> bt_id: Failed to send HCI RPA timeout command [00025488] <dbg> fp_fmdn_state: FMDN State: TX power set to 0 [dBm] [00025725] <dbg> fp_fmdn_state: FMDN State: enabled [00025909] <inf> dult_user: DULT enabled [00026056] <dbg> fast_pair: Fast Pair is enabled FMDN enabled [00026268] <inf> fp_fmdn: FMDN: state changed to unprovisioned has_account_key = 0 fmdn_provisioned change to 0 [00026688] <wrn> bt_hci_core: opcode 0x202e status 0x01 [00027381] <err> bt_id: Failed to send HCI RPA timeout command [00027615] <inf> fp_fmdn: Fast Pair: advertising in the discoverable mode00027853] <err> mpsl_init: MPSL ASSERT: 112, 1617 [00028039] <err> os: ***** HARD FAULT ***** [00028204] <err> os: Fault escalation (see below) [00028392] <err> os: ARCH_EXCEPT with reason 3 [00028570] <err> os: r0/a1: 0x00000003 r1/a2: 0x20008068 r2/a3: 0x00000000 [00028842] <err> os: r3/a4: 0x00000003 r12/ip: 0x00000000 r14/lr: 0x000584b7 [00029113] <err> os: xpsr: 0x01000018 [00029268] <err> os: Faulting instruction address (r15/pc): 0x0003fd60 [00029512] <err> os: >>> ZEPHYR FATAL ERROR 3: Kernel oops on CPU 0 [00029749] <err> os: Fault during interrupt handling [00029943] <err> os: Current thread: 0x20005bf8 (unknown) [00030150] <err> os: Halting system *** Booting nRF Connect SDK v2.7.0-5cb85570ca43 *** *** Using Zephyr OS v3.6.99-100befc70c74 *** [00000440] <inf> fs_nvs: 2 Sectors of 4096 bytes [00000612] <inf> fs_nvs: alloc wra: 1, e90 [00000766] <inf> fs_nvs: data wra: 1, 738 load params: 8493465600, 707788800 buttons_init! btn_wake_on = 1 device_state = 0 batt_volts = 3360 batt_volts adjust = 3460 percentage_batt_lvl = 100% battery level: 100% [00001593] <inf> fp_fmdn: FMDN: setting (0x64) battery level twi read tx failed! 0xbae0001 twi read rx failed 0xbae0001 twi read tx failed! 0xbae0001 twi read rx failed 0xbae0001 twi read tx failed! 0xbae0001 twi read rx failed 0xbae0001 ACC Chip not found! Platform initialized 0x10 0xff 0x9f 0xcf 0xdc 0x54 0xf3 0xac [00003225] <inf> bt_sdc_hci_driver: SoftDevice Controller build revision: d6 da c7 ae 08 db 72 6f 2a a3 26 49 2a 4d a8 b3 |......ro *.&I*M.. 98 0e 07 7f |.... [00004214] <inf> bt_hci_core: HW Platform: Nordic Semiconductor (0x0002) [00004456] <inf> bt_hci_core: HW Variant: nRF52x (0x0002) [00004655] <inf> bt_hci_core: Firmware: Standard Bluetooth controller (0x00) Version 214.51162 Build 1926957230 [00005033] <inf> bt_hci_core: No ID address. App must call settings_load() [00005351] <inf> bt_hci_core: Identity[0]: D0:66:BE:04:67:9D (random) [00005589] <inf> bt_hci_core: Identity[1]: FE:3F:E7:75:64:0C (random) [00005823] <inf> bt_hci_core: HCI: version 5.4 (0x0d) revision 0x11fb, manufacturer 0x0059 [00006119] <inf> bt_hci_core: LMP: version 5.4 (0x0d) subver 0x11fb Bluetooth initialized Starting the FMNA application [00006566] <inf> fmna: FMN pairing information is not present in the storage [00006909] <dbg> fmna: Advertising TX power set to 4 dBm [00007140] <dbg> fmna: Initializing FMN State to: Unpaired The FMN accessory transitioned to the unpaired state fmna_paired change to 0 [00007649] <inf> fmna: SW UUID: ed 56 19 8b 23 c7 49 14 ba 55 48 8c 13 86 6c 7c |.V..#.I. .UH...l| [00008134] <inf> fmna: SW Authentication Token (base64 format): [00008353] <inf> fmna: MYG/ME8CAQECAQEE (... 228 more chars ...) PkOgjHwrgyaleCE= [00008690] <inf> fmna: Serial Number: 61 30 31 31 38 32 66 30 36 35 30 34 36 37 39 64 |a01182f0 6504679d [00009116] <inf> fmna: Application firmware version: v2.2.3 [00009321] <wrn> fmna: The FMN stack is configured for qualification [00009560] <wrn> fmna: The qualification configuration should not be used for production FMNA initialized [00009909] <inf> fp_fmdn: Starting the FMDN application [00010101] <inf> fp_fmdn: FMDN initialized [00023211] <inf> fp_fmdn: FMDN: setting (0x64) battery level [00023445] <inf> fp_fmdn: Fast Pair: RPA expired [00023621] <inf> fp_fmdn: Fast Pair: setting RPA timeout to 812 [s] [00023872] <wrn> bt_hci_core: opcode 0x202e status 0x01 [00024074] <err> bt_id: Failed to send HCI RPA timeout command [00024340] <inf> fp_fmdn: Fast Pair: prepared the advertising set [00024577] <wrn> bt_hci_core: opcode 0x202e status 0x01 [00024779] <err> bt_id: Failed to send HCI RPA timeout command [00025047] <wrn> bt_hci_core: opcode 0x202e status 0x01 [00025250] <err> bt_id: Failed to send HCI RPA timeout command [00025499] <dbg> fp_fmdn_state: FMDN State: TX power set to 0 [dBm] [00025735] <dbg> fp_fmdn_state: FMDN State: enabled [00025919] <inf> dult_user: DULT enabled [00026066] <dbg> fast_pair: Fast Pair is enabled FMDN enabled [00026278] <inf> fp_fmdn: FMDN: state changed to unprovisioned has_account_key = 0 fmdn_provisioned change to 0 [00026698] <wrn> bt_hci_core: opcode 0x202e status 0x01 [00027391] <err> bt_id: Failed to send HCI RPA timeout command [00027625] <inf> fp_fmdn[00027713] <err> mpsl_init: MPSL ASSERT: 112, 1617 [00027899] <err> os: ***** HARD FAULT ***** [00028064] <err> os: Fault escalation (see below) [00028252] <err> os: ARCH_EXCEPT with reason 3 [00028429] <err> os: r0/a1: 0x00000003 r1/a2: 0x20008068 r2/a3: 0x00000000 [00028701] <err> os: r3/a4: 0x00000003 r12/ip: 0x00000000 r14/lr: 0x000584b7 [00028973] <err> os: xpsr: 0x01000018 [00029127] <err> os: Faulting instruction address (r15/pc): 0x0003fd60 [00029371] <err> os: >>> ZEPHYR FATAL ERROR 3: Kernel oops on CPU 0 [00029608] <err> os: Fault during interrupt handling [00029801] <err> os: Current thread: 0x20005bf8 (unknown) [00030008] <err> os: Halting system *** Booting nRF Connect SDK v2.7.0-5cb85570ca43 *** *** Using Zephyr OS v3.6.99-100befc70c74 *** [00000438] <inf> fs_nvs: 2 Sectors of 4096 bytes [00000610] <inf> fs_nvs: alloc wra: 1, e80 [00000765] <inf> fs_nvs: data wra: 1, 768 load params: 8493465600, 707788800 buttons_init! btn_wake_on = 0 device_state = 0 device_state = DEV_IDLE batt_volts = 3366 batt_volts adjust = 3466 percentage_batt_lvl = 100% battery level: 100% [00001668] <inf> fp_fmdn: FMDN: setting (0x64) battery level twi read tx failed! 0xbae0001 twi read rx failed 0xbae0001 twi read tx failed! 0xbae0001 twi read rx failed 0xbae0001 twi read tx failed! 0xbae0001 twi read rx failed 0xbae0001 ACC Chip not found! Platform initialized 0x10 0xff 0x9f 0xcf 0xdc 0x54 0xf3 0xac [00003307] <inf> bt_sdc_hci_driver: SoftDevice Controller build revision: d6 da c7 ae 08 db 72 6f 2a a3 26 49 2a 4d a8 b3 |......ro *.&I*M.. 98 0e 07 7f |.... [00004298] <inf> bt_hci_core: HW Platform: Nordic Semiconductor (0x0002) [00004542] <inf> bt_hci_core: HW Variant: nRF52x (0x0002) [00004740] <inf> bt_hci_core: Firmware: Standard Bluetooth controller (0x00) Version 214.51162 Build 1926957230 [00005118] <inf> bt_hci_core: No ID address. App must call settings_load() [00005441] <inf> bt_hci_core: Identity[0]: D0:66:BE:04:67:9D (random) [00005679] <inf> bt_hci_core: Identity[1]: CB:9B:D4:3C:D4:53 (random) [00005915] <inf> bt_hci_core: HCI: version 5.4 (0x0d) revision 0x11fb, manufacturer 0x0059 [00006212] <inf> bt_hci_core: LMP: version 5.4 (0x0d) subver 0x11fb Bluetooth initialized Starting the FMNA application [00006662] <inf> fmna: FMN pairing information is not present in the storage [00007007] <dbg> fmna: Advertising TX power set to 4 dBm [00007241] <dbg> fmna: Initializing FMN State to: Unpaired The FMN accessory transitioned to the unpaired state fmna_paired change to 0 [00007753] <inf> fmna: SW UUID: ed 56 19 8b 23 c7 49 14 ba 55 48 8c 13 86 6c 7c |.V..#.I. .UH...l| [00008241] <inf> fmna: SW Authentication Token (base64 format): [00008461] <inf> fmna: MYG/ME8CAQECAQEE (... 228 more chars ...) PkOgjHwrgyaleCE= [00008801] <inf> fmna: Serial Number: 61 30 31 31 38 32 66 30 36 35 30 34 36 37 39 64 |a01182f0 6504679d [00009227] <inf> fmna: Application firmware version: v2.2.3 [00009432] <wrn> fmna: The FMN stack is configured for qualification [00009671] <wrn> fmna: The qualification configuration should not be used for production FMNA initialized [00010022] <inf> fp_fmdn: Starting the FMDN application [00010214] <inf> fp_fmdn: FMDN initialized