SoftDevice Controller ASSERT: 53, 296

RE: SoftDevice Controller ASSERT: 53, 296

AHaug — Fri, 20 Dec 2024 10:56:16 GMT

Hi Tim,

I can verify that I've seen the discussion this has raised. I will monitor it.

Since the fix is within non-open source code material, I can't share anything about the commit here. The best approach willl thus be to keep in touch with the sales manager and/or raise your own private case (with relevant company info and request info)

Kind regards,
Andreas

RE: SoftDevice Controller ASSERT: 53, 296

mertzt — Thu, 19 Dec 2024 16:18:01 GMT

I've reached out to our local sales manager to request the backport. In the meantime, we are trying to gather as much information as possible.

Do you have any additional information about what commits address the issue?
Are there any known workarounds?

Thanks,
Tim

RE: SoftDevice Controller ASSERT: 53, 296

AHaug — Wed, 18 Dec 2024 11:56:35 GMT

Hi,

The fix is in NCS v2.8.0 and was determined previously to not be back ported to v2.6.1.

If you need it back ported, please reach out to your regional sales manager and request it through them

Kind regards,
Andreas

RE: SoftDevice Controller ASSERT: 53, 296

mertzt — Tue, 17 Dec 2024 19:49:25 GMT

Has this been addressed?

I am encountering the same assert on my project using an NRF5340 and NCS v2.6.1.

RE: SoftDevice Controller ASSERT: 53, 296

AHaug — Thu, 18 Jul 2024 06:53:29 GMT

Hi Mårten,

No worries, glad you found a middle ground that works for you. The patch will nonetheless be added to main shortly

Kind regards,
Andras

RE: SoftDevice Controller ASSERT: 53, 296

Maos — Wed, 17 Jul 2024 14:02:26 GMT

Hi!

Sorry for not responding sooner.

Thank you for confirming so we know that that there is nothing we need to change.

Unfortunately we had to go back to 2.5.3 where we don't see this issue so I can't test the workaround.

BR,
Mårten

RE: SoftDevice Controller ASSERT: 53, 296

AHaug — Mon, 15 Jul 2024 09:02:36 GMT

Hi,

Just a status update from us here: Verified as a bug and we're working on getting a fix to NCS main soon. We'll update you when the PR is available for monitoring

Could you try to remove the data length updates from the project? This should work around the issue.

Kind regards,
Andreas

RE: SoftDevice Controller ASSERT: 53, 296

Edvin — Fri, 05 Jul 2024 19:26:03 GMT

Yupp! There you go:

I will report it to our softdevice controller team, along with instructions on how to use it. This means that the next person handling this ticket will only have to relay the information from the softdevice team.

Best regards,

Edvin

RE: SoftDevice Controller ASSERT: 53, 296

Edvin — Fri, 05 Jul 2024 19:01:17 GMT

Hello Mårten. I have had the device running for a while now, but I have not seen any crashes yet. I then remembered that you mentioned the part about the RSSI. It is currently running in the office (and I am at home), but I can try to crank down the peripheral's TX power, to see if that does the trick. I will be out of office from next week, but I will write some notes for the person taking over. (I am sorry for the inconvenience, but I hope you understand it has to be that way in the summers). Hopefully the next person can have a flying start at this issue.

BR,

Edvin

RE: SoftDevice Controller ASSERT: 53, 296

Maos — Fri, 05 Jul 2024 13:37:58 GMT

Hi!

Any updates on this issue?
Have you been able to reproduce it?

BR,

Mårten

RE: SoftDevice Controller ASSERT: 53, 296

Edvin — Wed, 03 Jul 2024 21:17:49 GMT

Thank you! That is very helpful, and speeds things up a bit! I am sorry for getting back so late. I will try to run it tomorrow, to replicate the issue. Once that is done, it is a lot easier to either pinpoint, or forward the question to the developers, if they can reproduce it themselves.

You will hear from me tomorrow.

Best regards,

Edvin

RE: SoftDevice Controller ASSERT: 53, 296

Maos — Tue, 02 Jul 2024 15:04:34 GMT

Hi again!

I have now created a fw for the peripheral side as well.
Also suitable to link inas the fw binary to upload in the central build

I have also tested that I can reproduce the issue using these firmwares

At the moment the central is configured with RTT loggin so the crash dump can be viewed.

I did not have a dk available to test these on but by disabling various external signals as can be seen in the overlay file I was able to run it on our custom hardware building for nrf52840dk_nrd52840.

BR,
Mårten

devzone.nordicsemi.com/.../nrf_2D00_reproducer_2D00_peripheral.zip

RE: SoftDevice Controller ASSERT: 53, 296

Maos — Tue, 02 Jul 2024 12:53:34 GMT

Hi!

yes that is true.

We set the advertise parameters as follows and call bt_le_ext_adv_create:
static const uint32_t ADV_OPTS = (BT_LE_ADV_OPT_CONNECTABLE | BT_LE_ADV_OPT_EXT_ADV |
                                  BT_LE_ADV_OPT_CODED | BT_LE_ADV_OPT_USE_IDENTITY |
                                  BT_LE_ADV_OPT_ONE_TIME);

static struct bt_le_adv_param adv_param = BT_LE_ADV_PARAM_INIT(ADV_OPTS,
                                                               BT_GAP_ADV_FAST_INT_MIN_2,
                                                               BT_GAP_ADV_FAST_INT_MAX_2,
                                                               NULL);

And when a connection is made they are updated using bt_le_ext_adv_update_param to the following:
static const uint32_t ADV_DURING_CON_OPTS = (BT_LE_ADV_OPT_EXT_ADV | BT_LE_ADV_OPT_CODED |
                                             BT_LE_ADV_OPT_USE_IDENTITY);

static struct bt_le_adv_param adv_conn_param = BT_LE_ADV_PARAM_INIT(ADV_DURING_CON_OPTS,
                                                                    BT_GAP_ADV_FAST_INT_MIN_2,
                                                                    BT_GAP_ADV_FAST_INT_MAX_2,
                                                                    NULL);

But other than that I think the smp_svr example should work.
I will look into providing you with the peripherial side as well.

BR,

Mårten

RE: SoftDevice Controller ASSERT: 53, 296

Edvin — Tue, 02 Jul 2024 12:37:44 GMT

Hello Mårten,

Let me get this straight. This is the central application that is supposed to upload new FW to a peripheral, right? So I would need a peripheral device with an MCUBOOT bootloader running with this? Does the standard smp_svr sample work? Or does it need to be something special?

BR,
Edvin

RE: SoftDevice Controller ASSERT: 53, 296

Maos — Mon, 01 Jul 2024 11:24:02 GMT

Hi!

Great!

Yes, I understand, I will myself be on vacation from the 18th.

BR,
Mårten

RE: SoftDevice Controller ASSERT: 53, 296

Edvin — Fri, 28 Jun 2024 22:14:08 GMT

Hello Mårten,

Einar is out of office for a few weeks to come, so I will have a look at your ticket as soon as possible, hopefully on Monday. Please understand that we are a bit short staffed due to summer holidays.

Best regards,

Edvin

RE: SoftDevice Controller ASSERT: 53, 296

Maos — Fri, 28 Jun 2024 08:54:03 GMT

Hi!

I have created a reproducer fw, so this should be run as a central.
As I have removed all functionality that gets the fw to be sent during update I have made this compile the fw top transfer as ro-data.

Two things needs local adaption the mac address of the peripherial to connect and update.
And the path to the app_uppdate.bin file to send to the peripherial.
These are located in main.c line 102 and bin_handler.c line 19.

For the peripheral any client with support for dfu-smp and advertising in a compatible manner (coded phy) should be possible to use (and to fit in the central fw it should not be bigger than around 250k),

I have also attached some logs showing the output I get at two different locations running this fw, as I have had some problems recreate the issue in of the locations but as you can see from the logs I now can reproduce it in both locations.

BR,
Mårten

devzone.nordicsemi.com/.../nrf_2D00_reproducer.zip

devzone.nordicsemi.com/.../rttlog_2D00_office.log

devzone.nordicsemi.com/.../rttlog_2D00_home.log

RE: SoftDevice Controller ASSERT: 53, 296

Maos — Wed, 26 Jun 2024 14:52:34 GMT

Hi!

Great any suggestion or advice is appreciated.

I was actually thinking about asking if that was a possibility....
And am working on such a project (also as a way to try to reduce factors affecting it or eliminating mistakes on our side) but not quite finished, I will update here when it is ready.

BR,

Mårten

RE: SoftDevice Controller ASSERT: 53, 296

Einar Thorsrud — Wed, 26 Jun 2024 14:36:56 GMT

Hi Mårten,

I am checking internally to see what we can understand with this information, but I do not have any updates yet unforunately.

Is it possible for you to share a proejct that can reproduce this on a DK so that we can test on our side?

Br,

Einar

RE: SoftDevice Controller ASSERT: 53, 296

Maos — Mon, 24 Jun 2024 15:09:30 GMT

Hi

Seems the uart thing was false lead (maybe not so suprising), it might have probability of reproducing somewhat.

Also noticed I had commented out code that initialize nfc at the same time..

I did try to lower irq priority of both uart and nfc in device tree, couldn't find any clear information on this but hoped that a higher number meant lower priority but the assert was still triggered.

One detail I forgot to mention is that it onlly seem to happen if the peripheral is a bit away from the central, the rssi value reported by the SDK is somewhere betwen -85 and -95 when it happens.

Most of the time when this issue happens the connection has been lost once and then been able to reconnect again, and during this second connection this assert kicks in.

I think I have seen this on "first" connection but maybe it has been connected and done a "proper" disconnect but I missed to reset the test setup, so it is still on second connection.

I also noticed a place where the same connection maybe is disconnected twice (so disconnect returns -128).

I also tried with the 2.7.0-rc2 SDK and get a bit different ASSERT:
00> [00:00:38.858,306] [1;31m<err> bt_sdc_hci_driver: SoftDevice Controller ASSERT: 53, 345[0m
00> [00:00:38.858,337] [1;31m<err> os: ***** HARD FAULT *****
00> [00:00:38.858,337] [1;31m<err> os: Fault escalation (see below)[0m
00> [00:00:38.858,367] [1;31m<err> os: ARCH_EXCEPT with reason 3
00>
00> [00:00:38.858,367] [1;31m<err> os: r0/a1: 0x00000003 r1/a2: 0x00000000 r2/a3: 0x000e7241
00> [00:00:38.858,367] [1;31m<err> os: r3/a4: 0x00000003 r12/ip: 0x2001a238 r14/lr: 0xffffffff
00>
00> [00:00:38.858,398] [1;31m<err> os: xpsr: 0x01000011
00> [00:00:38.858,398] [1;31m<err> os: r4/v1: 0x20018168 r5/v2: 0x00033db9 r6/v3: 0x0000000a[0m
00>
00> [00:00:38.858,428] [1;31m<err> os: r7/v4: 0x20018168 r8/v5: 0x00000000 r9/v6: 0x00000000[0m
00> [00:00:38.858,428] [1;31m<err> os: r10/v7: 0x200009a0 r11/v8: 0x00000000 psp: 0x20018480
00> [00:00:38.858,459] [1;31m<err> os: EXC_RETURN: 0xfffffff1[0m
00> [00:00:38.858,459] [1;31m<err> os: Faulting instruction address (r15/pc): 0x00033df4[0m
00> [00:00:38.858,489] [1;31m<err> os: >>> ZEPHYR FATAL ERROR 3: Kernel oops on CPU 0[0m
00> [00:00:38.858,489] [1;31m<err> os: Fault during interrupt handling
00>
00>
00> [00:00:38.858,551] [1;31m<err> os: Current thread: 0x2000bd80 (unknown)[0m
00> [00:00:51.399,688] [1;31m<err> os: Halting system[0m

What can be interpreted from this one?

Also could the callback passed to bt_dfu_smp_command block the softdevice (assuming there is no explicit call to irq_lock or similar)?

BR,

Mårten

RE: SoftDevice Controller ASSERT: 53, 296

Einar Thorsrud — Fri, 21 Jun 2024 13:35:51 GMT

Hi Mårten,

It is good to hear you are making progress in narrowing down the issue. I do not have any suggestions from our side at the moment, but please let me know if you are able to narrow this down futher.

Br,

Einar

RE: SoftDevice Controller ASSERT: 53, 296

Maos — Thu, 20 Jun 2024 09:44:20 GMT

Hi!

Well we use the uart, nfc and bluetooth of the device and in the case of bluetooth scan for advertise data and if needed connect to the peripherial to either write a fairly small amount of data and/or do a OTA.
We only allow one connection doing OTA and one other in parallel.

I have also tried sdk version 2.5.3 and 2.6.0-rc1 and am not able to reproduce the issue on 2.5.3 but can reproduce it on 2.6.0-rc1.

I have tried to look over our code to see if there is some processing that is done when parsing the advertise data that should be deffered to later in some way, this does not seem to help the issue.

At the moment I am actually looking into the uart sending part.
The uart sending is done by copying the data to one buffer taken from a pool of buffers and put in a fifo that is read by a seperate thread that calls the uart tx function.

I did some test basically disabling the whole tx part (after some initial setup communication) and could not reproduce the issue.
At that test it looked like lowering the priority of the uart-tx-thread did not help.

I also noticed that by mistake the item put in the fifo was not (explicitly) word aligned, thou it probably would be because the second item in the struct after the intial (reserved) word was for some reason set to alignment 16.

So at the moment I am looking into moving the "disabling" of tx from the entry point to the thread to see how that affects reproducability and if setting the alignment correctly will make a difference, possibly in combination with lowering the tx thread prio.

This in one way feels a bit far fetched but this is the only change that seem to affect reproducability....

BR,
Mårten

RE: SoftDevice Controller ASSERT: 53, 296

Einar Thorsrud — Wed, 19 Jun 2024 08:54:39 GMT

Hi Mårten,

I checked with our SoftDevice Controller team and they did not have any more specific ideas other than wondering if you (or a module you use) perhaps disables interrupts for too long (as in a critical section with irq_lock()/irq_unlock(). Could that be the case?

Br,

Einar

RE: SoftDevice Controller ASSERT: 53, 296

Maos — Mon, 17 Jun 2024 12:05:47 GMT

Hi!

I have tried various settings to get a lead what could be causing this without much luck.

Regarding threads this is the status of them at the time the exception happens:

And this is the callstack:

I don't know if they provide any new lead on what is causing this.

I have tested this with the v2,4 and v2.6 stack and I can't reproduce it on the v2.4 stack (only code change during the test is three locations where we call zcbor_new_decode_state).

I have doing various adjustments to our thread priorites and also tested to enable CONFIG_BT_RECV_WORKQ_BT.

I tried to see if there where any interesting configs related to irq priorites but I couldn't find a working combination of settings (Mostly NFCT_IRQ_PRIORITY and ZERO_LATENCY_LEVELS).

(The device receive and send data on NFC but I don't need to blipp it to make this issue happen).

But none of these changes seem to solve the issue.

BR,

Mårten

RE: SoftDevice Controller ASSERT: 53, 296

Einar Thorsrud — Mon, 17 Jun 2024 09:10:03 GMT

Hi Mårten,

It is difficult to say what caused the issue, as the assert only tells us that the SoftDevice Controller was not able to handle a radio event on time. But in principle enything (with same or higher priority) could have prevented it from doing what it should.

[quote user="Maos"]Could doing to much processing in a scan_le_cb (recv) callback introduce such a problem?[/quote]

Yes, that is possible.

[quote user="Maos"]Or is this seperated and only a more systemwide depletion of resources would make the scan callback affect the this?[/quote]

That is also possible (though low priority threads will not cause this).

[quote user="Maos"]Also I haven't been able to test with the configs mentioned as they all depend on BT_LL_SW_SPLIT being set[/quote]

Yes, that is correct. I was too quick when I dug up these configs. They are not relevant for the SoftDevice Controller. In fact, there are no configs for setting these priorities (you can see all the SoftDevice Controller speicfic configs in nrf/subsys/bluetooth/controller/Kconfig).

Have you experimented and noticed if there are any configs that affect if you see this issue or not (related to SMP or bluetooth in general) that may perhaps point at something worth looking deeper into?