app_error_fault_handler() called with PC at 0x128A0

RE: app_error_fault_handler() called with PC at 0x128A0

MartinBL — Thu, 23 Aug 2018 08:56:50 GMT

Thanks. Let us continue in the other case.

Martin

RE: app_error_fault_handler() called with PC at 0x128A0

Eliot Stock — Wed, 22 Aug 2018 20:36:21 GMT

Thanks Martin.

1. #define SPI_DEFAULT_CONFIG_IRQ_PRIORITY 6

2. We have 13 timers configured across 8 modules, all using APP_TIMER_DEF(), app_timer_*(), etc.

3. Not directly. The only thing I can think of that we do that would cause a flash operation is bonding, and it happens only once, in general, for the lifetime of the device.

4. Never directly in our code. I haven't read all the SDK sources we link in to see whether SDK code we call does this.

5. Sure, but it's too much to link here so I've uploaded it in case ID 212717.

RE: app_error_fault_handler() called with PC at 0x128A0

MartinBL — Mon, 20 Aug 2018 11:34:32 GMT

Hi,

Sorry for the slow response. I have been on holiday for a couple of weeks. Have you made any progress?

Just a short recap: As mentioned before your assert indicates that something is messing with the mechanisms that the Softdevice uses to keep time. To keep time, the Softdevice uses RTC0 and TIMER0 and both peripherals use interrupts that run in priority 0 (highest). So the question is how your application is able to disrupt these timing critical things? It could be one of two root causes:

An inaccurate clock causes the nRF52's time to tick too slow or fast relative to the rest of the world. Since you are using prefabricated modules and are able to reproduce it on more than one device I find this option unlikely.
Something in your application is messing with the Softdevice's RTC and Timer interrupts, preventing the them from being executed on time.

Some questions:

What interrupt priority is your SPI running at?
What timers are you using?
Are you doing flash operations in your application?
Are you at any point disabling interrupts to execute time critical code? (some drivers and libraries in the SDK do this under the hood)
Do you mind uploading your code?

RE: app_error_fault_handler() called with PC at 0x128A0

Austin — Tue, 14 Aug 2018 01:29:00 GMT

It's really up to you which approach you take. My preference would be to use the latest SDK possible as there are often many improvements and bug fixes in comparison to prior releases. If you already have devices in production then this may not be feasible.

If you have a generally stable product apart from this issue, perhaps it would be a good idea to patch your SDK 14 with the updated logger frontend to first verify if this is in fact the issue you're facing. I haven't looked at the differences between SDK 14 and 14.2 so I'm not sure if there were other changes related to logging which may impact this patch. From memory, my logger frontend change was applicable to SDK 14.2.

RE: app_error_fault_handler() called with PC at 0x128A0

Eliot Stock — Tue, 14 Aug 2018 01:20:19 GMT

OK, I had in mind to migrate to SDK 14.2 pretty soon anyway, before I implement DFU OTA. Should I do that, patch the logger frontend with your patch, then do a load of testing again?

RE: app_error_fault_handler() called with PC at 0x128A0

Austin — Mon, 13 Aug 2018 23:17:28 GMT

It appears you're using SDK14. If you haven't already, have a look at the known issues list at https://devzone.nordicsemi.com/f/nordic-q-a/29796/what-are-sdk-14-x-0-known-issues/.

I had a number of issues with SDK14 related to logging using Segger RTT, especially with the use of APP_LOG_PUSH. The following thread https://devzone.nordicsemi.com/f/nordic-q-a/29103/nrf_log-fixes-in-sdk14-1-0 describes some of these issues. The accepted answer addresses some bugs with the logging module, but if you scroll down to my answer you will find a suggested nrf_log_frontend.c implementation which addresses a larger set of issues.

You could try to induce your issue quicker by increasing the amount of logging.

RE: app_error_fault_handler() called with PC at 0x128A0

Eliot Stock — Sun, 12 Aug 2018 19:51:28 GMT

Hi Martin, I've been able to repro this on a second device. What do we do now?

Never mind the issue with the RAM log, it was a bug in my logging implementation.

RE: app_error_fault_handler() called with PC at 0x128A0

Eliot Stock — Tue, 17 Jul 2018 10:22:31 GMT

Agreed, I don't understand why the last part of the RAM looks uninitialised. I'll do some more testing on the circular buffer.

Nope, I still can't reliably repro this, even on the first device, so getting it to repro on the second will be a long shot. Will keep testing with the same firmware.

No problem. Let's pick it up when you're back, but it may help if I add the code for the in-RAM log backend to a case.

RE: app_error_fault_handler() called with PC at 0x128A0

MartinBL — Tue, 17 Jul 2018 10:02:04 GMT

Thank you.

In your last log, one could say that the last part of the RAM just looks like random/uninitialized portions for memory. But you say that you are using a circular buffer and that, since there is no init stuff at the beginning, the buffer has already "circled around" at least once. So how come the last part of RAM looks so random if it has actually been used? Just a rhetorical question, but it is sort of contradictory isn't it?

Did you have any success replicating the first issue on a second device?

PS: I'll be on holiday until next Tuesday. As it is holiday season in Norway it is not certain that any of my colleagues will have time to pick up the case in the meantime.

RE: app_error_fault_handler() called with PC at 0x128A0

Eliot Stock — Mon, 16 Jul 2018 21:21:29 GMT

Sure, added above.

RE: app_error_fault_handler() called with PC at 0x128A0

MartinBL — Mon, 16 Jul 2018 08:48:41 GMT

Can you upload the code that shows how you handle the Softdevice assert and logging?

RE: app_error_fault_handler() called with PC at 0x128A0

Eliot Stock — Fri, 13 Jul 2018 22:23:41 GMT

Added some example logs above. I have code that assures me my retained RAM is working. In a brown out, I'd expect to lose the in-RAM log, but I'd also expect the Nordic to then reset and continue logging again from the top of my log memory region, but that's not what I'm seeing in the last log above.

RE: app_error_fault_handler() called with PC at 0x128A0

MartinBL — Fri, 13 Jul 2018 16:27:22 GMT

Can you elaborate on what you mean by "the log just stops"? Do you get any information at all or is it all lost?

As you can see in this table, the RAM doesn't survive all reset types. System resets as triggered in the error handler should be fine, but you will lose the content on a brown out for example.

RE: app_error_fault_handler() called with PC at 0x128A0

Eliot Stock — Wed, 11 Jul 2018 07:53:44 GMT

It's a Wisol SFM20R1 and uses the Nordic nRF52832 variant AAB0. I have one other surviving prototype board on which I can test, but it'll take a while because I can't repro this reliably even on the first board.

Any thoughts on the error logging issue? Can I send you the complete code for that, in a case?

RE: app_error_fault_handler() called with PC at 0x128A0

MartinBL — Wed, 11 Jul 2018 07:27:52 GMT

Do you have more than one module that you can try? Maybe this one particular module is an outlier with a broken crystal or bad solder joint or something causing the clock to run inaccurately. If the issue occurs on all modules on the other hand, I would be more inclined towards a firmware issue.

What is the serial number of the module?

RE: app_error_fault_handler() called with PC at 0x128A0

Eliot Stock — Tue, 10 Jul 2018 21:00:13 GMT

In that case I may be wrong about not being in a connection. I'm only ever in one connection at a time but I do a lot of logging stuff over a characteristic a bit like the Nordic UART service's RX characteristic (notify only). I don't know if that would have the radio busy enough to hit this, but I guess it's possible. I don't use the timeslot API.

I'm using a module from Wisol so I haven't chosen the crystal myself. I could ask them what they used.

It would help if I could get reliable logging in my app_error_fault_handler() function. At the moment I really only know this is happening on the desk because the debugger breaks in the error handler. In the field, I'd expect to see the same error logging in my in-memory log as I do in the gdb client, but instead the log just stops. My in-memory log is in retained RAM, so it survives a reset.

RE: app_error_fault_handler() called with PC at 0x128A0

MartinBL — Tue, 10 Jul 2018 08:48:06 GMT

Hi,

Thanks. I found the source of the assert. The Softdevice includes what we call a Radio Event Manager (REM), and an assert at 0x128A0 is the REM warning you that a radio event has overstayed its planned duration. This is more likely to happen in complex applications where you have multiple concurrent BLE links, have multiple radio protocols running concurrently, or are using the Timeslot API to allocate time slots to time sensitive peripheral operations. You say that you are not in a connection and just advertising, but do you use the timeslot API? Any other time sensitive things going on at the time?

Another possibility is that your clock is not accurate enough. If you are not doing anything timecritical as discussed above this possibility might be more likely. It could e.g. be that your crystal is incorrectly loaded. Are you using a custom device or a development kit?

RE: app_error_fault_handler() called with PC at 0x128A0

Eliot Stock — Mon, 09 Jul 2018 22:28:13 GMT

Thanks.

NORDIC_SDK_ROOT := $(HOME)/dev/nRF5_SDK_14.0.0_3bcc1f7

SOFTDEVICE = $(NORDIC_SDK_ROOT)/components/softdevice/s132/hex/s132_nrf52_5.0.0_softdevice.hex

I've seen it about four times in the last month, but I can't reliably repro it. Happy to send the hex file if it helps.

RE: app_error_fault_handler() called with PC at 0x128A0

MartinBL — Mon, 09 Jul 2018 12:06:46 GMT

Hi,

That is pretty strange. If you give me the version number of your Softdevice I can take a look under the hood and see what goes on at that specific address.

How frequently does it occur?

And what SDK are you using?