Random Fatal Error log message when powering up board

RE: Random Fatal Error log message when powering up board

Vidar Berg — Tue, 19 Nov 2019 08:20:12 GMT

Hello Robin,

The DEBUG flag is only used for pre-processor checks in the code, so I would suggest reviewing its usage in your code to make sure it doesn't have any unindented side effects if you haven't already done it. E.g., check that the program doesn't go into an infinite loop if your error handler gets invoked.

RE: Random Fatal Error log message when powering up board

Robin — Mon, 18 Nov 2019 17:40:03 GMT

Hello Vidar,

These error have not reoccured since I started with the DEBUG flag set. They may have been related to a bug I found wherein I was attempting read/write NVRAM near battery death. At this stage is is unlikely I will have time to chase it unless they become problematic again, or something changes in our schedule. For now the plan to move forward with DEBUG set for production code. Do you see any serious concerns with doing this?

Thanks

Robin @ TL

PS. You can close this case for now. I will be opening one today regarding pairing/bonding implications for ble_app_uart and ble_app_uart_central.

RE: Random Fatal Error log message when powering up board

Vidar Berg — Wed, 06 Nov 2019 15:44:09 GMT

Hi Robin,

[quote userid="74971" url="~/f/nordic-q-a/53920/random-fatal-error-log-message-when-powering-up-board/218857"]I am not sure you caught the gist of this latest finding. With the DEBUG flag there had need no reboots or errors. Or might I not be understanding your reply?[/quote]

You're right. The DEBUG flag could explain why you no longer get reboots, but it doesn't explain why the errors have stopped occurring. It's hard to say if that's a coincidence or not.

[quote userid="74971" url="~/f/nordic-q-a/53920/random-fatal-error-log-message-when-powering-up-board/218857"]Right now I am more concerned about why before DEBUG there were multiple reboots within a few hours, and since it, there had been none ( will be rechecking that this AM).[/quote]

Could you try to add logging in APP_UART_COMMUNICATION_ERROR without setting the DEBUG flag to see if this may be the error source?

Regards,

Vidar

RE: Random Fatal Error log message when powering up board

Robin — Wed, 06 Nov 2019 14:14:35 GMT

Hi Vidar,

I am not sure you caught the gist of this latest finding. With the DEBUG flag there had need no reboots or errors. Or might I not be understanding your reply?

Indeed, during the Cal and test processes there is code that writes (and therefore necessarily erases) NVRAM. I may have a handle on this, as I am doing battery life monitoring then at full charge and discharge saving the data to NVRAM. Under some battery conditions the battery may fully crash before the write. This could be the rare memory erasure.

Right now I am more concerned about why before DEBUG there were multiple reboots within a few hours, and since it, there had been none ( will be rechecking that this AM).

Thanks again for your time,

Robin @ TL

RE: Random Fatal Error log message when powering up board

Vidar Berg — Wed, 06 Nov 2019 07:38:55 GMT

Hello Robin,

It might be related. The DEBUG flag changes the error handling in the default app error handler: without the flag, the device resets itself by calling NVIC_SystemReset() to attempt to recover, and with the flag set, it enters the infinite loop app_error_save_and_stop() to allow the error information to be inspected with the debugger.

I know I indicated earlier that the error information (from app_error_save_and_stop) would be logged. But I had forgotten that crash logging was first introduced in SDK 14, so sorry for the confusion.

The default error handler from SDK 12.3.0 is shown below. If you want to verify if the behavior I described above with your app, you may insert an APP_ERROR_CHECK() macro in main() and pass an error code to it. Then try debugging with and without the DEBUG flag.

/**@brief Application main function.
 */
int main(void)
{
    uint32_t err_code;
    bool erase_bonds;

    // Initialize.
    APP_TIMER_INIT(APP_TIMER_PRESCALER, APP_TIMER_OP_QUEUE_SIZE, false);
    uart_init();

    APP_ERROR_CHECK(NRF_ERROR_INVALID_PARAM); //<-- will invoke error handler when input is not NRF_SUCCESS
    ...
}

Anyway, the "Fatal Error" message confirms the app is entering the fault handler, so the next step should be to find out where the error occurred. Since your app is based on ble_app_uart I'm wondering if you have kept the default handling for the APP_UART_COMMUNICATION_ERROR event in the UART callback? This is a common source of errors. A floating RX input, level change on startup, etc may lead to a UART Error condition (link to nrf52 but same applies to 51). To check if this may be the case I suggest that you insert

case APP_UART_COMMUNICATION_ERROR:

NRF_LOG_INFO(" UART comm error. Error source %d", p_event->data.error_communication);
APP_ERROR_HANDLER(p_event->data.error_communication); // <-- Will invoke fault handler. Handling of this error is application-specific. May be removed if it's OK to ignore com. errors.
break;

[quote user=""]2) Once booted, the contents of NVRAM (Board Cal, Sensor Cal, and board/battery life information) have been erased.[/quote]

This is probably not directly related to the code assertion. Is there anything in your code that may delete the data in certain conditions?

Regards,

Vidar

RE: Random Fatal Error log message when powering up board

Robin — Wed, 06 Nov 2019 03:45:56 GMT

Hello yet again Vidar,

I am not sure this is related, but it very well may be. I had powered 5 new boards yesterday and started burning them in. When I checked them this morning all but one had multiple reboots. At that juncture I elected to recompile with the DEBUG flag set, as you suggested, downloaded to all five boards, and restarted the burnin test on all of them, while monitoring the worst for errors. So far no errors. The only change I made to the code was the DEBUG flag. What do you think could be going on?

The failures the first attempt, before setting the flag happened within a few hours of the original burnin start. I am now more than half way through the (24hr) burnin restart, with no sign of problem.

Thanks for your input

Robin @ TL

RE: Random Fatal Error log message when powering up board

Robin — Tue, 05 Nov 2019 16:05:53 GMT

Vidar,

Where will these additional debug message appear? Are limited to the log config setting, or can they be directed to the BT UART port?

Thanks

Robin

RE: Random Fatal Error log message when powering up board

Robin — Tue, 05 Nov 2019 13:31:14 GMT

Ah, I see, thank you very much Vidar.

Robin

RE: Random Fatal Error log message when powering up board

Vidar Berg — Tue, 05 Nov 2019 07:35:26 GMT

Hello Robin,

The "DEBUG" flag does not impact the log level, that's a separate configuration option in the logger module. The flag is usually set like this if you use Keil:

RE: Random Fatal Error log message when powering up board

Robin — Mon, 04 Nov 2019 18:36:46 GMT

Hello again Vidar,

Thank you for the response. I am currently logging only "INFO" as I use this mode to trace the code execution. I will set the flag for "DEBUG". Will the info messages still print in DEBUG mode?

Thanks again, and happiest of Holday Seasons to you and yours.

Robin @ TL

RE: Random Fatal Error log message when powering up board

Vidar Berg — Mon, 04 Nov 2019 14:24:17 GMT

Hi,

Please try to compile your project with "DEBUG" defined if you haven't done so already. It should result in a more verbose crash log if used together with the default app error handler (Error module). Error information may include the file name and line number of where the error occurred.

Regards,

Vidar