hardfault issue with modified ble_app_uart_c example

[maxpmillion]

Greetings!

I have modified the "ble_app_uart_c" example from SDK v17.1.0 to be used to scan for and control our BLE-enabled devices via a terminal emulator. When we use the nRF52-DK as a simple UART-to-BLE bridge, we don't have any issues, but we also use the DK to send a firmware file over-the-air to upgrade our devices, and that's when I discovered this issue. Here's a brief overview of the update procedure:

A custom PC application reads 32-bit words from a text file and sends them one at a time to the nRF52-DK. The DK stores these values in a buffer until one page of data (264 bytes) is received. At that point the DK divides the page into 17 BLE packets, each containing 3 bytes of header data and 4 words of firmware data (except for the 17th one, which contains 2 words), and sends each packet to the device. The device then writes the page to flash and sends an acknowledgement, after which the process begins again until the entire firmware file is transferred. 

I am experiencing a hardfault during the OTA update at a consistent point (it gets ~20% of the way through the transfer, maybe 100 pages of data), and I am having trouble determining the root cause. Attached is a stack dump of when the fault occurs. If I am understanding this post  correctly, then the offending line of code is at 0x2'BB48, which, when cross-referencing the map file, is within nrf_queue_generic_pop(). I also attached the top-level application (see link below). I am using gcc, so the map file is under ble_app_uart_c/pca10040/s132/armgcc/_build/. I'm assuming this will eventually end up being my application code mishandling Nordic's libraries in some manner, but I was hoping that someone else might be able to take a look and offer some guidance for debugging this further. I tried to follow the hardfault handling instruction on the SDK InfoCenter page, but I don't think it made a difference. 

Any help would be greatly appreciated, at this point I'm out of ideas.

Thanks in advance,

Max

8863.ble_app_uart_c.zip

[Einar Thorsrud]

Hi,

I see you have logging enabled and HARDFAULT_HANDLER_ENABLED set to 1 in your sdk_config.h (assuming you are using the s132 project). Then we should get a sensible log output when a hardfault occurs. Can you check the log and upload it here?

[maxpmillion]

Good morning! 

I am indeed using the S132 project. Attached is the log from Segger's RTT Viewer. Let me know if that's sufficient.

Edit: I'm not seeing any additional messages regarding the hardfault reg dump in the log file. I'm also not seeing any objects named "hardfault", etc., in the map file. Is there another macro in sdk_config.h that I need to set? Or perhaps something I need to define in the makefile?

Upload1.log

[Einar Thorsrud]

Hi Max,

You should inspect CFSR to know more (see How to debug a HardFault on an ARM Cortex-M MCU). The HardFault handling library should do this for you though, and print it in the log. That is not the case in your log, and looking at your Makefile I see an explanation. It is missing hardfault_handler_gcc.c, which is needed.

PS: I see there are many dropped logs in your log, so perhaps it would be good to increase the log buffers and/or log less (there no reason to think it is related to the hardfault, though).

[maxpmillion]

Ahh thank you, I added the source file to the makefile. And thank you for linking the informative post! I'm going through the section describing how to use the PyCortexMDebug repo to further assess the problem, but I ran into an issue getting the svd_gdb.py script to run. I'll keep trying, but in the meantime, I tried another OTA run with these modifications:

1. The file "hardfault_handler_gcc.c" was added to the makefile
2. The "DEBUG" macro was added to the makefile
3. The macro NRF_LOG_DEFAULT_LEVEL was changed from 4 (debug) to 3 (info) in sdk_config.h

Attached is the log file and map file from this build. I see that it's now printing out more meaningful (at least to me) stack dump information. I have a feeling I'll want to use gdb to discern the line in the source file that's causing the issue, but I also wanted to bring your attention to the line "HARD FAULT at 0x0002C944" in the log file. Cross-referencing the map file, this at least narrows it down to the function "nrf_queue_generic_pop()". Is that information useful at all? Is there something more I can do? Thanks for all your help so far!

Upload2_HardfaultEnabled.log6761.nrf52832_xxaa.map

[Einar Thorsrud]

Hi,

The log show a bus fault at 0x600832F4 which is not within a valid memory range for the nRF52832. Perhaps a buffer overflow somewhere or similar that has corrupted some data so that a pointer points to this invalid memory (0x600832F4 in this case)?

[maxpmillion]

Sorry for the delay. I have a curious development in this case. I asked another member of our software group to try to capture when this issue occurs and get his insight on reading the trace log. Long story short, we found that disabling optimizations (by removing the "-O3" flag from the gcc makefile) alleviated the symptoms. My coworker has set up a debugging environment in Visual Studio Code, and we found that the crash was occurring at line 255 in the attached screenshot. The value of "read_pos" was 0x20040000. Perhaps a dereference operation was not performed correctly? I think our next move will be to try to generate assembly listings to try to gain more insight.

For more data points, we also tried building with IAR, and we saw the crash no matter what the optimization level was (though we're not sure that the crash occurred in the same location in the binary). For now, we at least have a working solution, but we will continue to investigate.

[Einar Thorsrud]

Thanks for keeping me posted. It would be interesting to know the root cause of this.

[maxpmillion]

Hey Einar,

So just to close this loop - it turns out that the root cause is actually not that interesting. You were right in your suspicion that it was a buffer overflow issue. We were mishandling incoming BLE data in the case of OTA updates, and overrunning a buffer. Thanks for all your help and patience! 

[Einar Thorsrud]

Hi Max,

Thanks for letting me know what the root cause turned out to be.