nRF9160 modem hard fault debugging

RE: nRF9160 modem hard fault debugging

Håkon Alseth — Thu, 18 Aug 2022 11:29:28 GMT

Hi,

[quote user="Robert K"]Im not quite sure how to see what Thread that was running[/quote]

As previously mentioned:

[quote user="hkn"]Content of R2, 537000870 (0x2001fba6), is in the RAM and must be checked in the build/zephyr/zephyr.map file manually to see which section/thread/etc that this might be.[/quote]

You can look for any of the faulting registers that has this 0x200XXXXX offset, as this will be in the RAM region.

If you're lucky, you can see it in the call stack trace as well.

Kind regards,

Håkon

RE: nRF9160 modem hard fault debugging

Robert K — Wed, 17 Aug 2022 18:29:07 GMT

Yes this was the latest output,

Im not quite sure how to see what Thread that was running, i'm using VS Code with the nRF Connect Extensions.

If I debug with Ozone and save the snapshot after the fault, would that be of any use?

RE: nRF9160 modem hard fault debugging

Håkon Alseth — Tue, 16 Aug 2022 12:23:39 GMT

Hi,

[quote user="Robert K"]I got R2 hex 0xD this time i ran it, and in the zephyr.map I got 8 occurences for this value:[/quote]

0xd can just be a random value that was used within one scope, it is not a guarantee that it can be resolved back to a specific function or RAM area.

[quote user="Robert K"]

Thank you, this is the output:

LR: ncs/v1.9.1/zephyr/include/arch/arm/aarch32/asm_inline_gcc.h:95
PC: ncs/v1.9.1/zephyr/include/drivers/adc.h:386

[/quote]

Was this output with the latest assert messages? Each time you compile and flash, the content of the CPU registers will change.

If they were, it indicates that you assert when you're trying to access the ADC / unlocking an IRQ:

https://github.com/nrfconnect/sdk-zephyr/blob/v2.7.99-ncs1-1/include/arch/arm/aarch32/asm_inline_gcc.h#L95

https://github.com/nrfconnect/sdk-zephyr/blob/v2.7.99-ncs1/include/drivers/adc.h#L386

Which thread was running? Have you tried debugging to see between which functions it faults? Ie. set a breakpoint, see if it hits, jump to the next function, etc. until it faults.

Kind regards,

Håkon

RE: nRF9160 modem hard fault debugging

Robert K — Mon, 15 Aug 2022 17:48:02 GMT

Thank you, this is the output:

LR: ncs/v1.9.1/zephyr/include/arch/arm/aarch32/asm_inline_gcc.h:95
PC: ncs/v1.9.1/zephyr/include/drivers/adc.h:386

I got R2 hex 0xD this time i ran it, and in the zephyr.map I got 8 occurences for this value:

 .rodata.lwm2m_engine_delete_obj_inst.str1.1
                0x00000000        0xd zephyr/subsys/net/lib/lwm2m/libsubsys__net__lib__lwm2m.a(lwm2m_engine.c.obj)
                
.rodata.str1.1
                0x00058cf4        0xd zephyr/libzephyr.a(stream_flash.c.obj)
                
.rodata.cmd_read.str1.1
                0x0005d6b3        0xd zephyr/drivers/flash/libdrivers__flash.a(flash_shell.c.obj)
                
.rodata.str1.1
                0x0005da85        0xd modules/nrf/lib/nrf_modem_lib/lib..__nrf__lib__nrf_modem_lib.a(nrf91_sockets.c.obj)
                
.rodata.fota_update_counter_update.str1.1
                0x0005fcbd        0xd modules/nrf/subsys/net/lib/lwm2m_client_utils/lib..__nrf__subsys__net__lib__lwm2m_client_utils.a(settings.c.obj)
                
.rodata.str1.1
                0x00060328        0xe modules/nrf/subsys/net/lib/lwm2m_client_utils/lib..__nrf__subsys__net__lib__lwm2m_client_utils.a(lwm2m_device.c.obj)
                                  0xd (size before relaxing)
                                  
                                  
 .rodata.str1.1
                0x00060c47        0xd modules/mcuboot/boot/bootutil/zephyr/libmcuboot_util.a(bootutil_public.c.obj)
                
.rodata._dtoa_r.str1.1
                0x00061de4        0xd c:/Users/Robert/ncs/v1.9.1/toolchain/opt/arm-none-eabi/lib/thumb/v8-m.main/nofp\libc_nano.a(lib_a-dtoa.o)
                                  0xf (size before relaxing)

RE: nRF9160 modem hard fault debugging

Håkon Alseth — Mon, 15 Aug 2022 09:14:27 GMT

Hi,

If you convert the decimal numbers to hex, specifically the contents of LR and PC, you should be able to do a lookup of the addresses using arm-none-eabi-addr2line:

LR: arm-none-eabi-addr2line -e path/to/build/zephyr/zephyr.elf 0x4a17b

PC: arm-none-eabi-addr2line -e path/to/build/zephyr/zephyr.elf 0x27326

Content of R2, 537000870 (0x2001fba6), is in the RAM and must be checked in the build/zephyr/zephyr.map file manually to see which section/thread/etc that this might be.

Could you post the output of the above?

Kind regards,

Håkon

RE: nRF9160 modem hard fault debugging

Robert K — Fri, 12 Aug 2022 15:40:00 GMT

This is what I can extract from when the debugger hits the breakpoint in VS Code:

0x20002ea0 <z_interrupt_stacks+1984>

RE: nRF9160 modem hard fault debugging

Håkon Alseth — Fri, 12 Aug 2022 12:03:30 GMT

Hi,

Could you try one of these solutions?

1. Switch over the UART based logging. This should print any fault scenarios (if present).

2. Enter debug mode, set a breakpoint at the function "z_arm_fatal_error", and if that breakpoint hits: share the content of either "esf_copy" of "esf"?

Kind regards,

Håkon

RE: nRF9160 modem hard fault debugging

Robert K — Fri, 12 Aug 2022 09:39:45 GMT

Hi, thank you for your reaponse.

I run RTT Viewer and it does not show any logs when the device resets. I only see my debug prints which prints just before the above mentioned function.

RE: nRF9160 modem hard fault debugging

Håkon Alseth — Fri, 12 Aug 2022 08:51:55 GMT

Hi,

Do you have a log or similar showing what happens?

If a fault condition occurs, you'll usually see a printout on the terminal.

Kind regards,

Håkon