Trouble viewing hard faults

RE: Trouble viewing hard faults

Didrik Rokhaug — Fri, 26 Aug 2022 12:55:46 GMT

These settings should work for the flash_primary region:

mcuboot:
    address: 0x0
    end_address: 0xc000
    placement:
      before:
      - mcuboot_primary
    region: flash_primary
    size: 0xc000
mcuboot_primary:
  address: 0xc000
  end_address: 0xee000
  orig_span: &id001
  - spm
  - app
  - mcuboot_pad
  region: flash_primary
  size: 0xe2000
  span: *id001
mcuboot_pad:
  address: 0xc000
  end_address: 0xc200
  placement:
    before:
    - mcuboot_primary_app
  region: flash_primary
  size: 0x200
mcuboot_primary_app:
  address: 0xc200
  end_address: 0xee000
  orig_span: &id002
  - app
  - spm
  region: flash_primary
  size: 0xe1e00
  span: *id002
spm:
  address: 0xc200
  end_address: 0x1c200
  inside:
  - mcuboot_primary_app
  placement:
    before:
    - app
  region: flash_primary
  size: 0x10000
app:
  address: 0x1c200
  end_address: 0xee000
  region: flash_primary
  size: 0xd1e00
memfault_storage:
  address: 0xee000
  align:
    start: 0x1000
  end_address: 0xfa000
  placement:
    before:
    - settings_storage
  region: flash_primary
  size: 0xc000
lwm2m_carrier:
  address: 0xfa000
  size: 0x3000
settings_storage:
    address: 0xfe000
    end_address: 0x100000
    placement:
      before:
      - end
    region: flash_primary
    size: 0x2000

(note that I have not been able to actually test them myself)

I had to shrink the memfault_storage partition to make room for the lwm2m_carrier partition. To make sure that the Memfault library uses the new size, you should probably also set CONFIG_PM_PARTITION_SIZE_MEMFAULT_STORAGE=0xc000 in your prj.conf. Shrinking the partition should not cause any problems, bu you might get less coredump data sent to memfault. You can avoid that by storing the coredump in no init RAM instead, similar to how it is done in this commit: https://github.com/simensrostad/fw-nrfconnect-nrf/commit/a366a615b16d93a961b94e9aec3873036d38c61e#

[quote user="esisk"]is there a compatibility matrix for what version of NCS works with which version of the MFW?[/quote]

This matrix shows which NCS version the various modem FW versions have been tested with: https://infocenter.nordicsemi.com/topic/comp_matrix_nrf9160/COMP/nrf9160/nrf9160_modem_fw.html

However, when dealing with carrier certifications (and especially Verizon), you have to use the combinations listed here: https://infocenter.nordicsemi.com/topic/comp_matrix_nrf9160/COMP/nrf9160/nrf9160_operator_certifications.html

RE: Trouble viewing hard faults

esisk — Thu, 25 Aug 2022 14:07:19 GMT

I've been having trouble updating devices via FOTAs due to the partitions already in place on those devices. I can probably figure out how to swing this. Is there a way to update the carrier lib without updating the version of NCS that I'm using? Also, I know there's a compatibility matrix for what MFW is approved by various carriers, but is there a compatibility matrix for what version of NCS works with which version of the MFW? It'd be great to upgrade from 1.7.0 to 2.x.x. We got our devices certified by Verizon on MFW v1.3.0

RE: Trouble viewing hard faults

Didrik Rokhaug — Thu, 25 Aug 2022 14:02:46 GMT

Hi, and sorry for the late reply.

First, it is good to hear that you managed to solve your original problem.

Your new problem is as you say caused by the carrier_lib not using the Partition Manager. This has been changed in version 0.30.0.

https://developer.nordicsemi.com/nRF_Connect_SDK/doc/latest/nrf/libraries/bin/lwm2m_carrier/CHANGELOG.html#liblwm2m-carrier-0-30-0

In the changelog, it is also explained how you can make the Partition Manager partition match the device tree partition:

To use the legacy NVS partition, you can add a pm_static.yml file to your project with the following content:

lwm2m_carrier:
  address: 0xfa000
  size: 0x3000
free:
  address: 0xfd000
  size: 0x3000

Best regards,

Didrik

RE: Trouble viewing hard faults

esisk — Wed, 17 Aug 2022 13:11:44 GMT

I ended up figuring it out. After some help from Memfault support, I got pointed in the right direction

"Ah, as far as the weird behavior around coredump capture, you've unfortunately run into a bug that's been since fixed in the Memfault SDK. You'll have to update the Memfault SDK module, here's some instructions:
https://docs.memfault.com/docs/mcu/nrf-connect-sdk-guide/#updating-the-memfault-sdk

The fix was made in v0.30.1 of the Memfault SDK:
https://github.com/memfault/memfault-firmware-sdk/blob/36a634536d2cd3c455545f0b97c926275772153c/CHANGES.md#changes-between-memfault-sdk-0301-and-sdk-0300---april-6-2022"

So if you're having the same issue, follow the steps there.

Also, another extremely nuanced error that I experienced once I fixed that. The coredump was getting generated and saved successfully to the 9160's internal flash; however, upon reboot, the LWM2M library was exhibiting undefined behavior when connecting to the network (i.e. LWM2M_EVENT_BOOTSTRAPPED then LWM2M_EVENT_DISCONNECTED over and over). I remembered a limitation of the library listed here Requirements and application limitations — nRF Connect SDK 1.7.0 documentation (nordicsemi.com). Specifically,

"The LwM2M carrier library uses the following NVS record key range: 0xCA00 - 0xCAFF. This range must not be used by the application."

After looking at partitions.yml and lwm2m_os.c, I could see that the partition generated for the coredump was in the same range as the storage partition listed in my board file, and lwm2m_os.c was using the storage partition for its file system. I'm a little confused as to why LWM2M is using this old partition from the board file, since I was under the impression that those definitions were deprecated in favor of the partition manager. Anyway, sure it's just a remnant of transitioning. Hope this helps someone else.

Edit: It'd also be nice if you could help me configure this partition file since it is a challenge. I actually discovered that the Settings storage is configured to go over the LWM2M partition as well which could lead to some weird behavior. Here's my dynamic partition file. I already have a static configuration file for my external flash but need a little help altering the primary flash partitions.

alerts:
  address: 0x2fc000
  device: MX25L3
  end_address: 0x31c000
  placement:
    after:
    - biometrics
  region: external_flash
  size: 0x20000
app:
  address: 0x1c200
  end_address: 0xee000
  region: flash_primary
  size: 0xd1e00
biometrics:
  address: 0xfc000
  device: MX25L3
  end_address: 0x2fc000
  placement:
    after:
    - indices
  region: external_flash
  size: 0x200000
external_flash:
  address: 0x400000
  end_address: 0x400000
  region: external_flash
  size: 0x0
gps:
  address: 0x31c000
  device: MX25L3
  end_address: 0x350000
  placement:
    after:
    - alerts
  region: external_flash
  size: 0x34000
indices:
  address: 0xf2000
  device: MX25L3
  end_address: 0xfc000
  placement:
    after:
    - mcuboot_secondary
  region: external_flash
  size: 0xa000
logging:
  address: 0x3a1000
  device: MX25L3
  end_address: 0x3b0000
  placement:
    after:
    - r2ws
  region: external_flash
  size: 0xf000
mcuboot:
  address: 0x0
  end_address: 0xc000
  placement:
    before:
    - mcuboot_primary
  region: flash_primary
  size: 0xc000
mcuboot_pad:
  address: 0xc000
  end_address: 0xc200
  placement:
    before:
    - mcuboot_primary_app
  region: flash_primary
  size: 0x200
mcuboot_primary:
  address: 0xc000
  end_address: 0xee000
  orig_span: &id001
  - spm
  - app
  - mcuboot_pad
  region: flash_primary
  size: 0xe2000
  span: *id001
mcuboot_primary_app:
  address: 0xc200
  end_address: 0xee000
  orig_span: &id002
  - app
  - spm
  region: flash_primary
  size: 0xe1e00
  span: *id002
mcuboot_secondary:
  address: 0x0
  device: MX25L3
  end_address: 0xf2000
  placement:
    align:
      start: 0x0
  region: external_flash
  size: 0xf2000
memfault_storage:
  address: 0xee000
  align:
    start: 0x1000
  end_address: 0xfe000
  placement:
    before:
    - settings_storage
  region: flash_primary
  size: 0x10000
nrf_modem_lib_ctrl:
  address: 0x20010000
  end_address: 0x200104e8
  inside:
  - sram_nonsecure
  placement:
    after:
    - spm_sram
    - start
  region: sram_primary
  size: 0x4e8
nrf_modem_lib_rx:
  address: 0x200124e8
  end_address: 0x200144e8
  inside:
  - sram_nonsecure
  placement:
    after:
    - nrf_modem_lib_tx
  region: sram_primary
  size: 0x2000
nrf_modem_lib_sram:
  address: 0x20010000
  end_address: 0x200144e8
  orig_span: &id003
  - nrf_modem_lib_ctrl
  - nrf_modem_lib_tx
  - nrf_modem_lib_rx
  region: sram_primary
  size: 0x44e8
  span: *id003
nrf_modem_lib_tx:
  address: 0x200104e8
  end_address: 0x200124e8
  inside:
  - sram_nonsecure
  placement:
    after:
    - nrf_modem_lib_ctrl
  region: sram_primary
  size: 0x2000
otp:
  address: 0xff8108
  end_address: 0xff83fc
  region: otp
  size: 0x2f4
r2ws:
  address: 0x360000
  device: MX25L3
  end_address: 0x3a1000
  placement:
    after:
    - statuses
  region: external_flash
  size: 0x41000
settings_storage:
  address: 0xfe000
  end_address: 0x100000
  placement:
    before:
    - end
  region: flash_primary
  size: 0x2000
spm:
  address: 0xc200
  end_address: 0x1c200
  inside:
  - mcuboot_primary_app
  placement:
    before:
    - app
  region: flash_primary
  size: 0x10000
spm_sram:
  address: 0x20000000
  end_address: 0x20010000
  inside:
  - sram_secure
  placement:
    after:
    - start
  region: sram_primary
  size: 0x10000
sram_nonsecure:
  address: 0x20010000
  end_address: 0x20040000
  orig_span: &id004
  - sram_primary
  - nrf_modem_lib_ctrl
  - nrf_modem_lib_tx
  - nrf_modem_lib_rx
  region: sram_primary
  size: 0x30000
  span: *id004
sram_primary:
  address: 0x200144e8
  end_address: 0x20040000
  region: sram_primary
  size: 0x2bb18
sram_secure:
  address: 0x20000000
  end_address: 0x20010000
  orig_span: &id005
  - spm_sram
  region: sram_primary
  size: 0x10000
  span: *id005
statuses:
  address: 0x350000
  device: MX25L3
  end_address: 0x360000
  placement:
    after:
    - gps
  region: external_flash
  size: 0x10000
unused:
  address: 0x3b0000
  device: MX25L3
  end_address: 0x400000
  placement:
    after:
    - logging
  region: external_flash
  size: 0x50000

RE: Trouble viewing hard faults

esisk — Tue, 16 Aug 2022 12:15:58 GMT

Hey Didrik, I should've clarified: I'm having trouble viewing the faults locally. I already have a proxy service forwarding the Memfault chunks that I'm sending over an MQTT connection to the Memfault cloud. I can view trace events and metrics just fine if I upload my symbol file. After I created this ticket yesterday, I did some further digging. I believe that the Memfault fault handler (which wraps around one of the Zephyr ones) was just not getting called because I did not have CONFIG_ASSERT=y as per https://docs.memfault.com/docs/mcu/zephyr-guide#zephyr-__assert Now, the handler is getting called; however, may have some lower level issue. I notice that everything is working fine until here:

void __wrap_z_fatal_error(unsigned int reason, const z_arch_esf_t *esf) {
  const struct __extra_esf_info *extra_info = &esf->extra_info;
  const _callee_saved_t *callee_regs = extra_info->callee;

  const uint32_t exc_return = extra_info->exc_return;
  const bool msp_was_active = (exc_return & (1 << 3)) == 0;

  sMfltRegState reg = {
    .exception_frame = (void *)(msp_was_active ? extra_info->msp : callee_regs->psp),
    .r4 = callee_regs->v1,
    .r5 = callee_regs->v2,
    .r6 = callee_regs->v3,
    .r7 = callee_regs->v4,
    .r8 = callee_regs->v5,
    .r9 = callee_regs->v6,
    .r10 = callee_regs->v7,
    .r11 = callee_regs->v8,
    .exc_return = exc_return ,
  };
  memfault_fault_handler(&reg, kMfltRebootReason_HardFault);
}

Specifically, the definition of the struct sMfltRegState reg. Upon further inspection, callee_regs is NULL so dereferencing them triggers another fatal error funnily enough. And since the Memfault handler has taken over, the 9160 is never reset. What I'm curious about is how is callee_regs NULL?

RE: Trouble viewing hard faults

Didrik Rokhaug — Tue, 16 Aug 2022 12:02:18 GMT

Hi,

Is your AWS integrated with Memfault?

As CONFIG_MEMFAULT_HTTP_ENABLE is disabled, the device will not upload the data directly to Memfault itself, instead you have to forward the data via something else (e.g. as explained here).

Best regards,

Didrik