Problem executing simple program on Thingy91

[AdBee]

Hello,

I recently acquired a Thingy91 board in order to make a hardware prototype. I already have experience developing with the nRF52840 so I took inspiration from my working project.

First I flashed the straight out "Hello World" sample from the v2.9.0 SDK to the nRF9160 using a J-Link external probe and the "west flash" command with success. I could launch the program and read "Hello World" from a serial terminal.

Then I tried to create a basic setup project by copying what worked on my nRF52840 project. These basic functionalities includes:

_ Versionning

_ Using sysbuild with MCUBOOT

_ Signing firmware with an ECDSA 256 key.

The project successfully built and flashed, but unfortunately, didn't execute. No output can be read from the serial terminal anymore, not even the early logs from the bootloader.

What's a bit worrisome is that I can't manage to get back to a state where things worked. When I try to flash the basic "Hello World" sample again, the program doesn't launch anymore (no output on the serial terminal).

Any idea what could be going on and how to fix this ? I tried switching to the most recent v3.0.1 SDK but with no success.

I'm adding the .tar.gz of the project.

thingy_minimal.tar.gz

Here is my build configuration:

And the output when I flash:

Flashing build_prod to 801051622
west flash -d /home/abourgaud/repos/haugenplus_official/user-node/application/build_prod --dev-id 801051622

-- west flash: rebuilding
[0/6] Performing build step for 'application'
ninja: no work to do.
[1/6] Performing build step for 'mcuboot'
ninja: no work to do.
[3/6] Completed 'application'
[5/6] No install step for 'mcuboot'
[6/6] Completed 'mcuboot'
WARNING: Specifying runner options for multiple domains is experimental.
If problems are experienced, please specify a single domain using '--domain <domain>'
-- west flash: using runner nrfjprog
-- runners.nrfjprog: Flashing file: /home/abourgaud/repos/haugenplus_official/user-node/application/build_prod/mcuboot/zephyr/zephyr.hex
[ ######               ]   0.000s | Erase file - Check image                               [ ####                 ]   0.000s | Check image validity - Initialize device info          [ ########             ]   0.000s | Check image validity - Check region 0 settings         [ ############         ]   0.000s | Check image validity - block 1 of 3                    [ ################     ]   0.007s | Check image validity - block 2 of 3                    [ #################### ]   0.028s | Check image validity - Finished                        [ #############        ]   0.000s | Erase file - Erasing                                   [ ######               ]   0.000s | Erasing non-volatile memory - block 1 of 2             [ #############        ]   0.553s | Erasing non-volatile memory - block 2 of 2             [ #################### ]   0.000s | Erasing non-volatile memory - Erase successful         [ #################### ]   1.291s | Erase file - Done erasing                                                          
[ ######               ]   0.000s | Program file - Checking image                          [ ####                 ]   0.000s | Check image validity - Initialize device info          [ ########             ]   0.000s | Check image validity - Check region 0 settings         [ ############         ]   0.963s | Check image validity - block 1 of 3                    [ ################     ]   0.361s | Check image validity - block 2 of 3                    [ #################### ]   0.004s | Check image validity - Finished                        [ #############        ]   0.000s | Program file - Programming                             [ ######               ]   0.000s | Programming image - block 1 of 2                       [ #############        ]   0.147s | Programming image - block 2 of 2                       [ #################### ]   0.000s | Programming image - Write successful                   [ #################### ]   0.330s | Program file - Done programming                                                    
[ ######               ]   0.000s | Verify file - Check image                              [ ####                 ]   0.000s | Check image validity - Initialize device info          [ ########             ]   0.000s | Check image validity - Check region 0 settings         [ ############         ]   1.224s | Check image validity - block 1 of 3                    [ ################     ]   0.459s | Check image validity - block 2 of 3                    [ #################### ]   0.004s | Check image validity - Finished                        [ #############        ]   0.000s | Verify file - Verifying                                [ ######               ]   0.000s | Verifying image - block 1 of 2                         [ #############        ]   0.144s | Verifying image - block 2 of 2                         [ #################### ]   0.000s | Verifying image - Verify successful                    [ #################### ]   0.304s | Verify file - Done verifying                                                       
-- runners.nrfjprog: Board with serial number 801051622 flashed successfully.
-- west flash: using runner nrfjprog
-- runners.nrfjprog: reset after flashing requested
-- runners.nrfjprog: Flashing file: /home/abourgaud/repos/haugenplus_official/user-node/application/build_prod/application/zephyr/zephyr.signed.hex
[ ######               ]   0.000s | Erase file - Check image                               [ #####                ]   0.000s | Check image validity - Initialize device info          [ ##########           ]   0.000s | Check image validity - Check region 0 settings         [ ###############      ]   0.000s | Check image validity - block 1 of 2                    [ #################### ]   0.028s | Check image validity - Finished                        [ #############        ]   0.000s | Erase file - Erasing                                   [ ##########           ]   0.000s | Erasing non-volatile memory - block 1 of 1             [ #################### ]   0.000s | Erasing non-volatile memory - Erase successful         [ #################### ]   1.510s | Erase file - Done erasing                                                          
[ ######               ]   0.000s | Program file - Checking image                          [ #####                ]   0.000s | Check image validity - Initialize device info          [ ##########           ]   0.000s | Check image validity - Check region 0 settings         [ ###############      ]   0.531s | Check image validity - block 1 of 2                    [ #################### ]   0.002s | Check image validity - Finished                        [ #############        ]   0.000s | Program file - Programming                             [ ##########           ]   0.000s | Programming image - block 1 of 1                       [ #################### ]   0.000s | Programming image - Write successful                   [ #################### ]   0.402s | Program file - Done programming                                                    
[ ######               ]   0.000s | Verify file - Check image                              [ #####                ]   0.000s | Check image validity - Initialize device info          [ ##########           ]   0.000s | Check image validity - Check region 0 settings         [ ###############      ]   0.685s | Check image validity - block 1 of 2                    [ #################### ]   0.002s | Check image validity - Finished                        [ #############        ]   0.000s | Verify file - Verifying                                [ ##########           ]   0.000s | Verifying image - block 1 of 1                         [ #################### ]   0.000s | Verifying image - Verify successful                    [ #################### ]   0.402s | Verify file - Done verifying                                                       
Applying system reset.
Run.
-- runners.nrfjprog: Board with serial number 801051622 flashed successfully.

Thank you for your time,

AdBee

[Øyvind]

Hello, 

The Nordic Thingy:91 USB connector is connected to the USB interface of the nRF52840 SoC. This enables PC communication and battery charging. The nRF52840 comes preprogrammed with the Connectivity Bridge application which  acts as a USB composite device, exposing two UART interfaces to a USB host as two CDC ACM devices.

Did you reprogram the nRF52840? I.e. build for thingy91/nrf52840?

Also, I see that you are building you project for the thingy91/nrf9160 and not the thingy91/nrf9160/ns. The latter is needed in order to communicate with the nRF91 modem from the application core. See Developing with the Thingy:91 for more information.

Kind regards,
Øyvind

[AdBee]

Hello,

Thank you for your fast reply.

I think by using the thingy91/nrf9160 board instead of the thingy/nrf9160/ns, I may have overwritten some useful default partitions.

Following your link, I reinstalled all default images using the nRFConnect for Desktop, reflashed my own code with the thingy/nrf9160/ns board and it appears that it solved my problem.

I have further questions though:

1/ When I switch off/on the device using the SW1 switch, my program doesn't restart from scratch as expected, but rather seems to resume to where it was. How is it ? Is there a bootloader configuration that somehow saves PC and other registers ? Is it possible to change this behavior so that my program actually restarts at every power on ? It is unclear to me how the peripherals behave when their parameters are not explicitly set by my init functions.

2/ With regards to the partitions: I am used to customize them in a pm_static file to enable features such as nvs or BLE settings. But with this "ns" setup it seems that a lot of things are done under the hood that I shouldn't break. How can I customize partitions in this environment ?

Thank you for your time,

AdBee

[Øyvind]

Hello, my apologies for the late reply. 

AdBee said:

The correct way to import it is to call #include <nrf_model_at.h>. I would suggest updating this doc and this one which are misleading about it.

Yes, I see what you man, which can be somewhat confusing. The best way is often to look at our samples in our SDK, but of course the documentation should reflect the code base. I will report internally.

AdBee said:

I am trying to interact with the LTE-M/GNSS module by using AT-commands.

Have you successfully been able to send AT commands programmatically? 

AdBee said:

Does the existence of the later file means that pm_static files are not used by default, but only if configs such as CONFIG_THINGY91_STATIC_PARTITIONS_FACTORY are provided ?

The build system should pick up on the file automatically. Have a look at Configuring static partitions in the partition manager. This is a KConfig symbol defined in nrf/boards/nordic/thingy91/Kconfig

config THINGY91_STATIC_PARTITIONS_FACTORY
	bool "Factory Thingy:91 partition layout"
	help
	   The default Thingy:91 partition layout used in the factory firmware. This ensures
	   firmware updates are compatible with Thingy:91 when flashing firmware over USB or over
	   the air.

Kind regards,
Øyvind

[AdBee]

Yes, I see what you man, which can be somewhat confusing. The best way is often to look at our samples in our SDK, but of course the documentation should reflect the code base. I will report internally.

I don't think there is a sample to demonstrate this functionnality, or at least I didn't find it.

Have you successfully been able to send AT commands programmatically? 

Yes with the nrf_modem_at_printf() function and the correct include it appears to work.

The only remaining question I have is about the absence of early logs. I assume it is because the hook that redirects outputs from the nRF9160 to the nRF52840 serial is set after some delay rather than at program start. Correct ?

Thank you for your support,

AdBee

[AdBee]

Hello,

I have been trying to implement a custom static partition following your leads, but with no success.

First I ran west build -t partition_manager_report with the default thingy/nrf9160/ns board:

  flash_primary (0x100000 - 1024kB): 
+-------------------------------------------------+
| 0x0: mcuboot (0xc000 - 48kB)                    |
+---0xc000: mcuboot_primary (0x69000 - 420kB)-----+
+---0xc000: tfm_secure (0xc000 - 48kB)------------+
| 0xc000: mcuboot_pad (0x200 - 512B)              |
+---0xc200: mcuboot_primary_app (0x68e00 - 419kB)-+
| 0xc200: tfm (0xbe00 - 47kB)                     |
| 0x18000: app (0x5d000 - 372kB)                  |
+---0x18000: tfm_nonsecure (0x5b000 - 364kB)------+
| 0x75000: mcuboot_secondary (0x69000 - 420kB)    |
| 0xde000: mcuboot_scratch (0x1e000 - 120kB)      |
| 0xfc000: EMPTY_0 (0x2000 - 8kB)                 |
+---0xfe000: nonsecure_storage (0x2000 - 8kB)-----+
| 0xfe000: settings_storage (0x2000 - 8kB)        |
+-------------------------------------------------+

  otp (0x2f4 - 756B): 
+------------------------------+
| 0xff8108: otp (0x2f4 - 756B) |
+------------------------------+

  sram_primary (0x40000 - 256kB): 
+--------------------------------------------------+
+---0x20000000: mcuboot_sram (0x8000 - 32kB)-------+
+---0x20000000: sram_secure (0x8000 - 32kB)--------+
| 0x20000000: tfm_sram (0x8000 - 32kB)             |
+---0x20008000: sram_nonsecure (0x38000 - 224kB)---+
+---0x20008000: nrf_modem_lib_sram (0x4568 - 17kB)-+
| 0x20008000: nrf_modem_lib_ctrl (0x4e8 - 1kB)     |
| 0x200084e8: nrf_modem_lib_tx (0x2080 - 8kB)      |
| 0x2000a568: nrf_modem_lib_rx (0x2000 - 8kB)      |
+--------------------------------------------------+
| 0x2000c568: sram_primary (0x33a98 - 206kB)       |
+--------------------------------------------------+

So far so good.

Then  tried to put a custom pm_static.yml file in my project directory, and following your doc I put

CONFIG_THINGY91_NO_PREDEFINED_LAYOUT=y

in my prj.conf file.

Running again west build -t partition_manager_report, I found the exact same result than previously, meaning that my pm_static.yml file was not taken into account.

I tried to move

CONFIG_THINGY91_NO_PREDEFINED_LAYOUT=y

to my sysbuild/mcuboot.conf file with no greater success.

I also added

SB_CONFIG_PARTITION_MANAGER=y

to my sysbuild.conf file which didn't help.

Lastly, I tried to switch the board from thingy91/nrf9160/ns to thingy91/nrf9160 but at this point the project didn't even compile (the <nrf_modem.h> library that I referenced for AT commands was not found)

What am I supposed to do ?

Ty,

AdBee

[Øyvind]

Hello AdBee, 

My apologies for the late reply. 

AdBee said:

I don't think there is a sample to demonstrate this functionnality, or at least I didn't find it.

No, there is no specific sample for this, but if you look at the samples that are using e.g. nrf_modem_at_printf() they will show the correct way to include the files. 

This is due to the build system and how CMakeLists.txt work in the background, removing parts of the path when found in a "include" folder. That said, the documentation for the AT commands should show how to use the AT commands.

AdBee said:

Then  tried to put a custom pm_static.yml file in my project directory, and following your doc I put

Did you set CONFIG_THINGY91_STATIC_PARTITIONS_FACTORY=n along with CONFIG_THINGY91_NO_PREDEFINED_LAYOUT=y?

Could you share your pm_static file?

[AdBee]

Hello,

Did you set CONFIG_THINGY91_STATIC_PARTITIONS_FACTORY=n along with CONFIG_THINGY91_NO_PREDEFINED_LAYOUT=y?

I didn't indeed. Doing this seems to have troubleshot the problem, thank you.

Running  west build -t partition_manager_report gets me:

flash_primary (0x100000 - 1024kB): 
+-------------------------------------------------+
| 0x0: mcuboot (0xc000 - 48kB)                    |
+---0xc000: mcuboot_primary (0x69000 - 420kB)-----+
+---0xc000: tfm_secure (0xc000 - 48kB)------------+
| 0xc000: mcuboot_pad (0x200 - 512B)              |
+---0xc200: mcuboot_primary_app (0x68e00 - 419kB)-+
| 0xc200: tfm (0xbe00 - 47kB)                     |
| 0x18000: app (0x5d000 - 372kB)                  |
+---0x18000: tfm_nonsecure (0x5b000 - 364kB)------+
| 0x75000: mcuboot_secondary (0x69000 - 420kB)    |
| 0xde000: user_partition (0x1e000 - 120kB)       |
| 0xfc000: EMPTY_0 (0x2000 - 8kB)                 |
+---0xfe000: nonsecure_storage (0x2000 - 8kB)-----+
| 0xfe000: settings_storage (0x2000 - 8kB)        |
+-------------------------------------------------+

  otp (0x2f4 - 756B): 
+------------------------------+
| 0xff8108: otp (0x2f4 - 756B) |
+------------------------------+

  sram_primary (0x40000 - 256kB): 
+--------------------------------------------------+
+---0x20000000: mcuboot_sram (0x8000 - 32kB)-------+
+---0x20000000: sram_secure (0x8000 - 32kB)--------+
| 0x20000000: tfm_sram (0x8000 - 32kB)             |
+---0x20008000: sram_nonsecure (0x38000 - 224kB)---+
+---0x20008000: nrf_modem_lib_sram (0x4568 - 17kB)-+
| 0x20008000: nrf_modem_lib_ctrl (0x4e8 - 1kB)     |
| 0x200084e8: nrf_modem_lib_tx (0x2080 - 8kB)      |
| 0x2000a568: nrf_modem_lib_rx (0x2000 - 8kB)      |
+--------------------------------------------------+
| 0x2000c568: sram_primary (0x33a98 - 206kB)       |
+--------------------------------------------------+

At heart, I took the default partitions and added a custom one for NVS that I called "user_partition". Since a scratch partition for MCUBOOT shouldn't be absolutely needed depending on the FOTA swap policy, I decided to remove it to make room for "user_partition".

I'm now getting this error at runtime when trying to mount the NVS partition:

[00:00:00.391,296] <err> os: ***** SECURE FAULT *****
[00:00:00.397,033] <err> os:   Address: 0xe0ff8
[00:00:00.402,313] <err> os:   Attribution unit violation
[00:00:00.408,477] <err> os: r0/a1:  0x20011518  r1/a2:  0x000e0ff8  r2/a3:  0x000e1000
[00:00:00.417,236] <err> os: r3/a4:  0x20011517 r12/ip:  0x0001cad5 r14/lr:  0x0001cb23
[00:00:00.425,994] <err> os:  xpsr:  0x81000000
[00:00:00.431,274] <err> os: Faulting instruction address (r15/pc): 0x0002de86
[00:00:00.439,239] <err> os: >>> ZEPHYR FATAL ERROR 41: Unknown error on CPU 0
[00:00:00.447,204] <err> os: Current thread: 0x2000d318 (unknown)
[00:00:00.454,040] <err> os: Halting system

which is pretty explicit about me violating the NS and/or trusted Firmware setup. Since I'm not very familiar with the concept, I am currently going through the documentation to see how I could fix this.

Have a good day.

[AdBee]

I solved my problem by moving my "user_partition" to the "nonsecure_storage" sector. My partition now looks like this:

  flash_primary (0x100000 - 1024kB): 
+-------------------------------------------------+
| 0x0: mcuboot (0xc000 - 48kB)                    |
+---0xc000: mcuboot_primary (0x69000 - 420kB)-----+
+---0xc000: tfm_secure (0xc000 - 48kB)------------+
| 0xc000: mcuboot_pad (0x200 - 512B)              |
+---0xc200: mcuboot_primary_app (0x68e00 - 419kB)-+
| 0xc200: tfm (0xbe00 - 47kB)                     |
+---0x18000: tfm_nonsecure (0x5b000 - 364kB)------+
| 0x18000: app (0x50000 - 320kB)                  |
| 0x68000: nvs_storage (0x6000 - 24kB)            |
| 0x75000: mcuboot_secondary (0x69000 - 420kB)    |
| 0xde000: EMPTY_0 (0x2000 - 8kB)                 |
+---0xe0000: nonsecure_storage (0x20000 - 128kB)--+
| 0xe0000: user_partition (0x1e000 - 120kB)       |
| 0xfe000: settings_storage (0x2000 - 8kB)        |
+-------------------------------------------------+

  otp (0x2f4 - 756B): 
+------------------------------+
| 0xff8108: otp (0x2f4 - 756B) |
+------------------------------+

  sram_primary (0x40000 - 256kB): 
+--------------------------------------------------+
+---0x20000000: mcuboot_sram (0x8000 - 32kB)-------+
+---0x20000000: sram_secure (0x8000 - 32kB)--------+
| 0x20000000: tfm_sram (0x8000 - 32kB)             |
+---0x20008000: sram_nonsecure (0x38000 - 224kB)---+
+---0x20008000: nrf_modem_lib_sram (0x4568 - 17kB)-+
| 0x20008000: nrf_modem_lib_ctrl (0x4e8 - 1kB)     |
| 0x200084e8: nrf_modem_lib_tx (0x2080 - 8kB)      |
| 0x2000a568: nrf_modem_lib_rx (0x2000 - 8kB)      |
+--------------------------------------------------+
| 0x2000c568: sram_primary (0x33a98 - 206kB)       |
+--------------------------------------------------+

I am still a bit confused about some partitions though, could you explain them to me ?

First I'd like to know why there is a "nvs_storage" partition which appeared in the tfm_nonsecure sector. I didn't explicitly requested this, and this is not the partition that is being mounted on my system (it is "user_partition" which works fine).

Secondly I'd like to know what the EMPTY_0 partition is for. This partition is set by default in the partitions.pm file so I kept it in my custom pm_static.yml file, but I'd like to why it's there.

Here is my pm_static.yml file:

app: {address: 0x18000, size: 0x5b000}
mcuboot:
  address: 0x0
  placement:
    before: [mcuboot_primary]
  size: 0xc000
mcuboot_pad:
  address: 0xc000
  placement:
    align: {start: 0x1000}
    before: [mcuboot_primary_app]
  size: 0x200
mcuboot_primary:
  address: 0xc000
  size: 0x69000
  span: [tfm, mcuboot_pad, app]
mcuboot_primary_app:
  address: 0xc200
  size: 0x68e00
  span: [app, tfm]
mcuboot_secondary:
  address: 0x75000
  placement:
    after: [mcuboot_primary]
    align: {start: 0x1000}
  share_size: [mcuboot_primary]
  size: 0x69000
EMPTY_0:
  address: 0xde000
  size: 0x2000
user_partition:
  address: 0xe0000
  placement:
    after: [EMPTY_0]
    align: {start: 0x1000}
  size: 0x1e000
settings_storage:
  address: 0xfe000
  placement:
    after: [user_partition]
  size: 0x2000
nonsecure_storage:
  address: 0xe0000
  size: 0x20000
  span: [user_partition, settings_storage]
tfm_secure:
  address: 0xc000
  size: 0xc000
  span: [mcuboot_pad, tfm]
tfm_nonsecure:
  address: 0x18000
  size: 0x5b000
  span: [app]
tfm:
  address: 0xc200
  size: 0xbe00

mcuboot_sram:
  address: 0x20000000
  end_address: 0x20008000
  orig_span: &id001
  - tfm_sram
  region: sram_primary
  size: 0x8000
  span: *id001
sram_secure:
  address: 0x20000000
  end_address: 0x20008000
  orig_span: &id004
  - tfm_sram
  region: sram_primary
  size: 0x8000
  span: *id004
tfm_sram:
  address: 0x20000000
  end_address: 0x20008000
  inside:
  - sram_secure
  placement:
    after:
    - start
  region: sram_primary
  size: 0x8000
sram_nonsecure:
  address: 0x20008000
  end_address: 0x20040000
  orig_span: &id003
  - sram_primary
  - nrf_modem_lib_ctrl
  - nrf_modem_lib_tx
  - nrf_modem_lib_rx
  region: sram_primary
  size: 0x38000
  span: *id003
nrf_modem_lib_sram:
  address: 0x20008000
  end_address: 0x2000c568
  orig_span: &id002
  - nrf_modem_lib_ctrl
  - nrf_modem_lib_tx
  - nrf_modem_lib_rx
  region: sram_primary
  size: 0x4568
  span: *id002
nrf_modem_lib_ctrl:
  address: 0x20008000
  end_address: 0x200084e8
  inside:
  - sram_nonsecure
  placement:
    after:
    - tfm_sram
    - start
  region: sram_primary
  size: 0x4e8
nrf_modem_lib_tx:
  address: 0x200084e8
  end_address: 0x2000a568
  inside:
  - sram_nonsecure
  placement:
    after:
    - nrf_modem_lib_ctrl
  region: sram_primary
  size: 0x2080
nrf_modem_lib_rx:
  address: 0x2000a568
  end_address: 0x2000c568
  inside:
  - sram_nonsecure
  placement:
    after:
    - nrf_modem_lib_tx
  region: sram_primary
  size: 0x2000
sram_primary:
  address: 0x2000c568
  end_address: 0x20040000
  region: sram_primary
  size: 0x33a98

Relevant prj.conf configurations:

CONFIG_FLASH=y
CONFIG_FLASH_PAGE_LAYOUT=y
CONFIG_FLASH_MAP=y
CONFIG_NVS=y

Thank you,

AdBee