AWS IoT example getting RSRP results in HARD FAULT

RE: AWS IoT example getting RSRP results in HARD FAULT

Martin Lesund — Wed, 30 Sep 2020 12:07:09 GMT

It is also now fixed in NCS master: https://github.com/nrfconnect/sdk-nrf/pull/2999

RE: AWS IoT example getting RSRP results in HARD FAULT

Elith — Wed, 30 Sep 2020 07:15:20 GMT

Perfect, thank you very much. I just found out myself that the problem was the sys_rand32_get().

Also a great tip with the sched_lock() functionality - that will be useful for other things I imagine 🙂

RE: AWS IoT example getting RSRP results in HARD FAULT

Martin Lesund — Wed, 30 Sep 2020 06:55:29 GMT

Hi Daniel,

Please try to change sys_rand32_get() to k_uptime_get_32() in this line.

Another option is to run k_sched_lock(), k_sched_unlock() around aws_iot_send(), but I recommend the first suggestion.

RE: AWS IoT example getting RSRP results in HARD FAULT

Elith — Wed, 30 Sep 2020 06:43:10 GMT

Alright, newest status this morning is: Completely new aws_iot project yields:

*** Booting Zephyr OS build v2.3.0-rc1-ncs1-2410-g7d20f2ebf259  ***
Flash regions           Domain          Permissions
00 02 0x00000 0x18000   Secure          rwxl
03 31 0x18000 0x100000  Non-Secure      rwxl

Non-secure callable region 0 placed in flash region 2 with size 32.

SRAM region             Domain          Permissions
00 07 0x00000 0x10000   Secure          rwxl
08 31 0x10000 0x40000   Non-Secure      rwxl

Peripheral              Domain          Status
00 NRF_P0               Non-Secure      OK
01 NRF_CLOCK            Non-Secure      OK
02 NRF_RTC0             Non-Secure      OK
03 NRF_RTC1             Non-Secure      OK
04 NRF_NVMC             Non-Secure      OK
05 NRF_UARTE1           Non-Secure      OK
06 NRF_UARTE2           Secure          SKIP
07 NRF_TWIM2            Non-Secure      OK
08 NRF_SPIM3            Non-Secure      OK
09 NRF_TIMER0           Non-Secure      OK
10 NRF_TIMER1           Non-Secure      OK
11 NRF_TIMER2           Non-Secure      OK
12 NRF_SAADC            Non-Secure      OK
13 NRF_PWM0             Non-Secure      OK
14 NRF_PWM1             Non-Secure      OK
15 NRF_PWM2             Non-Secure      OK
16 NRF_PWM3             Non-Secure      OK
17 NRF_WDT              Non-Secure      OK
18 NRF_IPC              Non-Secure      OK
19 NRF_VMC              Non-Secure      OK
20 NRF_FPU              Non-Secure      OK
21 NRF_EGU1             Non-Secure      OK
22 NRF_EGU2             Non-Secure      OK
23 NRF_DPPIC            Non-Secure      OK
24 NRF_GPIOTE1          Non-Secure      OK
25 NRF_REGULATORS       Non-Secure      OK

SPM: NS image at 0x18200
SPM: NS MSP at 0x20029c80
SPM: NS reset vector at 0x1d9cd
SPM: prepare to jump to Non-Secure image.
*** Booting Zephyr OS build v2.3.0-rc1-ncs1-2410-g7d20f2ebf259  ***
E: AT socket could not be opened
E: thingy91_magpio_configure failed with error: -14
The AWS IoT sample started, version: v1.0.0
LTE cell changed: Cell ID: 41423372, Tracking area: 24012
PSM parameter update: TAU: -1, Active time: -1
RRC mode: Connected
Network registration status: Connected - home network
PSM parameter update: TAU: 3600, Active time: 0
DATE_TIME_OBTAINED_NTP
AWS_IOT_EVT_CONNECTING
I: Created notify_next_topic $aws/things/nrf-074626/jobs/notify-next
I: Created get_topic $aws/things/nrf-074626/jobs/$next/get/#
I: previously subscribed to notify-next topic
I: Publish topic: $aws/things/nrf-074626/jobs/$next/get
I: Publish payload {"clientToken": ""}
AWS_IOT_EVT_CONNECTED
Persistent session enabled
Publishing: {
        "state":        {
                "reported":     {
                        "app_version":  "v1.0.0",
                        "batv": 4425,
                        "ts":   1601447933251
                }
        }
} to AWS IoT broker
AWS_IOT_EVT_READY
I: Checking for an available job
I: No queued jobs for this device
PSM parameter update: TAU: 1800, Active time: 60
RRC mode: Idle
Publishing: {
        "state":        {
                "reported":     {
                        "batv": 4195,
                        "ts":   1601447953252
                }
        }
} to AWS IoT broker
Next data publication in 20 seconds
RRC mode: Connected
RRC mode: Idle
Publishing: {
        "state":        {
                "reported":     {
                        "batv": 4421,
                        "ts":   1601447973329
                }
        }
} to AWS IoT broker
E: Exception occurred in Secure State
E: ***** HARD FAULT *****
E:   Fault escalation (see below)
E: ***** BUS FAULT *****
E:   Precise data bus error
E:   BFAR Address: 0x50015504
E: r0/a1:  0x0000000c  r1/a2:  0x00000090  r2/a3:  0x00000246
E: r3/a4:  0x000337f5 r12/ip:  0x20023108 r14/lr:  0x00000080
E:  xpsr:  0x20020400
E: s[ 0]:  0x00000000  s[ 1]:  0x00000000  s[ 2]:  0x00000000  s[ 3]:  0x00000000
E: s[ 4]:  0x00000000  s[ 5]:  0x00000000  s[ 6]:  0x00000000  s[ 7]:  0x00000000
E: s[ 8]:  0x00000000  s[ 9]:  0x00000000  s[10]:  0x00000000  s[11]:  0x00000000
E: s[12]:  0x00000000  s[13]:  0x00000000  s[14]:  0x00000000  s[15]:  0x00000000
E: fpscr:  0x00000000
E: Faulting instruction address (r15/pc): 0x20022490
E: >>> ZEPHYR FATAL ERROR 0: CPU exception on CPU 0
E: Current thread: 0x20022490 (unknown)
E: Resetting system
*** Booting Zephyr OS build v2.3.0-rc1-ncs1-2410-g7d20f2ebf259  ***
Flash regions           Domain          Permissions
00 02 0x00000 0x18000   Secure          rwxl
03 31 0x18000 0x100000  Non-Secure      rwxl

Non-secure callable region 0 placed in flash region 2 with size 32.

SRAM region             Domain          Permissions
00 07 0x00000 0x10000   Secure          rwxl
08 31 0x10000 0x40000   Non-Secure      rwxl

Peripheral              Domain          Status
00 NRF_P0               Non-Secure      OK
01 NRF_CLOCK            Non-Secure      OK
02 NRF_RTC0             Non-Secure      OK
03 NRF_RTC1             Non-Secure      OK
04 NRF_NVMC             Non-Secure      OK
05 NRF_UARTE1           Non-Secure      OK
06 NRF_UARTE2           Secure          SKIP
07 NRF_TWIM2            Non-Secure      OK
08 NRF_SPIM3            Non-Secure      OK
09 NRF_TIMER0           Non-Secure      OK
10 NRF_TIMER1           Non-Secure      OK
11 NRF_TIMER2           Non-Secure      OK
12 NRF_SAADC            Non-Secure      OK
13 NRF_PWM0             Non-Secure      OK
14 NRF_PWM1             Non-Secure      OK
15 NRF_PWM2             Non-Secure      OK
16 NRF_PWM3             Non-Secure      OK
17 NRF_WDT              Non-Secure      OK
18 NRF_IPC              Non-Secure      OK
19 NRF_VMC              Non-Secure      OK
20 NRF_FPU              Non-Secure      OK
21 NRF_EGU1             Non-Secure      OK
22 NRF_EGU2             Non-Secure      OK
23 NRF_DPPIC            Non-Secure      OK
24 NRF_GPIOTE1          Non-Secure      OK
25 NRF_REGULATORS       Non-Secure      OK

SPM: NS image at 0x18200
SPM: NS MSP at 0x20029c80
SPM: NS reset vector at 0x1d9cd
SPM: prepare to jump to Non-Secure image.
*** Booting Zephyr OS build v2.3.0-rc1-ncs1-2410-g7d20f2ebf259  ***
E: AT socket could not be opened
E: thingy91_magpio_configure failed with error: -14
The AWS IoT sample started, version: v1.0.0

The aws_iot_send() is responsible somehow.

RE: AWS IoT example getting RSRP results in HARD FAULT

Elith — Tue, 29 Sep 2020 12:11:16 GMT

I noticed that it might also happen when not using RSRP. I updated to the latest origin/master earlier today - perhaps an unintended introduction of a bug?

I get the following when compiling:

C:/ncs/nrf/subsys/net/lib/aws_iot/src/aws_iot.c: In function 'mqtt_evt_handler':
C:/ncs/nrf/subsys/net/lib/aws_iot/src/aws_iot.c:586:26: warning: 'err' may be used uninitialized in this function [-Wmaybe-uninitialized]
  586 |     aws_iot_evt.data.err = err;

I changed the err to just int err1 = 1; (and then changing the err to err1). Just in case, however it is not it.

I can provoke the error by unplugging the SIM card and plugging it back in (with some changes to the example such that it does not delete the thread on disconnect. It also happens sometimes when the cell tower changes.

It seems to be a problem in aws_iot_send(), as this is where it crashes (perhaps it is linked with the RSRP problem):

+CEREG: 5,1,"5DCC","0278120C",7,,,"00000000","00000110"
Publishing: {
        "state":        {
                "reported":     {
                        "device":       "nrf-074626",
                        "batv": 4445,
                        "rsrp": 255,
                        "mode": 1,
                        "datim":        "20/09/29,12:48:02+08",
                        "timestamp":    1601383681346
                }
        }
} to AWS IoT broker
*** Booting Zephyr OS build v2.3.0-rc1-ncs1-2410-g7d20f2ebf259  ***
Flash regions           Domain          Permissions
00 02 0x00000 0x18000   Secure          rwxl
03 31 0x18000 0x100000  Non-Secure      rwxl

Non-secure callable region 0 placed in flash region 2 with size 32.

SRAM region             Domain          Permissions
00 07 0x00000 0x10000   Secure          rwxl
08 31 0x10000 0x40000   Non-Secure      rwxl

Peripheral              Domain          Status
00 NRF_P0               Non-Secure      OK
01 NRF_CLOCK            Non-Secure      OK
02 NRF_RTC0             Non-Secure      OK
03 NRF_RTC1             Non-Secure      OK
04 NRF_NVMC             Non-Secure      OK
05 NRF_UARTE1           Non-Secure      OK
06 NRF_UARTE2           Secure          SKIP
07 NRF_TWIM2            Non-Secure      OK
08 NRF_SPIM3            Non-Secure      OK
09 NRF_TIMER0           Non-Secure      OK
10 NRF_TIMER1           Non-Secure      OK
11 NRF_TIMER2           Non-Secure      OK
12 NRF_SAADC            Non-Secure      OK
13 NRF_PWM0             Non-Secure      OK
14 NRF_PWM1             Non-Secure      OK
15 NRF_PWM2             Non-Secure      OK
16 NRF_PWM3             Non-Secure      OK
17 NRF_WDT              Non-Secure      OK
18 NRF_IPC              Non-Secure      OK
19 NRF_VMC              Non-Secure      OK
20 NRF_FPU              Non-Secure      OK
21 NRF_EGU1             Non-Secure      OK
22 NRF_EGU2             Non-Secure      OK
23 NRF_DPPIC            Non-Secure      OK
24 NRF_GPIOTE1          Non-Secure      OK
25 NRF_REGULATORS       Non-Secure      OK

SPM: NS image at 0x18200
SPM: NS MSP at 0x2002ac48
SPM: NS reset vector at 0x1b989
SPM: prepare to jump to Non-Secure image.
*** Booting Zephyr OS build v2.3.0-rc1-ncs1-2410-g7d20f2ebf259  ***
The AWS IoT sample started, version: v1.0.0

It seems that the aws_iot_send() bugs out when the connection is lost or changed or some state change related to the LTE and/or AWS iot connection occurs.