• State Not Answered
  • Replies 7 replies
  • Subscribers 96 subscribers
  • Views 383 views
  • Users 0 members are here
Attachments (0)
Nordic Case Info
  • Case ID: 362289
Options

Difficulty with Memfault UI managing FOTA images with different software_type values

Regan0
  • Target device: nRF9151
  • NCS 3.4.0
  • Memfault Integrated
  • FOTA architecture in place with four slots, 2 in internal flash and 2 in external flash
  • Background COAP updates of both 2 independent images to external flash before overwrite to internal partitions

We need to manage FOTA updates for two different images on the same physical device. Each image has its own software_type and version and must be released independently. mcuboot has no problem with this arrangement. The images have different roles, where one of them corresponds to nRF's 'app' nomenclature.

So far, I have only been able to represent this in the Memfault UI by adding an extra hardware device. This workaround does not reflect the actual product architecture, since both images belong to and are installed on the same device. It's meant that my device-side FOTA handling has to pretend to be two different hardware devices (yukk!).

Could you please clarify:

  • Is managing multiple independently versioned images on one device supported by the Nordic/Memfault integration? The REST API seems to indicate 'yes'.
  • How should releases and deployments be configured when each image has a different software_type and version? I can't add an additional OTA payload with a software_type other than the primary. Even doing this through REST methods won't work.
  • Is there a recommended way to target and track each image independently without creating a duplicate hardware device?
  • Why are there limitations in the Memfault UI or Nordic integration that prevent this workflow?

Our desired model is:

  • One physical device and hardware identity
  • Two (or more) independently updatable images. 
  • A distinct software_type and version for each image
  • Independent FOTA release, targeting, status, and rollback tracking

Have I missed something? Is this a bug or intentional? If this is in the pipeline but not yet exposed, I would be grateful to know that it is coming and when. Any documentation or configuration examples for this use case would be appreciated. 

Parents
  • 0

    Hi Ryan-

    Thanks for the question! We are in progress rolling out support for this use case.

    Note that it will enable updating individual system components semi-independently, but we still tie the overall "Release" version to the "primary" software type (in your case, this would be `app`). So specifically what you describe here:

    > Independent FOTA release, targeting, status, and rollback tracking

    Is not exactly how it's expected to be used (though possible to work around, it won't make reporting as seamless). It's intended to enable updating non-primary software_type components separately from primary, but still treating the system version as a holistic unit identified by primary software version. Hopefully that makes sense! We'd be glad to have you try the feature when it's shipped, expected in next 1-2 days. I'll post back here when that's available.

    Thanks!
    Noah

  • 0 in reply to NoahPendleton

    Any update to this, Noah?

    After a break of a few weeks, I've returned to managing FOTA updates and I've found that my secondary software_type updates are no longer working through nRFCloud. These failed updates are carried out through NRF_CLOUD_DL_TYPE_DL_CLIENT instead of NRF_CLOUD_DL_TYPE_FOTA (which is still working). It is possible that I've broken something. The error I get is -111 (downloader connection refusal), which seems server-related to me.

    Have any changes been made to FOTA management in nRFCloud in the past month?

  • 0 in reply to Regan0

    Hi  -

    Thank you for reporting- there was a backend change to the way those payloads were delivered, but we designed it to be invisible to devices (i.e. no functional changes from the device-facing part of the system). I'll forward this to my colleagues on the server side, they'll be able to diagnose faster than I. We'll get back to you.

    Thanks,
    Noah

Reply Children
  • 0 in reply to NoahPendleton

    Hi  -

    We suspect that the NRF_CLOUD_DL_TYPE_DL_CLIENT requests are hitting the wrong endpoint (i.e. CoAP proxy vs. HTTPs). Do you have any other logs you can share, maybe with the target URI of the failing request, so we can debug a bit more?

    Thanks!
    Noah

  • 0 in reply to NoahPendleton

    I've compiled a trace of the behaviour where it only attempts to download the secondary image. Sensitive elements have been redacted.

    [00:00:09.992,797] <dbg> dtls: nrfc_dtls_setup: Client IP address: 0.0.0.0
[00:00:09.993,255] <dbg> dtls: nrfc_dtls_setup: Setting socket options:
[00:00:09.993,682] <dbg> dtls: nrfc_dtls_setup:   hostname: coap.nrfcloud.com
[00:00:09.994,140] <dbg> dtls: nrfc_dtls_setup:   sectag: [REDACTED_SEC_TAG]
[00:00:09.994,781] <dbg> dtls: nrfc_dtls_setup:   Enable connection id
[00:00:09.995,574] <dbg> dtls: nrfc_dtls_setup:   Set handshake timeout 123
[00:00:09.996,398] <dbg> dtls: nrfc_dtls_setup:   Peer verify: 2
[00:00:09.996,795] <dbg> dtls: nrfc_dtls_setup:   TLS session cache: 1
[00:00:09.997,467] <dbg> dtls: nrfc_dtls_setup:   Keep open supported: 1
[00:00:09.998,107] <dbg> nrf_cloud_coap_transport: nrf_cloud_coap_connect_host_cb: Connect failed, errno: 118
[00:00:09.998,931] <dbg> nrf_cloud_dns: nrf_cloud_try_addresses: Failed to connect to IP address 2600:1f18:56ff:3d02::ffc5. Error: -111
[00:00:09.999,603] <dbg> nrf_cloud_dns: nrf_cloud_connect_host: Trying IPv4 addresses for coap.nrfcloud.com
[00:00:10.458,862] <dbg> nrf_cloud_dns: nrf_cloud_try_addresses: Trying IP address and port for server coap.nrfcloud.com: 34.200.153.177, port: 5684
[00:00:10.459,564] <dbg> nrf_cloud_coap_transport: nrf_cloud_coap_connect_host_cb: Creating socket type IPPROTO_DTLS_1_2
[00:00:10.460,418] <dbg> nrf_cloud_coap_transport: nrf_cloud_coap_connect_host_cb: sock = 0
[00:00:10.460,937] <dbg> dtls: nrfc_dtls_setup: Client IP address: 0.0.0.0
[00:00:10.461,425] <dbg> dtls: nrfc_dtls_setup: Setting socket options:
[00:00:10.461,853] <dbg> dtls: nrfc_dtls_setup:   hostname: coap.nrfcloud.com
[00:00:10.462,524] <dbg> dtls: nrfc_dtls_setup:   sectag: [REDACTED_SEC_TAG]
[00:00:10.462,951] <dbg> dtls: nrfc_dtls_setup:   Enable connection id
[00:00:10.463,623] <dbg> dtls: nrfc_dtls_setup:   Set handshake timeout 123
[00:00:10.464,294] <dbg> dtls: nrfc_dtls_setup:   Peer verify: 2
[00:00:10.464,721] <dbg> dtls: nrfc_dtls_setup:   TLS session cache: 1
[00:00:10.465,393] <dbg> dtls: nrfc_dtls_setup:   Keep open supported: 1
[00:00:13.080,261] <dbg> nrf_cloud_dns: nrf_cloud_connect_host: Connected to nRF Cloud host: coap.nrfcloud.com. Socket ID: 0
[00:00:13.080,871] <dbg> nrf_cloud_coap_transport: nrf_cloud_coap_transport_authenticate: Generate JWT
[00:00:13.156,127] <inf> nrf_cloud_coap_transport: Request authorization with JWT
[00:00:13.156,890] <dbg> nrf_cloud_coap_transport: client_transfer: CON POST auth/jwt?cver=1&dver=ncs-v3.4.1 Content-Format:PLAIN TEXT, 456 bytes out, Accept:none
[00:00:13.163,024] <dbg> nrf_cloud_coap_transport: client_transfer: Sent
                                                   [REDACTED_JWT_BYTES_AND_KEY_ID]
[00:00:13.716,461] <dbg> nrf_cloud_coap_transport: client_callback: result_code:2.01, offset:0x0, len:0x0, last_block:1
[00:00:13.717,132] <dbg> nrf_cloud_coap_transport: client_callback: Calling user's callback [REDACTED_ADDRESS]
[00:00:13.717,681] <inf> nrf_cloud_coap_transport: Authorization result_code: 2.01
[00:00:13.718,139] <dbg> nrf_cloud_coap_transport: client_callback: End of client transfer
[00:00:13.719,177] <dbg> nrf_cloud_coap_transport: client_transfer: Got callback
[00:00:13.719,635] <inf> nrf_cloud_coap_transport: Authorized
[00:00:13.720,214] <dbg> dtls: nrfc_dtls_cid_is_active: Full DTLS handshake performed
[00:00:13.720,886] <dbg> dtls: nrfc_dtls_cid_is_active: DTLS CID uplink
[00:00:13.721,282] <inf> nrf_cloud_coap_transport: DTLS CID is active
[00:00:13.721,862] <dbg> nrf_cloud_codec_internal: nrf_cloud_shadow_control_response_encode: Shadow response: {"control":{"memfaultEn":false}}
[00:00:13.722,869] <dbg> nrf_cloud_coap_transport: client_transfer: CON PATCH state/reported Content-Format:JSON, 32 bytes out, Accept:none
[00:00:13.725,891] <dbg> nrf_cloud_coap_transport: client_transfer: Sent
                                                   7b 22 63 6f 6e 74 72 6f  6c 22 3a 7b 22 6d 65 6d |{"contro l":{"mem
                                                   66 61 75 6c 74 45 6e 22  3a 66 61 6c 73 65 7d 7d |faultEn" :false}}
[00:00:14.278,503] <dbg> nrf_cloud_coap_transport: client_callback: result_code:2.04, offset:0x0, len:0x2, last_block:1
[00:00:14.279,144] <dbg> nrf_cloud_coap_transport: client_callback: payload received
                                                   7b 7d                                            |{}
[00:00:14.280,487] <dbg> nrf_cloud_coap_transport: client_callback: Calling user's callback [REDACTED_ADDRESS]
[00:00:14.281,005] <dbg> nrf_cloud_coap_transport: client_callback: End of client transfer
[00:00:14.282,012] <dbg> nrf_cloud_coap_transport: client_transfer: Got callback
[00:00:14.283,111] <dbg> nrf_cloud_coap_transport: client_transfer: CON PATCH state/reported Content-Format:JSON, 64 bytes out, Accept:none
[00:00:14.286,407] <dbg> nrf_cloud_coap_transport: client_transfer: Sent
                                                   7b 22 64 65 76 69 63 65  22 3a 7b 22 63 6f 6e 6e |{"device ":{"conn
                                                   65 63 74 69 6f 6e 49 6e  66 6f 22 3a 7b 22 70 72 |ectionIn fo":{"pr
                                                   6f 74 6f 63 6f 6c 22 3a  22 43 6f 41 50 22 2c 22 |otocol": "CoAP","
                                                   6d 65 74 68 6f 64 22 3a  22 4c 54 45 22 7d 7d 7d |method": "LTE"}}}
[00:00:14.836,456] <dbg> nrf_cloud_coap_transport: client_callback: result_code:2.04, offset:0x0, len:0x2, last_block:1
[00:00:14.837,066] <dbg> nrf_cloud_coap_transport: client_callback: payload received
                                                   7b 7d                                            |{}
[00:00:14.838,409] <dbg> nrf_cloud_coap_transport: client_callback: Calling user's callback [REDACTED_ADDRESS]
[00:00:14.838,928] <dbg> nrf_cloud_coap_transport: client_callback: End of client transfer
[00:00:14.839,813] <dbg> nrf_cloud_coap_transport: client_transfer: Got callback
[00:00:14.840,911] <dbg> nrf_cloud_coap_transport: client_transfer: CON PATCH state/reported Content-Format:JSON, 56 bytes out, Accept:none
[00:00:14.844,238] <dbg> nrf_cloud_coap_transport: client_transfer: Sent
                                                   7b 22 64 65 76 69 63 65  22 3a 7b 22 73 65 72 76 |{"device ":{"serv
                                                   69 63 65 49 6e 66 6f 22  3a 7b 22 75 69 22 3a 6e |iceInfo" :{"ui":n
                                                   75 6c 6c 2c 22 66 6f 74  61 5f 76 32 22 3a 5b 22 |ull,"fot a_v2":["
                                                   41 50 50 22 5d 7d 7d 7d                          |APP"]}}}
[00:00:15.396,514] <dbg> nrf_cloud_coap_transport: client_callback: result_code:2.04, offset:0x0, len:0x2, last_block:1
[00:00:15.397,125] <dbg> nrf_cloud_coap_transport: client_callback: payload received
                                                   7b 7d                                            |{}
[00:00:15.398,468] <dbg> nrf_cloud_coap_transport: client_callback: Calling user's callback [REDACTED_ADDRESS]
[00:00:15.398,986] <dbg> nrf_cloud_coap_transport: client_callback: End of client transfer
977,661] <dbg> nrf_cloud_coap_transport: client_transfer: Got callback
[00:00:18.305,053] <inf> cloud_coap: CoAP ready
[00:00:18.306,213] <dbg> nrf_cloud_coap_transport: client_transfer: CON GET ota/latest/url?device_serial=[REDACTED_DEVICE_SERIAL]&hardware_version=nrf9151dk&software_type=system&current_version=0.2.2 Content-Format:PLAIN TEXT, 0 bytes out, Accept:JSON
[00:00:18.998,535] <dbg> nrf_cloud_coap_transport: client_callback: result_code:2.05, offset:0x0, len:0x0, last_block:1
[00:00:18.999,176] <dbg> nrf_cloud_coap_transport: client_callback: Calling user's callback [REDACTED_ADDRESS]
[00:00:18.999,725] <dbg> nrf_cloud_coap_transport: client_callback: End of client transfer
[00:00:19.000,610] <dbg> nrf_cloud_coap_transport: client_transfer: Got callback
[00:00:19.001,708] <dbg> nrf_cloud_coap_transport: client_transfer: CON GET ota/latest/url?device_serial=[REDACTED_DEVICE_SERIAL]&hardware_version=nrf9151dk&software_type=app&current_version=0.2.2 Content-Format:PLAIN TEXT, 0 bytes out, Accept:JSON
[00:00:19.655,639] <dbg> nrf_cloud_coap_transport: client_callback: result_code:2.05, offset:0x0, len:0x0, last_block:1
[00:00:19.656,280] <dbg> nrf_cloud_coap_transport: client_callback: Calling user's callback [REDACTED_ADDRESS]
[00:00:19.656,829] <dbg> nrf_cloud_coap_transport: client_callback: End of client transfer
[00:00:19.658,508] <dbg> nrf_cloud_coap_transport: client_transfer: Got callback
[00:00:19.659,118] <inf> fota_memfault: Memfault image-0 is up to date
[00:00:19.667,907] <inf> fota: FOTA: system up to date; checking Lua
[00:00:19.942,840] <dbg> nrf_cloud_coap_transport: client_transfer: CON GET ota/latest/url?device_serial=[REDACTED_DEVICE_SERIAL]&hardware_version=nrf9151dk-lua&software_type=lua&current_version=0.0.5 Content-Format:PLAIN TEXT, 0 bytes out, Accept:JSON
[00:00:23.198,760] <dbg> nrf_cloud_coap_transport: client_callback: result_code:2.05, offset:0x0, len:0x99, last_block:1
[00:00:23.199,432] <dbg> nrf_cloud_coap_transport: client_callback: payload received
                                                   [REDACTED_SIGNED_OTA_URL_HEXDUMP]
[00:00:23.206,359] <dbg> nrf_cloud_coap_transport: client_callback: Calling user's callback [REDACTED_ADDRESS]
[00:00:23.206,970] <dbg> nrf_cloud_coap_transport: client_callback: End of client transfer
[00:00:23.211,456] <dbg> nrf_cloud_coap_transport: client_transfer: Got callback
[00:00:23.211,944] <inf> fota_memfault: Memfault image-1 update available
[00:00:23.218,109] <inf> fota: FOTA image-1 update available; downloading while Lua runs
[00:00:23.492,462] <dbg> nrf_cloud_coap_transport: client_transfer: CON GET ota/latest/url?device_serial=[REDACTED_DEVICE_SERIAL]&hardware_version=nrf9151dk-lua&software_type=lua&current_version=0.0.5 Content-Format:PLAIN TEXT, 0 bytes out, Accept:JSON
[00:00:24.143,737] <dbg> nrf_cloud_coap_transport: client_callback: result_code:2.05, offset:0x0, len:0x99, last_block:1
[00:00:24.144,378] <dbg> nrf_cloud_coap_transport: client_callback: payload received
                                                   [REDACTED_SIGNED_OTA_URL_HEXDUMP]
[00:00:24.151,275] <dbg> nrf_cloud_coap_transport: client_callback: Calling user's callback [REDACTED_ADDRESS]
[00:00:24.151,855] <dbg> nrf_cloud_coap_transport: client_callback: End of client transfer
[00:00:24.152,740] <dbg> nrf_cloud_coap_transport: client_transfer: Got callback
[00:00:24.153,381] <inf> fota_memfault: Lua FOTA proxy source=ota-cdn.memfault.com sec_tag=[REDACTED_SEC_TAG]
[00:00:24.154,022] <dbg> nrf_cloud_coap_transport: nrf_cloud_coap_transport_proxy_dl_uri_get: Proxy URI: [REDACTED_SIGNED_OTA_URL]
[00:00:24.155,212] <dbg> downloader: downloader_start: URL: coaps://coap.nrfcloud.com/proxy
[00:00:24.155,761] <dbg> downloader: state_set: state = 2 ()
[00:00:24.156,188] <dbg> downloader: dl_coap_init: Port not specified, using default: 5684
[00:00:24.156,799] <dbg> downloader: dl_socket_host_lookup: host lookup coap.nrfcloud.com, pdn id 0, family 2
[00:00:24.163,269] <inf> fota: FOTA download started
[00:00:24.689,697] <dbg> downloader: dl_socket_create_and_connect: family: 2, type: 2, proto: 273
[00:00:24.690,612] <dbg> downloader: dl_socket_create_and_connect: Socket opened, fd 3
[00:00:24.691,101] <inf> downloader: Setting up TLS credentials, sec tag count 1
[00:00:24.691,741] <dbg> downloader: dl_socket_create_and_connect: enabling CID
[00:00:24.692,535] <inf> downloader: Connecting to 2600:1f18:56ff:3d00::547c
[00:00:24.692,993] <dbg> downloader: dl_socket_create_and_connect: fd 3, addrlen 24, fam IPv6, port 5684
[00:00:24.694,000] <dbg> downloader: dl_socket_close: Socket closed, fd 3
[00:00:24.694,458] <inf> downloader: Failed to connect on IPv6 (err -118), attempting IPv4
[00:00:24.694,976] <dbg> downloader: dl_socket_host_lookup: host lookup coap.nrfcloud.com, pdn id 0, family 1
[00:00:25.196,411] <dbg> downloader: dl_socket_create_and_connect: family: 1, type: 2, proto: 273
[00:00:25.197,265] <dbg> downloader: dl_socket_create_and_connect: Socket opened, fd 2
[00:00:25.197,784] <inf> downloader: Setting up TLS credentials, sec tag count 1
[00:00:25.198,242] <dbg> downloader: dl_socket_create_and_connect: enabling CID
[00:00:25.199,005] <inf> downloader: Connecting to 34.200.153.177
[00:00:25.199,432] <dbg> downloader: dl_socket_create_and_connect: fd 2, addrlen 8, fam IPv4, port 5684
[00:00:26.420,959] <dbg> downloader: dl_socket_close: Socket closed, fd 2
[00:00:26.421,386] <err> downloader: Failed to connect, err -111
[00:00:26.421,783] <wrn> fota_memfault: Lua downloader error: -111
[00:00:26.422,241] <dbg> downloader: state_set: state = 1 ()
[00:00:26.428,222] <wrn> fota: FOTA download failed (-111); retry 1/3 in 60000 ms

Related
Terms of service