DFU issues on Android 10

RE: DFU issues on Android 10

Einar Thorsrud — Mon, 02 Dec 2019 15:21:30 GMT

Hi Anton,

Good question. It is not logical in my mind to send it before BLE_GAP_EVT_CONN_SEC_UPDATE, since that is when we know we are connected to a known peer (which has hopefully enabled service changed indications). You could try to send it before, but it anyway should not be before the peer has enabled indication (since you cannot know it is the same old peer before the link is encrypted). You could try to send in on the BLE_GAP_EVT_CONNECTED, or start an app_timer to send it a certain delay after.

I am surprised that the phone handles it like this. It would be interesting to know why this is so slow on the phone side, but no progress there yet.

RE: DFU issues on Android 10

AntonHellbe — Mon, 02 Dec 2019 08:58:49 GMT

Hi Einar,

Thanks for the update. We will try both solutions and see what works best.

Regarding sending a service changed indication earlier than BLE_GATTS_EVT_SYS_ATTR_MISSING or BLE_GAP_EVT_CONN_SEC_UPDATE, do you have any suggestions on what event would be appropriate to use for sending the service changed indication?

Br,
Anton

RE: DFU issues on Android 10

Einar Thorsrud — Fri, 29 Nov 2019 12:27:04 GMT

Hi,

Looking more closely at this with one of our Android developers he commented hat from the log is seems like the phone did receive SC indication, as it switched to 7.5 ms connection interval (which it does for service discovery), but it did receive it too late. The DFU service has already received cached services from the system and tries to use them. Service discovery was delayed 1.6 sec, but that seems not to be enough for the device, as 300 ms later the SC indication comes. Had it come earlier, the system would have cleared the cache, performed service discovery and returned the new services to the DFU library.

You can try to fix this in your app by including the library as a module, not from jcenter, and modify this line to be > 2000 ms. You can also try fixing in the bootloader by always sending a service changed indication earlier (if enabled), i.e. before getting the BLE_GATTS_EVT_SYS_ATTR_MISSING or BLE_GAP_EVT_CONN_SEC_UPDATE event.

RE: DFU issues on Android 10

Einar Thorsrud — Thu, 28 Nov 2019 09:46:33 GMT

Hi,

Thanks. All looks good here as far as I can see. I am wondering if there could be a timing issue in the Android DFU library itself, as you suggested earlier. Perhaps fixed by this. I am checking with the library developer and will get back to you.

RE: DFU issues on Android 10

AntonHellbe — Thu, 28 Nov 2019 08:03:19 GMT

devzone.nordicsemi.com/.../2816.sniffer.pcapng

Hi,

Here's the sniffer trace from wireshark. At package 6130 the connection to the bootloader starts.

Hopefully you can read the data, I'm not very experienced with wireshark so I dont know if this is the correct way to do it.

The sequence is here is write to DFU characteristic to "manually" put it in bootloader and then connect.

RE: DFU issues on Android 10

Einar Thorsrud — Wed, 27 Nov 2019 13:32:45 GMT

Hi,

I see. Then I think it would be very useful with a sniffer trace and a log from the bootloader to know exactly what is going on (we have based it on assumptions up to now).

Regarding problem 2, that is also a bit of a mystery. We may get a bit close if you look at the implementation in components\libraries\bootloader\dfu\nrf_dfu_svci_handler.c, an put breakpoints on all conditions that cause a NRF_ERROR_INVALID_STATE return value. I fail to see the reason, but reading this thread might suggest a link with problem 1.

RE: DFU issues on Android 10

AntonHellbe — Wed, 27 Nov 2019 10:29:03 GMT

Here's the changes I made to nrf_dfu_ble.c. I copied the service_changed_cccd function and In case the softdevice call to service_changed fails I call that function. If I understand the code in gatt_cache_manager.c that's how it works there aswell.

It does not seem to help.

static ret_code_t service_changed_cccd(uint16_t conn_handle, uint16_t * p_cccd)
{
    bool       sc_found = false;
    uint16_t   end_handle;

    ret_code_t err_code = sd_ble_gatts_initial_user_handle_get(&end_handle);
    ASSERT(err_code == NRF_SUCCESS);

    for (uint16_t handle = 1; handle < end_handle; handle++)
    {
        ble_uuid_t uuid;
        ble_gatts_value_t value = {.p_value = (uint8_t *)&uuid.uuid, .len = 2, .offset = 0};

        err_code = sd_ble_gatts_attr_get(handle, &uuid, NULL);
        if (err_code != NRF_SUCCESS)
        {
            return err_code;
        }
        else if (!sc_found && (uuid.uuid == BLE_UUID_GATT_CHARACTERISTIC_SERVICE_CHANGED))
        {
            sc_found = true;
        }
        else if (sc_found && (uuid.uuid == BLE_UUID_DESCRIPTOR_CLIENT_CHAR_CONFIG))
        {
            value.p_value = (uint8_t *)p_cccd;
            return sd_ble_gatts_value_get(conn_handle, handle, &value);
        }
    }
    return NRF_ERROR_NOT_FOUND;
}


#if (NRF_DFU_BLE_REQUIRES_BONDS)
static uint32_t service_changed_send(void)
{
    uint32_t err_code;

    NRF_LOG_DEBUG("Sending Service Changed indication");

    err_code = sd_ble_gatts_sys_attr_set(m_conn_handle,
                                         m_peer_data.sys_serv_attr,
                                         sizeof(m_peer_data.sys_serv_attr),
                                         BLE_GATTS_SYS_ATTR_FLAG_SYS_SRVCS);
    VERIFY_SUCCESS(err_code);

    err_code = sd_ble_gatts_sys_attr_set(m_conn_handle,
                                         NULL,
                                         0,
                                         BLE_GATTS_SYS_ATTR_FLAG_USR_SRVCS);
    VERIFY_SUCCESS(err_code);

    err_code = sd_ble_gatts_service_changed(m_conn_handle, m_dfu.service_handle, 0xFFFF);

    if (   (err_code == BLE_ERROR_INVALID_CONN_HANDLE)
        || (err_code == NRF_ERROR_INVALID_STATE)
        || (err_code == NRF_ERROR_BUSY))
    {
        uint16_t cccd;
        err_code = service_changed_cccd(m_conn_handle, &cccd);
        if ((err_code == NRF_SUCCESS) && cccd)
        {
            NRF_LOG_DEBUG("Service changed cccd returned err_code: %d", err_code);
            // Possible ATT_MTU exchange ongoing.
            // Do nothing, treat as busy.
            return err_code;
        }
        else
        {
            if (err_code != NRF_SUCCESS)
            {
                NRF_LOG_WARNING("Client did not have the Service Changed indication set to enabled."
                        "Error: 0x%08x", err_code);
                err_code = NRF_SUCCESS;

                //NRF_LOG_DEBUG("Unexpected error when looking for service changed CCCD: %s",
                //              nrf_strerror_get(err_code));
            }
            // CCCDs not enabled or an error happened. Drop indication.
            // Fallthrough.
        }
        /* These errors can be expected when trying to send a Service Changed indication */
        /* if the CCCD is not set to indicate. Thus, set the returning error code to success. */
    }

    return err_code;
}
#endif

The service changed notification is still sent from the events (no changes there):

case BLE_GAP_EVT_CONN_SEC_UPDATE:
case BLE_GATTS_EVT_SYS_ATTR_MISSING:
    {
        #if (NRF_DFU_BLE_REQUIRES_BONDS)
            err_code = service_changed_send();
        #else
            err_code = sd_ble_gatts_sys_attr_set(p_gap->conn_handle, NULL, 0, 0);
        #endif
        APP_ERROR_CHECK(err_code);
        NRF_LOG_DEBUG("Finished handling conn sec update");
    } break;

RE: DFU issues on Android 10

AntonHellbe — Wed, 27 Nov 2019 10:06:08 GMT

I will implement the fix and see what happens.

Regarding problem 2:

ret = nrf_dfu_set_peer_data(&m_peer_data);

is the function that returns INVALID_STATE, row 178 in ble_dfu_bonded.c

RE: DFU issues on Android 10

Einar Thorsrud — Wed, 27 Nov 2019 08:56:14 GMT

Hi,

The service changed indication in the bootloader in SDK 15.0 is sent by the call to service_changed_send() on line 882 in nrf_dfu_ble.c.

RE: DFU issues on Android 10

AntonHellbe — Tue, 26 Nov 2019 13:36:55 GMT

Hi,

I see. We could confirm with a sniffer trace, but your tests strengthen the theory of a timing difference. Different mobile devices, operating systems, SW/DFU library changes, all affect timing slightly. The fix is probably to do the same workaround as you did in your application in the bootloader as well.

- In the application I currently trigger pm_local_database_has_changed() in the peer manager event handler when we get the event CACHE_APPLY_FAILED, If I understand you correctly I should implement the same in the bootloader?

Currently the bootloader does not make use of the Peer Manager.

RE: DFU issues on Android 10

Einar Thorsrud — Tue, 26 Nov 2019 08:33:05 GMT

Hi,

For problem 2 to you get a return value 8, which is NRF_ERROR_INVALID_STATE. From which function?

RE: DFU issues on Android 10

AntonHellbe — Mon, 25 Nov 2019 14:04:58 GMT

Hi,

So I just finished a longer test session.

It seems like the GATT Service changed notifications is not working properly in the bootloader.

If I take these steps:

In the application, write to the DFU characteristic to enter bootloader
Enters bootloader
Reconnect
Still shows the attribute table from the application. Logs indicate that the Indication was sent successfully (returns NRF_SUCCESS inside nrf_dfu_ble.c)
Disconnect and then reconnect. Now displays the DFU service/characteristic (Sometimes requires more than one disconnect/reconnect in order to display the new attribute table).
Start FW upload

With the above sequence DFU was successful 10 out of 10 tries.

Automatic DFU from the application fails on either GATT_INVALID_HANDLE or GATT_ERROR which can be explained by the behavior above.

RE: DFU issues on Android 10

AntonHellbe — Mon, 25 Nov 2019 12:48:29 GMT

Update on problem 2:

<info> app: Retriving Peer data returned err_code: 0
<info> app: Setting peer data for bootloader returned err_code: 8
<error> app: Request to enter bootloader mode failed asynchroneously.

RE: DFU issues on Android 10

AntonHellbe — Mon, 25 Nov 2019 10:36:44 GMT

Update regarding problem 1:

Logs indicate that the Service Changed notifications are working correctly in the bootloader:

<debug> nrf_dfu_ble: Connected
<debug> nrf_dfu_ble: Received BLE_GAP_EVT_DATA_LENGTH_UPDATE_REQUEST.
<debug> nrf_dfu_ble: Received BLE_GAP_EVT_DATA_LENGTH_UPDATE (27, max_rx_time 328).
<debug> nrf_dfu_ble: Received BLE_GAP_EVT_PHY_UPDATE_REQUEST.
<debug> nrf_dfu_ble: Received BLE_GAP_EVT_PHY_UPDATE (RX:2, TX:2, status:0)
<debug> nrf_dfu_ble: Received BLE_GAP_EVT_CONN_PARAM_UPDATE
<debug> nrf_dfu_ble: max_conn_interval: 12
<debug> nrf_dfu_ble: min_conn_interval: 12
<debug> nrf_dfu_ble: slave_latency: 0
<debug> nrf_dfu_ble: conn_sup_timeout: 600
<debug> nrf_dfu_ble: Received BLE_GAP_EVT_SEC_INFO_REQUEST
<debug> nrf_dfu_ble: Received BLE_GAP_EVT_CONN_SEC_UPDATE
<debug> nrf_dfu_ble: Received BLE_GATTS_EVT_SYS_ATTR_MISSING
<debug> nrf_dfu_ble: Sending Service Changed indication
<debug> nrf_dfu_ble: Service changed notification returned err_code: 0
<debug> nrf_dfu_ble: Finished handling conn sec update
<debug> nrf_dfu_ble: Received BLE_GAP_EVT_CONN_PARAM_UPDATE
<debug> nrf_dfu_ble: max_conn_interval: 6
<debug> nrf_dfu_ble: min_conn_interval: 6
<debug> nrf_dfu_ble: slave_latency: 0
<debug> nrf_dfu_ble: conn_sup_timeout: 500
<debug> nrf_dfu_ble: Received BLE_GAP_EVT_CONN_PARAM_UPDATE
<debug> nrf_dfu_ble: max_conn_interval: 12
<debug> nrf_dfu_ble: min_conn_interval: 12
<debug> nrf_dfu_ble: slave_latency: 0
<debug> nrf_dfu_ble: conn_sup_timeout: 600
<debug> nrf_dfu_ble: Received BLE_GAP_EVT_CONN_PARAM_UPDATE
<debug> nrf_dfu_ble: max_conn_interval: 6
<debug> nrf_dfu_ble: min_conn_interval: 6
<debug> nrf_dfu_ble: slave_latency: 0
<debug> nrf_dfu_ble: conn_sup_timeout: 500
<debug> nrf_dfu_ble: Received BLE_GAP_EVT_CONN_PARAM_UPDATE
<debug> nrf_dfu_ble: max_conn_interval: 12
<debug> nrf_dfu_ble: min_conn_interval: 12
<debug> nrf_dfu_ble: slave_latency: 0
<debug> nrf_dfu_ble: conn_sup_timeout: 600

Edit:
Another intresting observation. According to Google Play nRF Connect uses:

- DFU Library 1.9.1 (improved DFU performance with devices based on SDK 15.x and 16)

Using nRF Connect DFU I received GATT ERROR (0x85)

Then I tried upgrading using nRF Toolbox which uses DFU Library 1.9.0 which went successfully without any issues.

RE: DFU issues on Android 10

AntonHellbe — Mon, 25 Nov 2019 10:20:30 GMT

Hi,

Problem 1: This is the exact same issue. But remember that this needs to work both in the application and in the bootloader. Have you done anything with this issue in the bootloader?

- No I have not modified the bootloader source. I can try to debug it and see if It successfully sends Service Changed Notifications before I do any modifications. In general I dont like to modify SDK files since it becomes hard to keep track of the changes which could be dangerous in the long run, especially if I do the changes myself (perhaps poor solutions).

Problem 2:

I will try to debug it further on the nRF side

RE: DFU issues on Android 10

Einar Thorsrud — Mon, 25 Nov 2019 10:00:07 GMT

Hi,

Problem 1: This is the exact same issue. But remember that this needs to work both in the application and in the bootloader. Have you done anything with this issue in the bootloader?

Problem 2: I see. Then I suggest you debug on the nRF side to see what happens before printing the "bootloader mode failed asynchronously" error message, as there are several issues that can lead to this being printed.

RE: DFU issues on Android 10

AntonHellbe — Mon, 25 Nov 2019 09:47:04 GMT

Hi Einar,

Problem 1:

I dont think Service changed colliding with MTU exchange is a problem. There was a bug in SDK V15.0 that I have resolved with help from your team that skipped Service changed indications if there was an ongoing MTU Exchange.

You can see that ticket here: Service changed indication MTU exchange

Problem 2:

The logs from the nRF52832 application that I provided in the post are me trying to write to the characteristic manually

RE: DFU issues on Android 10

Einar Thorsrud — Mon, 25 Nov 2019 09:41:40 GMT

Hi,

I have not found any reports of similar issues with Android 10 specifically, but a possible explanation is if Service changed collides with MTU exchange. The timing can be different between phones and SW versions, so perhaps you incidentally see this on Android 10 on the OnePlus. If so, this could actually be a similar issue in your problem 1 and 2. Sniffer traces could be useful to see if this theory makes sense.

For problem 2, can you try to trigger DFU mode by manually writing to the DFU characteristic in the nRF Connect app? Doing it manually should change the timing, and it would be interesting to see if you still get the issue with "Request to enter bootloader mode failed asynchroneously" or not.

For problem 1, it would be good to delay the serviced changed indication from the bootloader do see if that fixes the issue.

RE: DFU issues on Android 10

AntonHellbe — Mon, 25 Nov 2019 08:03:55 GMT

Hi again,

Here is another screenshot from my OnePlus 5 running Android 9, attempting DFU through the NRF Connect App.

I've never had any issues on this phone before.

RE: DFU issues on Android 10

AntonHellbe — Fri, 22 Nov 2019 12:59:18 GMT

Hi,

Doubled checked that three or four times now: NRF_SDH_BLE_SERVICE_CHANGED 1.

I found a similar issue here on devzone where the suggested solution was to move the DFU service to the top of the attribute table to line it up with the bootloader attribute table. In the application we are making changes to the attribute table quite often.

Accidently closed the tab with that issue so I dont have think link at the moment, will try to find it again.

Also In the application I have given the Peer Manager control over Service Changed Indications, could that have any consequences?

Edit: Found the link

/ Anton

RE: DFU issues on Android 10

Einar Thorsrud — Fri, 22 Nov 2019 12:52:26 GMT

Hi,

Both these issues could be explained by missing service changed. Have you made sure the NRF_SDH_BLE_SERVICE_CHANGED is set to the same value in both the applications and bootloaders sdk_config.h? What is it?