nrf_crypto_aes_crypt() fails when interrupted by SoftDevice

Question

Hello: 
 We have a BLE application on an NRF52840 with SD 140 and SDK 1.5.3. Our device acts as a BLE peripheral with bonding and LESC, and we additionally use block level aes encryption at the application level for all incoming and outgoing characteristics. For encryption, we are using nrf_crypto_aes_crypt() for our encrypt/decrypt operations. We have observed a case when we are encrypting packet of data using nrf_crypto_aes_crypt() when an incoming encrypted write occurs, which interrupts (from the soft device) our encryption process with an attempt to decrypt the incoming packet, also using using nrf_crypto_aes_crypt(). The interrupting operation fails with NRF_ERROR_CRYPTO_BUSY because of a mutex and can never succeed because of the priority inversion condition that exists until we release back to the original encrypt operation. What we've noted, however, is that when this occurs, the main loop's interrupted encryption also fails with NRF_ERROR_CRYPTO_BUSY for reasons we cannot describe. This failure should have been prevented when the interrupting decrypt call evaluates the mutex on lines 454/455 of cc310_backend_aes.c as far as we can tell. Is there something deeper that may cause a failure when we return from the interrupt context? For now, our approach is to enter a critical region with sd_nvic_critical_region_enter/exit() for encryption operations which appears to prevent the issue. But we are concerned that the mutex in cc310_backend_aes.c isn't working or is somehow stomping on the main context's crypto operations, or we are otherwise masking a larger issue. Any advice for additional troubleshooting or mitigation would be appreciated. 
 Thanks 
 Dan

greenlava · Answer

For sure. it turns out I was misreading the failure mode a little bit. The notes I had about the incoming characteristic Write decryption failure with NRF_ERROR_CRYPTO_BUSY because of the priority inversion were correct, but my read of the underlying data encryption from the main thread failing was incorrect.

Instead, what we observed was that since our Main loop loads the BLE TX buffers until it receives NRF_ERROR_RESOURCES, then counts on the BLE callback for BLE_GATTS_EVT_HVN_TX_COMPLETE to load additional buffers, repeating this cycle until all data has been transferred. Where the issue was is that the encryption operation takes a measurable amount of time, so our data stream from the Main Loop that initially loads all the buffers was being interrupted by the TX_COMPLETE event, and those encryption operations were failing if an encryption operation was already in progress.

For posterity, our fix was to send only 1 data packet to BLE TX buffer from the main thread, and allow the rest of the data transfer to occur via the higher priority event handler context.