Audio Recognition using Edge Impulse Machine Learning on the nRF52840 DK

Introduction

Machine Learning in the wireless edge-device space leverages the processing power of the local microprocessor to minimize the amount of data being transmitted. Since radio transmissions are high current events, this lowers the overall power being used and extends the battery life. Machine learning can be used in many applications where transmission of the raw data is less useful and more costly than only transmitting an indicator that a given state has been detected.

Setup

In this blog I used a X-NUCLEO-IKS02A1 expansion board to provide an IMP34DT05 microphone sensor for the nRF52840-DK. Using the Edge Impulse machine learning system and firmware, I trained the system to recognize the sound of a whistle in various environments. I then used Nordic UART service to implement a BLE peripheral to transmit the presence of the whistle to a mobile device. Other use-cases of this setup might be to detect a leaking water pipe, a barking dog, breaking glass, pre-detonation in an internal combustion engine or many others.

Build and Test the Audio Classifier

Following these steps from Edge Impulse, perform the following tasks to train and deploy an audio classification impulse to the nRF52840-DK:

1. Install the Edge Impulse Command Line Interface and open an Edge Impulse account.
   
   
2. Connect the X-NUCELO-IKS02A1 expansion board to the nRF52840-DK. I removed the center line of pins on the DK to prevent shorting to the expansion board.
   
   
3. Connect the DK to your PC's USB port using a micro-USB cable. Be sure to use the DK's debug port and not the nRF USB port. Switch the DK on.
   
   
4. Download the latest Edge Impulse Firmware and program the nrf52840-dk.bin file to the JLink drive of the DK. There are several ways to do this, but the simplest is to right-click on the file, select Send to and then click JLINK. After 20 seconds, press the DK’s reset button.
   
   
5. Open the Edge Impulse project web page and create a new audio project named “Whistle”. Name your project as you like, but I'll use the name "Whistle" from here on.
   
   
6. Open a command window and run edge-impulse-daemon to log into the Edge Impulse account and set the keys. This also connects the DK to the Whistle project.
   
   
7. Follow the steps here to build an audio classification task and collect data. Make sure to maximize the web page view so that you see the list on the left of the screen. When training the algorithm, the greater the amount of data you collect in as many different scenarios as possible will increase the likelihood of positively identifying your signal. The Edge Impulse system takes data directly from the hardware and uploads it to the cloud for analysis.
   
   
8. With the training data in place, Create an Impulse using the MFE signal processing block.
   
   
9. Once the data was processed, train the neural network to recognize the whistle among the background noises.
   
   
10. With the neural network trained, test the classifications using the Live Classification tab. It turns out that detecting a whistle is straightforward, so no additional training or retraining was required to achieve good performance. More subtle sounds will require substantial training to get accurate results.
    
    
11. Build the firmware and deploy the impulse to the nRF52840. This process downloads a zip file containing a .bin. I programmed the bin file to the DK by right-clicking on it, selecting Send to and then JLINK.
    
    
12. Close the previous Edge Impulse daemon command window (if it's still open) and open another command window. Only one serial connection can be open at one time.
    
    
13. Run edge-impulse-run-impulse. The results of the continuously running impulse will display in the command window.

Now that we have a trained machine learning impulse running and generating accurate results, it’s time to add that to a Nordic BLE peripheral and send the whistle detection to a mobile device. We'll use the Nordic UART Service to easily transmit the whistle detections to the UART app within nRF Connect for Mobile.

Modify the peripheral_uart Example to Send Whistle Detections

These steps will be added soon ...