Edge computing is a powerful way to be able to implement low power resource-saving machine learning in your application. Using Edge Impulse alongside the nRF52840 or nRF5340, you can do simple machine learning tasks such as mono audio processing.
This guide presents a minimal working sample, based on the Woodpecker detector module in the nRF Beehavior project. This sample is a simple implementation of how to use sound in an Edge Impulse model, and is based on an already trained Edge Impulse model that detects woodpeckers. It scans for the presence of woodpeckers and outputs the data as a probability from 0 to 1. In the Beehavior project, the output is then used to notify the beekeepers if there are woodpeckers close to the hive, which is dangerous for the bees.
Since the nRF52840 DK and nRF5340 DK does not have a built in microphone I used this shield: X-NUCLEO-IKS02A1
If you want to use/or train your own model, a nice place to start is here: Getting Started EI
You can check out the video documenting the Woodpecker detector module in the Beehavior Monitoring project here: YouTube: Saving our bees from woodpeckers // Beehavior Monitoring
Getting data from the microphone
The method used to obtain the data from the microphone is based on the DMIC Sample in nRF Connect SDK. Some modifications are made to be able to use the predictions in the Edge Impulse: Wrapper.
The DMIC Sample is made to sample at a rate of 100 ms, while the EI-Wrapper requires 1 second. As a work-around, the code runs 11 times to generate one second and store it to memory. The first 100 ms sample is discarded due to noise from the microphone starting. The full second is then fed to the EI-Wrapper.
Using the El-Wrapper with audio
The data is fed to the EI-Wrapper using the function ei_wrapper_add_data(). At this point, Edge Impulse is ready to start prediction on the sample data, and the function ei_wrapper_start_prediction() is called to start the prediction.
When the prediction is ready, the callback function for the module is triggered and will, in this case, store the prediction value for one specific event. The callback function then clears the data to be ready for the next event.
Configuration notes
Remember to set the prj.conf option: CONFIG_EI_WRAPPER_DATA_BUF_SIZE to be above the sample rate specified during the download of the model from Edge Impulse.
Depending on which board or shield you use, you have to change the Device Tree overlay to the correct pins using pinctrl.
&pinctrl {
pdm0_default_alt: pdm0_default_alt {
group1 {
psels = <NRF_PSEL(PDM_CLK, x_port, x_pin)>,
<NRF_PSEL(PDM_DIN, y_port, y_pin)>;
Replace x_port, x_pin, y_port and y_pin with the relevant pins for your setup.
The correct pins for the nRF52840 DK and the nRF5340 DK can be found from this datasheet X-NUCLEO-IKS02A1:
The "C9" header on the shield corresponds to pins P1.01-P1.08 on the nRF52840 DK and P1.00-P1.09 (omitting P1.02 and P1.03) on the nRF5340 DK.
For the nRF52840 DK with the X-NUCLEO-IKS02A1 this is the correct pinctrl settings:
&pinctrl {
pdm0_default_alt: pdm0_default_alt {
group1 {
psels = <NRF_PSEL(PDM_CLK, 1, 4)>,
<NRF_PSEL(PDM_DIN, 1, 5)>;
};
};
};
For the nRF5340 DK with the X-NUCLEO-IKS02A1, this is the correct pinctrl settings, found by looking at the pins on the DK:
&pinctrl {
pdm0_default_alt: pdm0_default_alt {
group1 {
psels = <NRF_PSEL(PDM_CLK, 1, 5)>,
<NRF_PSEL(PDM_DIN, 1, 6)>;
Output
The results of the model can be viewed through "Jlink RTTViewer" and looks like this, the result is a probability between 0 and 1 where 1 is 100% probability of woodpecker:
Project
The sample can be found here: https://github.com/NordicPlayground/nRF-Beehavior-Firmware/tree/master/Legacy_Modules/ei_sample