Feasibility: continuous audio capture + FFT processing + LTE-M clip upload on nRF9151

[Thijmen de Vries]

Hi,

We’re Hydro-Guard, a volunteer student team (TU Delft / University of Milan) building solar-powered acoustic recorders that detect illegal boats in nature parks (short video). We currently use an ESP32 and are considering the nRF9151 for our next generation.

What the device needs to do:

1. Continuously sample a digital MEMS mic (PDM or I²S) at 16 kHz.
2. Run an FFT-based detector on the application core: ~75 FFTs of 4096 points per 5 s clip, ~20–40 kB working memory.
3. Compress selected 5 s clips (~10–40 kB each) and upload them over LTE-M, up to ~100 clips/day.
4. Run on a small solar panel + battery.

Questions:

• Is there enough RAM left for audio buffers and this processing while the modem library and TLS are active?
• Can audio capture and processing keep running while the modem is transmitting?
• Are there samples or reference designs for audio capture + LTE-M upload on the nRF91 series?

We’re new to the Nordic platform, so any guidance on where to start is very welcome.

Thanks!
Thijmen, Hydro-Guard

[Øyvind]

Hello Thijmen,

First of all, welcome to the Nordic universe! Before starting I hope you are familiar with our academy.nordicsemi.com and docs.nordicsemi.com. In regards to the hardware design guidelines for the nRF9151, have a look at this link to ensure proper design around the solar panel + battery. 

In regards to what you are trying to achieve, there are no specific audio samples for the nRF9151. There is support for nRF9151 i2s and nRF9151 PDM using e.g. inter ic sound (i2s) API, which you could try on your side. Have a look at the Asset Tracker Template which might be a good starting point, documentation found here. You could also use the HTTP Client Sample as a starting point. 

• nRF9151 application core has 256 KB RAM total
• The Modem library requires shared memory that must reside in the lower 128 KB of RAM
  • Have a look at Cellular iot fundamentals lesson 3 nrf-modem-library-api

If there is not enough memory on the nRF9151 in order to reach your goals, then I suggest adding e.g. nRF54LM20B which can support FFT (Fast Fourier Transform) through the Axon NPU.

Or wait for the upcoming nRF92 Series which also adds the Axon NPU, our dedicated AI accelerator core, designed to increase the speed and efficiency of TensorFlow Lite models at the edge. It delivers up to 15× higher energy efficiency and up to 15× faster inference compared to running LiteRT on the CPU, and adds new capabilities such as audio classification. You can for example integrate voice control capabilities directly into your application running on the nRF92 Series cellular module, saving power and reducing the need for a host processor. (Edge AI for cellular)

Kind regards,
Øyvind

[Thijmen de Vries]

Hi Øyvind,

Thank you so much for the warm welcome and for taking the time to answer in such detail. This really helps us get started on the right foot.

Our nRF9151 DK is on its way, and we will start this weekend. Following your advice, we will begin with the HTTPS Client sample to get LTE and uploads working, then add our existing I²S microphone through the Zephyr I²S API. The hardware design guidelines for the solar panel and battery are exactly what we needed for the next step, so thank you for pointing us to those as well.

We would be happy to share our measurements and results with you once we have them. It would be great to show what the nRF9151 can do for protecting nature areas.

Thanks again, and kind regards,
Thijmen

[Øyvind]

Looking forward to hear about your work! Please do not hesitate to reach out if you need any further help/answers.

Kind regards,
Øyvind

[Thijmen de Vries]

Hi Øyvind,

As promised, here’s an update. The DK arrived and we spent the last two days on it. The good news is that a first version of the recorder already runs end to end on the nRF9151.

What works now

• Audio capture. Our I²S mic (ICS-43434) needs 64 SCK per frame (24-bit data in a 32-bit slot). The nRF9151 I²S has no 32-bit SWIDTH, so in master mode every 4th sample came out stale (25 % repeated samples). The workaround: one PWM sequence on the 16 MHz clock drives SCK and WS on the same pins, and I2S0 runs in slave mode (SWIDTH 24) and captures DOUT. That gives 15,625 Hz with zero repeated samples. We then resample to 16 kHz in firmware.
• Detector on the application core. Our FFT-based boat detector (FFT 1024) runs in real time: about 1.7 s of CPU per 5 s window, and about 1.9 s while LTE is active. Its output matches our Python reference on whole windows.
• RAM. It fits, but only just: 98.7 % of the 212 kB available to the app is in use, with the modem library and TLS active. Your note about the lower 128 kB was useful here.
• Upload. LTE-M with modem TLS sends 5 s clips to our backend over HTTPS (HTTP 200). Reusing one connection while clips are queued brought the cost per clip from ~8 s to ~2-2.7 s. PSM is granted on KPN LTE-M (band 20).

First PPK2 measurements

• Always listening plus the detector, with the modem in PSM: about 1.7 mA at 3.7 V (~6.3 mW).
• One upload on a new TLS connection: ~1.2 J, plus ~0.6 J for the radio tail before PSM. On a kept connection it’s ~0.37 J per clip.
• On the same daily workload as our ESP32 setup, that comes to roughly 0.3-0.4 Wh/day, against ~2.1 Wh/day for the ESP32. That’s about 6× lower, which matters a lot for a small solar panel. External flash writes and weak-signal conditions aren’t in these numbers yet.

A few questions, if you have time

1. Is PWM-generated SCK/WS with I²S in slave mode on the same pins a reasonable approach on the nRF9151, or is there a cleaner way to get 64 SCK per frame?
2. Our listening current is mostly the I²S and PWM running 24/7. Do you have tips to lower it, for example running from HFINT instead of HFXO, and how stable is HFINT’s frequency over temperature for audio?
3. A new TLS handshake is our most expensive step. Does the modem support TLS session resumption, and would you recommend it for ~5-15 uploads per hour?

Next steps

Based on these results, we’re now looking at developing our own board around the nRF9151: solar charging, battery, the I²S mic and external flash for an offline clip queue. We’d like to have our first devices deployed in a nature reserve by the end of November, so the timeline is tight.

Would Nordic be open to supporting us with this? As a volunteer student team we’d be very grateful for any help. In particular, a review of our schematic and layout (especially the antenna and power parts), or a contact with a field application engineer, would help us avoid mistakes on our first board. We’d be happy to share our design, measurements and field results in return, for example as a case study.

Two small things that cost us time and might help other newcomers. First, samples without their own partition overlay (like blinky) reserve space for an MCUboot that is never built, so TF-M ended up at the wrong address. The at_client overlay with nrf91_no_bootloader_partitions.dtsi fixed it. Second, opening SEGGER J-Link Commander tried to replace the on-board J-Link OB firmware and the board dropped off USB until we replugged it. nrfutil works fine.

Thanks again for pointing us in the right direction!

Kind regards,
Thijmen, Hydro-Guard