data compression

[ziv123]

hello Nordic

i am working with nrf52832 and nrf52840, with zephyr (ncs v2.1.2)

i am trying to compress vibration sensor data before sending it via ble (i have several 16k byte samples that i read from external flash before sending, i have some limitation of RAM size so i can not use very big bufs on RAM) .

i have tried to use the zephyr built in LZ4 module on my data (sensors data) with not such good results.

also i did not see any difference on ratio when using the default ' LZ4_compress_default(..) ' and the compress oriented API ' LZ4_compress_destSize(..) ' 

i gave both APIs 2k of data to compress and it did not compress at all. 

1.  what is the difference between the 2 APIs ?

2. i saw there is also lz4_hc (high compress) .c module that i am not sure how to use and if it can bring better results ?

3. if i would try to use a dictionary on a 16k bytes of sample could it give some significant improve on ratio ? if so, how can i use it i did not find any example or guide on that ?

if no, are there any recommended compression algorithms i can try/test that i can integrate to zephyr / ncs, (or already integrated ? ) is there some guide for such integration ?

hope to read you soon

best regards

Ziv 

[Sigurd]

Hi,

This lz4 library does not seem to be commonly used, and as far as I can see, we are not using it for any of our nordic libraries and samples. If you need support on this lz4 library, then I think it's better to check with the Zephyr community.

When it comes to your use-case here, I think it's better to just send the data as soon as you get it. The time you spend on buffering and compressing the data, you could have spend that time sending the data instead.

[ziv123]

Hi,

Sigurd said:

I think it's better to just send the data as soon as you get it. The time you spend on buffering and compressing the data, you could have spend that time sending the data instead.

though i appreciate your thoughts on the matter i don't understand the "time" relevance to that matter, since my motivation to run compression (p.s. - loss less)  is to send less bytes OTA, to save on connection time with the central and battery. and in any case the time it will take me to compress the chunks of data will not cause and delay in BLE transmission on waiting for the compression. the data is collected in the endpoint according to some scheduler and there is no need to transmit the data collected while collecting it, but only after all sampling process has finished so no real time requirements like that.

hope to read you soon

best regards

Ziv 

[Sigurd]

Hi,

ziv123 said:

since my motivation to run compression (p.s. - loss less)  is to send less bytes OTA, to save on connection time with the central and battery.

Ok, I see your use-case now.

if so, how can i use it i did not find any example or guide on that ?

Did you have a look at this sample? https://github.com/nrfconnect/sdk-zephyr/blob/main/samples/compression/lz4/src/main.c

[ziv123]

Hi, 

Sigurd said:

Did you have a look at this sample?

yes i did look at the sample, it shows the basic default use of the lib, does not show how to integrate a dictionary

[ziv123]

hello Sigurd

we gave up the lz4 cause it did not gave such good compression rate for our data.

i so in the ncs under the nrf dir that there is some reference to Brotli compression (by google) but i only saw include for the decoding, so its not clear for me if there is some implementation of Brotli compression algorithm in ncs ??

hope to read you soon

best regards

Ziv

[Sigurd]

Hi,

ziv123 said:

so its not clear for me if there is some implementation of Brotli compression algorithm in ncs ??

No, this is not implemented in NCS.