An issue with memory allocation in NRF52832 when working with JSON files

RE: An issue with memory allocation in NRF52832 when working with JSON files

ZaQa — Sun, 06 Aug 2023 09:19:53 GMT

Hi Håkon,

[quote userid="2115" url="~/f/nordic-q-a/102436/an-issue-with-memory-allocation-in-nrf52832-when-working-with-json-files/439719"]try switching to static allocation for everything else except cJSON_* calls.[/quote]

I have been addressing this problem since then and I am pleased to report that it has been resolved successfully. Based on your previous comments, I have made changes to the memory allocation and replaced all the static cJSON calls with dynamic memory allocation. The memory leakage issue has been eliminated, and everything is functioning correctly.

Thank you very much for taking the time to guide me. I truly appreciate your patience.

Warmest regards,

ZaQa

RE: An issue with memory allocation in NRF52832 when working with JSON files

Håkon Alseth — Thu, 03 Aug 2023 13:34:53 GMT

For instance: for each malloc, print a "+", and for each free call, print a "-".

That list shows your firmware is leaking memory, try switching to static allocation for everything else except cJSON_* calls.

Also, I would recommend that you add some waiting logic (wait for the callback with evt BLE_GATTS_EVT_HVN_TX_COMPLETE) after the ble_nus_data_send() call, as this is a non-blocking function.

Kind regards,

Håkon

RE: An issue with memory allocation in NRF52832 when working with JSON files

ZaQa — Thu, 03 Aug 2023 12:58:01 GMT

Hi,

[quote userid="2115" url="~/f/nordic-q-a/102436/an-issue-with-memory-allocation-in-nrf52832-when-working-with-json-files/439664"]This is the only place where you actually free the allocated memory, thus the HEAP goes up again. You need to adjust your heap size and/or handle your timing accordingly.[/quote]

I tried it again, but this time I increased the heap size to 16KB. However, I obtained the same result:

allocated address : 2000955c

allocated address : 2000949c

allocated address : 2000945c

allocated address : 2000934c

allocated address : 2000930c

allocated address : 200092dc

allocated address : 2000920c

allocated address : 200091dc

allocated address : 2000915c

allocated address : 2000910c

allocated address : 2000908c

allocated address : 2000904c

allocated address : 2000901c

allocated address : 20008fcc

allocated address : 20008f1c

allocated address : 20008eec

allocated address : 20008edc

allocated address : 20008eac

allocated address : 20008e7c

allocated address : 20008e6c

allocated address : 20008e3c

allocated address : 20008dbc

allocated address : 20008d7c

allocated address : 20008d0c

allocated address : 20008c8c

allocated address : 20008c2c

allocated address : 20008bfc

allocated address : 20008bec

allocated address : 20008bbc

allocated address : 20008b3c

allocated address : 20008aec

allocated address : 20008adc

allocated address : 20008acc

allocated address : 20008abc

allocated address : 20008aac

allocated address : 20008a9c

allocated address : 20008a8c

allocated address : 20008a7c

allocated address : 20008a6c

allocated address : 20008a5c

allocated address : 20008a4c

allocated address : 200089cc

allocated address : 2000898c

allocated address : 2000895c

allocated address : 2000892c

allocated address : 2000891c

allocated address : 200088ec

allocated address : 200088bc

allocated address : 2000889c

allocated address : 2000888c

allocated address : 2000887c

allocated address : 2000886c

allocated address : 2000885c

allocated address : 2000884c

allocated address : 2000883c

allocated address : 2000882c

allocated address : 2000881c

allocated address : 2000880c

allocated address : 200087fc

allocated address : 200087ec

allocated address : 2000879c

allocated address : 200086ec

allocated address : 2000861c

allocated address : 2000852c

allocated address : 200084bc

allocated address : 200083cc

allocated address : 2000837c

allocated address : 200082fc

allocated address : 200082bc

allocated address : 2000823c

allocated address : 2000818c

allocated address : 2000810c

allocated address : 200080cc

allocated address : 20007ffc

allocated address : 20007fac

allocated address : 20007f6c

allocated address : 20007f3c

allocated address : 20007f0c

allocated address : 20007edc

allocated address : 20007e9c

allocated address : 20007e4c

allocated address : 20007dac

allocated address : 20007d4c

allocated address : 20007cfc

allocated address : 20007ccc

allocated address : 20007c4c

allocated address : 20007c1c

allocated address : 20007b9c

allocated address : 20007a7c

allocated address : 20007a4c

allocated address : 200079bc

allocated address : 2000797c

allocated address : 2000791c

allocated address : 200078bc

allocated address : 200077ec

allocated address : 200077bc

allocated address : 200077ac

allocated address : 2000779c

allocated address : 2000777c

allocated address : 200076dc

allocated address : 2000767c

allocated address : 2000764c

allocated address : 2000761c

allocated address : 2000757c

allocated address : 2000754c

allocated address : 200074ec

allocated address : 2000747c

allocated address : 2000742c

allocated address : 200073ac

allocated address : 2000736c

allocated address : 2000733c

allocated address : 2000730c

allocated address : 200072dc

allocated address : 200072ac

allocated address : 2000727c

allocated address : 2000724c

allocated address : 2000721c

allocated address : 2000714c

allocated address : 2000705c

allocated address : 20006fbc

allocated address : 20006f0c

allocated address : 20006e4c

allocated address : 20006e1c

allocated address : 20006d3c

allocated address : 20006d2c

allocated address : 20006d1c

allocated address : 20006d0c

allocated address : 20006cbc

allocated address : 20006bfc

allocated address : 20006b3c

allocated address : 20006afc

allocated address : 20006aec

allocated address : 20006adc

allocated address : 20006acc

allocated address : 20006abc

allocated address : 20006aac

allocated address : 20006a9c

allocated address : 20006a8c

allocated address : 20006a7c

allocated address : 20006a6c

allocated address : 20006a5c

allocated address : 20006a4c

allocated address : 200069ac

allocated address : 2000691c

allocated address : 200068ec

allocated address : 200068cc

allocated address : 2000683c

allocated address : 200067dc

allocated address : 200067bc

allocated address : 2000678c

allocated address : 2000673c

allocated address : 2000670c

allocated address : 2000668c

allocated address : 200065fc

allocated address : 200065cc

allocated address : 2000656c

allocated address : 200064cc

allocated address : 2000646c

allocated address : 2000643c

allocated address : 2000640c

allocated address : 2000639c

allocated address : 2000634c

allocated address : 200062fc

allocated address : 200062cc

allocated address : 2000629c

allocated address : 2000623c

allocated address : 200061ec

allocated address : 2000617c

allocated address : 2000615c

allocated address : 200060ec

allocated address : 2000608c

allocated address : 2000603c

allocated address : 2000601c

allocated address : 20005ffc

allocated address : 20005fdc

allocated address : 20005fbc

allocated address : 20005f9c

allocated address : 20005f6c

allocated address : 20005f4c

allocated address : 20005f1c

allocated address : 20005efc

allocated address : 20005edc

allocated address : 20005eac

allocated address : 20005e8c

allocated address : 20005e6c

allocated address : 20005e5c

allocated address : 20005e3c

allocated address : 20005ddc

allocated address : 20005d8c

allocated address : 20005d2c

allocated address : 20005d0c

allocated address : 20005cdc

allocated address : 20005c3c

allocated address : 20005c0c

allocated address : 20005bac

allocated address : 20005b4c

allocated address : 20005b1c

allocated address : 20005aec

allocated address : 2000596c

allocated address : 20005a2c

allocated address : 200059ec

allocated address : 0

Finally, for the first and last time allocated address goes up from 2000596c to 20005a2c and then decreases to 0.
Is it possible to clear the entire heap and start anew, perhaps when the allocated address reaches 0?

[quote userid="2115" url="~/f/nordic-q-a/102436/an-issue-with-memory-allocation-in-nrf52832-when-working-with-json-files/439664"]Try to print + when alloc'ing, and - when free'ing memory to see how often one occurs vs. the other.[/quote]

I'm unsure how to ensure the allocated memory has been freed since the free() function doesn't return any value to confirm its success. Do you have any suggestions on how to address this issue?

Thank you for your time.

RE: An issue with memory allocation in NRF52832 when working with JSON files

Håkon Alseth — Thu, 03 Aug 2023 10:09:36 GMT

Hi,

Looking at it again I found one place where the memory is free'd:

[quote user="ZaQa"]allocated address : 20005b5c
allocated address : 20005b1c
allocated address : 20005b8c[/quote]

This is the only place where you actually free the allocated memory, thus the HEAP goes up again. You need to adjust your heap size and/or handle your timing accordingly.

Try to print + when alloc'ing, and - when free'ing memory to see how often one occurs vs. the other.

Kind regards,

Håkon

RE: An issue with memory allocation in NRF52832 when working with JSON files

ZaQa — Thu, 03 Aug 2023 09:20:06 GMT

Hi Håkon,

[quote userid="2115" url="~/f/nordic-q-a/102436/an-issue-with-memory-allocation-in-nrf52832-when-working-with-json-files/439637"]Which compiler are you using? is alloc used in other functions of your firmware?[/quote]

I'm currently using Segger Embedded Studio as IDE and GCC as the compiler. As for the second question, alloc is only utilized within the cJSON code and not elsewhere.

I appreciate your response.

ZaQa

RE: An issue with memory allocation in NRF52832 when working with JSON files

Håkon Alseth — Thu, 03 Aug 2023 08:54:42 GMT

Hi,

Dynamic memory handling is not always easy to handle, and should ideally only be used when there's no other option.

cJSON is one scenario where it makes sense to use dynamic allocation.

[quote user="ZaQa"]Upon thoroughly examining my code, I have come to the realization that the free() function does not release allocated space. Consequently, over time, the heap section becomes full and ultimately causes the code to fail.[/quote]

Which compiler are you using? is alloc used in other functions of your firmware?

Kind regards,

Håkon

RE: An issue with memory allocation in NRF52832 when working with JSON files

ZaQa — Wed, 02 Aug 2023 17:20:49 GMT

[quote userid="2115" url="~/f/nordic-q-a/102436/an-issue-with-memory-allocation-in-nrf52832-when-working-with-json-files/439497"]You must check your return here, if not; you risk losing data.[/quote]

That's correct. I investigated and now I'm verifying the value returned by the ble_nus_data_send function, which seems to be working properly.

Upon thoroughly examining my code, I have come to the realization that the free() function does not release allocated space. Consequently, over time, the heap section becomes full and ultimately causes the code to fail.

Could you please guide me on how to effectively resolve this issue? I would greatly appreciate your kind assistance.

RE: An issue with memory allocation in NRF52832 when working with JSON files

Håkon Alseth — Wed, 02 Aug 2023 12:26:11 GMT

Hi ZaQa,

[quote user="ZaQa"]It seems that multiple function callings and transmitting the mentioned data cause the heap section to fill up after several uses. I have traced some allocated memory addresses in this scenario:[/quote]

You can lock the function with a semaphore for instance.

The allocation print indicate that you're not free'ing the data? It only goes in one direction.

You have to debug to see where the memory is leaking.

[quote user=""] ble_nus_data_send(&m_nus, (uint8_t *)out , &len, m_conn_handle);[/quote]

Sorry, I assumed you were using NCS, but this function call points to nRF5 SDK.

You must check your return here, if not; you risk losing data.

In general, I would recommend that you check your returned pointers, also from the cJSON_* calls as well.

Kind regards,

Håkon

RE: An issue with memory allocation in NRF52832 when working with JSON files

ZaQa — Wed, 02 Aug 2023 07:17:14 GMT

Hi,
Thank you for your prompt response.

[quote userid="2115" url="~/f/nordic-q-a/102436/an-issue-with-memory-allocation-in-nrf52832-when-working-with-json-files/439326"]What is the heap size set to on your end?[/quote]

It's 8kb (0x20005910 - 0x2000790f).

It seems that multiple function callings and transmitting the mentioned data cause the heap section to fill up after several uses. I have traced some allocated memory addresses in this scenario:

allocated address : 200074dc
allocated address : 2000741c
allocated address : 200073dc
allocated address : 200072dc
allocated address : 2000728c
allocated address : 200071cc
allocated address : 2000717c
allocated address : 200070fc
allocated address : 2000709c
allocated address : 2000706c
allocated address : 20006fec
allocated address : 20006fdc
allocated address : 20006f2c
allocated address : 20006e9c
allocated address : 20006e2c
allocated address : 20006dac
allocated address : 20006d3c
allocated address : 20006d2c
allocated address : 20006c4c
allocated address : 20006bdc
allocated address : 20006b6c
allocated address : 20006b3c
allocated address : 20006afc
allocated address : 20006acc
allocated address : 20006a8c
allocated address : 20006a7c
allocated address : 200069cc
allocated address : 200068fc
allocated address : 200068cc
allocated address : 2000681c
allocated address : 2000677c
allocated address : 2000673c
allocated address : 2000669c
allocated address : 2000664c
allocated address : 2000657c
allocated address : 2000655c
allocated address : 200064dc
allocated address : 200063ec
allocated address : 200063cc
allocated address : 2000635c
allocated address : 200062ec
allocated address : 2000626c
allocated address : 2000619c
allocated address : 2000611c
allocated address : 2000609c
allocated address : 2000608c
allocated address : 2000601c
allocated address : 20005fac
allocated address : 20005f3c
allocated address : 20005edc
allocated address : 20005e9c
allocated address : 20005e4c
allocated address : 20005ddc
allocated address : 20005d7c
allocated address : 20005d1c
allocated address : 20005cec
allocated address : 20005c5c
allocated address : 20005bec
allocated address : 20005b5c
allocated address : 20005b1c
allocated address : 20005b8c
allocated address : 20005b2c
allocated address : 20005a4c

As you may have noticed, the beginning and end of these addresses are quite close to the start and end of the heap section. Is there anything I can do to prevent this? My goal is for users to be able to access this data as frequently as they desire through the mobile app, but it currently only works for a limited time.

I appreciate the time you have taken.

ZaQa

RE: An issue with memory allocation in NRF52832 when working with JSON files

Håkon Alseth — Tue, 01 Aug 2023 15:21:55 GMT

Hi,

[quote user=""]

I have read in a thread that applying cJSON_Init() to the main would be helpful, but when I added this function, the JSON parse data process failed:

cJSON_Parse(received_data) --> cJSON_ParseWithOpts --> cJSON_New_Item --> cJSON_malloc

I suspect that there is something wrong with memory allocation. Could you please help me with this problem? Is there any configuration or consideration that I should have taken but missed out on?

[/quote]

What is the heap size set to on your end?

Please note that you're using libc dynamic alloc functions, ie. "malloc()" instead of zephyr's kernel heap "k_malloc()"- this will use the heap by your selected libc (normally newlib or minimal libc, symbols NEWLIB_LIBC_MIN_REQUIRED_HEAP_SIZE / MINIMAL_LIBC_MALLOC_ARENA_SIZE)

Kind regards,

Håkon