Do not know what value of RAM to configure after adding new characteristic to custom BLE service

[Xander To]

Hi Nordic team,

I'm using S132 with nRF52832 on my custom board. The SDK is nRF5_SDK_17.1.0_ddde560

I have made a custom BLE service that worked for several months. Now I want to add a new characteristic to my custom service. I know what to do in firmware as the new characteristic is 99% similar to the existing custom characteristic inside the custom service.

Firstly, my custom board crashed as expected because I didn't reconfigure the RAM space and 

NRF_SDH_BLE_GATT_MAX_MTU_SIZE in sdk_config. Therefore, I tested this firmware on DK to get the debug message (I currently don't have the hardware to debug the custom board so all debug tasks are on the DK). Initially, here are the RAM configurations on DK.

The original NRF_SDH_BLE_GATT_MAX_MTU_SIZE value is 1408. Based on the Segger IDE suggestion, I changed the RAM_START to 0x20002D30. I keep the NRF_SDH_BLE_GATT_MAX_MTU_SIZE  unchanged. The firmware works on the DK. I can see the new characteristic on my custom service. So that's good!

However, weird things started when I uploaded the same firmware on the custom board. The original RAM configurations of custom board are 

I know the original RAM configuration between DK and custom board are not the same but since they worked fine in the past, I didn't bother to match them together. But today, after I changed the RAM configuration on custom board to be the same as DK's configuration , the board crashes. Below is the custom board RAM configuration after the change

I thought it's probably the RAM change causing the crash. So instead of matching the RAM configuration of 2 boards, I increase the original RAM location up by 8 (0x20002D30 - 0x20002D28), similar to what I did on the DK. However, the board still crashes.

I tried changing the 

NRF_SDH_BLE_GATTS_ATTR_TAB_SIZE to 1460. It still crashes.

I removed the BLE code in my firmware, it doesn't crash.

I added the BLE code back, but this time removed my new characteristic. It doesn't crash.

So that means my custom board is lacked of RAM to add the new characteristic and the amount of extra RAM required is not the same with the DK. Therefore, can you please give me a suggestion of what kind of RAM value I should change it to on the custom board? I program the DK with Segger IDE but program the custom board with GCC

Thank you for your help and suggestion.

Best regards,

Xander

[Einar Thorsrud]

Hi,

When you add more services or characteristics, the SoftDevice needs more RAM, which means that the application start address must be higher (and the size correspondingly smaller). The simplest way to do this is to increase the app start address significantly, and then adjust the RAM start address to what is calculated by the SoftDevice. See this post for details.

However, there are also other things that may need to be adjusted. What exactly happens when you write that the device "crash"? Have you debugged to see if there is a specific function call that returns an error? Sometimes it could be that you need to increase NRF_SDH_BLE_GATTS_ATTR_TAB_SIZE in sdk_config.h and/or NRF_SDH_BLE_VS_UUID_COUNT (if adding vendor specific UUIDs). Note that if you do this, you also need to adjust the application RAM start address (and size).

[Xander To]

Hi Einar,

I currently don’t have the debugging hardware until next month so I’m trying to debug without a debug messege. I just add a new characteristic to my existing custom service (no new service is added compared to the previous firmware version) so do I still have to change the NRF_SDH_BLE_VS_UUID_COUNT.

I will try your suggest approach. However, I’m still very confused why DK and custom board using same firmware but require different RAM/ Flash configuration? I don’t know the debug message but I’m sure the error happens during the new characteristic being added to the service as commenting this section will result in no more crash.

Best regards,

Xander

[Einar Thorsrud]

Hi Xander,

Xander To said:

I currently don’t have the debugging hardware until next month so I’m trying to debug without a debug messege.

What about the tool you use to program? Or is that via DFU? (If you have some form of SWD programmer, that can typically also be used to debug).

Xander To said:

I just add a new characteristic to my existing custom service (no new service is added compared to the previous firmware version) so do I still have to change the NRF_SDH_BLE_VS_UUID_COUNT.

Then there is no need to adjust NRF_SDH_BLE_VS_UUID_COUNT (NRF_SDH_BLE_GATTS_ATTR_TAB_SIZE could still be relevant, though).

Xander To said:

I will try your suggest approach. However, I’m still very confused why DK and custom board using same firmware but require different RAM/ Flash configuration?

If it is the exact same firmware, then you should not need to change the memory configuration. Is it the exact same firmware, or are there same changes? If so, what are those changes?

[Xander To]

Hi Einar,

I program the custom board with ST-Link and I cannot debug with ST-Link. I know I can debug the custom board by connecting to the DK pins, but I currently don’t have the wire to connect them together until next month. 
Yes, the firmware are exactly the same. I will double check this fact to ensure I didn’t mistakenly make any change between testing on DK and custom board.

Best regards,

Xander

[Turbo J]

OpenOCD should support debugging NRF52x chips with an ST-Link. Bit more difficult to set up than JLink, but doable.

RTT does not work with ST-Link though.

[Xander To]

Hi Einar,

Einar Thorsrud said:

The simplest way to do this is to increase the app start address significantly, and then adjust the RAM start address to what is calculated by the SoftDevice. See this post for details.

I just changed FLASH_START from 0x26000 to 0x56000 but the program just doesn't run at all. No crash is detected. I can compile and flash the code to the DK but it just doesn't run. Even a breakpoint doesn't help to see if the program can run and pause at the desire breakpoint.

Einar Thorsrud said:

If it is the exact same firmware, then you should not need to change the memory configuration. Is it the exact same firmware, or are there same changes? If so, what are those changes?

I can verify they use the exact 100% same firmware. I also copied the FLASH and RAM from the DK to the custom board.

Weird behavior detected: I tried removing the newly added characteristic to see if I can make the firmware not crash. However, it's still crashed. Then I changed the NRF_SDH_BLE_GATTS_ATTR_TAB_SIZE from 1508 back to 1408 then the app won't crash anymore on my custom board. However, on the DK, the app can run fine with NRF_SDH_BLE_GATTS_ATTR_TAB_SIZE = 1508. I changed this value from 1408 to 1508 because the app (with added characteristic) will crash on DK if I don't do so. This is extremely confusing as they always share same firmware, same RAM/Flash configurations, same sdk_config

Weirder behavior detected: This is something I have observed several times in the last week but I thought it was my mistake until today I clearly witnessed it. My firmware can be configured to enable/disable debug LEDs on the custom board. I just removed completely everything related to the newly added characteristic, this time the firmware still crashed and I'm so frustrated. Then I disable the LEDs, the firmware won't crash anymore and work fine. It makes me think maybe my LEDs code has issued. However, I press Ctrl + Z in VSCode ( just 1 press to enable the LEDs back in my personal config file), then this time firmware still works with LEDs. Therefore, I can confidently conclude that somehow, the same firmware can crash or cannot crash. This doesn't sound too strange for me. However, I stay at this firmware, trying to upload the code 10 times, and none of them crash, and this is super weird to me now. In conclusion, it looks like to make version A of a firmware work, I have to flash a version B to the board, then upload version A of firmware again and it works. This is insanely wrong to me. 

Best regards,

Xander

[Turbo J]

Einar mean to change RAM_START and not FLASH_START.

The latter won't work unless you modifiy the bootloader - softdevice is only able to start from the address specified in the sds or a bootloader via MBR or UICR.

That LED disabling helps sounds very sus to me -  a problem with the layout like not enough decoupling capacitors seems likely.

[Xander To]

Hi Turbo,

But is it possible if an external signal affect an output pin to crash the board? I don’t think the capacitors matter?

Best regards,

Xander

[Einar Thorsrud]

If you don't add characteristics or change the attribute table size compared ot the firmware you run on the DK, there is no need to change the RAM start address. So I suggest you don't make any changes there for now, to keep the changes to a minimal while trying to understand what is going wrong.

Then, you essentially have firmware that works on the DK but not on your custom board, even though the same firmware should have worked on both devices. That leads to the question about your HW and how it differs from the DK, and which HW features you use. Particularily this problematic LED is odd. Can you share your schematics and layout, and explain which LED that cause problems (if you have multiple)?

[Xander To]

Hi Einar,

I have tried removing any changes on RAM start address. However, the firmware after removing all change will work on both side. From this starting firmware (the working firmware before I made any change), adding the new characteristic will definitely crash the custom board and DK (should be due to not enough RAM). Increasing the RAM start address will make DK work and custom board still crashes. Staying at this point, I have a firmware that works on DK and crashes on custom board.

Below is the schematic of our custom board, including the 2 LEDs I mentioned before: LED.Blue and LED.Red. These LEDs are controlled by PPI module. Noted that for firmware using on DK, I have to disable LEDs as DK doesn't have LEDs on these pins. Firmware with disabled LEDs (to match firmware on DK) will still crash on custom board.

[Einar Thorsrud]

Hi,

Xander To said:

Firmware with disabled LEDs (to match firmware on DK) will still crash on custom board.

OK, then we can probably ignore the LEDs. But what did you not write earlier that you saw a difference in behaviour depending on a LED? 

Xander To said:

From this starting firmware (the working firmware before I made any change), adding the new characteristic will definitely crash the custom board and DK (should be due to not enough RAM). Increasing the RAM start address will make DK work and custom board still crashes.

That is most peculiar. This should not be related to the HW differences at all.

With the above two points in mind, could it be that this issue is triggered by something else altogether, and that it is easy to jump to conclusions when testing as behavior changes when you change something, but perhaps with different test runs you will get different behaviors even without any SW modifications?

This is a bit fuzzy and it is difficult to grasp what the issue really is. I would focus on getting debugging up and running as soon as possible. Note that as you have a DK, you can use that as a debugger for your custom board using the debug out port.