High RAM Application Questions

RE: High RAM Application Questions

Dave_couling — Wed, 12 Aug 2015 04:06:45 GMT

I think I may have found the issue. Perhaps some insight from Nordic could help. My application uses the app_button module which internally uses the GPIOTE for timing. I also use GPIOTE for PWM generation. I just noticed that my application had APP_GPIOTE_MAX_USERS 2

I believe that under a specific race condition a button push and buzzer tone attempting to generate at the same time may have caused the odd Lock-up issue. If I think I may have found the issue. Perhaps some insight from Nordic could help. My application uses the app_button module which internally uses the GPIOTE for timing. I also use GPIOTE for PWM generation. I just noticed that my application had APP_GPIOTE_MAX_USERS 2

I believe that under a specific race condition a button push and buzzer tone attempting to generate at the same time may have caused the odd Lock-up issue. Does this sound legit?

RE: High RAM Application Questions

Dave_couling — Mon, 10 Aug 2015 05:42:30 GMT

Hi RK,

Currently I ignore the error NRF_ERROR_NO_MEM and move-on, effectively skipping the event_put. I currently don't drop events. I'll have to look more closely at this. Perhaps there is something there.

I'm more concerned that an app_timer timeout handler cannot execute due to the FULL queue. This is a condition I can't trap.

RE: High RAM Application Questions

RK — Mon, 10 Aug 2015 03:14:02 GMT

That's odd - I can connect to running boards without a power cycle, it connects then I can halt. I'll try one over battery power later and check it's true in that case as well. There are other options for getting debug info later but they require manually erasing a flash page, dumping the registers and the contents of the stack page, you can try that, it's 30-40 lines of assembler. Then you pull the data off and laboriously cull through it.

So you think that it's the scheduler queue filling up. What do you do if app_sched_event_put() returns NRF_ERROR_NO_MEM? Do you use APP_ERR_CHECK() on it and go to the error handler? can you drop events if you can't queue them, can you at least make it light a different LED so you know this is the case?

A quick look at the code for app_timer says it only calls the scheduler on timeout events.

RE: High RAM Application Questions

Dave_couling — Mon, 10 Aug 2015 02:56:21 GMT

Unforuntely we cannot get stack traces as there is a power-cycle uponing access the SWD pins. I believe the issue may again be related to the Scheduler. We use BLE event to trigger scheduler events. These events start app_timers. It is likely a condition where several BLE events (Stack uses the scheduler as well) causing several scheduler events which somehow fills up the Scheduler Queue and then odd things happen. I don't really know how to design around this. I cannot increase the Scheduler Queue anymore without RAM overflows.

When using app_timers with the scheduler, do app_timer_start and app_timer_stop use the scheduler as well? or just the timeout_handler? Trying to track down how I could be filling up the queue.

Thanks for the help guys.

-DC

RE: High RAM Application Questions

RK — Sat, 08 Aug 2015 03:07:42 GMT

One of those hard debugging situations, it only breaks when you're not looking. What errors cause the LED to turn on and the next push to reset? APP_ERROR_CHECK(), hardfault, all of the above? That might help fractionally to narrow it down, but you probably have lots of app error checks so it may not.

You can run release firmware on a board and still get a debugger on it when it fails. That at least gives you a stack trace and there's enough information in the original binary file to figure out where you were, especially if you've built release code (ie optimized) with debug info.

You need some kind of post-mortem when the thing fails, somehow, it's the only way.

RE: High RAM Application Questions

Dave_couling — Sat, 08 Aug 2015 02:18:19 GMT

You absolutely correct, I was speaking in terms of external perception. When this happens a GPIO does raise (red LED turns on). Upon on the next button push the device restarts.

We can't get this to occur on the bench, have only seen it in field tests with release firmware.

RE: High RAM Application Questions

RK — Sat, 08 Aug 2015 02:09:23 GMT

CPUs don't freeze, they get in loops, they go to fault handlers when they detect error conditions, and they are very good at finding seemingly impossible race conditions, but they don't just stop working. Your CPU is running and it's doing something even though your board is unresponsive. What does your hardfault handler do, just loop, how about your app error handler, does it turn on some LEDs or raise a GPIO line you can check externally? When the device becomes unresponsive can you get a debugger attached to it and find out where it is?

RE: High RAM Application Questions

Dave_couling — Sat, 08 Aug 2015 01:28:30 GMT

We have seen (seemly at random) when I go to press the button, the chip seems to grind to a halt. This can happen at any point. Within 2hrs of programming or +3weeks. I had suspected that something to do with the app_button module was causing this. Since the Button Module uses the scheduler, along with App_timers and SoftDevice. Could these 3 items all using the Scheduler at the same time potentially cause a CPU freeze?

I have recently bumped up the SCHED_QUEUE_SIZE to 30 (from 18). Is there a rule of thumb for setting this? We have 20 app_timers, 2 other misc app_sched_event_put handlers, app_button and the SoftDevice. Thanks for the help.

RE: High RAM Application Questions

Dave_couling — Fri, 07 Aug 2015 07:12:13 GMT

I have seen this blog, thought using Eclipse it is not quite as useful. I do not have any malloc() is my code.

RE: High RAM Application Questions

Øyvind Karlsen — Fri, 07 Aug 2015 06:28:59 GMT

Hi,

Have you seen the RAM reduction measures described in this blog?

RE: High RAM Application Questions

RK — Fri, 07 Aug 2015 02:59:03 GMT

if you look in the .ld file there should be a check that ram doesn't overflow into stack. As long as you pass that, and your stack stays within limits during runtime, you're good to go. That's one of the benefits of having static RAM allocation instead of using heap/malloc() etc, if it links, it runs.

RE: High RAM Application Questions

Dave_couling — Fri, 07 Aug 2015 02:53:06 GMT

My question was specifically about run-time RAM usage. The project compiles, links and programs fine. But since I've designedd such a RAM heavy application I was to make sure I have enough headroom for whatever is happening during run-time. I've set my APP_TIMER_MAX to 4 above what I actually use, and set the queue to 2 more than the total number of app_timers, just to be sure.

More than anything I just want to make sure there isn't some consideration i should be making when using so many app_timers (besides the obvious ones like settings the max, and using the scheduler). Thanks for the reply RK

RE: High RAM Application Questions

RK — Fri, 07 Aug 2015 01:30:03 GMT

Don't totally understand the question. If the app links without error and fits in the available space, it works, the link script has a check for this. The startup file is entirely generic and just copies things from one place to another, it doesn't matter if it copies 1 byte, 10 bytes or 16Kb.

What issues were you concerned about, perhaps I'm just missing the point?

That is a lot of RAM usage, the map file should tell you what it's all used for, although they're annoyingly tricky to read, you might want to confirm to yourself that it is app timer usage (they use that much??) and not something else.