mqttsn with openthread and freertos

[EdgarGrimberg]

I am trying to hook openthread with freeRTOS.

Short story: the current state of my implementation is that I get a hardfault after succesfully calling mqttsn_client_search_gateway, but before receiving any data on the other end on the mqtt-sn gateway. The stack at this point starts in

> strace
warning: Couldn't determine the static tracepoint marker to probe
Static tracepoint 16 at 0x7d8a: file nRF5_SDK\external\freertos\portable\GCC\nrf52\port.c, line 104.

Long story:

From freeRTOS, I am mostly interested in tasks and message queues.

My skeleton implementation has 2 tasks:

1. the openthread processing thread:

static SemaphoreHandle_t threadMutex = nullptr;

static void thread_stack_task(void*)

{

    while (1)

    {

        xSemaphoreTake(threadMutex, portMAX_DELAY);

        thread_process();

        xSemaphoreGive(threadMutex);

        UNUSED_RETURN_VALUE(ulTaskNotifyTake(pdTRUE, portMAX_DELAY));

    }

}

and starting it like in one of the examples:

threadMutex = xSemaphoreCreateMutex();

// Start thread stack execution.

    if (pdPASS != xTaskCreate(thread_stack_task, "THR", THREAD_STACK_TASK_STACK_SIZE, NULL, THREAD_STACK_TASK_PRIORITY, &thread_stack_task_handle))

    {

        APP_ERROR_HANDLER(NRF_ERROR_NO_MEM);

    }

The other task is a state machine to handle different states in the connection process:

- initializing the thread structures

- joining a thread network: role changes to CHILD or ROUTER

- searching for a mqtt-sn gateway

- connecting the mqtt client, etc.

Here's how I start the state machine task (ConnectionStateMachine is a singleton for some other reasons later on) :

    csm = ConnectionStateMachine::GetInstance();

    csm->SetThreadMutex(threadMutex);

    if (pdPASS != xTaskCreate(

        [](void* stateMachine)->void

        { 

            ((ConnectionStateMachine *)stateMachine)->StateMachine();

        }, 

        "CSM", THREAD_STACK_TASK_STACK_SIZE, csm, THREAD_STACK_TASK_PRIORITY, &thread_stack_task_handle))

    {

        APP_ERROR_HANDLER(NRF_ERROR_NO_MEM);

    }

and the StateMachine method:

void ConnectionStateMachine::StateMachine()

{

    while (true)

    {

        Event event;

        if(pdTRUE == xQueueReceive(messasgeQueue, &event, portMAX_DELAY))

        {

            xSemaphoreTake(m_threadMutex, portMAX_DELAY);

            switch (state)

            {

            case StateUnitialized:

            {

                if(event == EventInitialized)

                {

                    //auto transition to an initialized state

                    ThreadInstanceInit();

                    m_client.Init();

                    state = StateInitialized;

                }

                break;

            }

where state is the state, and event is the transition.

To transition from a state, one needs to call:

bool ConnectionStateMachine::Transition(Event desiredTransition)

{

    bool retValue;

    //return false if the queue is full (in the middle of processing a state transition)

    retValue = (xQueueSend(messasgeQueue, &desiredTransition, (TickType_t)0) == pdTRUE ? true : false);

    return retValue;

}

The idea is that the callback in openthread and mqtt are on the context of the Thread task, and I can quickly wake up my StateMachine by adding a message to the queue:

static void state_changed_callback(uint32_t flags, void * p_context)

{

    if (flags & OT_CHANGED_THREAD_ROLE)

    {

        otDeviceRole role = otThreadGetDeviceRole((otInstance *)p_context);

        if (role == OT_DEVICE_ROLE_CHILD || role == OT_DEVICE_ROLE_ROUTER)

        {

            ConnectionStateMachine::GetInstance()->Transition(ConnectionStateMachine::EventConnected);

        }

    }

}

I can reach this part, and even transtion to the next state, where I call mqttsn_client_init and then call mqttsn_client_search_gateway successfully, with a return code NRF_SUCCESS. After this, I do not see any data transmitted to the gateway, not reaching the event handler callback set in the init function, and quickly (a few seconds) crashing in a hardfault originating from nRF5_SDK\external\freertos\portable\GCC\nrf52\port.c:104 in 

void xPortPendSVHandler( void )

At this point I have no ideas what to try to debug anymore. Any help is highly appreciated.

Thanks,

Edgar

[EdgarGrimberg]

Forgot to mention, the same build environment minus freeRTOS and re-arranged code to look more like the example code works fine.

[Jørgen Holmefjord]

Hi,

It looks like the hardfault happens when xPortPendSVHandler is executing. This will attempt to pop the current TCB value from the stack, but if the stack or stack pointer is corrupted at this point, it could lead to hardfault. First step is to increase the heap size in FreeRTOSConfig.h file and increase the default task stack size

You can try the following as a starting point:

#define configMINIMAL_STACK_SIZE ( 400 )
#define configTOTAL_HEAP_SIZE ( 1024 * 40 )

If this does not work, try to increase the stack size in 200 byte steps, and heap size in 1-2 kB steps. This can be fine-tuned at a later point once you get everything working correctly.

Best regards,
Jørgen

[EdgarGrimberg]

Hi Jørgen.

Thanks for the suggestions. Here's what I did:

- Double the heap in freeRTOS:

#define configTOTAL_HEAP_SIZE ( 1024 * 64 )

- Increase the stack sizes for:  * my state machine twice as I had, still crashes:

#define CSM_STACK_TASK_STACK_SIZE     (( 1024 * 16 ) / sizeof(StackType_t))   /**< FreeRTOS task stack size is determined in multiples of StackType_t. */

  * the openthread task, increased with (1024 * 2) / sizeof(StackType_t), still crashes. increased even with (1024 * 4) / sizeof(StackType_t), the same result:

#define THREAD_STACK_TASK_STACK_SIZE     (( 1024 * 12 ) / sizeof(StackType_t))

Even more, I've added the hard fault handlers from the SDK, and here's the log:

00> <info> app: Succesfully send the search for gate<error> hardfault: HARD FAULT at 0x00008D52
00>
00> <error> hardfault:   R0:  0x20040000  R1:  0x00000000  R2:  0x00000000  R3:  0x20015410
00>
00> <error> hardfault:   R12: 0x20020BD0  LR:  0x00008D45  PSR: 0x4100000E
00>
00> <error> hardfault: Cause: Data bus error (PC value stacked for the exception return points to the instruction that caused the fault).
00>
00> <error> hardfault: Bus Fault Address: 0x20040000

Setting a breakpoint here, I can see the following variables:and Still chasing a memory corruption, I've enabled more debugging from freeRTOS:

#define configCHECK_FOR_STACK_OVERFLOW                                            1

#define configUSE_MALLOC_FAILED_HOOK                                              1

and implemented my hooks:

void vApplicationMallocFailedHook( void )

{

    NRF_LOG_ERROR("malloc failed");

    while(1);

}

void vApplicationStackOverflowHook( void )

{

    NRF_LOG_ERROR("stack overflow");

    while(1);

}

but I never reach the infinite loops, only the hard faultThanks,Edgar

[EdgarGrimberg]

OK, I think I got it. I was single stepping through the mqttsn code, when stack overflow handler kicked. Really lucky here!

Here's what the current TCB was showing:

So it was the timer task that was running low on stack. The solution, of course, was to increase the task size for the timer task.

In FreeRTOSConfig.h, I bumped the timer task stack depth to 160:

#define configTIMER_TASK_STACK_DEPTH  ( 160 )

And it all went fine from there.

Thanks for pointing me into the right direction, really appreciate the support!

Regards,

Edgar