Using LIBUARTE during flash operations

RE: Using LIBUARTE during flash operations

ovrebekk — Tue, 25 Jun 2019 14:10:03 GMT

That is excellent news. If you have more questions, or encounter some other issue, just let me know 🙂

Best regards
Torbjørn

RE: Using LIBUARTE during flash operations

oguzcan oguz — Tue, 25 Jun 2019 09:52:18 GMT

Hi,

With this patch we haven't seen the issue, yet. So it seems to work for now; in case we notice anything, we will update you further. Thank you.

Regards,

Oguzcan

RE: Using LIBUARTE during flash operations

ovrebekk — Tue, 25 Jun 2019 08:33:09 GMT

Hi again

The developer discovered a potential race condition in the code that might be related to your issue.

He implemented a possible fix and forwarded it to me so you can test it:
nrf_libuarte_async.c

Could you please try this implementation out with the timeout set to 5ms and see if the problem is still there?

Best regards
Torbjørn

RE: Using LIBUARTE during flash operations

ovrebekk — Tue, 25 Jun 2019 07:53:42 GMT

As it seems the developer did not agree with my hypothesis. The driver does use the double buffering technique, and they have tested it with interrupts of up to 100ms to make sure it can handle long periods of 'stalls' without losing data.

He asked me if you could try to enable logging for this module, and see if any errors are returned during operation?

Can you give some more information about the data corruption you see?
You mentioned the data appears to be mixed, do you mean that every other byte is from the first burst or the second?

If the problem persists I think I will have to setup a small test application here to try to reproduce the issue.

Best regards
Torbjørn

RE: Using LIBUARTE during flash operations

oguzcan oguz — Mon, 24 Jun 2019 12:56:04 GMT

When setting priority to _PRIO_APP_LOW, the issue is still there. I did more testing with the 2 ms, and that seems to be reliable in our test test setup - problem is that there might be sequences where the second burst is equal to that timout as well.

Please double check your hypothesis with the designers.

RE: Using LIBUARTE during flash operations

ovrebekk — Mon, 24 Jun 2019 07:26:52 GMT

Hi Roy

[quote user="RoyCreemers"]A: The "irq_prio" is filled in with 5[/quote]

That's a bit odd. Legal values when using the SoftDevice are 2, 3, 6 and 7.

Defines for setting IRQ priority can be found in app_util_platform.h in the SDK:

#define _PRIO_SD_HIGH 0
#define _PRIO_SD_MID 1
#define _PRIO_APP_HIGH 2
#define _PRIO_APP_MID 3
#define _PRIO_SD_LOW 4
#define _PRIO_SD_LOWEST 5
#define _PRIO_APP_LOW 6
#define _PRIO_APP_LOWEST 7
#define _PRIO_THREAD 15

If the problem seems to be solved when the timeout is shorter than the time between bursts I expect the issue to be triggered when the DMA buffer fills up in the middle of a burst, but the hardware should handle this if you set up two buffers and configure the DMA to switch to the second buffer automatically once the first fills up.

I can double check with the designers if the driver is set up to do this.

Have you done more testing at 2ms to see if this setting seems reliable?

Best regards
Torbjørn

RE: Using LIBUARTE during flash operations

RoyCreemers — Fri, 21 Jun 2019 12:14:29 GMT

Let me start with answering some of your previous questions:

Q:To summarize, you confirm that this can happen with or without flash erase running?
A:This happens without flash erase.

Q:You confirm that this will only happen when using the libuarte_async library?
A: Yes. When using the nrfx_uarte driver directly, we did not have this issue.

Q:How do you process UART events in the application? Are you running a lot of event processing in the events directly, or do you process it later in main/thread context?
A: Data is put in a buffer, and processed in the main thread. No processing is handled in IRQ context. Same for BLE events: everything is queued for processing outside of interrupt context.

Q: Do you know what interrupt priority the UART events are returned in
A: The "irq_prio" is filled in with 5

The corruption only happens on the receive action. I looks like the problem occurs in case I receive new data during the timeout. What I see only the logic analyzer is that we often have a UART sequence looking llike the following

<~200 bytes on UART Rx> <no communication for ~5 ms> <~200 bytes on UART Rx> <no communication for ~X ms)

Initialily, I had the timeout set to 5 ms. I noticed the error very often. It looked like the data from the first and second burst where mixed.

Once I set the timeout to 15 ms, I see it very rarely: actually only when my "no communication" window is equal to X.

Now I reduced it to 2 ms, it works without issues (in the 15 minutes I tried this), probably because this "no communication" window is very unlikely to happen.

RE: Using LIBUARTE during flash operations

ovrebekk — Fri, 21 Jun 2019 11:29:15 GMT

Hi Roy

That is good to hear 🙂

One of the SoftDevice developers confirmed that the assert was related to a delayed interrupt in the SoftDevice, which happens if you configure interrupts to run at the highest priorities (should be reserved for the SoftDevice only).

As soon as you have more information on the corruption issue just let me know.

I assume you are talking about UART corruption?

Does it happen both on RX and TX?

Does it happen with or without flash access enabled?

And any more details you think might be relevant.

Best regards
Torbjørn

RE: Using LIBUARTE during flash operations

RoyCreemers — Fri, 21 Jun 2019 09:22:26 GMT

Hi Torbjørn,

I was preparing all your questions with answers, till I reached the final question about the interrupt priority. When searching through our code, I realized that we did not specificy the 'int_prio' field (it was added in the patch we received). It was undefined and pointed out to '112' (or some other high value). Once I changed it to irq priority 5, the issue is gone!! Thanks!

Unfortunately I still see that data gets corrupted some times. I will dive into this first to see if I can get a reproducable scenario. For now, it reproduces more often in case I reduce the timeout.

RE: Using LIBUARTE during flash operations

ovrebekk — Fri, 21 Jun 2019 08:12:03 GMT

Hi Roy

When you say 0x16e5a or close, do you mean this changes from time to time, or you're not sure the exact address?

The exact address can be important when reporting this to the stack developers.

To summarize, you confirm that this can happen with or without flash erase running?

You confirm that this will only happen when using the libuarte_async library?

How do you process UART events in the application?
Are you running a lot of event processing in the events directly, or do you process it later in main/thread context?

Do you know what interrupt priority the UART events are returned in?

Best regards
Torbjørn

RE: Using LIBUARTE during flash operations

RoyCreemers — Thu, 20 Jun 2019 14:38:36 GMT

The asserts seem to be exclusively from the softdevice. Return values of advertising start & stop are always ok (NRF_SUCCESS). Don't know if there is much relevant information in the fault itself.

'id' is always '1' (NRF_FAULT_ID_SD_ASSERT)

pc = 0x16e5a (or close)

info = 0;

)

RE: Using LIBUARTE during flash operations

ovrebekk — Thu, 20 Jun 2019 14:06:26 GMT

Hi Roy

Are you able to log these asserts, or use the debugger to trace from where they occur in the Nordic device?

Do you know if the sd_ble_gap_adv_start or sd_ble_gap_adv_stop functions return an error, or if the assert are exclusively generated by the SoftDevice?

I agree receiving your whole code base is unlikely to be helpful, but it might be interesting if you can isolate and send the parts of the code running on the nRF52 side that controls the SoftDevice (including starting and stopping advertising).

Best regards
Torbjørn

RE: Using LIBUARTE during flash operations

RoyCreemers — Wed, 19 Jun 2019 08:21:47 GMT

Sharing the code is possible, but I'm not sure this will help a lot (of course I will share if you think is has benefits). It is part of quite a big C++ application, involving multiple MCUs.

Our "application core" has a button which triggers an event to toggle advertising. This is communicated by UART to the Nordic BLE MCU. In his turn, this sends back a response over UART, and toggles advertising (so in this case there is a high change that there is some UART activity during softdevice activity).

The start of the advertisement is done by:

sd_ble_gap_adv_start(advertisementHandle, APP_BLE_CONN_CFG_TAG);

the stop is done by:

sd_ble_gap_adv_stop(advertisementHandle);

The scenario above can easily be reproduced here, the assertion occurs at least one out of 10 times. I share your oppinion that they can also occur at other times (I also saw some softdevice assertions during our automated testing, not related to advertisement... just during normal BLE transmittions)

RE: Using LIBUARTE during flash operations

ovrebekk — Wed, 19 Jun 2019 08:03:22 GMT

Hi Roy

How often do you get these asserts, and do they appear to be connected to any particular event or state in your application?

You mention that they occur when you toggle between advertising and standby, but I get the impression they also occur at other times?

Are you able to share the code you use for toggling between advertising and standby?

Best regards
Torbjørn

RE: Using LIBUARTE during flash operations

RoyCreemers — Tue, 18 Jun 2019 13:49:42 GMT

After testing some more, it seems that there are still issues. The device logs:

<error> app: Fatal error
<warning> app: System reset

... and reboots.

I tried to build for debug, but then it looks like more issues are popping up (UART data loss/corruption). When I repeat this several time and ignore the UART error, it looks like an issue in the softdevice:

<error> app: SOFTDEVICE: ASSERTION FAILED

Some more info:

- I got none of there errors when using the nrfx_uart driver

- When increasing the timeout of the UART (e.g. 5000us to 15000us) the problem of the UART data corruption seems to be reduced/dissapeared

- The assertion of the softdevice is not related to flash operations. In case I toggle between advertising & standby, I also get the assertion.

RE: Using LIBUARTE during flash operations

ovrebekk — Tue, 18 Jun 2019 12:44:47 GMT

No problem, I am happy to hear you got it working 🙂

RE: Using LIBUARTE during flash operations

RoyCreemers — Tue, 18 Jun 2019 11:12:55 GMT

Thanks for the response. With this patch, the UART works as desired!

RE: Using LIBUARTE during flash operations

ovrebekk — Tue, 18 Jun 2019 07:12:04 GMT

Please find the patched files attached:
nrf_libuarte_async_patch.zip

Can you give them a try and see if it solves the issue?

Best regards
Torbjørn

RE: Using LIBUARTE during flash operations

ovrebekk — Fri, 14 Jun 2019 13:40:36 GMT

Thanks.

The developer has discovered a bug that is probably related to the issue you are seeing.

I will try to prepare a patch next week so you can check if the problem is the same.

Best regards
Torbjørn

RE: Using LIBUARTE during flash operations

RoyCreemers — Thu, 13 Jun 2019 06:07:44 GMT

Hereby the sdk_config.h file

devzone.nordicsemi.com/.../2318.sdk_5F00_config.h

RE: Using LIBUARTE during flash operations

ovrebekk — Wed, 12 Jun 2019 13:52:47 GMT

Are you able to share the sdk_config.h project file of your application so I can try to replicate your setup?

I have forwarded the issue to the developers also, but they need some more time to investigate. Hopefully I will hear back from them in a couple of days.

Best regards
Torbjørn

RE: Using LIBUARTE during flash operations

RoyCreemers — Tue, 11 Jun 2019 12:02:30 GMT

Response on behalf of my collegue: The baudrate is 115200. There is no risk that the UART Rx buffer could fill up. The connected MCU can only send one single message (which is less <400 bytes),then it waits for a response. Unfortunately there is no possibility that to use hardware flow control

RE: Using LIBUARTE during flash operations

ovrebekk — Tue, 11 Jun 2019 11:53:21 GMT

What baudrate is the UART1 running at?

How large is the UART utilization from the connected MCU? Do you think there is a risk that the UART RX buffer could fill up during those ~85 milliseconds?

Are you using hardware flow control or not?

Best regards
Torbjørn