nrf_log fixes in SDK14.1.0

RE: nrf_log fixes in SDK14.1.0

Mark Leavitt — Sat, 05 May 2018 23:57:59 GMT

Austin, you are a hero and genius!

My SDK 14.2 app was crashing after 5-15 minutes, first symptom being log messages that were garbled (they looked like recycled earlier messages -- corrupted pointer?), then a hard lockup soon after. After replacing the standard nrf_log_frontend.c with your code 7522 above, my app has been running for hours now without a glitch.

In SDK 15.0 notes, it looks like the logger *backend* has new features, but there's no mention of bug fixes to the *frontend*. So I'm sticking to 14.2 + AustinFix until I hear otherwise. Thank you!

RE: nrf_log fixes in SDK14.1.0

Keton — Wed, 07 Mar 2018 16:32:22 GMT

if I remember correctly I've placed link to this topic in comment section under the list, but maybe this was for SDK13.x. Feel free to add it if needed.

RE: nrf_log fixes in SDK14.1.0

John C — Wed, 07 Mar 2018 15:54:48 GMT

If I read that list correctly it looks like all log-related bugs are marked as Fixed in SDK 14.1.

This bug, or similar, has evidently persisted into 14.2 - Do we need to make a new entry in the list?

RE: nrf_log fixes in SDK14.1.0

John C — Wed, 07 Mar 2018 15:40:36 GMT

Thanks for the reference.

RE: nrf_log fixes in SDK14.1.0

Keton — Wed, 07 Mar 2018 12:14:56 GMT

there are lists such as this one: https://devzone.nordicsemi.com/f/nordic-q-a/29796/what-are-sdk-14-x-0-known-issues for every major SDK version

RE: nrf_log fixes in SDK14.1.0

John C — Wed, 07 Mar 2018 11:42:36 GMT

@Nordic: Are bugs like this being tracked against SDK releases somewhere that we can see?

It would be reassuring to see an triaged list of known bugs and each releases' ongoing support status. This would avoid peeps spending time re-discovering known bugs and enable them to judge whether to patch, wait for a fix or just avoid. This particular bug was not a great introduction to NRF52 as the symptoms varied with the text I was logging! This module is clearly labelled as experimental, so having bugs is no shame at all, but I would still give the fix a reasonably high priority because I think it quite likely to bite and discourage other peeps evaluating your products.

RE: nrf_log fixes in SDK14.1.0

Keton — Mon, 22 Jan 2018 13:45:21 GMT

@Krzsztof after a period of testing I can confirm that problem is resolved. Thanks for providing the patches!

RE: nrf_log fixes in SDK14.1.0

Austin — Mon, 08 Jan 2018 05:15:57 GMT

There are number of issues with logging, even with SDK 14.2.0. The change in Krzysztof's answer goes a long way to fixing issues related to buffer overflow in normal operating conditions. This change fixes issues when using NRF_LOG_PUSH which could incorrectly return a pointer to non-contiguous memory and result in memory overwrites outside of the log buffer.

I've attempted to address some additional issues with the nrf log front end including the following:

Add consistent handling of log buffer allocations by always setting the allocated log buffer type as INAVLID.
Ensuring that buffer full can be distinguished from buffer empty condition.
Better handling of errors when there is no log buffer space remaining.
Added notes which indicate additional design issues with the logging system which could be addressed in future.

See the attached nrf_log_frontend.c. This file is based on SDK 14.2.0 and addresses some of the issues listed above. All '// NOTE:' comment lines indcate changes from SDK 14.2.0 with reasons for the change.

Updated 22 Feb 2018: Fixed additional bug in nrf_log_frontend.c related to buffering of hex data and rewrote buffering used by nrf_log_push. See new attachment

devzone.nordicsemi.com/.../7522.nrf_5F00_log_5F00_frontend.c

RE: nrf_log fixes in SDK14.1.0

Austin — Fri, 05 Jan 2018 11:26:34 GMT

@Krzsztof Thanks for the update, I was developing my own fix for this today but will take your change instead. There are quite a number of other issues with data races and edge conditions in this module. I'm hoping to propose some changes early next week as another answer to this question.

RE: nrf_log fixes in SDK14.1.0

Krzysztof Chruscinski — Fri, 05 Jan 2018 10:45:54 GMT

yes, it is 1 word.

RE: nrf_log fixes in SDK14.1.0

Keton — Fri, 05 Jan 2018 10:42:48 GMT

SDK 13.1 nrf_log implementation has no definition of PUSHED_HEADER_SIZE. Can I assume that it's equal to 1 word as per comment on line 438 of nrf_log_internal.h in SDK14.2?

RE: nrf_log fixes in SDK14.1.0

Keton — Thu, 04 Jan 2018 21:54:27 GMT

could you please provide either whole body of cont_buf_prealloc() or git patch? I'm not sure what to do with the rest of the critical section.

RE: nrf_log fixes in SDK14.1.0

Keton — Wed, 20 Dec 2017 10:31:20 GMT

Since Nordic provided the 'fixed' snippet at the top of this post I was hoping that someone who knows how this is supposed to work may come up with correct available_words estimate equation. Original doesn't handle overflows correctly, your proposed fix doesn't handle no overflow situations. Please at least advise is there situation where they both fail? I'm not that familiar with nrf_log internals to be able to judge that.

As for migrating to more recent SDK/backporting parts of SDK14 this is exactly what I'm trying to avoid. The product I'm currently developing is near release so we cannot afford any significant changes.

BTW it may be a good idea to consider having long term support version of the SDK with focus on no bugs and stability every once in a while. IMHO some developers (including me) would really appreciate codebase that maybe is not feature complete but is thoroughly tested both by Nordic and by the community and it's known to contain (almost) no bugs.

RE: nrf_log fixes in SDK14.1.0

Krzysztof Chruscinski — Wed, 20 Dec 2017 08:12:02 GMT

Hi,

we have made significant changes in nrf_log in SDK 14.0 and then bunch of stability fixes in SDK 14.1 It is hard to provide the patch for them. You can try to port latest nrf_log to SDK 13. Here is the migration guide which might be helpful:

RE: nrf_log fixes in SDK14.1.0

Keton — Tue, 19 Dec 2017 15:59:19 GMT

It seems that fix posted above is only valid when NRF_LOG_DEFERRED_BUFSIZE is smaller than logged data amount. For large NRF_LOG_DEFERRED_BUFSIZE patch above leads to a hardfault while original buffer size estimate works.

I know that Nordic devs are busy preparing SDK15 release but for all users like me who are stuck with SDK13 I'd really appreciate that someone who knows internals of nrf_log module takes a look if this approach is correct.

diff --git a/components/libraries/log/src/nrf_log_frontend.c b/components/libraries/log/src/nrf_log_frontend.c
index e8f80c10..e6f5dfd7 100644
--- a/components/libraries/log/src/nrf_log_frontend.c
+++ b/components/libraries/log/src/nrf_log_frontend.c
@@ -379,8 +379,25 @@ static inline uint32_t * cont_buf_prealloc(uint32_t len32,

     CRITICAL_REGION_ENTER();
     *p_wr_idx = m_log_data.wr_idx;
+
     uint32_t available_words = (m_log_data.mask + 1) -
                                (m_log_data.wr_idx & m_log_data.mask);
+
+       /*
+               This is more robust attempt of fixing NRF_LOG_DEFFERED=1 hardfault.
+               Nordic posted fix here: devzone.nordicsemi.com/.../
+               The change works only when NRF_LOG_DEFERRED_BUFSIZE isn't large enough to fit all log data before log processing happens.
+               When NRF_LOG_DEFERRED_BUFSIZE is large i.e. 4096 words proposed fix leads to hardfault by overflowing subtraction result
+
+               This is an attempt of more robust fix which assumes that smaller free space estimate is correct.
+       */
+
+       //this is how Nordic proposes to fix it
+       uint32_t available_words_alt=(m_log_data.mask + 1) - (m_log_data.wr_idx - m_log_data.rd_idx);
+
+       //assume that smaller value is correct
+       if(available_words_alt<available_words) available_words=available_words_alt;
+
     if (len32 <= available_words)
     {
         // buffer will fit as is
--

RE: nrf_log fixes in SDK14.1.0

Keton — Tue, 19 Dec 2017 10:56:52 GMT

Any updates on this? I have a project where I have to use deffered logging due to timing constraints and NRF_LOG hardfaults every time.

RE: nrf_log fixes in SDK14.1.0

Keton — Thu, 19 Oct 2017 14:58:50 GMT

thanks, it solved the problem for test case I've linked but still I'm able to crash MCU by doing bunch of: NRF_LOG_RAW_DEBUG("%s\r\n",nrf_log_push(str)); where str is: char str[150]. When I'm using SEGGER_RTT_Write instead of nrf_log it doesn't crash so it's most likely another memory access issue in nrf_log.

RE: nrf_log fixes in SDK14.1.0

Krzysztof Chruscinski — Thu, 19 Oct 2017 11:11:52 GMT

Hi,

you can try to find function cont_buf_prealloc in nrf_log_frontend.c and update following code:

uint32_t available_words = (m_log_data.mask + 1) -
                           (m_log_data.wr_idx & m_log_data.mask);

should be changed to:

uint32_t available_words = (m_log_data.mask + 1) -
                            (m_log_data.wr_idx - m_log_data.rd_idx);

That should help.

UPDATE Actually, the fix provided was not good and there was still a bug in SDK 14.2.0 in that function. Here is the updated version the function:

static inline uint32_t * cont_buf_prealloc(uint32_t len32,
                                       uint32_t * p_offset,
                                       uint32_t * p_wr_idx){
uint32_t * p_buf = NULL;

len32 += PUSHED_HEADER_SIZE; // Increment because 32bit header is needed to be stored.

CRITICAL_REGION_ENTER();
*p_wr_idx = m_log_data.wr_idx;
uint32_t available_words = (m_log_data.mask + 1) -
                            (m_log_data.wr_idx - m_log_data.rd_idx);
uint32_t cont_words =  (m_log_data.mask + 1) - (m_log_data.wr_idx & m_log_data.mask);

//available space is continuous
uint32_t curr_pos_available = (available_words <= cont_words) ? available_words : cont_words;
uint32_t start_pos_available = (available_words <= cont_words) ? 0 : (available_words - cont_words);
if (len32 <= curr_pos_available)
{
    // buffer will fit as is
    p_buf              = &m_log_data.buffer[(m_log_data.wr_idx + 1) & m_log_data.mask];
    m_log_data.wr_idx += len32;
    *p_offset          = 0;
}
else if (len32 <= start_pos_available)
{
    // wraping to the begining of the buffer
    m_log_data.wr_idx += (len32 + cont_words);
    *p_offset          = cont_words;
    p_buf              = m_log_data.buffer;
}

CRITICAL_REGION_EXIT();

return p_buf;}