nRF9160 Date Time Desync Issue

RE: nRF9160 Date Time Desync Issue

Didrik Rokhaug — Tue, 07 Feb 2023 13:11:31 GMT

Hi JLCIOT, and sorry for the late reply.

Unfortunately, replies to very old cases have a tendency to be missed.

The PR with the fix (which I linked to in my previous reply) was merged August 6th, 2021, and has been part of all NCS versions starting from v1.7.0.

Best regards,

Didrik

RE: nRF9160 Date Time Desync Issue

JLCIOT — Fri, 03 Feb 2023 02:08:47 GMT

We are having this same issue on our boards. Would someone provide an update on if the recommended fix solved this problem!!!!!

RE: nRF9160 Date Time Desync Issue

Didrik Rokhaug — Fri, 06 Aug 2021 09:02:45 GMT

Hi anicare-tero,

No, I don't think there should be any negative consequences of changing the type from int to tint64_t.

We have opened a PR with this update: https://github.com/nrfconnect/sdk-nrf/pull/5245

As Simen explains in the commit, k_uptime_get() returns an int64_t, so storing the value in an int will cause an overflow after approximately 25 days, as you are seeing.

RE: nRF9160 Date Time Desync Issue

anicare-tero — Fri, 06 Aug 2021 08:32:59 GMT

Hi,

I just found out yesterday after reading this discussion that I have this same problem.

nRF Connect SDK version: 1.6.0
Modem firmware version: 1.2.3

Our device search GPS once a week and after a GPS fix it sets the GPS datetime to Date-Time with date_time_set(). Then I'm calculating the delay for the next week's target time and getting the current UTC time with date_time_now().

I have also set CONFIG_DATE_TIME_UPDATE_INTERVAL_SECONDS=0 so the library won't update the Date-Time without commanding it to do it.

Here you can see from the logs that after 25 days it's not responding correctly anymore:

Jul 06 14:41:24 [00:02:06.592,620] <inf> application: Current UTC time from GPS has been set: 06.07.2021 11:41:24	// date_time_set()
Jul 06 14:41:24 [00:02:06.987,121] <inf> application: uptime_days: 0
Jul 06 14:41:27 [00:02:09.473,358] <inf> application: Current UTC time: 06.07.2021 11:41:26							// date_time_now()
Jul 06 14:41:27 [00:02:09.483,032] <inf> application: Calculated target UTC time: 11.07.2021 04:48:26

Jul 11 07:50:01 [03:57:16.162,139] <inf> application: Current UTC time from GPS has been set: 11.07.2021 04:50:00	// date_time_set()
Jul 11 07:50:01 [03:57:16.545,410] <inf> application: uptime_days: 4
Jul 11 07:50:03 [03:57:18.905,487] <inf> application: Current UTC time: 11.07.2021 04:50:02							// date_time_now()
Jul 11 07:50:03 [03:57:18.915,191] <inf> application: Calculated target UTC time: 18.07.2021 04:48:02

Jul 18 08:12:02 [26:41:23.327,484] <inf> application: Current UTC time from GPS has been set: 18.07.2021 05:12:02	// date_time_set()
Jul 18 08:12:03 [26:41:23.711,059] <inf> application: uptime_days: 11
Jul 18 08:12:05 [26:41:26.098,937] <inf> application: Current UTC time: 18.07.2021 05:12:04							// date_time_now()
Jul 18 08:12:05 [26:41:26.108,581] <inf> application: Calculated target UTC time: 25.07.2021 04:48:04

Jul 25 07:49:30 [12:16:26.470,001] <inf> application: Current UTC time from GPS has been set: 25.07.2021 04:49:30	// date_time_set()
Jul 25 07:49:31 [12:16:26.853,454] <inf> application: uptime_days: 18
Jul 25 07:49:33 [12:16:29.483,276] <inf> application: Current UTC time: 25.07.2021 04:49:33							// date_time_now()
Jul 25 07:49:33 [12:16:29.492,980] <inf> application: Calculated target UTC time: 01.08.2021 04:48:33

Aug 01 07:50:00 [34:39:01.667,724] <inf> application: Current UTC time from GPS has been set: 01.08.2021 04:50:00	// date_time_set()
Aug 01 07:50:01 [34:39:02.051,330] <inf> application: uptime_days: 25
######################################################################################################################################
Aug 01 07:50:03 [34:39:04.418,701] <inf> application: Current UTC time: 19.09.2021 21:52:50							// date_time_now()
Aug 01 07:50:03 [34:39:04.428,405] <inf> application: Calculated target UTC time: 26.09.2021 04:48:50
######################################################################################################################################

It's sad that testing this issue lasts for so long. Didrik Rokhaug do you think there could be any disadvantages or problems when changing that last_date_time_update from int to int64_t? I think we have to do a next firmware release in this month so we have not enough time to test this fix completely before that.

RE: nRF9160 Date Time Desync Issue

Didrik Rokhaug — Thu, 05 Aug 2021 15:02:40 GMT

Hi,

I've had a talk with the developer of the date_time library, and he suggested that it might be the last_date_time_update that overflows.

Could you try to change the type from int to int64_t for the last_date_time_update field in the time_aux struct (line 70 in date_time.c)?

RE: nRF9160 Date Time Desync Issue

Sebastian Stein — Tue, 03 Aug 2021 09:48:39 GMT

Hi,

sorry for jumping into the discussion. However, I am experiencing the same behavior with a nrf9160 DK, which I run in a continuous duty test.

Meta information:

nrf9160 DK version: 0.9.0 | 2019.20
nrf Connect SDK version: 1.5.1
Modem firmware version: 1.2.3

Date time library related prj.conf settings:

CONFIG_DATE_TIME=y
CONFIG_DATE_TIME_MODEM=y
CONFIG_DATE_TIME_UPDATE_INTERVAL_SECONDS=3600

Issue description:

At the month boundary between July and August, the date time jumped to late September and is not corrected despite of the update period of 3600 s. This issue occurred after 36 days (5 hour precision) after the continuous duty test started. The time offset is around 4294959 s.

Every 10 minutes, the DK acquires sensor data and sends them into a cloud service. The cloud service received the last valid data set at 2021-07-30 05:31:26 UTC. The date time has an offset since then. In the following you find two examples of raw data sets, which the cloud service has received since then.

// Received at 2021-08-02 15:51:47 UTC
// The unix time stamps translate to 2021-09-21 08:54:26 UTC
{

   "ver":"0.01",

   "snd":[

      "1632214466,LmLrs21,t0,19.100,0",

      "1632214466,LmLrs21,t1,55.900,0",

      "1632214466,LmLrs21,t2,993.600,0"

   ]

}

// Received at 2021-08-02 16:01:47 UTC
// The unix time stamps translate to 2021-09-21 09:04:26 UTC
{

   "ver":"0.01",

   "snd":[

      "1632215066,LmLrs21,t0,19.300,0",

      "1632215066,LmLrs21,t1,55.500,0",

      "1632215066,LmLrs21,t2,993.600,0"

   ]

}

Furthermore, I gather logging information and send them to a server of mine. I noticed two things in the log messages. Firstly, the time stamps of the log messages exhibit the same date time offset. Secondly, the modem seemed to had some issues with querying the time from a ntp server. However, this querying issue does not overlap with the first occurrence of the date time offset. Below you find a slightly censored log file. The time is given in UTC. Please note the jump in the time stamps between line 4 and 5 and please note the date time log messages beginning at line 12.

[2021-07-29 23:21:26] <dbg> lmSdi12.sdi12IssueCommandSequence: Sensor 4 - seq. retries: 1
[2021-07-30 02:01:26] <dbg> lmSdi12.sdi12IssueCommandSequence: Sensor 4 - seq. retries: 1
[2021-07-30 03:31:26] <dbg> lmSdi12.sdi12IssueCommandSequence: Sensor 4 - seq. retries: 1
[2021-07-30 04:01:26] <dbg> lmSdi12.sdi12IssueCommandSequence: Sensor 4 - seq. retries: 1
[2021-09-18 18:24:34] <wrn> cloudService: Could not send data, data buffered
[2021-09-18 18:24:34] <wrn> cloudUpdateTask: Sending failed
[2021-09-18 18:34:34] <inf> cloudService: Buffered data was send
[2021-09-19 13:04:26] <dbg> lmSdi12.sdi12IssueCommandSequence: Sensor 4 - seq. retries: 1
[2021-09-19 18:34:33] <wrn> cloudService: Could not send data, data buffered
[2021-09-19 18:34:33] <wrn> cloudUpdateTask: Sending failed
[2021-09-19 18:44:34] <inf> cloudService: Buffered data was send
[2021-09-20 00:43:30] <wrn> date_time: sntp_query, error: -116
[2021-09-20 00:44:00] <wrn> date_time: getaddrinfo, error: -11
[2021-09-20 00:44:30] <wrn> date_time: getaddrinfo, error: -11
[2021-09-20 00:44:30] <wrn> date_time: getaddrinfo, error: -11
[2021-09-20 00:44:30] <wrn> date_time: getaddrinfo, error: -11
[2021-09-20 00:44:30] <wrn> date_time: Not getting time from any NTP server
[2021-09-20 00:44:30] <err> at_cmd: Failed to send AT command (err:9)
[2021-09-20 00:44:33] <wrn> cloudService: Could not send data, data buffered
[2021-09-20 00:44:33] <wrn> cloudUpdateTask: Sending failed
[2021-09-20 00:54:34] <inf> cloudService: Buffered data was send
[2021-09-20 05:43:30] <wrn> date_time: sntp_query, error: -116
[2021-09-21 20:44:26] <dbg> lmSdi12.sdi12IssueCommandSequence: Sensor 4 - seq. retries: 1

RE: nRF9160 Date Time Desync Issue

Jstone40649 — Fri, 23 Jul 2021 12:31:35 GMT

Unfortunately no progress yet, since it takes so long to reproduce were finding it difficult to debug.

1. It's about 8 weeks, so for example on July 5th (7-5) the time reported as august 24th (8-24)

2. Yes, it appears to be consistent between devices.

3. It's my understand that the date_time library can pull from several sources, I think we have modem time and NTP enabled. We have not been tracking the source of each individual update, I'll add that to our testing considerations going forward.
4. The error seems to appear 30 days after initial power-up, but there's no other specific boundary.

5. Before uploading we format the date-time to a JSON string, the cloud stores that string directly along side the date-time as the server sees it.

6. We're working on something like this, but we don't have long-term serial access generally so we're working to get that info pushed up through the cloud.

7. We don't have a huge sample size for this, but it seems to trigger around 30 days after powerup.

Thanks for your help looking into this, I'll keep looking into it on our end and I'll try to add in those extra considerations you mentioned.

RE: nRF9160 Date Time Desync Issue

Didrik Rokhaug — Fri, 23 Jul 2021 09:10:59 GMT

Hi, and sorry for the late reply.

have you had any progress on this issue?

I would like to have the developer behind the date_time library look at this ticket, but unfortunately, he is on vacation for another week.

In the meantime, I have some more questions:

1. How big is the time offset? What time values do you get?

2. Is the time offset the same on all the failing devices?

3. What source do you get the time from? Does the source change when the error appears?

4. When did the error appear? E.g. was it at a month boundary?

5. How are the timestamps handled on the cloud side?

6. Could you add printks/log statements in the date_time library that prints the time received from the source?

7. Does the error appear at the same time, or after the same runtime in different devices?

Best regards,

Didrik

RE: nRF9160 Date Time Desync Issue

Jstone40649 — Wed, 14 Jul 2021 12:33:17 GMT

Config wise we have: the date time interval at 3600, Date time modem and ntp are both set, the stack size for the date time thread is 1024, and the ntp query time is 5 seconds.

The snippet covers most of the code related to getting date-times, and works as expected in testing.

static void date_time_event_handler(const struct date_time_evt *evt)
{
	switch (evt->type) {
	case DATE_TIME_OBTAINED_MODEM:
		printk("DATE_TIME_OBTAINED_MODEM\n");
		break;
	case DATE_TIME_OBTAINED_NTP:
		printk("DATE_TIME_OBTAINED_NTP\n");
		break;
	case DATE_TIME_OBTAINED_EXT:
		printk("DATE_TIME_OBTAINED_EXT\n");
		break;
	case DATE_TIME_NOT_OBTAINED:
		printk("DATE_TIME_NOT_OBTAINED\n");
		break;
	default:
		break;
	}

	/** Do not depend on obtained time, continue upon any event from the
	 *  date time library.
	 */
	k_sem_give(&date_time_obtained);
}

int dateTimeErrs = 0;

static void date_time_update(void)
{
    int err;

    // Start datetime update
    date_time_update_async(date_time_event_handler);

    // wait, DATE_TIME_TIMEOUT_S seconds for the datetime update to return
    err = k_sem_take(&date_time_obtained, K_SECONDS(DATE_TIME_TIMEOUT_S));
    if (err) {
            printk("Date time, no callback event within %d seconds\n",
                    DATE_TIME_TIMEOUT_S);
    }
    else
    {
        int64_t unixTime;   
        err = date_time_now(&unixTime);
        if (err)
        {
			dateTimeErrs++;
        }
        else
        {
			// We're deviding by 1000 to get unix time in seconds instead of milliseconds.
			// TODO Consider finding an alternative method.
			ClockService_SetTimeUnix(unixTime/1000);
        }
		LOG_ERR("DateTimeErrors: %d", dateTimeErrs);
    }
}

And then a little down the line we have this code, to convert from the unix timestamp to our structure

bool DateTime_FromUnix(DateTime* value, int64_t unixTime)
{
    time_t epoch = unixTime;
    struct tm timestamp = *localtime(&epoch);  
    return DateTime_Initialize(value, 1900 + timestamp.tm_year, 1 + timestamp.tm_mon, timestamp.tm_mday, timestamp.tm_hour, timestamp.tm_min, timestamp.tm_sec, 0);
}

I don't think we have any place where the internal variable could be comprimised. Memory leaks or problems like it could make sense, but we've got multiple devices reporting the same behaviour, and the datetime, while now a month off, is still consistent with the passing of time, but with an offset. I'd expect that memory issues would be lead to more random outcomes.

RE: nRF9160 Date Time Desync Issue

Didrik Rokhaug — Tue, 13 Jul 2021 14:52:52 GMT

Hi,

What time sources are you getting your time from?

What date_time related Kconfig options have you set, and how do you use the library in your code?

[quote user=""]Do you have any insight as to what could cause our clock to skip forward like that, or why the date_time_update_async calls would not correct the error?[/quote]

In general, memory leaks, or other memory related errors could cause all kind of weird behavior. Could it be possible that there is something else than the date_time library that writes to the date_time library's internal variable?

Best regards,

Didrik