nRF9160 data_time library behavior

[ClemensG]

Hi,

in my current project I need to have a wall clock time, thus I am using the date_time library. While trying to understand it I came across a few issues/questions.

All in here: ncs 2.1.2

The reason is to compensate for crystal clock drift (esp the clock_gettime based on this) of long running system each time I connect to the network to send my data maybe once a day. The modem is kept in psm the other time.
Ideal would be: update the time from the cellular network time if available and if not use an ntp server. All of this automatically as soon as I connect (get out of psm).

By reading the docs my first try was:

CONFIG_DATE_TIME=y
CONFIG_DATE_TIME_MODEM=y
CONFIG_DATE_TIME_NTP=y
CONFIG_DATE_TIME_AUTO_UPDATE=y
CONFIG_DATE_TIME_TOO_OLD_SECONDS= some value > 0
CONFIG_DATE_TIME_UPDATE_INTERVAL_SECONDS = 0  (to disable automatic time polling, only when I come out of psm)

First issue here is the ASSERT  in date_time_core.c line 19 does not allow me to set CONFIG_DATE_TIME_TOO_OLD_SECONDS to
a value bigger than CONFIG_DATE_TIME_UPDATE_INTERVAL_SECONDS. But according to the documentation this should be possible
(Kconfig help for CONFIG_DATE_TIME_UPDATE_INTERVAL_SECONDS: "Setting this option to 0 disables sequential date time updates."
And by looking at the source code I do no see a reason why this assert should not pass for CONFIG_DATE_TIME_UPDATE_INTERVAL_SECONDS=0
(I might have overseen something).

Second issue: On a regular basis the lib only updates automatically on %XTIME notifications from the network. I guess I do have no influence on
when this happens. I see it when I register to the network but not when coming out of psm (at least not every time at short intervals). The lib also
forces an update (besides XTIME) when the modem registers to the network (and no XTIME was received before) but this should ideally only
happen once at startup.
I had somehow expected it to query NTP when no network time information (XTIME) is received when coming out of psm in automatic mode.
The documentation states "Automatic update when connecting to LTE network if the option CONFIG_DATE_TIME_AUTO_UPDATE is set". Maybe
this should be clarified to "...when registering to the network..." (so others don't have the misunderstanding as I)

Third issue (not really affecting me but maybe others): as soon as the first %XTIME notification was received once the lib will always trusts the modem time
if the modem provides a time by +CCLK (noticed by reading the source code). Isn't that an issue when changing the network (asset tracking applications) to
a network that does not provide a time?
Or does the modem invalidates it's time when it hasn't received a time from the network for a longer period? There must be a clock drift, too...
Somehow I would expect the lib to fallback to NTP then (when configured) but it doesn't (when the modem still provides a time).

In the end I am a little bit puzzled on how to achieve a robust and deterministic behavior with the library. After writing these questions and reading over
the documentation again the documentation seems to be mostly correct but still not very intuitive, see my misunderstandings above.

For me I will most likely do my updates manually by date_time update_async().

Whishlist for future versions:
- Query NTP if last XTIME event is older than CONFIG_XYZ
- Also have this functionality callable (like update_async now). So you can be sure to have a "fresh" time after the call and not a silent fallback to the modem's time.

Two question for the end: How accurate is the modem clock? A comment in the lib states "... as modem clock is more accurate than
application side clock". I somehow assumed it is also only based on the LFXO which is also the base for the zephyr timekeeping when the modem is enabled.
How often are time updates send typically from the network if they are provided?

Thanks for following me up to here.

Regads

Clemens

   

[Michal]

Hello Clemens,

Thank for the comprehensive information.

I am looking into it, but it will take me some time to research the whole topic.

I will return back to you when I'll have more information.

Best regards,

Michal

[ClemensG]

Hi Michal,

Thanks for looking into it. I guess I found (a not very nice way) of how I can life with the current
implementation, I will still give some thoughts as I was just dealing with it:

Application user view: I guess there are two major types of usages for having the wall clock time in an IoT system.
a) The ones who do event (external, or random) driven sampling and want to timestamp these events
   -> wakeup at an event, do something and timestamp it with the current wall clock time
b) The ones who want to wakeup and do things at times aligned to wall clock time
   -> Wakeup at ("precicely") 11:15:00 do something and timestamp this with 11:15:00, next wakeup at 12:30:00
   
For the a) case it guess it's ok if the wall clock suddenly jumps while getting updated in the background as
the current implementation it meant to be used. The timestamp will be just the one the date_time_now() returns.
   
The b) base is a little bit trickier: Here a jumping wall clock might be ugly. The reason is, that I can only
do my timing (wakeup) by using zephyrs timers based on kernel ticks an not on the wall clock.
Thus an implementation (with current possibilities would be):
1) Update wall clock
2) Take current wall clock time
3) calc time diff to desired wakeup/sampling point
4) k_sleep()/k_timer_start() with the time diff
5) sample and timestamp them with the current wall clock

Within all of the above it is clear, that the wall clock time aquired by the lib is not accurate
compared to real UTC/TAI as the update procedure has delays which are not accounted for. But let's
assume we will take this time (best we have) as the one we thin it'S right.

If in the above sample an automatic time update happens between 4) and 5) things look ugly. Due
to crystal clock drift and variable delays while updating the time, my timestamp in 5) suddenly
is 11:15:02 instead of the desired 11:15:00 which at least looks ugly in the result.

Two things i'd love to see (whishlist):
- for step 1) (update time) an api call with date_time_update(timeout_s) would be nice.
  Blocking until either updated or timed out. As calculating the sampling scheduling in
  2) and 3) must be done after updating the wall clock. Okay, it's possible right now but
  I need to implement an event handler just to tell me when it's done and at the same time.
  Of cause all of this taking into account my previous post.

- A huge current limitation currently is the fact that zephyr does not allow absolute
  timeouts/delays/timers (timeout at tick count x) but only relative. Also these timeouts
  never base on the CLOCK_REALTIME and there is no atmonic way of getting the diff of
  CLOCK_REALTMIME and uptime ticks (as far as I know).
  Thus all calculations above do have non compensated runtime delays (when does a k_sleep()
  relative time starts?) or can be way of when getting interrupted by a higher priority thread
  (think about a long interruption at step 3)).
  As far as I know this is not solvable right now in the lib (needs zephyr support) but
  at least a note in the documentation might help.
  Also date_time_uptime_to_unix_time_ms() might give really wrong results if preempted inbetween
  k_uptime_get() and date_time_now(). Either put it into a critical section or ask the caller to
  do so (documentation).

BTW: This feels wrong to use the Q&A forum for that. Any better places to give such kind of feedback?
 
Regards,

Clemens

[Michal]

Hello Clemens,

Thank you again for the information. This is the correct place for feedback, so don't worry.

I am not 100% sure about any Zephyr-specific requests/feedback, that should probably go directly to the Zephyr Project. They have a contact form with a mailing list and discord server and there are also the github issues.

I am still looking into this, so I have to ask for patience.

Best regards,

Michal