Mailing List Archive

On Wed, Apr 4, 2012 at 10:33 AM, Steven D'Aprano <steve@pearwood.info> wrote:
> Python has worked pretty well without high res and monotonic clocks for 20
> years. Let's not rush into a suboptimal design based on who can outlast
> everyone else in this discussion.

+1

FWIW, I'd be fine with underscore prefixes on *any* additions to the
relevant module APIs for 3.3.

Cheers,
Nick.

--
Nick Coghlan   |   ncoghlan@gmail.com   |   Brisbane, Australia
_______________________________________________
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

Le 04/04/2012 02:33, Steven D'Aprano a écrit :
> Judging by the hundreds of emails regarding PEP 418, the disagreements
> about APIs, namings, and even what characteristics clocks should have, I
> believe that the suggestion is too divisive (and confusing!) to be
> accepted or rejected at this time.

Oh, I just "rewrote" the PEP before reading your email. Sorry for the
noise with this PEP :-) I just read again all emails related to this PEP
to complete the PEP. The PEP should now list all proposed API designs. I
hope that I did not forget anything.

I failed to propose a consistent and clear API because I (and Guido!)
wanted to fallback to the system clock. Falling back to the system clock
is a problem when you have to define the function in the documentation
or if you don't want to use the system clock (but do something else) if
no monotonic clock is available.

So I rewrote the PEP to simplify it:

* Don't fallback to system clock: time.monotonic() is always monotonic
(cannot go backward), but it is not always available. You have to write
a classic try/except ImportError which has a nice advantage: your
program will work on Python older than 3.3 ;-)

* Remove the time.perf_counter() function (it was called
time.highres() before). "highres" notion was confusing. I only wrote the
function to expose QueryPerformanceCounter (while it was already
accessible using os.clock()). The function was not well defined. Another
PEP should be written or at least the subject should be discussed after
the PEP 418 (monotonic clock).

* Rename time.steady() to time.monotonic(), again :-)

> Everyone has a different opinion, everyone believes their opinion holds
> for the majority, and it isn't practical for anyone to read the entire
> discussion.

I read most emails and I can say that:

* There is a need for a monotonic clock
* Most people prefer to handle explicitly the fallback if no monotonic
clock is available
* Most people don't want to call the new function "steady" because it
stands for something different

> I propose for Python 3.3:
>
> 1) the os module should expose lightweight wrappers around whatever
> clocks the operating system provides;

Python 3.3 has already time.clock_gettime() and time.clock_getres() with
CLOCK_REALTIME, CLOCK_MONOTONIC, CLOCK_MONOTONIC_RAW, CLOCK_HIGHRES.

mach_absolute_time() and GetTickCount/GetTick64 are not available yet.

> 3) we postpone PEP 418 until there is some real-world experience with
> using the os clocks from Python and we can develop a consensus of what
> is actually needed rather than what people think we need (i.e. probably
> in 3.4);

Many applications already implement their own "monotonic" clock". Some
libraries provide also such clock for Python. On UNIX, it's always using
clock_gettime(MONOTONIC). On Windows, it's sometimes GetTickCount,
sometimes QueryPerformanceCounter. On Mac OS X, it's always
mach_absolute_time(). I didn't find a library supporting Solaris.

> 4) if the standard library has need for a "use the best clock available,
> for some definition of best, and fall back to time() if not" clock, then
> the time module should do the simplest thing that could possible work,
> flagged as a private function:

In the last version of my PEP, time.monotonic() is simply defined as "a
monotonic clock (cannot go backward)". There is no more "... best ..."
in its definition.

Victor
_______________________________________________
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

On 2012-04-03, at 8:33 PM, Steven D'Aprano wrote:

> 1) the os module should expose lightweight wrappers around whatever clocks the operating system provides;

+1. That should make it flexible enough to those who really need it.

> 2) the time module should NOT provide any further clocks other than the existing time() and clock() functions (but see point 4 below);
>
> 3) we postpone PEP 418 until there is some real-world experience with using the os clocks from Python and we can develop a consensus of what is actually needed rather than what people think we need (i.e. probably in 3.4);
>
> 4) if the standard library has need for a "use the best clock available, for some definition of best, and fall back to time() if not" clock, then the time module should do the simplest thing that could possible work, flagged as a private function:

+1 on overall idea too.

-
Yury
_______________________________________________
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

Finally! We've come full circle.

+1 for monotonic as just described by Victor.

Lock it in before the paint dries.
On Apr 4, 2012 10:05 AM, "Matt Joiner" <anacrolix@gmail.com> wrote:

> Finally! We've come full circle.
>
> +1 for monotonic as just described by Victor.
>

On Wed, Apr 04, 2012 at 03:28:34AM +0200, Victor Stinner wrote:
> Le 04/04/2012 02:33, Steven D'Aprano a écrit :
> >Judging by the hundreds of emails regarding PEP 418, the disagreements
> >about APIs, namings, and even what characteristics clocks should have, I
> >believe that the suggestion is too divisive (and confusing!) to be
> >accepted or rejected at this time.
>
> Oh, I just "rewrote" the PEP before reading your email. Sorry for the
> noise with this PEP :-) I just read again all emails related to this PEP
> to complete the PEP. The PEP should now list all proposed API designs. I
> hope that I did not forget anything.

I think the PEP is a good, important PEP, and thank you for being the
PEP's champion. But in my opinion, this is too big to rush it and risk
locking in a sub-standard API for the next decade or two.


> >I propose for Python 3.3:
> >
> >1) the os module should expose lightweight wrappers around whatever
> >clocks the operating system provides;
>
> Python 3.3 has already time.clock_gettime() and time.clock_getres() with
> CLOCK_REALTIME, CLOCK_MONOTONIC, CLOCK_MONOTONIC_RAW, CLOCK_HIGHRES.

Why does it already have these things when the PEP is not accepted?

(This is not a rhetorical question, perhaps there is a good reason why
these have been added independently of the PEP.)

If I remember correctly, Guido earlier mentioned that he only wanted to
see one or two (I forget which) new clocks, and I see in 3.3.0a1 there
are already at least five new clocks:

monotonic or clock_gettime(CLOCK_MONOTONIC) # Are these the same thing?
wallclock
clock_gettime(CLOCK_PROCESS_CPUTIME_ID)
clock_gettime(CLOCK_REALTIME)
clock_gettime(CLOCK_THREAD_CPUTIME_ID)

plus the old ways, time.time and time.clock. (Neither of which seems
to have a clock-id.)


> mach_absolute_time() and GetTickCount/GetTick64 are not available yet.

That will make potentially 10 different clocks in the time module.


It may be that, eventually, Python should support all these ten
different clocks. (Personally, I doubt that the average Python
programmer cares about the difference between time() and clock(), let
alone another eight more.) But there's no rush. I think we should start
by supporting OS-specific clocks in the os module, and then once we have
some best-practice idioms, we can promote some of them to the time
module.


--
Steven
_______________________________________________
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

On Wed, 4 Apr 2012 18:09:40 +1000
Steven D'Aprano <steve@pearwood.info> wrote:
> > Python 3.3 has already time.clock_gettime() and time.clock_getres() with
> > CLOCK_REALTIME, CLOCK_MONOTONIC, CLOCK_MONOTONIC_RAW, CLOCK_HIGHRES.
>
> Why does it already have these things when the PEP is not accepted?
>
> (This is not a rhetorical question, perhaps there is a good reason why
> these have been added independently of the PEP.)

Because these are thin (low-level) wrappers around the corresponding
POSIX APIs, so there is no reason not to add them.

I know you were asking for such wrappers to be in the "os" module, but
my understanding is that time-related functions should preferably go
into the "time" module. "os" is already full of very diverse stuff, and
documentation-wise it is better if time-related functions end up in a
time-related module. Otherwise we'll end up having to cross-link
manually, which is always cumbersome (for us) and less practical (for
the reader).

Regards

Antoine.


_______________________________________________
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

>> Why does it already have these things when the PEP is not accepted?
>> ...
>> (This is not a rhetorical question, perhaps there is a good reason why
>> these have been added independently of the PEP.)

time.clock_gettime() & friends were added by the issue #10278. The
function was added before someone asked (me) to write a PEP. The need
of a PEP came later, when time.wallclock() and time.monotonic()
functions were added.

> Because these are thin (low-level) wrappers around the corresponding
> POSIX APIs, so there is no reason not to add them.

time.clock_gettime() can be used for other purpose than a monotonic
clock. For example, CLOCK_THREAD_CPUTIME_ID is the only available
function to get the "Thread-specific CPU-time clock". It also gives
access to CLOCK_MONOTONIC_RAW which is not used by the
time.monotonic() function proposed in the PEP.

Victor
_______________________________________________
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

> I failed to propose a consistent and clear API because I (and Guido!) wanted
> to fallback to the system clock. Falling back to the system clock is a
> problem when you have to define the function in the documentation or if you
> don't want to use the system clock (but do something else) if no monotonic
> clock is available.

Well, it was not only Guido and me.

Nick Coghlan wrote:
"However, I think Victor's right to point out that the *standard
library* needs to have a fallback to maintain backwards compatibility
if time.monotonic() isn't available, and it seems silly to implement
the same fallback logic in every module where we manipulate timeouts."
and
"Since duplicating that logic in every module that handles timeouts
would be silly, it makes sense to provide an obvious way to do it in
the time module."

Michael Foord wrote:
"It is this always-having-to-manually-fallback-depending-on-os that I
was hoping your new functionality would avoid. Is time.try_monotonic()
suitable for this usecase?"

The following functions / libraries fall back to the system clock if
no monotonic clock is available:
- QElapsedTimer class of the Qt library
- g_get_monotonic_time() of the glib library
- monotonic_clock library
- AbsoluteTime.now (third-party Ruby library),
"AbsoluteTime.monotonic?" tells if AbsoluteTime.now is monotonic

Extract of the glib doc: "Otherwise, we make a best effort that
probably involves returning the wall clock time (with at least
microsecond accuracy, subject to the limitations of the OS kernel)."

--

Only the python-monotonic-time fails with an OSError if no monotonic
clock is available.

System.nanoTime() of Java has few garantee: "Returns the current value
of the most precise available system timer, in nanoseconds. This
method can only be used to measure elapsed time and is not related to
any other notion of system or wall-clock time. The value returned
represents nanoseconds since some fixed but arbitrary time (perhaps in
the future, so values may be negative)." I don't even know if it is
monotonic, steady or has an high resolution.

Note: Boost.Chrono.high_resolution_clock falls back to the system
clock if no steady clock is available. (But the high-resolution clock
idea was deferred, it's something different than a monotonic or steady
clock.)

Victor
_______________________________________________
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

I am fine with the PEP as it is now (2012-04-04 15:34 GMT).

A question:

Since the only monotonic clock that can be adjusted by NTP is Linux'
CLOCK_MONOTONIC, if we avoid it, then time.monotonic() would always
give a clock that isn't adjusted by NTP. That would however mean we
wouldn't support monotonic clocks on systems that run a Linux that is
older than mid-2008. Is this generally seen as a problem?

//Lennart
_______________________________________________
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

On 2012-04-03, at 9:28 PM, Victor Stinner wrote:

> In the last version of my PEP, time.monotonic() is simply defined as "a monotonic clock (cannot go backward)". There is no more "... best ..." in its definition.

I like the last version of the PEP ;)

-
Yury
_______________________________________________
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

Lennart Regebro wrote:
> Since the only monotonic clock that can be adjusted by NTP is Linux'
> CLOCK_MONOTONIC, if we avoid it, then time.monotonic() would always
> give a clock that isn't adjusted by NTP.

I thought we decided that NTP adjustment isn't an issue, because
it's always gradual.

--
Greg

_______________________________________________
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

> I failed to propose a consistent and clear API because I (and Guido!) wanted
> to fallback to the system clock. Falling back to the system clock is a
> problem when you have to define the function in the documentation or if you
> don't want to use the system clock (but do something else) if no monotonic
> clock is available.

More details why it's hard to define such function and why I dropped
it from the PEP.

If someone wants to propose again such function ("monotonic or
fallback to system" clock), two issues should be solved:

- name of the function
- description of the function

At least, "monotonic" and "steady" names are not acceptable names for
such function, even if the function has an optional "strict=False" or
"fallback=True" parameter. By the way, someone complained that having
a boolean parameter requires to create a new function if you want to
call it without an argument (use a lambda function, functools.partial,
or anything else). Example:

get_time = lambda: try_monotonic(fallback=True)
t = get_time()

The description should give the least guarantees.

If the function doesn't promise anything (or only a few weak
properties), it is harder to decide which clock do you need for your
use case: time.clock(), time.time(), time.monotonic(), time.<name of
the monotonic-of-fallback function>, ...

Victor
_______________________________________________
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

On Apr 4, 2012 7:28 PM, "Victor Stinner" <victor.stinner@gmail.com> wrote:
>
> More details why it's hard to define such function and why I dropped
> it from the PEP.
>
> If someone wants to propose again such function ("monotonic or
> fallback to system" clock), two issues should be solved:
>
> - name of the function
> - description of the function

Maybe I missed it, but did anyone ever give a reason why the fallback
couldn't be to Steven D'Aprano's monotonic wrapper algorithm over the
system clock? (Given a suitable minimum delta.) That function appeared to
me to provide a sufficiently monotonic clock for timeout purposes, if
nothing else.

On 04Apr2012 22:23, PJ Eby <pje@telecommunity.com> wrote:
| On Apr 4, 2012 7:28 PM, "Victor Stinner" <victor.stinner@gmail.com> wrote:
| > More details why it's hard to define such function and why I dropped
| > it from the PEP.
| >
| > If someone wants to propose again such function ("monotonic or
| > fallback to system" clock), two issues should be solved:
| >
| > - name of the function
| > - description of the function
|
| Maybe I missed it, but did anyone ever give a reason why the fallback
| couldn't be to Steven D'Aprano's monotonic wrapper algorithm over the
| system clock? (Given a suitable minimum delta.) That function appeared to
| me to provide a sufficiently monotonic clock for timeout purposes, if
| nothing else.

It was pointed out (by Nick Coglan I think?) that if the system clock
stepped backwards then a timeout would be extended by at least that
long. For example, code that waited (by polling the synthetic clock)
for 1s could easily wait an hour if the system clock stepped back that
far. Probaby undesirable.

I think synthetic clocks are an extra task; they will all have side
effects of one kind of another.

A system monotonic clock, by contrast, may have access to some clock
hardware that doesn't step when the "main" system clock gets adjusted,
and can stay monotonic. A synthetic clock without such access can't
behave as nicely.

If synthetic clocks get handed out as fallback there should be some way
for the user to know, or a big glaring negative guarrentee in the docs
and on platforms without a system monotonic clock you might get a clock
with weird (but monotonic!) behaviours if you use the fallback.

Cheers,
--
Cameron Simpson <cs@zip.com.au> DoD#743
http://www.cskk.ezoshosting.com/cs/

Tachyon: A gluon that's not completely dry.
_______________________________________________
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

On Thu, Apr 5, 2012 at 1:41 PM, Cameron Simpson <cs@zip.com.au> wrote:
> It was pointed out (by Nick Coglan I think?) that if the system clock
> stepped backwards then a timeout would be extended by at least that
> long.

Guido pointed it out (it was in a reply to me, though).

Cheers,
Nick.

--
Nick Coghlan   |   ncoghlan@gmail.com   |   Brisbane, Australia
_______________________________________________
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

On Thu, Apr 5, 2012 at 00:45, Greg Ewing <greg.ewing@canterbury.ac.nz> wrote:
> Lennart Regebro wrote:
>>
>> Since the only monotonic clock that can be adjusted by NTP is Linux'
>> CLOCK_MONOTONIC, if we avoid it, then time.monotonic() would always
>> give a clock that isn't adjusted by NTP.
>
> I thought we decided that NTP adjustment isn't an issue, because
> it's always gradual.

Well, in timings it is an issue, but perhaps not a big one. :-)
In any case, which one we use will not change the API, so if it is
decided it is an issue, we can always more to CLOCK_MONOTONIC_RAW in
the future, once Linux < 2.6.26 (or whatever it was) is deemed
unsupported.

//Lennart
_______________________________________________
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

>>> Since the only monotonic clock that can be adjusted by NTP is Linux'
>>> CLOCK_MONOTONIC, if we avoid it, then time.monotonic() would always
>>> give a clock that isn't adjusted by NTP.
>>
>> I thought we decided that NTP adjustment isn't an issue, because
>> it's always gradual.
>
> Well, in timings it is an issue, but perhaps not a big one. :-)
> In any case, which one we use will not change the API, so if it is
> decided it is an issue, we can always more to CLOCK_MONOTONIC_RAW in
> the future, once Linux < 2.6.26 (or whatever it was) is deemed
> unsupported.

I prefer to use CLOCK_MONOTONIC, not because it is also available for
older Linux kernels, but because it is more reliable. Even if the
underlying clock source is unstable (unstable frequency), a delta of
two reads of the CLOCK_MONOTONIC clock is a result in *seconds*,
whereas CLOCK_MONOTONIC_RAW may use an unit a little bit bigger or
smaller than a second. time.monotonic() unit is the second, as written
in its documentation.

Linux is the OS providing the most reliable monotonic clock, why
should you use a not reliable monotonic clock instead?

NTP doesn't step CLOCK_MONOTONIC, it only slew it.
http://www.python.org/dev/peps/pep-0418/#ntp-adjustment

Victor
_______________________________________________
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

2012/4/5 PJ Eby <pje@telecommunity.com>:
>> More details why it's hard to define such function and why I dropped
>> it from the PEP.
>>
>> If someone wants to propose again such function ("monotonic or
>> fallback to system" clock), two issues should be solved:
>>
>>  - name of the function
>>  - description of the function
>
> Maybe I missed it, but did anyone ever give a reason why the fallback
> couldn't be to Steven D'Aprano's monotonic wrapper algorithm over the system
> clock?  (Given a suitable minimum delta.)  That function appeared to me to
> provide a sufficiently monotonic clock for timeout purposes, if nothing
> else.


Did you read the following section of the PEP?
http://www.python.org/dev/peps/pep-0418/#working-around-operating-system-bugs

Did I miss something? If yes, could you write a patch for the PEP please?

Victor
_______________________________________________
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

Cameron Simpson wrote:
> On 04Apr2012 22:23, PJ Eby <pje@telecommunity.com> wrote:
> | On Apr 4, 2012 7:28 PM, "Victor Stinner" <victor.stinner@gmail.com> wrote:
> | > More details why it's hard to define such function and why I dropped
> | > it from the PEP.
> | >
> | > If someone wants to propose again such function ("monotonic or
> | > fallback to system" clock), two issues should be solved:
> | >
> | > - name of the function
> | > - description of the function
> |
> | Maybe I missed it, but did anyone ever give a reason why the fallback
> | couldn't be to Steven D'Aprano's monotonic wrapper algorithm over the
> | system clock? (Given a suitable minimum delta.) That function appeared to
> | me to provide a sufficiently monotonic clock for timeout purposes, if
> | nothing else.
>
> It was pointed out (by Nick Coglan I think?) that if the system clock
> stepped backwards then a timeout would be extended by at least that
> long. For example, code that waited (by polling the synthetic clock)
> for 1s could easily wait an hour if the system clock stepped back that
> far. Probaby undesirable.

Steven D'Aprano's synthetic clock is able to partially avoid that
situation -- worst case is a timeout of double what you asked for -- so
10 seconds instead of 5 (which is much better than 3600!).

~Ethan~
_______________________________________________
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

On Thu, 05 Apr 2012 08:32:22 -0700
Ethan Furman <ethan@stoneleaf.us> wrote:
>
> Steven D'Aprano's synthetic clock is able to partially avoid that
> situation -- worst case is a timeout of double what you asked for -- so
> 10 seconds instead of 5 (which is much better than 3600!).

The remaining issue is that the clock is not system-wide, it's
interpreter-specific.

Regards

Antoine.


_______________________________________________
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

On Wed, Apr 4, 2012 at 11:41 PM, Cameron Simpson <cs@zip.com.au> wrote:

> On 04Apr2012 22:23, PJ Eby <pje@telecommunity.com> wrote:
> | On Apr 4, 2012 7:28 PM, "Victor Stinner" <victor.stinner@gmail.com>
> wrote:
> | > More details why it's hard to define such function and why I dropped
> | > it from the PEP.
> | >
> | > If someone wants to propose again such function ("monotonic or
> | > fallback to system" clock), two issues should be solved:
> | >
> | > - name of the function
> | > - description of the function
> |
> | Maybe I missed it, but did anyone ever give a reason why the fallback
> | couldn't be to Steven D'Aprano's monotonic wrapper algorithm over the
> | system clock? (Given a suitable minimum delta.) That function appeared
> to
> | me to provide a sufficiently monotonic clock for timeout purposes, if
> | nothing else.
>
> It was pointed out (by Nick Coglan I think?) that if the system clock
> stepped backwards then a timeout would be extended by at least that
> long. For example, code that waited (by polling the synthetic clock)
> for 1s could easily wait an hour if the system clock stepped back that
> far. Probaby undesirable.
>

Steven D'Aprano's algorithm doesn't do that. If the system clock steps
backwards, it still stepped forward by a specified minimum delta. The
amount of time that a timeout was extended would be a function of the
polling frequency, not the presence of absence of backward steps in the
underlying clock.

On Thu, Apr 5, 2012 at 6:34 AM, Victor Stinner <victor.stinner@gmail.com>wrote:

> 2012/4/5 PJ Eby <pje@telecommunity.com>:
> >> More details why it's hard to define such function and why I dropped
> >> it from the PEP.
> >>
> >> If someone wants to propose again such function ("monotonic or
> >> fallback to system" clock), two issues should be solved:
> >>
> >> - name of the function
> >> - description of the function
> >
> > Maybe I missed it, but did anyone ever give a reason why the fallback
> > couldn't be to Steven D'Aprano's monotonic wrapper algorithm over the
> system
> > clock? (Given a suitable minimum delta.) That function appeared to me
> to
> > provide a sufficiently monotonic clock for timeout purposes, if nothing
> > else.
>
>
> Did you read the following section of the PEP?
>
> http://www.python.org/dev/peps/pep-0418/#working-around-operating-system-bugs
>
> Did I miss something? If yes, could you write a patch for the PEP please?
>

What's missing is that if you're using a monotonic clock for timeouts, then
a monotonically-adjusted system clock can do that, subject to the polling
frequency -- it does not break just because the system clock is set
backwards; it simply loses time proportional to the frequency with which it
is polled.

For timeout purposes in a single process, such a clock is useful. It just
isn't suitable for benchmarks, or for interprocess coordination.

On Thu, Apr 5, 2012 at 9:48 AM, PJ Eby <pje@telecommunity.com> wrote:
>
>
> On Thu, Apr 5, 2012 at 6:34 AM, Victor Stinner <victor.stinner@gmail.com>
> wrote:
>>
>> 2012/4/5 PJ Eby <pje@telecommunity.com>:
>> >> More details why it's hard to define such function and why I dropped
>> >> it from the PEP.
>> >>
>> >> If someone wants to propose again such function ("monotonic or
>> >> fallback to system" clock), two issues should be solved:
>> >>
>> >>  - name of the function
>> >>  - description of the function
>> >
>> > Maybe I missed it, but did anyone ever give a reason why the fallback
>> > couldn't be to Steven D'Aprano's monotonic wrapper algorithm over the
>> > system
>> > clock?  (Given a suitable minimum delta.)  That function appeared to me
>> > to
>> > provide a sufficiently monotonic clock for timeout purposes, if nothing
>> > else.
>>
>>
>> Did you read the following section of the PEP?
>>
>> http://www.python.org/dev/peps/pep-0418/#working-around-operating-system-bugs
>>
>> Did I miss something? If yes, could you write a patch for the PEP please?
>
>
> What's missing is that if you're using a monotonic clock for timeouts, then
> a monotonically-adjusted system clock can do that, subject to the polling
> frequency -- it does not break just because the system clock is set
> backwards; it simply loses time proportional to the frequency with which it
> is polled.

Depending on the polling frequency sounds like a bad idea, since you
can't know that you're the only user of the clock. Also depending on
the use case, too short a timeout may be worse than too long a
timeout. E.g. imagine hitting a website that usually takes 2 seconds
to respond, and setting a timeout to e.g. 4 seconds to bail. If the
timeout somehow gets reduced to 1 second it will appear as if the
website died, whereas if the timeout was increased to 1 hour, nothing
bad would happen unless the website *actually* started having truly
bad response times.

> For timeout purposes in a single process, such a clock is useful.  It just
> isn't suitable for benchmarks, or for interprocess coordination.

I think it would be better if the proposed algorithm (or whatever
algorithm to "fix" timeouts) was implemented by the
application/library code using the timeout (or provided as a separate
library function), rather than by the clock, since the clock can't
know what fallback behavior the app/lib needs.

--
--Guido van Rossum (python.org/~guido)
_______________________________________________
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

On Thu, 5 Apr 2012 09:56:19 -0700
Guido van Rossum <guido@python.org> wrote:
>
> > For timeout purposes in a single process, such a clock is useful.  It just
> > isn't suitable for benchmarks, or for interprocess coordination.
>
> I think it would be better if the proposed algorithm (or whatever
> algorithm to "fix" timeouts) was implemented by the
> application/library code using the timeout (or provided as a separate
> library function), rather than by the clock, since the clock can't
> know what fallback behavior the app/lib needs.

Agreed with providing it as a separate library function.

Regards

Antoine.


_______________________________________________
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

>> > For timeout purposes in a single process, such a clock is useful.  It just
>> > isn't suitable for benchmarks, or for interprocess coordination.
>>
>> I think it would be better if the proposed algorithm (or whatever
>> algorithm to "fix" timeouts) was implemented by the
>> application/library code using the timeout (or provided as a separate
>> library function), rather than by the clock, since the clock can't
>> know what fallback behavior the app/lib needs.
>
> Agreed with providing it as a separate library function.

I changed time.monotonic() to not fallback to the system clock exactly
for this reason: Python cannot guess what the developer expects, or
how the developer will use the clock.

Instead of implementing your own clock in your application, it's maybe
easier to patch your OS? I suppose that you are running on GNU/Hurd,
because I didn't find yet other OS not providing a monotonic clock :-)

If you are using an OS that doesn't provide a monotonic clock, do you
really need to implement your own in your application?

Victor
_______________________________________________
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

Victor Stinner wrote:
> I changed time.monotonic() to not fallback to the system clock exactly
> for this reason: Python cannot guess what the developer expects, or
> how the developer will use the clock.

Which is exactly why I like Cameron Simpson's approach to selecting a
clock -- let the developer/user decide what kind of clock they need, and
ask for one that matches their criteria.

~Ethan~
_______________________________________________
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

On Thu, Apr 5, 2012 at 12:56 PM, Guido van Rossum <guido@python.org> wrote:

> Depending on the polling frequency sounds like a bad idea, since you
> can't know that you're the only user of the clock. Also depending on
> the use case, too short a timeout may be worse than too long a
> timeout.


Given a small enough delta, the timeout won't be too short. (Steven's
original code sample I believed used either 0 or 1 as a delta, but it could
be as small a fraction as will add correctly in the datatype used.) And
the worst case polling frequency is the length of the timeout, meaning you
can't end up with more than double your intended timeout.

In the opposite scenario, where the time is polled in a tight loop, then as
long as Python doesn't sample the raw clock so often that the summed
fractional deltas exceeds the real clock speed, the timeout won't be
shortened by any appreciable amount. In fact, this can be guaranteed by
measuring time as a (raw, increment) tuple, where the increment can be an
arbitrarily-large integer. Each new time value is greater than the one
before, yet the real component remains untouched. With this approach, the
timeout can only be delayed for however long the system clock *stops*, and
the timeout can only be shortened by the system clock skipping ahead.

Okay, having thought that out, I now agree that there are too many fine
points to make this cover enough of the use cases without needing
parameters.

Or more to the point, "If the implementation is hard to explain, it's a bad
idea." ;-)

Folks:

Good job, Victor Stinner on baking the accumulated knowledge of this
thread into PEP 418. Even though I'm very interested in the topic, I
haven't been able to digest the whole thread(s) on the list and
understand what the current collective understanding is. The detailed
PEP document helps a lot.

I think there are still some mistakes, either in our collective
understanding as reflected by the PEP, or in my own head.

For starters, I still don't understand the first, most basic thing:
what do people mean when they say "monotonic clock"? I don't
understand the current text of PEP 418 with regard to the definition
of that word.

Allow me to resort to an analogy. There is an infinitely long,
perfectly straight and flat racetrack. There is a flag that gets
dragged along it at a constant rate, with the label "REAL TIME" on the
flag. There are some runners, each with a different label on their
chest:

Runner A: a helicopter hovers over Runner A. Occasionally it picks him
up and plops him down right next to the flag. Also, he wears a headset
and listens to instructions from his coach to run a little faster or
slower, as necessary, to remain abreast of the flag.

Runner B: a helicopter hovers over Runner B. If he is behind the flag,
it will pick him up and plop him down right next to the flag. However,
if he is ahead of the flag it will not pick him up.

Runner C: no helicopter ever picks up Runner C, but he does wear a
headset and listens to instructions from his coach to run a little
faster or a little slower. His coach tells him to run a little faster
if he is behind the flag or run a little slower if he is in front of
the flag, with the goal of eventually having him right next to the
flag.

Runner D: like Runner C, he never gets picked up, but he listens to
instructions to run a little faster or a little slower. However,
instead of telling him to run faster in order to catch up to the flag,
or to run slower in order to "catch down" to the flag, his coach
instead tells him to run a little faster if he is moving slower than
the flag is moving, and to run a little slower if he is moving faster
than the flag is moving. Note that this is very different from Runner
C, in that it is not intended to cause him to eventually be right next
to the flag, and indeed if it is done right it guarantees that he will
*never* be right next to the flag, although he will be moving just as
fast as the flag is moving.

Runner E: no helicopter, no headset. He just proceeds at his own pace,
blissfully unaware of the exhortations of others.

Now: which ones of these five runners do you call "monotonic"? Which
ones do you call "steady"?

Regards,

Zooko
_______________________________________________
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

On 05Apr2012 09:56, Guido van Rossum <guido@python.org> wrote:
| On Thu, Apr 5, 2012 at 9:48 AM, PJ Eby <pje@telecommunity.com> wrote:
| > What's missing is that if you're using a monotonic clock for timeouts, then
| > a monotonically-adjusted system clock can do that, subject to the polling
| > frequency -- it does not break just because the system clock is set
| > backwards; it simply loses time proportional to the frequency with which it
| > is polled.
|
| Depending on the polling frequency sounds like a bad idea, since you
| can't know that you're the only user of the clock.

You can if you're handed a shiny new "clock" object in some way, with a
not-a-singleton guarrentee. Of course, such a clock is immediately
_less_ reliable to synchornisation with other clock users:-)

| Also depending on
| the use case, too short a timeout may be worse than too long a
| timeout. [...]
|
| > For timeout purposes in a single process, such a clock is useful.  It just
| > isn't suitable for benchmarks, or for interprocess coordination.
|
| I think it would be better if the proposed algorithm (or whatever
| algorithm to "fix" timeouts) was implemented by the
| application/library code using the timeout (or provided as a separate
| library function), rather than by the clock, since the clock can't
| know what fallback behavior the app/lib needs.

Absolutely. I for one would be happy with a clocktools module or
something offering a bunch of synthetic clocks. Especially if they were
compatible in API with whatever clock objects the core time module
clocks used, so that a user _could_ add them into the pick-a-clock
decision easily.

Cheers,
--
Cameron Simpson <cs@zip.com.au> DoD#743
http://www.cskk.ezoshosting.com/cs/

I'd be careful who you call smart or not smart. Smart isn't knowing how to
save six bytes. Smart is knowing WHEN. - Peter Cherna, Amiga O.S. Development
_______________________________________________
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

On 05Apr2012 13:39, Zooko Wilcox-O'Hearn <zooko@zooko.com> wrote:
| Good job, Victor Stinner on baking the accumulated knowledge of this
| thread into PEP 418. Even though I'm very interested in the topic, I
| haven't been able to digest the whole thread(s) on the list and
| understand what the current collective understanding is.

There isn't a collective understanding :-) That's why all the noise!

| The detailed
| PEP document helps a lot.

Yes indeed, though like all of us I think it could (always) use more
detail on my pet concerns.

| I think there are still some mistakes, either in our collective
| understanding as reflected by the PEP, or in my own head.
|
| For starters, I still don't understand the first, most basic thing:
| what do people mean when they say "monotonic clock"? I don't
| understand the current text of PEP 418 with regard to the definition
| of that word.

A monotonic clock never returns t0 > t1 for t0, t1 being two adjacent
polls of the clock. On its own it says nothing about steadiness or
correlation with real world time.

_Quality of implementation_ says that the montonic() call should try to
return a close that is monotonic and _also_ steady and preferably
precise. How these things are balancer is a matter of policy.

| Allow me to resort to an analogy. There is an infinitely long,
| perfectly straight and flat racetrack. There is a flag that gets
| dragged along it at a constant rate, with the label "REAL TIME" on the
| flag. There are some runners, each with a different label on their
| chest:
|
| Runner A: a helicopter hovers over Runner A. Occasionally it picks him
| up and plops him down right next to the flag. Also, he wears a headset
| and listens to instructions from his coach to run a little faster or
| slower, as necessary, to remain abreast of the flag.

If he always runs forwards, it is montonic. Not very steady when the
helicopter comes to play.

| Runner B: a helicopter hovers over Runner B. If he is behind the flag,
| it will pick him up and plop him down right next to the flag. However,
| if he is ahead of the flag it will not pick him up.

Seems like runner A without instruction. Monotonic. Not very steady.

| Runner C: no helicopter ever picks up Runner C, but he does wear a
| headset and listens to instructions from his coach to run a little
| faster or a little slower. His coach tells him to run a little faster
| if he is behind the flag or run a little slower if he is in front of
| the flag, with the goal of eventually having him right next to the
| flag.

If he always runs forward, monotonic. And steady.

| Runner D: like Runner C, he never gets picked up, but he listens to
| instructions to run a little faster or a little slower. However,
| instead of telling him to run faster in order to catch up to the flag,
| or to run slower in order to "catch down" to the flag, his coach
| instead tells him to run a little faster if he is moving slower than
| the flag is moving, and to run a little slower if he is moving faster
| than the flag is moving. Note that this is very different from Runner
| C, in that it is not intended to cause him to eventually be right next
| to the flag, and indeed if it is done right it guarantees that he will
| *never* be right next to the flag, although he will be moving just as
| fast as the flag is moving.
|
| Runner E: no helicopter, no headset. He just proceeds at his own pace,
| blissfully unaware of the exhortations of others.
|
| Now: which ones of these five runners do you call "monotonic"? Which
| ones do you call "steady"?

If they all run forwards, they're all monotonic.

If their coach or helicopter can move then _backwards_ they're not
monotonic. If the helicopter can move them an arbitrary (but matching
the game plan) distance, they're not steady. Otherwise they are steady,
if the runner's speed is always sufficiently close to the flag speed
(this threshold and the criteria for measuring it as subject to debate,
forming policy).

And "high resolution" has its own flavours, though generally less
contentious.

Cheers,
--
Cameron Simpson <cs@zip.com.au> DoD#743
http://www.cskk.ezoshosting.com/cs/

If you don't shoot the fish in your barrel, your barrel will soon be
full of fish. - Tim Mefford
_______________________________________________
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

On Apr 5, 2012 11:01 AM, "Guido van Rossum" <guido@python.org> wrote:
> I think it would be better if the proposed algorithm (or whatever
> algorithm to "fix" timeouts) was implemented by the
> application/library code using the timeout (or provided as a separate
> library function), rather than by the clock, since the clock can't
> know what fallback behavior the app/lib needs.

+1

-eric

Cameron Simpson wrote:

> A monotonic clock never returns t0 > t1 for t0, t1 being two adjacent
> polls of the clock. On its own it says nothing about steadiness or
> correlation with real world time.

No, no, no.

This is the strict mathematical meaning of the word "monotonic",
but the way it's used in relation to OS clocks, it seems to
mean rather more than that.

A clock whose only guarantee is that it never goes backwards
is next to useless. For things like benchmarks and timeouts,
the important thing about a clock that it *keeps going forward*
at a reasonably constant rate. On the other hand, it can have
an arbitrary starting point and doesn't have to be related
to any external time standard.

I'm assuming this is what Linux et al mean when they talk
about a "monotonic clock", because anything else doesn't make
sense.

So if we're going to use the term "monotonic" at all, I think we
should explicitly define it as having this meaning, i.e.
both mathematically monotonic and steady. Failure to be clear
about this has caused a huge amount of confusion in this thead
so far.

--
Greg
_______________________________________________
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

On 06Apr2012 13:14, Greg Ewing <greg.ewing@canterbury.ac.nz> wrote:
| Cameron Simpson wrote:
| > A monotonic clock never returns t0 > t1 for t0, t1 being two adjacent
| > polls of the clock. On its own it says nothing about steadiness or
| > correlation with real world time.
|
| No, no, no.
| This is the strict mathematical meaning of the word "monotonic",
| but the way it's used in relation to OS clocks, it seems to
| mean rather more than that.

It's like my next paragraph didn't exist...

_Quality of implementation_ says that the montonic() call should try to
return a close that is monotonic and _also_ steady and preferably precise.
How these things are balancer is a matter of policy.

To forstall things, right at the bottom of this I'm going to say i agree
with you about objectives, but not about terminology. I maintain that
"monotonic" still means what I said, and that it is the combination of
the word with "clock" that brings in your other criteria.

| A clock whose only guarantee is that it never goes backwards
| is next to useless. For things like benchmarks and timeouts,
| the important thing about a clock that it *keeps going forward*
| at a reasonably constant rate. [...]
| I'm assuming this is what Linux et al mean when they talk
| about a "monotonic clock", because anything else doesn't make
| sense.

This is why I say there's no global understanding. Why assume?
On a Linux box, "man 3 clock_getres" says (sorry, this is a bit wordy):

All implementations support the system-wide real-time clock, which
is identified by CLOCK_REALTIME. Its time represents seconds and
nanoseconds since the Epoch. When its time is changed, timers for
a relative interval are unaffected, but timers for an absolute point
in time are affected.

More clocks may be implemented. The interpretation of the
corresponding time values and the effect on timers is unspecified.

Sufficiently recent versions of glibc and the Linux kernel support
the following clocks:

CLOCK_REALTIME
System-wide real-time clock. Setting this clock requires
appropriate privileges.

CLOCK_MONOTONIC
Clock that cannot be set and represents monotonic time since
some unspecified starting point.

CLOCK_MONOTONIC_RAW (since Linux 2.6.28; Linux-specific)
Similar to CLOCK_MONOTONIC, but provides access to a raw hard-
ware-based time that is not subject to NTP adjustments.

CLOCK_PROCESS_CPUTIME_ID
High-resolution per-process timer from the CPU.

CLOCK_THREAD_CPUTIME_ID
Thread-specific CPU-time clock.

The first paragraph is very clear about real time (wall clock, and what
time.time() does, being "seconds since the epoch"). The CLOCK_MONOTONIC*
modes clearly imply steadiness.

"man 3p clock_getres" (POSIX clock_getres) is even more verbose and general.

| So if we're going to use the term "monotonic" at all, I think we
| should explicitly define it as having this meaning, i.e.
| both mathematically monotonic and steady.

I think if it is too "unsteady" then it is not "time".

So I actually side with you in the requirement for a "clock", but
monotonic alone does not mean that. Quality of implementation _may_ mean
we don't offer something abjectly erratic.

| Failure to be clear
| about this has caused a huge amount of confusion in this thead
| so far.

And burdening the word "monotonic" itself with exciting new meanings
doesn't help. "monotonic clock", sure, _that_ has additional
connotations, but not just the word monotonic.

Cheers,
--
Cameron Simpson <cs@zip.com.au> DoD#743
http://www.cskk.ezoshosting.com/cs/

We had the experience, but missed the meaning. - T.S. Eliot
_______________________________________________
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

Greg Ewing wrote:
> Cameron Simpson wrote:
>
>> A monotonic clock never returns t0 > t1 for t0, t1 being two adjacent
>> polls of the clock. On its own it says nothing about steadiness or
>> correlation with real world time.
>
> No, no, no.
>
> This is the strict mathematical meaning of the word "monotonic",
> but the way it's used in relation to OS clocks, it seems to
> mean rather more than that.
>
> A clock whose only guarantee is that it never goes backwards
> is next to useless. For things like benchmarks and timeouts,
> the important thing about a clock that it *keeps going forward*

That would be a *strictly* monotonic clock, as opposed to merely monotonic.

And yes, a merely monotonic clock could just return a constant value, forever:

9, 9, 9, 9, 9, ...

and yes, such a thing would be useless.

Various people have suggested caching the last value of time() and re-using it
if the new value is in the past. This will give a monotonic clock, but since
it can give constant timestamps for an indefinite period, it's usefulness is
limited.

I earlier put forward an alternate implementation which gives no more than one
such constant tick in a row. If you know the hardware resolution of the clock,
you can even avoid that single constant tick by always advancing the timestamp
by that minimum resolution:

_prev = _prev_raw = 0
_res = 1e-9 # nanosecond resolution
def monotonic():
global _prev, _prev_raw
raw = time()
delta = max(_res, raw - _prev_raw)
_prev_raw = raw
_prev += delta
return _prev

Even if time() jumps backwards, or stays constant, monotonic() here will be
strictly monotonic.



--
Steven
_______________________________________________
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

Cameron Simpson wrote:

> I maintain that
> "monotonic" still means what I said, and that it is the combination of
> the word with "clock" that brings in your other criteria.

I'm not trying to redefine the word "monotonic" in general.
All I'm saying is that *if* the PEP is going to talk about
a "monotonic clock" or "monotonic time", it should clearly
define what this means, because it's not obvious to everyone
that it implies something more than the mathematical meaning.

Alternatively, don't use the word "monotonic" at all, and
find a better term.

> CLOCK_MONOTONIC
> Clock that cannot be set and represents monotonic time since
> some unspecified starting point.

Which doesn't help very much, because it talks about "monotonic
time" without saying what that means. Googling for that phrase
doesn't seem turn up anything very useful. Apparently we're
supposed to just know.

--
Greg
_______________________________________________
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

Steven D'Aprano wrote:
> Greg Ewing wrote:
>
>> the important thing about a clock that it *keeps going forward*
>
> That would be a *strictly* monotonic clock, as opposed to merely monotonic.

Well, yes, but even that's not enough -- it needs to go forward
at a reasonably constant rate, otherwise it's just as useless.
If it had enough resolution, it could go forward by one femtosecond
every hour for a while and still call itself strictly monotonic...

--
Greg
_______________________________________________
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

On 6 April 2012 02:50, Cameron Simpson <cs@zip.com.au> wrote:
(Quoted from the Linux manpage)

> All implementations support the system-wide real-time clock, which
> is identified by CLOCK_REALTIME. Its time represents seconds and
> nanoseconds since the Epoch. When its time is changed, timers for
> a relative interval are unaffected, but timers for an absolute point
> in time are affected.
>

This made me think. They make a distinction between "timers for a relative
interval" and "timers for an absolute point in time".

But that's not right - *all* of the clock calls we are talking about here
return a *single* number. Interpreting that as an absolute time needs an
epoch. Or to put it another way, clock values are always meaningless
without context - whereas clock *differences* are what actually carry
meaning (in terms of a duration).

On that basis, I'd say that

- A clock that doesn't get adjusted or slewed is a tick counter (which
technically doesn't have any relationship to time, although the tick
frequency can be used to convert to seconds, but see the next entry)
- A clock that gets slewed but not adjusted is a seconds counter (or at
least, the nearest approximation the system can provide - presumably better
than a tick counter)
- A clock that gets adjusted is not an interval timer at all, but an
absolute timer (which therefore shouldn't really be used for benchmarking
or timeouts)

It seems to me that what *I* would most often need are the second two of
these (to at least as high a precision as my app needs, which may vary but
"to the highest precision possible" would do :-)) I'd be happy for a
seconds counter to fallback to a tick counter converted to seconds using
its frequency - slewing is simply an accuracy improvement process, as far
as I can see.

It seems to me that the current time.time() and time.wallclock() are the
right names for my "absolute timer" and "seconds timer" above. Whether
their implementations match my definitions I'm not sure, but that's what
I'd hope. One thing I would expect is that time.wallclock() would never go
backwards (so differences are always positive). The various other debates
about monotonic, steady, etc, seem to me to be only relevant for specialist
uses that I don't care about.

As regards suspension, if I'm timing intervals and the system suspends, I'd
be happy to say all bets are off. Similarly with timeouts. If I cared, I'd
simply make sure the system didn't suspend :-)

As far as comparability between different threads or processes are
concerned, I would expect absolute time (time.time) to be the same across
threads or processes (but wouldn't generally write apps that were affected
if it weren't - at least by small amounts), but I wouldn't expect
time.wallclock values obtained in different threads or processes to be
comparable (mostly because I can't think of a case where I'd compare them).
Where VMs or multiple machines are involved, I wouldn't even expect
absolute time to match (but that's the job of NTP, and if time.time follows
NTP, there's no reason why there would be an issue even there).

Summary: I'm happy with time.time and time.wallclock. The rest of this
debate doesn't matter for my usecases (and I suspect many other people's in
practice).

[Update, after I downloaded and installed 3.3a2] Bah, looks like
time.wallclock is gone. (Actually, looks like it was documented but not
implemented in 3.3a1!). Actually, the docs and the implementation don't
match - clock_gettime is documented as available, but it's not (at least on
Windows). I still prefer time.wallclock() as described above and in the
3.3a1 documentation. I thought I knew what was going on, but now I'm
confused. My comments above still stand, though.

Paul.

On Thu, Apr 5, 2012 at 12:32, Victor Stinner <victor.stinner@gmail.com> wrote:
> I prefer to use CLOCK_MONOTONIC, not because it is also available for
> older Linux kernels, but because it is more reliable. Even if the
> underlying clock source is unstable (unstable frequency), a delta of
> two reads of the CLOCK_MONOTONIC clock is a result in *seconds*,
> whereas CLOCK_MONOTONIC_RAW may use an unit a little bit bigger or
> smaller than a second.

Aha. OK, CLOCK_MONOTONIC it is then.

//Lennart
_______________________________________________
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

On 4 April 2012 10:33, Steven D'Aprano <steve@pearwood.info> wrote:

> try:
> from os import bestclock as _bestclock
> except ImportError:
> _bestclock = time
>

My problem here is that "best clock" means different things to different
people (as the number of emails shows).

I think exposing specific clocks is also useful (sometimes people may need
a steady clock, and early failure is better than clock skew). However, I
propose a loosely classified set of clocks built on top of the specific
clocks, all of which can fall back to the lowest precision/non-monotonic
clock if needed.

1. The "steadiest" clock on the system. Ideally this would be a steady
clock, but may not be.

2. The "most precise" clock on the system. This would have the finest-grain
tick available on the system.

3. The "highest performance" (or maybe "lowest latency") clock. This would
be whichever clock on the system returned its results fastest.

I'm not sure if there are more that would be needed ("most accurate" comes
to mind, but feels like it's arbitrarily choosing between steadiest and
most precise, so I don't think it's valid).

By choosing relative terms, it caters to people's desire to have the "best"
clock, but doesn't set the expectation that the behaviour is guaranteed.

Tim Delaney

> 1. The "steadiest" clock on the system. Ideally this would be a steady
> clock, but may not be.

time.monotonic() as proposed in the PEP 418 *is* the steadiest
available clock, but we cannot say that it is steady :-)

> 2. The "most precise" clock on the system. This would have the finest-grain
> tick available on the system.

It's discussed in the "Deferred API: time.perf_counter()" section. It
would be nice to provide such clock, but I don't feel able right now
to propose ab API for such requirement. It's unclear to me if it must
be monotonic, steady, count elapsed time during a sleep or not, etc.

It is already very hard to propose one single time function
(time.monotonic), so I chose to simplify the PEP and not propose two
functions but only one :-)

> 3. The "highest performance" (or maybe "lowest latency") clock. This would
> be whichever clock on the system returned its results fastest.

Linux provides CLOCK_REALTIME_COARSE and CLOCK_MONOTONIC_COARSE
clocks, reading the ACPI Power Management clock is known to be slow.
But should the clock be monotonic or not? Return seconds or CPU ticks?
If the clock is not well defined, it's useless or at least, not
portable. Exposing CLOCK_REALTIME_COARSE and CLOCK_MONOTONIC_COARSE
constants should be enough.

Victor
_______________________________________________
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

On 4/5/2012 6:32 AM, Victor Stinner wrote:
>>>> Since the only monotonic clock that can be adjusted by NTP is Linux'
>>>> CLOCK_MONOTONIC, if we avoid it, then time.monotonic() would always
>>>> give a clock that isn't adjusted by NTP.
>>>
>>> I thought we decided that NTP adjustment isn't an issue, because
>>> it's always gradual.
>>
>> Well, in timings it is an issue, but perhaps not a big one. :-)
>> In any case, which one we use will not change the API, so if it is
>> decided it is an issue, we can always more to CLOCK_MONOTONIC_RAW in
>> the future, once Linux< 2.6.26 (or whatever it was) is deemed
>> unsupported.
>
> I prefer to use CLOCK_MONOTONIC, not because it is also available for
> older Linux kernels, but because it is more reliable. Even if the
> underlying clock source is unstable (unstable frequency), a delta of
> two reads of the CLOCK_MONOTONIC clock is a result in *seconds*,
> whereas CLOCK_MONOTONIC_RAW may use an unit a little bit bigger or
> smaller than a second. time.monotonic() unit is the second, as written
> in its documentation.
>

I believe the above is only true for sufficiently large time deltas. One
of the major purposes of NTP slewing is to give up some short term
accuracy in order to achieve long term accuracy (e.g. whenever the clock
is found to be ahead of real time it is purposefully ticked slower than
real time).

So for benchmarking it would not be surprising to be better off with the
non-adjusted clock. Ideally there would be a clock that was slewed "just
enough" to try and achieve short term accuracy, but I don't know of
anything providing that.

Janzert

_______________________________________________
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

2012/4/7 Janzert <janzert@janzert.com>:
> On 4/5/2012 6:32 AM, Victor Stinner wrote:
>> I prefer to use CLOCK_MONOTONIC, not because it is also available for
>> older Linux kernels, but because it is more reliable. Even if the
>> underlying clock source is unstable (unstable frequency), a delta of
>> two reads of the CLOCK_MONOTONIC clock is a result in *seconds*,
>> whereas CLOCK_MONOTONIC_RAW may use an unit a little bit bigger or
>> smaller than a second. time.monotonic() unit is the second, as written
>> in its documentation.
>
> I believe the above is only true for sufficiently large time deltas. One of
> the major purposes of NTP slewing is to give up some short term accuracy in
> order to achieve long term accuracy (e.g. whenever the clock is found to be
> ahead of real time it is purposefully ticked slower than real time).

I don't think that NTP works like that. NTP only uses very smooth adjustements:

""slewing": change the clock frequency to be slightly faster or slower
(which is done with adjtime()). Since the slew rate is limited to 0.5
ms/s, each second of adjustment requires an amortization interval of
2000 s. Thus, an adjustment of many seconds can take hours or days to
amortize."
http://www.python.org/dev/peps/pep-0418/#ntp-adjustment

> So for benchmarking it would not be surprising to be better off with the
> non-adjusted clock. Ideally there would be a clock that was slewed "just
> enough" to try and achieve short term accuracy, but I don't know of anything
> providing that.

time.monotonic() is not written for benchmarks. It does not have the
highest frequecency, it's primary property is that is monotonic. A
side effect is that it is usually the steadiest clock.

For example, on Windows time.monotonic() has only an accuracy of 15 ms
(15 milliseconds not 15 microseconds).

If you consider that the PEP should also solve the issue of
benchmarking clock, we should continue the work on this section:
http://www.python.org/dev/peps/pep-0418/#deferred-api-time-perf-counter

Victor
_______________________________________________
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

Victor Stinner wrote:
> 2012/4/7 Janzert <janzert@janzert.com>:
>> On 4/5/2012 6:32 AM, Victor Stinner wrote:
>>> I prefer to use CLOCK_MONOTONIC, not because it is also available for
>>> older Linux kernels, but because it is more reliable. Even if the
>>> underlying clock source is unstable (unstable frequency), a delta of
>>> two reads of the CLOCK_MONOTONIC clock is a result in *seconds*,
>>> whereas CLOCK_MONOTONIC_RAW may use an unit a little bit bigger or
>>> smaller than a second. time.monotonic() unit is the second, as written
>>> in its documentation.
>> I believe the above is only true for sufficiently large time deltas. One of
>> the major purposes of NTP slewing is to give up some short term accuracy in
>> order to achieve long term accuracy (e.g. whenever the clock is found to be
>> ahead of real time it is purposefully ticked slower than real time).
>
> I don't think that NTP works like that. NTP only uses very smooth adjustements:
>
> ""slewing": change the clock frequency to be slightly faster or slower
> (which is done with adjtime()). Since the slew rate is limited to 0.5
> ms/s, each second of adjustment requires an amortization interval of
> 2000 s. Thus, an adjustment of many seconds can take hours or days to
> amortize."
> http://www.python.org/dev/peps/pep-0418/#ntp-adjustment


That is incorrect. NTP by default will only slew the clock for small
discrepancies. For large discrepancies, it will step the clock, causing the
time to jump. By default, "large" here means more than 128 milliseconds.

Yes, milliseconds.

http://www.ntp.org/ntpfaq/NTP-s-config-tricks.htm#AEN4249


In any case, NTP is not the only thing that adjusts the clock, e.g. the
operating system will adjust the time for daylight savings.


--
Steven

_______________________________________________
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

2012/4/7 Steven D'Aprano <steve@pearwood.info>:
> Victor Stinner wrote:
>>
>> 2012/4/7 Janzert <janzert@janzert.com>:
>>>
>>> On 4/5/2012 6:32 AM, Victor Stinner wrote:
>>>>
>>>> I prefer to use CLOCK_MONOTONIC, not because it is also available for
>>>> older Linux kernels, but because it is more reliable. Even if the
>>>> underlying clock source is unstable (unstable frequency), a delta of
>>>> two reads of the CLOCK_MONOTONIC clock is a result in *seconds*,
>>>> whereas CLOCK_MONOTONIC_RAW may use an unit a little bit bigger or
>>>> smaller than a second. time.monotonic() unit is the second, as written
>>>> in its documentation.
>>>
>>> I believe the above is only true for sufficiently large time deltas. One
>>> of
>>> the major purposes of NTP slewing is to give up some short term accuracy
>>> in
>>> order to achieve long term accuracy (e.g. whenever the clock is found to
>>> be
>>> ahead of real time it is purposefully ticked slower than real time).
>>
>> I don't think that NTP works like that. NTP only uses very smooth
>> adjustements:
>>
>> ""slewing": change the clock frequency to be slightly faster or slower
>> (which is done with adjtime()). Since the slew rate is limited to 0.5
>> ms/s, each second of adjustment requires an amortization interval of
>> 2000 s. Thus, an adjustment of many seconds can take hours or days to
>> amortize."
>> http://www.python.org/dev/peps/pep-0418/#ntp-adjustment
>

> That is incorrect. NTP by default will only slew the clock for small
> discrepancies. For large discrepancies, it will step the clock, causing the
> time to jump. By default, "large" here means more than 128 milliseconds.
>
> Yes, milliseconds.
>
> http://www.ntp.org/ntpfaq/NTP-s-config-tricks.htm#AEN4249

We are talking about CLOCK_MONOTONIC. Steping is disabled on this clock.

Victor
_______________________________________________
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

On 07Apr2012 20:40, Steven D'Aprano <steve@pearwood.info> wrote:
| Victor Stinner wrote:
| > I don't think that NTP works like that. NTP only uses very smooth adjustements:
[...]
| > http://www.python.org/dev/peps/pep-0418/#ntp-adjustment
|
| That is incorrect. NTP by default will only slew the clock for small
| discrepancies. For large discrepancies, it will step the clock, causing the
| time to jump. By default, "large" here means more than 128 milliseconds.
| Yes, milliseconds.
| http://www.ntp.org/ntpfaq/NTP-s-config-tricks.htm#AEN4249
|
| In any case, NTP is not the only thing that adjusts the clock, e.g. the
| operating system will adjust the time for daylight savings.

Ignoring the discussion of NTP, daylight saving is a _presentation_
issue. It is _display_. The OS clock does not change for daylight
saving! Think: "seconds since the epoch". This is a continuous function.
Daylight saving presentation occurs turning a seconds-since-epoch into a
human decomposed time (hours, etc).

Now, AFAIR, Windows used to run its system clock in "local time"
i.e. it did have to jump its clock for daylight saving. Hopefully that
is long gone.

UNIX never did this. It ran in seconds since the epoch (in its case, start of
01jan1970 GMT). Printing dates and timestamps to humans needed daylight
saving knowledge etc.

Cheers,
--
Cameron Simpson <cs@zip.com.au> DoD#743
http://www.cskk.ezoshosting.com/cs/

Stan Malyshev <stas@netcom.com> wrote:
| You're dragging a peg in a blind hairpin when you see a patch of coolant
| up ahead, and you hear the airbrakes of an oncoming 18-wheeler.
| What do you do? WHAT DO YOU DO? ("Speed", anyone?)
Shoot my pillion? - Vociferous Mole <stevegr@Starbase.NeoSoft.COM>
_______________________________________________
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

On Apr 7, 2012, at 3:40 AM, Steven D'Aprano wrote:

> In any case, NTP is not the only thing that adjusts the clock, e.g. the operating system will adjust the time for daylight savings.

Daylight savings time is not a clock adjustment, at least not in the sense this thread has mostly been talking about the word "clock". It doesn't affect the "seconds from epoch" measurement, it affects the way in which the clock is formatted to the user.

-glyph

On Sat, Apr 7, 2012 at 4:56 PM, Glyph Lefkowitz <glyph@twistedmatrix.com>wrote:

> On Apr 7, 2012, at 3:40 AM, Steven D'Aprano wrote:
>
> In any case, NTP is not the only thing that adjusts the clock, e.g. the
> operating system will adjust the time for daylight savings.
>
>
> Daylight savings time is not a clock adjustment, at least not in the sense
> this thread has mostly been talking about the word "clock". It doesn't
> affect the "seconds from epoch" measurement, it affects the way in which
> the clock is formatted to the user.
>
> -glyph
>

even on windows where the system hardware clock is maintained in local time?

Is it still? I thought they fixed that ages ago?

On Mon, Apr 9, 2012 at 4:42 PM, Gregory P. Smith <greg@krypto.org> wrote:
>
> On Sat, Apr 7, 2012 at 4:56 PM, Glyph Lefkowitz <glyph@twistedmatrix.com>
> wrote:
>>
>> On Apr 7, 2012, at 3:40 AM, Steven D'Aprano wrote:
>>
>> In any case, NTP is not the only thing that adjusts the clock, e.g. the
>> operating system will adjust the time for daylight savings.
>>
>>
>> Daylight savings time is not a clock adjustment, at least not in the sense
>> this thread has mostly been talking about the word "clock".  It doesn't
>> affect the "seconds from epoch" measurement, it affects the way in which the
>> clock is formatted to the user.
>>
>> -glyph
>
>
> even on windows where the system hardware clock is maintained in local time?
>
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev@python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> http://mail.python.org/mailman/options/python-dev/guido%40python.org
>



--
--Guido van Rossum (python.org/~guido)
_______________________________________________
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

On Mon, Apr 9, 2012 at 4:46 PM, Guido van Rossum <guido@python.org> wrote:

> Is it still? I thought they fixed that ages ago?
>

sadly, no. http://www.cl.cam.ac.uk/~mgk25/mswish/ut-rtc.html

On Mon, Apr 9, 2012 at 4:42 PM, Gregory P. Smith <greg@krypto.org> wrote:
> >
> > On Sat, Apr 7, 2012 at 4:56 PM, Glyph Lefkowitz <glyph@twistedmatrix.com
> >
> > wrote:
> >>
> >> On Apr 7, 2012, at 3:40 AM, Steven D'Aprano wrote:
> >>
> >> In any case, NTP is not the only thing that adjusts the clock, e.g. the
> >> operating system will adjust the time for daylight savings.
> >>
> >>
> >> Daylight savings time is not a clock adjustment, at least not in the
> sense
> >> this thread has mostly been talking about the word "clock". It doesn't
> >> affect the "seconds from epoch" measurement, it affects the way in
> which the
> >> clock is formatted to the user.
> >>
> >> -glyph
> >
> >
> > even on windows where the system hardware clock is maintained in local
> time?
> >
> >
> > _______________________________________________
> > Python-Dev mailing list
> > Python-Dev@python.org
> > http://mail.python.org/mailman/listinfo/python-dev
> > Unsubscribe:
> > http://mail.python.org/mailman/options/python-dev/guido%40python.org
> >
>
>
>
> --
> --Guido van Rossum (python.org/~guido)
>

>> In any case, NTP is not the only thing that adjusts the clock, e.g. the
>> operating system will adjust the time for daylight savings.
>
> Daylight savings time is not a clock adjustment, at least not in the sense
> this thread has mostly been talking about the word "clock".  It doesn't
> affect the "seconds from epoch" measurement, it affects the way in which the
> clock is formatted to the user.

Ah yes, you're right. The system clock uses the UTC time zone on Linux
and Windows, and it is not affected by DST.

Victor
_______________________________________________
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

On 4/7/2012 5:49 AM, Victor Stinner wrote:
> 2012/4/7 Janzert<janzert@janzert.com>:
>> On 4/5/2012 6:32 AM, Victor Stinner wrote:
>>> I prefer to use CLOCK_MONOTONIC, not because it is also available for
>>> older Linux kernels, but because it is more reliable. Even if the
>>> underlying clock source is unstable (unstable frequency), a delta of
>>> two reads of the CLOCK_MONOTONIC clock is a result in *seconds*,
>>> whereas CLOCK_MONOTONIC_RAW may use an unit a little bit bigger or
>>> smaller than a second. time.monotonic() unit is the second, as written
>>> in its documentation.
>>
>> I believe the above is only true for sufficiently large time deltas. One of
>> the major purposes of NTP slewing is to give up some short term accuracy in
>> order to achieve long term accuracy (e.g. whenever the clock is found to be
>> ahead of real time it is purposefully ticked slower than real time).
>
> I don't think that NTP works like that. NTP only uses very smooth adjustements:
>
> ""slewing": change the clock frequency to be slightly faster or slower
> (which is done with adjtime()). Since the slew rate is limited to 0.5
> ms/s, each second of adjustment requires an amortization interval of
> 2000 s. Thus, an adjustment of many seconds can take hours or days to
> amortize."
> http://www.python.org/dev/peps/pep-0418/#ntp-adjustment
>

Right, the description in that paragraph is exactly what I was referring
to above. :) It is unfortunate that a clock with a resolution of 1ns may
be purposefully thrown off by 500,000ns per second in the short term.

In practice you are probably correct that it is better to take the
slewed clock even though it may have purposeful short term inaccuracy
thrown in than it is to use the completely unadjusted one.

>> So for benchmarking it would not be surprising to be better off with the
>> non-adjusted clock. Ideally there would be a clock that was slewed "just
>> enough" to try and achieve short term accuracy, but I don't know of anything
>> providing that.
>
> time.monotonic() is not written for benchmarks. It does not have the
> highest frequecency, it's primary property is that is monotonic. A
> side effect is that it is usually the steadiest clock.
>
> For example, on Windows time.monotonic() has only an accuracy of 15 ms
> (15 milliseconds not 15 microseconds).
>

Hmm, I just realized an unfortunate result of that. Since it means
time.monotonic() will be too coarse to be useful for frame rate level
timing on windows.

Janzert

_______________________________________________
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com