Mailing List Archive

Docs: audit event table empty
Hi,

the table with auditing events does not render on docs.python.org,
https://docs.python.org/3.9/library/audit_events.html. Steve and I are
going to present the auditing feature tomorrow at EuroPython. It would
be helpful to have the table available.

It works on Steve's and my local machine without any issues. I suspect
it's either an outdated sphinx version or caching issue. Could somebody
from the docs team or shell access to the docs machine please look into
the matter?

Thanks!
Christian

_______________________________________________
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-leave@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/POWT35ULU2CPELWQ6BRTLTU5H3YKHQZW/
Re: Docs: audit event table empty [ In reply to ]
Hi Christian,

> the table with auditing events does not render on docs.python.org,
> https://docs.python.org/3.9/library/audit_events.html. Steve and I are
> going to present the auditing feature tomorrow at EuroPython. It would
> be helpful to have the table available.

This was not an easy one... and it may be a Sphinx issue, yet I'm still not sure, maybe Steve can shed some light on it:

It's the "-j" option of sphinx-build (to parallelize) that causes the issue. I double checked it (full commands at the end of the message in case someone want to reproduce it):

- Run with -j4 ? No table
- Run without -j ? Table is here
- Run again with -j4 ? No table!
- Run again without -j ? Table is back!

I'm patching docsbuild-scripts to stop using -j4 with is not really helpfull anyway as docsbuild script is parallelizing by starting multiple sphinx-build (for multiple languages / versions).

I also copied the file and invalidated the cache, so https://docs.python.org/3.9/library/audit_events.html is good again.

If I'm too slow testing locally and releasing a new docsbuild_script.py, the cron MAY break the file again, don't hesitate to poke me if it happen without me noticing first.

Full test:

docsbuild@docs:/srv/docsbuild/3.9/cpython-en/Doc$ /srv/docsbuild/venv/bin/sphinx-build -b html -d build/doctrees -D latex_elements.papersize= -D latex_engine=xelatex -D latex_elements.inputenc= -D latex_elements.fontenc= -j4 -q -Ea -A daily=1 -A switchers=1 . build/html
docsbuild@docs:/srv/docsbuild/3.9/cpython-en/Doc$ grep breakpoint build/html/library/audit_events.html
docsbuild@docs:/srv/docsbuild/3.9/cpython-en/Doc$ /srv/docsbuild/venv/bin/sphinx-build -b html -d build/doctrees -D latex_elements.papersize= -D latex_engine=xelatex -D latex_elements.inputenc= -D latex_elements.fontenc= -q -Ea -A daily=1 -A switchers=1 . build/html
docsbuild@docs:/srv/docsbuild/3.9/cpython-en/Doc$ grep breakpoint build/html/library/audit_events.html
<tr class="row-odd"><td><p>builtins.breakpoint</p></td>
<td><p><code class="docutils literal notranslate"><span class="pre">breakpointhook</span></code></p></td>
<td><p><a class="reference internal" href="functions.html#breakpoint">[1]</a></p></td>
docsbuild@docs:/srv/docsbuild/3.9/cpython-en/Doc$ /srv/docsbuild/venv/bin/sphinx-build -b html -d build/doctrees -D latex_elements.papersize= -D latex_engine=xelatex -D latex_elements.inputenc= -D latex_elements.fontenc= -j4 -q -Ea -A daily=1 -A switchers=1 . build/html
docsbuild@docs:/srv/docsbuild/3.9/cpython-en/Doc$ grep breakpoint build/html/library/audit_events.html
docsbuild@docs:/srv/docsbuild/3.9/cpython-en/Doc$ /srv/docsbuild/venv/bin/sphinx-build -b html -d build/doctrees -D latex_elements.papersize= -D latex_engine=xelatex -D latex_elements.inputenc= -D latex_elements.fontenc= -q -Ea -A daily=1 -A switchers=1 . build/html
docsbuild@docs:/srv/docsbuild/3.9/cpython-en/Doc$ grep breakpoint build/html/library/audit_events.html
<tr class="row-odd"><td><p>builtins.breakpoint</p></td>
<td><p><code class="docutils literal notranslate"><span class="pre">breakpointhook</span></code></p></td>
<td><p><a class="reference internal" href="functions.html#breakpoint">[1]</a></p></td>

Bests,
-- 
Julien Palard
https://mdk.fr
_______________________________________________
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-leave@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/Q23O23HXTD7MSQGQW4Z5RO4XK5XYW2LZ/
Re: Docs: audit event table empty [ In reply to ]
On 09/07/2019 14.02, Julien Palard wrote:
> Hi Christian,
>
>> the table with auditing events does not render on docs.python.org,
>> https://docs.python.org/3.9/library/audit_events.html. Steve and I are
>> going to present the auditing feature tomorrow at EuroPython. It would
>> be helpful to have the table available.
>
> This was not an easy one... and it may be a Sphinx issue, yet I'm still not sure, maybe Steve can shed some light on it:
>
> It's the "-j" option of sphinx-build (to parallelize) that causes the issue. I double checked it (full commands at the end of the message in case someone want to reproduce it):
>
> - Run with -j4 ? No table
> - Run without -j ? Table is here
> - Run again with -j4 ? No table!
> - Run again without -j ? Table is back!
>
> I'm patching docsbuild-scripts to stop using -j4 with is not really helpfull anyway as docsbuild script is parallelizing by starting multiple sphinx-build (for multiple languages / versions).
>
> I also copied the file and invalidated the cache, so https://docs.python.org/3.9/library/audit_events.html is good again.
>
> If I'm too slow testing locally and releasing a new docsbuild_script.py, the cron MAY break the file again, don't hesitate to poke me if it happen without me noticing first.

Fantastic! It'll do for now. Thank you!

I'll follow up with you after EuroPython.

Christian
_______________________________________________
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-leave@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/6IW5QY7VRSILMJ5R475VSM25V7OAVQDB/
Re: Docs: audit event table empty [ In reply to ]
On Tue, Jul 9, 2019 at 5:34 PM Julien Palard via Python-Dev <
python-dev@python.org> wrote:

> Hi Christian,
>
> > the table with auditing events does not render on docs.python.org,
> > https://docs.python.org/3.9/library/audit_events.html. Steve and I are
> > going to present the auditing feature tomorrow at EuroPython. It would
> > be helpful to have the table available.
>
> This was not an easy one... and it may be a Sphinx issue, yet I'm still
> not sure, maybe Steve can shed some light on it:
>
> It's the "-j" option of sphinx-build (to parallelize) that causes the
> issue. I double checked it (full commands at the end of the message in case
> someone want to reproduce it):
>
> - Run with -j4 ? No table
> - Run without -j ? Table is here
> - Run again with -j4 ? No table!
> - Run again without -j ? Table is back!
>

I can reproduce the same with table not appearing in parallel builds and
table is successfully built on non-parallel builds. On adding -j4 to
ALLSPHINXOPTS in my Makefile in Docs/ folder there is no table generated.
Currently, the audit events are collected in a dictionary and are stored in
self.state.document.settings.env that is initialized with empty dict in
AuditEvent directive [0].

* On non-parallel builds when I use print statement with hasattr to check
for all_audit_events, it's False for first time and then remains True with
tuple set for rest of the collection. In process_audit_events for
non-parallel builds the attribute is set with a dictionary of all audit
events.
* On parallel builds when I use print statement with hasattr to check for
all_audit_events, it returns True and False randomly throughout the build.
I guess it's due to attribute not being properly initialized. In
process_audit_events for parallel builds the all_audit_events attribute is
not at all set during table creation and falls back to empty tuple writing
no rows to the table.

I am not sure if modifying the env attribute on parallel builds is safe
since I couldn't find any guarantees from the sphinx docs. The default docs
Makefile is not parallel which could be the reason it was not reproducible
locally.

Thanks for fixing this.

[0]
https://github.com/python/cpython/blob/73ea54620a6f91c3f2e53880373dd47813691a21/Doc/tools/extensions/pyspecific.py#L187


> I'm patching docsbuild-scripts to stop using -j4 with is not really
> helpfull anyway as docsbuild script is parallelizing by starting multiple
> sphinx-build (for multiple languages / versions).
>
> I also copied the file and invalidated the cache, so
> https://docs.python.org/3.9/library/audit_events.html is good again.
>
> If I'm too slow testing locally and releasing a new docsbuild_script.py,
> the cron MAY break the file again, don't hesitate to poke me if it happen
> without me noticing first.
>
> Full test:
>
> docsbuild@docs:/srv/docsbuild/3.9/cpython-en/Doc$
> /srv/docsbuild/venv/bin/sphinx-build -b html -d build/doctrees -D
> latex_elements.papersize= -D latex_engine=xelatex -D
> latex_elements.inputenc= -D latex_elements.fontenc= -j4 -q -Ea -A daily=1
> -A switchers=1 . build/html
> docsbuild@docs:/srv/docsbuild/3.9/cpython-en/Doc$ grep breakpoint
> build/html/library/audit_events.html
> docsbuild@docs:/srv/docsbuild/3.9/cpython-en/Doc$
> /srv/docsbuild/venv/bin/sphinx-build -b html -d build/doctrees -D
> latex_elements.papersize= -D latex_engine=xelatex -D
> latex_elements.inputenc= -D latex_elements.fontenc= -q -Ea -A daily=1 -A
> switchers=1 . build/html
> docsbuild@docs:/srv/docsbuild/3.9/cpython-en/Doc$ grep breakpoint
> build/html/library/audit_events.html
> <tr class="row-odd"><td><p>builtins.breakpoint</p></td>
> <td><p><code class="docutils literal notranslate"><span
> class="pre">breakpointhook</span></code></p></td>
> <td><p><a class="reference internal"
> href="functions.html#breakpoint">[1]</a></p></td>
> docsbuild@docs:/srv/docsbuild/3.9/cpython-en/Doc$
> /srv/docsbuild/venv/bin/sphinx-build -b html -d build/doctrees -D
> latex_elements.papersize= -D latex_engine=xelatex -D
> latex_elements.inputenc= -D latex_elements.fontenc= -j4 -q -Ea -A daily=1
> -A switchers=1 . build/html
> docsbuild@docs:/srv/docsbuild/3.9/cpython-en/Doc$ grep breakpoint
> build/html/library/audit_events.html
> docsbuild@docs:/srv/docsbuild/3.9/cpython-en/Doc$
> /srv/docsbuild/venv/bin/sphinx-build -b html -d build/doctrees -D
> latex_elements.papersize= -D latex_engine=xelatex -D
> latex_elements.inputenc= -D latex_elements.fontenc= -q -Ea -A daily=1 -A
> switchers=1 . build/html
> docsbuild@docs:/srv/docsbuild/3.9/cpython-en/Doc$ grep breakpoint
> build/html/library/audit_events.html
> <tr class="row-odd"><td><p>builtins.breakpoint</p></td>
> <td><p><code class="docutils literal notranslate"><span
> class="pre">breakpointhook</span></code></p></td>
> <td><p><a class="reference internal"
> href="functions.html#breakpoint">[1]</a></p></td>
>
> Bests,
> --
> Julien Palard
> https://mdk.fr
> _______________________________________________
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-leave@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/Q23O23HXTD7MSQGQW4Z5RO4XK5XYW2LZ/
>


--
Regards,
Karthikeyan S
Re: Docs: audit event table empty [ In reply to ]
On 7/9/2019 6:44 AM, Christian Heimes wrote:
> Hi,
>
> the table with auditing events does not render on docs.python.org,
> https://docs.python.org/3.9/library/audit_events.html. Steve and I are
> going to present the auditing feature tomorrow at EuroPython. It would
> be helpful to have the table available.
>
> It works on Steve's and my local machine without any issues. I suspect
> it's either an outdated sphinx version or caching issue. Could somebody
> from the docs team or shell access to the docs machine please look into
> the matter?

After they did so, this now works for me. Looks great.

--
Terry Jan Reedy
_______________________________________________
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-leave@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/JWFXUUO3PXX4ATLDKAPVGLWW7NH7FB6Y/
Re: Docs: audit event table empty [ In reply to ]
On 7/9/2019 3:53 PM, Terry Reedy wrote:

>> https://docs.python.org/3.9/library/audit_events.html. Steve and I are
>> going to present the auditing feature tomorrow at EuroPython.

That should be interesting. My experiment with the following

>>> def audit(event, args): print(event, args)

>>> import sys; sys.addaudithook(audit)

produced a line, on IDLE, like this:
builtins.id (1501408069568,)

builtins.id is not in the table. Should it be?



--
Terry Jan Reedy
_______________________________________________
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-leave@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/3WJUP7G5TGIGN22WSH2HFDHMIXDJ7AJ6/