Mailing List Archive

importlib is now bootstrapped (and what that means)
My multi-year project -- started in 2006 according to my blog -- to rewrite
import in pure Python and then bootstrap it into CPython as *the*
implementation of __import__() is finally over (mostly)! Hopefully I didn't
break too much code in the process. =)

Now this is "mostly" finished because the single incompatibility that
importlib has is that it doesn't check the Windows registry for paths to
search since I lack a Windows installation to develop and test on. If
someone can tackle that issue that would be greatly appreciated (
http://bugs.python.org/issue14578).

Next up is how to maintain/develop for all of this. So the Makefile will
regenerate Python/importlib.h whenever Lib/importlib/_bootstrap.py or
Python/freeze_importlib.py changes. So if you make a change to importlib
make sure to get it working and tested before running 'make' again else you
will generate a bad frozen importlib (if you do mess up you can also revert
the changes to importlib.h and re-run make; a perk to having importlib.h
checked in). Otherwise keep in mind that you can't use any module that
isn't a builtin (sys.builtin_module_names) in importlib._bootstrap since
you can't import something w/o import working. =)

Where does this leave imp and Python/import.c? I want to make imp into _imp
and then implement as much as possible in pure Python (either in importlib
itself or in Lib/imp.py). Once that has happened then more C code in
import.c can be gutted (see http://bugs.python.org/issue13959 for tracking
this work which I will start piecemeal shortly).

I have some ideas on how to improve things for import, but I'm going to do
them as separate emails to have separate discussion threads on them so all
of this is easier to follow (e.g. actually following through on PEP 302 and
exposing the import machinery as importers instead of having anything be
implicit, etc.).

And the only outstanding point of contention in all of this is that some
people don't like having freeze_importlib.py in Python/ and instead want it
in Tools/. I didn't leave it in Tools/ as I have always viewed that Python
should build w/o the Tools directory, but maybe the Linux distros actually
ship with it and thus this is an unneeded worry. Plus the scripts to
generate the AST are in Parser so there is precedent for what I have done.

Anyway, I will write up the What's New entry and double-check the language
spec for updating once all of the potential changes I want to talk about in
other emails have been resolved.
Re: importlib is now bootstrapped (and what that means) [ In reply to ]
On 14.04.2012 20:12, Brett Cannon wrote:
> My multi-year project -- started in 2006 according to my blog -- to rewrite
> import in pure Python and then bootstrap it into CPython as *the* implementation
> of __import__() is finally over (mostly)! Hopefully I didn't break too much code
> in the process. =)
>
> Now this is "mostly" finished because the single incompatibility that importlib
> has is that it doesn't check the Windows registry for paths to search since I
> lack a Windows installation to develop and test on. If someone can tackle that
> issue that would be greatly appreciated (http://bugs.python.org/issue14578).
>
> Next up is how to maintain/develop for all of this. So the Makefile will
> regenerate Python/importlib.h whenever Lib/importlib/_bootstrap.py or
> Python/freeze_importlib.py changes. So if you make a change to importlib make
> sure to get it working and tested before running 'make' again else you will
> generate a bad frozen importlib (if you do mess up you can also revert the
> changes to importlib.h and re-run make; a perk to having importlib.h checked
> in). Otherwise keep in mind that you can't use any module that isn't a builtin
> (sys.builtin_module_names) in importlib._bootstrap since you can't import
> something w/o import working. =)

We've just now talked on IRC about this regeneration. Since both files --
_bootstrap.py and importlib.h -- are checked in, a make run can try to re-
generate importlib.h. This depends on the timestamps of the two files, which I
don't think Mercurial makes any guarantees about.

We have other instances of this (e.g. the Objects/typeslots.inc file is
generated and checked in), but in the case of importlib, we have to use the
./python binary for freezing to avoid bytecode incompatibilities, which
obviously is a problem if ./python isn't built yet.

> And the only outstanding point of contention in all of this is that some people
> don't like having freeze_importlib.py in Python/ and instead want it in Tools/.
> I didn't leave it in Tools/ as I have always viewed that Python should build w/o
> the Tools directory, but maybe the Linux distros actually ship with it and thus
> this is an unneeded worry. Plus the scripts to generate the AST are in Parser so
> there is precedent for what I have done.

I would have no objections to Python/. There is also e.g. Objects/typeslots.py.

Georg

_______________________________________________
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com
Re: importlib is now bootstrapped (and what that means) [ In reply to ]
Brett Cannon, 14.04.2012 20:12:
> My multi-year project -- started in 2006 according to my blog -- to rewrite
> import in pure Python and then bootstrap it into CPython as *the*
> implementation of __import__() is finally over (mostly)! Hopefully I didn't
> break too much code in the process. =)

Well, some at least.

The new import cache broke Cython's load of on-the-fly compiled extension
modules, which naively used "__import__(module_name)" after building them.
I could fix that by moving to "imp.load_dynamic()" (we know where we put
the compiled module anyway), although I just noticed that that's not
actually documented. So I hope that won't break later on.

The next thing I noticed is that the old-style level -1 import no longer
works, which presumably breaks a *lot* of Cython compiled modules. It used
to work in the master branch until two days ago, now it raises a
ValueError. We may be able to fix this by copying over CPython's old import
code into Cython, but I actually wonder if this was really intended. If
this feature wasn't deliberately broken in Py3.0, why break it now?

Stefan

_______________________________________________
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com
Re: importlib is now bootstrapped (and what that means) [ In reply to ]
On Mon, 16 Apr 2012 09:54:41 +0200
Stefan Behnel <stefan_ml@behnel.de> wrote:
>
> The new import cache broke Cython's load of on-the-fly compiled extension
> modules, which naively used "__import__(module_name)" after building them.
> I could fix that by moving to "imp.load_dynamic()" (we know where we put
> the compiled module anyway), although I just noticed that that's not
> actually documented. So I hope that won't break later on.

You can call importlib.invalidate_caches().
http://docs.python.org/dev/library/importlib.html#importlib.invalidate_caches

> The next thing I noticed is that the old-style level -1 import no longer
> works, which presumably breaks a *lot* of Cython compiled modules. It used
> to work in the master branch until two days ago, now it raises a
> ValueError. We may be able to fix this by copying over CPython's old import
> code into Cython, but I actually wonder if this was really intended. If
> this feature wasn't deliberately broken in Py3.0, why break it now?

Regressions should be reported on the bug tracker IMHO.

Regards

Antoine.


_______________________________________________
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com
Re: importlib is now bootstrapped (and what that means) [ In reply to ]
Antoine Pitrou, 16.04.2012 13:13:
> On Mon, 16 Apr 2012 09:54:41 +0200
> Stefan Behnel wrote:
>>
>> The new import cache broke Cython's load of on-the-fly compiled extension
>> modules, which naively used "__import__(module_name)" after building them.
>> I could fix that by moving to "imp.load_dynamic()" (we know where we put
>> the compiled module anyway), although I just noticed that that's not
>> actually documented. So I hope that won't break later on.
>
> You can call importlib.invalidate_caches().
> http://docs.python.org/dev/library/importlib.html#importlib.invalidate_caches

Well, yes, but imp.load_dynamic() would be the right thing to do for us. Is
there a reason why it's not documented?

I would like to avoid changing the code to load_dynamic() now and then
having to realise that that's going to die in 3.3 final because it somehow
got in the way of the importlob rewrites and is not being considered a
valuable enough public API.

New doc bug ticket:

http://bugs.python.org/issue14594


>> The next thing I noticed is that the old-style level -1 import no longer
>> works, which presumably breaks a *lot* of Cython compiled modules. It used
>> to work in the master branch until two days ago, now it raises a
>> ValueError. We may be able to fix this by copying over CPython's old import
>> code into Cython, but I actually wonder if this was really intended. If
>> this feature wasn't deliberately broken in Py3.0, why break it now?
>
> Regressions should be reported on the bug tracker IMHO.

It was meant as more of a question for now, but here it goes:

http://bugs.python.org/issue14592

Stefan

_______________________________________________
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com
Re: importlib is now bootstrapped (and what that means) [ In reply to ]
> We have other instances of this (e.g. the Objects/typeslots.inc file
> is generated and checked in), but in the case of importlib, we have
> to use the ./python binary for freezing to avoid bytecode
> incompatibilities, which obviously is a problem if ./python isn't
> built yet.

As for dependencies on byte code: we could consider using Cython instead
of freeze (not sure whether Cython would build the bootstrap correctly;
it may need to be fixed first). With that, we would get semi-readable
source code, which should also play more nicely with hg diffs. On the
down side, we would depend on Cython for evolving .

As for the timestamp issue: I think we should add a target "make touch"
or some such which checks whether the respective files are unmodified
in the local clone, and if so, arranges to touch generated files that
are older than their sources. If this is done by a plain shell script,
one could also make this a post-update Mercurial hook.

Regards,
Martin

_______________________________________________
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com
Re: importlib is now bootstrapped (and what that means) [ In reply to ]
On Mon, Apr 16, 2012 at 10:07, "Martin v. Löwis" <martin@v.loewis.de> wrote:

> > We have other instances of this (e.g. the Objects/typeslots.inc file
> > is generated and checked in), but in the case of importlib, we have
> > to use the ./python binary for freezing to avoid bytecode
> > incompatibilities, which obviously is a problem if ./python isn't
> > built yet.
>
> As for dependencies on byte code: we could consider using Cython instead
> of freeze (not sure whether Cython would build the bootstrap correctly;
> it may need to be fixed first). With that, we would get semi-readable
> source code, which should also play more nicely with hg diffs. On the
> down side, we would depend on Cython for evolving .
>

We could also just store the raw source code and use that if we are all
willing to pay the performance cost of parsing and compiling the code at
every startup.


>
> As for the timestamp issue: I think we should add a target "make touch"
> or some such which checks whether the respective files are unmodified
> in the local clone, and if so, arranges to touch generated files that
> are older than their sources. If this is done by a plain shell script,
> one could also make this a post-update Mercurial hook.
>

So like execute hg diff on the dependent files and if nothing changed then
touch the auto-generated file w/ 'touch' to prevent future attempts to
execute the target?

-Brett


>
> Regards,
> Martin
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev@python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> http://mail.python.org/mailman/options/python-dev/brett%40python.org
>
Re: importlib is now bootstrapped (and what that means) [ In reply to ]
On Mon, 16 Apr 2012 11:21:34 -0400, Brett Cannon <brett@python.org> wrote:
> On Mon, Apr 16, 2012 at 10:07, "Martin v. Löwis" <martin@v.loewis.de> wrote:
>
> > > We have other instances of this (e.g. the Objects/typeslots.inc file
> > > is generated and checked in), but in the case of importlib, we have
> > > to use the ./python binary for freezing to avoid bytecode
> > > incompatibilities, which obviously is a problem if ./python isn't
> > > built yet.
> >
> > As for dependencies on byte code: we could consider using Cython instead
> > of freeze (not sure whether Cython would build the bootstrap correctly;
> > it may need to be fixed first). With that, we would get semi-readable
> > source code, which should also play more nicely with hg diffs. On the
> > down side, we would depend on Cython for evolving .
> >
>
> We could also just store the raw source code and use that if we are all
> willing to pay the performance cost of parsing and compiling the code at
> every startup.

I don't see how depending on Cython is better than depending on having
an existing Python. If the only benefit is semi-readable code, surely
we do have source code for the pre-frozen module, and it is just a matter
of convincing hg that the bytecode is binary, not text?

Brett's earlier thought of compiling from source as a *fallback* makes
sense to me. I'd rather not add overhead to startup that we can avoid.

--David
Re: importlib is now bootstrapped (and what that means) [ In reply to ]
On Mon, 16 Apr 2012 12:15:16 -0400
"R. David Murray" <rdmurray@bitdance.com> wrote:
>
> I don't see how depending on Cython is better than depending on having
> an existing Python. If the only benefit is semi-readable code, surely
> we do have source code for the pre-frozen module, and it is just a matter
> of convincing hg that the bytecode is binary, not text?
>
> Brett's earlier thought of compiling from source as a *fallback* makes
> sense to me. I'd rather not add overhead to startup that we can avoid.

Compiling from source at which point, though?
In essence, that would mean reimplement Python/freeze_importlib.py in C?
We could even compile it to a separate executable that gets built
before the Python executable (like pgen) :-)

Regards

Antoine.


_______________________________________________
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com
Re: importlib is now bootstrapped (and what that means) [ In reply to ]
On Mon, Apr 16, 2012 at 12:31, Antoine Pitrou <solipsis@pitrou.net> wrote:

> On Mon, 16 Apr 2012 12:15:16 -0400
> "R. David Murray" <rdmurray@bitdance.com> wrote:
> >
> > I don't see how depending on Cython is better than depending on having
> > an existing Python. If the only benefit is semi-readable code, surely
> > we do have source code for the pre-frozen module, and it is just a matter
> > of convincing hg that the bytecode is binary, not text?
> >
> > Brett's earlier thought of compiling from source as a *fallback* makes
> > sense to me. I'd rather not add overhead to startup that we can avoid.
>
>
In reply to David, one trick with this, though, is that frozen modules
don't store the magic number of the bytecode, so that would need to change
in order to make this fully feasible.


> Compiling from source at which point, though?
>

At startup of the interpreter.


> In essence, that would mean reimplement Python/freeze_importlib.py in C?
> We could even compile it to a separate executable that gets built
> before the Python executable (like pgen) :-)
>

So a mini Python that just knew how to compile to bytecode and nothing more?

-Brett


>
> Regards
>
> Antoine.
>
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev@python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> http://mail.python.org/mailman/options/python-dev/brett%40python.org
>
Re: importlib is now bootstrapped (and what that means) [ In reply to ]
> I don't see how depending on Cython is better than depending on having
> an existing Python. If the only benefit is semi-readable code, surely
> we do have source code for the pre-frozen module, and it is just a matter
> of convincing hg that the bytecode is binary, not text?

Cython-generated C code would likely be more stable (and produce
compiler errors if it gets stale), whereas importlib.h needs to be
regenerated with byte code changes.

Having source code has the advantage that it becomes possible to
single-step through the import process in C debugger. Single-stepping
with pdb would, of course, be better than that, but I doubt it's
feasible.

In addition, there might be a performance gain with Cython over ceval.

Regards,
Martin

_______________________________________________
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com
Re: importlib is now bootstrapped (and what that means) [ In reply to ]
> So like execute hg diff on the dependent files and if nothing changed
> then touch the auto-generated file w/ 'touch' to prevent future attempts
> to execute the target?

Exactly. There might be something better than hg diff, perhaps some form
of hg status.

Regards,
Martin
_______________________________________________
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com
Re: importlib is now bootstrapped (and what that means) [ In reply to ]
On Mon, Apr 16, 2012 at 13:04, "Martin v. Löwis" <martin@v.loewis.de> wrote:

> > I don't see how depending on Cython is better than depending on having
> > an existing Python. If the only benefit is semi-readable code, surely
> > we do have source code for the pre-frozen module, and it is just a matter
> > of convincing hg that the bytecode is binary, not text?
>
> Cython-generated C code would likely be more stable (and produce
> compiler errors if it gets stale), whereas importlib.h needs to be
> regenerated with byte code changes.
>
> Having source code has the advantage that it becomes possible to
> single-step through the import process in C debugger. Single-stepping
> with pdb would, of course, be better than that, but I doubt it's
> feasible.
>
> In addition, there might be a performance gain with Cython over ceval.
>

The other benefit is maintainability. In order to hit my roughly 5% startup
speed I had to rewrite chunks of __import__() in C code and then delegate
to importlib's Python code in cases where sys.modules was not hit. Using
Cython would mean that can all go away and the differences between the C
and Python code would become (supposedly) non-existent, making tweaks
easier (e.g. when I made the change to hit sys.modules less when a loader
returned the desired module it was annoying to have to change importlib
*and* import.c).
Re: importlib is now bootstrapped (and what that means) [ In reply to ]
On Mon, Apr 16, 2012 at 13:08, "Martin v. Löwis" <martin@v.loewis.de> wrote:

> > So like execute hg diff on the dependent files and if nothing changed
> > then touch the auto-generated file w/ 'touch' to prevent future attempts
> > to execute the target?
>
> Exactly. There might be something better than hg diff, perhaps some form
> of hg status.
>

Yeah, hg status is probably better. Now someone just needs to write the
shell script. =)
Re: importlib is now bootstrapped (and what that means) [ In reply to ]
On Mon, 16 Apr 2012 13:33:45 -0400
Brett Cannon <brett@python.org> wrote:
> On Mon, Apr 16, 2012 at 13:08, "Martin v. Löwis" <martin@v.loewis.de> wrote:
>
> > > So like execute hg diff on the dependent files and if nothing changed
> > > then touch the auto-generated file w/ 'touch' to prevent future attempts
> > > to execute the target?
> >
> > Exactly. There might be something better than hg diff, perhaps some form
> > of hg status.
> >
>
> Yeah, hg status is probably better. Now someone just needs to write the
> shell script. =)

Wouldn't it be better if Python could compile regardless of the
presence of a hg repository?

Regards

Antoine.


_______________________________________________
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com
Re: importlib is now bootstrapped (and what that means) [ In reply to ]
On Mon, Apr 16, 2012 at 11:32 AM, Brett Cannon <brett@python.org> wrote:
>
>
> On Mon, Apr 16, 2012 at 13:04, "Martin v. Löwis" <martin@v.loewis.de> wrote:
>>
>> > I don't see how depending on Cython is better than depending on having
>> > an existing Python.  If the only benefit is semi-readable code, surely
>> > we do have source code for the pre-frozen module, and it is just a
>> > matter
>> > of convincing hg that the bytecode is binary, not text?
>>
>> Cython-generated C code would likely be more stable (and produce
>> compiler errors if it gets stale), whereas importlib.h needs to be
>> regenerated with byte code changes.
>>
>> Having source code has the advantage that it becomes possible to
>> single-step through the import process in C debugger. Single-stepping
>> with pdb would, of course, be better than that, but I doubt it's
>> feasible.
>>
>> In addition, there might be a performance gain with Cython over ceval.
>
>
> The other benefit is maintainability. In order to hit my roughly 5% startup
> speed I had to rewrite chunks of __import__() in C code and then delegate to
> importlib's Python code in cases where sys.modules was not hit. Using Cython
> would mean that can all go away and the differences between the C and Python
> code would become (supposedly) non-existent, making tweaks easier (e.g. when
> I made the change to hit sys.modules less when a loader returned the desired
> module it was annoying to have to change importlib *and* import.c).

+1 on reducing the complexity of the import code.

-eric
_______________________________________________
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com
Re: importlib is now bootstrapped (and what that means) [ In reply to ]
On Apr 16, 2012, at 07:44 PM, Antoine Pitrou wrote:

>Wouldn't it be better if Python could compile regardless of the
>presence of a hg repository?

If you want it in your $DISTRO, yes please!

-Barry
_______________________________________________
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com
Re: importlib is now bootstrapped (and what that means) [ In reply to ]
On Mon, 16 Apr 2012 13:51:35 -0400, Barry Warsaw <barry@python.org> wrote:
> On Apr 16, 2012, at 07:44 PM, Antoine Pitrou wrote:
>
> >Wouldn't it be better if Python could compile regardless of the
> >presence of a hg repository?
>
> If you want it in your $DISTRO, yes please!

My impression is that our usual solution for this is to make sure the
timestamps are correct in distributed tarballs, so that the hg-dependent
machinery is not invoked when building from a release tarball. Is this
case any different?

--David
_______________________________________________
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com
Re: importlib is now bootstrapped (and what that means) [ In reply to ]
"Martin v. Löwis", 16.04.2012 16:07:
>> We have other instances of this (e.g. the Objects/typeslots.inc file
>> is generated and checked in), but in the case of importlib, we have
>> to use the ./python binary for freezing to avoid bytecode
>> incompatibilities, which obviously is a problem if ./python isn't
>> built yet.
>
> As for dependencies on byte code: we could consider using Cython instead
> of freeze (not sure whether Cython would build the bootstrap correctly;
> it may need to be fixed first).

I assume that this would be needed rather early during the interpreter
startup, so you're right that there may be obstacles due to expectations in
the generated C code that may not be there yet. But I think it's worth a
try. Cython is rather predictable in its requirements given the Python
source code. Just run the module through Cython and see what it gives you.
(Note that you may need the latest github master version to make everything
work nicely in Py3.3, I keep adapting things there as I find them - our CI
server runs daily tests against the latest CPython master.)

One thing Cython does during module initialisation is to import the
builtins module using PyModule_AddModule(). Does that work already at this
point? It uses it mostly to cache builtins that the module uses. That can
be disabled, though, and I also wouldn't mind letting Cython put a
preprocessor guard into that code that would let it do something else based
on a macro that CPython defines at this point, maybe even just
Py_BUILD_CORE. We already do these things for all sorts of purposes.

And, obviously, the module init function can also be called directly
instead of running it as part of an import. That's commonly done when using
Cython modules in an embedded Python runtime.


> With that, we would get semi-readable
> source code, which should also play more nicely with hg diffs.

There's still a Cython patch hanging around on github that aims to keep the
C code from changing all that drastically on Python source code changes
(e.g. due to source line references etc.). Might be worth integrating for
something like this. There's also a switch that disables the (helpful)
reproduction of the Python code context in C code comments, in case that
gets in the way for diffs.


> On the down side, we would depend on Cython for evolving .

Right, although not as a strict dependency. The code would still work just
fine in plain Python. But it would depend on Cython for performance. And we
usually recommend to ship the generated C sources anyway to avoid a user
dependency on Cython, so this use case is quite normal.

Stefan

_______________________________________________
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com
Re: importlib is now bootstrapped (and what that means) [ In reply to ]
Hi,

2012/4/16 Stefan Behnel <stefan_ml@behnel.de>

> > On the down side, we would depend on Cython for evolving .
>
> Right, although not as a strict dependency. The code would still work just
> fine in plain Python.


Not quite, we are talking of the imp module here...

--
Amaury Forgeot d'Arc
Re: importlib is now bootstrapped (and what that means) [ In reply to ]
Am 16.04.2012 19:44, schrieb Antoine Pitrou:
> On Mon, 16 Apr 2012 13:33:45 -0400
> Brett Cannon <brett@python.org> wrote:
>> On Mon, Apr 16, 2012 at 13:08, "Martin v. Löwis" <martin@v.loewis.de> wrote:
>>
>>>> So like execute hg diff on the dependent files and if nothing changed
>>>> then touch the auto-generated file w/ 'touch' to prevent future attempts
>>>> to execute the target?
>>>
>>> Exactly. There might be something better than hg diff, perhaps some form
>>> of hg status.
>>>
>>
>> Yeah, hg status is probably better. Now someone just needs to write the
>> shell script. =)
>
> Wouldn't it be better if Python could compile regardless of the
> presence of a hg repository?

Who says it couldn't?

Regards,
Martin
_______________________________________________
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com
Re: importlib is now bootstrapped (and what that means) [ In reply to ]
On 16.04.2012 18:15, R. David Murray wrote:
> On Mon, 16 Apr 2012 11:21:34 -0400, Brett Cannon<brett@python.org> wrote:
>> On Mon, Apr 16, 2012 at 10:07, "Martin v. Löwis"<martin@v.loewis.de> wrote:
>>
>> > > We have other instances of this (e.g. the Objects/typeslots.inc file
>> > > is generated and checked in), but in the case of importlib, we have
>> > > to use the ./python binary for freezing to avoid bytecode
>> > > incompatibilities, which obviously is a problem if ./python isn't
>> > > built yet.
>> >
>> > As for dependencies on byte code: we could consider using Cython instead
>> > of freeze (not sure whether Cython would build the bootstrap correctly;
>> > it may need to be fixed first). With that, we would get semi-readable
>> > source code, which should also play more nicely with hg diffs. On the
>> > down side, we would depend on Cython for evolving .

That is an interesting idea. We already depend on external tools, e.g.
autotools, for updating checked-in files, so why not Cython.

>> We could also just store the raw source code and use that if we are all
>> willing to pay the performance cost of parsing and compiling the code at
>> every startup.
>
> I don't see how depending on Cython is better than depending on having
> an existing Python.

No, it's not just an existing Python, it is (at least currently) the same
version of Python being built. Therefore I wrote about the bootstrapping
problems when bytecode changes.

Depending on Cython is better in that it breaks the bootstrapping cycle,
but on the other hand the C code may need to be regenerated when the C API
changes in an incompatible way.

> If the only benefit is semi-readable code, surely
> we do have source code for the pre-frozen module, and it is just a matter
> of convincing hg that the bytecode is binary, not text?

The benefit is also (presumably) better performance.

Georg

_______________________________________________
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com
Re: importlib is now bootstrapped (and what that means) [ In reply to ]
On Tue, 17 Apr 2012 01:11:14 +0200
Georg Brandl <g.brandl@gmx.net> wrote:
>
> No, it's not just an existing Python, it is (at least currently) the same
> version of Python being built. Therefore I wrote about the bootstrapping
> problems when bytecode changes.
>
> Depending on Cython is better in that it breaks the bootstrapping cycle,
> but on the other hand the C code may need to be regenerated when the C API
> changes in an incompatible way.

Cython OTOH probably needs Python 2.x, which isn't that great for
building Python 3. And requiring Cython for developing is not very
contributor-friendly.

Regards

Antoine.


_______________________________________________
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com
Re: importlib is now bootstrapped (and what that means) [ In reply to ]
On Mon, Apr 16, 2012 at 20:27, Antoine Pitrou <solipsis@pitrou.net> wrote:

> On Tue, 17 Apr 2012 01:11:14 +0200
> Georg Brandl <g.brandl@gmx.net> wrote:
> >
> > No, it's not just an existing Python, it is (at least currently) the same
> > version of Python being built. Therefore I wrote about the bootstrapping
> > problems when bytecode changes.
> >
> > Depending on Cython is better in that it breaks the bootstrapping cycle,
> > but on the other hand the C code may need to be regenerated when the C
> API
> > changes in an incompatible way.
>
> Cython OTOH probably needs Python 2.x, which isn't that great for
> building Python 3. And requiring Cython for developing is not very
> contributor-friendly.
>

Well, required to regenerate _frozen_importlib, but nothing else. I mean
making fixes go into importlib directly and get tested that way, not
through __import__(). So really Cython would only be needed when
importlib._bootstrap has been changed and you are making a commit.

-Brett


>
> Regards
>
> Antoine.
>
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev@python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> http://mail.python.org/mailman/options/python-dev/brett%40python.org
>
Re: importlib is now bootstrapped (and what that means) [ In reply to ]
Amaury Forgeot d'Arc, 16.04.2012 22:43:
> 2012/4/16 Stefan Behnel
>>> On the down side, we would depend on Cython for evolving .
>>
>> Right, although not as a strict dependency. The code would still work just
>> fine in plain Python.
>
> Not quite, we are talking of the imp module here...

Hmm, right, after writing the above, I figured that it would at least have
to do something like "import sys" in order to deal with the import config
(path, meta path, ...). That obviously won't work in pure Python at this point.

Stefan

_______________________________________________
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

1 2  View All