Mailing List Archive

1 2  View All
Re: importlib is now bootstrapped (and what that means) [ In reply to ]
Antoine Pitrou, 17.04.2012 02:27:
> On Tue, 17 Apr 2012 01:11:14 +0200
> Georg Brandl wrote:
>> No, it's not just an existing Python, it is (at least currently) the same
>> version of Python being built. Therefore I wrote about the bootstrapping
>> problems when bytecode changes.
>>
>> Depending on Cython is better in that it breaks the bootstrapping cycle,
>> but on the other hand the C code may need to be regenerated when the C API
>> changes in an incompatible way.
>
> Cython OTOH probably needs Python 2.x, which isn't that great for
> building Python 3.

It uses 2to3 at install time, so you get a Py3 version out of it. No need
to have Py2 installed in order to use it.


> And requiring Cython for developing is not very
> contributor-friendly.

Brett Cannon answered that one. If you ship the C sources, developers will
only be impacted when they want to modify source code that gets compiled
with Cython.

Stefan

_______________________________________________
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com
Re: importlib is now bootstrapped (and what that means) [ In reply to ]
On Mon, 16 Apr 2012 20:41:56 -0400
Brett Cannon <brett@python.org> wrote:
> On Mon, Apr 16, 2012 at 20:27, Antoine Pitrou <solipsis@pitrou.net> wrote:
>
> > On Tue, 17 Apr 2012 01:11:14 +0200
> > Georg Brandl <g.brandl@gmx.net> wrote:
> > >
> > > No, it's not just an existing Python, it is (at least currently) the same
> > > version of Python being built. Therefore I wrote about the bootstrapping
> > > problems when bytecode changes.
> > >
> > > Depending on Cython is better in that it breaks the bootstrapping cycle,
> > > but on the other hand the C code may need to be regenerated when the C
> > API
> > > changes in an incompatible way.
> >
> > Cython OTOH probably needs Python 2.x, which isn't that great for
> > building Python 3. And requiring Cython for developing is not very
> > contributor-friendly.
> >
>
> Well, required to regenerate _frozen_importlib, but nothing else. I mean
> making fixes go into importlib directly and get tested that way, not
> through __import__(). So really Cython would only be needed when
> importlib._bootstrap has been changed and you are making a commit.

That's still a large dependency to bring in, while we already have a
working solution.
I'd understand using Cython to develop some new extension module which
requires linking against a C library (and is thus impossible to write
in pure Python). But for importlib that's totally non-necessary.

I guess I'm -1 on it.

Regards

Antoine.
_______________________________________________
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com
Re: importlib is now bootstrapped (and what that means) [ In reply to ]
On 4/17/2012 5:52 AM, Antoine Pitrou wrote:
> On Mon, 16 Apr 2012 20:41:56 -0400
> Brett Cannon <brett@python.org> wrote:
>> On Mon, Apr 16, 2012 at 20:27, Antoine Pitrou <solipsis@pitrou.net> wrote:
>>
>>> On Tue, 17 Apr 2012 01:11:14 +0200
>>> Georg Brandl <g.brandl@gmx.net> wrote:
>>>>
>>>> No, it's not just an existing Python, it is (at least currently) the same
>>>> version of Python being built. Therefore I wrote about the bootstrapping
>>>> problems when bytecode changes.
>>>>
>>>> Depending on Cython is better in that it breaks the bootstrapping cycle,
>>>> but on the other hand the C code may need to be regenerated when the C
>>> API
>>>> changes in an incompatible way.
>>>
>>> Cython OTOH probably needs Python 2.x, which isn't that great for
>>> building Python 3. And requiring Cython for developing is not very
>>> contributor-friendly.
>>>
>>
>> Well, required to regenerate _frozen_importlib, but nothing else. I mean
>> making fixes go into importlib directly and get tested that way, not
>> through __import__(). So really Cython would only be needed when
>> importlib._bootstrap has been changed and you are making a commit.
>
> That's still a large dependency to bring in, while we already have a
> working solution.
> I'd understand using Cython to develop some new extension module which
> requires linking against a C library (and is thus impossible to write
> in pure Python). But for importlib that's totally non-necessary.
>
> I guess I'm -1 on it.

I agree. If the problem we're trying to solve is that the generated file
isn't always rebuilt, bringing in a large dependency like Cython seems
like overkill to me.

We basically have a working solution now (thanks, Brett). I think we
should focus on getting it polished. Maybe we can bring in Cython in a
later release, if in the 3.4 timeframe it still seems like we have a
problem to solve. I suspect things will be working fine.

Eric.

_______________________________________________
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com
Re: importlib is now bootstrapped (and what that means) [ In reply to ]
On Tue, 17 Apr 2012 01:11:14 +0200, Georg Brandl <g.brandl@gmx.net> wrote:
> On 16.04.2012 18:15, R. David Murray wrote:
> > I don't see how depending on Cython is better than depending on having
> > an existing Python.
>
> No, it's not just an existing Python, it is (at least currently) the same
> version of Python being built. Therefore I wrote about the bootstrapping
> problems when bytecode changes.

Ah, yes, I had missed that subtlety.

--David
_______________________________________________
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com
Re: importlib is now bootstrapped (and what that means) [ In reply to ]
On Tue, Apr 17, 2012 at 06:43, Eric V. Smith <eric@trueblade.com> wrote:

> On 4/17/2012 5:52 AM, Antoine Pitrou wrote:
> > On Mon, 16 Apr 2012 20:41:56 -0400
> > Brett Cannon <brett@python.org> wrote:
> >> On Mon, Apr 16, 2012 at 20:27, Antoine Pitrou <solipsis@pitrou.net>
> wrote:
> >>
> >>> On Tue, 17 Apr 2012 01:11:14 +0200
> >>> Georg Brandl <g.brandl@gmx.net> wrote:
> >>>>
> >>>> No, it's not just an existing Python, it is (at least currently) the
> same
> >>>> version of Python being built. Therefore I wrote about the
> bootstrapping
> >>>> problems when bytecode changes.
> >>>>
> >>>> Depending on Cython is better in that it breaks the bootstrapping
> cycle,
> >>>> but on the other hand the C code may need to be regenerated when the C
> >>> API
> >>>> changes in an incompatible way.
> >>>
> >>> Cython OTOH probably needs Python 2.x, which isn't that great for
> >>> building Python 3. And requiring Cython for developing is not very
> >>> contributor-friendly.
> >>>
> >>
> >> Well, required to regenerate _frozen_importlib, but nothing else. I mean
> >> making fixes go into importlib directly and get tested that way, not
> >> through __import__(). So really Cython would only be needed when
> >> importlib._bootstrap has been changed and you are making a commit.
> >
> > That's still a large dependency to bring in, while we already have a
> > working solution.
> > I'd understand using Cython to develop some new extension module which
> > requires linking against a C library (and is thus impossible to write
> > in pure Python). But for importlib that's totally non-necessary.
> >
> > I guess I'm -1 on it.
>
> I agree. If the problem we're trying to solve is that the generated file
> isn't always rebuilt, bringing in a large dependency like Cython seems
> like overkill to me.
>

Actually Cython would help with a subtle maintenance burden of maintaining
*any* C code for import. Right now,
Python/import.c:PyImport_ImportModuleLevelObject() is an accelerated C
version of importlib.__import__() through checking sys.modules, after which
it calls into the Python code. Cython would do away with that C
acceleration code (which I have already had to modify once and Antoine
found a couple refleaks in).


>
> We basically have a working solution now (thanks, Brett). I think we
> should focus on getting it polished. Maybe we can bring in Cython in a
> later release, if in the 3.4 timeframe it still seems like we have a
> problem to solve. I suspect things will be working fine.


I don't view this discussion as work/not work but more of work/work better
(possibly; I have severe bias here to cut the C code down to zilch since I
don't want to write anymore of it so I'm definitely not going to make any
final call on this topic).
Re: importlib is now bootstrapped (and what that means) [ In reply to ]
On Tue, 17 Apr 2012 11:41:32 -0400
Brett Cannon <brett@python.org> wrote:
>
> Actually Cython would help with a subtle maintenance burden of maintaining
> *any* C code for import. Right now,
> Python/import.c:PyImport_ImportModuleLevelObject() is an accelerated C
> version of importlib.__import__() through checking sys.modules, after which
> it calls into the Python code. Cython would do away with that C
> acceleration code (which I have already had to modify once and Antoine
> found a couple refleaks in).

Would it? That's assuming Cython would be smart enough to do the
required optimizations.

Regards

Antoine.


_______________________________________________
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com
Re: importlib is now bootstrapped (and what that means) [ In reply to ]
On Tue, Apr 17, 2012 at 13:39, Antoine Pitrou <solipsis@pitrou.net> wrote:

> On Tue, 17 Apr 2012 11:41:32 -0400
> Brett Cannon <brett@python.org> wrote:
> >
> > Actually Cython would help with a subtle maintenance burden of
> maintaining
> > *any* C code for import. Right now,
> > Python/import.c:PyImport_ImportModuleLevelObject() is an accelerated C
> > version of importlib.__import__() through checking sys.modules, after
> which
> > it calls into the Python code. Cython would do away with that C
> > acceleration code (which I have already had to modify once and Antoine
> > found a couple refleaks in).
>
> Would it? That's assuming Cython would be smart enough to do the
> required optimizations.
>

Yes, it is an assumption I'm making. I also assume we wouldn't make a
change like this w/o taking the time to run importlib through Cython and
seeing how the performance numbers come out.

-Brett


>
> Regards
>
> Antoine.
>
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev@python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> http://mail.python.org/mailman/options/python-dev/brett%40python.org
>
Re: importlib is now bootstrapped (and what that means) [ In reply to ]
On 04/14/2012 02:12 PM, Brett Cannon wrote:
> My multi-year project -- started in 2006 according to my blog -- to
> rewrite import in pure Python and then bootstrap it into CPython as
> *the* implementation of __import__() is finally over (mostly)!

Maybe I'm missing something, but it seems that I need to run
importlib._bootstrap._install(sys, _imp) manually in order to make
__import__ be importlib's version. Is that not supposed to happen
automatically?

Eric.
_______________________________________________
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com
Re: importlib is now bootstrapped (and what that means) [ In reply to ]
On Fri, Apr 20, 2012 at 09:54, Eric V. Smith <eric@trueblade.com> wrote:

> On 04/14/2012 02:12 PM, Brett Cannon wrote:
> > My multi-year project -- started in 2006 according to my blog -- to
> > rewrite import in pure Python and then bootstrap it into CPython as
> > *the* implementation of __import__() is finally over (mostly)!
>
> Maybe I'm missing something, but it seems that I need to run
> importlib._bootstrap._install(sys, _imp) manually in order to make
> __import__ be importlib's version. Is that not supposed to happen
> automatically?


It's happening automatically. If you look in Python/import.c you will
notice that the code that __import__() eventually calls is calling out into
the Python code. There is still some C code in order to accelerate the case
of hitting sys.modules.
Re: importlib is now bootstrapped (and what that means) [ In reply to ]
On Fri, Apr 20, 2012 at 7:54 AM, Eric V. Smith <eric@trueblade.com> wrote:
> On 04/14/2012 02:12 PM, Brett Cannon wrote:
>> My multi-year project -- started in 2006 according to my blog -- to
>> rewrite import in pure Python and then bootstrap it into CPython as
>> *the* implementation of __import__() is finally over (mostly)!
>
> Maybe I'm missing something, but it seems that I need to run
> importlib._bootstrap._install(sys, _imp) manually in order to make
> __import__ be importlib's version. Is that not supposed to happen
> automatically?

In the default tip (3.3a2+), importlib.__import__ is already
bootstrapped, so you don't need mess with anything. As well, in any
of the 3.x versions you can bind builtins.__import__ to
importlib.__import__.

If you are making changes to importlib (essentially, changes in
Lib/importlib/_bootstrap.py), you must re-build (make) cpython in
order for your changes to get pulled into the frozen copy of
importlib. Until you do that, the built-in import machinery will be
the one that existed before your changes. You could also re-bind
builtins.__import__ to try out the changes without having to re-build,
but ultimately your changes will have to get frozen (into
Python/importlib.h) and will be part of the commit of your changes to
importlib.

Likely you already know all this, but just in case... :)

-eric
_______________________________________________
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com
Re: importlib is now bootstrapped (and what that means) [ In reply to ]
On Fri, Apr 20, 2012 at 11:02, Eric Snow <ericsnowcurrently@gmail.com>wrote:

> On Fri, Apr 20, 2012 at 7:54 AM, Eric V. Smith <eric@trueblade.com> wrote:
> > On 04/14/2012 02:12 PM, Brett Cannon wrote:
> >> My multi-year project -- started in 2006 according to my blog -- to
> >> rewrite import in pure Python and then bootstrap it into CPython as
> >> *the* implementation of __import__() is finally over (mostly)!
> >
> > Maybe I'm missing something, but it seems that I need to run
> > importlib._bootstrap._install(sys, _imp) manually in order to make
> > __import__ be importlib's version. Is that not supposed to happen
> > automatically?
>
> In the default tip (3.3a2+), importlib.__import__ is already
> bootstrapped, so you don't need mess with anything. As well, in any
> of the 3.x versions you can bind builtins.__import__ to
> importlib.__import__.
>
> If you are making changes to importlib (essentially, changes in
> Lib/importlib/_bootstrap.py), you must re-build (make) cpython in
> order for your changes to get pulled into the frozen copy of
> importlib. Until you do that, the built-in import machinery will be
> the one that existed before your changes. You could also re-bind
> builtins.__import__ to try out the changes without having to re-build,
> but ultimately your changes will have to get frozen (into
> Python/importlib.h) and will be part of the commit of your changes to
> importlib.
>
> Likely you already know all this, but just in case... :)


And if you want to run a test using importlib instead of the frozen code
you can use importlib.test.regrtest to handle the injection for you.
Re: importlib is now bootstrapped (and what that means) [ In reply to ]
On 04/20/2012 10:59 AM, Brett Cannon wrote:
>
>
> On Fri, Apr 20, 2012 at 09:54, Eric V. Smith <eric@trueblade.com
> <mailto:eric@trueblade.com>> wrote:
>
> On 04/14/2012 02:12 PM, Brett Cannon wrote:
> > My multi-year project -- started in 2006 according to my blog -- to
> > rewrite import in pure Python and then bootstrap it into CPython as
> > *the* implementation of __import__() is finally over (mostly)!
>
> Maybe I'm missing something, but it seems that I need to run
> importlib._bootstrap._install(sys, _imp) manually in order to make
> __import__ be importlib's version. Is that not supposed to happen
> automatically?
>
>
> It's happening automatically. If you look in Python/import.c you will
> notice that the code that __import__() eventually calls is calling out
> into the Python code. There is still some C code in order to accelerate
> the case of hitting sys.modules.

Okay. But I'm running make, and that's succeeding (and it looks like it
does the right thing), yet it doesn't appear to be picking up my changes
to _bootstrap.py automatically. I'll keep investigating.

Eric.
_______________________________________________
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

1 2  View All