Mailing List Archive

On Wed, May 23, 2012 at 3:02 PM, Brett Cannon <brett@python.org> wrote:

> If I understand the proposal correctly, this would be a change in
> NamespaceLoader in how it sets __path__ and in no way affect any other code
> since __import__() just grabs the object on __path__ and passes as an
> argument to the meta path finders which just iterate over the object, so I
> have no objections to it.


That's not *quite* the proposal (but almost). The change would also mean
that __import__() instead passes a ModulePath (aka Nick's LazyIterable)
instance to the meta path finders, which just iterate over it. But other
than that, yes.

On Wed, May 23, 2012 at 3:35 PM, PJ Eby <pje@telecommunity.com> wrote:

> On Wed, May 23, 2012 at 3:02 PM, Brett Cannon <brett@python.org> wrote:
>
>> If I understand the proposal correctly, this would be a change in
>> NamespaceLoader in how it sets __path__ and in no way affect any other code
>> since __import__() just grabs the object on __path__ and passes as an
>> argument to the meta path finders which just iterate over the object, so I
>> have no objections to it.
>
>
> That's not *quite* the proposal (but almost). The change would also mean
> that __import__() instead passes a ModulePath (aka Nick's LazyIterable)
> instance to the meta path finders, which just iterate over it. But other
> than that, yes.
>

And why does __import__() need to construct that? I thought NamespaceLoader
was going to be making these "magical" __path__ objects that detected
changes and thus update themselves as necessary and just stick them on the
object. Why specifically does __import__() need to play a role?

On 05/23/2012 03:56 PM, Brett Cannon wrote:
>
>
> On Wed, May 23, 2012 at 3:35 PM, PJ Eby <pje@telecommunity.com
> <mailto:pje@telecommunity.com>> wrote:
>
> On Wed, May 23, 2012 at 3:02 PM, Brett Cannon <brett@python.org
> <mailto:brett@python.org>> wrote:
>
> If I understand the proposal correctly, this would be a change
> in NamespaceLoader in how it sets __path__ and in no way affect
> any other code since __import__() just grabs the object on
> __path__ and passes as an argument to the meta path finders
> which just iterate over the object, so I have no objections to it.
>
>
> That's not *quite* the proposal (but almost). The change would also
> mean that __import__() instead passes a ModulePath (aka Nick's
> LazyIterable) instance to the meta path finders, which just iterate
> over it. But other than that, yes.
>
>
> And why does __import__() need to construct that? I thought
> NamespaceLoader was going to be making these "magical" __path__ objects
> that detected changes and thus update themselves as necessary and just
> stick them on the object. Why specifically does __import__() need to
> play a role?

Assume that we're talking about importing either a top-level namespace
package named 'parent' and a nested namespace package parent.child.

The problem is that NamespaceLoader is just passed the parent path
(typically sys.path, but if a sub-package then parent.__path__). The
concern is that if the parent path object is replaced:
sys.path = sys.path + ['new-dir']
or
parent.__path__ = ['new-dir']
then the NamespaceLoader instance can no longer detect changes to
parent_path.

So the proposed solution is for NamespaceLoader to be told the name of
the parent module ('sys' or 'parent') and the attribute name to use to
find the path ('path' or '__path__').

Here's another suggestion: instead of modifying the finder/loader code
to pass these names through, assume that we can always find
(module_name, attribute_name) with this code:

def find_parent_path_names(module):
parent, dot, me = module.__name__.rpartition('.')
if dot == '':
return 'sys', 'path'
return parent, '__path__'

>>> import glob
>>> find_parent_path_names(glob)
('sys', 'path')
>>> import unittest.test.test_case
>>> find_parent_path_names(unittest.test.test_case)
('unittest.test', '__path__')

I guess it's a little more fragile than passing in these names to
NamespaceLoader, but it requires less code to change.

I think I'll whip this up in the pep-420 branch.

Eric.
_______________________________________________
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

> Here's another suggestion: instead of modifying the finder/loader code
> to pass these names through, assume that we can always find
> (module_name, attribute_name) with this code:
>
> def find_parent_path_names(module):
> parent, dot, me = module.__name__.rpartition('.')
> if dot == '':
> return 'sys', 'path'
> return parent, '__path__'
>
>>>> import glob
>>>> find_parent_path_names(glob)
> ('sys', 'path')
>>>> import unittest.test.test_case
>>>> find_parent_path_names(unittest.test.test_case)
> ('unittest.test', '__path__')
>
> I guess it's a little more fragile than passing in these names to
> NamespaceLoader, but it requires less code to change.
>
> I think I'll whip this up in the pep-420 branch.

I tried this approach and it works fine. The only caveat is that it
assumes that the parent path can always be computed as described above,
independent of what's passed in to PathFinder.load_module(). I think
that's reasonable, since load_module() itself hard-codes sys.path if the
supplied path is missing.

I've checked this in to the pep-420 branch. I prefer this approach over
Nick's because it doesn't require any changes to any existing
interfaces. The changes are contained to the namespace package code and
don't affect other code in importlib.

Assuming this approach is acceptable, I'm done with the PEP except for
adding some examples.

And I'm done with the implementation except for adding tests and a few
small tweaks.

Eric.
_______________________________________________
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

On Wed, May 23, 2012 at 8:24 PM, Eric V. Smith <eric@trueblade.com> wrote:

> I tried this approach and it works fine. The only caveat is that it
> assumes that the parent path can always be computed as described above,
> independent of what's passed in to PathFinder.load_module(). I think
> that's reasonable, since load_module() itself hard-codes sys.path if the
> supplied path is missing.
>

Technically, PEP 302 says that finders aren't allowed to assume their
parent packages are imported:

""" However, the find_module() method isn't necessarily always called
during an actual import: meta tools that analyze import dependencies (such
as freeze, Installer or py2exe) don't actually load modules, so a finder
shouldn't *depend* on the parent package being available in sys.modules."""

OTOH, that's finders, and I think we're dealing with loaders here.
Splitting hairs, perhaps, but at least it's in a good cause. ;-)


I've checked this in to the pep-420 branch. I prefer this approach over
> Nick's because it doesn't require any changes to any existing
> interfaces. The changes are contained to the namespace package code and
> don't affect other code in importlib.
>
> Assuming this approach is acceptable, I'm done with the PEP except for
> adding some examples.
>
> And I'm done with the implementation except for adding tests and a few
> small tweaks.
>

Yay!

On 5/23/2012 8:58 PM, PJ Eby wrote:
> On Wed, May 23, 2012 at 8:24 PM, Eric V. Smith <eric@trueblade.com
> <mailto:eric@trueblade.com>> wrote:
>
> I tried this approach and it works fine. The only caveat is that it
> assumes that the parent path can always be computed as described above,
> independent of what's passed in to PathFinder.load_module(). I think
> that's reasonable, since load_module() itself hard-codes sys.path if the
> supplied path is missing.
>
>
> Technically, PEP 302 says that finders aren't allowed to assume their
> parent packages are imported:
>
> """ However, the find_module() method isn't necessarily always called
> during an actual import: meta tools that analyze import dependencies
> (such as freeze, Installer or py2exe) don't actually load modules, so a
> finder shouldn't /depend/ on the parent package being available in
> sys.modules."""
>
> OTOH, that's finders, and I think we're dealing with loaders here.
> Splitting hairs, perhaps, but at least it's in a good cause. ;-)

I guess I could store the passed-in parent path, and use that if it
can't be found through sys.modules.

I'm not sure I can conjure up code to test this.

_______________________________________________
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

On Wed, May 23, 2012 at 9:02 PM, Eric V. Smith <eric@trueblade.com> wrote:

> On 5/23/2012 8:58 PM, PJ Eby wrote:
> > On Wed, May 23, 2012 at 8:24 PM, Eric V. Smith <eric@trueblade.com
> > <mailto:eric@trueblade.com>> wrote:
> >
> > I tried this approach and it works fine. The only caveat is that it
> > assumes that the parent path can always be computed as described
> above,
> > independent of what's passed in to PathFinder.load_module(). I think
> > that's reasonable, since load_module() itself hard-codes sys.path if
> the
> > supplied path is missing.
> >
> >
> > Technically, PEP 302 says that finders aren't allowed to assume their
> > parent packages are imported:
> >
> > """ However, the find_module() method isn't necessarily always called
> > during an actual import: meta tools that analyze import dependencies
> > (such as freeze, Installer or py2exe) don't actually load modules, so a
> > finder shouldn't /depend/ on the parent package being available in
> > sys.modules."""
> >
> > OTOH, that's finders, and I think we're dealing with loaders here.
> > Splitting hairs, perhaps, but at least it's in a good cause. ;-)
>
> I guess I could store the passed-in parent path, and use that if it
> can't be found through sys.modules.
>
> I'm not sure I can conjure up code to test this.
>

I actually was suggesting that we change PEP 302, if it became an issue.
;-)

On Thu, May 24, 2012 at 11:02 AM, Eric V. Smith <eric@trueblade.com> wrote:
> On 5/23/2012 8:58 PM, PJ Eby wrote:
>> OTOH, that's finders, and I think we're dealing with loaders here.
>> Splitting hairs, perhaps, but at least it's in a good cause.  ;-)
>
> I guess I could store the passed-in parent path, and use that if it
> can't be found through sys.modules.
>
> I'm not sure I can conjure up code to test this.

I don't think there's a need to change anything from your current
strategy, but we should be clear in the docs:

1. Finders should *not* assume their parent packages have been loaded
(and should not load them implicitly)
2. Loaders *can* assume their parent packages have already been loaded
and are present in sys.modules (and can complain if they're not there)

Cheers,
Nick.

--
Nick Coghlan   |   ncoghlan@gmail.com   |   Brisbane, Australia
_______________________________________________
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

I've reviewed the updates to the PEP and have accepted it. Congrats all!

I know the implementation is lagging behind a bit, that's not a
problem. Just get it into the next 3.3 alpha release!

--
--Guido van Rossum (python.org/~guido)
_______________________________________________
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

On 5/24/2012 2:33 PM, Guido van Rossum wrote:
> I've reviewed the updates to the PEP and have accepted it. Congrats all!

Thanks to the many people who helped: Martin, Barry, Guido, Jason, Nick,
PJE, and others. I'm sure I've offended someone by leaving them out, and
I apologize in advance.

But special thanks to Brett. Without his work on importlib, this never
would have happened (as Barry, Jason, and I demonstrated on a two or
three occasions)!

> I know the implementation is lagging behind a bit, that's not a
> problem. Just get it into the next 3.3 alpha release!

It's only missing a few small things. I'll get it committed in the next
day or so.

Eric.


_______________________________________________
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

On Thu, May 24, 2012 at 12:33 PM, Guido van Rossum <guido@python.org> wrote:
> I've reviewed the updates to the PEP and have accepted it. Congrats all!

Congrats!

-eric
_______________________________________________
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com