Mailing List Archive

1 2  View All
Re: What is a public API? [ In reply to ]
Steve Dower wrote:
> So I apologise for mentioning that people care about import performance.
> Let's ignore them/that issue for now and worry instead about making sure
> people (including us!) know what the canonical reference for
> public/internal is.

Good point, the discussion about __all__, adding the @public decorator, and worrying about performance impacts might be jumping too far ahead.

For now, if most of the core devs are in agreement with the current unwritten rule of "unless explicitly documented public, all imports are private even if not prefixed with an underscore", I think the first priority should be to document it officially somewhere. That way, other developers and any potential confused users can be referred to it.

It might not be the best long term solution, but it would require no code changes to be made and provide a written canonical reference for differentiating public vs private. I'm not certain as to where the most appropriate location for the rule would be, let me know if anyone has any suggestions.
_______________________________________________
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-leave@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/5XFBD2DECM234BYIEVCFQXCZP57S4FSY/
Re: What is a public API? [ In reply to ]
On Jul 23, 2019, at 12:02, Steve Dower <steve.dower@python.org> wrote:
>
> Even if the performance impact is zero, commits that span the entire codebase for not-very-impactful changes have a negative impact on readability (for example, someone will suddenly become responsible for every single module as far as some blame tools are concerned - including github's suggested reviewers). I'm inclined to think this one would be primarily negative.

If we were to adopt @public, its inclusion in the stdlib would follow the precedence we already have for non-functional changes (e.g. whitespace, code cleanup, etc.). It definitely shouldn’t be done willy nilly but if the opportunity arises, e.g. because someone is already fixing bugs or modernizing a module, then it would be fair game to add @public decorators. Of course, you can’t do that if it’s not available. :)

> We already maintain separate documentation from the source code, and this is the canonical reference for what is public or not. Until we make a new policy for __all__ to be the canonical reference, touching every file to use it is premature (let alone adding a builtin).

Agreed, sort of. We’ve had lots of cases of grey areas though, where the documentation doesn’t match the source. The question always becomes whether the source or the documentation is the source of truth. For any individual case, we don’t always come down on the same side of that question.

> So I apologise for mentioning that people care about import performance. Let's ignore them/that issue for now and worry instead about making sure people (including us!) know what the canonical reference for public/internal is.

+1

-Barry
Re: What is a public API? [ In reply to ]
Barry Warsaw wrote:
> On Jul 23, 2019, at 12:02, Steve Dower steve.dower@python.org wrote:
> > Even if the performance impact is zero, commits that
> > span the entire codebase for not-very-impactful changes have a negative impact on
> > readability (for example, someone will suddenly become responsible for every single module
> > as far as some blame tools are concerned - including github's suggested reviewers). I'm
> > inclined to think this one would be primarily negative.
> > If we were to adopt @public, its inclusion in the stdlib would follow the
> precedence we already have for non-functional changes (e.g. whitespace, code cleanup,
> etc.). It definitely shouldn’t be done willy nilly but if the opportunity arises, e.g.
> because someone is already fixing bugs or modernizing a module, then it would be fair game
> to add @public decorators. Of course, you can’t do that if it’s not available. :)
> > We already maintain separate documentation from the
> > source code, and this is the canonical reference for what is public or not. Until we make
> > a new policy for __all__ to be the canonical reference, touching every file to use it is
> > premature (let alone adding a builtin).
> > Agreed, sort of. We’ve had lots of cases of grey areas though, where the
> documentation doesn’t match the source. The question always becomes whether the source or
> the documentation is the source of truth. For any individual case, we don’t always come
> down on the same side of that question.
> > So I apologise for mentioning that people care about
> > import performance. Let's ignore them/that issue for now and worry instead about making
> > sure people (including us!) know what the canonical reference for public/internal is.
> > +1

Since this is for the stdlib, then PEP 8 would be the place to spell out how to denote public/private appropriately.
_______________________________________________
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-leave@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/Z7JMM7BB2YDP4FPU4WSLASCNQG2TXWR4/
Re: What is a public API? [ In reply to ]
> On Jul 13, 2019, at 1:56 PM, Serhiy Storchaka <storchaka@gmail.com> wrote:
>
> I thought that the name in a module is in the public interface if:
>
> * It doesn't start with an underscore and the module does not have __all__.
> * It is included in the module's __all__ list.
> * It is explicitly documented as a part of the public interface.
>
> help() uses more complex rules, but it believes __all__ if it defined.
>
> But seems there are different views on this.
>
> * Raymond suggested to add an underscore the two dozens of names in the calendar module not included in __all__.
> https://bugs.python.org/issue28292#msg347758
>
> I do not like this idea, because it looks like a code churn and makes the code less readable.
>
> * Gregory suggests to document codecs.escape_decode() despite it is not included in __all__.
> https://bugs.python.org/issue30588
>
> I do not like this idea, because this function always was internal, its only purpose was implementing the "string-escape" codec which was removed in Python 3 (for reasons). In Python 3 it is only used for supporting the old pickle protocol 0.
>
> Could we strictly define what is considered a public module interface in Python?

My apologies for not having read this very large thread before posting, but hopefully this small note won't be adding too much fuel to the fire:

Earlier this year I created an extremely small project called "publication" (https://pypi.org/project/publication/ <https://pypi.org/project/publication/>, https://github.com/glyph/publication <https://github.com/glyph/publication>) which attempts to harmonize the lofty ideal of "only the names explicitly mentioned in __all__ and in the documentation!" with the realpolitik of "anything I can reach where I don't have to type a leading underscore is fair game". It simply removes the ability to externally invoke non-__all__-exported names without importing an explicitly named "._private" namespace. It does not add any new syntactic idiom like a @public decorator (despite the aesthetic benefits of doing something like that) so that existing IDEs, type checkers, refactoring tools, code browsers etc can use the existing __all__ idiom and not break. It intentionally doesn't try hard to hide the implementation; it's still Python and if you demonstrate that you know what you're doing you're welcome to all the fiddly internals, it just makes sure you know that that's what you're getting.

While I am perhaps infamously a stdlib contrarian ;-) this project is a single module with extremely straightforward, stable semantics which I would definitely not mind being copy/pasted into the stdlib wholesale, either under a different (private, experimental) name, or even under its current one if folks like it. I'd be very pleased if this could solve the issue for the calendar module.

Thanks for reading,

-glyph
Re: What is a public API? [ In reply to ]
On Wed, 24 Jul 2019 at 06:32, Kyle Stanley <aeros167@gmail.com> wrote:
>
> Steve Dower wrote:
> > So I apologise for mentioning that people care about import performance.
> > Let's ignore them/that issue for now and worry instead about making sure
> > people (including us!) know what the canonical reference for
> > public/internal is.
>
> Good point, the discussion about __all__, adding the @public decorator, and worrying about performance impacts might be jumping too far ahead.
>
> For now, if most of the core devs are in agreement with the current unwritten rule of "unless explicitly documented public, all imports are private even if not prefixed with an underscore", I think the first priority should be to document it officially somewhere. That way, other developers and any potential confused users can be referred to it.

It's not an unwritten rule, as it already has its own subsection in
PEP 8: https://www.python.org/dev/peps/pep-0008/#public-and-internal-interfaces

The main question in this thread is what to do about standard library
modules that were written before those documented guidelines were put
in place, and hence have multiple internal APIs that lack the leading
underscore (and I don't think that's a question with a generic
answer).

Cheers,
Nick.

--
Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
_______________________________________________
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-leave@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/6TNG7J4QEOBCVLGSZ6VBOKKQQBHKPZPF/
Re: What is a public API? [ In reply to ]
On 16.07.19 00:32, Barry Warsaw wrote:
> On Jul 13, 2019, at 19:09, Raymond Hettinger <raymond.hettinger@gmail.com> wrote:
>>
>> In some modules, we've been careful to use both __all__ and to use an underscore prefix to indicate private variables and helper functions (collections and random for example). IMO, when a module has shown that care, future maintainers should stick with that practice.
>>
>> The calendar module is an example of where that care was taken for many years and then a recent patch went against that practice. This came to my attention when an end-user questioned which functions were for internal use only and posted their question on Twitter. On the tracker, I then made a simple request to restore the module's convention but you seem steadfastly resistant to the suggestion.
>>
>> When we do have evidence of user confusion (as in the case with the calendar module), we should just fix it. IMO, it would be an undue burden on the user to have to check every method in dir() against the contents of __all__ to determine what is public (see below). Also, as a maintainer of the module, I would not have found it obvious whether the functions were public or not. The non-public functions look just like the public ones.
>>
>> It's true that the practices across the standard library have historically been loose and varied (__all__ wasn't always used and wasn't always kept up-to-date, some modules took care with private underscore names and some didn't). To me this has mostly worked out fine and didn't require a strict rule for all modules everywhere. IMO, there is no need to sweep through the library and change long-standing policies on existing modules.
>
> EIBTI <wink>
>
> Shameless plug: https://public.readthedocs.io/en/latest/


Hey, what a fantastic little module!
I'll hurry and use that a lot! Especially the builtins idea
is really great :-P

Cheers - Chris


p.s.: How about adding @private as well?
There are cases where I would like to do the opposite:

__all__ = dir()

@private
_some_private_func_1(...): ...
...
@private
_some_private_func_n(...): ...

not-too-seriously yours - Chris
--
Christian Tismer :^) tismer@stackless.com
Software Consulting : http://www.stackless.com/
Karl-Liebknecht-Str. 121 : https://github.com/PySide
14482 Potsdam : GPG key -> 0xFB7BEE0E
phone +49 173 24 18 776 fax +49 (30) 700143-0023
Re: What is a public API? [ In reply to ]
Interesting idea!

https://gitlab.com/warsaw/public/issues/3

-Barry

> On Aug 9, 2019, at 09:55, Christian Tismer <tismer@stackless.com> wrote:
>
> Signed PGP part
> On 16.07.19 00:32, Barry Warsaw wrote:
>> On Jul 13, 2019, at 19:09, Raymond Hettinger <raymond.hettinger@gmail.com> wrote:
>>>
>>> In some modules, we've been careful to use both __all__ and to use an underscore prefix to indicate private variables and helper functions (collections and random for example). IMO, when a module has shown that care, future maintainers should stick with that practice.
>>>
>>> The calendar module is an example of where that care was taken for many years and then a recent patch went against that practice. This came to my attention when an end-user questioned which functions were for internal use only and posted their question on Twitter. On the tracker, I then made a simple request to restore the module's convention but you seem steadfastly resistant to the suggestion.
>>>
>>> When we do have evidence of user confusion (as in the case with the calendar module), we should just fix it. IMO, it would be an undue burden on the user to have to check every method in dir() against the contents of __all__ to determine what is public (see below). Also, as a maintainer of the module, I would not have found it obvious whether the functions were public or not. The non-public functions look just like the public ones.
>>>
>>> It's true that the practices across the standard library have historically been loose and varied (__all__ wasn't always used and wasn't always kept up-to-date, some modules took care with private underscore names and some didn't). To me this has mostly worked out fine and didn't require a strict rule for all modules everywhere. IMO, there is no need to sweep through the library and change long-standing policies on existing modules.
>>
>> EIBTI <wink>
>>
>> Shameless plug: https://public.readthedocs.io/en/latest/
>
>
> Hey, what a fantastic little module!
> I'll hurry and use that a lot! Especially the builtins idea
> is really great :-P
>
> Cheers - Chris
>
>
> p.s.: How about adding @private as well?
> There are cases where I would like to do the opposite:
>
> __all__ = dir()
>
> @private
> _some_private_func_1(...): ...
> ...
> @private
> _some_private_func_n(...): ...
>
> not-too-seriously yours - Chris
> --
> Christian Tismer :^) tismer@stackless.com
> Software Consulting : http://www.stackless.com/
> Karl-Liebknecht-Str. 121 : https://github.com/PySide
> 14482 Potsdam : GPG key -> 0xFB7BEE0E
> phone +49 173 24 18 776 fax +49 (30) 700143-0023
>
>
>
Re: What is a public API? [ In reply to ]
Nick Coghlan wrote:
> It's not an unwritten rule, as it already has its own subsection in
> PEP 8: https://www.python.org/dev/peps/pep-0008/#public-and-internal-interfaces
> The main question in this thread is what to do about standard library
> modules that were written before those documented guidelines were put
> in place, and hence have multiple internal APIs that lack the leading
> underscore (and I don't think that's a question with a generic
> answer).
> Cheers,
> Nick.

Oh I see, thanks for the clarification. I've read over most of PEP 8 a few times at this point, but somehow I missed this part: "All undocumented
interfaces should be assumed to be internal". Apologies for that.

Personally, I think the stdlib modules which were written before the rule
should be gradually updated, with each one being it's own issue with
the respective experts for each of the modules carefully monitoring the changes.

It would also be appropriate to provide any user attempting to import
a module that is going to be prepended with an underscore with
warnings, and at least a couple of versions to update their code.
_______________________________________________
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-leave@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/ICAXVGLOHF7H3GSL7OONI3Y6XDBBB54Z/
Re: What is a public API? [ In reply to ]
Kyle Stanley wrote:
> It would also be appropriate to provide any user attempting to import
> a module that is going to be prepended with an underscore with
> warnings, and at least a couple of versions to update their code.

Clarification: When I mentioned prepending a module with an
underscore, I meant for functions and classes within the module,
not the module itself.

It might be difficult to implement this in a way which does
not cause an excessive number of warnings, but I think it's
definitely worthwhile to aim towards having a fully consistent
standard for differentiating public and private interfaces
across all of stdlib.
_______________________________________________
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-leave@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/2O7PZB5BGRUPPTPH4Q4MQIXUWT734BQY/

1 2  View All