Mailing List Archive

libpython added to ABI tracker
Hi,

The libpython library has been added to the ABI tracker:
http://upstream-tracker.org/versions/python.html

The page lists library versions and changes in API/ABI.

--
Andrey Ponomarenko, NTC IT ROSA.

_______________________________________________
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: https://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com
Re: libpython added to ABI tracker [ In reply to ]
Hi,

2014-03-28 9:31 GMT+01:00 Andrey Ponomarenko <aponomarenko@rosalab.ru>:
> The libpython library has been added to the ABI tracker:
> http://upstream-tracker.org/versions/python.html
>
> The page lists library versions and changes in API/ABI.

Nice!

By the way, would it be possible to add a second page for the stable
ABI of Python?

Victor
_______________________________________________
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: https://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com
Re: libpython added to ABI tracker [ In reply to ]
On Fri, 28 Mar 2014 12:31:59 +0400
Andrey Ponomarenko <aponomarenko@rosalab.ru> wrote:
> Hi,
>
> The libpython library has been added to the ABI tracker:
> http://upstream-tracker.org/versions/python.html
>
> The page lists library versions and changes in API/ABI.

Thanks. Do note that most of these changes are on private (internal)
APIs that normal user code should never invoke / rely on (for example
all those APIs starting with an underscore).

Regards

Antoine.


_______________________________________________
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: https://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com
Re: libpython added to ABI tracker [ In reply to ]
On 29 March 2014 02:25, Antoine Pitrou <solipsis@pitrou.net> wrote:
> On Fri, 28 Mar 2014 12:31:59 +0400
> Andrey Ponomarenko <aponomarenko@rosalab.ru> wrote:
>> Hi,
>>
>> The libpython library has been added to the ABI tracker:
>> http://upstream-tracker.org/versions/python.html
>>
>> The page lists library versions and changes in API/ABI.
>
> Thanks. Do note that most of these changes are on private (internal)
> APIs that normal user code should never invoke / rely on (for example
> all those APIs starting with an underscore).

That's where having the stable ABI tracked as well would really come
into its own, as any regressions in that *are* a big deal.

I also consider it a very good sign that the maintenance releases are
very stable from an ABI perspective - that's exactly what it supposed
to happen :)

Cheers,
Nick.
_______________________________________________
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: https://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com
Re: libpython added to ABI tracker [ In reply to ]
Hi,

Victor Stinner wrote:
> Hi,
>
> 2014-03-28 9:31 GMT+01:00 Andrey Ponomarenko <aponomarenko@rosalab.ru>:
>> The libpython library has been added to the ABI tracker:
>> http://upstream-tracker.org/versions/python.html
>>
>> The page lists library versions and changes in API/ABI.
> Nice!
>
> By the way, would it be possible to add a second page for the stable
> ABI of Python?
>
> Victor

The public libpython API changes will be tracked here:
http://upstream-tracker.org/versions/python_public_api.html

For now I've excluded only symbols starting with an underscore. What
other symbols should be excluded?

Thanks.

--
Andrey Ponomarenko, NTC IT ROSA.

_______________________________________________
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: https://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com
Re: libpython added to ABI tracker [ In reply to ]
2014-03-31 13:38 GMT+02:00 Andrey Ponomarenko <aponomarenko@rosalab.ru>:
> The public libpython API changes will be tracked here:
> http://upstream-tracker.org/versions/python_public_api.html
>
> For now I've excluded only symbols starting with an underscore. What other
> symbols should be excluded?

It's not a matter of underscore. You should define Py_LIMITED_API to
0x03020000 to test the stable ABI of Python 3.2.

http://docs.python.org/dev/c-api/stable.html

Victor
_______________________________________________
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: https://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com
Re: libpython added to ABI tracker [ In reply to ]
On 1 Apr 2014 01:38, "Victor Stinner" <victor.stinner@gmail.com> wrote:
>
> 2014-03-31 13:38 GMT+02:00 Andrey Ponomarenko <aponomarenko@rosalab.ru>:
> > The public libpython API changes will be tracked here:
> > http://upstream-tracker.org/versions/python_public_api.html
> >
> > For now I've excluded only symbols starting with an underscore. What
other
> > symbols should be excluded?
>
> It's not a matter of underscore. You should define Py_LIMITED_API to
> 0x03020000 to test the stable ABI of Python 3.2.
>
> http://docs.python.org/dev/c-api/stable.html

Well, we have more than one ABI, with different guarantees. The "no leading
underscore" one we promise not to change in maintenance releases, but we
only preserve *API* compatibility in feature releases (mostly due to
structs changing size).

The "stable ABI" (aka Py_LIMITED_API) is the one where we promise to hide
all the memory layout details and treat it as "additive only" so that
binaries built with previous releases keep working. That should never break
ABI compatibility, and only get new additions if the macro definition is
bumped up to match the newer release.

Cheers,
Nick.

P.S. I understand it was Anatoly that put the process in motion to get this
set up. Thanks for doing that Anatoly, it's a genuinely good idea.

>
> Victor
> _______________________________________________
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com
Re: libpython added to ABI tracker [ In reply to ]
Nick Coghlan wrote:
> On 1 Apr 2014 01:38, "Victor Stinner" <victor.stinner@gmail.com> wrote:
>> 2014-03-31 13:38 GMT+02:00 Andrey Ponomarenko <aponomarenko@rosalab.ru>:
>>> The public libpython API changes will be tracked here:
>>> http://upstream-tracker.org/versions/python_public_api.html
>>>
>>> For now I've excluded only symbols starting with an underscore. What
> other
>>> symbols should be excluded?
>> It's not a matter of underscore. You should define Py_LIMITED_API to
>> 0x03020000 to test the stable ABI of Python 3.2.
>>
>> http://docs.python.org/dev/c-api/stable.html
> Well, we have more than one ABI, with different guarantees. The "no leading
> underscore" one we promise not to change in maintenance releases, but we
> only preserve *API* compatibility in feature releases (mostly due to
> structs changing size).
>
> The "stable ABI" (aka Py_LIMITED_API) is the one where we promise to hide
> all the memory layout details and treat it as "additive only" so that
> binaries built with previous releases keep working. That should never break
> ABI compatibility, and only get new additions if the macro definition is
> bumped up to match the newer release.
>
> Cheers,
> Nick.
>
> P.S. I understand it was Anatoly that put the process in motion to get this
> set up. Thanks for doing that Anatoly, it's a genuinely good idea.

The stable libpython ABI with Py_LIMITED_API=0x03020000 will be tracked
at http://upstream-tracker.org/versions/python_stable_api.html

I also added source-compatibility reports to the "public" API tracker:
http://upstream-tracker.org/versions/python_public_api.html

Thanks.

--
Andrey Ponomarenko, NTC IT ROSA.

_______________________________________________
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: https://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com
Re: libpython added to ABI tracker [ In reply to ]
On 1 April 2014 21:23, Andrey Ponomarenko <aponomarenko@rosalab.ru> wrote:
>
> Nick Coghlan wrote:
>>
>> On 1 Apr 2014 01:38, "Victor Stinner" <victor.stinner@gmail.com> wrote:
>>>
>>> 2014-03-31 13:38 GMT+02:00 Andrey Ponomarenko <aponomarenko@rosalab.ru>:
>>>>
>>>> The public libpython API changes will be tracked here:
>>>> http://upstream-tracker.org/versions/python_public_api.html
>>>>
>>>> For now I've excluded only symbols starting with an underscore. What
>>
>> other
>>>>
>>>> symbols should be excluded?
>>>
>>> It's not a matter of underscore. You should define Py_LIMITED_API to
>>> 0x03020000 to test the stable ABI of Python 3.2.
>>>
>>> http://docs.python.org/dev/c-api/stable.html
>>
>> Well, we have more than one ABI, with different guarantees. The "no
>> leading
>> underscore" one we promise not to change in maintenance releases, but we
>> only preserve *API* compatibility in feature releases (mostly due to
>> structs changing size).
>>
>> The "stable ABI" (aka Py_LIMITED_API) is the one where we promise to hide
>> all the memory layout details and treat it as "additive only" so that
>> binaries built with previous releases keep working. That should never
>> break
>> ABI compatibility, and only get new additions if the macro definition is
>> bumped up to match the newer release.
>>
>> Cheers,
>> Nick.
>
>
> The stable libpython ABI with Py_LIMITED_API=0x03020000 will be tracked at
> http://upstream-tracker.org/versions/python_stable_api.html

Thanks! The "leading underscore means private" convention is also
applicable here (that's a general guideline for Python related APIs).

Interesting to see the UCS2 removal there for 3.3. That's a genuine
removal from the public ABI as part of PEP 393. I guess the reason
nobody complained is because most 3.2 Linux builds used the UCS4 ABI
instead, and the stable ABI hadn't seen broad adoption on Windows in
the 3.2->3.3 time frame.

Regarding the warnings for this one - is there a way for the checker
to warn if data structures are exposed directly, rather than as opaque
types? It's fine if there isn't, it would just be cool if there was -
one of the premises of the stable ABI is that it *doesn't* expose the
type definitions directly to consuming code, just the pointers to them
(hence allowing the struct size to change without actually breaking
compatibility with the defined ABI).

Regardless, this service already shows we've made some mistakes with
the stable ABI in previous releases - it is indicating there are new
symbols in the stable ABI for 3.3 and 3.4 that aren't properly guarded
with version constraints. That means it is currently possible to set
Py_LIMITED_API=0x03020000 and get something that won't actually run
properly on 3.2.

Georg, Larry, Benjamin - should checking these be added to PEP 101, so
we don't get the same thing happening for 3.5?

> I also added source-compatibility reports to the "public" API tracker:
> http://upstream-tracker.org/versions/python_public_api.html

Thanks again for setting these up!

Cheers,
Nick.

--
Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
_______________________________________________
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: https://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com
Re: libpython added to ABI tracker [ In reply to ]
On Tue, Apr 1, 2014, at 4:45, Nick Coghlan wrote:
> Georg, Larry, Benjamin - should checking these be added to PEP 101, so
> we don't get the same thing happening for 3.5?

I would like it if we could put this in the testsuite somehow.
_______________________________________________
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: https://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com
Re: libpython added to ABI tracker [ In reply to ]
On 2 Apr 2014 00:54, "Benjamin Peterson" <benjamin@python.org> wrote:
>
> On Tue, Apr 1, 2014, at 4:45, Nick Coghlan wrote:
> > Georg, Larry, Benjamin - should checking these be added to PEP 101, so
> > we don't get the same thing happening for 3.5?
>
> I would like it if we could put this in the testsuite somehow.

Perhaps an independent automated daily or weekly check like Antoine's daily
refleak hunter would be better?

Cheers,
Nick.
Re: libpython added to ABI tracker [ In reply to ]
On Tue, Apr 1, 2014, at 13:59, Nick Coghlan wrote:
> On 2 Apr 2014 00:54, "Benjamin Peterson" <benjamin@python.org> wrote:
> >
> > On Tue, Apr 1, 2014, at 4:45, Nick Coghlan wrote:
> > > Georg, Larry, Benjamin - should checking these be added to PEP 101, so
> > > we don't get the same thing happening for 3.5?
> >
> > I would like it if we could put this in the testsuite somehow.
>
> Perhaps an independent automated daily or weekly check like Antoine's
> daily
> refleak hunter would be better?

As long as it fails loudly.
_______________________________________________
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: https://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com
Re: libpython added to ABI tracker [ In reply to ]
Nick Coghlan wrote:
> Regarding the warnings for this one - is there a way for the checker
> to warn if data structures are exposed directly, rather than as opaque
> types? It's fine if there isn't, it would just be cool if there was -
> one of the premises of the stable ABI is that it *doesn't* expose the
> type definitions directly to consuming code, just the pointers to them
> (hence allowing the struct size to change without actually breaking
> compatibility with the defined ABI).

The ABI of the library can be dumped by the abi-compliance-checker basic
tool to a text file in the human readable format, so anyone can analyse
it in order to find problems of any kind.

Homepage of the tool: https://github.com/lvc/abi-compliance-checker

To dump the libpython ABI type:

$> abi-compliance-checker -l libpython -dump descriptor.xml

The descriptor.xml input file ({RELPATH} - path to the python install
tree, i.e. installation "prefix"):

<version>
3.4.0
</version>

<headers>
{RELPATH}/include
</headers>

<libs>
{RELPATH}/lib/libpython3.4m.so.1.0
</libs>

<skip_including>
ast.h
Python-ast.h
asdl.h
pyexpat.h
pymactoolbox.h
</skip_including>

<gcc_options>
-DPy_LIMITED_API=0x03020000
</gcc_options>

I've already created the sample dump of the libpython-3.4.0 stable ABI
here: http://upstream-tracker.org/downloads/libpython_3.4.0.abi.tar.gz

In order to analyse data types in the ABI please read 'TypeInfo' section
of the dump.

I see several structures with exposed definition in the stable ABI v3.4.0:

struct PyStructSequence_Desc
struct grammar
struct PyStructSequence_Field
struct _node
struct cchar_t
struct PyType_Spec
struct PyType_Slot
struct dfa
struct labellist
struct PyMethodDef
struct _Py_Identifier
struct state
struct PyVarObject
struct arc
struct label
struct PyModuleDef
struct PyModuleDef_Base
struct PyMemberDef
struct PyGetSetDef
struct _object
struct PyCursesWindowObject


Thanks.

--
Andrey Ponomarenko, NTC IT ROSA.

_______________________________________________
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: https://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com
Re: libpython added to ABI tracker [ In reply to ]
Am 01.04.14 13:45, schrieb Nick Coghlan:
> Interesting to see the UCS2 removal there for 3.3. That's a genuine
> removal from the public ABI as part of PEP 393. I guess the reason
> nobody complained is because most 3.2 Linux builds used the UCS4 ABI
> instead, and the stable ABI hadn't seen broad adoption on Windows in
> the 3.2->3.3 time frame.

Not really. The intention was that the stable ABI wouldn't have any
UCS2/UCS4 denotation in the function names, see

http://hg.python.org/cpython/file/9186f4a18584/PC/python3.def

Functions that explicitly referred to Py_UNICODE were banned from
the ABI; functions that were mapped but shouldn't have been mapped
were meant to be unmapped.

However, it seems that this wasn't properly implemented, see

http://bugs.python.org/issue17432

> Regardless, this service already shows we've made some mistakes with
> the stable ABI in previous releases - it is indicating there are new
> symbols in the stable ABI for 3.3 and 3.4 that aren't properly guarded
> with version constraints. That means it is currently possible to set
> Py_LIMITED_API=0x03020000 and get something that won't actually run
> properly on 3.2.

Depends on the operating system. On Windows, the import library has
much less additions; anything declared in the header files that
is not in python3.def is a bug in the header files (by default).

Regards,
Martin

_______________________________________________
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: https://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com
Re: libpython added to ABI tracker [ In reply to ]
On 3 April 2014 00:18, Andrey Ponomarenko <aponomarenko@rosalab.ru> wrote:
>
> Nick Coghlan wrote:
>>
>> Regarding the warnings for this one - is there a way for the checker
>> to warn if data structures are exposed directly, rather than as opaque
>> types? It's fine if there isn't, it would just be cool if there was -
>> one of the premises of the stable ABI is that it *doesn't* expose the
>> type definitions directly to consuming code, just the pointers to them
>> (hence allowing the struct size to change without actually breaking
>> compatibility with the defined ABI).
>
>
> The ABI of the library can be dumped by the abi-compliance-checker basic
> tool to a text file in the human readable format, so anyone can analyse it
> in order to find problems of any kind.

Thanks! I've now filed http://bugs.python.org/issue21142 as an RFE -
I'll definitely be looking into setting up a regular run of this to
help us avoid any new issues with managing the evolution of the stable
ABI (and the public ABI in general).

Cheers,
Nick.



--
Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
_______________________________________________
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: https://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com
Re: libpython added to ABI tracker [ In reply to ]
Nick Coghlan wrote:
> On 3 April 2014 00:18, Andrey Ponomarenko <aponomarenko@rosalab.ru> wrote:
>> Nick Coghlan wrote:
>>> Regarding the warnings for this one - is there a way for the checker
>>> to warn if data structures are exposed directly, rather than as opaque
>>> types? It's fine if there isn't, it would just be cool if there was -
>>> one of the premises of the stable ABI is that it *doesn't* expose the
>>> type definitions directly to consuming code, just the pointers to them
>>> (hence allowing the struct size to change without actually breaking
>>> compatibility with the defined ABI).
>>
>> The ABI of the library can be dumped by the abi-compliance-checker basic
>> tool to a text file in the human readable format, so anyone can analyse it
>> in order to find problems of any kind.
> Thanks! I've now filed http://bugs.python.org/issue21142 as an RFE -
> I'll definitely be looking into setting up a regular run of this to
> help us avoid any new issues with managing the evolution of the stable
> ABI (and the public ABI in general).

I wrote some instructions on how to setup local ABI checker in the
comment to issue 21142.

The main idea is to create ABI dump of the reference libpython version
(say, 3.2) and compare it with the ABI dump of the current version. The
ABI dump of the reference version can be stored somewhere in the source
tree.

Thanks.

--
Andrey Ponomarenko, NTC IT ROSA.

_______________________________________________
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: https://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com