Mailing List Archive

Porting c extension - PyBuffer_New() deprecated in python3. What's the replacement?
Hi

I need to convert the following existing c extension code to support
Python 3.

// --- existing code ------

// PyBuffer_New() deprecated in python3
if (!(pyBuf = PyBuffer_New(len)))
{
return NULL;
}

// should use memoryview object in python3
if (PyObject_AsWriteBuffer(pyBuf, &cbuf, &len))
{
Py_DECREF(pyBuf);
return NULL ;
}

// fill in cbuf
...

return pyBuf ;

//-----------

I'm somewhat confounded in finding an equivalent (PyBuffer_New()) for
creating a buffer of size len that has continuous memory in the c
extension function for python3. cbuf is manipulated/filled in using c,
after which the created pyBuf is then returned. So far, I haven't found
much in the way of examples/doc for porting the deprecated
Python-/C-level buffer API calls to the new C-level buffer
API/memoryview object model.

Any guidance or direction to existing doc/example is much appreciated.

TIA.




--
https://mail.python.org/mailman/listinfo/python-list
Re: Porting c extension - PyBuffer_New() deprecated in python3. What's the replacement? [ In reply to ]
Mark Heieis, 11.01.2014 06:47:
> I need to convert the following existing c extension code to support Python 3.
>
> // --- existing code ------
>
> // PyBuffer_New() deprecated in python3
> if (!(pyBuf = PyBuffer_New(len)))
> {
> return NULL;
> }
>
> // should use memoryview object in python3
> if (PyObject_AsWriteBuffer(pyBuf, &cbuf, &len))
> {
> Py_DECREF(pyBuf);
> return NULL ;
> }
>
> // fill in cbuf
> ...
>
> return pyBuf ;
>
> //-----------
>
> I'm somewhat confounded in finding an equivalent (PyBuffer_New()) for
> creating a buffer of size len that has continuous memory in the c extension
> function for python3. cbuf is manipulated/filled in using c, after which
> the created pyBuf is then returned. So far, I haven't found much in the way
> of examples/doc for porting the deprecated Python-/C-level buffer API calls
> to the new C-level buffer API/memoryview object model.
>
> Any guidance or direction to existing doc/example is much appreciated.

If the extension isn't huge, you should consider rewriting it in Cython.
That can usually be done quite quickly - the main thing is to figure out
what the verbose C code actually does and write it down in much simpler
Python code. And it will make it easy to make the code portable and fast.
Also likely safer and more generic and versatile, because Cython covers
away a lot of the annoying boilerplate, ref-counting issues, type
conversions, etc.

For your specific problem at hand, you could use Cython's memory views:

http://docs.cython.org/src/userguide/memoryviews.html

They allow you to convert the input value to a 1-dim char buffer (or
whatever you need, but you mentioned the old Py2 buffer interface, which
can't do much more) by saying

cdef char[:] my_memview = some_python_object

If you need to pass the unpacked buffer into C code, you can get the
address as "&my_memview[0]" (i.e. the address of the first item in the
buffer). Memory views themselves support fast slicing and indexing, so you
can efficiently work with them using the normal Python slicing/indexing syntax.

In case what you actually receive are not arbitrary buffers but simple byte
strings or bytearray instances, you can even use the normal byte string
coercion in Cython and simply say

cdef char* c_string = some_python_byte_string_object

and then use that pointer to pass it on into C.

I've written a string processing tutorial for Cython here:

http://docs.cython.org/src/tutorial/strings.html

These things may take a moment to learn, especially if you are used to
doing everything in excessive manual detail in C code, but once you are
through that, you should get things done much more quickly than when trying
to do them by hand.

Stefan


--
https://mail.python.org/mailman/listinfo/python-list
Re: Porting c extension - PyBuffer_New() deprecated in python3. What's the replacement? [ In reply to ]
Stefan,

Thank-you for the reply. I hadn't considered cpython, unfortunately the
extension is too large a project to port at the moment. I ended up
replacing the PyBuffer_New() segment with malloc() and passing back an
object from PyByteArray_FromStringAndSize(). It seems to work.

mrh.

On 2014-01-11 01:10, Stefan Behnel wrote:
> Mark Heieis, 11.01.2014 06:47:
>> I need to convert the following existing c extension code to support Python 3.
>>
>> // --- existing code ------
>>
>> // PyBuffer_New() deprecated in python3
>> if (!(pyBuf = PyBuffer_New(len)))
>> {
>> return NULL;
>> }
>>
>> // should use memoryview object in python3
>> if (PyObject_AsWriteBuffer(pyBuf, &cbuf, &len))
>> {
>> Py_DECREF(pyBuf);
>> return NULL ;
>> }
>>
>> // fill in cbuf
>> ...
>>
>> return pyBuf ;
>>
>> //-----------
>>
>> I'm somewhat confounded in finding an equivalent (PyBuffer_New()) for
>> creating a buffer of size len that has continuous memory in the c extension
>> function for python3. cbuf is manipulated/filled in using c, after which
>> the created pyBuf is then returned. So far, I haven't found much in the way
>> of examples/doc for porting the deprecated Python-/C-level buffer API calls
>> to the new C-level buffer API/memoryview object model.
>>
>> Any guidance or direction to existing doc/example is much appreciated.
> If the extension isn't huge, you should consider rewriting it in Cython.
> That can usually be done quite quickly - the main thing is to figure out
> what the verbose C code actually does and write it down in much simpler
> Python code. And it will make it easy to make the code portable and fast.
> Also likely safer and more generic and versatile, because Cython covers
> away a lot of the annoying boilerplate, ref-counting issues, type
> conversions, etc.
>
> For your specific problem at hand, you could use Cython's memory views:
>
> http://docs.cython.org/src/userguide/memoryviews.html
>
> They allow you to convert the input value to a 1-dim char buffer (or
> whatever you need, but you mentioned the old Py2 buffer interface, which
> can't do much more) by saying
>
> cdef char[:] my_memview = some_python_object
>
> If you need to pass the unpacked buffer into C code, you can get the
> address as "&my_memview[0]" (i.e. the address of the first item in the
> buffer). Memory views themselves support fast slicing and indexing, so you
> can efficiently work with them using the normal Python slicing/indexing syntax.
>
> In case what you actually receive are not arbitrary buffers but simple byte
> strings or bytearray instances, you can even use the normal byte string
> coercion in Cython and simply say
>
> cdef char* c_string = some_python_byte_string_object
>
> and then use that pointer to pass it on into C.
>
> I've written a string processing tutorial for Cython here:
>
> http://docs.cython.org/src/tutorial/strings.html
>
> These things may take a moment to learn, especially if you are used to
> doing everything in excessive manual detail in C code, but once you are
> through that, you should get things done much more quickly than when trying
> to do them by hand.
>
> Stefan
>
>

--
https://mail.python.org/mailman/listinfo/python-list