Mailing List Archive

Interprocess comunication
I'm trying to call an external process to filter some of my data, i.e., I'm
trying to pass some information to the called process, and have this information
back transformed. I started testing with the linux 'cat' command, in this way:


->>> import subprocess as sp
->>> p = sp.Popen(["cat"],stdin=sp.PIPE,stdout=sp.PIPE,close_fds=True)
->>> (fi,fo) = (p.stdin, p.stdout)
->>> fi.write("SOMETHING\n")
->>> fi.flush()
->>> fo.readline()
'SOMETHING\n'
->>> fi.write("OTHER\n")
->>> fi.flush()
->>> fo.readline()
'OTHER\n'
->>> fi.write("NEXT\n")
->>> fi.flush()
->>> fo.readline()
'NEXT\n'
->>> fi.write("NEXT1\n")
->>> fi.flush()
->>> s = fo.readline()
->>> s
'NEXT1\n'

Up to this point it worked as expected. However, when I tryied with the methods
that write and read several lines, apparently the process got stalled:

->>> fi.writelines(["uno\n","dos\n","tres\n"])
->>> fi.flush()
->>> s = fo.readlines()
.
.
.

Do you have any comments on this?

Thanks,

--Sergio


--
http://mail.python.org/mailman/listinfo/python-list
Re: Interprocess comunication [ In reply to ]
On 7 June 2012 17:04, Julio Sergio <juliosergio@gmail.com> wrote:

> I'm trying to call an external process to filter some of my data, i.e., I'm
> trying to pass some information to the called process, and have this
> information
> back transformed. I started testing with the linux 'cat' command, in this
> way:
>
>
> ->>> import subprocess as sp
> ->>> p = sp.Popen(["cat"],stdin=sp.PIPE,stdout=sp.PIPE,close_fds=True)
> ->>> (fi,fo) = (p.stdin, p.stdout)
> ->>> fi.write("SOMETHING\n")
> ->>> fi.flush()
> ->>> fo.readline()
> 'SOMETHING\n'
> ->>> fi.write("OTHER\n")
> ->>> fi.flush()
> ->>> fo.readline()
> 'OTHER\n'
> ->>> fi.write("NEXT\n")
> ->>> fi.flush()
> ->>> fo.readline()
> 'NEXT\n'
> ->>> fi.write("NEXT1\n")
> ->>> fi.flush()
> ->>> s = fo.readline()
> ->>> s
> 'NEXT1\n'
>
> Up to this point it worked as expected. However, when I tryied with the
> methods
> that write and read several lines, apparently the process got stalled:
>
> ->>> fi.writelines(["uno\n","dos\n","tres\n"])
>
->>> fi.flush()
>
->>> s = fo.readlines()
>

The readlines() method is intended to read an entire file and split it into
lines, so it blocks until EOF is found. If you want to use readlines(), you
should first close the subprocesses stdin:

>>> fi.writelines(["uno\n","dos\n","tres\n"])
>>> fi.flush()
>>> fi.close()
>>> s = fo.readlines()

Although now you can no longer use the subprocess for reading or writing.
If that is not what you want in your actual problem then you'll need to
avoid readlines().


> .
> .
> .
>
> Do you have any comments on this?
>
> Thanks,
>
> --Sergio
>
>
> --
> http://mail.python.org/mailman/listinfo/python-list
>
Re: Interprocess comunication [ In reply to ]
On 07/06/2012 17:04, Julio Sergio wrote:
> I'm trying to call an external process to filter some of my data, i.e., I'm
> trying to pass some information to the called process, and have this information
> back transformed. I started testing with the linux 'cat' command, in this way:
>
>
> ->>> import subprocess as sp
> ->>> p = sp.Popen(["cat"],stdin=sp.PIPE,stdout=sp.PIPE,close_fds=True)
> ->>> (fi,fo) = (p.stdin, p.stdout)
> ->>> fi.write("SOMETHING\n")
> ->>> fi.flush()
> ->>> fo.readline()
> 'SOMETHING\n'
> ->>> fi.write("OTHER\n")
> ->>> fi.flush()
> ->>> fo.readline()
> 'OTHER\n'
> ->>> fi.write("NEXT\n")
> ->>> fi.flush()
> ->>> fo.readline()
> 'NEXT\n'
> ->>> fi.write("NEXT1\n")
> ->>> fi.flush()
> ->>> s = fo.readline()
> ->>> s
> 'NEXT1\n'
>
> Up to this point it worked as expected. However, when I tryied with the methods
> that write and read several lines, apparently the process got stalled:
>
> ->>> fi.writelines(["uno\n","dos\n","tres\n"])
> ->>> fi.flush()
> ->>> s = fo.readlines()
> .
> .
> .
>
> Do you have any comments on this?
>
I believe it's waiting for the end of the input, i.e. for the pipe to
close.

Have you tried calling fo.readline() 3 times instead?
--
http://mail.python.org/mailman/listinfo/python-list
Re: Interprocess comunication [ In reply to ]
On Thu, 2012-06-07 at 16:04 +0000, Julio Sergio wrote:

> Up to this point it worked as expected. However, when I tryied with the methods
> that write and read several lines, apparently the process got stalled:
>
> ->>> fi.writelines(["uno\n","dos\n","tres\n"])
> ->>> fi.flush()
> ->>> s = fo.readlines()
> .
> .
> .

readlines() reads all the lines from the filehandle, but the filehandle
hasn't signalled that it is done writing lines, so fo is waiting until
fi is complete. You either need to keep reading one line at a time, and
manually release control when there's nothing more to read, or you need
to do an fi.close() before you try to use fo.readlines().



--
http://mail.python.org/mailman/listinfo/python-list
Re: Interprocess comunication [ In reply to ]
MRAB <python <at> mrabarnett.plus.com> writes:

>
> I believe it's waiting for the end of the input, i.e. for the pipe to
> close.
>
> Have you tried calling fo.readline() 3 times instead?
>

yeah! It worked!...
A question remains: what is then the purpose of fo.readlines(...)?

Thanks,
--Sergio




--
http://mail.python.org/mailman/listinfo/python-list
Re: Interprocess comunication [ In reply to ]
J. Cliff Dyer <jcd <at> sdf.lonestar.org> writes:

>
> readlines() reads all the lines from the filehandle, but the filehandle
> hasn't signalled that it is done writing lines, so fo is waiting until
> fi is complete. You either need to keep reading one line at a time, and
> manually release control when there's nothing more to read, or you need
> to do an fi.close() before you try to use fo.readlines().
>
>

Thanks... It worked as you said. Here is how I coded it:

->>> fi.writelines(["uno\n","dos\n","tres\n"])
->>> fi.close()
->>> l = fo.readlines()
->>> l
['uno\n', 'dos\n', 'tres\n']


--Sergio

--
http://mail.python.org/mailman/listinfo/python-list
Re: Interprocess comunication [ In reply to ]
It is for reading all the lines from a complete file. If the file is
still being written to, it doesn't have an end yet. File objects do
many things besides RPC. Also, there are instances where all you want
to do is block until the file is done, and then get all the content.
readlines will do that, too.



On Thu, 2012-06-07 at 16:39 +0000, Julio Sergio wrote:
> MRAB <python <at> mrabarnett.plus.com> writes:
>
> >
> > I believe it's waiting for the end of the input, i.e. for the pipe to
> > close.
> >
> > Have you tried calling fo.readline() 3 times instead?
> >
>
> yeah! It worked!...
> A question remains: what is then the purpose of fo.readlines(...)?
>
> Thanks,
> --Sergio
>
>
>
>


--
http://mail.python.org/mailman/listinfo/python-list
Re: Interprocess comunication [ In reply to ]
Julio Sergio wrote:

> J. Cliff Dyer <jcd <at> sdf.lonestar.org> writes:
>
>>
>> readlines() reads all the lines from the filehandle, but the filehandle
>> hasn't signalled that it is done writing lines, so fo is waiting until
>> fi is complete. You either need to keep reading one line at a time, and
>> manually release control when there's nothing more to read, or you need
>> to do an fi.close() before you try to use fo.readlines().
>>
>>
>
> Thanks... It worked as you said. Here is how I coded it:
>
> ->>> fi.writelines(["uno\n","dos\n","tres\n"])
> ->>> fi.close()
> ->>> l = fo.readlines()
> ->>> l
> ['uno\n', 'dos\n', 'tres\n']

I believe this may hang on the fi.writelines(...) line if you write "too
much" data.

>From the subprocess documentation:

"""
Warning

Use communicate() rather than .stdin.write, .stdout.read or .stderr.read to
avoid deadlocks due to any of the other OS pipe buffers filling up and
blocking the child process.
"""

--
http://mail.python.org/mailman/listinfo/python-list