Mailing List Archive

Receiving large files with @tornado.web.stream_request_body
I was using tornado 3 before. I see that file streaming was included in
version 4. This is something I need for posting large files.

I have a problem receiving multiple files with a single POST request.
Client side could be something like this:

<form action="/postfiles">
First file: <input type="file" name="fname1"><br>
Second file: <input type="file" name="fname2"><br>
Third file: <input type="file" name="fname3"><br>
<input type="submit" value="Submit">
</form>

Suppose the user has seleted 3 large ISO files, I would like to stream
them into files on the server side. This is what I was trying:

@tornado.web.stream_request_body
class PostFilesHandler(tornado.web.RequestHandler):
def post(self):
for postfile in self.request.files:
print("File info:",postfile) # There is no
postfile["body"] here!!!

def prepare(self):
self.temp_file = tempfile.NamedTemporaryFile(delete=False)

def data_received(self, chunk):
self.temp_file.write(chunk) # This is great but which file is
this???


The problem is obvious: the data_received method receives raw data, but
how do I tell when data body for a file ends and another begins? How can
I save the body of all posted files into different named temp files, and
then access them from from the post() method?

The main reason for using this is to limit memory usage. Some files
might be GB sized and I don't want to store them in memory. Can please
someone point me to a working example?

Thanks

Laszlo


--
This message has been scanned for viruses and
dangerous content by MailScanner, and is
believed to be clean.

--
https://mail.python.org/mailman/listinfo/python-list
Re: Receiving large files with @tornado.web.stream_request_body [ In reply to ]
@tornado.web.stream_request_body
> class PostFilesHandler(tornado.web.RequestHandler):
> def post(self):
> for postfile in self.request.files:
> print("File info:",postfile) # There is no
> postfile["body"] here!!!
>
> def prepare(self):
> self.temp_file = tempfile.NamedTemporaryFile(delete=False)
>
> def data_received(self, chunk):
> self.temp_file.write(chunk) # This is great but which file is
> this???
>
>
I have found out that the raw post data is stored there. Extracting file
contents from that raw file is possible. However, the provisional
email.contentmanager class does not provide methods for extracting
streams from a mime message. Which is bad, because nobody wants to load
a huge mime message into memory (for example, a DVD iso file...)



--
This message has been scanned for viruses and
dangerous content by MailScanner, and is
believed to be clean.

--
https://mail.python.org/mailman/listinfo/python-list
Re: Receiving large files with @tornado.web.stream_request_body [ In reply to ]
2014.09.24. 16:31 keltezéssel, Nagy László Zsolt írta:
> @tornado.web.stream_request_body
>> class PostFilesHandler(tornado.web.RequestHandler):
>> def post(self):
>> for postfile in self.request.files:
>> print("File info:",postfile) # There is no
>> postfile["body"] here!!!
>>
>> def prepare(self):
>> self.temp_file = tempfile.NamedTemporaryFile(delete=False)
>>
>> def data_received(self, chunk):
>> self.temp_file.write(chunk) # This is great but which file is
>> this???
>>
>>
> I have found out that the raw post data is stored there. Extracting
> file contents from that raw file is possible. However, the provisional
> email.contentmanager class does not provide methods for extracting
> streams from a mime message. Which is bad, because nobody wants to
> load a huge mime message into memory (for example, a DVD iso file...)

I wrote my own solution in the meantime, that can parse any large POST
request into temporary files without using too much memory. If anyone is
interested I can post it here.


--
This message has been scanned for viruses and
dangerous content by MailScanner, and is
believed to be clean.

--
https://mail.python.org/mailman/listinfo/python-list