Mailing List Archive

[issue14310] Socket duplication for windows
New submission from Kristján Valur Jónsson <kristjan@ccpgames.com>:

winsock natively supports the duplciation of sockets for use on different processes. This patch proposes to add that functionality:
socket.duplicate(target_pid) gets a bytes representation of the duplicate socket, usable for the target process.
socket.socket(x,y,z,data) creates the socket object from the bytes representation.

The patch contains a test using multiprocessing that works.
Note that multiprocessing already contains its own code to achieve this, and that code can possibly be simplified with this patch.

Note also the new function "duplicate." Perhaps "dup(target_pid) would be better? But it would have different semantics.
Also notice how we overload the "fromfd" parameter in socket.socket() to recreate a socket from a bytes representation. Maybe this is not ideal?
Looking for thoughs here.

----------
components: Interpreter Core
files: duplicate.patch
keywords: patch
messages: 155842
nosy: krisvale
priority: normal
severity: normal
status: open
title: Socket duplication for windows
type: enhancement
versions: Python 3.4
Added file: http://bugs.python.org/file24854/duplicate.patch

_______________________________________
Python tracker <report@bugs.python.org>
<http://bugs.python.org/issue14310>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com
[issue14310] Socket duplication for windows [ In reply to ]
Changes by Giampaolo Rodola' <g.rodola@gmail.com>:


----------
nosy: +giampaolo.rodola

_______________________________________
Python tracker <report@bugs.python.org>
<http://bugs.python.org/issue14310>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com
[issue14310] Socket duplication for windows [ In reply to ]
sbt <shibturn@gmail.com> added the comment:

It appears that the 4th argument of the socket constructor is undocumented, so presumably one is expected to use fromfd() instead.

Maybe you could have a frominfo(info) function (to match fromfd(fd,...)) and a dupinfo(pid) method.

(It appears that multiprocessing uses DuplicateHandle() instead of WSADuplicateSocket() for duplicating socket handles on Windows. That should be fixed.)

----------
nosy: +sbt

_______________________________________
Python tracker <report@bugs.python.org>
<http://bugs.python.org/issue14310>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com
[issue14310] Socket duplication for windows [ In reply to ]
Kristján Valur Jónsson <kristjan@ccpgames.com> added the comment:

I'm adding an updated version of the patch, taking into account issues raised from reviewers. No documentation yet.

----------
Added file: http://bugs.python.org/file24940/duplicate.patch

_______________________________________
Python tracker <report@bugs.python.org>
<http://bugs.python.org/issue14310>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com
[issue14310] Socket duplication for windows [ In reply to ]
Kristján Valur Jónsson <kristjan@ccpgames.com> added the comment:

I like the idea of having a frominfo() function. Though "info" is maybe too generic? Also, dupinfo() is probably better than duplicate, even though there is no dup involved.

But I also wonder if we can make this windows/unix agnostic?
on unix, you would achieve the same thing by:
1) getting fd with fileno()
2) sending it over a unix domain socket using sendmsg() (used for duplicating fds betwerrn processes
3) calling fromfd() on the target host.

We could have a uniform process (more or less) if dupinfo(pid) would return a string containing the fileno on linux, and frominfo() were to accept such a string too. The differeince between linux/windows would then be that linux needs to transfer the "info" between processes using the "magic" sendmsg() control info ona UNIX domain sock, whereas on Windows one can use whatever RPC one wishes. To implement this, we could therefore have special "sendinfo" and "recvinfo" functions.

The process would then become, on either platform:
1) info = dupinfo(pid)
2) sendinfo(info)/info=recvinfo()
3) frominfo(info)

But again: "info" is very generic here. any suggestions for a better name?

----------

_______________________________________
Python tracker <report@bugs.python.org>
<http://bugs.python.org/issue14310>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com
[issue14310] Socket duplication for windows [ In reply to ]
Kristján Valur Jónsson <kristjan@ccpgames.com> added the comment:

On second thought, sendinfo/recvinfo makes no sense. On unix, one has to create and manage the necessary unix domain sockets, and such code belongs higher up somewhere. So dupinfo(pid), and frominfo() I'll add.

----------

_______________________________________
Python tracker <report@bugs.python.org>
<http://bugs.python.org/issue14310>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com
[issue14310] Socket duplication for windows [ In reply to ]
Kristján Valur Jónsson <kristjan@ccpgames.com> added the comment:

Here is a new version of thepatch. We now have socket.socket.share(pid) and socket.fromshare(data) to do these.
I think this captures the functionality better than "duplicate" or duppid() since there is no actual duplication involved until the fromshare() function is called.

Still no documentation. What do you think?

----------
Added file: http://bugs.python.org/file24944/duplicate.patch

_______________________________________
Python tracker <report@bugs.python.org>
<http://bugs.python.org/issue14310>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com
[issue14310] Socket duplication for windows [ In reply to ]
sbt <shibturn@gmail.com> added the comment:

> I think this captures the functionality better than "duplicate" or
> duppid() since there is no actual duplication involved until the
> fromshare() function is called.

Are you saying the WSADuplicateSocket() call in share() doesn't duplicate the handle in to the target process? I am pretty sure it does.

(Delaying handle duplication till WSASocket() is called in the target process would be rather problematic since then you cannot close the original socket in the source processes until you know the duplication has occurred.)

----------

_______________________________________
Python tracker <report@bugs.python.org>
<http://bugs.python.org/issue14310>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com
[issue14310] Socket duplication for windows [ In reply to ]
Kristján Valur Jónsson <kristjan@ccpgames.com> added the comment:

Good point. The docs don't actually say, except that the docs show a pretty elaborate process dance, where the source process only closes its side of the socket once the target process has called WSASocket().

If duplication happened early, then there would have to be a way to "unduplicate" it in the source process if, say, IPC somehow failed. There is currently no api to undo the effects of WSADuplicateSocket().

It's easy enough to test, of course. I´ll do some test using my unittest.

----------

_______________________________________
Python tracker <report@bugs.python.org>
<http://bugs.python.org/issue14310>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com
[issue14310] Socket duplication for windows [ In reply to ]
sbt <shibturn@gmail.com> added the comment:

> If duplication happened early, then there would have to be a way to
> "unduplicate" it in the source process if, say, IPC somehow failed.
> There is currently no api to undo the effects of WSADuplicateSocket().

If this were a normal handle then you could use the DUPLICATE_CLOSE_SOURCE flag with DuplicateHandle() to close it. But using DuplicateHandle() with socket handles is discouraged.

I find the ability to duplicate and close handles in unrelated processes of the same user rather surprising.

----------

_______________________________________
Python tracker <report@bugs.python.org>
<http://bugs.python.org/issue14310>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com
[issue14310] Socket duplication for windows [ In reply to ]
Kristján Valur Jónsson <kristjan@ccpgames.com> added the comment:

A new patch with fixes and documentation.
And yes, socket.share() does in fact perform duplication, because socket.close() can be performed immediately afterwards.

----------
Added file: http://bugs.python.org/file24965/duplicate.patch

_______________________________________
Python tracker <report@bugs.python.org>
<http://bugs.python.org/issue14310>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com
[issue14310] Socket duplication for windows [ In reply to ]
Kristján Valur Jónsson <kristjan@ccpgames.com> added the comment:

hm, patch doesn't work, removing the [mq] section

----------
Added file: http://bugs.python.org/file24966/duplicate.patch

_______________________________________
Python tracker <report@bugs.python.org>
<http://bugs.python.org/issue14310>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com
[issue14310] Socket duplication for windows [ In reply to ]
Changes by Kristján Valur Jónsson <kristjan@ccpgames.com>:


Removed file: http://bugs.python.org/file24966/duplicate.patch

_______________________________________
Python tracker <report@bugs.python.org>
<http://bugs.python.org/issue14310>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com
[issue14310] Socket duplication for windows [ In reply to ]
Changes by Kristján Valur Jónsson <kristjan@ccpgames.com>:


----------
Removed message: http://bugs.python.org/msg156418

_______________________________________
Python tracker <report@bugs.python.org>
<http://bugs.python.org/issue14310>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com
[issue14310] Socket duplication for windows [ In reply to ]
Kristján Valur Jónsson <kristjan@ccpgames.com> added the comment:

So, what is the consensus, does this go in or out?
What about the "share()" and "fromshare()" functions, are people ok with that?

----------
versions: +Python 3.3 -Python 3.4

_______________________________________
Python tracker <report@bugs.python.org>
<http://bugs.python.org/issue14310>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com
[issue14310] Socket duplication for windows [ In reply to ]
Kristján Valur Jónsson <kristjan@ccpgames.com> added the comment:

Still no responses? I'll submit this then, unless someone objects.

----------

_______________________________________
Python tracker <report@bugs.python.org>
<http://bugs.python.org/issue14310>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com
[issue14310] Socket duplication for windows [ In reply to ]
Antoine Pitrou <pitrou@free.fr> added the comment:

I would like to see this discussed on python-dev.

----------
nosy: +pitrou

_______________________________________
Python tracker <report@bugs.python.org>
<http://bugs.python.org/issue14310>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com
[issue14310] Socket duplication for windows [ In reply to ]
Kristján Valur Jónsson <kristjan@ccpgames.com> added the comment:

You are right, I was thinking more of pyobject attributes. I'll fix this then.

----------

_______________________________________
Python tracker <report@bugs.python.org>
<http://bugs.python.org/issue14310>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com
[issue14310] Socket duplication for windows [ In reply to ]
Changes by Santoso Wijaya <santoso.wijaya@gmail.com>:


----------
nosy: +santa4nt

_______________________________________
Python tracker <report@bugs.python.org>
<http://bugs.python.org/issue14310>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com
[issue14310] Socket duplication for windows [ In reply to ]
Changes by Kristján Valur Jónsson <kristjan@ccpgames.com>:


----------
Removed message: http://bugs.python.org/msg157189

_______________________________________
Python tracker <report@bugs.python.org>
<http://bugs.python.org/issue14310>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com
[issue14310] Socket duplication for windows [ In reply to ]
Kristján Valur Jónsson <kristjan@ccpgames.com> added the comment:

Not much happening on python-dev.
Perhaps this is not so controversial.

http://www.mail-archive.com/python-dev@python.org/msg66206.html

----------

_______________________________________
Python tracker <report@bugs.python.org>
<http://bugs.python.org/issue14310>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com
[issue14310] Socket duplication for windows [ In reply to ]
Antoine Pitrou <pitrou@free.fr> added the comment:

I have one question: you're simply doing a binary copy of the WSAPROTOCOL_INFO structure. Does it mean sharing wouldn't work between e.g. different MSVC builds of Python (if the structure is different), or between a 32-bit and a 64-bit Python?

----------

_______________________________________
Python tracker <report@bugs.python.org>
<http://bugs.python.org/issue14310>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com
[issue14310] Socket duplication for windows [ In reply to ]
Antoine Pitrou <pitrou@free.fr> added the comment:

Comments about the patch:
- versionadded should be 3.3, not 3.4
- MultiprocessingFunc would deserve a meaningful, PEP8 name; it should also probably be a classmethod on TestSocketSharing
- please use PEP8 for comments (a space after the "#")
- it would be nice to have a test for the ValueError when passing a bytes object of the wrong size
- also, there should be a test that the "family", "type" and "proto" attributes are correctly set on the fromshare() result (perhaps "timeout" too)

----------

_______________________________________
Python tracker <report@bugs.python.org>
<http://bugs.python.org/issue14310>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com
[issue14310] Socket duplication for windows [ In reply to ]
Kristján Valur Jónsson <kristjan@ccpgames.com> added the comment:

The binary copy:
This is a winsock2 feature. There is no difference in layout between 32 bits and 64 bits. I also don't think there is a difference between SDK versions. However, strangely, there is a difference between unicode and non-unicode builds. But since we are using this in python3, which always uses the same unicode setting, this shouldn't be an issue.

I'll take your other comments into account and produce a new patch.
Similar to socket.dup(), the socket shared is assumed to be in "blockin" mode (this his how its timeout is initialized). We can test for that too.

----------

_______________________________________
Python tracker <report@bugs.python.org>
<http://bugs.python.org/issue14310>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com
[issue14310] Socket duplication for windows [ In reply to ]
Kristján Valur Jónsson <kristjan@ccpgames.com> added the comment:

Here is a new patch, with more tests.
Note that the process worker function can't be a member function because of how multiprocessing works on Windows.

----------
Added file: http://bugs.python.org/file25143/duplicate.patch

_______________________________________
Python tracker <report@bugs.python.org>
<http://bugs.python.org/issue14310>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

1 2  View All