Mailing List Archive

Decentralized data center
Hello, this topic is from foundation-l, I think it is more suited on
wikitech-l.

-------- Message original --------

What about sharing the whole databases among the millions of users, in
some p2p net with a lot of redundancies?, something like a dense, cloudy
internet of databases who remains whole even if it looses part of
itself? Does it sound unwordly?
It could be a good complement to the server based versions.

Le 22/01/2012 20:50, Jussi-Ville Heiskanen a écrit :
> The simple option that will just blow all this talk fo lobbying away,
> is to migrate outside US jurisdiction entirely. It does entail some
> costs, and may well not be optimal, on many fronts.
>
> A medium option is to do a plan on the lines of the actions that
> Google has already put into force, of diversifying datacenters that
> have our non-fungible assets, so that for enforcement they would
> have to invade sovreign territory. But for a non-profit, our best line
> would be to say that we are making those plans, but actually want
> to keep the US have the PR benefit of being able to say that WMF
> like entities find the US best to be incorporated in. And then grin
> very hard, so they know we mean business. Follow up with saying
> the very real contingency plans can not wait on their realizing they
> have the wrong end of the stick, so we have to act now.
>
> So we will put a few fallback datacenters elsewhere, just so our
> various communities and chapters realize we aren't going to be
> bullied by US jurisdiction. But we have a much more expansive
> plan which we tell we will eventually realize. But the legislators
> in the US have to understand we are doing this all so they realize
> what they are working on is harmful to prosperity around the globe.
>
> And if they play ball, (we won't give a cent of tribute, sorry) we will
> not accelerate the rate at which we realize the full international
> nature of the Wikimedia Foundation.
>
> That is pretty much the line of "education" that might be effective,
> without costing the Foundation a single backhander.



_______________________________________________
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: Decentralized data center [ In reply to ]
On Mon, Jan 23, 2012 at 9:38 AM, cyrano <cyrano.fawkes@gmail.com> wrote:
> Hello, this topic is from foundation-l, I think it is more suited on
> wikitech-l.
>
> -------- Message original --------
>
> What about sharing the whole databases among the millions of users, in
> some p2p net with a lot of redundancies?, something like a dense, cloudy
> internet of databases who remains whole even if it looses part of
> itself? Does it sound unwordly?
> It could be a good complement to the server based versions.

this sounds nice but just wouldn't work at all. we need to have
reliable databases with a consistant latency.

>
> Le 22/01/2012 20:50, Jussi-Ville Heiskanen a écrit :
>> The simple option that will just blow all this talk fo lobbying away,
>> is to migrate outside US jurisdiction entirely. It does entail some
>> costs, and may well not be optimal, on many fronts.

s/some/lots of/

>>
>> A medium option is to do a plan on the lines of the actions that
>> Google has already put into  force, of diversifying datacenters that
>> have our non-fungible assets, so that for enforcement they would
>> have to invade sovreign territory. But for a non-profit, our best line
>> would be to say that we are making those plans, but actually want
>> to keep the US have the PR benefit of being able to say that WMF
>> like entities find the US best to be incorporated in. And then grin
>> very hard, so they know we mean business. Follow up with saying
>> the very real contingency plans can not wait on their realizing they
>> have the wrong end of the stick, so we have to act now.
>>
>> So we will put a few fallback datacenters elsewhere, just so our
>> various communities and chapters realize we aren't going to be
>> bullied by US jurisdiction. But we have a much more expansive
>> plan which we tell we will eventually realize. But the legislators
>> in the US have to understand we are doing this all so they realize
>> what they are working on is harmful to prosperity around the globe.
>>

again, expensive!

>> And if they play ball, (we won't give a cent of tribute, sorry) we will
>> not accelerate the rate at which we realize the full international
>> nature of the Wikimedia Foundation.
>>
>> That is pretty much the line of "education" that might be effective,
>> without costing the Foundation a single backhander.
>
>
>
> _______________________________________________
> Wikitech-l mailing list
> Wikitech-l@lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l



--
Leslie Carr
Wikimedia Foundation
AS 14907, 43821

_______________________________________________
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: Decentralized data center [ In reply to ]
Hoi,
What we need is a database with a robust and fool proof synchronisation
process. Latency is important when all the databases have to be
synchronised all the time. When articles only need to be the latest when
they are readied for editing it is less of an issue.

When the data is shared over many computers in for instance a peer to peer
network, the real cost can be high but when the cost is shared by
volunteers and organisations volunteering it is no longer much of an issue
for the central organisation.

Databases of such a design are not that rare. There has been research on
this for Wikipedia usage in Amsterdam where it is considered feasible.
Thanks,
Gerard


On 23 January 2012 18:48, Leslie Carr <lcarr@wikimedia.org> wrote:

> On Mon, Jan 23, 2012 at 9:38 AM, cyrano <cyrano.fawkes@gmail.com> wrote:
> > Hello, this topic is from foundation-l, I think it is more suited on
> > wikitech-l.
> >
> > -------- Message original --------
> >
> > What about sharing the whole databases among the millions of users, in
> > some p2p net with a lot of redundancies?, something like a dense, cloudy
> > internet of databases who remains whole even if it looses part of
> > itself? Does it sound unwordly?
> > It could be a good complement to the server based versions.
>
> this sounds nice but just wouldn't work at all. we need to have
> reliable databases with a consistant latency.
>
> >
> > Le 22/01/2012 20:50, Jussi-Ville Heiskanen a écrit :
> >> The simple option that will just blow all this talk fo lobbying away,
> >> is to migrate outside US jurisdiction entirely. It does entail some
> >> costs, and may well not be optimal, on many fronts.
>
> s/some/lots of/
>
> >>
> >> A medium option is to do a plan on the lines of the actions that
> >> Google has already put into force, of diversifying datacenters that
> >> have our non-fungible assets, so that for enforcement they would
> >> have to invade sovreign territory. But for a non-profit, our best line
> >> would be to say that we are making those plans, but actually want
> >> to keep the US have the PR benefit of being able to say that WMF
> >> like entities find the US best to be incorporated in. And then grin
> >> very hard, so they know we mean business. Follow up with saying
> >> the very real contingency plans can not wait on their realizing they
> >> have the wrong end of the stick, so we have to act now.
> >>
> >> So we will put a few fallback datacenters elsewhere, just so our
> >> various communities and chapters realize we aren't going to be
> >> bullied by US jurisdiction. But we have a much more expansive
> >> plan which we tell we will eventually realize. But the legislators
> >> in the US have to understand we are doing this all so they realize
> >> what they are working on is harmful to prosperity around the globe.
> >>
>
> again, expensive!
>
> >> And if they play ball, (we won't give a cent of tribute, sorry) we will
> >> not accelerate the rate at which we realize the full international
> >> nature of the Wikimedia Foundation.
> >>
> >> That is pretty much the line of "education" that might be effective,
> >> without costing the Foundation a single backhander.
> >
> >
> >
> > _______________________________________________
> > Wikitech-l mailing list
> > Wikitech-l@lists.wikimedia.org
> > https://lists.wikimedia.org/mailman/listinfo/wikitech-l
>
>
>
> --
> Leslie Carr
> Wikimedia Foundation
> AS 14907, 43821
>
> _______________________________________________
> Wikitech-l mailing list
> Wikitech-l@lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
>
_______________________________________________
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: Decentralized data center [ In reply to ]
On Mon, Jan 23, 2012 at 11:41 AM, Gerard Meijssen
<gerard.meijssen@gmail.com> wrote:
> Hoi,
> What we need is a database with a robust and fool proof synchronisation
> process. Latency is important when all the databases have to be
> synchronised all the time. When articles only need to be the latest when
> they are readied for editing it is less of an issue.
>
> When the data is shared over many computers in for instance a peer to peer
> network, the real cost can be high but when the cost is shared by
> volunteers and organisations volunteering it is no longer much of an issue
> for the central organisation.
>
> Databases of such a design are not that rare. There has been research on
> this for Wikipedia usage in Amsterdam where it is considered feasible.
> Thanks,
>     Gerard
>
>
> On 23 January 2012 18:48, Leslie Carr <lcarr@wikimedia.org> wrote:
>
>> On Mon, Jan 23, 2012 at 9:38 AM, cyrano <cyrano.fawkes@gmail.com> wrote:
>> > Hello, this topic is from foundation-l, I think it is more suited on
>> > wikitech-l.
>> >
>> > -------- Message original --------
>> >
>> > What about sharing the whole databases among the millions of users, in
>> > some p2p net with a lot of redundancies?, something like a dense, cloudy
>> > internet of databases who remains whole even if it looses part of
>> > itself? Does it sound unwordly?
>> > It could be a good complement to the server based versions.
>>
>> this sounds nice but just wouldn't work at all.  we need to have
>> reliable databases with a consistant latency.
>>
>> >
>> > Le 22/01/2012 20:50, Jussi-Ville Heiskanen a écrit :
>> >> The simple option that will just blow all this talk fo lobbying away,
>> >> is to migrate outside US jurisdiction entirely. It does entail some
>> >> costs, and may well not be optimal, on many fronts.
>>
>> s/some/lots of/
>>
>> >>
>> >> A medium option is to do a plan on the lines of the actions that
>> >> Google has already put into  force, of diversifying datacenters that
>> >> have our non-fungible assets, so that for enforcement they would
>> >> have to invade sovreign territory. But for a non-profit, our best line
>> >> would be to say that we are making those plans, but actually want
>> >> to keep the US have the PR benefit of being able to say that WMF
>> >> like entities find the US best to be incorporated in. And then grin
>> >> very hard, so they know we mean business. Follow up with saying
>> >> the very real contingency plans can not wait on their realizing they
>> >> have the wrong end of the stick, so we have to act now.
>> >>
>> >> So we will put a few fallback datacenters elsewhere, just so our
>> >> various communities and chapters realize we aren't going to be
>> >> bullied by US jurisdiction. But we have a much more expansive
>> >> plan which we tell we will eventually realize. But the legislators
>> >> in the US have to understand we are doing this all so they realize
>> >> what they are working on is harmful to prosperity around the globe.
>> >>
>>
>> again, expensive!
>>
>> >> And if they play ball, (we won't give a cent of tribute, sorry) we will
>> >> not accelerate the rate at which we realize the full international
>> >> nature of the Wikimedia Foundation.
>> >>
>> >> That is pretty much the line of "education" that might be effective,
>> >> without costing the Foundation a single backhander.


Byzantine fault tolerance and proving provenance of the data are major
issues with distributed peer-to-peer systems; we have bad enough
problems with vandals now, without cloudsourcing our database storage
mechanisms...

Today, this is a computer science solution, not an operations
solution. The Foundation doesn't exist to be a testbed for making
computer science technology operationally mature.


--
-george william herbert
george.herbert@gmail.com

_______________________________________________
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: Decentralized data center [ In reply to ]
Hoi,
Vandals are not a technical problem.

Asynchronous updating has been in production in for instance the Novell NDS
for many many, many years. Calling it Byzantine and research only does not
negate the fact that our existing solution is not the only one. There are
reasons why we stick with what we have but it does not follow that other
avenues are not open to us.
Thanks,
Gerard

On 24 January 2012 00:23, George Herbert <george.herbert@gmail.com> wrote:

> On Mon, Jan 23, 2012 at 11:41 AM, Gerard Meijssen
> <gerard.meijssen@gmail.com> wrote:
> > Hoi,
> > What we need is a database with a robust and fool proof synchronisation
> > process. Latency is important when all the databases have to be
> > synchronised all the time. When articles only need to be the latest when
> > they are readied for editing it is less of an issue.
> >
> > When the data is shared over many computers in for instance a peer to
> peer
> > network, the real cost can be high but when the cost is shared by
> > volunteers and organisations volunteering it is no longer much of an
> issue
> > for the central organisation.
> >
> > Databases of such a design are not that rare. There has been research on
> > this for Wikipedia usage in Amsterdam where it is considered feasible.
> > Thanks,
> > Gerard
> >
> >
> > On 23 January 2012 18:48, Leslie Carr <lcarr@wikimedia.org> wrote:
> >
> >> On Mon, Jan 23, 2012 at 9:38 AM, cyrano <cyrano.fawkes@gmail.com>
> wrote:
> >> > Hello, this topic is from foundation-l, I think it is more suited on
> >> > wikitech-l.
> >> >
> >> > -------- Message original --------
> >> >
> >> > What about sharing the whole databases among the millions of users, in
> >> > some p2p net with a lot of redundancies?, something like a dense,
> cloudy
> >> > internet of databases who remains whole even if it looses part of
> >> > itself? Does it sound unwordly?
> >> > It could be a good complement to the server based versions.
> >>
> >> this sounds nice but just wouldn't work at all. we need to have
> >> reliable databases with a consistant latency.
> >>
> >> >
> >> > Le 22/01/2012 20:50, Jussi-Ville Heiskanen a écrit :
> >> >> The simple option that will just blow all this talk fo lobbying away,
> >> >> is to migrate outside US jurisdiction entirely. It does entail some
> >> >> costs, and may well not be optimal, on many fronts.
> >>
> >> s/some/lots of/
> >>
> >> >>
> >> >> A medium option is to do a plan on the lines of the actions that
> >> >> Google has already put into force, of diversifying datacenters that
> >> >> have our non-fungible assets, so that for enforcement they would
> >> >> have to invade sovreign territory. But for a non-profit, our best
> line
> >> >> would be to say that we are making those plans, but actually want
> >> >> to keep the US have the PR benefit of being able to say that WMF
> >> >> like entities find the US best to be incorporated in. And then grin
> >> >> very hard, so they know we mean business. Follow up with saying
> >> >> the very real contingency plans can not wait on their realizing they
> >> >> have the wrong end of the stick, so we have to act now.
> >> >>
> >> >> So we will put a few fallback datacenters elsewhere, just so our
> >> >> various communities and chapters realize we aren't going to be
> >> >> bullied by US jurisdiction. But we have a much more expansive
> >> >> plan which we tell we will eventually realize. But the legislators
> >> >> in the US have to understand we are doing this all so they realize
> >> >> what they are working on is harmful to prosperity around the globe.
> >> >>
> >>
> >> again, expensive!
> >>
> >> >> And if they play ball, (we won't give a cent of tribute, sorry) we
> will
> >> >> not accelerate the rate at which we realize the full international
> >> >> nature of the Wikimedia Foundation.
> >> >>
> >> >> That is pretty much the line of "education" that might be effective,
> >> >> without costing the Foundation a single backhander.
>
>
> Byzantine fault tolerance and proving provenance of the data are major
> issues with distributed peer-to-peer systems; we have bad enough
> problems with vandals now, without cloudsourcing our database storage
> mechanisms...
>
> Today, this is a computer science solution, not an operations
> solution. The Foundation doesn't exist to be a testbed for making
> computer science technology operationally mature.
>
>
> --
> -george william herbert
> george.herbert@gmail.com
>
> _______________________________________________
> Wikitech-l mailing list
> Wikitech-l@lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
>
_______________________________________________
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: Decentralized data center [ In reply to ]
If I remember correctly, a backup of database is in a public folder,
so anyone can download it. Some people do it, to run analysis and do
some interesting science/tech projects. If the FBI and the Interpol
put the wikimedia people in jail (thats not going to happend), anybody
in a country withouth extradition treaty to USA can recreate it.

So, the wikimedia organization can be killed, but the data is public
and mirrored "enough" so probably can't be killed as easy.

Random link (not related):
https://tahoe-lafs.org/trac/tahoe-lafs

--
--
ℱin del ℳensaje.

_______________________________________________
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: Decentralized data center [ In reply to ]
Actually we still want mirrors of the revision texts and the mysql
tables [1], and we do not yet have a mirror of the image data. If
anyone has contacts at a university with good bandwidth and 6T + of
space lying around for a good cause...

Ariel

[1] http://meta.wikimedia.org/wiki/Mirroring_Wikimedia_project_XML_dumps

Στις 25-01-2012, ημέρα Τετ, και ώρα 13:58 +0100, ο/η Tei έγραψε:
> If I remember correctly, a backup of database is in a public folder,
> so anyone can download it. Some people do it, to run analysis and do
> some interesting science/tech projects. If the FBI and the Interpol
> put the wikimedia people in jail (thats not going to happend), anybody
> in a country withouth extradition treaty to USA can recreate it.
>
> So, the wikimedia organization can be killed, but the data is public
> and mirrored "enough" so probably can't be killed as easy.
>
> Random link (not related):
> https://tahoe-lafs.org/trac/tahoe-lafs
>
> --
> --
> ℱin del ℳensaje.
>
> _______________________________________________
> Wikitech-l mailing list
> Wikitech-l@lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l



_______________________________________________
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l