Mailing List Archive

Bugzilla vandalism
People who get mail from Bugzilla may have noticed some unusual
activity at around midnight UTC today. Someone created an account with
email address "tim.starling@rocketmail.com", using my name, and
proceeded to vandalise 88 bugs in 4 minutes. The vandalism consisted
of changes to random fields, such as status, component, CC, keywords,
dependencies, etc.

The scale of the problem was not immediately apparent since outgoing
email (and hence wikibugs IRC) was backlogged for about half an hour.

The vandalism has now been reverted. Some statuses and resolutions
were reverted by direct SQL queries. These reversions did not result
in a log entry or email.

The user's IP address was a Tor exit node. I blocked the IP address in
iptables, but when I found out it was an exit node, I also disabled
account creation entirely, so that we could stop the vandalism by user
account locks. It remains disabled for now.

About an hour before this incident, a user from a different Tor exit
node (jacob-craddy@mail.com) filed 21 new bugs in 40 seconds. All were
closed as "invalid".

Reverting hundreds of bug property changes was labour intensive. It
points to the need for better tools to deal with malicious behaviour
in Bugzilla. I looked into the possibility of writing an automated
revert tool as a command-line perl script integrated with Bugzilla,
but it looked like it would be fairly complicated:

* The bug history log has an irregular free form text format which is
not designed for automated reversions.

* The update interface (Bugzilla::Bug->set_*()) is not ideal for this
application, it mixes backend access methods such as database field
name maps with business logic such as agreement between product and
component.

-- Tim Starling


_______________________________________________
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: Bugzilla vandalism [ In reply to ]
2011/11/22 Tim Starling <tstarling@wikimedia.org>:
> People who get mail from Bugzilla may have noticed some unusual
> activity at around midnight UTC today. Someone created an account with
> email address "tim.starling@rocketmail.com", using my name, and
> proceeded to vandalise 88 bugs in 4 minutes. The vandalism consisted
> of changes to random fields, such as status, component, CC, keywords,
> dependencies, etc.

Thank you for the update. I noticed it and i assumed that it was
vandalism, but it's good to read the details.

--
Amir Elisha Aharoni · אָמִיר אֱלִישָׁע אַהֲרוֹנִי
http://aharoni.wordpress.com
‪“We're living in pieces,
I want to live in peace.” – T. Moore‬

_______________________________________________
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: Bugzilla vandalism [ In reply to ]
On Tue, Nov 22, 2011 at 7:12 AM, Tim Starling <tstarling@wikimedia.org> wrote:
> The user's IP address was a Tor exit node. I blocked the IP address in
> iptables, but when I found out it was an exit node, I also disabled
> account creation entirely, so that we could stop the vandalism by user
> account locks. It remains disabled for now.
>

I don't suppose we could block Tor indefinitely from write actions
on BZ?

-Chad

_______________________________________________
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: Bugzilla vandalism [ In reply to ]
On 22/11/11 23:34, Chad wrote:
> On Tue, Nov 22, 2011 at 7:12 AM, Tim Starling <tstarling@wikimedia.org> wrote:
>> The user's IP address was a Tor exit node. I blocked the IP address in
>> iptables, but when I found out it was an exit node, I also disabled
>> account creation entirely, so that we could stop the vandalism by user
>> account locks. It remains disabled for now.
>>
>
> I don't suppose we could block Tor indefinitely from write actions
> on BZ?

We can't even block Tor from write actions in MediaWiki, despite
having an extension which is meant to do exactly that. See bug 30716.
I haven't found any robust way to do it for Bugzilla or Apache, and we
should probably fix our own software before we try patching someone
else's.

-- Tim Starling


_______________________________________________
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: Bugzilla vandalism [ In reply to ]
On Tue, Nov 22, 2011 at 7:46 AM, Tim Starling <tstarling@wikimedia.org> wrote:
> On 22/11/11 23:34, Chad wrote:
>> On Tue, Nov 22, 2011 at 7:12 AM, Tim Starling <tstarling@wikimedia.org> wrote:
>>> The user's IP address was a Tor exit node. I blocked the IP address in
>>> iptables, but when I found out it was an exit node, I also disabled
>>> account creation entirely, so that we could stop the vandalism by user
>>> account locks. It remains disabled for now.
>>>
>>
>> I don't suppose we could block Tor indefinitely from write actions
>> on BZ?
>
> We can't even block Tor from write actions in MediaWiki, despite
> having an extension which is meant to do exactly that. See bug 30716.
> I haven't found any robust way to do it for Bugzilla or Apache, and we
> should probably fix our own software before we try patching someone
> else's.
>

I saw that bug this morning. The script that used to run on check.torproject.org
seems to be available[0] but I haven't tried it yet. At a quick
glance, it needs at
least one or two fixes--such as loading the exit data from an HTTP request,
rather than reading a file on the system. If it indeed works, it
should be trivial
to set this script up to run for us.

-Chad

[0] https://svn.torproject.org/svn/check/trunk/cgi-bin/TorBulkExitList.py

_______________________________________________
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: Bugzilla vandalism [ In reply to ]
On 22/11/11 13:12, Tim Starling wrote:
> proceeded to vandalise 88 bugs in 4 minutes. The vandalism consisted

> node (jacob-craddy@mail.com) filed 21 new bugs in 40 seconds. All were

Perhaps some throttling could be helpful here? I don't see a legitimate
user filing more than a bug per minute or a change per, say, 15s.

_______________________________________________
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: Bugzilla vandalism [ In reply to ]
On Tue, Nov 22, 2011 at 8:16 AM, Nikola Smolenski <smolensk@eunet.rs> wrote:
> On 22/11/11 13:12, Tim Starling wrote:
>> proceeded to vandalise 88 bugs in 4 minutes. The vandalism consisted
>
>> node (jacob-craddy@mail.com) filed 21 new bugs in 40 seconds. All were
>
> Perhaps some throttling could be helpful here? I don't see a legitimate
> user filing more than a bug per minute or a change per, say, 15s.
>

I can't seem to find any built-in BZ support for such a thing--nor can
I even find a bug (other than this tangentially-related WONTFIX[0])
requesting it.

Maybe something like mod_bandwidth would be useful here?

-Chad

[0] https://bugzilla.mozilla.org/216738

_______________________________________________
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: Bugzilla vandalism [ In reply to ]
----- Original Message -----
> From: "Tim Starling" <tstarling@wikimedia.org>
> Reverting hundreds of bug property changes was labour intensive. It
> points to the need for better tools to deal with malicious behaviour
> in Bugzilla. I looked into the possibility of writing an automated
> revert tool as a command-line perl script integrated with Bugzilla,
> but it looked like it would be fairly complicated:

Might there be some relatively easy way to hack rate limiting into the code?

In general, except for triage WONTFIX runs, I shouldn't think any given user
would need to comment on or status-change a bug more than about every minute
or two, possibly with an exponential backoff.

Cheers,
-- jra
--
Jay R. Ashworth Baylink jra@baylink.com
Designer The Things I Think RFC 2100
Ashworth & Associates http://baylink.pitas.com 2000 Land Rover DII
St Petersburg FL USA http://photo.imageinc.us +1 727 647 1274

_______________________________________________
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: Bugzilla vandalism [ In reply to ]
Keep in mind that you can bulk-modify bugs; updating assignments,
department, components etc may change dozens of bugs at once quite
legitimately.

-- brion
On Nov 22, 2011 5:51 AM, "Jay Ashworth" <jra@baylink.com> wrote:

> ----- Original Message -----
> > From: "Tim Starling" <tstarling@wikimedia.org>
> > Reverting hundreds of bug property changes was labour intensive. It
> > points to the need for better tools to deal with malicious behaviour
> > in Bugzilla. I looked into the possibility of writing an automated
> > revert tool as a command-line perl script integrated with Bugzilla,
> > but it looked like it would be fairly complicated:
>
> Might there be some relatively easy way to hack rate limiting into the
> code?
>
> In general, except for triage WONTFIX runs, I shouldn't think any given
> user
> would need to comment on or status-change a bug more than about every
> minute
> or two, possibly with an exponential backoff.
>
> Cheers,
> -- jra
> --
> Jay R. Ashworth Baylink
> jra@baylink.com
> Designer The Things I Think RFC
> 2100
> Ashworth & Associates http://baylink.pitas.com 2000 Land
> Rover DII
> St Petersburg FL USA http://photo.imageinc.us +1 727 647
> 1274
>
> _______________________________________________
> Wikitech-l mailing list
> Wikitech-l@lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
>
_______________________________________________
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: Bugzilla vandalism [ In reply to ]
On 11/22/11 5:16 AM, Nikola Smolenski wrote:
> Perhaps some throttling could be helpful here? I don't see a legitimate
> user filing more than a bug per minute or a change per, say, 15s.

I've already done this. I switched over to a local git repo a few weeks
ago, using git-svn to interface with our svn repo. So I commit changes
locally as I go, and then infrequently batch-commit them back to SVN. On
Nov 2 I committed ten changes in less than a minute.

When we move to git it may be more normal for developers to commit like
this. It's possible to batch all your local changes into one big change,
but I think smaller changes are better for reviewing anyway.

And who knows what automated scripts might do, even today.

I do agree that more than a bug per minute might be excessive, unless
there is some legit reason why a bot would do that (perhaps looking for
linting errors?).

--
Neil Kandalgaonkar ( <neilk@wikimedia.org>

_______________________________________________
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: Bugzilla vandalism [ In reply to ]
On 22/11/11 14:51, Jay Ashworth wrote:
> Might there be some relatively easy way to hack rate limiting into the code?
>
> In general, except for triage WONTFIX runs, I shouldn't think any given user
> would need to comment on or status-change a bug more than about every minute
> or two, possibly with an exponential backoff.
>
> Cheers,
> -- jra

After fixing one bug, going to close another as a duplicate less than a
minute after the previous save, doesn't seem strange. There can even be
a few of them.


_______________________________________________
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: Bugzilla vandalism [ In reply to ]
----- Original Message -----
> From: "Platonides" <Platonides@gmail.com>

> > Might there be some relatively easy way to hack rate limiting into
> > the code?
> >
> > In general, except for triage WONTFIX runs, I shouldn't think any given user
> > would need to comment on or status-change a bug more than about every minute
> > or two, possibly with an exponential backoff.
>
> After fixing one bug, going to close another as a duplicate less than a
> minute after the previous save, doesn't seem strange. There can even be
> a few of them.

Fair point.

I don't have any non-vandalism clickstream stats, so I can't tell you whether
that invalidates the suggestion, or merely means you have to tune it better.

"Allow 5 changes in 3 minutes, and then require a 30 second delay that doubles
after every three additional changes"?

You see what I mean.

Cheers,
-- jra
--
Jay R. Ashworth Baylink jra@baylink.com
Designer The Things I Think RFC 2100
Ashworth & Associates http://baylink.pitas.com 2000 Land Rover DII
St Petersburg FL USA http://photo.imageinc.us +1 727 647 1274

_______________________________________________
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: Bugzilla vandalism [ In reply to ]
On Tue, Nov 22, 2011 at 4:11 PM, Platonides <Platonides@gmail.com> wrote:
> After fixing one bug, going to close another as a duplicate less than a
> minute after the previous save, doesn't seem strange. There can even be
> a few of them.

But from Tor? Sure, walking into a bank and asking for lots of money
is common too, unless you are wearing a mask ;)

_______________________________________________
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: Bugzilla vandalism [ In reply to ]
On Tue, Nov 22, 2011 at 4:12 AM, Tim Starling <tstarling@wikimedia.org> wrote:
> I looked into the possibility of writing an automated
> revert tool as a command-line perl script integrated with Bugzilla,
> but it looked like it would be fairly complicated:

Good news: there's a bug filed for this in Bugzilla's tracker:
https://bugzilla.mozilla.org/show_bug.cgi?id=363346

Bad news: it's 5 years old, without a lot of visible interest.

Also....

On Tue, Nov 22, 2011 at 5:25 AM, Chad <innocentkiller@gmail.com> wrote:
> On Tue, Nov 22, 2011 at 8:16 AM, Nikola Smolenski <smolensk@eunet.rs> wrote:
>> Perhaps some throttling could be helpful here? I don't see a legitimate
>> user filing more than a bug per minute or a change per, say, 15s.
>
> I can't seem to find any built-in BZ support for such a thing--nor can
> I even find a bug (other than this tangentially-related WONTFIX[0])
> requesting it.

New feature request filed:
https://bugzilla.mozilla.org/show_bug.cgi?id=704753

Anyone with some Perl chops want to take on a project?

Rob

_______________________________________________
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: Bugzilla vandalism [ In reply to ]
On 23/11/11 06:46, trouble daemon wrote:
> On Tue, Nov 22, 2011 at 4:11 PM, Platonides <Platonides@gmail.com> wrote:
>> After fixing one bug, going to close another as a duplicate less than a
>> minute after the previous save, doesn't seem strange. There can even be
>> a few of them.
>
> But from Tor? Sure, walking into a bank and asking for lots of money
> is common too, unless you are wearing a mask ;)

According to http://xkcd.com/980, Batman has a fortune of $6,500,000,000

I was just pointing out a common case (probably more for hexmode or
sumanah). You can then tune it to not hit false positives.


_______________________________________________
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: Bugzilla vandalism [ In reply to ]
----- Original Message -----
> From: "Platonides" <Platonides@gmail.com>

> On 23/11/11 06:46, trouble daemon wrote:
> > On Tue, Nov 22, 2011 at 4:11 PM, Platonides <Platonides@gmail.com>
> > wrote:
> >> After fixing one bug, going to close another as a duplicate less
> >> than a
> >> minute after the previous save, doesn't seem strange. There can
> >> even be
> >> a few of them.
> >
> > But from Tor? Sure, walking into a bank and asking for lots of money
> > is common too, unless you are wearing a mask ;)
>
> According to http://xkcd.com/980, Batman has a fortune of
> $6,500,000,000
>
> I was just pointing out a common case (probably more for hexmode or
> sumanah). You can then tune it to not hit false positives.

Or you can just put in a manual override if it *thinks* it has a positive,
whether a captcha (of increasing frequency), or "go to IRC and get a cookie
from a human that's good for an hour".

Cheers,
-- jra
--
Jay R. Ashworth Baylink jra@baylink.com
Designer The Things I Think RFC 2100
Ashworth & Associates http://baylink.pitas.com 2000 Land Rover DII
St Petersburg FL USA http://photo.imageinc.us +1 727 647 1274

_______________________________________________
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: Bugzilla vandalism [ In reply to ]
We are now getting mail to noc@ by users asking us to create accounts for them.
I would have created one but don't have that permission on admin.cgi . Who does?
Do we even want to do that manually now? Could also be a vandal asking
for it, even though unlikely :p

---
from Annaïg DENIS adenis@alliance-libre.org
to noc@wikimedia.org
cc Antoine MUSSO <amusso@free.fr>
date Thu, Nov 24, 2011 at 2:46 PM
subject Account for bugzilla
mailed-by alliance-libre.org
Important mainly because of the people in the conversation.

hide details 2:46 PM (18 minutes ago)

hi,

I have a bug to report on wikimedia but I can't create an account on
https://bugzilla.wikimedia.org/createaccount.cgi

Could you tell me what to do?

Thanks

Regards

Annaïg DENIS

_______________________________________________
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: Bugzilla vandalism [ In reply to ]
As I pointed out on IRC, but just repeating it in here in case someone
else actually wants to attempt it (Regarding the create via admin
panel).

[00:22] <p858snake|l> mutante: don't use that function to create accounts
[00:23] <p858snake|l> it doesn't send the emails to create it or lets
you set the passwords, so you have useless accounts
[00:24] <p858snake|l> unless bz has actually fixed that function in
the last version or two since i looked about it

_______________________________________________
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: Bugzilla vandalism [ In reply to ]
On Thu, Nov 24, 2011 at 3:29 PM, K. Peachey <p858snake@gmail.com> wrote:

> <p858snake|l> it doesn't send the emails to create it or lets you set the passwords, so you have useless accounts
> <p858snake|l> unless bz has actually fixed that function in the last version or two since i looked about it

It's still like that :/

Bugzilla Guide - 4.0.2 Release (which we are running)

--> 3.2.2.2.2. Accounts created by an administrator

"Adding a user this way will not send an email informing them of their
username and password. While useful for creating dummy accounts
(watchers which shuttle mail to another system, for instance, or email
addresses which are a mailing list), in general it is preferable to
log out and use the "New Account" button to create users, as it will
pre-populate all the required fields and also notify the user of her
account name and password."

http://www.bugzilla.org/docs/4.0/en/html/useradmin.html

--
--
Daniel Zahn <dzahn@wikimedia.org>

_______________________________________________
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: Bugzilla vandalism [ In reply to ]
On Thu, Nov 24, 2011 at 6:11 AM, Daniel Zahn <dzahn@wikimedia.org> wrote:
> We are now getting mail to noc@ by users asking us to create accounts for them.
> I would have created one but don't have that permission on admin.cgi . Who does?
> Do we even want to do that manually now? Could also be a vandal asking
> for it, even though unlikely :p

So, here's the solution for now, and probably for a while:

1. New account creation has been re-enabled
2. All existing accounts have been explicitly given the "editbugs" permission
3. New accounts only get the ability to file bugs, comment on bugs,
and mark bugs as confirmed

We intend to be very liberal with who we bless with "editbugs", but
not so liberal as to incur more vandalism. Having proper vandalism
fighting solutions (e.g. especially bug 363346 and probably bug 704753
in bugzilla.mozilla.org) seems like a necessary precondition to going
back to the old way of doing things.

Rob

_______________________________________________
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: Bugzilla vandalism [ In reply to ]
On Thu, Nov 24, 2011 at 8:15 PM, Rob Lanphier <robla@wikimedia.org> wrote:
> So, here's the solution for now, and probably for a while:
>
> 1.  New account creation has been re-enabled
> 2.  All existing accounts have been explicitly given the "editbugs" permission
> 3.  New accounts only get the ability to file bugs, comment on bugs,
> and mark bugs as confirmed

Nice! Thanks apergos and robla.
@Antoine:I have mailed Annaïg to try again now.

--
--
Daniel Zahn <dzahn@wikimedia.org>

_______________________________________________
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: Bugzilla vandalism [ In reply to ]
Στις 24-11-2011, ημέρα Πεμ, και ώρα 21:30 +0100, ο/η Daniel Zahn έγραψε:
> On Thu, Nov 24, 2011 at 8:15 PM, Rob Lanphier <robla@wikimedia.org> wrote:
> > So, here's the solution for now, and probably for a while:
> >
> > 1. New account creation has been re-enabled
> > 2. All existing accounts have been explicitly given the "editbugs" permission
> > 3. New accounts only get the ability to file bugs, comment on bugs,
> > and mark bugs as confirmed
>
> Nice! Thanks apergos and robla.
> @Antoine:I have mailed Annaïg to try again now.
>

Not so nice, as it turns out. They were back today adding and removing
people from CC lists. I've disabled new account creation again, the two
accounts they created are blocked, and Siebrand cleaned up the bugs,
about 80 of them.

Ariel


_______________________________________________
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: Bugzilla vandalism [ In reply to ]
On Fri, Nov 25, 2011 at 8:56 AM, Ariel T. Glenn <ariel@wikimedia.org> wrote:
>
> Στις 24-11-2011, ημέρα Πεμ, και ώρα 21:30 +0100, ο/η Daniel Zahn έγραψε:
> > On Thu, Nov 24, 2011 at 8:15 PM, Rob Lanphier <robla@wikimedia.org> wrote:
> > > So, here's the solution for now, and probably for a while:
> > >
> > > 1.  New account creation has been re-enabled
> > > 2.  All existing accounts have been explicitly given the "editbugs" permission
> > > 3.  New accounts only get the ability to file bugs, comment on bugs,
> > > and mark bugs as confirmed
> >
> > Nice! Thanks apergos and robla.
> > @Antoine:I have mailed Annaïg to try again now.
> >
>
> Not so nice, as it turns out. They were back today adding and removing
> people from CC lists. I've disabled new account creation again, the two
> accounts they created are blocked, and Siebrand cleaned up the bugs,
> about 80 of them.
>
There is also another, impersonating brion, which does not appear to
have been cleaned up.


Bryan

_______________________________________________
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: Bugzilla vandalism [ In reply to ]
On Fri, Nov 25, 2011 at 7:06 PM, Bryan Tong Minh
<bryan.tongminh@gmail.com> wrote:
> There is also another, impersonating brion, which does not appear to
> have been cleaned up.
Most of that has been (Its what Ariel was referring to just before),
But it was done mostly with email off to prevent people getting
flooded with bogus./unwanted emails about it.

_______________________________________________
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: Bugzilla vandalism [ In reply to ]
Στις 25-11-2011, ημέρα Παρ, και ώρα 19:08 +1000, ο/η K. Peachey έγραψε:
> On Fri, Nov 25, 2011 at 7:06 PM, Bryan Tong Minh
> <bryan.tongminh@gmail.com> wrote:
> > There is also another, impersonating brion, which does not appear to
> > have been cleaned up.
> Most of that has been (Its what Ariel was referring to just before),
> But it was done mostly with email off to prevent people getting
> flooded with bogus./unwanted emails about it.

Fixed about another 40 of those. That should be the lot of them.

Ariel



_______________________________________________
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

1 2  View All