Mailing List Archive

Five-year WMF targets exclude non-Wikipedia projects
Despite repeated assurances at Wikimania, on lists and on strategywiki,
that the strategic plan was going to consider all Wikimedia projects as
important, now at
http://wikimediafoundation.org/wiki/Resolution:Five-year_targets the
second target, «Increase the amount of information we offer» considers
only the number of Wikipedia articles.
«We're aware of the challenges around bot-created articles, articles of
low quality, etc., and the limited focus on Wikipedia, so this metric
shouldn't be seen in isolation, but is an important indicator.» Yes, but
a wrong one.

I'm, very, very disappointed: I have to conclude that all the words on
community participation etc. were only empty rhetoric.

Nemo

_______________________________________________
foundation-l mailing list
foundation-l@lists.wikimedia.org
Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/foundation-l
Re: Five-year WMF targets exclude non-Wikipedia projects [ In reply to ]
On 10 Oct 2010, at 11:33, Federico Leva (Nemo) wrote:

> Despite repeated assurances at Wikimania, on lists and on strategywiki,
> that the strategic plan was going to consider all Wikimedia projects as
> important, now at
> http://wikimediafoundation.org/wiki/Resolution:Five-year_targets the
> second target, «Increase the amount of information we offer» considers
> only the number of Wikipedia articles.
> «We're aware of the challenges around bot-created articles, articles of
> low quality, etc., and the limited focus on Wikipedia, so this metric
> shouldn't be seen in isolation, but is an important indicator.» Yes, but
> a wrong one.
>
> I'm, very, very disappointed: I have to conclude that all the words on
> community participation etc. were only empty rhetoric.

It's a shame that the number of Wikipedia articles is the only entry under that heading, but this appears to be a vastly simplified document that is very black and white - every single objective only has one unit of measure, whereas there should be several for every one of them. I would hope that the Foundation's board recognised this (either officially or unofficially) during their consideration of it, and that the extrapolation of saying that community participation was only empty rhetoric is not a good extrapolation (I sincerely doubt it is - that reassurance will have been based in reality).

In any case, I think one of the major benefits of the strategy exercise was to get Wikimedians considering where Wikimedia should be in 5 years and setting their individual aims accordingly. Getting the WMF Board to recognise those aims is only a secondary consideration, really, as it's the community that drives Wikimedia's success and breadth/depth/etc. of content.

Mike Peel
_______________________________________________
foundation-l mailing list
foundation-l@lists.wikimedia.org
Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/foundation-l
Re: Five-year WMF targets exclude non-Wikipedia projects [ In reply to ]
On Sun, Oct 10, 2010 at 9:33 PM, Federico Leva (Nemo)
<nemowiki@gmail.com> wrote:
> Despite repeated assurances at Wikimania, on lists and on strategywiki,
> that the strategic plan was going to consider all Wikimedia projects as
> important, now at
> http://wikimediafoundation.org/wiki/Resolution:Five-year_targets the
> second target, «Increase the amount of information we offer» considers
> only the number of Wikipedia articles.
> «We're aware of the challenges around bot-created articles, articles of
> low quality, etc., and the limited focus on Wikipedia, so this metric
> shouldn't be seen in isolation, but is an important indicator.» Yes, but
> a wrong one.
>
> I'm, very, very disappointed: I have to conclude that all the words on
> community participation etc. were only empty rhetoric.

I am likewise disappointed. The five year plan _should_ have seen the
other projects as the most likely source of new talent, contributors
and innovation, and should have focused on developing them.

Even worse is the third target, which is _wrong_ because the
Foundation hasn't included other projects in its considerations. It
says we don't have baselines for quality. On Wikisource, we _do_ have
empirical data on quality built into in the Proofreading system.

http://wikisource.org/wiki/Wikisource:ProofreadPage_Statistics

Also, how was the 100,000 contributors per month with 5+ edits
calculated? Using simple addition of monthly counts for each project,
I can only see 90,000 contributors with >=5 edits. The figure would
be lower as the contributors to smaller projects usually intersects
the contributors to the larger projects.

http://wikimediafoundation.org/wiki/File:Wikimedia_Five-Year_Targets.ods

--
John Vandenberg

_______________________________________________
foundation-l mailing list
foundation-l@lists.wikimedia.org
Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/foundation-l
Re: Five-year WMF targets exclude non-Wikipedia projects [ In reply to ]
On 10 October 2010 09:33, Federico Leva (Nemo) <nemowiki@gmail.com> wrote:
> Despite repeated assurances at Wikimania, on lists and on strategywiki,
> that the strategic plan was going to consider all Wikimedia projects as
> important, now at
> http://wikimediafoundation.org/wiki/Resolution:Five-year_targets the
> second target, «Increase the amount of information we offer» considers
> only the number of Wikipedia articles.
> «We're aware of the challenges around bot-created articles, articles of
> low quality, etc., and the limited focus on Wikipedia, so this metric
> shouldn't be seen in isolation, but is an important indicator.» Yes, but
> a wrong one.
>
> I'm, very, very disappointed: I have to conclude that all the words on
> community participation etc. were only empty rhetoric.

This was a concious decision and I believe it is explained in the FAQs
or somewhere (Sue certainly mentioned it in at least one of the
(many!) presentions I've seen her do about the plan - there are slides
for those somewhere too). In summary (from memory), the reason was
basically one of "bang for your buck". The vast majority of our users
are using Wikipedia and not the other projects, which means even a
small improvement to Wikipedia is likely to have more impact than even
a large improvement to one of the other projects. Sue was very clear
that prioritising Wikipedia only applies to the WMF. The community
can, and should, continue to improve the other projects, the WMF just
feels that its limited resources are better used where they will have
more impact.

_______________________________________________
foundation-l mailing list
foundation-l@lists.wikimedia.org
Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/foundation-l
Re: Five-year WMF targets exclude non-Wikipedia projects [ In reply to ]
It is good to see discussion of the targets. There is also a final
strategic plan document, which is almost finished and which the Board
reviewed at our meeting over the weekend. There were small wording
changes in the final plan.


Mike Peel wrote:
> I think one of the major benefits of the strategy exercise was
> to get Wikimedians considering where Wikimedia should be in 5 years
> and setting their individual aims accordingly. Getting the WMF Board to
> recognise those aims is only a secondary consideration, really, as it's the
> community that drives Wikimedia's success and breadth/depth/etc. of content.

Yes, this may have been the greatest benefit to the process. I would
say "setting individual *and* project aims accordingly" -- every
project (both sister Projects in general and individual language
projects) are encouraged to come up with their own targets and
priorities for the next five years.


> On Sun, Oct 10, 2010, Federico Leva (Nemo)
> <nemowiki@gmail.com> wrote:
>> Despite repeated assurances at Wikimania, on lists and on strategywiki,
>> that the strategic plan was going to consider all Wikimedia projects as
>> important, now at
>> http://wikimediafoundation.org/wiki/Resolution:Five-year_targets the
>> second target, «Increase the amount of information we offer» considers
>> only the number of Wikipedia articles.

The problems with focusing only on Wikipedia articles were noted. The
text of this target in the final strategic plan refers to growth in
articles available 'on Wikimedia projects', not just on Wikipedia.
This is still only a very rough estimate of growth in meaningfully
available knowledge. [1]

The target does not say anything about the growth of Commons, though
this shows up elsewhere in strategy discussions. I hope the Commons
community will develop its own targets and priorities for growth, of
both its collections and its community.


On Sun, Oct 10, 2010 at 11:20 PM, John Vandenberg <jayvdb@gmail.com> wrote:
> I am likewise disappointed.  The five year plan _should_ have seen the
> other projects as the most likely source of new talent, contributors
> and innovation, and should have focused on developing them.

That is how I see the focus on innovation, by the way, including
"other users of MediaWiki" along with "the other projects".


> Even worse is the third target, which is _wrong_ because the
> Foundation hasn't included other projects in its considerations.  It
> says we don't have baselines for quality.  On Wikisource, we _do_ have
> empirical data on quality built into in the Proofreading system.
< http://wikisource.org/wiki/Wikisource:ProofreadPage_Statistics

True, and a number of the largest Wikipedias also have quality
measures used in their article assessments. But there is no baseline
that applies across all projects, as many do not have such measures.

The third target is not fully specified yet -- an important part of
this focus will be identifying and visualizing quality measures on
different projects, connecting them to one another, and sharing good
practices.


> Also, how was the 100,000 contributors per month with 5+ edits
> calculated?  Using simple addition of monthly counts for each project,
> I can only see 90,000 contributors with >=5 edits.  The figure would
> be lower as the contributors to smaller projects usually intersects
> the contributors to the larger projects.

A good point, which also came up over the weekend. As I understand
it, the baseline may be updated in the strategic plan when it is
published, with the target remaining 'doubling by 2015' by a
consistent measure.

Sam.


[1] It is worth noting that Wikipedia, thanks to having the
preponderance of editors and traffic, is sometimes used as a casual
shorthand for the effective size of Wikimedia, even within our
community. This is a skewing of focus that requires effort to
overcome -- but the effort is worth it, as Wikipedia alone will not
fulfill our mission. Better communication about the sister projects'
work and news may help.

This holds true for public discussions as well. I was speaking at a
library conference last week, and mentioned Wikisource. A librarian
interrupted with enthusiasm, "there's also a Wikisource?" and later
had ideas about how to contribute digitizations. Many potential
partners in disseminating knowledge may be able to contribute directly
to one of our projects, but not the others.

_______________________________________________
foundation-l mailing list
foundation-l@lists.wikimedia.org
Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/foundation-l
Re: Five-year WMF targets exclude non-Wikipedia projects [ In reply to ]
Hi folks,

Sorry to have been absent from this discussion thus far: I didn't
realize Sam was going to post the targets when he did, and so I am
playing a little catch-up here.

Below are some questions and answers re the targets that might be
helpful for the discussion. (Erik wrote most of this, and I've just
now added a few bits.) If you read this and there are still issues
that you want addressed, please just say so :-)

Thanks,
Sue



http://wikimediafoundation.org/wiki/Resolution:Five-year_targets

What’s the purpose of these five-year targets?

The July 2015 targets approved by the Board represent [[big hairy
audacious goal]]s for Wikimedia. They are intended to reflect
Wikimedia’s core mission -- “to empower and engage people around the
world to collect and develop free educational content and to
disseminate it effectively and globally” -- and to capture it in terms
of concrete, ambitious outcomes that are very hard --but ideally not
impossible-- to achieve.

The development of very ambitious five-year targets is intended to
help inspire and focus energy across the Wikimedia movement, in
connection with the priorities identified through Wikimedia’s
strategic planning process. They’re also intended to help persuade
people who aren’t yet active members of the Wikimedia movement (for
example, readers who don’t yet edit or donate, grant-making
institutions, “GLAM” organizations) to get involved and support us.

How do you expect these targets to be used?

These targets will be part of the printed “summary” version of the
strategy plan that we will distribute internally and to external
interested parties -- e.g., Wikipedia Academy attendees, grant-making
institutions, etc. We will give out the document, including the
targets, so that people understand where the Wikimedia movement is
focusing its energy, and how they can help.

The targets will also help us, we hope, to focus our own work, and
measure whether we’re being successful. In the context of an overall
Wikimedia dashboard, we can begin to highlight if key performance
indicators are significantly deviating from our expectations -- e.g.
if overall article growth flattens, or the number of editors declines.

Are these the only targets used by the Wikimedia Foundation?

No. These five targets are called out as very ambitious long-term
outcomes which, if we achieve them together, will indicate that
Wikimedia has made great strides in serving its mission. They are
intentionally high level, focused on information and the people who
develop and receive it, as opposed to operations.

There are other key performance indicators which we must examine on an
ongoing basis, including but not limited to:
* engagement and retention of the editor community
* site uptime and load times in different geographies
* financial health of the Wikimedia movement
* availability of secure off-site copies of all data
* number and quality of multimedia files in all our projects
* number and quality of information in Wikimedia’s other projects
* demographic composition of the editor community
* our collective ability to develop and operationalize innovative technology

The Wikimedia report-card, at http://stats.wikimedia.org/reportcard/ ,
is our primary instrument for tracking key performance indicators; its
formats and the indicators which are included are still evolving. The
Wikimedia report-card is much more detailed than the five-year
targets: it tracks more information at a more granular level, more
frequently.

How were these targets developed?

The development of five-year targets was the last major piece of work
that needed to be concluded before the strategy project wrapped up.

To that end, the targets were developed primarily by the staff of the
Wikimedia Foundation (mainly Sue Gardner, Erik Moeller, and Barry
Newstead), for Board approval. The first source for material was the
discussions held on the strategy wiki, focused on performance
indicators and goals (e.g.,
<http://strategy.wikimedia.org/wiki/Strategic_Plan/Movement_Priorities>).
Another source was the set of internal metrics that the Wikimedia
community has used for a long time, including “number of articles” and
“active editors”, as well as first baseline estimates for gender
participation and country-of-origin breakdown that have been
established more recently through surveys, log analysis, and other
methods.

People’s general views about target-setting, as well as their
assessment of the value of different possible measures, were surfaced
through surveys of foundation-l and internal-l readers, Advisory Board
members, Board members and staff members [1]. That helped shape both
our general approach to target-setting, and the actual measures and
numbers we wanted to use. Once a draft set of targets was created, Sue
gave it to the Board, where it was discussed at some length and then
approved.

[1] See Sue’s blog post at
<http://suegardner.org/2010/08/16/how-wikimedia-will-measure-success-over-the-next-five-years/>
for some background.

Can these targets be modified?

While other key performance indicators (see above) are intended to be
flexibly adjusted as we better understand which indicators are useful
and which ones aren’t, the five-year targets voted on by the Board are
intended to be stable. That’s why they are so high-level. The
underlying measurements and methodologies for each target will
continue to be refined over time -- more so in categories where we
lack clear and consistent baselines, such as article quality.

Why was “number of Wikipedia articles” used as a target for “amount of
information we offer”?

We were looking for a single, clearly understandable indicator that we
weren’t just reaching more people, growing our community and
increasing its diversity, and improving perceived quality of content,
but that we were also successfully expanding the breadth and depth of
information available to readers.

We fully understand that the number of Wikipedia articles is an
imperfect measure -- it’s especially imperfect if taken by itself. On
the other hand, it’s fairly well understood what a Wikipedia article
is, what it isn’t, and what the potential challenges with counting
Wikipedia articles are (such as mass-creation of articles based on
some data source): we’ve used article counts in Wikipedia since the
project started, and have developed a collective expertise how to
manage and how to interpret them.

Taken by itself, the number of Wikipedia articles answers one
question, and one question only: How likely is it that Wikimedia’s
flagship project, Wikipedia, is going to have any information on a
given term? It doesn’t answer whether the information is useful, of
high quality, or even whether it should be in Wikipedia at all. But
the other targets help to answer that question: We are measuring
whether we are increasing the reach of our projects, the perceived
quality of the information provided, and the number and diversity of
contributors. If we succeed along all those dimensions, the number of
Wikipedia articles is a useful indicator of our overall breadth and
depth. This is not to diminish the work done in other categories of
content (such as media files), or the work done in Wikipedia’s sister
projects. The “number of Wikipedia articles” is expected to be useful
as an _indicator_ of the overall amount of information we offer: that
is its primary usefulness.

What does a “25% increase in quality” actually mean?

We’ve not yet established which assessment systems will be the most
useful, scalable, and effective, and until we have done so, this
measure should be seen principally as a placeholder for a large,
ambitious, relative increase in perceived quality of a large number of
assessed articles over the five-year time period (most simply, in the
context of a scoring system like
<http://www.mediawiki.org/wiki/Article_feedback/Public_Policy_Pilot>,
a 25% increase of average scores for assessed articles over the
five-year time period).

Wikimedia projects to-date have primarily used internal measures of
quality (“good articles”, “featured articles”, patrolled edits,
WikiProject classification, etc.). These measures are of critical
importance to the organization of Wikimedia’s editorial efforts. At
the same time, they only measure what we, ourselves, think about the
quality of the product that we’ve built. It’s equally important that
we obtain measures of what others think about the quality provided --
this includes our audience, but it also includes individuals with
verifiable expertise regarding the subject matter domain(s) an article
relates to.



--
Sue Gardner
Executive Director
Wikimedia Foundation

415 839 6885 office
415 816 9967 cell

Imagine a world in which every single human being can freely share in
the sum of all knowledge.  Help us make it a reality!

http://wikimediafoundation.org/wiki/Donate

_______________________________________________
foundation-l mailing list
foundation-l@lists.wikimedia.org
Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/foundation-l
Re: Five-year WMF targets exclude non-Wikipedia projects [ In reply to ]
Hello,

2010/10/12 Thomas Dalton <thomas.dalton@gmail.com>:
> On 10 October 2010 09:33, Federico Leva (Nemo) <nemowiki@gmail.com> wrote:
>> Despite repeated assurances at Wikimania, on lists and on strategywiki,
>> that the strategic plan was going to consider all Wikimedia projects as
>> important, now at
>> http://wikimediafoundation.org/wiki/Resolution:Five-year_targets the
>> second target, «Increase the amount of information we offer» considers
>> only the number of Wikipedia articles.
>> «We're aware of the challenges around bot-created articles, articles of
>> low quality, etc., and the limited focus on Wikipedia, so this metric
>> shouldn't be seen in isolation, but is an important indicator.» Yes, but
>> a wrong one.
>>
>> I'm, very, very disappointed: I have to conclude that all the words on
>> community participation etc. were only empty rhetoric.
>
> This was a concious decision and I believe it is explained in the FAQs
> or somewhere (Sue certainly mentioned it in at least one of the
> (many!) presentions I've seen her do about the plan - there are slides
> for those somewhere too). In summary (from memory), the reason was
> basically one of "bang for your buck". The vast majority of our users
> are using Wikipedia and not the other projects, which means even a
> small improvement to Wikipedia is likely to have more impact than even
> a large improvement to one of the other projects.

That's an unproven assumption. It might even be the opposite, i.e.
reinforcing Wikipedia might only increase the gap between the
projects.

> Sue was very clear
> that prioritising Wikipedia only applies to the WMF.

That's a bad decision. The WMF should try to balance the projects,
when the community has not done it alone.

> The community
> can, and should, continue to improve the other projects, the WMF just
> feels that its limited resources are better used where they will have
> more impact.

Regards,

Yann

_______________________________________________
foundation-l mailing list
foundation-l@lists.wikimedia.org
Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/foundation-l
Re: Five-year WMF targets exclude non-Wikipedia projects [ In reply to ]
> > The vast majority of our users
> > are using Wikipedia and not the other projects, which means even a
> > small improvement to Wikipedia is likely to have more impact than even
> > a large improvement to one of the other projects.
>
> That's an unproven assumption. It might even be the opposite, i.e.
> reinforcing Wikipedia might only increase the gap between the
> projects.
>

I definitely agree with Yann. The topic is complex, I think that both
assumptions are someway true. Surely, more people using Wikipedia, more
potential viewers on links to sisterprojects, thus more potential users. But
the more Wikipedia is huge, the more it will steal attention form other wiki
project. I'm sure the everyone of you struggle explaining people that
Wiki*m*edia is not Wikipedia, and viceversa. Why this should not apply to
sister projects too?


> > Sue was very clear
> > that prioritising Wikipedia only applies to the WMF.
>

I was the one who raised his hand during Sue's presentation in Gdansk,
and, as far as I understood, she agreed in mentioning sister-projects in the
targets. If the new slogan of the WMF shifted from "Let it happen" from
"Make it happen",
well, this should be the case.


> > The community
> > can, and should, continue to improve the other projects, the WMF just
> > feels that its limited resources are better used where they will have
> > more impact.
>

I understand this, but we are not asking the 90% of the limited resources,
just a recognition that we exist and we are doing our best to create useful
and reliable (free) content.

Aubrey
WM Italy
_______________________________________________
foundation-l mailing list
foundation-l@lists.wikimedia.org
Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/foundation-l