Mailing List Archive

Office Hours on the article feedback tool
Hey guys

So, on Thursday we're going to be holding an Office Hours session on IRC to
discuss the Article Feedback Tool and what we're planning to do with it -
namely, scrapping it and replacing it with an entirely new version ;).
Attending will be Fabrice Florin, the contractor leading development on the
new version, Howie Fung, the WMF's product manager, and myself. If you're
interested in the AFT, whether because you think the existing version is
good or because you think it's really bad, we'd love for you to attend -
every opinion and viewpoint is welcome. The session will be held in
#wikimedia-office at 19:00 GMT/UTC, and I hope to see a lot of you there :).

Thanks

Oliver Keyes
Community Liason, Product Development
Wikimedia Foundation
_______________________________________________
foundation-l mailing list
foundation-l@lists.wikimedia.org
Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/foundation-l
Re: Office Hours on the article feedback tool [ In reply to ]
On 26 October 2011 11:04, Oliver Keyes <scire.facias@gmail.com> wrote:

> So, on Thursday we're going to be holding an Office Hours session on IRC to
> discuss the Article Feedback Tool and what we're planning to do with it -
> namely, scrapping it and replacing it with an entirely new version ;).


*slaps own forehead*

So is the data to be thrown away too?

(Is there anywhere to look up the data en masse?)


- d.

_______________________________________________
foundation-l mailing list
foundation-l@lists.wikimedia.org
Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/foundation-l
Re: Office Hours on the article feedback tool [ In reply to ]
No, the data will remain; you can find it at
http://toolserver.org/~catrope/articlefeedback/ (we really need to advertise
that more widely, actually).

To be clear, we're not talking about junking the idea; we will still have an
"Article Feedback Tool" that lets readers provide feedback to editors. The
goal is more to move away from a subjective rating system, and towards
something the editors can look at and go "huh, that's a reasonable
suggestion as to how to fix the article, I'll go do that" or "aw, that's
really nice! I'm glad they liked it so much"

O.

On Wed, Oct 26, 2011 at 11:09 AM, David Gerard <dgerard@gmail.com> wrote:

> On 26 October 2011 11:04, Oliver Keyes <scire.facias@gmail.com> wrote:
>
> > So, on Thursday we're going to be holding an Office Hours session on IRC
> to
> > discuss the Article Feedback Tool and what we're planning to do with it -
> > namely, scrapping it and replacing it with an entirely new version ;).
>
>
> *slaps own forehead*
>
> So is the data to be thrown away too?
>
> (Is there anywhere to look up the data en masse?)
>
>
> - d.
>
> _______________________________________________
> foundation-l mailing list
> foundation-l@lists.wikimedia.org
> Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/foundation-l
>
_______________________________________________
foundation-l mailing list
foundation-l@lists.wikimedia.org
Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/foundation-l
Re: Office Hours on the article feedback tool [ In reply to ]
On Wed, 26 Oct 2011 11:04:23 +0100, Oliver Keyes <scire.facias@gmail.com>
wrote:
> Hey guys
>
> So, on Thursday we're going to be holding an Office Hours session on IRC
to
> discuss the Article Feedback Tool and what we're planning to do with it
-
> namely, scrapping it and replacing it with an entirely new version ;).
> Attending will be Fabrice Florin, the contractor leading development on
the
> new version, Howie Fung, the WMF's product manager, and myself. If
you're
> interested in the AFT, whether because you think the existing version is
> good or because you think it's really bad, we'd love for you to attend -
> every opinion and viewpoint is welcome. The session will be held in
> #wikimedia-office at 19:00 GMT/UTC, and I hope to see a lot of you there
> :).
>

I personally can not attend but I guess there are many users interested in
the topic, and some brief update on a dedicated Meta page (or even
publication of the IRC log) would be much appreciated.

Cheers
Yaroslav

_______________________________________________
foundation-l mailing list
foundation-l@lists.wikimedia.org
Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/foundation-l
Re: Office Hours on the article feedback tool [ In reply to ]
On Wed, Oct 26, 2011 at 12:57 PM, Yaroslav M. Blanter <putevod@mccme.ru>wrote:

>
>
>
> I personally can not attend but I guess there are many users interested in
> the topic, and some brief update on a dedicated Meta page (or even
> publication of the IRC log) would be much appreciated.
>
> Cheers
> Yaroslav
>
> Sure; we're sticking some content up at the moment, and I'll be sure to
log the hours (and then stick them up at the usual location

Oliver
_______________________________________________
foundation-l mailing list
foundation-l@lists.wikimedia.org
Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/foundation-l
Re: Office Hours on the article feedback tool [ In reply to ]
On Wed, Oct 26, 2011 at 11:09, David Gerard <dgerard@gmail.com> wrote:
> *slaps own forehead*
>
> So is the data to be thrown away too?
>
> (Is there anywhere to look up the data en masse?)
>

It's all on the Toolserver and should be in the dumps too.

If you have any specific requirements for retrieving certain subsets
of the data, do ask and someone with Toolserver access can run queries
against the data and provide the results.

--
Tom Morris
<http://tommorris.org/>

_______________________________________________
foundation-l mailing list
foundation-l@lists.wikimedia.org
Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/foundation-l
Re: Office Hours on the article feedback tool [ In reply to ]
On 26 October 2011 06:04, Oliver Keyes <scire.facias@gmail.com> wrote:

> Hey guys
>
> So, on Thursday we're going to be holding an Office Hours session on IRC
> ..<snip>...The session will be held in
> #wikimedia-office at 19:00 GMT/UTC, and I hope to see a lot of you there
> :).
>
>
I realise that sometimes it is a challenge to arrange these Office Hours in
a more spread out way; however, of the last five sessions (including this
one), four of them have occurred at a time that severely limits
participation from North and South American editors, as they come during our
"business day". This particular topic area is very much of interest to our
N/S American editors who work on projects where the Article Feedback Tool is
in use and has raised concerns, and ones where the "new and improved"
version will be placed.

I'd very much urge trying to spread out the time of Office Hours generally.
I'd also like to suggest consideration be given to doing a "double" office
hour session for topic areas that impact projects globally and involve
editors from just about every time zone. Reading IRC "minutes" is not the
same as being involved in the discussion.

Thanks!

Risker
_______________________________________________
foundation-l mailing list
foundation-l@lists.wikimedia.org
Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/foundation-l
Re: Office Hours on the article feedback tool [ In reply to ]
On Wed, Oct 26, 2011 at 1:59 PM, Risker <risker.wp@gmail.com> wrote:

> On 26 October 2011 06:04, Oliver Keyes <scire.facias@gmail.com> wrote:
>
> > Hey guys
> >
> > So, on Thursday we're going to be holding an Office Hours session on IRC
> > ..<snip>...The session will be held in
> > #wikimedia-office at 19:00 GMT/UTC, and I hope to see a lot of you there
> > :).
> >
> >
> I realise that sometimes it is a challenge to arrange these Office Hours in
> a more spread out way; however, of the last five sessions (including this
> one), four of them have occurred at a time that severely limits
> participation from North and South American editors, as they come during
> our
> "business day". This particular topic area is very much of interest to our
> N/S American editors who work on projects where the Article Feedback Tool
> is
> in use and has raised concerns, and ones where the "new and improved"
> version will be placed.
>
> I'd very much urge trying to spread out the time of Office Hours generally.
> I'd also like to suggest consideration be given to doing a "double" office
> hour session for topic areas that impact projects globally and involve
> editors from just about every time zone. Reading IRC "minutes" is not the
> same as being involved in the discussion.
>
> Thanks!
>
> Risker
> _______________________________________________
> foundation-l mailing list
> foundation-l@lists.wikimedia.org
> Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/foundation-l
>
That's certainly true - I'll see if we can hold a second and more N/SA
orientated one next week :).
_______________________________________________
foundation-l mailing list
foundation-l@lists.wikimedia.org
Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/foundation-l
Office Hours on the article feedback tool [ In reply to ]
> ------------------------------
>
> Message: 6
> Date: Wed, 26 Oct 2011 11:11:57 +0100
> From: Oliver Keyes <scire.facias@gmail.com>
> Subject: Re: [Foundation-l] Office Hours on the article feedback tool
> To: Wikimedia Foundation Mailing List
> <foundation-l@lists.wikimedia.org>
> Message-ID:
> <CAPYupWA34CujYan_vV_cHgYxWfCT3EJnb4d-Nrav_U20QejZ1A@mail.gmail.com
> >
> Content-Type: text/plain; charset=ISO-8859-1
>
> No, the data will remain; you can find it at
> http://toolserver.org/~catrope/articlefeedback/ (we really need to
> advertise
> that more widely, actually).
>
> To be clear, we're not talking about junking the idea; we will still have
> an
> "Article Feedback Tool" that lets readers provide feedback to editors. The
> goal is more to move away from a subjective rating system, and towards
> something the editors can look at and go "huh, that's a reasonable
> suggestion as to how to fix the article, I'll go do that" or "aw, that's
> really nice! I'm glad they liked it so much"
>
> O.
>
>
As someone who was never exactly a fan of the Article Feedback Tool I'm glad
to hear that the current version is to be canned. The sort of subjective
ratings it could produce were never going to be useful at improving
articles, certainly not useful enough to justify the screen space. My fear
was that it might divert people from improving articles to complaining about
them. Since we skipped a key stage in the testing we will never know whether
it did that. I didn't realise at the time that it was going to abuse our
readers trust by collecting shed loads of data that we weren't going to use.

We took a big risk in implementing the Article Feedback Tool without first
testing to see whether it would do more harm than good. It is hard to tell
in hindsight whether it has been negative or neutral in effect. Yes
recruitment of new editors has fallen sharply - September's new editors on
EN wiki are down to levels not seen since 2005
http://stats.wikimedia.org/EN/TablesWikipediaEN.htm#editdistribution but
things were on the decline anyway so we don't know whether and to what
extent the Article Feedback tool exacerbated the trend. My concern about
turning it into something that collects more meaningful comments is that
this could exacerbate the pernicious trend from improving articles to
tagging them for others to improve. I appreciate that there are various
competing theories as to why the community went off the boil circa 2007, but
for me and anyone else who considers that the trend to template rather than
improve articles has been a major cause of community decline, an "improved"
version of the Article Feedback Tool is a worrying prospect.

Can we make sure that any new generation Article Feedback tool is properly
tested, and that testing includes:

1. Implementing it on a random group of articles and comparing them with
a control sample to see which group of articles had the more edits from
newbies;
2. Whether the collecting of feedback on ways to improve the article
generates additional comments or diverts some editors away from actually
fixing the article.
3. Which group of articles recruited the most new editors to the pedia.

Please don't implement it if the testing shows that it diverts people from
fixing articles to pointing out things that others can fix.

On a broader note I suggested some time ago that for the community to give
meaningful input into article development we need a process for the
community to give feedback on the priority of various potential
developments. Wikimania does something like that in the way the program is
put together. The image filter "referendum" came close in that it asked
people to rate the image filter for importance, unfortunately it didn't
include other proposals so that people could put them in order of relevant
importance (we also need a quite separate question for whether you think
something is worth doing at all). In your new role as liaison between the
community and the development team please could you initiate something like
that, so that those of us who would give a higher priority to global
watchlists or enhancing catalot so that it works on uncategorised articles
can say so?

Regards

WereSpielChequers
_______________________________________________
foundation-l mailing list
foundation-l@lists.wikimedia.org
Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/foundation-l
Re: Office Hours on the article feedback tool [ In reply to ]
On Wed, 26 Oct 2011 15:33:30 +0100, WereSpielChequers
<werespielchequers@gmail.com> wrote:

>
> Can we make sure that any new generation Article Feedback tool is
properly
> tested, and that testing includes:
>
> 1. Implementing it on a random group of articles and comparing them
> with
> a control sample to see which group of articles had the more edits
from
> newbies;
> 2. Whether the collecting of feedback on ways to improve the article
> generates additional comments or diverts some editors away from
actually
> fixing the article.
> 3. Which group of articles recruited the most new editors to the
pedia.
>
> Please don't implement it if the testing shows that it diverts people
from
> fixing articles to pointing out things that others can fix.
>

And I think this time showing it to the Research Committee prior to
running the tests would be a good idea.

Cheers
Yaroslav

_______________________________________________
foundation-l mailing list
foundation-l@lists.wikimedia.org
Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/foundation-l
Re: Office Hours on the article feedback tool [ In reply to ]
On Wed, Oct 26, 2011 at 3:46 PM, Yaroslav M. Blanter <putevod@mccme.ru>wrote:

> On Wed, 26 Oct 2011 15:33:30 +0100, WereSpielChequers
> <werespielchequers@gmail.com> wrote:
>
> >
> > Can we make sure that any new generation Article Feedback tool is
> properly
> > tested, and that testing includes:
> >
> > 1. Implementing it on a random group of articles and comparing them
> > with
> > a control sample to see which group of articles had the more edits
> from
> > newbies;
> > 2. Whether the collecting of feedback on ways to improve the article
> > generates additional comments or diverts some editors away from
> actually
> > fixing the article.
> > 3. Which group of articles recruited the most new editors to the
> pedia.
> >
> > Please don't implement it if the testing shows that it diverts people
> from
> > fixing articles to pointing out things that others can fix.
> >
>
> And I think this time showing it to the Research Committee prior to
> running the tests would be a good idea.
>
> Cheers
> Yaroslav
>
> I'll bring these points up with the folks :). If you have any others, do
come to office-hours.

O.

> _______________________________________________
>
_______________________________________________
foundation-l mailing list
foundation-l@lists.wikimedia.org
Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/foundation-l
Re: Office Hours on the article feedback tool [ In reply to ]
On Wed, Oct 26, 2011 at 3:11 AM, Oliver Keyes <scire.facias@gmail.com> wrote:
> To be clear, we're not talking about junking the idea; we will still have an
> "Article Feedback Tool" that lets readers provide feedback to editors. The
> goal is more to move away from a subjective rating system, and towards
> something the editors can look at and go "huh, that's a reasonable
> suggestion as to how to fix the article, I'll go do that" or "aw, that's
> really nice! I'm glad they liked it so much"

And, the idea is to experiment with some alternative approaches in
parallel with the existing deployment, not to scrap the existing
deployment and start over immediately. We don't know yet which reader
feedback mechanisms are going to be the most useful to meet the two
core objectives (engaging readers as much and as usefully as possible
in article development, and measuring change-over-time in quality).
Initial wireframes to be tested against each other can be found here:

http://www.mediawiki.org/wiki/Article_feedback/Version_5/Feature_Requirements
--
Erik Möller
VP of Engineering and Product Development, Wikimedia Foundation

Support Free Knowledge: http://wikimediafoundation.org/wiki/Donate

_______________________________________________
foundation-l mailing list
foundation-l@lists.wikimedia.org
Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/foundation-l
Re: Office Hours on the article feedback tool [ In reply to ]
The AFT v.4 data is documented here: http://www.mediawiki.org/wiki/Article_feedback/Data

Dario

On Oct 26, 2011, at 5:53 AM, Tom Morris wrote:

> On Wed, Oct 26, 2011 at 11:09, David Gerard <dgerard@gmail.com> wrote:
>> *slaps own forehead*
>>
>> So is the data to be thrown away too?
>>
>> (Is there anywhere to look up the data en masse?)
>>
>
> It's all on the Toolserver and should be in the dumps too.
>
> If you have any specific requirements for retrieving certain subsets
> of the data, do ask and someone with Toolserver access can run queries
> against the data and provide the results.
>
> --
> Tom Morris
> <http://tommorris.org/>
>
> _______________________________________________
> foundation-l mailing list
> foundation-l@lists.wikimedia.org
> Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/foundation-l


_______________________________________________
foundation-l mailing list
foundation-l@lists.wikimedia.org
Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/foundation-l
Re: Office Hours on the article feedback tool [ In reply to ]
Hi WereSpielChequers,

I worked on the data analysis for previous AFT versions and I believe I've already answered on a number of occasions your questions as to what we could test and what we couldn't in the previous phase, but I am happy to do this here and clarify what the research plans for the next version are.

Subjective ratings

We have definitely seen a lot of love/hate rating happen in the case of popular articles (e.g. Lady Gaga, Justin Bieber). Teasing apart ratings on the quality of the article and rater attitudes towards the topic of the article is pretty hard given the fact that an average enwiki article gets a very small number of ratings per day and articles that get a sufficient number of ratings tend to be attracting particularly opinionated or polarized visitors.

To give you a measure of the problem: of the 3.7M articles in the main namespace of the English Wikipedia only 40 articles (0.001%) obtain 10 or more ratings per day. The vast majority of articles don't get any rating for several days or weeks or ever. FInding ways to increase the volume of ratings per article is one of the issues we're discussing in the context of v.5.

The second problem is that we don't have enough observations on multiple ratings by the same user. Only 0.02% of unique raters rate more than one article and that means that on a single article basis we cannot easily filter out users who only rated a topic they love or hate and still have enough good data to process. This is unfortunate: the more rating data we can get per rater, the more we can identify gaming or rating biases and control them in public article feedback reports.

Effects of AFT on participations

I ran a number of pre/post analyses comparing editing activity before and after AFT was activated on a random sample of English Wikipedia articles, controlling for page views before and after the activation and found no statistically significant difference in the volume of edits. As I noted elsewhere the comparison between two random samples of articles is problematic because we cannot easily control for the multiple factors that affect editing activity in independent samples of articles so any result you may get out of this coarse analysis would be questionable. I agree that's a very important issue and the proper way to address it is by a/b testing different AFT interfaces (including no AFT widget whatsoever) for the same article and measuring the effects on edit activity for the same articles across different user groups: this is one of the plans we are considering for v.5

Another important limitation of AFT v.4 is that we only collected aggregate event counts for call to actions and we didn't mark edits or new accounts created via AFT, which means that we couldn't directly study the effects of AFT as an on-ramping tool for new editors (e.g. how many readers it is converting to registered users and what is the quality of edits generated via the AFT. i.e. how many users who create an account via AFT call to actions actually end up becoming editors? What is their survival compared to users who create an account in a standard way? And how many among the edits created via AFT are vandalism? How many are good faith tests that get reverted? These are all questions that we will be addressing as of v.5.

We'll be still working on analyzing the current AFT data to support the design of v.5. In particular, we will be focusing on (1) correlations between consistent low ratings and poor quality or vandalism or the likelihood of an article to be nominated for deletion and (2) the relation between ratings and changes in other quality-related metrics on a per-article basis.

I have also pitched the existing data to a number of external researchers interesting in article quality measurements and/or rating systems and I invite you to do the same.

Hope this helps. I look forward to a more in-depth discussion during the office hours.

Dario

On Oct 26, 2011, at 7:33 AM, WereSpielChequers wrote:

>> ------------------------------
>>
>> Message: 6
>> Date: Wed, 26 Oct 2011 11:11:57 +0100
>> From: Oliver Keyes <scire.facias@gmail.com>
>> Subject: Re: [Foundation-l] Office Hours on the article feedback tool
>> To: Wikimedia Foundation Mailing List
>> <foundation-l@lists.wikimedia.org>
>> Message-ID:
>> <CAPYupWA34CujYan_vV_cHgYxWfCT3EJnb4d-Nrav_U20QejZ1A@mail.gmail.com
>>>
>> Content-Type: text/plain; charset=ISO-8859-1
>>
>> No, the data will remain; you can find it at
>> http://toolserver.org/~catrope/articlefeedback/ (we really need to
>> advertise
>> that more widely, actually).
>>
>> To be clear, we're not talking about junking the idea; we will still have
>> an
>> "Article Feedback Tool" that lets readers provide feedback to editors. The
>> goal is more to move away from a subjective rating system, and towards
>> something the editors can look at and go "huh, that's a reasonable
>> suggestion as to how to fix the article, I'll go do that" or "aw, that's
>> really nice! I'm glad they liked it so much"
>>
>> O.
>>
>>
> As someone who was never exactly a fan of the Article Feedback Tool I'm glad
> to hear that the current version is to be canned. The sort of subjective
> ratings it could produce were never going to be useful at improving
> articles, certainly not useful enough to justify the screen space. My fear
> was that it might divert people from improving articles to complaining about
> them. Since we skipped a key stage in the testing we will never know whether
> it did that. I didn't realise at the time that it was going to abuse our
> readers trust by collecting shed loads of data that we weren't going to use.
>
> We took a big risk in implementing the Article Feedback Tool without first
> testing to see whether it would do more harm than good. It is hard to tell
> in hindsight whether it has been negative or neutral in effect. Yes
> recruitment of new editors has fallen sharply - September's new editors on
> EN wiki are down to levels not seen since 2005
> http://stats.wikimedia.org/EN/TablesWikipediaEN.htm#editdistribution but
> things were on the decline anyway so we don't know whether and to what
> extent the Article Feedback tool exacerbated the trend. My concern about
> turning it into something that collects more meaningful comments is that
> this could exacerbate the pernicious trend from improving articles to
> tagging them for others to improve. I appreciate that there are various
> competing theories as to why the community went off the boil circa 2007, but
> for me and anyone else who considers that the trend to template rather than
> improve articles has been a major cause of community decline, an "improved"
> version of the Article Feedback Tool is a worrying prospect.
>
> Can we make sure that any new generation Article Feedback tool is properly
> tested, and that testing includes:
>
> 1. Implementing it on a random group of articles and comparing them with
> a control sample to see which group of articles had the more edits from
> newbies;
> 2. Whether the collecting of feedback on ways to improve the article
> generates additional comments or diverts some editors away from actually
> fixing the article.
> 3. Which group of articles recruited the most new editors to the pedia.
>
> Please don't implement it if the testing shows that it diverts people from
> fixing articles to pointing out things that others can fix.
>
> On a broader note I suggested some time ago that for the community to give
> meaningful input into article development we need a process for the
> community to give feedback on the priority of various potential
> developments. Wikimania does something like that in the way the program is
> put together. The image filter "referendum" came close in that it asked
> people to rate the image filter for importance, unfortunately it didn't
> include other proposals so that people could put them in order of relevant
> importance (we also need a quite separate question for whether you think
> something is worth doing at all). In your new role as liaison between the
> community and the development team please could you initiate something like
> that, so that those of us who would give a higher priority to global
> watchlists or enhancing catalot so that it works on uncategorised articles
> can say so?
>
> Regards
>
> WereSpielChequers
> _______________________________________________
> foundation-l mailing list
> foundation-l@lists.wikimedia.org
> Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/foundation-l

_______________________________________________
foundation-l mailing list
foundation-l@lists.wikimedia.org
Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/foundation-l