Mailing List Archive

Fwd: [WikimediaMobile] Number crunching: Upload errors on mobile
Since nobody reads mobile-l


---------- Forwarded message ----------
From: Jon Robson <jdlrobson@gmail.com>
Date: Sat, Jun 8, 2013 at 4:17 AM
Subject: [WikimediaMobile] Number crunching: Upload errors on mobile
To: mobile-l <mobile-l@lists.wikimedia.org>


I ran some data crunching on a sample of 4429 photo uploads from
mobile web. In this sample 2821 uploads succeeded and 36% (1608/4429)
of attempted uploads failed. This is very high and unacceptable.

Looking closely 53% of all errors were due to problems with invalid or
'anonymous' tokens. This will occur when the client is unable to get a
token using CORS from Commons due to not being logged in there.
Luckily Chris Steipp and the rest of the platform team have pushed a
change that should significantly reduce this error:
https://gerrit.wikimedia.org/r/#/c/57662/

CentralAuth related errors (861)
####################################
Anonymous token. 598
Invalid token 263

The next big offender was 'Missing filename' accounting for 22% of all
upload errors. Unfortunately this is ambiguous as it could mean a
variety of things - it simply means that an upload was attempted and
the response didn't report the filename. I've pushed a patch to try to
understand what errors we are running into:
https://gerrit.wikimedia.org/r/67545

Other errors (467)
###################################
Missing filename 347
This file did not pass file verification 69
Blank error message given 36
The file you submitted was empty 15

There are various other errors all listed below for your enjoyment.
Some due to bad choices of name, permissions problems and attempts to
upload certain file types we do not accept. The good news is we
probably don't want these uploads to succeed as they hint at vandalism
attempts or uploads by poorly educated users.

The server problems section is worth a look though - although a small
percentage "The modification you tried to make was aborted by an
extension hook 61". These errors are occurring on the following wiki
projects:
* sv.m.wikipedia.org
* de.m.wikipedia.org
* test.m.wikipedia.org
* en.m.wikipedia.org
* ar.m.wikipedia.org
* es.m.wikipedia.org
* ja.m.wikipedia.org
* he.m.wikipedia.org
* fr.m.wikipedia.org
* nl.m.wikipedia.org
Any ideas what may be causing that error?

***
Other errors:

Users uploading with bad or unclear filenames (113)
####################################
"titleblacklist-custom-filename" 48
(https://commons.m.wikimedia.org/wiki/Template:Titleblacklist-custom-filename/en)
"titleblacklist-forbidden-edit" 37
(https://en.m.wikipedia.org/wiki/MediaWiki_talk:Titleblacklist-forbidden-edit)
Filename exists 25 (I suspect they used a common filename)
Unknown error: "titleblacklist-custom-double-apostrophe" 3

(Out of interest is there any API to check whether a filename will be accepted?)

Server problems (81):
####################################
The modification you tried to make was aborted by an extension hook 61
Database query error 10
An internal error occurred 9
error: Internal Server Error 1

Permission based errors (69)
####################################
The "autoconfirmed" right is required to edit this page 36
You have been blocked from editing 25
The "protect" right is required to edit this page 7
Unknown error: "globalblocking-ipblocked" 1

(These users should not be seeing the upload button!)

Users uploading with bad file types (17):
###########################################
Filetype not permitted: MOV 8
Filetype not permitted: webp 4
Filetype not permitted: mp3 3
Filetype not permitted: xml 1
Filetype not permitted: bmp 1

_______________________________________________
Mobile-l mailing list
Mobile-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mobile-l


--
Yuvi Panda T
http://yuvi.in/blog

_______________________________________________
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: Fwd: [WikimediaMobile] Number crunching: Upload errors on mobile [ In reply to ]
>
> The server problems section is worth a look though - although a small
> percentage "The modification you tried to make was aborted by an
> extension hook 61". These errors are occurring on the following wiki
> projects:
> * sv.m.wikipedia.org
> * de.m.wikipedia.org
> * test.m.wikipedia.org
> * en.m.wikipedia.org
> * ar.m.wikipedia.org
> * es.m.wikipedia.org
> * ja.m.wikipedia.org
> * he.m.wikipedia.org
> * fr.m.wikipedia.org
> * nl.m.wikipedia.org
> Any ideas what may be causing that error?

I suspect that is caused by UploadBlacklist extension, which
blacklists about 23 files by their sha hash. According to the config
file, there's a log at "udp://$wmfUdp2logDest/upload-blacklist", so
you can probably check if that guess is right.

--bawolff

_______________________________________________
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: Fwd: [WikimediaMobile] Number crunching: Upload errors on mobile [ In reply to ]
On 08.06.2013, 21:07 Brian wrote:


> I suspect that is caused by UploadBlacklist extension, which
> blacklists about 23 files by their sha hash. According to the config
> file, there's a log at "udp://$wmfUdp2logDest/upload-blacklist", so
> you can probably check if that guess is right.

$ grep -v 'MISS' upload-blacklist.log
$

--
Best regards,
Max Semenik ([[User:MaxSem]])


_______________________________________________
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: Fwd: [WikimediaMobile] Number crunching: Upload errors on mobile [ In reply to ]
Update on error numbers:

111 errors since we pushed the fix on Tuesday 2pm to avoid anonymous token
problems.

Things are looking much better and come under 3 types of error
Here is the new breakdown

1) Errors due to bad filenames / bad filetypes

Missing filename: Bad filename 67
> This is now the biggest cause of errors. I suggest we log the file names
users are trying to upload with to identify what is going wrong. A bug is
open https://bugzilla.wikimedia.org/show_bug.cgi?id=49544

There are various other errors which are probably not worth our time as
they suggest vandal edits:

This file did not pass file verification 4
Unknown error: "titleblacklist-forbidden-edit" 5
Unknown error: "titleblacklist-custom-filename" 1
Missing filename: Filename exists 1
Missing filename: Duplicate archive 1
Filetype not permitted: MOV 1
> The file type not permitted errors could be solved by checking the image
in preview mode correctly loaded and whether it has a width that isn't 0.

2) Errors due to tokens


Bad token name. 15
> These have all occurred on en.m.wikipedia.org, ru.m.wikipedia.org and
meta.m.wikimedia.org
> 5 of them were on the uploads page
Investigating with Chris it seems this can happen when a user has logged in
on mobile, left the page for some time and thus their login has expired.
Requesting a central auth token without being logged in will cause this
error. We should explore checking login status before starting the upload
workflow and redirect the user to the login page.

Invalid token 5
> We are caching tokens that have since expired. We should explore
invalidating tokens.

3) Other errors
These bugs are all pretty mysterious and it's not clear what causes them

* The modification you tried to make was aborted by an extension hook 7
* error: 3
> Note This occurs when an error happens but the error is missing an 'info'
property.
e.g. the response is:
{ error: {} }

An internal error occurred 1


On Sat, Jun 8, 2013 at 3:49 PM, Max Semenik <maxsem.wiki@gmail.com> wrote:
> On 08.06.2013, 21:07 Brian wrote:
>
>
>> I suspect that is caused by UploadBlacklist extension, which
>> blacklists about 23 files by their sha hash. According to the config
>> file, there's a log at "udp://$wmfUdp2logDest/upload-blacklist", so
>> you can probably check if that guess is right.
>
> $ grep -v 'MISS' upload-blacklist.log
> $
>
> --
> Best regards,
> Max Semenik ([[User:MaxSem]])
>
>
> _______________________________________________
> Wikitech-l mailing list
> Wikitech-l@lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l

--
Jon Robson
http://jonrobson.me.uk
@rakugojon
_______________________________________________
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l