Mailing List Archive

[Site issue] s3 wikis read-only until replication catches up
Wikimedia wikis hosted on the "s3" cluster (pretty much all but the
very large wikis, click on the "s3" box in
https://noc.wikimedia.org/dbtree/ to get a full list) are currently in
read-only mode due to severe replication lag. This problem was
apparently caused by a logic issue in the job queue, which should now
be fixed, but it will still take at least 2-3 hours for replication to
catch up. We apologize for the inconvenience, and are continuing to
monitor the situation.

Erik
--
Erik Möller
VP of Engineering and Product Development, Wikimedia Foundation

Support Free Knowledge: https://wikimediafoundation.org/wiki/Donate

_______________________________________________
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Site issue] s3 wikis read-only until replication catches up [ In reply to ]
Στις 11-12-2012, ημέρα Τρι, και ώρα 01:10 -0800, ο/η Erik Moeller
έγραψε:
> Wikimedia wikis hosted on the "s3" cluster (pretty much all but the
> very large wikis, click on the "s3" box in
> https://noc.wikimedia.org/dbtree/ to get a full list) are currently in
> read-only mode due to severe replication lag. This problem was
> apparently caused by a logic issue in the job queue, which should now
> be fixed, but it will still take at least 2-3 hours for replication to
> catch up. We apologize for the inconvenience, and are continuing to
> monitor the situation.
>
> Erik

Forgot to follow up here as well: editing was restored at around 11 am
UTC, and all databases are back to normal. The job runners have been in
operation during this entire time and seem not to cause any db issues
now.

Ariel


_______________________________________________
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l