Mailing List Archive

Pending Changes deployment strategy (Re: Pending Changes (nee Flagged Protection) launch on June 14 (PDT))
On Fri, Jun 4, 2010 at 3:12 PM, Platonides <Platonides@gmail.com> wrote:

> Rob Lanphier wrote:
> > A full test pass with all of the different configurations isn't going to
> be
> > possible, so some help with testing the different configurations would be
> > wonderful. We'll have a fast fallback plan in place should we
> accidentally
> > break the other wikis, but obviously it'd be better to get it right the
> > first time.
>
> A bit off topic but, what makes it impossible? It shouldn't be /that/
> hard. Fixing it may be useful for future deployments.


Well, it's possible, but it would delay the deployment. One plan we
discussed on #wikimedia-tech was to deploy the FlaggedRevs_alpha branch to
en.wikipedia.org, rather than deploying the FlaggedRevs trunk.
FlaggedRevs_alpha is what is currently running on
flaggedrevs.labs.wikimedia

A couple of different ways of going about it:
1. Deploy FlaggedRevs, but then drop back to FlaggedRevs_alpha if there's
obvious breakage on one of the other wikis we can't fix quickly
2. Deploy FlaggedRevs_alpha from the start

Right now, we really don't want to delay deployment, since delays beget
delays. Thoughts on a preferred strategy?

Rob
_______________________________________________
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: Pending Changes deployment strategy (Re: Pending Changes (nee Flagged Protection) launch on June 14 (PDT)) [ In reply to ]
2010/6/5 Rob Lanphier <robla@wikimedia.org>:
> A couple of different ways of going about it:
> 1. Deploy FlaggedRevs, but then drop back to FlaggedRevs_alpha if there's
> obvious breakage on one of the other wikis we can't fix quickly
> 2. Deploy FlaggedRevs_alpha from the start
>
> Right now, we really don't want to delay deployment, since delays beget
> delays.  Thoughts on a preferred strategy?
>
I don't think strategy #1 makes any sense at all. What I'd do is:
* Update FlaggedRevs_alpha to the desired state and check it on the labs wikis
* Switch wikis from FlaggedRevs to FlaggedRevs_alpha
* The old FlaggedRevs dir now sits around not being used. Leave it
like this for a while so we can easily switch wikis back to it
* After some time, update FlaggedRevs to be identical to
FlaggedRevs_alpha, and switch the wikis back, so we don't drag this
_alpha thing along forever.

The rationale behind this is that FlaggedRevs is a known-good state
more so than FlaggedRevs_alpha.

Roan Kattouw (Catrope)

_______________________________________________
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: Pending Changes deployment strategy (Re: Pending Changes (nee Flagged Protection) launch on June 14 (PDT)) [ In reply to ]
On Fri, Jun 4, 2010 at 3:49 PM, Roan Kattouw <roan.kattouw@gmail.com> wrote:

> I don't think [deploying the new FlaggedRevs to all wikis] makes any sense
> at all. What I'd do is:
> * Update FlaggedRevs_alpha to the desired state and check it on the labs
> wikis
> * Switch wikis from FlaggedRevs to FlaggedRevs_alpha
> * The old FlaggedRevs dir now sits around not being used. Leave it
> like this for a while so we can easily switch wikis back to it
> * After some time, update FlaggedRevs to be identical to
> FlaggedRevs_alpha, and switch the wikis back, so we don't drag this
> _alpha thing along forever.
>
> The rationale behind this is that FlaggedRevs is a known-good state
> more so than FlaggedRevs_alpha.
>

Well phooey. We discussed this option today in our status update, and based
on what Aaron says, this might not be an option. I didn't fully understand
the issue, so my apologies if I get this wrong, but I'll give it a shot
anyway.

What Aaron says is that the use of FlaggedRevs_alpha on
flaggedrevs.labs.wikimedia.org involves some more hacks than merely using a
different branch. There are also some configuration changes that disable
the message cache and possibly some other stuff. Without the hacks in
place, there's a race condition that can cause different messages to show up
randomly.

So, now we've got two different options:
1. If it's even possible, figure out how to work around the message cache
race condition
2. Try to deploy to all production wikis at once next week.

Neither of those gives me warm fuzzies right now, but #2 seems to be our new
plan of record. Thoughts?

Rob
_______________________________________________
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: Pending Changes deployment strategy (Re: Pending Changes (nee Flagged Protection) launch on June 14 (PDT)) [ In reply to ]
2010/6/8 Rob Lanphier <robla@wikimedia.org>:
> What Aaron says is that the use of FlaggedRevs_alpha on
> flaggedrevs.labs.wikimedia.org involves some more hacks than merely using a
> different branch.  There are also some configuration changes that disable
> the message cache and possibly some other stuff.  Without the hacks in
> place, there's a race condition that can cause different messages to show up
> randomly.
>
Ah, crap, totally forgot about the message cache issues.

> So, now we've got two different options:
> 1.  If it's even possible, figure out how to work around the message cache
> race condition
> 2.  Try to deploy to all production wikis at once next week.
>
> Neither of those gives me warm fuzzies right now, but #2 seems to be our new
> plan of record.  Thoughts?
>
Yeah, #2 is probably easiest. The message cache currently can't be
coerced into storing two different values for the same message, to be
used on different wikis, AFAIK.

Roan Kattouw (Catrope)

_______________________________________________
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: Pending Changes deployment strategy (Re: Pending Changes (nee Flagged Protection) launch on June 14 (PDT)) [ In reply to ]
On Mon, Jun 7, 2010 at 3:15 PM, Roan Kattouw <roan.kattouw@gmail.com> wrote:

> 2010/6/8 Rob Lanphier <robla@wikimedia.org>:
> > So, now we've got two different options [for deploying FlaggedRevs]:
> > 1. If it's even possible, figure out how to work around the message
> cache
> > race condition [and deploy FlaggedRevs_alpha only to en.wikipedia.org]
> > 2. Try to deploy to all production wikis at once next week.
> >
> > Neither of those gives me warm fuzzies right now, but #2 seems to be our
> new
> > plan of record. Thoughts?
> >
> Yeah, #2 is probably easiest. The message cache currently can't be
> coerced into storing two different values for the same message, to be
> used on different wikis, AFAIK.


One idea I'd like to float: we might be able to reconcile the messages
between the two branches, and using the "Mediawiki:" namespace to override
the messages we need to override (pretty much what we're already doing). I
have no idea just how awful that would be, and there may be other risks I'm
not accounting for, but I'm throwing that idea out there just in case it
might work.

We really don't want to delay the launch at this point, but we also don't
want to break any of the other production wikis in the process of rolling
this out to en.wikipedia.org, so any ideas you all have on risk
mitigation/minimization would be greatly appreciated.

Thanks!
Rob
_______________________________________________
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: Pending Changes deployment strategy (Re: Pending Changes (nee Flagged Protection) launch on June 14 (PDT)) [ In reply to ]
2010/6/8 Rob Lanphier <robla@robla.net>:
> One idea I'd like to float:  we might be able to reconcile the messages
> between the two branches, and using the "Mediawiki:" namespace to override
> the messages we need to override (pretty much what we're already doing).  I
> have no idea just how awful that would be, and there may be other risks I'm
> not accounting for, but I'm throwing that idea out there just in case it
> might work.
>
For a one-wiki rollout, this would work. However, it's probably more
complex than just updating the software across the board and crossing
our fingers.

> We really don't want to delay the launch at this point, but we also don't
> want to break any of the other production wikis in the process of rolling
> this out to en.wikipedia.org, so any ideas you all have on risk
> mitigation/minimization would be greatly appreciated.
>
IMO you should ascertain with some degree of confidence that your
updated code won't break vanilla FR setups (using a labs wiki or
something), then just roll it out.

Roan Kattouw (Catrope)

_______________________________________________
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l