Mailing List Archive

1 2 3  View All
Re: API BREAKING CHANGE: Default continuation mode for action=query will change at the end of this month [ In reply to ]
On Tue, Jun 9, 2015 at 6:04 AM, Ilya Korniyko <intracer@gmail.com> wrote:
> It would be nice if MediaWiki API _AND_ pywikipedia bot do not deprecate
> at once.
>
> Now it looks as
> API: we are deprecating what we promised to deprecated long ago - ok
> pywikipedia compat: did not handle the deprecation of API before, and are
> not going to fix copy-pasted in tens of places (not one place, it's never
> that simple) query builders to support "rawcontinue", we announce compat as
> discontinued together with the old style API.
> API deprecation was not coordinated with client library deprecation - not ok
>
> If there is one year gap between two deprecations - ok, bot writers can
> choose either compat or core, and their bots can still work.
> Most users don't use APi directly so it should be the problem of
> coordination between API and clients developers.

On the pywikipedia side, it has been unofficially deprecated for a
while, and the intention was to decommission compat as gracefully as
possible, with Wikimania as the final killing ground.

The pywikibot developers roughly hashed out a decommissioning plan at
the Lyon hackathon, and worked with WMF staff to start doing impact
analysis.
https://phabricator.wikimedia.org/T101214

Then the MW API continuation breakage was announced post Lyon. Hmm.

The continuation problem in pywikipedia / compat is not so much those
50 or so occurrences where continuation bugs appear to exist, but
1. testing those 50 or so occurrences
2. finding the other less obvious occurrences
3. scripts people have written using compat's query.GetData may also
be broken, as it doesnt do continuation.

With a lot of wasted effort, 1 & 2 might be resolved so that 'compat'
code works, and someone could create a hack on query.GetData which
adds continuation for scripts not yet adapted to do continuation.
However the active pywikibot developers (approx. 5 people?) are
focused on making core better , including about 10 complex patches
under review that improve its query continuation algorithms, and
making core more compatible with 'compat' to ease the pain of
switching to core.

If anyone submits patches for compat continuation bugs affecting them,
they will be reviewed (usually by people familiar with compat) with
the presumption that the patch author has tested it, and merged if
there are no major problems.

--
John Vandenberg

_______________________________________________
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: API BREAKING CHANGE: Default continuation mode for action=query will change at the end of this month [ In reply to ]
Maybe someone with enough time and knowledge can fork compat and keep it alive...

> Date: Tue, 9 Jun 2015 13:03:03 +1000
> From: jayvdb@gmail.com
> To: wikitech-l@lists.wikimedia.org
> Subject: Re: [Wikitech-l] API BREAKING CHANGE: Default continuation mode for action=query will change at the end of this month
>
> On Tue, Jun 9, 2015 at 6:04 AM, Ilya Korniyko <intracer@gmail.com> wrote:
> > It would be nice if MediaWiki API _AND_ pywikipedia bot do not deprecate
> > at once.
> >
> > Now it looks as
> > API: we are deprecating what we promised to deprecated long ago - ok
> > pywikipedia compat: did not handle the deprecation of API before, and are
> > not going to fix copy-pasted in tens of places (not one place, it's never
> > that simple) query builders to support "rawcontinue", we announce compat as
> > discontinued together with the old style API.
> > API deprecation was not coordinated with client library deprecation - not ok
> >
> > If there is one year gap between two deprecations - ok, bot writers can
> > choose either compat or core, and their bots can still work.
> > Most users don't use APi directly so it should be the problem of
> > coordination between API and clients developers.
>
> On the pywikipedia side, it has been unofficially deprecated for a
> while, and the intention was to decommission compat as gracefully as
> possible, with Wikimania as the final killing ground.
>
> The pywikibot developers roughly hashed out a decommissioning plan at
> the Lyon hackathon, and worked with WMF staff to start doing impact
> analysis.
> https://phabricator.wikimedia.org/T101214
>
> Then the MW API continuation breakage was announced post Lyon. Hmm.
>
> The continuation problem in pywikipedia / compat is not so much those
> 50 or so occurrences where continuation bugs appear to exist, but
> 1. testing those 50 or so occurrences
> 2. finding the other less obvious occurrences
> 3. scripts people have written using compat's query.GetData may also
> be broken, as it doesnt do continuation.
>
> With a lot of wasted effort, 1 & 2 might be resolved so that 'compat'
> code works, and someone could create a hack on query.GetData which
> adds continuation for scripts not yet adapted to do continuation.
> However the active pywikibot developers (approx. 5 people?) are
> focused on making core better , including about 10 complex patches
> under review that improve its query continuation algorithms, and
> making core more compatible with 'compat' to ease the pain of
> switching to core.
>
> If anyone submits patches for compat continuation bugs affecting them,
> they will be reviewed (usually by people familiar with compat) with
> the presumption that the patch author has tested it, and merged if
> there are no major problems.
>
> --
> John Vandenberg
>
> _______________________________________________
> Wikitech-l mailing list
> Wikitech-l@lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l

_______________________________________________
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: API BREAKING CHANGE: Default continuation mode for action=query will change at the end of this month [ In reply to ]
On Tue, 2015-06-09 at 11:32 +0200, Steinsplitter Wiki wrote:
> Maybe someone with enough time and knowledge can fork compat and keep
> it alive...

As far as I understand it, there is no need to fork anything (no
disagreement with existing maintainers; no contribution barriers).

Anybody interested could have kept compat (more) alive over the last
years. Or could in the future.

andre
--
Andre Klapper | Wikimedia Bugwrangler
http://blogs.gnome.org/aklapper/


_______________________________________________
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: API BREAKING CHANGE: Default continuation mode for action=query will change at the end of this month [ In reply to ]
On Wed, Jun 3, 2015 at 12:43 PM, Brad Jorsch (Anomie) <bjorsch@wikimedia.org
> wrote:

> On Wed, Jun 3, 2015 at 7:29 AM, John Mark Vandenberg <jayvdb@gmail.com>
> wrote:
>
>> If possible, could you compile a list of bots affected at a lower
>> threshold - maybe 1,000. That will give us a better idea of the scale
>> of bots operators that will be affected when this lands - currently in
>> one months time.
>>
>
> I already have the list of *accounts* affected: there are 510 with between
> 1000 and 10000 hits. Of those, 454 do not contain "bot" (case
> insensitively), so they might be human users with user scripts, or AWB if
> that's not fixed (someone please check!), or the like. For comparison, in
> the over-10000 group there were 30 such that I filtered out.
>
> I'll want to check with Legal to make sure the additional release of
> account names is still compliant with the privacy policy (I'm almost but
> not entirely sure it would be ok).
>

Legal recommended we only post the list of bots, not the human accounts.
These are:

AHbot
AsuraBot
Autobot
BattyBot
Bibcode_Bot
Bottuzzu
ChenzwBot
Cydebot
DickensBot
DrTrigonBot
DSisyphBot
DumbBOT
DYKHousekeepingBot
DYKUpdateBot
FBot
GiftBot
GrantsBot
HangsnaBot
HangsnaBot2
ImageRemovalBot
InceptionBot
JackBot
JBot
Jimmy-bot
Kenrick95Bot
KrBot
KrinkleBot
LivingBot
MalafayaBot
MaraBot
MauroBot
MBHbot
Mr.Z-bot
NowCommons-Sichtbot
Olafbot
PereBot
PseudoBot
QianBot
Rainbot
Reports_bot
RFF-Bot
Salebot
Sanjeev_bot
SemperBlottoBot
SergoBot
SHBot
Steenthbot
TurkászBot
UWCTransferBot
VlsergeyBot
VriuBot
YiFeiBot
Yobot
ZacheBot
Zlobot

Note this list is still from May 23–29; a bot appearing in this list may
have been updated since then.


--
Brad Jorsch (Anomie)
Software Engineer
Wikimedia Foundation
_______________________________________________
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: API BREAKING CHANGE: Default continuation mode for action=query will change at the end of this month [ In reply to ]
Il 09/06/2015 11:32, Steinsplitter Wiki ha scritto:
> Maybe someone with enough time and knowledge can fork compat and keep it alive...
>

Everyone is free to fork it, of course, but in this case it'd only bring
more fragmentation to the ecosystem.
As John said, patches are always welcome.

_______________________________________________
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: API BREAKING CHANGE: Default continuation mode for action=query will change at the end of this month [ In reply to ]
On Wed, 03 Jun 2015 18:29:08 +0700, John Mark Vandenberg wrote:

> On Wed, Jun 3, 2015 at 3:42 AM, Brad Jorsch (Anomie)
> <bjorsch@wikimedia.org> wrote:
>> ...
>> I've compiled a list of bots that have hit the deprecation warning more
>> than 10000 times over the course of the week May 23–29. If you are
>> responsible for any of these bots, please fix them. If you know who is,
>> please make sure they've seen this notification. Thanks.
>
> Thank you Brad for doing impact analysis and providing a list of the 71
> bots with more than 10,000 problems per week. We can try to solve those
> by working with the bot operators.
>
> If possible, could you compile a list of bots affected at a lower
> threshold - maybe 1,000. That will give us a better idea of the scale
> of bots operators that will be affected when this lands - currently in
> one months time.
>
> Will the deploy date be moved back if the impact doesnt diminish by bots
> being fixed?

Should someone contact those bots on their talk page to notify the owners or
do we hope everyone reads this mailing list?

Also, this change will affect not only bots but also every piece of software
or tool that was written or published prior to the change and relies on the
API to fetch data. So there are two issues with that:
* The maintainer of the tool has no time/interest to compile and publish new
releases of versions which were considered stable. This means the tool will
die if the source code is not available or no one wants to take over.
* Previously published programs that were considered stable and are executed
after July 1st. The user may not be aware that this tool is no longer stable
and requires an update. As far as I understood this API change: The results
from the API will just be somewhat different and not produce an error. So
the software will not crash but may produce weird behavior instead. Is there
any solution to this?

Thanks,
Marco
_______________________________________________
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: API BREAKING CHANGE: Default continuation mode for action=query will change at the end of this month [ In reply to ]
On Wed, Jun 17, 2015 at 9:15 PM, Marco <maic23@live.de> wrote:
> On Wed, 03 Jun 2015 18:29:08 +0700, John Mark Vandenberg wrote:
>
>> On Wed, Jun 3, 2015 at 3:42 AM, Brad Jorsch (Anomie)
>> <bjorsch@wikimedia.org> wrote:
>>> ...
>>> I've compiled a list of bots that have hit the deprecation warning more
>>> than 10000 times over the course of the week May 23–29. If you are
>>> responsible for any of these bots, please fix them. If you know who is,
>>> please make sure they've seen this notification. Thanks.
>>
>> Thank you Brad for doing impact analysis and providing a list of the 71
>> bots with more than 10,000 problems per week. We can try to solve those
>> by working with the bot operators.
>>
>> If possible, could you compile a list of bots affected at a lower
>> threshold - maybe 1,000. That will give us a better idea of the scale
>> of bots operators that will be affected when this lands - currently in
>> one months time.
>>
>> Will the deploy date be moved back if the impact doesnt diminish by bots
>> being fixed?
>
> Should someone contact those bots on their talk page to notify the owners or
> do we hope everyone reads this mailing list?

I see Whatamidoing (WMF) has already done this for many bots.

I saw one of the problematic bots says they are using AutoWikiBrowser.
maybe the bot is using an old version of AWB?

> Also, this change will affect not only bots but also every piece of software
> or tool that was written or published prior to the change and relies on the
> API to fetch data.
> So there are two issues with that:
> * The maintainer of the tool has no time/interest to compile and publish new
> releases of versions which were considered stable. This means the tool will
> die if the source code is not available or no one wants to take over.
> * Previously published programs that were considered stable and are executed
> after July 1st. The user may not be aware that this tool is no longer stable
> and requires an update. As far as I understood this API change: The results
> from the API will just be somewhat different and not produce an error. So
> the software will not crash but may produce weird behavior instead. Is there
> any solution to this?

The API currently emits a warning if a query continuation mode isnt selected.

I guess on July 1 the API could emit an error, and not return any query data.
Then the data isnt going to cause weird behaviour - it will break, properly.

--
John Vandenberg

_______________________________________________
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: API BREAKING CHANGE: Default continuation mode for action=query will change at the end of this month [ In reply to ]
On Wed, Jun 17, 2015 at 7:44 PM, John Mark Vandenberg <jayvdb@gmail.com>
wrote:

>
> The API currently emits a warning if a query continuation mode isnt
> selected.
>
> I guess on July 1 the API could emit an error, and not return any query
> data.
> Then the data isnt going to cause weird behaviour - it will break,
> properly.
>
>
Hmm, this is actually an interesting idea - would it make sense to error on
missing "continue" or "rawcontinue" for all action=query for about a month
or two, so that everyone notices it right away and gets updated, and than
resume with the new behavior?
_______________________________________________
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: API BREAKING CHANGE: Default continuation mode for action=query will change at the end of this month [ In reply to ]
On Wed, Jun 17, 2015 at 10:13 PM, Yuri Astrakhan <yastrakhan@wikimedia.org>
wrote:

> On Wed, Jun 17, 2015 at 7:44 PM, John Mark Vandenberg <jayvdb@gmail.com>
> wrote:
>
> >
> > The API currently emits a warning if a query continuation mode isnt
> > selected.
> >
> > I guess on July 1 the API could emit an error, and not return any query
> > data.
> > Then the data isnt going to cause weird behaviour - it will break,
> > properly.
> >
> >
> Hmm, this is actually an interesting idea - would it make sense to error on
> missing "continue" or "rawcontinue" for all action=query for about a month
> or two, so that everyone notices it right away and gets updated, and than
> resume with the new behavior?


I saw at least one major bot had all of its actual continuation uses fixed
after the first notification, but did not bother to fix cases where it was
receiving the warning but was never going to continue the query. Others may
well have done the same.

For example, if a client wants to get the current content for enwiki's Main
Page, it might hit
https://en.wikipedia.org/w/api.php?action=query&prop=revisions&rvprop=content&rvlimit=1&titles=Main%20Page.
That returns continuation (and the warning) in case the client would want
further revisions, but the client here doesn't. Similar cases could be made
for fetching the most recent protection log entry of a page, the imageinfo
for the current version of an image, and so on.

The current plan, as shown in https://gerrit.wikimedia.org/r/#/c/160223/,
is to output a different warning for some period of time after the
changeover. People investigating the sudden failure of their
bot/script/gadget would hopefully see that.


--
Brad Jorsch (Anomie)
Software Engineer
Wikimedia Foundation
_______________________________________________
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: API BREAKING CHANGE: Default continuation mode for action=query will change at the end of this month [ In reply to ]
There is no such a thing like "properly break". That's like if someone
was happily sad. Come on

On Thu, Jun 18, 2015 at 4:09 PM, Brad Jorsch (Anomie)
<bjorsch@wikimedia.org> wrote:
> On Wed, Jun 17, 2015 at 10:13 PM, Yuri Astrakhan <yastrakhan@wikimedia.org>
> wrote:
>
>> On Wed, Jun 17, 2015 at 7:44 PM, John Mark Vandenberg <jayvdb@gmail.com>
>> wrote:
>>
>> >
>> > The API currently emits a warning if a query continuation mode isnt
>> > selected.
>> >
>> > I guess on July 1 the API could emit an error, and not return any query
>> > data.
>> > Then the data isnt going to cause weird behaviour - it will break,
>> > properly.
>> >
>> >
>> Hmm, this is actually an interesting idea - would it make sense to error on
>> missing "continue" or "rawcontinue" for all action=query for about a month
>> or two, so that everyone notices it right away and gets updated, and than
>> resume with the new behavior?
>
>
> I saw at least one major bot had all of its actual continuation uses fixed
> after the first notification, but did not bother to fix cases where it was
> receiving the warning but was never going to continue the query. Others may
> well have done the same.
>
> For example, if a client wants to get the current content for enwiki's Main
> Page, it might hit
> https://en.wikipedia.org/w/api.php?action=query&prop=revisions&rvprop=content&rvlimit=1&titles=Main%20Page.
> That returns continuation (and the warning) in case the client would want
> further revisions, but the client here doesn't. Similar cases could be made
> for fetching the most recent protection log entry of a page, the imageinfo
> for the current version of an image, and so on.
>
> The current plan, as shown in https://gerrit.wikimedia.org/r/#/c/160223/,
> is to output a different warning for some period of time after the
> changeover. People investigating the sudden failure of their
> bot/script/gadget would hopefully see that.
>
>
> --
> Brad Jorsch (Anomie)
> Software Engineer
> Wikimedia Foundation
> _______________________________________________
> Wikitech-l mailing list
> Wikitech-l@lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l

_______________________________________________
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: API BREAKING CHANGE: Default continuation mode for action=query will change at the end of this month [ In reply to ]
On Thu, Jun 18, 2015 at 12:13 PM, Yuri Astrakhan
<yastrakhan@wikimedia.org> wrote:
> On Wed, Jun 17, 2015 at 7:44 PM, John Mark Vandenberg <jayvdb@gmail.com>
> wrote:
>
>>
>> The API currently emits a warning if a query continuation mode isnt
>> selected.
>>
>> I guess on July 1 the API could emit an error, and not return any query
>> data.
>> Then the data isnt going to cause weird behaviour - it will break,
>> properly.
>>
>>
> Hmm, this is actually an interesting idea - would it make sense to error on
> missing "continue" or "rawcontinue" for all action=query for about a month
> or two, so that everyone notices it right away and gets updated, and than
> resume with the new behavior?

Not 'all', please, but perhaps all queries where there is actual
continuation data like is used now for the existing warning. I would
really to avoid the API failing at all, ever, for
"?action=query&meta=siteinfo" - that is how clients get the API
generator version, and only when we have the API generator version do
we know what features are supported by the API.

Or just not flick the switch on July 1, and only default to the new
continuation mode for formatversion=2.

--
John Vandenberg

_______________________________________________
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: API BREAKING CHANGE: Default continuation mode for action=query will change at the end of this month [ In reply to ]
I'm reluctant to interject again, but as a client of the API I feel the
need to speak up. Let's work through an example: Clients A, B, & C are
using API 1 when the Server adds API 2:

1. Current, "change the default" approach:
1. All clients start receiving warnings as *extra payload data* (how
does this even work for API's w/o a response payload?)
2. Clients B & C start using API 2
3. Server changes API 1 to behave like API 2
4. Client A is broken, so switches to API 2
5. Server removes API 2 (becomes the new default)
6. Clients A, B, & C have to switch to API 1
2. Possible alternative:
1. Clients get an email (as they currently do) about the new API
being available
2. Clients A & B migrate to API 2
3. Since there's only 1 client left (and/or the deadline has passed)
Server removes API 1, *explicitly breaking* Client C
4. Someone notices Client C is broken and moves it to API 2 (or
whatever the currently-supported API is)

Am I missing something? What are your thoughts on 1 vs. 2?

On Thu, Jun 18, 2015 at 10:17 AM, John Mark Vandenberg <jayvdb@gmail.com>
wrote:

> On Thu, Jun 18, 2015 at 12:13 PM, Yuri Astrakhan
> <yastrakhan@wikimedia.org> wrote:
> > On Wed, Jun 17, 2015 at 7:44 PM, John Mark Vandenberg <jayvdb@gmail.com>
> > wrote:
> >
> >>
> >> The API currently emits a warning if a query continuation mode isnt
> >> selected.
> >>
> >> I guess on July 1 the API could emit an error, and not return any query
> >> data.
> >> Then the data isnt going to cause weird behaviour - it will break,
> >> properly.
> >>
> >>
> > Hmm, this is actually an interesting idea - would it make sense to error
> on
> > missing "continue" or "rawcontinue" for all action=query for about a
> month
> > or two, so that everyone notices it right away and gets updated, and than
> > resume with the new behavior?
>
> Not 'all', please, but perhaps all queries where there is actual
> continuation data like is used now for the existing warning. I would
> really to avoid the API failing at all, ever, for
> "?action=query&meta=siteinfo" - that is how clients get the API
> generator version, and only when we have the API generator version do
> we know what features are supported by the API.
>
> Or just not flick the switch on July 1, and only default to the new
> continuation mode for formatversion=2.
>
> --
> John Vandenberg
>
> _______________________________________________
> Wikitech-l mailing list
> Wikitech-l@lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
>



--
EN Wikipedia user page: https://en.wikipedia.org/wiki/User:Brian.gerstle
IRC: bgerstle
_______________________________________________
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: API BREAKING CHANGE: Default continuation mode for action=query will change at the end of this month [ In reply to ]
On Thu, Jun 18, 2015 at 10:37 AM, Brian Gerstle <bgerstle@wikimedia.org>
wrote:

> 1. Current, "change the default" approach:
> 1. All clients start receiving warnings as *extra payload data* (how
> does this even work for API's w/o a response payload?)
>

What module in api.php doesn't have a response payload? action=opensearch
with format=json is the only example I can think of.


> 5. Server removes API 2 (becomes the new default)
> 6. Clients A, B, & C have to switch to API 1
>

You're making false assumptions here. For example, with this continuation
change, explicitly supplying '&continue=' in the initial query is not going
to suddenly stop working. Nor is '&rawcontinue=', for that matter.

The same goes for a hypothetical future change to make '&formatversion=2'
the default for format=json. We wouldn't suddenly make clients have to
supply '&formatversion=1' to get the formatversion=2 behavior, or make
supplying a formatversion be an error or warning.


--
Brad Jorsch (Anomie)
Software Engineer
Wikimedia Foundation
_______________________________________________
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: API BREAKING CHANGE: Default continuation mode for action=query will change at the end of this month [ In reply to ]
Brad Jorsch (Anomie) wrote:
>On Thu, Jun 18, 2015 at 10:37 AM, Brian Gerstle <bgerstle@wikimedia.org>
>wrote:
>> 1. Current, "change the default" approach:
>> 1. All clients start receiving warnings as *extra payload data*
>> (how does this even work for API's w/o a response payload?)
>
>What module in api.php doesn't have a response payload? action=opensearch
>with format=json is the only example I can think of.

I would think any request to api.php using format=none would lack a
response payload? However I'm not totally sure that I'm reading and using
the term response payload in the same way that you two might be.

MZMcBride



_______________________________________________
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: API BREAKING CHANGE: Default continuation mode for action=query will change at the end of this month [ In reply to ]
On Thu, Jun 18, 2015 at 11:33 AM, MZMcBride <z@mzmcbride.com> wrote:

> Brad Jorsch (Anomie) wrote:
> >On Thu, Jun 18, 2015 at 10:37 AM, Brian Gerstle <bgerstle@wikimedia.org>
> >wrote:
> >> 1. Current, "change the default" approach:
> >> 1. All clients start receiving warnings as *extra payload data*
> >> (how does this even work for API's w/o a response payload?)
> >
> >What module in api.php doesn't have a response payload? action=opensearch
> >with format=json is the only example I can think of.
>
> I would think any request to api.php using format=none would lack a
> response payload? However I'm not totally sure that I'm reading and using
> the term response payload in the same way that you two might be.
>

Haha, true. But if someone is using format=none, they're explicitly not
caring to know if their request worked or not. ;)

--
Brad Jorsch (Anomie)
Software Engineer
Wikimedia Foundation
_______________________________________________
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: API BREAKING CHANGE: Default continuation mode for action=query will change at the end of this month [ In reply to ]
I guess it comes down to is this: if we're going to continue supporting old
behavior, they should be accessible via the same old requests. *This
removes the need for existing clients to be updated in the first place*.
If we eventually want to delete the old code keeping the old behavior
separated from the new will make it clear & explicit what's being dropped
and what to use instead. For example, dropping formatversion=1 means
clients need to use formatversion=2 (or whatever other supported versions).

Lastly, changing the default behavior to make things sane for new
developers is, IMO, a bad trade-off because they'll eventually get tripped
by us pulling the rug out from under their feet by *breaking backwards
compatibility stable APIs*. Those sorts of changes should be reserved for
experimental or even beta APIs. Continuing queries seems like a stable—and
pervasive—part of the API.


On Thu, Jun 18, 2015 at 11:40 AM, Brad Jorsch (Anomie) <
bjorsch@wikimedia.org> wrote:

> On Thu, Jun 18, 2015 at 11:33 AM, MZMcBride <z@mzmcbride.com> wrote:
>
> > Brad Jorsch (Anomie) wrote:
> > >On Thu, Jun 18, 2015 at 10:37 AM, Brian Gerstle <bgerstle@wikimedia.org
> >
> > >wrote:
> > >> 1. Current, "change the default" approach:
> > >> 1. All clients start receiving warnings as *extra payload data*
> > >> (how does this even work for API's w/o a response payload?)
> > >
> > >What module in api.php doesn't have a response payload?
> action=opensearch
> > >with format=json is the only example I can think of.
> >
> > I would think any request to api.php using format=none would lack a
> > response payload? However I'm not totally sure that I'm reading and using
> > the term response payload in the same way that you two might be.
> >
>
> Haha, true. But if someone is using format=none, they're explicitly not
> caring to know if their request worked or not. ;)
>
> --
> Brad Jorsch (Anomie)
> Software Engineer
> Wikimedia Foundation
> _______________________________________________
> Wikitech-l mailing list
> Wikitech-l@lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
>



--
EN Wikipedia user page: https://en.wikipedia.org/wiki/User:Brian.gerstle
IRC: bgerstle
_______________________________________________
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: API BREAKING CHANGE: Default continuation mode for action=query will change at the end of this month [ In reply to ]
On Thu, Jun 18, 2015 at 9:26 AM, Brian Gerstle <bgerstle@wikimedia.org>
wrote:

> I guess it comes down to is this: if we're going to continue supporting
> old behavior, they should be accessible via the same old requests. *This
> removes the need for existing clients to be updated in the first place*.
> If we eventually want to delete the old code keeping the old behavior
> separated from the new will make it clear & explicit what's being dropped
> and what to use instead. ...
>
> Lastly, changing the default behavior to make things sane for new
> developers is, IMO, a bad trade-off


That seems the crux of it. Because the MediaWiki API isn't versioned (and
there are good reasons for it), the recommended "best practices" form of an
API request evolves, something like

api.php?continue=& formatversion=2 &utf8= &*your_actual_params*

and over time the best practices boilerplate at the front gets longer
unless we change a default and break old clients. Examples and
documentation should show best practices; T103015 is to use formatversion=2
and "https" in all example API requests. (We should have used &continue= in
examples for the last year, it's too late now.)

The above is actually a real URL and shows three different approaches:
1. As the e-mail subject says we're going to make continue= the default in
a few weeks so you won't need to add it (but clients MUST add rawcontinue=
to get the old behavior).
2. formatversion=2 is the future but won't be the default for a while.
3. If you request formatversion=2 then results default to utf8, so you
don't need to specify utf8. (Note formatversion=2 only applies to
format=json or php.)

Which approach to take is a judgement call, I'm interject-opinion-reluctant
:)



> because they'll eventually get tripped by us pulling the rug out from
> under their feet by *breaking backwards compatibility stable APIs*.


Or, over time the best practices boilerplate endlessly expands:
responselayout=clean &reporterrors=schema &facebookoverlordmode= ... Does
that make our API "user-hostile"? IMO it just makes it wordy.



> Those sorts of changes should be reserved for experimental or even beta
> APIs. Continuing queries seems like a stable—and pervasive—part of the API.
>

Cheers,
--
=S Page WMF Tech writer
_______________________________________________
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: API BREAKING CHANGE: Default continuation mode for action=query will change at the end of this month [ In reply to ]
Before playing around with api serious bugs should be fixed... I see xxXx unresolved bugs on pahricator. It is frustrating....

> Date: Thu, 18 Jun 2015 14:56:25 -0700
> From: spage@wikimedia.org
> To: wikitech-l@lists.wikimedia.org
> Subject: Re: [Wikitech-l] API BREAKING CHANGE: Default continuation mode for action=query will change at the end of this month
>
> On Thu, Jun 18, 2015 at 9:26 AM, Brian Gerstle <bgerstle@wikimedia.org>
> wrote:
>
> > I guess it comes down to is this: if we're going to continue supporting
> > old behavior, they should be accessible via the same old requests. *This
> > removes the need for existing clients to be updated in the first place*.
> > If we eventually want to delete the old code keeping the old behavior
> > separated from the new will make it clear & explicit what's being dropped
> > and what to use instead. ...
> >
> > Lastly, changing the default behavior to make things sane for new
> > developers is, IMO, a bad trade-off
>
>
> That seems the crux of it. Because the MediaWiki API isn't versioned (and
> there are good reasons for it), the recommended "best practices" form of an
> API request evolves, something like
>
> api.php?continue=& formatversion=2 &utf8= &*your_actual_params*
>
> and over time the best practices boilerplate at the front gets longer
> unless we change a default and break old clients. Examples and
> documentation should show best practices; T103015 is to use formatversion=2
> and "https" in all example API requests. (We should have used &continue= in
> examples for the last year, it's too late now.)
>
> The above is actually a real URL and shows three different approaches:
> 1. As the e-mail subject says we're going to make continue= the default in
> a few weeks so you won't need to add it (but clients MUST add rawcontinue=
> to get the old behavior).
> 2. formatversion=2 is the future but won't be the default for a while.
> 3. If you request formatversion=2 then results default to utf8, so you
> don't need to specify utf8. (Note formatversion=2 only applies to
> format=json or php.)
>
> Which approach to take is a judgement call, I'm interject-opinion-reluctant
> :)
>
>
>
> > because they'll eventually get tripped by us pulling the rug out from
> > under their feet by *breaking backwards compatibility stable APIs*.
>
>
> Or, over time the best practices boilerplate endlessly expands:
> responselayout=clean &reporterrors=schema &facebookoverlordmode= ... Does
> that make our API "user-hostile"? IMO it just makes it wordy.
>
>
>
> > Those sorts of changes should be reserved for experimental or even beta
> > APIs. Continuing queries seems like a stable—and pervasive—part of the API.
> >
>
> Cheers,
> --
> =S Page WMF Tech writer
> _______________________________________________
> Wikitech-l mailing list
> Wikitech-l@lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l

_______________________________________________
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: API BREAKING CHANGE: Default continuation mode for action=query will change at the end of this month [ In reply to ]
On Fri, Jun 19, 2015 at 7:56 AM, S Page <spage@wikimedia.org> wrote:
> On Thu, Jun 18, 2015 at 9:26 AM, Brian Gerstle <bgerstle@wikimedia.org>
> wrote:
>
>> I guess it comes down to is this: if we're going to continue supporting
>> old behavior, they should be accessible via the same old requests. *This
>> removes the need for existing clients to be updated in the first place*.
>> If we eventually want to delete the old code keeping the old behavior
>> separated from the new will make it clear & explicit what's being dropped
>> and what to use instead. ...
>>
>> Lastly, changing the default behavior to make things sane for new
>> developers is, IMO, a bad trade-off
>
>
> That seems the crux of it. Because the MediaWiki API isn't versioned (and
> there are good reasons for it), the recommended "best practices" form of an
> API request evolves, something like
>
> api.php?continue=& formatversion=2 &utf8= &*your_actual_params*

Why doesnt formatversion=2 assume continue= and utf8= ?

--
John Vandenberg

_______________________________________________
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: API BREAKING CHANGE: Default continuation mode for action=query will change at the end of this month [ In reply to ]
On Fri, Jun 19, 2015 at 5:42 AM, John Mark Vandenberg <jayvdb@gmail.com>
wrote:

> Why doesnt formatversion=2 assume continue= and utf8= ?
>

It does assume utf8= (there's an ascii= to request the old mode in
combination with formatversion=2).

It doesn't assume continue= because that belongs to ApiQuery, not
ApiFormatJson or ApiFormatPhp.


--
Brad Jorsch (Anomie)
Software Engineer
Wikimedia Foundation
_______________________________________________
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: API BREAKING CHANGE: Default continuation mode for action=query will change at the end of this month [ In reply to ]
On Wed, 2015-06-10 at 15:01 -0400, Brad Jorsch (Anomie) wrote:
> Legal recommended we only post the list of bots, not the human
> accounts. These are:

> AHbot
> AsuraBot
[...]
> Note this list is still from May 23–29; a bot appearing in this list
> may have been updated since then.


As there are seven days left, does someone have time / capacity to
provide an updated list of bots that likely haven't seen updates yet?

Also, in [1], Sitic came up with a query of de.wp *gadgets* affected:
https://de.wikipedia.org/w/index.php?title=Spezial:Suche&limit=100&offset=0&ns2=1&ns8=1&search=%22query-continue%22+-%22rawcontinue%22+intitle%3A*.js

Has there been on-wiki outreach to *gadget* maintainers?

Thanks in advance!
andre

[1] https://de.wikipedia.org/wiki/Wikipedia:Bots/Notizen#Hinweis_an_alle_Bot-Betreiber_.28API-.C3.84nderung.29

--
Andre Klapper | Wikimedia Bugwrangler
http://blogs.gnome.org/aklapper/


_______________________________________________
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: API BREAKING CHANGE: Default continuation mode for action=query will change at the end of this month [ In reply to ]
Hi Brad,

Brad Jorsch (Anomie) schreef op 2-6-2015 om 22:42:
> As has been announced several times (most recently at
> https://lists.wikimedia.org/pipermail/wikitech-l/2015-April/081559.html),
> the default continuation mode for action=query requests to api.php will be
> changing to be easier for new coders to use correctly.
>
> ...
> CategorizationBot
> ...
> ErfgoedBot
Pulled the plug on both bots. You won't see them in your logs again.
Some background at
https://commons.wikimedia.org/wiki/Commons:Village_pump#Pulled_the_plug_on_CategorizationBot_.26_ErfgoedBot

Maarten


_______________________________________________
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: API BREAKING CHANGE: Default continuation mode for action=query will change at the end of this month [ In reply to ]
I run GrantsBot, which is listed here.

I've updated all GrantsBot API requests to use rawcontinue=1. But as I read
through this thread, it's not clear to me that that's the problem. I can't
find a single instance in my code where I'm actually continuing a query.
Does this breaking change only apply be an issue if you were using
querycontinue in the first place?

I'm sure this has been covered elsewhere or I'm missing something obvious,
but I'd be grateful for specific confirmation :)

Best,
Jonathan

On Wed, Jun 10, 2015 at 12:01 PM, Brad Jorsch (Anomie) <
bjorsch@wikimedia.org> wrote:

> On Wed, Jun 3, 2015 at 12:43 PM, Brad Jorsch (Anomie) <
> bjorsch@wikimedia.org
> > wrote:
>
> > On Wed, Jun 3, 2015 at 7:29 AM, John Mark Vandenberg <jayvdb@gmail.com>
> > wrote:
> >
> >> If possible, could you compile a list of bots affected at a lower
> >> threshold - maybe 1,000. That will give us a better idea of the scale
> >> of bots operators that will be affected when this lands - currently in
> >> one months time.
> >>
> >
> > I already have the list of *accounts* affected: there are 510 with
> between
> > 1000 and 10000 hits. Of those, 454 do not contain "bot" (case
> > insensitively), so they might be human users with user scripts, or AWB if
> > that's not fixed (someone please check!), or the like. For comparison, in
> > the over-10000 group there were 30 such that I filtered out.
> >
> > I'll want to check with Legal to make sure the additional release of
> > account names is still compliant with the privacy policy (I'm almost but
> > not entirely sure it would be ok).
> >
>
> Legal recommended we only post the list of bots, not the human accounts.
> These are:
>
> AHbot
> AsuraBot
> Autobot
> BattyBot
> Bibcode_Bot
> Bottuzzu
> ChenzwBot
> Cydebot
> DickensBot
> DrTrigonBot
> DSisyphBot
> DumbBOT
> DYKHousekeepingBot
> DYKUpdateBot
> FBot
> GiftBot
> GrantsBot
> HangsnaBot
> HangsnaBot2
> ImageRemovalBot
> InceptionBot
> JackBot
> JBot
> Jimmy-bot
> Kenrick95Bot
> KrBot
> KrinkleBot
> LivingBot
> MalafayaBot
> MaraBot
> MauroBot
> MBHbot
> Mr.Z-bot
> NowCommons-Sichtbot
> Olafbot
> PereBot
> PseudoBot
> QianBot
> Rainbot
> Reports_bot
> RFF-Bot
> Salebot
> Sanjeev_bot
> SemperBlottoBot
> SergoBot
> SHBot
> Steenthbot
> TurkászBot
> UWCTransferBot
> VlsergeyBot
> VriuBot
> YiFeiBot
> Yobot
> ZacheBot
> Zlobot
>
> Note this list is still from May 23–29; a bot appearing in this list may
> have been updated since then.
>
>
> --
> Brad Jorsch (Anomie)
> Software Engineer
> Wikimedia Foundation
> _______________________________________________
> Wikitech-l mailing list
> Wikitech-l@lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
>



--
Jonathan T. Morgan
Senior Design Researcher
Wikimedia Foundation
User:Jmorgan (WMF) <https://meta.wikimedia.org/wiki/User:Jmorgan_(WMF)>
_______________________________________________
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: API BREAKING CHANGE: Default continuation mode for action=query will change at the end of this month [ In reply to ]
Jonathan, in theory, if you never use query continuation, you don't need to
do anything with your code - you just need to tell the community that the
bot is ok to use the new continuation system. This way noone will block the
bot just in case. It is much better than to use "rawcontinue", because
that flag will keep telling us someone needs the old system.

On Mon, Jun 29, 2015 at 8:01 PM, Jonathan Morgan <jmorgan@wikimedia.org>
wrote:

> I run GrantsBot, which is listed here.
>
> I've updated all GrantsBot API requests to use rawcontinue=1. But as I read
> through this thread, it's not clear to me that that's the problem. I can't
> find a single instance in my code where I'm actually continuing a query.
> Does this breaking change only apply be an issue if you were using
> querycontinue in the first place?
>
> I'm sure this has been covered elsewhere or I'm missing something obvious,
> but I'd be grateful for specific confirmation :)
>
> Best,
> Jonathan
>
> On Wed, Jun 10, 2015 at 12:01 PM, Brad Jorsch (Anomie) <
> bjorsch@wikimedia.org> wrote:
>
> > On Wed, Jun 3, 2015 at 12:43 PM, Brad Jorsch (Anomie) <
> > bjorsch@wikimedia.org
> > > wrote:
> >
> > > On Wed, Jun 3, 2015 at 7:29 AM, John Mark Vandenberg <jayvdb@gmail.com
> >
> > > wrote:
> > >
> > >> If possible, could you compile a list of bots affected at a lower
> > >> threshold - maybe 1,000. That will give us a better idea of the scale
> > >> of bots operators that will be affected when this lands - currently in
> > >> one months time.
> > >>
> > >
> > > I already have the list of *accounts* affected: there are 510 with
> > between
> > > 1000 and 10000 hits. Of those, 454 do not contain "bot" (case
> > > insensitively), so they might be human users with user scripts, or AWB
> if
> > > that's not fixed (someone please check!), or the like. For comparison,
> in
> > > the over-10000 group there were 30 such that I filtered out.
> > >
> > > I'll want to check with Legal to make sure the additional release of
> > > account names is still compliant with the privacy policy (I'm almost
> but
> > > not entirely sure it would be ok).
> > >
> >
> > Legal recommended we only post the list of bots, not the human accounts.
> > These are:
> >
> > AHbot
> > AsuraBot
> > Autobot
> > BattyBot
> > Bibcode_Bot
> > Bottuzzu
> > ChenzwBot
> > Cydebot
> > DickensBot
> > DrTrigonBot
> > DSisyphBot
> > DumbBOT
> > DYKHousekeepingBot
> > DYKUpdateBot
> > FBot
> > GiftBot
> > GrantsBot
> > HangsnaBot
> > HangsnaBot2
> > ImageRemovalBot
> > InceptionBot
> > JackBot
> > JBot
> > Jimmy-bot
> > Kenrick95Bot
> > KrBot
> > KrinkleBot
> > LivingBot
> > MalafayaBot
> > MaraBot
> > MauroBot
> > MBHbot
> > Mr.Z-bot
> > NowCommons-Sichtbot
> > Olafbot
> > PereBot
> > PseudoBot
> > QianBot
> > Rainbot
> > Reports_bot
> > RFF-Bot
> > Salebot
> > Sanjeev_bot
> > SemperBlottoBot
> > SergoBot
> > SHBot
> > Steenthbot
> > TurkászBot
> > UWCTransferBot
> > VlsergeyBot
> > VriuBot
> > YiFeiBot
> > Yobot
> > ZacheBot
> > Zlobot
> >
> > Note this list is still from May 23–29; a bot appearing in this list may
> > have been updated since then.
> >
> >
> > --
> > Brad Jorsch (Anomie)
> > Software Engineer
> > Wikimedia Foundation
> > _______________________________________________
> > Wikitech-l mailing list
> > Wikitech-l@lists.wikimedia.org
> > https://lists.wikimedia.org/mailman/listinfo/wikitech-l
> >
>
>
>
> --
> Jonathan T. Morgan
> Senior Design Researcher
> Wikimedia Foundation
> User:Jmorgan (WMF) <https://meta.wikimedia.org/wiki/User:Jmorgan_(WMF)>
> _______________________________________________
> Wikitech-l mailing list
> Wikitech-l@lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
>
_______________________________________________
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: API BREAKING CHANGE: Default continuation mode for action=query will change at the end of this month [ In reply to ]
On Mon, Jun 29, 2015 at 1:53 PM, Yuri Astrakhan <yastrakhan@wikimedia.org>
wrote:

> It is much better than to use "rawcontinue", because
> that flag will keep telling us someone needs the old system.
>

I doubt we'll ever get rid of rawcontinue. There's practically no code to
maintain there (it's just a one-line function[1] and a trivial if-block to
call it[2]) so it's highly unlikely to be worth even minimal effort.

[1]:
https://git.wikimedia.org/blob/mediawiki%2Fcore.git/26288853/includes%2Fapi%2FApiContinuationManager.php#L163
[2]:
https://git.wikimedia.org/blob/mediawiki%2Fcore.git/26288853/includes%2Fapi%2FApiQuery.php#L295


--
Brad Jorsch (Anomie)
Software Engineer
Wikimedia Foundation
_______________________________________________
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

1 2 3  View All