Mailing List Archive

ResourceLoader, now in trunk!
MediaWiki Developers,

Over the past couple of months, Roan Kattouw and I (Trevor Parscal) have
been working on a JavaScript and CSS delivery system called
ResourceLoader. We're really excited about this technology, and hope
others will be too.

This system has been proving itself to be able to seriously improve
front-end performance. Just for starters, we're talking about taking the
Vector skin from 35 requests @ 30kB gzipped to 1 request @ 9.4kB gzipped
(see http://www.mediawiki.org/wiki/ResourceLoader/Benchmarks)

We are looking to make this the standard way to deliver Javascript, CSS,
and small images in MediaWiki and on Wikimedia projects, and we're
seeking your comments and help.

== Background ==

The goals of the project were to improve front-end performance, reduce
the complexity of developing JavaScript libraries and user-interfaces,
and get the ball rolling on a rewrite/refactoring of all JavaScript and
CSS code in MediaWiki.

What's wrong with things as they are now?

* Too many individual requests are being made. All JavaScript, CSS and
image resources are being loaded individually, which causes poor
performance on the cluster and users experience the site as being slow.
* We are wasting too much bandwidth. We are sending JavaScript and CSS
resources with large amounts of unneeded whitespace and comments.
* We are purging our caches too much. Many user interface changes
require purging page caches to take effect and many assets are
unnecessarily being purged from client machines due to the use of a
single style version for all assets
* We are sending people code they don't even use. Lots of JavaScript is
being sent to clients whose browsers will either crash when it arrives
(BlackBerry comes to mind), just not use it at all (older versions of
many browsers) while parsing it unnecessarily (this is slow on older
browsers, especially IE 6) or isn't even being completely utilized
(UsabilityInitiative's plugins.combined.min.js for instance)
* Internationalization in JavaScript is a mess. Developers are using
many different ways -- most of which are not ideal -- to get their
translated messages to the client.
* Right-to-left support in CSS is akward. Stylesheets for right-to-left
must to be either hand-coded in a separate stylesheet, generated each
time a change is made by running CSSJanus, or an extra style-sheet which
contains a series of over-rides.
* There's more! These and other issues were captured in our requirements
gathering process (see
http://www.mediawiki.org/wiki/ResourceLoader/Requirements )

What does ResourceLoader do to solve this?

* Combines resources together. Multiple scripts, styles, messages to be
delivered in a single request, either at initial page load or
dynamically; in both cases resolving dependencies automatically.
* Allows minification of JavaScript and CSS.
* Dramatically reduces the number of requests for small images. Small
images linked to from CSS code can be automatically in-lined as data
URLs (when the developer marks it with a special comment), and it's done
automatically as the file is served without requiring the developer to
do such steps manually.
* Allows deployment changes to all pages for all users within minutes,
without purging any HTML. ResourceLoader provides a short-expiry
start-up script which then decides to continue loading more JavaScript
or not, and if so has a complete manifest of all scripts and styles on
the server and their most recent versions, Also, this startup script
will be able to be inlined using ESI (see
http://en.wikipedia.org/wiki/Edge_Side_Includes ) when using Squid or
Varnish, reducing requests and improving performance even further.
* Provides a standard way to deliver translated messages to the client,
bundling them together with the code that uses them.
* Performs automatic left-to-right/right-to-left flipping for CSS files.
In most cases the developer won't have to do anything before deploying.
* Does all kinds of other cool tricks, which should soon make everyone's
lives better

What do you want from me?

* Help by porting existing code! While ResourceLoader and traditional
methods of adding scripts to MediaWiki output can co-exist, the
performance gains of ResourceLoader are directly related to the amount
of software utilizing it. There's some more stuff in core that needs to
be tweaked to utilize the ResourceLoader system, such as user scripts
and site CSS. We also need extensions to start using it, especially
those we are deploying on Wikimedia sites or thinking about deploying
soon. Only basic documentation exists on how to port extensions, but
much more will be written very shortly and we (Roan and I) be leading by
example by porting the UsabilityInitiative extensions ourselves. If you
need help, we're usually on IRC. (See
http://www.mediawiki.org/wiki/ResourceLoader/Getting_Started )
* Help writing new code! While wikibits.js is now also known as the
"mediawiki.legacy.wikibits" module, the functionality that it and
basically all other existing MediaWiki JavaScript code provide is being
deprecated, in favor of new modules which take advantage of jQuery and
can be written using a lot less code while eliminating the current
dependence on a large number of globally accessible variables and
functions (see
http://www.mediawiki.org/wiki/ResourceLoader/JavaScript_Deprecations )
* Some patience and understanding... Please... While we are integrating
into trunk, things might break unexpectedly. We're diligently tracking
down issues and resolving them as fast as we can, but help in this
regard is much needed and really appreciated. But most of all, we're
sorry if something gets screwed up, and we're trying our best to make
this integration smooth.
* Enthusiasm!

Documentation is coming online as fast as we can write it. There's a
very detailed design specification document at
http://www.mediawiki.org/wiki/ResourceLoader/Design_Specification and
more information in general at
http://www.mediawiki.org/wiki/ResourceLoader , where we will be adding
more and more documentation as time goes on. If you can help with
documentation, please feel free to edit boldly - just try not to modify
the design specification unless you are also modifying the software :)

While this project has been bootstrapped by Roan and myself in a branch,
we're really excited about bringing it to trunk and hope the community
can start taking advantage of the new features right away.

Tracking bug for tracking things that ResourceLoader will fix:
http://bugzilla.wikimedia.org/show_bug.cgi?id=24415

Bugzilla Component:
https://bugzilla.wikimedia.org/buglist.cgi?query_format=advanced&bug_status=NEW&bug_status=ASSIGNED&bug_status=REOPENED&bug_status=VERIFIED&component=Resource%20Loader&resolution=---&product=MediaWiki

- Trevor (and Roan, who's committing the merge to SVN right now)

_______________________________________________
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: ResourceLoader, now in trunk! [ In reply to ]
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

On 9/3/2010 10:59 PM, Trevor Parscal wrote:
> - Trevor (and Roan, who's committing the merge to SVN right now)
>

Pssst, it's broken.

- --
var src=mediaWiki.config.get('server')+'/load.php
- --

Hurm, what's server?

>>> mediaWiki.config.get('server')
"http://www.thedarkcitadel.com/w/load.php"

http://www.thedarkcitadel.com/w/load.php/load.php => 404 => broken
legacy scripts => no working JavaScript
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.14 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iQEcBAEBCAAGBQJMgc5/AAoJEL+AqFCTAyc2LEMIAKQaeR9jMLXdXE+s366n2Sj/
HHW6Wu8bJ8gQ60gMdF1RYkIFWQLRfVyb63vl5cw4QinhBdOHrkXzzicfy7UhGqcf
JWNeAb/qF8y+ZBb4HI2af6OdvHoGhow4udzB2yUGIShhQEPpTfnHNFgR0M/mMo8L
5jhyizIaVc/yWo0EeYQG4RTU4BzjyQCWZlR/bFksCLpgyQ/deF/voPL2XFGRUPXR
31/lOlG8+9BiT741AbFsUNaa6VhFH0iZ/5UExigs/9HyJwq/E9lcBTQow3mIEXXN
oYWOAX/AXmE80E+VclVSh7DqEGiLZSVBX8RW4kGCJr9Fj+KzQhvEGsGEDuS7yro=
=LHAO
-----END PGP SIGNATURE-----

_______________________________________________
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: ResourceLoader, now in trunk! [ In reply to ]
PHP Path-info made this work, despite being horribly broken.

Fixed in r72355. Thanks for poking!

- Trevor

On 9/3/10 9:43 PM, Q wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA256
>
> On 9/3/2010 10:59 PM, Trevor Parscal wrote:
>> - Trevor (and Roan, who's committing the merge to SVN right now)
>>
> Pssst, it's broken.
>
> - --
> var src=mediaWiki.config.get('server')+'/load.php
> - --
>
> Hurm, what's server?
>
>>>> mediaWiki.config.get('server')
> "http://www.thedarkcitadel.com/w/load.php"
>
> http://www.thedarkcitadel.com/w/load.php/load.php => 404 => broken
> legacy scripts => no working JavaScript
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v2.0.14 (MingW32)
> Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
>
> iQEcBAEBCAAGBQJMgc5/AAoJEL+AqFCTAyc2LEMIAKQaeR9jMLXdXE+s366n2Sj/
> HHW6Wu8bJ8gQ60gMdF1RYkIFWQLRfVyb63vl5cw4QinhBdOHrkXzzicfy7UhGqcf
> JWNeAb/qF8y+ZBb4HI2af6OdvHoGhow4udzB2yUGIShhQEPpTfnHNFgR0M/mMo8L
> 5jhyizIaVc/yWo0EeYQG4RTU4BzjyQCWZlR/bFksCLpgyQ/deF/voPL2XFGRUPXR
> 31/lOlG8+9BiT741AbFsUNaa6VhFH0iZ/5UExigs/9HyJwq/E9lcBTQow3mIEXXN
> oYWOAX/AXmE80E+VclVSh7DqEGiLZSVBX8RW4kGCJr9Fj+KzQhvEGsGEDuS7yro=
> =LHAO
> -----END PGP SIGNATURE-----
>
> _______________________________________________
> Wikitech-l mailing list
> Wikitech-l@lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l


_______________________________________________
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: ResourceLoader, now in trunk! [ In reply to ]
On 4 September 2010 06:59, Trevor Parscal <tparscal@wikimedia.org> wrote:
>  MediaWiki Developers,

> * Right-to-left support in CSS is akward. Stylesheets for right-to-left
> must to be either hand-coded in a separate stylesheet, generated each
> time a change is made by running CSSJanus, or an extra style-sheet which
> contains a series of over-rides.

> * Performs automatic left-to-right/right-to-left flipping for CSS files.
> In most cases the developer won't have to do anything before deploying.

Does this affect in any way the possibility of fixing the
long-standing LTR-RTL problem, where page direction depends on the
content language? It should depend on the user language instead, and
it should be possible to have content in different direction. Or is
this just automation of the work which was previously done by hand or
not done at all?

-Niklas

--
Niklas Laxström

_______________________________________________
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: ResourceLoader, now in trunk! [ In reply to ]
When you combine ResourceLoader's ability to flip css on the fly with
changes made in r72366, what you are asking for should work with just a
bit more tweaking. r72367 and 72368 make strides towards solving the
issue of going to Arabic Wikipedia and using English as your language,
ensuring we don't just flip on RTL, we flip when the user language is
not the same direction as the content language, and only do this for CSS
hosted on the wiki. Unfortunately because site CSS is still not passed
through ResourceLoader this is not a complete solution.

- Trevor

On 9/4/10 1:11 AM, Niklas Laxström wrote:
> On 4 September 2010 06:59, Trevor Parscal<tparscal@wikimedia.org> wrote:
>> MediaWiki Developers,
>> * Right-to-left support in CSS is akward. Stylesheets for right-to-left
>> must to be either hand-coded in a separate stylesheet, generated each
>> time a change is made by running CSSJanus, or an extra style-sheet which
>> contains a series of over-rides.
>> * Performs automatic left-to-right/right-to-left flipping for CSS files.
>> In most cases the developer won't have to do anything before deploying.
> Does this affect in any way the possibility of fixing the
> long-standing LTR-RTL problem, where page direction depends on the
> content language? It should depend on the user language instead, and
> it should be possible to have content in different direction. Or is
> this just automation of the work which was previously done by hand or
> not done at all?
>
> -Niklas
>


_______________________________________________
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: ResourceLoader, now in trunk! [ In reply to ]
On 09/04/2010 05:59 AM, Trevor Parscal wrote:
> Over the past couple of months, Roan Kattouw and I (Trevor Parscal) have
> been working on a JavaScript and CSS delivery system called
> ResourceLoader. We're really excited about this technology, and hope
> others will be too.

Cool! There have been many complaints with the introduction of Vector
that Wikimedia sites are taking longer to load, hopefully this will be
fixed soon. :)
Re: ResourceLoader, now in trunk! [ In reply to ]
church.of.emacs.ml wrote:
> On 09/04/2010 05:59 AM, Trevor Parscal wrote:
>> Over the past couple of months, Roan Kattouw and I (Trevor Parscal) have
>> been working on a JavaScript and CSS delivery system called
>> ResourceLoader. We're really excited about this technology, and hope
>> others will be too.
>
> Cool! There have been many complaints with the introduction of Vector
> that Wikimedia sites are taking longer to load, hopefully this will be
> fixed soon. :)

It will be interesting to see the reactions for the sites which get
vector and the resourceloader at the same time.


_______________________________________________
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: ResourceLoader, now in trunk! [ In reply to ]
On 9/4/10 8:31 AM, Platonides wrote:
> church.of.emacs.ml wrote:
>> On 09/04/2010 05:59 AM, Trevor Parscal wrote:
>>> Over the past couple of months, Roan Kattouw and I (Trevor Parscal) have
>>> been working on a JavaScript and CSS delivery system called
>>> ResourceLoader. We're really excited about this technology, and hope
>>> others will be too.
>> Cool! There have been many complaints with the introduction of Vector
>> that Wikimedia sites are taking longer to load, hopefully this will be
>> fixed soon. :)
> It will be interesting to see the reactions for the sites which get
> vector and the resourceloader at the same time.
>
>
> _______________________________________________
> Wikitech-l mailing list
> Wikitech-l@lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Well, given we've recently switched all projects to Vector (are there
any we have yet to switch that I don't know of?) that's not really
possible anymore.

- Trevor

_______________________________________________
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: ResourceLoader, now in trunk! [ In reply to ]
> Well, given we've recently switched all projects to Vector (are there
> any we have yet to switch that I don't know of?) that's not really
> possible anymore.
>

We could try asking someone from outside Wikimedia :p

Conrad

_______________________________________________
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: ResourceLoader, now in trunk! [ In reply to ]
On Fri, Sep 3, 2010 at 11:59 PM, Trevor Parscal <tparscal@wikimedia.org> wrote:
> Over the past couple of months, Roan Kattouw and I (Trevor Parscal) have
> been working on a JavaScript and CSS delivery system called
> ResourceLoader. We're really excited about this technology, and hope
> others will be too.

Awesome! Now maybe our front-end code won't be absolutely horrible.
If only browsers supported some way of debugging minified code sanely
. . .

> What do you want from me?
>
> . . .
> * Enthusiasm!

I'll stick to this, since I've never done too much front-end work. :)

On Sat, Sep 4, 2010 at 10:17 AM, church.of.emacs.ml
<church.of.emacs.ml@googlemail.com> wrote:
> Cool! There have been many complaints with the introduction of Vector
> that Wikimedia sites are taking longer to load, hopefully this will be
> fixed soon. :)

That's mainly due to JS execution time, isn't it? A resource loader
won't speed up page loads much if at all on a hot cache, so this
probably won't offset that.

_______________________________________________
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: ResourceLoader, now in trunk! [ In reply to ]
At 2010-09-04 05:59, Trevor Parscal wrote:
> * Allows minification of JavaScript and CSS.

Will JS/CSS developers/testers be able to disable this in preferences?

> * Allows deployment changes to all pages for all users within minutes,
> without purging any HTML. ResourceLoader provides a short-expiry
> start-up script which then decides to continue loading more JavaScript
> or not, and if so has a complete manifest of all scripts and styles on
> the server and their most recent versions, Also, this startup script
> will be able to be inlined using ESI (see
> http://en.wikipedia.org/wiki/Edge_Side_Includes ) when using Squid or
> Varnish, reducing requests and improving performance even further.

So any change to MediaWiki:Common.js will refresh all script that are
minified?

> * Help writing new code! While wikibits.js is now also known as the
> "mediawiki.legacy.wikibits" module, the functionality that it and
> basically all other existing MediaWiki JavaScript code provide is being
> deprecated, in favor of new modules which take advantage of jQuery and
> can be written using a lot less code while eliminating the current
> dependence on a large number of globally accessible variables and
> functions (see
> http://www.mediawiki.org/wiki/ResourceLoader/JavaScript_Deprecations )

Will jQuery be available for older skins too? If so won't it cause
problems for mobile browsers? AFAIK mobile jQuery will not be ready for
at least next few years.

I'm asking here because those that make gadgets (well me at least ;-))
still try to make them work on Vector and Monobook.

> * Some patience and understanding... Please... While we are integrating
> into trunk, things might break unexpectedly. We're diligently tracking
> down issues and resolving them as fast as we can, but help in this
> regard is much needed and really appreciated. But most of all, we're
> sorry if something gets screwed up, and we're trying our best to make
> this integration smooth.
> * Enthusiasm!

Sorry for not being very enthusiastic ;-). I see this as solving some
problems but creating new ones too. Not really complaining here, I think
minification is really needed now to get Wikimedia sites loading faster.

Regards,
Nux.

_______________________________________________
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: ResourceLoader, now in trunk! [ In reply to ]
2010/9/5 Aryeh Gregor <Simetrical+wikilist@gmail.com>:
> On Fri, Sep 3, 2010 at 11:59 PM, Trevor Parscal <tparscal@wikimedia.org> wrote:
>> Over the past couple of months, Roan Kattouw and I (Trevor Parscal) have
>> been working on a JavaScript and CSS delivery system called
>> ResourceLoader. We're really excited about this technology, and hope
>> others will be too.
>
> Awesome!  Now maybe our front-end code won't be absolutely horrible.
> If only browsers supported some way of debugging minified code sanely
> . . .
Or just use ?debug=true and we'll serve you unminified, uncombined code.

Roan Kattouw (Catrope)

_______________________________________________
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: ResourceLoader, now in trunk! [ In reply to ]
2010/9/6 Maciej Jaros <egil@wp.pl>:
>  At 2010-09-04 05:59, Trevor Parscal wrote:
>> * Allows minification of JavaScript and CSS.
>
> Will JS/CSS developers/testers be able to disable this in preferences?
>
Like I said in my previous reply to Aryeh, you can use debug mode.
This is a feature we probably should've mentioned in the OP.

>> * Allows deployment changes to all pages for all users within minutes,
>> without purging any HTML. ResourceLoader provides a short-expiry
>> start-up script which then decides to continue loading more JavaScript
>> or not, and if so has a complete manifest of all scripts and styles on
>> the server and their most recent versions,  Also, this startup script
>> will be able to be inlined using ESI (see
>> http://en.wikipedia.org/wiki/Edge_Side_Includes ) when using Squid or
>> Varnish, reducing requests and improving performance even further.
>
> So any change to MediaWiki:Common.js will refresh all script that are
> minified?
>
We currently serve site JS (Common.js and skinname.js combined; this
combining was done before RL too) separately because it needs to run
at the very end. This is because it assumes it runs in the global
scope, so it can't be wrapped in a closure. We also can't concatenate
unwrapped site JS to the end of wrapped modules, because that will
result in the site JS being executed before e.g. wikibits, which will
also break. The only reliable way at this time is to include site JS
separately after the mediaWiki.loader.go(); call.

This means it's currently not combined with anything else, so
invalidations of the site scripts won't invalidate anything else.
Treating site JS like a normal module is only possible once site
scripts have been ported to act like modules so they won't explode
when wrapped in a closure. Depending on wikibits is fine as we already
have dependency handling.

>> * Help writing new code! While wikibits.js is now also known as the
>> "mediawiki.legacy.wikibits" module, the functionality that it and
>> basically all other existing MediaWiki JavaScript code provide is being
>> deprecated, in favor of new modules which take  advantage of jQuery and
>> can be written using a lot less code while eliminating the current
>> dependence on a large number of globally accessible variables and
>> functions (see
>> http://www.mediawiki.org/wiki/ResourceLoader/JavaScript_Deprecations )
>
> Will jQuery be available for older skins too? If so won't it cause
> problems for mobile browsers? AFAIK mobile jQuery will not be ready for
> at least next few years.
>
Yes, jQuery will be served on every page load. However, the (short)
startup script that contains the module registration information and
loads the jquery and mediawiki modules checks the browser version for
compatibility before doing the latter. Currently the only browser it
considers incompatible is IE <6, but this list should probably be
extended to include other stone age browsers and mobile browsers. For
the latter, we can even do the redirect to the mobile site in that
place so we don't load a ton of JS before deciding to redirect people.
OTOH, it might be nice to put a mobile redirect script in the <head>
so the browser doesn't have to render the entire page before it hits
the redirect script.

> I'm asking here because those that make gadgets (well me at least ;-))
> still try to make them work on Vector and Monobook.
>
That's one of the objectives here: to provide a common base library
that everything in MW can build on, regardless of skin or whatever. To
that end, we started serving jQuery on every page on Wikimedia wikis
last week, and we already did so in trunk ($wgJQueryOnEveryPage) for a
few months IIRC.

> Sorry for not being very enthusiastic ;-). I see this as solving some
> problems but creating new ones too. Not really complaining here, I think
> minification is really needed now to get Wikimedia sites loading faster.
>
Which problems would this introduce? The ones you mentioned in this
post have either already been solved or are trivially solvable.

Roan Kattouw (Catrope)

_______________________________________________
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: ResourceLoader, now in trunk! [ In reply to ]
Roan Kattouw wrote:
> For the latter, we can even do the redirect to the mobile site in that
> place so we don't load a ton of JS before deciding to redirect people.
> OTOH, it might be nice to put a mobile redirect script in the <head>
> so the browser doesn't have to render the entire page before it hits
> the redirect script.

I was under the impression the goal for the mobile redirection was to make
it entirely server-side, not client-side. That is, the goal isn't to move
the script to <head>, it's to remove the need for a script altogether. More
info at https://bugzilla.wikimedia.org/show_bug.cgi?id=24859

MZMcBride



_______________________________________________
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: ResourceLoader, now in trunk! [ In reply to ]
2010/9/6 MZMcBride <z@mzmcbride.com>:
> I was under the impression the goal for the mobile redirection was to make
> it entirely server-side, not client-side. That is, the goal isn't to move
> the script to <head>, it's to remove the need for a script altogether. More
> info at https://bugzilla.wikimedia.org/show_bug.cgi?id=24859
>
Ultimately, yes. We discussed this briefly last week while giving Mark
a rundown of the resource loader, and Mark said that server-side
UA-based redirects are definitely feasible, but only after we move to
Varnish for our text caching (we currently use Varnish on bits for
JS/CSS caching and Squid for text and image caching). From what I
understood and remember, the Varnish migration is not really a top
priority right now and is expected to be completed in a year or so. In
the meantime, we can make the mobile redirect suck slightly less.

Roan Kattouw (Catrope)

_______________________________________________
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: ResourceLoader, now in trunk! [ In reply to ]
On Mon, Sep 6, 2010 at 10:15 AM, Roan Kattouw <roan.kattouw@gmail.com> wrote:
> Or just use ?debug=true and we'll serve you unminified, uncombined code.

Then you have to hope that the bug still occurs. Probably it will,
but you never know. Plus, it would be nice if there weren't that
extra step. Of course, even if they de-minified things, you'd still
have no comments and so on, but at least the line number would mean
something.

_______________________________________________
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: ResourceLoader, now in trunk! [ In reply to ]
Roan Kattouw wrote:
> Yes, jQuery will be served on every page load. However, the (short)
> startup script that contains the module registration information and
> loads the jquery and mediawiki modules checks the browser version for
> compatibility before doing the latter. Currently the only browser it
> considers incompatible is IE <6, but this list should probably be
> extended to include other stone age browsers and mobile browsers. For
> the latter, we can even do the redirect to the mobile site in that
> place so we don't load a ton of JS before deciding to redirect people.
> OTOH, it might be nice to put a mobile redirect script in the <head>
> so the browser doesn't have to render the entire page before it hits
> the redirect script.

Gadgets should be minifiable, too.
Perhaps having an update add an standalone flag to the gadgets
definitions, and minify everything without it.
It would also be nice to be able to specify a filter there on which the
is run (Special:Upload, namespace 2...)


_______________________________________________
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: ResourceLoader, now in trunk! [ In reply to ]
2010/9/6 Aryeh Gregor <Simetrical+wikilist@gmail.com>:
> Then you have to hope that the bug still occurs.  Probably it will,
> but you never know.  Plus, it would be nice if there weren't that
> extra step.  Of course, even if they de-minified things, you'd still
> have no comments and so on, but at least the line number would mean
> something.
>
Actually, the line number would not mean a great deal because the
scripts would still be combined, comments frequently take up entire
lines and the deminifier cannot account for blank lines, statements
broken over multiple lines, or multiple statements on one line. All
this means it's impossible to map the line number to a source line and
file, although I agree it does map it to a statement: someone else can
add a breakpoint at the same line of the same combined+minified output
(provided they're also hitting the same load.php URL) and have that
break at the exact statement the reporter got their error on.

But all that is just theorizing about deminifying debuggers, which I
don't think exist right now. The only thing we can do in practice is
change the environment to remove the impediments to debugging we
introduced and hope that doesn't make any bugs go away. We do this not
only by obtaining each module's JS unminified in a separate request
(unfortunately, we can't request CSS style sheets separately for
dynamically requested modules because the technique needed for this is
not as widely supported), but also by ensuring that all manipulation
of JS and CSS we do (such as wrapping JS in a
mediaWiki.loader.implement() call, direction flipping in CSS and image
embedding in CSS) don't introduce or remove any lines, thereby
preserving line numbers.

Roan Kattouw (Catrope)

_______________________________________________
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: ResourceLoader, now in trunk! [ In reply to ]
2010/9/6 Platonides <Platonides@gmail.com>:
> Gadgets should be minifiable, too.
Should be a matter of porting the Gadgets extension. Will need a bit
of smartness because gadgets are wiki pages rather than files, but
it's definitely doable.

> Perhaps having an update add an standalone flag to the gadgets
> definitions, and minify everything without it.
You mean something like a permanent debug or compatibility mode for
selected gadgets?

> It would also be nice to be able to specify a filter there on which the
> is run (Special:Upload, namespace 2...)
>
That's actually something you can do with a custom loader script. This
allows you to write the code calling mediaWiki.loader.register()
yourself rather than having it generated by the resource loader
(default), and to dynamically generate dependency information on the
fly (by putting your dependency generation code in a closure that's
only run when the module's dependency information is needed). At the
point that script is run you don't have access to environment
variables like the title and namespace yet, but the dependency
registration closure does, so you can hack around it by adding an
empty module that either does or doesn't depend on a second module
with the actual code (eww, maybe we should add better support for
this; typically this would be checked on the server side before adding
the module or on the client side before requesting the module, though,
but Gadgets can't do that easily).

Unfortunately, I don't think these loader scripts currently allow you
to override some but not all parts of the registration call, it's
either all or nothing. This is particularly annoying since the last
modified timestamp of a module is hard to obtain on the client side.
Filed this as https://bugzilla.wikimedia.org/show_bug.cgi?id=25085 .

Roan Kattouw (Catrope)

_______________________________________________
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: ResourceLoader, now in trunk! [ In reply to ]
Roan Kattouw wrote:
> 2010/9/6 Platonides <Platonides@gmail.com>:
>> Gadgets should be minifiable, too.
> Should be a matter of porting the Gadgets extension. Will need a bit
> of smartness because gadgets are wiki pages rather than files, but
> it's definitely doable.
>
>> Perhaps having an update add an standalone flag to the gadgets
>> definitions, and minify everything without it.
> You mean something like a permanent debug or compatibility mode for
> selected gadgets?

Yes. If Common.js has issues for minimifying, gadgets could as well. I
first thought at it as "legacy", but there might be some reason for
wanting it loaded separately. As far as not providing any flag,
minifyies it, that shouldn't be a problem.


>> It would also be nice to be able to specify a filter there on which the
>> is run (Special:Upload, namespace 2...)
>>
> That's actually something you can do with a custom loader script. This
> allows you to write the code calling mediaWiki.loader.register()
> yourself rather than having it generated by the resource loader
> (default), and to dynamically generate dependency information on the
> fly (by putting your dependency generation code in a closure that's
> only run when the module's dependency information is needed). At the
> point that script is run you don't have access to environment
> variables like the title and namespace yet, but the dependency
> registration closure does, so you can hack around it by adding an
> empty module that either does or doesn't depend on a second module
> with the actual code (eww, maybe we should add better support for
> this; typically this would be checked on the server side before adding
> the module or on the client side before requesting the module, though,
> but Gadgets can't do that easily).

Maybe adding a gadget module which in turns loads the others as needed?
I'm not familiar with the resource loaded code to determine the best way
to design it.

> Unfortunately, I don't think these loader scripts currently allow you
> to override some but not all parts of the registration call, it's
> either all or nothing. This is particularly annoying since the last
> modified timestamp of a module is hard to obtain on the client side.
> Filed this as https://bugzilla.wikimedia.org/show_bug.cgi?id=25085 .
>
> Roan Kattouw (Catrope)

And I don't understand what this last paragraph is about :) You are
trying to reuse the same loader for all gadgets?


_______________________________________________
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: ResourceLoader, now in trunk! [ In reply to ]
2010/9/7 Platonides <Platonides@gmail.com>:
> Yes. If Common.js has issues for minimifying, gadgets could as well. I
> first thought at it as "legacy", but there might be some reason for
> wanting it loaded separately. As far as not providing any flag,
> minifyies it, that shouldn't be a problem.
>
It's likely that gadgets will have issues similar to Common.js's, yes,
in which case it could be handy to still load them in legacy mode
unless the gadgets definition page indicates it's RL-compatible.
(That's another feature request: we should introduce that concept of
legacy mode so we can combine scripts loaded in that mode rather than
loading all of them separately.)

> Maybe adding a gadget module which in turns loads the others as needed?
That could work, yes, but it'd mean the loader only finds out which
gadgets it needs after the gadget module runs, which means it has to
do an extra request back to the server to get them.

> I'm not familiar with the resource loaded code to determine the best way
> to design it.
>
Basically the relevant facts are:
* Module information is defined server-side and transfered to the
client-side loader through either a standard
mediaWiki.loader.register() call generated by the server or a custom
registration script (called 'loader script' in the code) if defined
* If server-side PHP code knows it needs a module, it can ask
OutputPage to include it on a page. It does this by adding a call to
mediaWiki.loader.load()
* Near the bottom of the page (below everything but legacy-mode
scripts, currently only site JS), mediaWiki.loader.go() is called. It
consolidates all load() calls, resolves dependencies and obtains all
required modules from the server in one request
* Once these modules are loaded, their code is executed (in the
correct order to satisfy dependencies)
* On-demand loading after go() has happened can be done with
mw.loader.using( 'modulename', function() { code using module } );
This will load the module and its dependencies, execute them, then
execute the function provided to using().

This means that a gadget module that figures out which other modules
to load could only use the on-demand mechanism: the other mechanisms
are only available after the gadget module has been loaded. Using
on-demand loading results in an additional request and should only be
done when loading a relatively large portion of code that is only
required upon a user action (e.g. jquery.ui.dialog when the user
clicks one of the edit toolbar buttons associated with dialogs). Using
on-demand loading for things that are immediately required on page
load should be avoided if possible (and this usually is possible).

>> Unfortunately, I don't think these loader scripts currently allow you
>> to override some but not all parts of the registration call, it's
>> either all or nothing. This is particularly annoying since the last
>> modified timestamp of a module is hard to obtain on the client side.
>> Filed this as https://bugzilla.wikimedia.org/show_bug.cgi?id=25085 .
>>
>> Roan Kattouw (Catrope)
>
> And I don't understand what this last paragraph is about :) You are
> trying to reuse the same loader for all gadgets?
>
What I was saying is that when you use a custom registration script,
that script is responsible for providing the loader with all the
information it needs (module name, dependencies and last modified
timestamp) without any help from the server (usually all of this
information is defined on the server which then generates code to
inform the loader). This is a problem, especially for the timestamp
parameter, so I filed a bug requesting we make the information defined
on the server side available to client-side custom loaders.

Roan Kattouw (Catrope)

_______________________________________________
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: ResourceLoader, now in trunk! [ In reply to ]
AT 2010-09-06 18:26, Roan Kattouw wrote:
> 2010/9/6 Maciej Jaros<egil@wp.pl>:
>> At 2010-09-04 05:59, Trevor Parscal wrote:
>>> * Allows minification of JavaScript and CSS.
>> Will JS/CSS developers/testers be able to disable this in preferences?
>>
> Like I said in my previous reply to Aryeh, you can use debug mode.
> This is a feature we probably should've mentioned in the OP.

This is good, but an option set in preferences would be more efficient
when debugging more then one page. But if it would read _GET or _COOKIE
then it would be fine for me, too.

>>> * Help writing new code! While wikibits.js is now also known as the
>>> "mediawiki.legacy.wikibits" module, the functionality that it and
>>> basically all other existing MediaWiki JavaScript code provide is being
>>> deprecated, in favor of new modules which take advantage of jQuery and
>>> can be written using a lot less code while eliminating the current
>>> dependence on a large number of globally accessible variables and
>>> functions (see
>>> http://www.mediawiki.org/wiki/ResourceLoader/JavaScript_Deprecations )
>> Will jQuery be available for older skins too? If so won't it cause
>> problems for mobile browsers? AFAIK mobile jQuery will not be ready for
>> at least next few years.
>>
> Yes, jQuery will be served on every page load. However, the (short)
> startup script that contains the module registration information and
> loads the jquery and mediawiki modules checks the browser version for
> compatibility before doing the latter. Currently the only browser it
> considers incompatible is IE<6, but this list should probably be
> extended to include other stone age browsers and mobile browsers. For
> the latter, we can even do the redirect to the mobile site in that
> place so we don't load a ton of JS before deciding to redirect people.
> OTOH, it might be nice to put a mobile redirect script in the<head>
> so the browser doesn't have to render the entire page before it hits
> the redirect script.

That seems like a good idea. I've just found some data about jQuery tests:
http://jquerymobile.com/gbs/
As seems I was a little over pessimistic about jQuery for mobile
devices. None the less you can use the table on the page for
redirections. I think everything below B should be redirected to mobile
version. B might be left on monobook and A should work fine with any
skin (or least surrvive).

>> I'm asking here because those that make gadgets (well me at least ;-))
>> still try to make them work on Vector and Monobook.
>>
> That's one of the objectives here: to provide a common base library
> that everything in MW can build on, regardless of skin or whatever. To
> that end, we started serving jQuery on every page on Wikimedia wikis
> last week, and we already did so in trunk ($wgJQueryOnEveryPage) for a
> few months IIRC.

Cool. Just announced it on pl.wiki news page.

>> Sorry for not being very enthusiastic ;-). I see this as solving some
>> problems but creating new ones too. Not really complaining here, I think
>> minification is really needed now to get Wikimedia sites loading faster.
> Which problems would this introduce? The ones you mentioned in this
> post have either already been solved or are trivially solvable.

I fear mainly that there will be reproduction issues and I'm probably
just a bit superstitious ;-). I'm also a bit worried how current various
loading mechanism will accompany new ones.

Regards,
Nux.

_______________________________________________
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: ResourceLoader, now in trunk! [ In reply to ]
Roan Kattouw wrote:
>> Maybe adding a gadget module which in turns loads the others as needed?
> That could work, yes, but it'd mean the loader only finds out which
> gadgets it needs after the gadget module runs, which means it has to
> do an extra request back to the server to get them.

I see. It would have to be a hook into the resource loader.


_______________________________________________
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: ResourceLoader, now in trunk! [ In reply to ]
* Aryeh Gregor <Simetrical+wikilist@gmail.com> [Mon, 6 Sep 2010 14:31:45
-0400]:
> On Mon, Sep 6, 2010 at 10:15 AM, Roan Kattouw <roan.kattouw@gmail.com>
> wrote:
> > Or just use ?debug=true and we'll serve you unminified, uncombined
> code.
>
> Then you have to hope that the bug still occurs. Probably it will,
> but you never know. Plus, it would be nice if there weren't that
> extra step. Of course, even if they de-minified things, you'd still
> have no comments and so on, but at least the line number would mean
> something.
>
Do browsers support streamed and combined (archive-like) gz-encoded
content? Then probably minifying will not be neccessary. Also, it would
be great if these high-level JS-libraries like jQuery actually were
ported into DOM API level (native browser's implementation instead of
extra JS layer). However, these questions are to FF/IE/Opera
developers...
Dmitriy

_______________________________________________
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: ResourceLoader, now in trunk! [ In reply to ]
2010/9/7 Platonides <Platonides@gmail.com>:
> I see. It would have to be a hook into the resource loader.
You don't need a hook in the server-side PHP part of the resource
loader, as determining which modules to load on that side is already
supported. The Gadgets extension's PHP code could inspect variables
like $wgTitle->getNamespace() and determine whether to load a certain
gadget or not. This is easy as far as the resource loader goes, but it
does mean the server has to make this decision based on information in
the gadget's code or the gadget definition page. This is significantly
less flexible than allowing a gadget to just provide a JS function
that returns true or false, but the fact that the decision has to be
made client-side makes it a bit trickier. It's still very much doable,
although the way I proposed is a little bit hacky and we may want to
have the client-side loader support this better.

Or did you mean a hook into the client-side loader?

Roan Kattouw (Catrope)

_______________________________________________
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

1 2 3  View All