Mailing List Archive

Module storage is coming
The roll-out of 1.23wmf2 to your favorite Wikimedia wiki will
inaugurate the era of ResourceLoader module storage -- an era which
will be marked by terrifying and hilarious new bugs, intermittent
client-side failures, and generally inexcusable disruptions to user
experience. Sounds exciting? Read on!

What is it?
-----------
"Module storage" refers to the use of localStorage in ResourceLoader
as an auxiliary to the browser cache. ResourceLoader, remember, is the
MediaWiki subsystem that delivers JavaScript and CSS to your browser.
One of its primary functions is to minimize the total number of
network requests that your browser needs to make in order to render
the page, and to make the remaining requests as slim as possible. One
of the ways ResourceLoader does this is by making a list of all the
different components your browser needs and then transmitting them in
bulk.

The downside of this approach is that a small update to MediaWiki's
code, touching perhaps one or two components, will often force the
browser to throw out (and re-retrieve) a bunch of unrelated modules
that did not change and did not ostensibly need to be re-retrieved.
This is because the browser cache operates on the level of network
requests; it is not intelligent enough to decompose a request into its
constituitive parts and to cache these parts separately from one
another.

Starting with the 1.23wmf2 branch, ResourceLoader will check if your
browser implements localStorage, a mechanism for storing structure
data. If it does, ResourceLoader will use it as an auxiliary cache,
capable of versioning individual components.

Why?
----
The primary goals are:

* Destabalize MediaWiki's front-end code, by coupling it to an
inconsistently-implemented and poorly-understood HTML5 API.
* Make debugging fun again, by adding another layer of caching to
MediaWiki. Yes!!

However, as with all new features, unintended side-effects are
possible. Specific concerns for module storage include the risk of
making the network footprint of page loads smaller, resulting in
improved latency.

But seriously,
--------------
* Module storage is enabled only if the underlying browser supports
localStorage (~89% of browsers in use, according to
<http://caniuse.com/#feat=namevalue-storage>). Users with older
browsers will not see a benefit, but they will not suffer a penalty
either.
* Module storage may or may not improve site performance. We need to
test it against a wide range of browsers and platforms to know for
certain. If it doesn't improve performance, we'll blame it on you,
your poor choice of browsers, and your uncooperative attitude, but
then we'll scrap it.

How can I test it?
------------------
Glad you asked! Module storage is enabled by default on the beta
cluster, and on test & test2 wikis. The simplest way you can help is
by being alert to to performance regressions and front-end code
breakage. If you know how to use your browser's JavaScript console,
you can get stats about module storage usage on the current page by
checking the value of 'mw.loader.store.stats'.

When the code is rolled out across the cluster, it will be disabled by
default, but you will be able to opt-in by manually setting a cookie
in your browser. I will follow up with the exact details. If module
storage proves stable enough for production use, we'll seek to asses
its utility by running a controlled study of some kind. If we can
demonstrate that module storage leads to a significant improvement in
performance, we'll enable by it default.

---
Ori Livneh
ori@wikimedia.org

_______________________________________________
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: Module storage is coming [ In reply to ]
MediaWiki: Manually fixing broken browser functionality since 2012.

Also I do wonder how useful this actually is. Does site JavaScript really
change that often? I suppose we'll find out after testing.

*-- *
*Tyler Romeo*
Stevens Institute of Technology, Class of 2016
Major in Computer Science


On Sun, Nov 3, 2013 at 8:27 PM, Ori Livneh <ori@wikimedia.org> wrote:

> The roll-out of 1.23wmf2 to your favorite Wikimedia wiki will
> inaugurate the era of ResourceLoader module storage -- an era which
> will be marked by terrifying and hilarious new bugs, intermittent
> client-side failures, and generally inexcusable disruptions to user
> experience. Sounds exciting? Read on!
>
> What is it?
> -----------
> "Module storage" refers to the use of localStorage in ResourceLoader
> as an auxiliary to the browser cache. ResourceLoader, remember, is the
> MediaWiki subsystem that delivers JavaScript and CSS to your browser.
> One of its primary functions is to minimize the total number of
> network requests that your browser needs to make in order to render
> the page, and to make the remaining requests as slim as possible. One
> of the ways ResourceLoader does this is by making a list of all the
> different components your browser needs and then transmitting them in
> bulk.
>
> The downside of this approach is that a small update to MediaWiki's
> code, touching perhaps one or two components, will often force the
> browser to throw out (and re-retrieve) a bunch of unrelated modules
> that did not change and did not ostensibly need to be re-retrieved.
> This is because the browser cache operates on the level of network
> requests; it is not intelligent enough to decompose a request into its
> constituitive parts and to cache these parts separately from one
> another.
>
> Starting with the 1.23wmf2 branch, ResourceLoader will check if your
> browser implements localStorage, a mechanism for storing structure
> data. If it does, ResourceLoader will use it as an auxiliary cache,
> capable of versioning individual components.
>
> Why?
> ----
> The primary goals are:
>
> * Destabalize MediaWiki's front-end code, by coupling it to an
> inconsistently-implemented and poorly-understood HTML5 API.
> * Make debugging fun again, by adding another layer of caching to
> MediaWiki. Yes!!
>
> However, as with all new features, unintended side-effects are
> possible. Specific concerns for module storage include the risk of
> making the network footprint of page loads smaller, resulting in
> improved latency.
>
> But seriously,
> --------------
> * Module storage is enabled only if the underlying browser supports
> localStorage (~89% of browsers in use, according to
> <http://caniuse.com/#feat=namevalue-storage>). Users with older
> browsers will not see a benefit, but they will not suffer a penalty
> either.
> * Module storage may or may not improve site performance. We need to
> test it against a wide range of browsers and platforms to know for
> certain. If it doesn't improve performance, we'll blame it on you,
> your poor choice of browsers, and your uncooperative attitude, but
> then we'll scrap it.
>
> How can I test it?
> ------------------
> Glad you asked! Module storage is enabled by default on the beta
> cluster, and on test & test2 wikis. The simplest way you can help is
> by being alert to to performance regressions and front-end code
> breakage. If you know how to use your browser's JavaScript console,
> you can get stats about module storage usage on the current page by
> checking the value of 'mw.loader.store.stats'.
>
> When the code is rolled out across the cluster, it will be disabled by
> default, but you will be able to opt-in by manually setting a cookie
> in your browser. I will follow up with the exact details. If module
> storage proves stable enough for production use, we'll seek to asses
> its utility by running a controlled study of some kind. If we can
> demonstrate that module storage leads to a significant improvement in
> performance, we'll enable by it default.
>
> ---
> Ori Livneh
> ori@wikimedia.org
>
> _______________________________________________
> Wikitech-l mailing list
> Wikitech-l@lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
_______________________________________________
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: Module storage is coming [ In reply to ]
On Nov 4, 2013 2:28 AM, "Ori Livneh" <ori@wikimedia.org> wrote:
>
> The roll-out of 1.23wmf2 to your favorite Wikimedia wiki will
> inaugurate the era of ResourceLoader module storage -- an era which
> will be marked by terrifying and hilarious new bugs, intermittent
> client-side failures, and generally inexcusable disruptions to user
> experience. Sounds exciting? Read on!
>
> What is it?
> -----------
> "Module storage" refers to the use of localStorage in ResourceLoader
> as an auxiliary to the browser cache. ResourceLoader, remember, is the
> MediaWiki subsystem that delivers JavaScript and CSS to your browser.
> One of its primary functions is to minimize the total number of
> network requests that your browser needs to make in order to render
> the page, and to make the remaining requests as slim as possible. One
> of the ways ResourceLoader does this is by making a list of all the
> different components your browser needs and then transmitting them in
> bulk.
>
> The downside of this approach is that a small update to MediaWiki's
> code, touching perhaps one or two components, will often force the
> browser to throw out (and re-retrieve) a bunch of unrelated modules
> that did not change and did not ostensibly need to be re-retrieved.
> This is because the browser cache operates on the level of network
> requests; it is not intelligent enough to decompose a request into its
> constituitive parts and to cache these parts separately from one
> another.
>
> Starting with the 1.23wmf2 branch, ResourceLoader will check if your
> browser implements localStorage, a mechanism for storing structure
> data. If it does, ResourceLoader will use it as an auxiliary cache,
> capable of versioning individual components.
>
> Why?
> ----
> The primary goals are:
>
> * Destabalize MediaWiki's front-end code, by coupling it to an
> inconsistently-implemented and poorly-understood HTML5 API.
> * Make debugging fun again, by adding another layer of caching to
> MediaWiki. Yes!!
>
> However, as with all new features, unintended side-effects are
> possible. Specific concerns for module storage include the risk of
> making the network footprint of page loads smaller, resulting in
> improved latency.
>
> But seriously,
> --------------
> * Module storage is enabled only if the underlying browser supports
> localStorage (~89% of browsers in use, according to
> <http://caniuse.com/#feat=namevalue-storage>). Users with older
> browsers will not see a benefit, but they will not suffer a penalty
> either.
> * Module storage may or may not improve site performance. We need to
> test it against a wide range of browsers and platforms to know for
> certain. If it doesn't improve performance, we'll blame it on you,
> your poor choice of browsers, and your uncooperative attitude, but
> then we'll scrap it.
>
> How can I test it?
> ------------------
> Glad you asked! Module storage is enabled by default on the beta
> cluster, and on test & test2 wikis. The simplest way you can help is
> by being alert to to performance regressions and front-end code
> breakage. If you know how to use your browser's JavaScript console,
> you can get stats about module storage usage on the current page by
> checking the value of 'mw.loader.store.stats'.
>
> When the code is rolled out across the cluster, it will be disabled by
> default, but you will be able to opt-in by manually setting a cookie
> in your browser. I will follow up with the exact details. If module
> storage proves stable enough for production use, we'll seek to asses
> its utility by running a controlled study of some kind. If we can
> demonstrate that module storage leads to a significant improvement in
> performance, we'll enable by it default.
>

Sounds awesome. I'll wait patiently for hilarity to ensue.

> ---
> Ori Livneh
> ori@wikimedia.org
>
> _______________________________________________
> Wikitech-l mailing list
> Wikitech-l@lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
_______________________________________________
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: Module storage is coming [ In reply to ]
On 11/03/2013 08:27 PM, Ori Livneh wrote:
> * Module storage is enabled only if the underlying browser supports
> localStorage (~89% of browsers in use, according to
> <http://caniuse.com/#feat=namevalue-storage>). Users with older
> browsers will not see a benefit, but they will not suffer a penalty
> either.

Ori, Thank you for this fun-to-read introduction to Module Storage.

I wonder if this will have any affect on low bandwidth users. It seems
like this would help if someone had only a 2G/GPRS connection, but do a
lower percentage of those users have browsers that can use Module
Storage? If so, is there anything that we can to do address this issue?

I imagine there are things we could do in the software, but those things
would take work and time that might not make sense for limited
resources. Would it make sense to see if Mozilla has any interest in
helping to spread HTML5-capable browsers via USB keychains and/or local
Mozillians?

Hrm... I should probably go ask them about that. But I'm curious about
your perspective and to see if we have any information on the bandwidth
available to various users.

Mark.


_______________________________________________
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: Module storage is coming [ In reply to ]
On Mon, Nov 4, 2013 at 1:50 PM, Mark A. Hershberger <mah@nichework.com>wrote:

> Ori, Thank you for this fun-to-read introduction to Module Storage.


+1

Željko
_______________________________________________
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: Module storage is coming [ In reply to ]
On 04.11.2013, 16:50 Mark wrote:

> Ori, Thank you for this fun-to-read introduction to Module Storage.

> I wonder if this will have any affect on low bandwidth users. It seems
> like this would help if someone had only a 2G/GPRS connection, but do a
> lower percentage of those users have browsers that can use Module
> Storage? If so, is there anything that we can to do address this issue?

> I imagine there are things we could do in the software, but those things
> would take work and time that might not make sense for limited
> resources. Would it make sense to see if Mozilla has any interest in
> helping to spread HTML5-capable browsers via USB keychains and/or local
> Mozillians?

> Hrm... I should probably go ask them about that. But I'm curious about
> your perspective and to see if we have any information on the bandwidth
> available to various users.

Most of our mobile traffic is genrated by iOS and Android - both of
which support LocalStorage, so it will work for them too. The
situation for mobile users in developing countries is less certain -
lots of various browsers with wildly varied support of modern
features. Still, it's a huge improvement for the majority of users.

--
Best regards,
Max Semenik ([[User:MaxSem]])


_______________________________________________
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: Module storage is coming [ In reply to ]
On 11/04/2013 10:10 AM, Max Semenik wrote:
> Most of our mobile traffic is genrated by iOS and Android - both of
> which support LocalStorage, so it will work for them too.

People using 2G/GPRS are not necessarily using a mobile device. I've
heard from at least one user (hence
https://bugzilla.wikimedia.org/55842) that he is using his cell phone as
a modem.

I suspect this is not a small percentage of users in less developed
areas where cell phone towers are plentiful but cables are not.

Non-mobile UAs on mobile IPs is what I'm asking about.

Mark.


_______________________________________________
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: Module storage is coming [ In reply to ]
I'm looking forward to seeing how this plays out. The only downside I
can so far see is that the amount of browser storage available varies
drastically [1] and I wonder whether this will cause upsets for those
browsers with extremely strict limits.

I've also been toying with the idea of using localStorage and cache
manifests in mobile for offline usage in mobile for some time and this
will compete for that space and make that experiment if it ever
happens even more interesting. :-)

[1] http://dev-test.nemikor.com/web-storage/support-test/

On Mon, Nov 4, 2013 at 7:31 AM, Mark A. Hershberger <mah@nichework.com> wrote:
> On 11/04/2013 10:10 AM, Max Semenik wrote:
>> Most of our mobile traffic is genrated by iOS and Android - both of
>> which support LocalStorage, so it will work for them too.
>
> People using 2G/GPRS are not necessarily using a mobile device. I've
> heard from at least one user (hence
> https://bugzilla.wikimedia.org/55842) that he is using his cell phone as
> a modem.
>
> I suspect this is not a small percentage of users in less developed
> areas where cell phone towers are plentiful but cables are not.
>
> Non-mobile UAs on mobile IPs is what I'm asking about.
>
> Mark.
>
>
> _______________________________________________
> Wikitech-l mailing list
> Wikitech-l@lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l



--
Jon Robson
http://jonrobson.me.uk
@rakugojon

_______________________________________________
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: Module storage is coming [ In reply to ]
On Mon, Nov 4, 2013 at 11:13 AM, Jon Robson <jdlrobson@gmail.com> wrote:
> I'm looking forward to seeing how this plays out. The only downside I
> can so far see is that the amount of browser storage available varies
> drastically [1] and I wonder whether this will cause upsets for those
> browsers with extremely strict limits.

Or, for that matter, if it will fill up the allowed storage so user
scripts and gadgets can't make effective use of it.


--
Brad Jorsch (Anomie)
Software Engineer
Wikimedia Foundation

_______________________________________________
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: Module storage is coming [ In reply to ]
On 04/11/13 15:31, Mark A. Hershberger wrote:
> On 11/04/2013 10:10 AM, Max Semenik wrote:
>> Most of our mobile traffic is genrated by iOS and Android - both of
>> which support LocalStorage, so it will work for them too.
> People using 2G/GPRS are not necessarily using a mobile device. I've
> heard from at least one user (hence
> https://bugzilla.wikimedia.org/55842) that he is using his cell phone as
> a modem.
>
> I suspect this is not a small percentage of users in less developed
> areas where cell phone towers are plentiful but cables are not.
>
> Non-mobile UAs on mobile IPs is what I'm asking about.
>
> Mark.
>
>
> _______________________________________________
> Wikitech-l mailing list
> Wikitech-l@lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Er, that's a fairly standard speed for low-cost wired plans as well in
much of the US. I'd actually be happy to have something that fast, but I
couldn't afford it.

Well, it's the US's version of 'low cost', anyway.

_______________________________________________
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: Module storage is coming [ In reply to ]
I found a usability problem with some versions of the webkit/chrome
inspector for localStorage due to this, it is being tracked here:

https://bugs.webkit.org/show_bug.cgi?id=123750


On Mon, Nov 4, 2013 at 7:20 PM, Isarra Yos <zhorishna@gmail.com> wrote:

> On 04/11/13 15:31, Mark A. Hershberger wrote:
>
>> On 11/04/2013 10:10 AM, Max Semenik wrote:
>>
>>> Most of our mobile traffic is genrated by iOS and Android - both of
>>> which support LocalStorage, so it will work for them too.
>>>
>> People using 2G/GPRS are not necessarily using a mobile device. I've
>> heard from at least one user (hence
>> https://bugzilla.wikimedia.org/55842) that he is using his cell phone as
>> a modem.
>>
>> I suspect this is not a small percentage of users in less developed
>> areas where cell phone towers are plentiful but cables are not.
>>
>> Non-mobile UAs on mobile IPs is what I'm asking about.
>>
>> Mark.
>>
>>
>> _______________________________________________
>> Wikitech-l mailing list
>> Wikitech-l@lists.wikimedia.org
>> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
>>
>
> Er, that's a fairly standard speed for low-cost wired plans as well in
> much of the US. I'd actually be happy to have something that fast, but I
> couldn't afford it.
>
> Well, it's the US's version of 'low cost', anyway.
>
>
> _______________________________________________
> Wikitech-l mailing list
> Wikitech-l@lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
>
_______________________________________________
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: Module storage is coming [ In reply to ]
On Mon, Nov 4, 2013 at 4:50 AM, Mark A. Hershberger <mah@nichework.com> wrote:
> Hrm... I should probably go ask them about that. But I'm curious about
> your perspective and to see if we have any information on the bandwidth
> available to various users.

There is some, but not enough for us to know in advance what sort of
impact this will have. But Aaron Hafaker and I are working on it, and
we are going to be rigorous about measuring it, and we will report
back.

On Mon, Nov 4, 2013 at 9:03 AM, Brad Jorsch (Anomie)
<bjorsch@wikimedia.org> wrote:
> On Mon, Nov 4, 2013 at 11:13 AM, Jon Robson <jdlrobson@gmail.com> wrote:
>> I'm looking forward to seeing how this plays out. The only downside I
>> can so far see is that the amount of browser storage available varies
>> drastically [1] and I wonder whether this will cause upsets for those
>> browsers with extremely strict limits.
>
> Or, for that matter, if it will fill up the allowed storage so user
> scripts and gadgets can't make effective use of it.

You can find out the current byte size of the module store by
evaluating "mw.loader.inspect('store')" in a JavaScript console on the
wikis on which it is enabled (test, test2, beta cluster, and twn).
testwiki's JavaScript payload is usually a superset of the modules
deployed on various wikis, and the size of a fully warm module cache
is 710 kB. This should leave plenty of room on all but the most
restrictive platforms. However, if it does end up soaking up so much
space that it compromises the functionality of other scripts, I think
we could simply modify the implementation to make it honor some soft
limit, possibly one that is determined in reference to the user agent.

On Tue, Nov 5, 2013 at 1:00 AM, Derk-Jan Hartman
<d.j.hartman+wmf_ml@gmail.com> wrote:
> I found a usability problem with some versions of the webkit/chrome
> inspector for localStorage due to this, it is being tracked here:
>
> https://bugs.webkit.org/show_bug.cgi?id=123750

Thank you for doing that.

---
Ori Livneh

_______________________________________________
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: Module storage is coming [ In reply to ]
On Sun, Nov 3, 2013 at 5:27 PM, Ori Livneh <ori@wikimedia.org> wrote:
> How can I test it?
> ------------------
> Glad you asked! Module storage is enabled by default on the beta
> cluster, and on test & test2 wikis.

It's also enabled on MediaWiki.org now, the last such wiki before
doing performance testing. I figured it'd be OK because it has been
running on beta / test / test2 for over a week with no bugs being
reported. Please report back if you notice anything good or bad.

_______________________________________________
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: Module storage is coming [ In reply to ]
Can you explain why you use LocalStorage for this? It seems to me like
this is the wrong solution and you should use cache manifests instead.
LocalStorage is a quite limited area for _data_ storage and it will
create problems if we start wasting that space for _code_ storage.

John

On Mon, Nov 4, 2013 at 2:27 AM, Ori Livneh <ori@wikimedia.org> wrote:
> The roll-out of 1.23wmf2 to your favorite Wikimedia wiki will
> inaugurate the era of ResourceLoader module storage -- an era which
> will be marked by terrifying and hilarious new bugs, intermittent
> client-side failures, and generally inexcusable disruptions to user
> experience. Sounds exciting? Read on!
>
> What is it?
> -----------
> "Module storage" refers to the use of localStorage in ResourceLoader
> as an auxiliary to the browser cache. ResourceLoader, remember, is the
> MediaWiki subsystem that delivers JavaScript and CSS to your browser.
> One of its primary functions is to minimize the total number of
> network requests that your browser needs to make in order to render
> the page, and to make the remaining requests as slim as possible. One
> of the ways ResourceLoader does this is by making a list of all the
> different components your browser needs and then transmitting them in
> bulk.
>
> The downside of this approach is that a small update to MediaWiki's
> code, touching perhaps one or two components, will often force the
> browser to throw out (and re-retrieve) a bunch of unrelated modules
> that did not change and did not ostensibly need to be re-retrieved.
> This is because the browser cache operates on the level of network
> requests; it is not intelligent enough to decompose a request into its
> constituitive parts and to cache these parts separately from one
> another.
>
> Starting with the 1.23wmf2 branch, ResourceLoader will check if your
> browser implements localStorage, a mechanism for storing structure
> data. If it does, ResourceLoader will use it as an auxiliary cache,
> capable of versioning individual components.
>
> Why?
> ----
> The primary goals are:
>
> * Destabalize MediaWiki's front-end code, by coupling it to an
> inconsistently-implemented and poorly-understood HTML5 API.
> * Make debugging fun again, by adding another layer of caching to
> MediaWiki. Yes!!
>
> However, as with all new features, unintended side-effects are
> possible. Specific concerns for module storage include the risk of
> making the network footprint of page loads smaller, resulting in
> improved latency.
>
> But seriously,
> --------------
> * Module storage is enabled only if the underlying browser supports
> localStorage (~89% of browsers in use, according to
> <http://caniuse.com/#feat=namevalue-storage>). Users with older
> browsers will not see a benefit, but they will not suffer a penalty
> either.
> * Module storage may or may not improve site performance. We need to
> test it against a wide range of browsers and platforms to know for
> certain. If it doesn't improve performance, we'll blame it on you,
> your poor choice of browsers, and your uncooperative attitude, but
> then we'll scrap it.
>
> How can I test it?
> ------------------
> Glad you asked! Module storage is enabled by default on the beta
> cluster, and on test & test2 wikis. The simplest way you can help is
> by being alert to to performance regressions and front-end code
> breakage. If you know how to use your browser's JavaScript console,
> you can get stats about module storage usage on the current page by
> checking the value of 'mw.loader.store.stats'.
>
> When the code is rolled out across the cluster, it will be disabled by
> default, but you will be able to opt-in by manually setting a cookie
> in your browser. I will follow up with the exact details. If module
> storage proves stable enough for production use, we'll seek to asses
> its utility by running a controlled study of some kind. If we can
> demonstrate that module storage leads to a significant improvement in
> performance, we'll enable by it default.
>
> ---
> Ori Livneh
> ori@wikimedia.org
>
> _______________________________________________
> Wikitech-l mailing list
> Wikitech-l@lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l

_______________________________________________
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: Module storage is coming [ In reply to ]
Cache manifests are extremely inflexible. The HTTP caching we already
have is more flexible than cache manifests. So cache manifests won't
help make any improvements.

~Daniel Friesen (Dantman, Nadir-Seen-Fire) [http://danielfriesen.name/]

On 2013-11-07 12:19 AM, John Erling Blad wrote:
> Can you explain why you use LocalStorage for this? It seems to me like
> this is the wrong solution and you should use cache manifests instead.
> LocalStorage is a quite limited area for _data_ storage and it will
> create problems if we start wasting that space for _code_ storage.
>
> John

_______________________________________________
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: Module storage is coming [ In reply to ]
From personal experience don't touch cache manifests with a barge pole...

Bear in mind the majority of browsers provide at least 5mb of local storage
and we are talking about caching a few kB at most of minified JavaScript....
On 7 Nov 2013 00:35, "Daniel Friesen" <daniel@nadir-seen-fire.com> wrote:

> Cache manifests are extremely inflexible. The HTTP caching we already
> have is more flexible than cache manifests. So cache manifests won't
> help make any improvements.
>
> ~Daniel Friesen (Dantman, Nadir-Seen-Fire) [http://danielfriesen.name/]
>
> On 2013-11-07 12:19 AM, John Erling Blad wrote:
> > Can you explain why you use LocalStorage for this? It seems to me like
> > this is the wrong solution and you should use cache manifests instead.
> > LocalStorage is a quite limited area for _data_ storage and it will
> > create problems if we start wasting that space for _code_ storage.
> >
> > John
>
> _______________________________________________
> Wikitech-l mailing list
> Wikitech-l@lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
_______________________________________________
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: Module storage is coming [ In reply to ]
What should I be aware of as a developer? That is, if I'm running a local
MW and hacking on a resources (an extension / JavaScript / etc) will things
Just Work? Or should I take active steps to disable Module Storage so that
I don't inadvertently run the 'old' version of something I'm hacking on?

From what I understand, it's currently disabled by default, etc, so I don't
have to worry yet. But I guess I'm asking in advance to ensure that it's
well documented by the time we actually turn this on by default and
developers have to think about it.
--scott


On Sun, Nov 3, 2013 at 8:27 PM, Ori Livneh <ori@wikimedia.org> wrote:

> The roll-out of 1.23wmf2 to your favorite Wikimedia wiki will
> inaugurate the era of ResourceLoader module storage -- an era which
> will be marked by terrifying and hilarious new bugs, intermittent
> client-side failures, and generally inexcusable disruptions to user
> experience. Sounds exciting? Read on!
>
> What is it?
> -----------
> "Module storage" refers to the use of localStorage in ResourceLoader
> as an auxiliary to the browser cache. ResourceLoader, remember, is the
> MediaWiki subsystem that delivers JavaScript and CSS to your browser.
> One of its primary functions is to minimize the total number of
> network requests that your browser needs to make in order to render
> the page, and to make the remaining requests as slim as possible. One
> of the ways ResourceLoader does this is by making a list of all the
> different components your browser needs and then transmitting them in
> bulk.
>
> The downside of this approach is that a small update to MediaWiki's
> code, touching perhaps one or two components, will often force the
> browser to throw out (and re-retrieve) a bunch of unrelated modules
> that did not change and did not ostensibly need to be re-retrieved.
> This is because the browser cache operates on the level of network
> requests; it is not intelligent enough to decompose a request into its
> constituitive parts and to cache these parts separately from one
> another.
>
> Starting with the 1.23wmf2 branch, ResourceLoader will check if your
> browser implements localStorage, a mechanism for storing structure
> data. If it does, ResourceLoader will use it as an auxiliary cache,
> capable of versioning individual components.
>
> Why?
> ----
> The primary goals are:
>
> * Destabalize MediaWiki's front-end code, by coupling it to an
> inconsistently-implemented and poorly-understood HTML5 API.
> * Make debugging fun again, by adding another layer of caching to
> MediaWiki. Yes!!
>
> However, as with all new features, unintended side-effects are
> possible. Specific concerns for module storage include the risk of
> making the network footprint of page loads smaller, resulting in
> improved latency.
>
> But seriously,
> --------------
> * Module storage is enabled only if the underlying browser supports
> localStorage (~89% of browsers in use, according to
> <http://caniuse.com/#feat=namevalue-storage>). Users with older
> browsers will not see a benefit, but they will not suffer a penalty
> either.
> * Module storage may or may not improve site performance. We need to
> test it against a wide range of browsers and platforms to know for
> certain. If it doesn't improve performance, we'll blame it on you,
> your poor choice of browsers, and your uncooperative attitude, but
> then we'll scrap it.
>
> How can I test it?
> ------------------
> Glad you asked! Module storage is enabled by default on the beta
> cluster, and on test & test2 wikis. The simplest way you can help is
> by being alert to to performance regressions and front-end code
> breakage. If you know how to use your browser's JavaScript console,
> you can get stats about module storage usage on the current page by
> checking the value of 'mw.loader.store.stats'.
>
> When the code is rolled out across the cluster, it will be disabled by
> default, but you will be able to opt-in by manually setting a cookie
> in your browser. I will follow up with the exact details. If module
> storage proves stable enough for production use, we'll seek to asses
> its utility by running a controlled study of some kind. If we can
> demonstrate that module storage leads to a significant improvement in
> performance, we'll enable by it default.
>
> ---
> Ori Livneh
> ori@wikimedia.org
>
> _______________________________________________
> Wikitech-l mailing list
> Wikitech-l@lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l




--
(http://cscott.net)
_______________________________________________
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: Module storage is coming [ In reply to ]
On Thu, Nov 7, 2013 at 8:05 AM, Jon Robson <jdlrobson@gmail.com> wrote:

> From personal experience don't touch cache manifests with a barge pole...
>
> Bear in mind the majority of browsers provide at least 5mb of local storage
> and we are talking about caching a few kB at most of minified
> JavaScript....
> On 7 Nov 2013 00:35, "Daniel Friesen" <daniel@nadir-seen-fire.com> wrote:
>

What I'm seeing in Chromium/Chrome is that when I invoke VisualEditor one
time localStorage hits ~3.5MB immediately.

Hit shift-reload a few times in quick succession and it's pretty easy to
max out localStorage for Chromium at 5.xMB. Default for Chrome seems to be
10MB.
_______________________________________________
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: Module storage is coming [ In reply to ]
I'm not sure why shift reloading would make the cache grow since the
same page should load exactly the same modules - if that's not the
case that points at a bug somewhere.

That said since there are a potentially infinite amount of
modules/gadgets, many of which are rarely used that can be loaded in
localStorage it could be a good idea to control this somehow - e.g.
only cache commonly loaded modules . Ori have you thought much about
this approach? I could imagine two approaches:

1) only caching modules loaded by startup module
2) keeping a counter for each module name to count how many times it
loads and when it hits a certain threshold only then the css/js is
cached. For instance there is not much point in caching a bit of JS
that runs on
3) a mixture of these 2 approaches

I was just reminded also of an old talk I went to [1] which had
awesome ideas for maximising storage that might be worth exploring.
Since JavaScript uses UTF-16 for text encoding the idea was
essentially to convert to UTF-8/ASCII to get up to 50% compression
ratio. There were also ideas about bit shifting base64. Worth
exploring and making this even more of a dangerous experiment?!?! :D

[1] http://lanyrd.com/2012/londonjs-10/srdzy/

On Thu, Nov 7, 2013 at 8:38 AM, Chris McMahon <cmcmahon@wikimedia.org> wrote:
> On Thu, Nov 7, 2013 at 8:05 AM, Jon Robson <jdlrobson@gmail.com> wrote:
>
>> From personal experience don't touch cache manifests with a barge pole...
>>
>> Bear in mind the majority of browsers provide at least 5mb of local storage
>> and we are talking about caching a few kB at most of minified
>> JavaScript....
>> On 7 Nov 2013 00:35, "Daniel Friesen" <daniel@nadir-seen-fire.com> wrote:
>>
>
> What I'm seeing in Chromium/Chrome is that when I invoke VisualEditor one
> time localStorage hits ~3.5MB immediately.
>
> Hit shift-reload a few times in quick succession and it's pretty easy to
> max out localStorage for Chromium at 5.xMB. Default for Chrome seems to be
> 10MB.
> _______________________________________________
> Wikitech-l mailing list
> Wikitech-l@lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l



--
Jon Robson
http://jonrobson.me.uk
@rakugojon

_______________________________________________
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: Module storage is coming [ In reply to ]
On Thu, Nov 7, 2013 at 12:22 PM, Jon Robson <jdlrobson@gmail.com> wrote:

> I'm not sure why shift reloading would make the cache grow since the
> same page should load exactly the same modules - if that's not the
> case that points at a bug somewhere.
>

Seems like a bug, to me.


> That said since there are a potentially infinite amount of
> modules/gadgets, many of which are rarely used that can be loaded in
> localStorage it could be a good idea to control this somehow - e.g.
> only cache commonly loaded modules . Ori have you thought much about
> this approach? I could imagine two approaches:
>
> 1) only caching modules loaded by startup module
> 2) keeping a counter for each module name to count how many times it
> loads and when it hits a certain threshold only then the css/js is
> cached. For instance there is not much point in caching a bit of JS
> that runs on
> 3) a mixture of these 2 approaches
>

It seems like it would also be useful for gadgets/extensions to be able to
interact with the resource loader, to effectively say, "reserve X bytes for
me". Maybe this is just having a programmatic way for gadgets/extensions
to throw out the LRU bytes in the resource loader cache if they try to save
something to local storage and fail. Alternatively: a priority scheme, so
gadgets can automatically get prioritized access to the limited amount of
localStorage.


> I was just reminded also of an old talk I went to [1] which had
> awesome ideas for maximising storage that might be worth exploring.
> Since JavaScript uses UTF-16 for text encoding the idea was
> essentially to convert to UTF-8/ASCII to get up to 50% compression
> ratio. There were also ideas about bit shifting base64. Worth
> exploring and making this even more of a dangerous experiment?!?! :D
>

I've played around with some of these ideas in the past. I'm not sure it's
worth it, in the end. The spec doesn't actually say how the localstorage
accounting works in the browser, so even though strings are "nominally"
UTF-16, it's not clear that browsers are using "size in UTF-16 encoding" as
their internal quota metric. More likely they are using size in whatever
internal representation they have. As far as the spec is concerned, the
browser can compress the stored data itself to maximize space available to
user scripts... or just increase the quota allowance. I've had some
conversations about this with the mobile firefox guys.

If you wanted to play around with this, though,
https://github.com/cscott/nell-colors/blob/master/src/lzw.js contains an
implementation of lzw compression optimized for storage in UTF-16 strings.

If you really want to maximize storage, you'd be better off using the
IndexedDB API, which allows you to directly store byte arrays. Then you
could use all the fancy compression algorithms in
https://github.com/cscott/compressjs to really squeeze things down. (Or
just use the byte arrays to ensure that the script is stored in UTF8.)

It's a tradeoff, though. A bandwidth-limited user might tolerate the
slowdown of compression, but if you're trying to use the cache to speed
things up for a high-bandwidth desktop user it's hard to make compression
pay for itself.
--scott





>
> [1] http://lanyrd.com/2012/londonjs-10/srdzy/
>
> On Thu, Nov 7, 2013 at 8:38 AM, Chris McMahon <cmcmahon@wikimedia.org>
> wrote:
> > On Thu, Nov 7, 2013 at 8:05 AM, Jon Robson <jdlrobson@gmail.com> wrote:
> >
> >> From personal experience don't touch cache manifests with a barge
> pole...
> >>
> >> Bear in mind the majority of browsers provide at least 5mb of local
> storage
> >> and we are talking about caching a few kB at most of minified
> >> JavaScript....
> >> On 7 Nov 2013 00:35, "Daniel Friesen" <daniel@nadir-seen-fire.com>
> wrote:
> >>
> >
> > What I'm seeing in Chromium/Chrome is that when I invoke VisualEditor one
> > time localStorage hits ~3.5MB immediately.
> >
> > Hit shift-reload a few times in quick succession and it's pretty easy to
> > max out localStorage for Chromium at 5.xMB. Default for Chrome seems to
> be
> > 10MB.
> > _______________________________________________
> > Wikitech-l mailing list
> > Wikitech-l@lists.wikimedia.org
> > https://lists.wikimedia.org/mailman/listinfo/wikitech-l
>
>
>
> --
> Jon Robson
> http://jonrobson.me.uk
> @rakugojon
>
> _______________________________________________
> Wikitech-l mailing list
> Wikitech-l@lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
>



--
(http://cscott.net)
_______________________________________________
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: Module storage is coming [ In reply to ]
We may want to make some considerations for multiple wiki on the same
origin though.

For example a wiki that uses paths for multiple languages:
wiki.example.com/en/Foo
wiki.example.com/fr/Foo
wiki.example.com/de/Foo
...

On a setup like this all the wikis will share the same localStorage
origin and a number of them will fill up the localStorage with cache
entries for each wiki if a user visits them all.
Most of these cache entries will likely be duplicates of the contents of
other wiki.
So we may want to consider a setup where the current localStorage cache
stores hashes and then stores the actual module data inside a different
localStorage key which is shared among wiki.

~Daniel Friesen (Dantman, Nadir-Seen-Fire) [http://danielfriesen.name/]

On 2013-11-07 7:05 AM, Jon Robson wrote:
> From personal experience don't touch cache manifests with a barge pole...
>
> Bear in mind the majority of browsers provide at least 5mb of local storage
> and we are talking about caching a few kB at most of minified JavaScript....

_______________________________________________
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: Module storage is coming [ In reply to ]
That is a statement, not an explanation.
John

On Thu, Nov 7, 2013 at 4:05 PM, Jon Robson <jdlrobson@gmail.com> wrote:
> From personal experience don't touch cache manifests with a barge pole...
>
> Bear in mind the majority of browsers provide at least 5mb of local storage
> and we are talking about caching a few kB at most of minified JavaScript....
> On 7 Nov 2013 00:35, "Daniel Friesen" <daniel@nadir-seen-fire.com> wrote:
>
>> Cache manifests are extremely inflexible. The HTTP caching we already
>> have is more flexible than cache manifests. So cache manifests won't
>> help make any improvements.
>>
>> ~Daniel Friesen (Dantman, Nadir-Seen-Fire) [http://danielfriesen.name/]
>>
>> On 2013-11-07 12:19 AM, John Erling Blad wrote:
>> > Can you explain why you use LocalStorage for this? It seems to me like
>> > this is the wrong solution and you should use cache manifests instead.
>> > LocalStorage is a quite limited area for _data_ storage and it will
>> > create problems if we start wasting that space for _code_ storage.
>> >
>> > John
>>
>> _______________________________________________
>> Wikitech-l mailing list
>> Wikitech-l@lists.wikimedia.org
>> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
> _______________________________________________
> Wikitech-l mailing list
> Wikitech-l@lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l

_______________________________________________
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: Module storage is coming [ In reply to ]
That is a statement, not an explanation.
Please provide a valid explanation why you want to do this.
John

On Thu, Nov 7, 2013 at 9:35 AM, Daniel Friesen
<daniel@nadir-seen-fire.com> wrote:
> Cache manifests are extremely inflexible. The HTTP caching we already
> have is more flexible than cache manifests. So cache manifests won't
> help make any improvements.
>
> ~Daniel Friesen (Dantman, Nadir-Seen-Fire) [http://danielfriesen.name/]
>
> On 2013-11-07 12:19 AM, John Erling Blad wrote:
>> Can you explain why you use LocalStorage for this? It seems to me like
>> this is the wrong solution and you should use cache manifests instead.
>> LocalStorage is a quite limited area for _data_ storage and it will
>> create problems if we start wasting that space for _code_ storage.
>>
>> John
>
> _______________________________________________
> Wikitech-l mailing list
> Wikitech-l@lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l

_______________________________________________
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: Module storage is coming [ In reply to ]
I'm not interested in writing an entire explanation for you on how cache
manifests work and their faults when you could simply do a web search
for one of the many existing tutorials.
The issues with using cache manifests to try and do this should be
perfectly understandable once someone understands how cache manifests work.
They are unusable for this technique.
And Ori has already explained the advantage of being able to have
modules cached separately.

You're basically suggesting we try to make orange juice out of apples
instead of oranges.

~Daniel Friesen (Dantman, Nadir-Seen-Fire) [http://danielfriesen.name/]

On 2013-11-08 12:46 AM, John Erling Blad wrote:
> That is a statement, not an explanation.
> Please provide a valid explanation why you want to do this.
> John
>
> On Thu, Nov 7, 2013 at 9:35 AM, Daniel Friesen
> <daniel@nadir-seen-fire.com> wrote:
>> Cache manifests are extremely inflexible. The HTTP caching we already
>> have is more flexible than cache manifests. So cache manifests won't
>> help make any improvements.
>>
>> ~Daniel Friesen (Dantman, Nadir-Seen-Fire) [http://danielfriesen.name/]
>>
>> On 2013-11-07 12:19 AM, John Erling Blad wrote:
>>> Can you explain why you use LocalStorage for this? It seems to me like
>>> this is the wrong solution and you should use cache manifests instead.
>>> LocalStorage is a quite limited area for _data_ storage and it will
>>> create problems if we start wasting that space for _code_ storage.
>>>
>>> John
>> _______________________________________________
>> Wikitech-l mailing list
>> Wikitech-l@lists.wikimedia.org
>> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
> _______________________________________________
> Wikitech-l mailing list
> Wikitech-l@lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l


_______________________________________________
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: Module storage is coming [ In reply to ]
Le 08/11/13 10:33, Daniel Friesen a écrit :
> I'm not interested in writing an entire explanation for you on how cache
> manifests work and their faults when you could simply do a web search
> for one of the many existing tutorials.

If you could explain to the newbie there like me what a cache manifest
here, that would help the discussion. Not everyone has the time to
attempt to figure out every technical matters, much less figuring it wrong.

So what is a cache manifest? :D


--
Antoine "hashar" Musso


_______________________________________________
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

1 2  View All