Mailing List Archive

scaled media (thumbs) as *temporary* files, not stored forever
So it's time to have this discussion again. At least, I think we're
having it again, though I could not find previous threads on this list
about the subject.

In short, scaled media is currently generated on the fly for any size
and for any user. The resulting files are kept around forever or until
we run perilously short of space, at which point we make some guesses
about what we can toss and then do a mass purge. Last time we did so, we
had the rotation bug going at the same time, which made for a real fine
mess.

A little bit of crunching shows me that we have about 6 million images
in use on the projects, and yet we manage to have around 130 million
thumbnails. Just for fun I checked to see how many thumbs each image
has, what sizes we are looking at, etc. Here's the results.

Some "standard" sizes are most popular, with between 200K and 640K media
files having thumbs scaled to each of these widths:
75, 120, 150, 180, 200, 220, 320, 640, 800, 1024, and 1280 pixels

But there's plenty of "odd" sizes with lots of thumbs too. For example,
over 65K files with width 181px, 20K with width 138px.

As an experiment and before having this data, I purged from ms5 (no
longer in use for thumbs) 1/16 of the thumbs that were greater than
100px wide but not one of these widths:
120px, 200px, 220px, 250px, 320px, 640px, 800px
We got back over 300GB of space.

The other thing about delivering any scaled version on demand is that we
have some media files with several hundred different thumb sizes in
there. Here's a few of the top offenders for your entertainment:

2514 wikipedia/commons/thumb/f/f9/Orange_and_cross_section.jpg
2285 wikipedia/commons/thumb/f/fb/Thrermal_grease.jpg
2218 wikipedia/commons/thumb/f/fc/Blue_sport.jpg
2071 wikipedia/commons/thumb/f/f3/Flag_of_Switzerland.svg
2062 wikipedia/commons/thumb/f/f2/Flag_of_Costa_Rica.svg
2034 wikipedia/commons/thumb/f/f8/Wiktionary-logo-en.svg
1915 wikipedia/commons/thumb/f/f6/VeulesLesRoses.JPG
1689 wikipedia/commons/thumb/f/fa/Wikibooks-logo.svg
1447 wikipedia/commons/thumb/f/fa/Wikiquote-logo.svg
1371 wikipedia/commons/thumb/f/f0/Mori_Uncanny_Valley.svg
1249 wikipedia/commons/thumb/f/f5/Grand_prismatic_spring.jpg
1246 wikipedia/commons/thumb/f/f3/Mature.jpg
1191 wikipedia/commons/thumb/f/f7/Kirchdorf_in_Tirol.JPG
1187 wikipedia/commons/thumb/f/f8/Camille_Cabral_pour_les_Trans.JPG
1143 wikipedia/commons/thumb/f/f7/Profanity.svg
1079 wikipedia/commons/thumb/f/f2/HSV_color_solid_cone.png
1040 wikipedia/commons/thumb/f/f2/Carmen_Electra.jpg
1032 wikipedia/commons/thumb/f/f1/Pink_eye.jpg
1001 wikipedia/commons/thumb/f/f6/USNS_Medgar_Evers_announcement.jpg

I'd comment on some of those but I'd be too snarky.

So there are some things we could change:

1. We could generate and keep only certain sizes, tossing the rest.
2. We could keep *nothing*, scaling all media as required.
3. We could have a cron job that was clever about tossing thumbs every
day (not sure how easy it would be to be clever).
4. ??

In any of these cases, the squids will have copies of recently requested
scaled media, so we won't be scaling the same file to the same size over
and over in a short time frame.

What do folks think about how to proceed?

Ariel



_______________________________________________
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: scaled media (thumbs) as *temporary* files, not stored forever [ In reply to ]
On 31 August 2012 13:36, Ariel T. Glenn <ariel@wikimedia.org> wrote:

> 1. We could generate and keep only certain sizes, tossing the rest.
> 2. We could keep *nothing*, scaling all media as required.
> 3. We could have a cron job that was clever about tossing thumbs every
> day (not sure how easy it would be to be clever).
> 4. ??
> In any of these cases, the squids will have copies of recently requested
> scaled media, so we won't be scaling the same file to the same size over
> and over in a short time frame.


To be obvious for #3:

* Do we know access times for these files? Can stuff be purged that
hasn't been accessed in x time? What values of x would be good?
* More generally: what's the tradeofff between generating a thumbnail
afresh and keeping an old copy around until it's needed? Just how
CPU-stressed is the thumbnailer?


- d.

_______________________________________________
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: scaled media (thumbs) as *temporary* files, not stored forever [ In reply to ]
On Fri, Aug 31, 2012 at 7:36 AM, Ariel T. Glenn <ariel@wikimedia.org> wrote:

> So there are some things we could change:
>
> 1. We could generate and keep only certain sizes, tossing the rest.
>

Heck yes. Generate some standard sizes at upload time and let the browser
scale if a funny size is demanded. Modern browsers scale photos nicely, not
like the nearest-neighbor ugliness from 2002.

This'll simplify our thumbnail-serving architecture, remove some DoS
vectors, and if we pick the next size up makes things look better when
zooming or on high-density screens.

Downside: diagrams and charts done as PNGs or JPGs might not look as sharp
at non-standard sizes.

We should also start considering serving SVGs directly to supporting
browsers, so they always look nice at any size -- and at any zoom level.
(Downside of this: this means we have to start thinking about size and
rendering efficiency in SVGs -- don't use a 6 megabyte super-detailed map
for something that's going to be shown at 200px most of the time!)

-- brion
_______________________________________________
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: scaled media (thumbs) as *temporary* files, not stored forever [ In reply to ]
If we make a cron job, could we also have it purge all SVG thumbnails older than say 5 years?

Ryan Kaldari

On Aug 31, 2012, at 5:36 AM, "Ariel T. Glenn" <ariel@wikimedia.org> wrote:

> So it's time to have this discussion again. At least, I think we're
> having it again, though I could not find previous threads on this list
> about the subject.
>
> In short, scaled media is currently generated on the fly for any size
> and for any user. The resulting files are kept around forever or until
> we run perilously short of space, at which point we make some guesses
> about what we can toss and then do a mass purge. Last time we did so, we
> had the rotation bug going at the same time, which made for a real fine
> mess.
>
> A little bit of crunching shows me that we have about 6 million images
> in use on the projects, and yet we manage to have around 130 million
> thumbnails. Just for fun I checked to see how many thumbs each image
> has, what sizes we are looking at, etc. Here's the results.
>
> Some "standard" sizes are most popular, with between 200K and 640K media
> files having thumbs scaled to each of these widths:
> 75, 120, 150, 180, 200, 220, 320, 640, 800, 1024, and 1280 pixels
>
> But there's plenty of "odd" sizes with lots of thumbs too. For example,
> over 65K files with width 181px, 20K with width 138px.
>
> As an experiment and before having this data, I purged from ms5 (no
> longer in use for thumbs) 1/16 of the thumbs that were greater than
> 100px wide but not one of these widths:
> 120px, 200px, 220px, 250px, 320px, 640px, 800px
> We got back over 300GB of space.
>
> The other thing about delivering any scaled version on demand is that we
> have some media files with several hundred different thumb sizes in
> there. Here's a few of the top offenders for your entertainment:
>
> 2514 wikipedia/commons/thumb/f/f9/Orange_and_cross_section.jpg
> 2285 wikipedia/commons/thumb/f/fb/Thrermal_grease.jpg
> 2218 wikipedia/commons/thumb/f/fc/Blue_sport.jpg
> 2071 wikipedia/commons/thumb/f/f3/Flag_of_Switzerland.svg
> 2062 wikipedia/commons/thumb/f/f2/Flag_of_Costa_Rica.svg
> 2034 wikipedia/commons/thumb/f/f8/Wiktionary-logo-en.svg
> 1915 wikipedia/commons/thumb/f/f6/VeulesLesRoses.JPG
> 1689 wikipedia/commons/thumb/f/fa/Wikibooks-logo.svg
> 1447 wikipedia/commons/thumb/f/fa/Wikiquote-logo.svg
> 1371 wikipedia/commons/thumb/f/f0/Mori_Uncanny_Valley.svg
> 1249 wikipedia/commons/thumb/f/f5/Grand_prismatic_spring.jpg
> 1246 wikipedia/commons/thumb/f/f3/Mature.jpg
> 1191 wikipedia/commons/thumb/f/f7/Kirchdorf_in_Tirol.JPG
> 1187 wikipedia/commons/thumb/f/f8/Camille_Cabral_pour_les_Trans.JPG
> 1143 wikipedia/commons/thumb/f/f7/Profanity.svg
> 1079 wikipedia/commons/thumb/f/f2/HSV_color_solid_cone.png
> 1040 wikipedia/commons/thumb/f/f2/Carmen_Electra.jpg
> 1032 wikipedia/commons/thumb/f/f1/Pink_eye.jpg
> 1001 wikipedia/commons/thumb/f/f6/USNS_Medgar_Evers_announcement.jpg
>
> I'd comment on some of those but I'd be too snarky.
>
> So there are some things we could change:
>
> 1. We could generate and keep only certain sizes, tossing the rest.
> 2. We could keep *nothing*, scaling all media as required.
> 3. We could have a cron job that was clever about tossing thumbs every
> day (not sure how easy it would be to be clever).
> 4. ??
>
> In any of these cases, the squids will have copies of recently requested
> scaled media, so we won't be scaling the same file to the same size over
> and over in a short time frame.
>
> What do folks think about how to proceed?
>
> Ariel
>
>
>
> _______________________________________________
> Wikitech-l mailing list
> Wikitech-l@lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l

_______________________________________________
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: scaled media (thumbs) as *temporary* files, not stored forever [ In reply to ]
> >
> > 1. We could generate and keep only certain sizes, tossing the rest.
> >
>
> Heck yes. Generate some standard sizes at upload time and let the browser
> scale if a funny size is demanded. Modern browsers scale photos nicely,
not
> like the nearest-neighbor ugliness from 2002.

+1 I was very surprised to learn any thumbnail sizes could be generated. We
should standardise on a tiny, small,medium high and original resolutions. 5
sizes seems more than enough.
>
> This'll simplify our thumbnail-serving architecture, remove some D
> _______________________________________________
> Wikitech-l mailing list
> Wikitech-l@lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
_______________________________________________
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: scaled media (thumbs) as *temporary* files, not stored forever [ In reply to ]
On 31 August 2012 18:21, Jon Robson <jdlrobson@gmail.com> wrote:

> +1 I was very surprised to learn any thumbnail sizes could be generated. We
> should standardise on a tiny, small,medium high and original resolutions. 5
> sizes seems more than enough.


I would suggest checking against the corpus before saying "x should be
enough for anyone". There are also occasional but valid uses for
non-standard sizes.


- d.

_______________________________________________
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: scaled media (thumbs) as *temporary* files, not stored forever [ In reply to ]
On 31/08/2012 08:57, Brion Vibber wrote:
> Heck yes. Generate some standard sizes at upload time and let the
> browser scale if a funny size is demanded. Modern browsers scale
> photos nicely, not like the nearest-neighbor ugliness from 2002.

As a graphist, I must say this does not seem like a good idea. Only
rendering certain sizes and having the browser then scale the weird ones
will still result in fuzzy images, because no matter how good the
renderer, every time a bitmap image is scaled down, sharpness is lost.
This is part of why there is so much emphasis placed on using vectors
even in a static environment - with those, the first scale down is also
avoided, and there is a very visible difference in clarity even there.
But while only rendering certain sizes and then having the browser scale
those would defeat that purpose, having to scale down bitmaps twice
would look even worse, regardless of subject.

--
-— Isarra


_______________________________________________
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: scaled media (thumbs) as *temporary* files, not stored forever [ In reply to ]
>I would suggest checking against the corpus before saying "x should be
>enough for anyone". There are also occasional but valid uses for
>non-standard sizes.

Perhaps generate the standard thumbnail sizes at upload time and then generate
and cache tumbnails of non-standard times until they haven't been accessed for a
few days.

Doesn't take up much space, like just deleting all the thumbnails that are
generated, and it saves time down the road as the most common thumbnails are
already generated.

I wouldn't rely too much on the Squids. They do an excellent job, but thinking
of those of us who use Mediawiki outside the foundation, I would much rather see
an intelligent thumbnail generating and caching scheme that doesn't rely on
Squid being present.

Thank you,
Derric Atzrott


_______________________________________________
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: scaled media (thumbs) as *temporary* files, not stored forever [ In reply to ]
On Fri, 31 Aug 2012 05:36:18 -0700, Ariel T. Glenn <ariel@wikimedia.org>
wrote:

> So it's time to have this discussion again. At least, I think we're
> having it again, though I could not find previous threads on this list
> about the subject.
>
> In short, scaled media is currently generated on the fly for any size
> and for any user. The resulting files are kept around forever or until
> we run perilously short of space, at which point we make some guesses
> about what we can toss and then do a mass purge. Last time we did so, we
> had the rotation bug going at the same time, which made for a real fine
> mess.
>
> A little bit of crunching shows me that we have about 6 million images
> in use on the projects, and yet we manage to have around 130 million
> thumbnails. Just for fun I checked to see how many thumbs each image
> has, what sizes we are looking at, etc. Here's the results.
>
> Some "standard" sizes are most popular, with between 200K and 640K media
> files having thumbs scaled to each of these widths:
> 75, 120, 150, 180, 200, 220, 320, 640, 800, 1024, and 1280 pixels
>
> But there's plenty of "odd" sizes with lots of thumbs too. For example,
> over 65K files with width 181px, 20K with width 138px.
>
> As an experiment and before having this data, I purged from ms5 (no
> longer in use for thumbs) 1/16 of the thumbs that were greater than
> 100px wide but not one of these widths:
> 120px, 200px, 220px, 250px, 320px, 640px, 800px
> We got back over 300GB of space.
>
> The other thing about delivering any scaled version on demand is that we
> have some media files with several hundred different thumb sizes in
> there. Here's a few of the top offenders for your entertainment:
>
> 2514 wikipedia/commons/thumb/f/f9/Orange_and_cross_section.jpg
> 2285 wikipedia/commons/thumb/f/fb/Thrermal_grease.jpg
> 2218 wikipedia/commons/thumb/f/fc/Blue_sport.jpg
> 2071 wikipedia/commons/thumb/f/f3/Flag_of_Switzerland.svg
> 2062 wikipedia/commons/thumb/f/f2/Flag_of_Costa_Rica.svg
> 2034 wikipedia/commons/thumb/f/f8/Wiktionary-logo-en.svg
> 1915 wikipedia/commons/thumb/f/f6/VeulesLesRoses.JPG
> 1689 wikipedia/commons/thumb/f/fa/Wikibooks-logo.svg
> 1447 wikipedia/commons/thumb/f/fa/Wikiquote-logo.svg
> 1371 wikipedia/commons/thumb/f/f0/Mori_Uncanny_Valley.svg
> 1249 wikipedia/commons/thumb/f/f5/Grand_prismatic_spring.jpg
> 1246 wikipedia/commons/thumb/f/f3/Mature.jpg
> 1191 wikipedia/commons/thumb/f/f7/Kirchdorf_in_Tirol.JPG
> 1187 wikipedia/commons/thumb/f/f8/Camille_Cabral_pour_les_Trans.JPG
> 1143 wikipedia/commons/thumb/f/f7/Profanity.svg
> 1079 wikipedia/commons/thumb/f/f2/HSV_color_solid_cone.png
> 1040 wikipedia/commons/thumb/f/f2/Carmen_Electra.jpg
> 1032 wikipedia/commons/thumb/f/f1/Pink_eye.jpg
> 1001 wikipedia/commons/thumb/f/f6/USNS_Medgar_Evers_announcement.jpg
>
> I'd comment on some of those but I'd be too snarky.
>
> So there are some things we could change:
>
> 1. We could generate and keep only certain sizes, tossing the rest.
> 2. We could keep *nothing*, scaling all media as required.
> 3. We could have a cron job that was clever about tossing thumbs every
> day (not sure how easy it would be to be clever).
> 4. ??
>
> In any of these cases, the squids will have copies of recently requested
> scaled media, so we won't be scaling the same file to the same size over
> and over in a short time frame.
>
> What do folks think about how to proceed?
>
> Ariel

Another idea I've played with was development of a LRU filesystem.
Probably a FUSE module. You would mount it at thumbs/ and unused files
would periodically disappear.

--
~Daniel Friesen (Dantman, Nadir-Seen-Fire) [http://daniel.friesen.name]


_______________________________________________
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: scaled media (thumbs) as *temporary* files, not stored forever [ In reply to ]
> Another idea I've played with was development of a LRU filesystem. Probably
> a FUSE module. You would mount it at thumbs/ and unused files would
> periodically disappear.
>

We don't mount these filesystems anymore. We use an object store call
swift. FUSE also doesn't exist outside of Linux, right? So, this
likely wouldn't be terribly useful as a core feature.

- Ryan

_______________________________________________
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: scaled media (thumbs) as *temporary* files, not stored forever [ In reply to ]
How hard/easy is it to determine when a thumb file has last been
accessed (by the squids or any means for that matter)?

If easy, why not have some process delete the thumbs that have not been
accessed for (squid expiration time + 1 day)? That ensures thumbs live
for as long as needed (or until purged), without adding to the scaler's
workload. Basically let the thumb files expire as they do on the squids.

I imagine the first run would be a mammoth job, but subsequent runs
shouldn't be stressing at all.

--
Erwin Dokter


_______________________________________________
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: scaled media (thumbs) as *temporary* files, not stored forever [ In reply to ]
On Fri, Aug 31, 2012 at 1:48 PM, Derric Atzrott
<datzrott@alizeepathology.com> wrote:
> Perhaps generate the standard thumbnail sizes at upload time

I believe the status quo is no thumbs (of any size) are generated at
upload time. They are all just done on demand.

-Jeremy

_______________________________________________
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: scaled media (thumbs) as *temporary* files, not stored forever [ In reply to ]
On Fri, Aug 31, 2012 at 1:22 PM, Jeremy Baron <jeremy@tuxmachine.com> wrote:
> On Fri, Aug 31, 2012 at 1:48 PM, Derric Atzrott
> <datzrott@alizeepathology.com> wrote:
>> Perhaps generate the standard thumbnail sizes at upload time
>
> I believe the status quo is no thumbs (of any size) are generated at
> upload time. They are all just done on demand.
>
That's correct in theory. In practice (for uploads using
Special:Upload at least) the user is redirected to the file
description page after a successful upload, which contains a thumb of
the file of a given size (800px?), which is then immediately generated
on demand :) . Special:UploadWizard has similar behavior: the success
page contains 200px(?) thumbs of all the images the user uploaded.

Roan

_______________________________________________
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: scaled media (thumbs) as *temporary* files, not stored forever [ In reply to ]
Maybe we could have "large","medium","small" etc as aliases for
standard/popular sizes to encourage using less of the non-standard
ones?

--
Daniel Zahn <dzahn@wikimedia.org>
Operations Engineer

_______________________________________________
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: scaled media (thumbs) as *temporary* files, not stored forever [ In reply to ]
Also wondering if there are any thumbnails that are larger than their
actual images, and if yes to get rid of them.

_______________________________________________
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: scaled media (thumbs) as *temporary* files, not stored forever [ In reply to ]
On Fri, Aug 31, 2012 at 3:52 PM, Daniel Zahn <dzahn@wikimedia.org> wrote:

> Maybe we could have "large","medium","small" etc as aliases for
> standard/popular sizes to encourage using less of the non-standard
> ones?
>

I kinda like this. It would also be nice if simply including an image
defaulted to some sane size, even without using explicit "|thumb".

In fact, we should think about just redoing how images get included in the
first place maybe. :P

I'd kind of like to see something like this:

{{#media:Foobar.jpg}} <- default to a nice size, displayed in some
nice-looking way suitable to the output. Sane framing and positioning
typical for most usages.

{{#media:Foobar.jpg|caption=Hello this is my caption about [[stuff]].
Enjoy!}} <- caption should probably be an explicitly named parameter

* Consider having *no size option* at all. :)
* Definitely don't have "left" "right" or "center" options.
* Consider making it easy to collect multiple related photos together, like
<gallery>.

Or maybe we should just use <gallery> more aggressively and make it a
billion times prettier...


For the more icon-like uses, maybe an explicit inline-media function:

{{#inline-media:Foobar.svg|24x24px}}

Anyway.... this needs more thought. But for a lot of images, we don't
really *need* to be manually specifying every detail of their layout. It
feels like it would be nicer to say "stuff these photos, with these
captions, into this section of the article" and let the wiki deal with
laying them out.

Note that in mobile/tablet contexts it's also very handy to be able to
extract just the photos and provide them for separate browsing; this has
influenced my thinking on this for sure.

-- brion
_______________________________________________
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: scaled media (thumbs) as *temporary* files, not stored forever [ In reply to ]
On Fri, Aug 31, 2012 at 4:25 PM, Daniel Zahn <dzahn@wikimedia.org> wrote:

> Also wondering if there are any thumbnails that are larger than their
> actual images, and if yes to get rid of them.
>

For raster image formats, we don't generate thumbs larger than the original
-- we just use the original image and let the browser stretch it to the
requested size.

For vector image formats (SVG, possibly PDF) we can and do generate
thumbnails larger than the canonical width and height, so that tiny SVG
files can be scaled up. But a hard limit is enforced, I think 2048px or
something.

-- brion
_______________________________________________
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: scaled media (thumbs) as *temporary* files, not stored forever [ In reply to ]
On Fri, Aug 31, 2012 at 5:52 PM, Brion Vibber <brion@pobox.com> wrote:

> Note that in mobile/tablet contexts it's also very handy to be able to
> extract just the photos and provide them for separate browsing; this has
> influenced my thinking on this for sure.
>

^ in particular, distinguishing between "editorial" photo/diagram content
and icon-like uses of images would be a huge help. We don't want to stick
template icons in an image gallery of photos on an article.

-- brion
_______________________________________________
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: scaled media (thumbs) as *temporary* files, not stored forever [ In reply to ]
On Fri, Aug 31, 2012 at 10:26 AM, David Gerard <dgerard@gmail.com> wrote:
> On 31 August 2012 18:21, Jon Robson <jdlrobson@gmail.com> wrote:
>
>> +1 I was very surprised to learn any thumbnail sizes could be generated. We
>> should standardise on a tiny, small,medium high and original resolutions. 5
>> sizes seems more than enough.
>
>
> I would suggest checking against the corpus before saying "x should be
> enough for anyone". There are also occasional but valid uses for
> non-standard sizes.
>
>
> - d.

True, but if we were to institute a cleanup program to rationalize
sizes to a limited set, even if particular users object on a page here
or there and we leave it be, the vast bulk of the work is easy and
gets us the size usage win.

The questions to me are what sizes to standardize towards, and how can
we tell where things are used at what size?


--
-george william herbert
george.herbert@gmail.com

_______________________________________________
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: scaled media (thumbs) as *temporary* files, not stored forever [ In reply to ]
Please don't. The current syntax is nice, concise, consistent and not
overflowing with special characters. The proposed one is verbose and
"looks technical". But defaulting to thumb seems like a good idea to
me (but we ought to make some usage stats first :) ).

2012/9/1, Brion Vibber <brion@pobox.com>:
> On Fri, Aug 31, 2012 at 3:52 PM, Daniel Zahn <dzahn@wikimedia.org> wrote:
>
>> Maybe we could have "large","medium","small" etc as aliases for
>> standard/popular sizes to encourage using less of the non-standard
>> ones?
>>
>
> I kinda like this. It would also be nice if simply including an image
> defaulted to some sane size, even without using explicit "|thumb".
>
> In fact, we should think about just redoing how images get included in the
> first place maybe. :P
>
> I'd kind of like to see something like this:
>
> {{#media:Foobar.jpg}} <- default to a nice size, displayed in some
> nice-looking way suitable to the output. Sane framing and positioning
> typical for most usages.
>
> {{#media:Foobar.jpg|caption=Hello this is my caption about [[stuff]].
> Enjoy!}} <- caption should probably be an explicitly named parameter
>
> * Consider having *no size option* at all. :)
> * Definitely don't have "left" "right" or "center" options.
> * Consider making it easy to collect multiple related photos together, like
> <gallery>.
>
> Or maybe we should just use <gallery> more aggressively and make it a
> billion times prettier...
>
>
> For the more icon-like uses, maybe an explicit inline-media function:
>
> {{#inline-media:Foobar.svg|24x24px}}
>
> Anyway.... this needs more thought. But for a lot of images, we don't
> really *need* to be manually specifying every detail of their layout. It
> feels like it would be nicer to say "stuff these photos, with these
> captions, into this section of the article" and let the wiki deal with
> laying them out.
>
> Note that in mobile/tablet contexts it's also very handy to be able to
> extract just the photos and provide them for separate browsing; this has
> influenced my thinking on this for sure.
>
> -- brion
> _______________________________________________
> Wikitech-l mailing list
> Wikitech-l@lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
>

--
Wysłane z mojego urządzenia przenośnego

-- Matma Rex

_______________________________________________
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: scaled media (thumbs) as *temporary* files, not stored forever [ In reply to ]
Ariel wrote:
> But there's plenty of "odd" sizes with lots of thumbs too. For example,
> over 65K files with width 181px

181px images are tipically made when 180px images are "bad", with the
squid/scaler refusing to deliver the latest version. Adding 1px there
forces a regeneration of the file, solving the issue.
Going up to 65K seems too much, though.


On 31/08/12 22:35, Roan Kattouw wrote:
> On Fri, Aug 31, 2012 at 1:22 PM, Jeremy Baron <jeremy@tuxmachine.com> wrote:
>> I believe the status quo is no thumbs (of any size) are generated at
>> upload time. They are all just done on demand.
>>
> That's correct in theory. In practice (for uploads using
> Special:Upload at least) the user is redirected to the file
> description page after a successful upload, which contains a thumb of
> the file of a given size (800px?), which is then immediately generated
> on demand :) . Special:UploadWizard has similar behavior: the success
> page contains 200px(?) thumbs of all the images the user uploaded.
>
> Roan

Plus another tiny one at the file history section.

I remember seeing a change about rounding the requested image size some
months ago, so the "request any size" may not be true any longer.


_______________________________________________
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: scaled media (thumbs) as *temporary* files, not stored forever [ In reply to ]
On Fri, Aug 31, 2012 at 6:17 PM, Bartosz Dziewoński <matma.rex@gmail.com>wrote:

> Please don't. The current syntax is nice, concise, consistent and not
> overflowing with special characters. The proposed one is verbose and
> "looks technical".


The current syntax is actually hard to machine-parse, with lots of
language-specific overrides and weird options that combine in non-obvious
ways. Not to mention that it overrides the simple link syntax... I wish
when we'd renamed Image: to File: that we'd left the magic behavior only on
the Image: alias so that links would be rationalized. :(

Keep in mind also that in the glorious unicorn-filled future most people
who aren't doing low-level template work will rarely see the markup.
They'll just push a button to insert an image.


> But defaulting to thumb seems like a good idea to
> me (but we ought to make some usage stats first :) ).
>

Yeah... changing existing behavior gets scary. :)

-- brion
_______________________________________________
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: scaled media (thumbs) as *temporary* files, not stored forever [ In reply to ]
On Fri, Aug 31, 2012 at 4:38 PM, Brion Vibber <brion@pobox.com> wrote:
> Keep in mind also that in the glorious unicorn-filled future most people
> who aren't doing low-level template work will rarely see the markup.
> They'll just push a button to insert an image.

Is that really a rainbow coming out of the Unicorn, Brion? It seems
like an unlikely place to...... Nevermind.

>> But defaulting to thumb seems like a good idea to
>> me (but we ought to make some usage stats first :) ).
>>
>
> Yeah... changing existing behavior gets scary. :)

Usage stats win. So - how can we get those?


--
-george william herbert
george.herbert@gmail.com

_______________________________________________
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: scaled media (thumbs) as *temporary* files, not stored forever [ In reply to ]
On 1 September 2012 00:38, Brion Vibber <brion@pobox.com> wrote:

> The current syntax is actually hard to machine-parse, with lots of
> language-specific overrides and weird options that combine in non-obvious
> ways. Not to mention that it overrides the simple link syntax... I wish
> when we'd renamed Image: to File: that we'd left the magic behavior only on
> the Image: alias so that links would be rationalized. :(


PROBLEM: Thumbnails are taking up a lot of disk space.
SOLUTION: Completely revamp image syntax in a backwards-incompatible manner.

There's something excessive there ...

Is it really the case that the server can't get atime of the images?
iIf it can, then we *know* which thumbs may as well be trashed and
regenerated as and when someone cares. If it can't, why not?


- d.

_______________________________________________
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: scaled media (thumbs) as *temporary* files, not stored forever [ In reply to ]
On Fri, Aug 31, 2012 at 9:31 PM, David Gerard <dgerard@gmail.com> wrote:

> On 1 September 2012 00:38, Brion Vibber <brion@pobox.com> wrote:
>
> > The current syntax is actually hard to machine-parse, with lots of
> > language-specific overrides and weird options that combine in non-obvious
> > ways. Not to mention that it overrides the simple link syntax... I wish
> > when we'd renamed Image: to File: that we'd left the magic behavior only
> on
> > the Image: alias so that links would be rationalized. :(
>
>
> PROBLEM: Thumbnails are taking up a lot of disk space.
> SOLUTION: Completely revamp image syntax in a backwards-incompatible
> manner.
>
> There's something excessive there ...
>

Nah, we're just years behind on modernizing image handling. Such
improvements need to be done at some point regardless of this particular
question, but it would help with it in certain ways.

Is it really the case that the server can't get atime of the images?
> iIf it can, then we *know* which thumbs may as well be trashed and
> regenerated as and when someone cares. If it can't, why not?
>

I'd tend to think that image scaler CPU time is more precious than disk
space used by thumbs; in theory we don't actually need to "store" thumbs if
we just cache them and have a suitably large cache. A caching HTTP proxy
should have some sort of LRU-or-other system to discard old things that
aren't being used; my main worry would be about whether the system can
survive the load of a cache being cleared (say due to downtime, upgrades,
incompatible storage formats for upgrades, or whatever).

Of course if you don't have to scale anything on demand on the server side,
suddenly the entire problem disappears. Just something to think about.

-- brion
_______________________________________________
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

1 2 3  View All