Mailing List Archive

Upcoming SyntaxHighlight_GeSHi changes
Hello,

Over the course of the next two days, a major update to the
SyntaxHighlight_GeSHi extension will be rolled out to Wikimedia wikis. The
change swaps geshi, the unmaintained PHP library which performs the lexical
analysis and output formatting of code, for another library, called
Pygments.

The roll-out will remove support for 31 languages while adding support for
several hundred languages not previously supported, including Dart, Rust,
Julia, APL, Mathematica, SNOBOL, Puppet, Dylan, Racket, Swift, and many
others. See <https://people.wikimedia.org/~ori/geshi_changes.txt> for a
full list. The languages that will lose support are mostly obscure, with
the notable exception of ALGOL68, Oz, and MMIX.

The change is expected to slightly improve the time it takes to load and
render all pages on all wikis (not just those that contain code blocks!),
at the cost of a slight penalty (about a tenth of a second) on the time it
takes to save edits which introduce or modify a block of highlighted code
to an article.

Lastly, the way the extension handles unfamiliar languages will change.
Previously, if the specified language was not supported by the extension,
instead of a code block, the extension would print an error message. From
now on, it will simply output a plain, unhighlighted block of monospaced
code.

The wikitext syntax for highlighting code will remain the same.

-- Ori
_______________________________________________
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: Upcoming SyntaxHighlight_GeSHi changes [ In reply to ]
Ori Livneh wrote:
>Over the course of the next two days, a major update to the
>SyntaxHighlight_GeSHi extension will be rolled out to Wikimedia wikis. The
>change swaps geshi, the unmaintained PHP library which performs the
>lexical analysis and output formatting of code, for another library,
>called Pygments.
>
>The roll-out will remove support for 31 languages while adding support for
>several hundred languages not previously supported, including Dart, Rust,
>Julia, APL, Mathematica, SNOBOL, Puppet, Dylan, Racket, Swift, and many
>others. See <https://people.wikimedia.org/~ori/geshi_changes.txt> for a
>full list. The languages that will lose support are mostly obscure, with
>the notable exception of ALGOL68, Oz, and MMIX.

Thanks for writing this e-mail and putting together an easy-to-read list.

I use the SyntaxHighlight_GeSHi MediaWiki extension on a few non-Wikimedia
wikis. If I just run "git pull" in the extension's directory in a few
days, is that sufficient? The upgrade path isn't totally clear to me from
your e-mail. I'm also curious if the Pygments version of the extension
will change the required MediaWiki core version needed.

>Lastly, the way the extension handles unfamiliar languages will change.
>Previously, if the specified language was not supported by the extension,
>instead of a code block, the extension would print an error message. From
>now on, it will simply output a plain, unhighlighted block of monospaced
>code.

Actually, not instead of, but in addition to. In general, your change
seems like it will be an improvement. My only feature request would be for
a tracking category to be auto-populated. Something similar to "Pages
containing invalid syntax highlight languages" which would then allow wiki
editors to find and address these pages (by setting lang="text" or by
filing Phabricator Maniphest tasks to add support for missing languages).

For what it's worth, I occasionally found the verbose error message
helpful if I tried, for example, <syntaxhighlight lang="html"> instead of
lang="html4strict" or lang="html5". I'm not sure if Pygments has
additional aliases or is a bit more lenient in cases like this?

MZMcBride



_______________________________________________
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: Upcoming SyntaxHighlight_GeSHi changes [ In reply to ]
On Mon, Jun 22, 2015 at 6:37 PM, MZMcBride <z@mzmcbride.com> wrote:

> I use the SyntaxHighlight_GeSHi MediaWiki extension on a few non-Wikimedia
> wikis. If I just run "git pull" in the extension's directory in a few
> days, is that sufficient?


Almost. There are updated instructions in
https://github.com/wikimedia/mediawiki-extensions-SyntaxHighlight_GeSHi/blob/master/README#L16-40
.



> The upgrade path isn't totally clear to me from
> your e-mail. I'm also curious if the Pygments version of the extension
> will change the required MediaWiki core version needed.
>

It requires MediaWiki 1.25, but the previous version did as well:

https://github.com/wikimedia/mediawiki-extensions-SyntaxHighlight_GeSHi/blob/f3476de0158e464db442213d6731b691a2dd8af1/SyntaxHighlight_GeSHi.php#L12


> >Lastly, the way the extension handles unfamiliar languages will change.
> >Previously, if the specified language was not supported by the extension,
> >instead of a code block, the extension would print an error message. From
> >now on, it will simply output a plain, unhighlighted block of monospaced
> >code.
>
> Actually, not instead of, but in addition to.


Correct. My description of the current behavior was inaccurate. Thanks.


> My only feature request would be for
> a tracking category to be auto-populated. Something similar to "Pages
> containing invalid syntax highlight languages" which would then allow wiki
> editors to find and address these pages (by setting lang="text" or by
> filing Phabricator Maniphest tasks to add support for missing languages).
>

Sounds reasonable. Could you file a task?
_______________________________________________
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: Upcoming SyntaxHighlight_GeSHi changes [ In reply to ]
On Tue, Jun 23, 2015 at 10:48 AM, Ori Livneh <ori@wikimedia.org> wrote:
> Hello,
>
> Over the course of the next two days, a major update to the
> SyntaxHighlight_GeSHi extension will be rolled out to Wikimedia wikis. The
> change swaps geshi, the unmaintained PHP library which performs the lexical
> analysis and output formatting of code, for another library, called
> Pygments.
>
> The roll-out will remove support for 31 languages while adding support for
> several hundred languages not previously supported, including Dart, Rust,
> Julia, APL, Mathematica, SNOBOL, Puppet, Dylan, Racket, Swift, and many
> others. See <https://people.wikimedia.org/~ori/geshi_changes.txt> for a
> full list.

A very welcome list of additions!

> The languages that will lose support are mostly obscure, with
> the notable exception of ALGOL68, Oz, and MMIX.

There are a few more exceptions, such as bnf, dot, pcre, email, and bibtex.

Hmm, algol support was recently added to Pygments.

https://bitbucket.org/birkenfeld/pygments-main/issue/1090/update-to-m2-lexer-in

Perhaps it needs to be backported into their stable branch, and a new
minor release pushed out for use on Wikimedia?

Since it is short, here is the full list of languages being de-supported.

6502acme
68000devpac
algol68
arm
avisynth
bibtex
bnf
cil
dot
e
email
euphoria
gml
ldif
lolcode
mirc
mmix
mpasm
oz
parigp
pcre
pic16
pli
q
robots
sas
teraterm
typoscript
unicon
whois
xpp

Of the English Wikipedia articles about those concept that I have
looked at, so far they all use <source> with the appropriate language
set, so they will all regress down to plain monospaced text.

Have you identified how many times each of these have been used on
Wikimedia projects? Perhaps that might help us identify the priority
of languages needing to be added to Pygments (and which ones are
entirely useless.

Some of them have bugs raised in Pygments
https://bitbucket.org/birkenfeld/pygments-main/issue/1024/add-dot-lexer-graphviz-language

> Lastly, the way the extension handles unfamiliar languages will change.
> Previously, if the specified language was not supported by the extension,
> instead of a code block, the extension would print an error message. From
> now on, it will simply output a plain, unhighlighted block of monospaced
> code.

Ugh, is there a way to configure pygments to have fallbacks for
languages which are substantially based on another? e.g. xpp is
basically java, and looks quite good when I tested it on betalabs. I
am sure that Pygments has some parser close to 'email', as they do
support a 'http session' language.

--
John Vandenberg

_______________________________________________
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: Upcoming SyntaxHighlight_GeSHi changes [ In reply to ]
On 2015-06-22 10:30 PM, John Mark Vandenberg wrote:
> I am sure that Pygments has some parser close to 'email', as they do
> support a 'http session' language.
That lexer:
https://bitbucket.org/birkenfeld/pygments-main/src/652e08e410f4f4f3a61add91f75ce5ed9d134711/pygments/lexers/textfmts.py?at=default#cl-113

Does highlight some specific http things like the request method. The
auto-detection also tests for the presence of the `GET /`, etc... pattern.

However it would be trivial to split that into a common lexer base or
just copy and modify it, so that the highlighter instead highlights
patterns in MIME instead of HTTP.

~Daniel Friesen (Dantman, Nadir-Seen-Fire) [http://danielfriesen.name/]


_______________________________________________
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: Upcoming SyntaxHighlight_GeSHi changes [ In reply to ]
On Tue, Jun 23, 2015 at 2:48 AM, Ori Livneh <ori@wikimedia.org> wrote:

> Lastly, the way the extension handles unfamiliar languages will change.
> Previously, if the specified language was not supported by the extension,
> instead of a code block, the extension would print an error message. From
> now on, it will simply output a plain, unhighlighted block of monospaced
> code.
>
>
That's really nice, but I have a request.
Previously, the error message contained the list of languages supported by
the extension, so it was easy to find which value you should use.
Without the error message, how do we easily get the list of languages that
can really be used ?
By easily, I mean directly accessible from the page we are currently
editing.
Maybe something like a small icon in the code block with either a tooltip
listing languages or a link to a page with the list of languages.

Thanks
Nico
_______________________________________________
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: Upcoming SyntaxHighlight_GeSHi changes [ In reply to ]
Also for those that prefer the client to do the work I recently made an
extension to use the highlight.js module at highlightjs.org.
https://www.mediawiki.org/wiki/Extension:HighlightJS

On 22/06/15 21:48, Ori Livneh wrote:
> Hello,
>
> Over the course of the next two days, a major update to the
> SyntaxHighlight_GeSHi extension will be rolled out to Wikimedia wikis. The
> change swaps geshi, the unmaintained PHP library which performs the lexical
> analysis and output formatting of code, for another library, called
> Pygments.
>
> The roll-out will remove support for 31 languages while adding support for
> several hundred languages not previously supported, including Dart, Rust,
> Julia, APL, Mathematica, SNOBOL, Puppet, Dylan, Racket, Swift, and many
> others. See <https://people.wikimedia.org/~ori/geshi_changes.txt> for a
> full list. The languages that will lose support are mostly obscure, with
> the notable exception of ALGOL68, Oz, and MMIX.
>
> The change is expected to slightly improve the time it takes to load and
> render all pages on all wikis (not just those that contain code blocks!),
> at the cost of a slight penalty (about a tenth of a second) on the time it
> takes to save edits which introduce or modify a block of highlighted code
> to an article.
>
> Lastly, the way the extension handles unfamiliar languages will change.
> Previously, if the specified language was not supported by the extension,
> instead of a code block, the extension would print an error message. From
> now on, it will simply output a plain, unhighlighted block of monospaced
> code.
>
> The wikitext syntax for highlighting code will remain the same.
>
> -- Ori
> _______________________________________________
> Wikitech-l mailing list
> Wikitech-l@lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l


_______________________________________________
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: Upcoming SyntaxHighlight_GeSHi changes [ In reply to ]
>https://people.wikimedia.org/~ori/geshi_changes.txt

What about languages that aren't listed in your "Supported" or "Unsupported" lists? We use "lang=tsql" in many pages on our wiki, for T-SQL, but it's not clear to me what the replacement is. Thanks.

DanB

_______________________________________________
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: Upcoming SyntaxHighlight_GeSHi changes [ In reply to ]
On Mon, Jun 22, 2015 at 10:30 PM, John Mark Vandenberg <jayvdb@gmail.com>
wrote:

> Ugh, is there a way to configure pygments to have fallbacks for
> languages which are substantially based on another? e.g. xpp is
> basically java, and looks quite good when I tested it on betalabs. I
> am sure that Pygments has some parser close to 'email', as they do
> support a 'http session' language.
>

Yes. We maintain a mapping from GeSHi lexer names to compatible Pygment
lexers:

https://github.com/wikimedia/mediawiki-extensions-SyntaxHighlight_GeSHi/blob/master/SyntaxHighlight_GeSHi.compat.php#L26-96

Making xpp render as Java would be as simple as adding "xpp" => "java". If
you submit a patch, I will be happy to merge it.
_______________________________________________
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: Upcoming SyntaxHighlight_GeSHi changes [ In reply to ]
On Tue, Jun 23, 2015 at 7:55 AM, Daniel Barrett <danb@cimpress.com> wrote:

> >https://people.wikimedia.org/~ori/geshi_changes.txt
>
> What about languages that aren't listed in your "Supported" or
> "Unsupported" lists?


They'll continue to work as before.
_______________________________________________
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: Upcoming SyntaxHighlight_GeSHi changes [ In reply to ]
On Tue, Jun 23, 2015 at 7:23 AM, Aran <aran@organicdesign.co.nz> wrote:

> Also for those that prefer the client to do the work I recently made an
> extension to use the highlight.js module at highlightjs.org.
> https://www.mediawiki.org/wiki/Extension:HighlightJS


This is very nice, and may be a target for a future migration. It'd be
great if the extension had a server-side implementation as well in the form
of a node.js service that MediaWiki could call.
_______________________________________________
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: Upcoming SyntaxHighlight_GeSHi changes [ In reply to ]
Il 23/06/2015 07:30, John Mark Vandenberg ha scritto:
> Since it is short, here is the full list of languages being de-supported.
>
> 6502acme
> 68000devpac
> algol68
> arm
> avisynth
> bibtex
> bnf
> cil
> dot
> e
> email
> euphoria
> gml
> ldif
> lolcode

Nooooooooooooo!!! I need LOLCODE!!!!!!!

> mirc
> mmix
> mpasm
> oz
> parigp
> pcre
> pic16
> pli
> q
> robots
> sas
> teraterm
> typoscript
> unicon
> whois
> xpp

_______________________________________________
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: Upcoming SyntaxHighlight_GeSHi changes [ In reply to ]
Ori Livneh wrote:
>On Mon, Jun 22, 2015 at 6:37 PM, MZMcBride <z@mzmcbride.com> wrote:
>> My only feature request would be for a tracking category to be
>>auto-populated. Something similar to "Pages containing invalid syntax
>>highlight languages" which would then allow wiki editors to find and
>>address these pages (by setting lang="text" or by filing Phabricator
>>Maniphest tasks to add support for missing languages).
>
>Sounds reasonable. Could you file a task?

Sure, filed as <https://phabricator.wikimedia.org/T103586>.

>>For what it's worth, I occasionally found the verbose error message
>>helpful if I tried, for example, <syntaxhighlight lang="html"> instead of
>>lang="html4strict" or lang="html5". I'm not sure if Pygments has
>>additional aliases or is a bit more lenient in cases like this?

I tested with the Pygments-backed version of the extension and "html",
"html5", and "html4strict" all seemed to work. Very nice!

Nicolas Vervelle wrote:
>Previously, the error message contained the list of languages supported by
>the extension, so it was easy to find which value you should use.
>Without the error message, how do we easily get the list of languages that
>can really be used ?
>By easily, I mean directly accessible from the page we are currently
>editing.

When using VisualEditor, there should just be a drop-down menu of
available languages, of course. But when editing the wikitext source, I'm
not sure how we'd expose such a list.

>Maybe something like a small icon in the code block with either a tooltip
>listing languages or a link to a page with the list of languages.

I guess the challenge here is that we only need the list of languages
available when editing, not when viewing the page. And even when editing,
we only need the list of languages available if <syntaxhighlight> is being
invoked. Consequently, exposing a list of supported languages seems
tricky. We could maybe do something on page preview?

MZMcBride



_______________________________________________
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: Upcoming SyntaxHighlight_GeSHi changes [ In reply to ]
On 24 June 2015 at 03:44, MZMcBride <z@mzmcbride.com> wrote:

> Nicolas Vervelle wrote:
> >Previously, the error message contained the list of languages supported by
> >the extension, so it was easy to find which value you should use.
> >Without the error message, how do we easily get the list of languages that
> >can really be used ?
> >By easily, I mean directly accessible from the page we are currently
> >editing.
>
> When using VisualEditor, there should just be a drop-down menu of
> available languages, of course.


Yeah, I'm hoping to get this as a follow-up to
https://gerrit.wikimedia.org/r/#/c/214078/
_______________________________________________
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: Upcoming SyntaxHighlight_GeSHi changes [ In reply to ]
(Now filed as https://phabricator.wikimedia.org/T103756)

On 24 June 2015 at 16:03, Alex Monk <krenair@gmail.com> wrote:

> On 24 June 2015 at 03:44, MZMcBride <z@mzmcbride.com> wrote:
>
>> Nicolas Vervelle wrote:
>> >Previously, the error message contained the list of languages supported
>> by
>> >the extension, so it was easy to find which value you should use.
>> >Without the error message, how do we easily get the list of languages
>> that
>> >can really be used ?
>> >By easily, I mean directly accessible from the page we are currently
>> >editing.
>>
>> When using VisualEditor, there should just be a drop-down menu of
>> available languages, of course.
>
>
> Yeah, I'm hoping to get this as a follow-up to
> https://gerrit.wikimedia.org/r/#/c/214078/
>
>
_______________________________________________
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: Upcoming SyntaxHighlight_GeSHi changes [ In reply to ]
A nodejs service would be nice, but probably not on the cards in the
near future - note however that the extension does include a simple
method of ensuring that the highlighting module is called for blocks
even if they're inserted into the page after load time such as in Ajax
calls like live preview.


On 23/06/15 12:14, Ori Livneh wrote:
> On Tue, Jun 23, 2015 at 7:23 AM, Aran <aran@organicdesign.co.nz> wrote:
>
>> Also for those that prefer the client to do the work I recently made an
>> extension to use the highlight.js module at highlightjs.org.
>> https://www.mediawiki.org/wiki/Extension:HighlightJS
>
> This is very nice, and may be a target for a future migration. It'd be
> great if the extension had a server-side implementation as well in the form
> of a node.js service that MediaWiki could call.
> _______________________________________________
> Wikitech-l mailing list
> Wikitech-l@lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l


_______________________________________________
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: Upcoming SyntaxHighlight_GeSHi changes [ In reply to ]
On Mon, Jun 22, 2015 at 8:48 PM, Ori Livneh <ori@wikimedia.org> wrote:

> Over the course of the next two days, a major update to the
> SyntaxHighlight_GeSHi extension will be rolled out to Wikimedia wikis. The
> change swaps geshi, the unmaintained PHP library which performs the lexical
> analysis and output formatting of code, for another library, called
> Pygments.
>

... Please tell me we're not really going to have the final state here be
an extension named "SyntaxHighlight_GeSHi" that doesn't use GeSHi anymore.


--
Brad Jorsch (Anomie)
Software Engineer
Wikimedia Foundation
_______________________________________________
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: Upcoming SyntaxHighlight_GeSHi changes [ In reply to ]
Il 29/06/2015 20:01, Brad Jorsch (Anomie) ha scritto:
> On Mon, Jun 22, 2015 at 8:48 PM, Ori Livneh <ori@wikimedia.org> wrote:
>
>> Over the course of the next two days, a major update to the
>> SyntaxHighlight_GeSHi extension will be rolled out to Wikimedia wikis. The
>> change swaps geshi, the unmaintained PHP library which performs the lexical
>> analysis and output formatting of code, for another library, called
>> Pygments.
>>
> ... Please tell me we're not really going to have the final state here be
> an extension named "SyntaxHighlight_GeSHi" that doesn't use GeSHi anymore.
>
>

Here we are.

_______________________________________________
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: Upcoming SyntaxHighlight_GeSHi changes [ In reply to ]
On Mon, 29 Jun 2015 20:01:08 +0200, Brad Jorsch (Anomie)
<bjorsch@wikimedia.org> wrote:

> ... Please tell me we're not really going to have the final state here be
> an extension named "SyntaxHighlight_GeSHi" that doesn't use GeSHi
> anymore.

Yeah, that would suck somewhat.

https://phabricator.wikimedia.org/T103614


--
Bartosz Dziewoński

_______________________________________________
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: Upcoming SyntaxHighlight_GeSHi changes [ In reply to ]
On Mon, Jun 29, 2015 at 11:04 AM, Ricordisamoa <ricordisamoa@openmailbox.org
> wrote:

> Il 29/06/2015 20:01, Brad Jorsch (Anomie) ha scritto:
>
>> On Mon, Jun 22, 2015 at 8:48 PM, Ori Livneh <ori@wikimedia.org> wrote:
>>
>> Over the course of the next two days, a major update to the
>>> SyntaxHighlight_GeSHi extension will be rolled out to Wikimedia wikis.
>>> The
>>> change swaps geshi, the unmaintained PHP library which performs the
>>> lexical
>>> analysis and output formatting of code, for another library, called
>>> Pygments.
>>>
>>> ... Please tell me we're not really going to have the final state here
>> be
>> an extension named "SyntaxHighlight_GeSHi" that doesn't use GeSHi anymore.
>>
>> Here we are.


The mixed case, nonsense word, and inconsistent word separation were simply
too dear to let go. It puts a song in your heart every time you need to
type it.
_______________________________________________
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: Upcoming SyntaxHighlight_GeSHi changes [ In reply to ]
On Tue, Jun 23, 2015 at 10:48 AM, Ori Livneh <ori@wikimedia.org> wrote:
> Hello,
>
> Over the course of the next two days, a major update to the
> SyntaxHighlight_GeSHi extension will be rolled out to Wikimedia wikis. The
> change swaps geshi, the unmaintained PHP library which performs the lexical
> analysis and output formatting of code, for another library, called
> Pygments.
>
> The roll-out will remove support for 31 languages while adding support for
> several hundred languages not previously supported, including Dart, Rust,
> Julia, APL, Mathematica, SNOBOL, Puppet, Dylan, Racket, Swift, and many
> others. See <https://people.wikimedia.org/~ori/geshi_changes.txt> for a
> full list. The languages that will lose support are mostly obscure, with
> the notable exception of ALGOL68, Oz, and MMIX.

I was surprised to find other languages not in your text file that
appear to no longer be supported.

I've gone through the geshi php files looking for assembler languages
only so far:
https://gerrit.wikimedia.org/r/224379

How/Why were these excluded in your list?

I've found reasonable fallbacks for many of the other unsupported
languages, and as previously mentioned they have added support for
some of these languages.
Is Wikimedia going to only use the official releases of Pygments?
Their last release was six months ago.
Is someone in communication with them about pushing out a new release?
If they dont release them soon, will Wikimedia maintain its own
version of the software?

--
John Vandenberg

_______________________________________________
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: Upcoming SyntaxHighlight_GeSHi changes [ In reply to ]
On Mon, Jul 13, 2015 at 1:30 PM, John Mark Vandenberg <jayvdb@gmail.com> wrote:
> On Tue, Jun 23, 2015 at 10:48 AM, Ori Livneh <ori@wikimedia.org> wrote:
>> Hello,
>>
>> Over the course of the next two days, a major update to the
>> SyntaxHighlight_GeSHi extension will be rolled out to Wikimedia wikis. The
>> change swaps geshi, the unmaintained PHP library which performs the lexical
>> analysis and output formatting of code, for another library, called
>> Pygments.
>>
>> The roll-out will remove support for 31 languages while adding support for
>> several hundred languages not previously supported, including Dart, Rust,
>> Julia, APL, Mathematica, SNOBOL, Puppet, Dylan, Racket, Swift, and many
>> others. See <https://people.wikimedia.org/~ori/geshi_changes.txt> for a
>> full list. The languages that will lose support are mostly obscure, with
>> the notable exception of ALGOL68, Oz, and MMIX.
>
> I was surprised to find other languages not in your text file that
> appear to no longer be supported.
>
> I've gone through the geshi php files looking for assembler languages
> only so far:
> https://gerrit.wikimedia.org/r/224379
>
> How/Why were these excluded in your list?

I've encountered more of these on Wikipedia, so, ...

Here is a list of 59 geshi supported languages which were omitted from
the above list of 31 languages being de-supported by the switch to
Pygments.

4cs
6502kickass
6502tasm
aimms
apt_sources
autoconf
caddcl
cfdg
cuesheet
dcl
dcpu16
dcs
epc
ezt
f1
falcon
fo
freeswitch
gdb
genero
genie
gettext
hicest
hq9plus
icon
inno
intercal
ispfpanel
jcl
kixtart
klonec
klonecpp
lb
lotusformulas
lotusscript
lscript
lsl2
m68k
magiksf
nagios
objeck
oxygene
parasail
per
pf
pixelbender
powerbuilder
proftpd
providex
rbs
scl
spark
stonescript
uscript
vbscript
vedit
whitespace
xorg_conf
z80

--
John Vandenberg

_______________________________________________
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: Upcoming SyntaxHighlight_GeSHi changes [ In reply to ]
On Tue, Jul 14, 2015 at 4:35 AM, John Mark Vandenberg <jayvdb@gmail.com>
wrote:

> On Mon, Jul 13, 2015 at 1:30 PM, John Mark Vandenberg <jayvdb@gmail.com>
> wrote:
> > On Tue, Jun 23, 2015 at 10:48 AM, Ori Livneh <ori@wikimedia.org> wrote:
> >> Hello,
> >>
> >> Over the course of the next two days, a major update to the
> >> SyntaxHighlight_GeSHi extension will be rolled out to Wikimedia wikis.
> The
> >> change swaps geshi, the unmaintained PHP library which performs the
> lexical
> >> analysis and output formatting of code, for another library, called
> >> Pygments.
> >>
> >> The roll-out will remove support for 31 languages while adding support
> for
> >> several hundred languages not previously supported, including Dart,
> Rust,
> >> Julia, APL, Mathematica, SNOBOL, Puppet, Dylan, Racket, Swift, and many
> >> others. See <https://people.wikimedia.org/~ori/geshi_changes.txt> for a
> >> full list. The languages that will lose support are mostly obscure, with
> >> the notable exception of ALGOL68, Oz, and MMIX.
> >
> > I was surprised to find other languages not in your text file that
> > appear to no longer be supported.
> >
> > I've gone through the geshi php files looking for assembler languages
> > only so far:
> > https://gerrit.wikimedia.org/r/224379
> >
> > How/Why were these excluded in your list?
>
> I've encountered more of these on Wikipedia, so, ...
>
> Here is a list of 59 geshi supported languages which were omitted from
> the above list of 31 languages being de-supported by the switch to
> Pygments.


These haven't been supported since
https://gerrit.wikimedia.org/r/#/c/197449/ -- see
https://phabricator.wikimedia.org/T93025 .
_______________________________________________
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: Upcoming SyntaxHighlight_GeSHi changes [ In reply to ]
Hi,

Is this related to the fact that some pages are now categorized in "Pages
with syntax highlighting errors
<https://en.wikipedia.org/wiki/Category:Pages_with_syntax_highlighting_errors>"
?
How can we find what is wrong in the page ?
I tried several things to fix Wikipedia talk:WPCleaner, but didn't manage
to remove the categorization.
https://en.wikipedia.org/wiki/Wikipedia_talk:WPCleaner

Is it also related to a change of behavior when there are nowiki tags
inside the text ?
I think they were necessary before in some cases, and are simply not
recognized (treated as plain text) now.

Nico

On Tue, Jun 23, 2015 at 2:48 AM, Ori Livneh <ori@wikimedia.org> wrote:

> Hello,
>
> Over the course of the next two days, a major update to the
> SyntaxHighlight_GeSHi extension will be rolled out to Wikimedia wikis. The
> change swaps geshi, the unmaintained PHP library which performs the lexical
> analysis and output formatting of code, for another library, called
> Pygments.
>
> The roll-out will remove support for 31 languages while adding support for
> several hundred languages not previously supported, including Dart, Rust,
> Julia, APL, Mathematica, SNOBOL, Puppet, Dylan, Racket, Swift, and many
> others. See <https://people.wikimedia.org/~ori/geshi_changes.txt> for a
> full list. The languages that will lose support are mostly obscure, with
> the notable exception of ALGOL68, Oz, and MMIX.
>
> The change is expected to slightly improve the time it takes to load and
> render all pages on all wikis (not just those that contain code blocks!),
> at the cost of a slight penalty (about a tenth of a second) on the time it
> takes to save edits which introduce or modify a block of highlighted code
> to an article.
>
> Lastly, the way the extension handles unfamiliar languages will change.
> Previously, if the specified language was not supported by the extension,
> instead of a code block, the extension would print an error message. From
> now on, it will simply output a plain, unhighlighted block of monospaced
> code.
>
> The wikitext syntax for highlighting code will remain the same.
>
> -- Ori
> _______________________________________________
> Wikitech-l mailing list
> Wikitech-l@lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
_______________________________________________
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: Upcoming SyntaxHighlight_GeSHi changes [ In reply to ]
> On 15 jul. 2015, at 18:43, Nicolas Vervelle <nvervelle@gmail.com> wrote:
>
> Hi,
>
> Is this related to the fact that some pages are now categorized in "Pages
> with syntax highlighting errors
> <https://en.wikipedia.org/wiki/Category:Pages_with_syntax_highlighting_errors>"
> ?
> How can we find what is wrong in the page ?
> I tried several things to fix Wikipedia talk:WPCleaner, but didn't manage
> to remove the categorization.
> https://en.wikipedia.org/wiki/Wikipedia_talk:WPCleaner
>

Fixed
https://en.wikipedia.org/w/index.php?title=Wikipedia_talk%3AWPCleaner&type=revision&diff=671587123&oldid=671579619


> Is it also related to a change of behavior when there are nowiki tags
> inside the text ?
> I think they were necessary before in some cases, and are simply not
> recognized (treated as plain text) now.

Yes, they are no longer necessary and behave as you describe.

DJ


> On Tue, Jun 23, 2015 at 2:48 AM, Ori Livneh <ori@wikimedia.org> wrote:
>
>> Hello,
>>
>> Over the course of the next two days, a major update to the
>> SyntaxHighlight_GeSHi extension will be rolled out to Wikimedia wikis. The
>> change swaps geshi, the unmaintained PHP library which performs the lexical
>> analysis and output formatting of code, for another library, called
>> Pygments.
>>
>> The roll-out will remove support for 31 languages while adding support for
>> several hundred languages not previously supported, including Dart, Rust,
>> Julia, APL, Mathematica, SNOBOL, Puppet, Dylan, Racket, Swift, and many
>> others. See <https://people.wikimedia.org/~ori/geshi_changes.txt> for a
>> full list. The languages that will lose support are mostly obscure, with
>> the notable exception of ALGOL68, Oz, and MMIX.
>>
>> The change is expected to slightly improve the time it takes to load and
>> render all pages on all wikis (not just those that contain code blocks!),
>> at the cost of a slight penalty (about a tenth of a second) on the time it
>> takes to save edits which introduce or modify a block of highlighted code
>> to an article.
>>
>> Lastly, the way the extension handles unfamiliar languages will change.
>> Previously, if the specified language was not supported by the extension,
>> instead of a code block, the extension would print an error message. From
>> now on, it will simply output a plain, unhighlighted block of monospaced
>> code.
>>
>> The wikitext syntax for highlighting code will remain the same.
>>
>> -- Ori
>> _______________________________________________
>> Wikitech-l mailing list
>> Wikitech-l@lists.wikimedia.org
>> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
> _______________________________________________
> Wikitech-l mailing list
> Wikitech-l@lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l