Mailing List Archive

[perl #112792] Perl Major Release Doc Flaws (5.8.0->5.10.0)
# New Ticket Created by Linda Walsh
# Please include the string: [perl #112792]
# in the subject line of all future correspondence about this issue.
# <URL: https://rt.perl.org:443/rt3/Ticket/Display.html?id=112792 >



This is a bug report for perl from perl-diddler@tlinx.org,
generated with the help of perlbug 1.39 running under perl 5.14.2.


-----------------------------------------------------------------
[Please describe your issue here]


If someone has not upgraded through each point release of perl --
a perfectly reasonable expectation under older releases where
point releases can out often, to address what was supposed to be
minor details while keeping compatibility within the major
5.X series (with interface changes going into larger 'point' releases,
like 5.8->5.10 5.10->5.12 -- but then only with "use newfeature"),

This practice was not (and often, is not) adhered to resulting in
chaos and a rain of crap upon users.

In 5.8.0, full UTF8 support on I/O was introduced:
Previously in Perl 5.6 to use Unicode one would say "use utf8" and then
the operations (like string concatenation) were Unicode-aware in that
lexical scope.

This was found to be an inconvenient interface, and in Perl 5.8 the
Unicode model has completely changed: now the "Unicodeness" is bound to
the data itself, and for most of the time "use utf8" is not needed at
all. The only remaining use of "use utf8" is when the Perl script
itself has been written in the UTF-8 encoding of Unicode. (UTF-8 has
not been made the default since there are many Perl scripts out there
that are using various national eight-bit character sets, which would
be illegal in UTF-8.)

There has been no mention of a change in any major release since
then.

The fact that it is NOT this way in 5.10, .12, .14 (likely .16), ..
is a complete failure in the release process in documenting major changes from release to release.

Only if one is involved in the day-to-day internal politics and development of perl and keeps track with every 'minor bug fix release' (of the sort that isn't to break compatibilty) or make incompatible interface changes, might one have seen in some footnote or other that this was no longer the case.

Apparently, someone slipped in a major UI change into a "minor release", and got it through approvals -- something that should never have happened.

But maybe it was somehow consider an 'emergency so dire that normal release processes had to be ignored'.

But if that was the case, one would have expected to see some note, or mention of this major change when 5.10.0 came out -- with major UI changes since
last main release.

This was not the case. I couldn't figure out why so many progs broke that were re-written to take advantage of the specification of 5.8.0.


I may have updated at 5.8.4 or 5.8.6 -- But usually when my distros upgrade
it's to 5.MAJ.<X for some X>0. I many do minor release upgrades by pulling down my own perl between distro releases, especially when I see a feature like full UTF-8 support based on ENV/local, because it was the direction of the future and as furter proof of that opinion, is now the HTML5 standard.

So I can easily see my jumping at the new support -- and finding it great! -- but then not seeing some footnote, that a Major incompatibility was introduced in 5.8.1. This major step forward was quietly removed.

No mention in the next 5.10.X version my distro released...(there might have
been a 5.8.9 in there with no mention of the issue or to read previous docs --I think 5.8 was during the 'slowdown' time).

Even more ironic is that 5.10.0 talks about more functions being converted to UTF-88 (Win32::GetLongPathName), rather than latin1 so instead of a mention of perl's backstep, we see what looks like progress. From then on, I see only mention of increasing compatbility... *BLECH*.


Slipping these types of changes in without notes in major releases seems completely vacuous at best, if not irresponsible. Too often changes are being made in the perl package by "perl-community insiders", who have no realistic view of how their changes are seen or will affect those who are not day-to-day members of the perl community, or users who use perl without being "of perl"...



>From my point of view, 5.10, 5.12, and 5.14 are all broken as no mention was
made of the change intro'ed to 5.8. That's the source of some of my recent confusion (and a whole subclass of unicode bugs caused by this incompatible change.

Someone may have thought they were getting pleasing their 'core friends' by reverting that functionality, but others were relying on it and who didn't get the dday-to-day sub-release messages, that sound like a posix or w3c committee yanking a standard around from meeting to meeting. Justing by the living Html5 website, I get that this practice isn't very well received by people trying to actually use released versions.

>From my perspective, I'll have to make sure the env var PERLIO is set, and either fail if not, or set it to the default as specified in the in the last major release to mention a change in this area.

That will ensure compatibility by default with major releases, putting the onus of change on those who slipped in major UI changes in a minor release, to invoke a 'change method' (set the ENV var, whatever), as it should have been in the first place.

It's not that I even agree or disagree with the engineering reasons that went into the decision -- but the way the process was done was (and still is) decidedly broken.

If perl wants to move forward, the whole community needs to be more responsible about intro'ing changes in major releases, with deprecation announcements and compatibility options if at all possible (there may be some that are impossible, I dunno, but this wasn't one of them).

This is a severe bug in the way *releases are handled*, -- i.e. I'm not so dillusional to think that anyone would follow the major specifications as described in the major releases' release notes.

The release note process ("What's New") **SHOULD** be vetted for missing references to major UI changes that were slipped in during "minor releases" so no one else will find these changes "suprising" years later.

I may be the only one on this list that this is news to, but I doubt I'm the only perl user in the world who has missed 'inconsequential, minor'[sic] changes like this.

This may only be a call for changing the process forward to either roll all **major changes** into the next V's what's new (non-minor changes that were done
in 'minor' releases, that wouldn't have been seen for those looking through major release notes to see what major changes have been wrought.

Yeah, this may look like a rant. But if you think what I'm saying has no merit, you are part of the problem that is killing perl.





[Please do not change anything below this line]
-----------------------------------------------------------------
---
Flags:
category=core
severity=high
---
This perlbug was built using Perl 5.14.2 - Sat Oct 29 15:59:10 UTC 2011
It is being executed now by Perl 5.14.2 - Sat Oct 29 15:55:30 UTC 2011.

Site configuration information for perl 5.14.2:

Configured by abuild at Sat Oct 29 15:55:30 UTC 2011.

Summary of my perl5 (revision 5 version 14 subversion 2) configuration:

Platform:
osname=linux, osvers=3.1.0-rc10-1-default, archname=x86_64-linux-thread-multi
uname='linux build33 3.1.0-rc10-1-default #1 smp thu oct 20 08:17:26 utc 2011 (f6d77d4) x86_64 x86_64 x86_64 gnulinux '
config_args='-ds -e -Dprefix=/usr -Dvendorprefix=/usr -Dinstallusrbinperl -Dusethreads -Di_db -Di_dbm -Di_ndbm -Di_gdbm -Duseshrplib=true -Doptimize=-fmessage-length=0 -O2 -Wall -D_FORTIFY_SOURCE=2 -fstack-protector -funwind-tables -fasynchronous-unwind-tables -g -Wall -pipe -Accflags=-DPERL_USE_SAFE_PUTENV -Dotherlibdirs=/usr/lib/perl5/site_perl'
hint=recommended, useposix=true, d_sigaction=define
useithreads=define, usemultiplicity=define
useperlio=define, d_sfio=undef, uselargefiles=define, usesocks=undef
use64bitint=define, use64bitall=define, uselongdouble=undef
usemymalloc=n, bincompat5005=undef
Compiler:
cc='cc', ccflags ='-D_REENTRANT -D_GNU_SOURCE -DPERL_USE_SAFE_PUTENV -fno-strict-aliasing -pipe -fstack-protector -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64',
optimize='-fmessage-length=0 -O2 -Wall -D_FORTIFY_SOURCE=2 -fstack-protector -funwind-tables -fasynchronous-unwind-tables -g -Wall -pipe',
cppflags='-D_REENTRANT -D_GNU_SOURCE -DPERL_USE_SAFE_PUTENV -fno-strict-aliasing -pipe -fstack-protector'
ccversion='', gccversion='4.6.2', gccosandvers=''
intsize=4, longsize=8, ptrsize=8, doublesize=8, byteorder=12345678
d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=16
ivtype='long', ivsize=8, nvtype='double', nvsize=8, Off_t='off_t', lseeksize=8
alignbytes=8, prototype=define
Linker and Libraries:
ld='cc', ldflags =' -L/usr/local/lib64 -fstack-protector'
libpth=/lib64 /usr/lib64 /usr/local/lib64
libs=-lm -ldl -lcrypt -lpthread
perllibs=-lm -ldl -lcrypt -lpthread
libc=/lib64/libc-2.14.1.so, so=so, useshrplib=true, libperl=libperl.so
gnulibc_version='2.14.1'
Dynamic Linking:
dlsrc=dl_dlopen.xs, dlext=so, d_dlsymun=undef, ccdlflags='-Wl,-E -Wl,-rpath,/usr/lib/perl5/5.14.2/x86_64-linux-thread-multi/CORE'
cccdlflags='-fPIC', lddlflags='-shared -L/usr/local/lib64 -fstack-protector'

Locally applied patches:


---
@INC for perl 5.14.2:
/usr/lib/perl5/site_perl/5.14.2/x86_64-linux-thread-multi
/usr/lib/perl5/site_perl/5.14.2
/usr/lib/perl5/vendor_perl/5.14.2/x86_64-linux-thread-multi
/usr/lib/perl5/vendor_perl/5.14.2
/usr/lib/perl5/5.14.2/x86_64-linux-thread-multi
/usr/lib/perl5/5.14.2
/usr/lib/perl5/site_perl/5.14.2/x86_64-linux-thread-multi
/usr/lib/perl5/site_perl/5.14.2
/usr/lib/perl5/site_perl
.

---
Environment for perl 5.14.2:
HOME=/home/law
LANG=en_US.UTF-8
LANGUAGE (unset)
LC_COLLATE=C
LC_CTYPE=en_US.UTF-8
LD_LIBRARY_PATH (unset)
LOGDIR (unset)
PATH=.:/sbin:/usr/local/sbin:/opt/lsb/bin:/home/law/bin:/usr/local/bin:/usr/bin:/bin:/usr/X11R6/bin:/usr/games:/opt/kde3/bin:/usr/lib/mit/bin:/usr/lib/mit/sbin:/usr/lib/qt3/bin:/usr/sbin:/etc/local/func_lib:/home/law/lib:/home/law/bin/lib
PERL_BADLANG (unset)
SHELL=/bin/bash
Re: [perl #112792] Perl Major Release Doc Flaws (5.8.0->5.10.0) [ In reply to ]
On Sun, May 06, 2012 at 12:35:15PM -0700, Linda Walsh wrote:
> This is a severe bug in the way *releases are handled*, -- i.e. I'm
> not so dillusional to think that anyone would follow the major
> specifications as described in the major releases' release notes.
>
> The release note process ("What's New") **SHOULD** be vetted for
> missing references to major UI changes that were slipped in during
> "minor releases" so no one else will find these changes "suprising"
> years later.
>
> I may be the only one on this list that this is news to, but I doubt
> I'm the only perl user in the world who has missed 'inconsequential,
> minor'[sic] changes like this.
>
> This may only be a call for changing the process forward to either
> roll all **major changes** into the next V's what's new (non-minor
> changes that were done in 'minor' releases, that wouldn't have been
> seen for those looking through major release notes to see what major
> changes have been wrought.

It's true that in the past, the way minor releases have been done has
not been very clear or transparent. This process, however, has been
fixed since at least 5.12 (possibly even 5.10). Introducing major
changes in the 5.8.0 -> 5.8.1 transition was probably a mistake, but
there's not really anything to be done about that now, and the relevant
process has been fixed since then.

-doy
Re: [perl #112792] Perl Major Release Doc Flaws (5.8.0->5.10.0) [ In reply to ]
On Sun, May 06, 2012 at 12:35:15PM -0700, Linda Walsh wrote:
> Yeah, this may look like a rant.

Yes, its an unhelpful rant.

I read about halfway through before giving up, being unable to determine
what it was you were complaining about having changed without sufficient
documentation.

Could you please provide a succinct description, in a few lines, about
what functionally changed in which release, that you consider either
bad or under-documented.

--
Indeed sir, I certainly have the capability to boogie. But -- and I wish
to make this absolutely clear -- only if you sing a certain song.
[perl #112792] Perl Major Release Doc Flaws (5.8.0->5.10.0) [ In reply to ]
On Sun May 06 12:53:34 2012, davem wrote:
> On Sun, May 06, 2012 at 12:35:15PM -0700, Linda Walsh wrote:
> > Yeah, this may look like a rant.
>
> Yes, its an unhelpful rant.

Hey, at least she's consistent.




---
via perlbug: queue: perl5 status: open
https://rt.perl.org:443/rt3/Ticket/Display.html?id=112792
Re: [perl #112792] Perl Major Release Doc Flaws (5.8.0->5.10.0) [ In reply to ]
Jesse Luehrs via RT wrote:
> On Sun, May 06, 2012 at 12:35:15PM -0700, Linda Walsh wrote:
>
>> This is a severe bug in the way *releases are handled*....
>> The release note process ("What's New") **SHOULD** be vetted for
>> missing references to major UI changes...
>> .. a call for **changing the process forward** to either
>> roll all **major changes** into the next V's what's new (non-minor
>> changes that were done in 'minor' releases, that wouldn't have been
>> seen for those looking through major release notes to see what major
>> changes have been wrought.
>>
> ---
> It's true that in the past, the way minor releases have been done has
> not been very clear or transparent. This process, however, has been
> fixed since at least 5.12 (possibly even 5.10). Introducing major
> changes in the 5.8.0 -> 5.8.1 transition was probably a mistake, but
> there's not really anything to be done about that now, and the relevant
> process has been fixed since then.
> ---
>
How was it fixed? I'm not saying it wasn't, but what is being done
differently now that wasn't before? I'm glad some people had the sense
to address this.
Some of us miss a 1-time important announcement and wondering why things
don't
work they way they were documented to for the next 5 years. >;^>

Sidenote:
I'm glad at least some people in this project have a >5th grade
reading level. I know my prose is extended, and I know, that
sometimes, it could be shorter, but many times, I can't express
what I want to say, with the details I feel are needed, in Reader's
Digest form.

I have to write for a general audience -- many of who will look at
the technical details of what I am saying, and if I don't cross as
many t's and dot as many i's as possible, I'm sure to get
misunderstandings or more questions. Thus I try to aim at enough
detail to answer questions that might arise (and end giving details
that, for most people never enter their mind). But it doesn't mean
it wasn't in someone's mind... thoroughness is rarely appreciated
as when it matches perfectly, it's not noticed, if it is overdone,
it's though to be needless verbiage, and if it is not enough, then
some wonder why you didn't
give more information in the first pace.

Is there a webpage of these processes now where I can go read about
current procedures?






>
>
Re: [perl #112792] Perl Major Release Doc Flaws (5.8.0->5.10.0) [ In reply to ]
On Sun, May 06, 2012 at 03:40:28PM -0700, Linda Walsh wrote:
> Is there a webpage of these processes now where I can go read about
> current procedures?

perldoc perlpolicy

-doy
Re: [perl #112792] Perl Major Release Doc Flaws (5.8.0->5.10.0) [ In reply to ]
> How was it fixed?

We didn't do it again.

Additionally, this was a doc flaw between 5.8.0 and 5.8.1. It was, I'll
entirely admit, a crappy mistake, but it was one made by entirely different
people than the people currently in charge of the release process.

Therefore, filing this bug against 5.14.2 is incredibly unkind and unfair
to the people doing their best to avoid such a problem in future, and
factually incorrect as well.

The fact that people actually tried to respond in spite of your accusation
that anybody who didn't consider your incompetent and offensive bug report
to be valid was "part of the problem that is killing perl" was, frankly,
stunningly impressive to me.

> I'm glad at least some people in this project have a >5th grade reading
> level.

Most of us do. That's why we find it so disappointing that you appear not
to have such a writing level. I had to read your original report twice
to determine the gist of it; while I appreciate that you need to supply
sufficient details that people utterly incapable of useful participation
in this process don't ask you stupid questions afterwards, I'd much prefer
it if you'd keep such clarification out of the ticket.

If your audience is so stupid they're unable to follow a link to a clarifying
web page, then I would suggest that perhaps you attempted to present a short,
coherent first paragraph that explains the crux of your complaint to those
of us who are likely to have the capacity to help resolve it, and then only
after that provide the dozen clarifying paragraphs for those who are
ignorant of the required context (or simply hard of thinking).

But, short form: perl 5.8.0 was a disaster. The people involved at the time
did their best to fix it for 5.8.1; they were not entirely successful.

This is unfortunate.

However, what already *did* nearly kill perl was a failure to release at all
and an atrophying of the perl5-porters community directly caused by excess
negativity both on-list and from the world around us. That's now gradually
improving, a process I've not been directly part of but have done my best
to support indirectly through shifting furniture out of the way for the
people getting things done.

Filing such a horribly negative bug criticising something that was a many
years old mistake, with a massively entitled attitude and an implication that
all of the people that are trying to make sure this doesn't happen again
are "killing perl" is to turn reality on its head.

These are the people who have rescued perl from a slow death.

These are the people who are making every effort to ensure that a perl5
major version release is correctly documented, stable, well tested, and
as such does not contain the sort of disastrous structural problem that
caused you so much pain between 5.8.0 and 5.8.1.

Your brand of unconstructive, overly verbose, entitled bullshit is unfair,
unkind, and fundamentally useless to the important goal of moving perl
forwards, and doing so in a way that allows us to retain, nurture and
expand the team that have done such good work over the past few years.

I strongly suggest that you stop and think about how your words are
coming across to these ensthusiastic, unpaid volunteers.

If you have the slightest shred of empathy and a functioning theory of
mind, I hope that by considering things from their point of view you'll
be able to realise why you've been getting less than ideal reactions.

Once you have realised this, I'd suggest that stepping back, apologising
for your appalling behaviour to date, and attempting to interact with these
people as fellow lovers of perl and fellow human beings in future with at
least some trivial level of respect would be your best path forwards.

If you can do that, you'll be far more likely to get the sort of responses
you were originally looking for, and perhaps your comments can lead to some
actual positive results for perl - and I choose to believe that you do
genuinely care about that.

Sadly, your current attitude achieves nothing except for burning huge amounts
of unpaid volunteer time spelunking through the layers of verbiage to
discover that your only real point is "fuck you, 5.8.1 was bad so I'm going
to throw a tantrum at all the people trying to make 5.16+ good" and that
instead of a useful contribution, they've just wasted time to achieve nothing
except get an undeserved slap in the face that's likely to substantially
effect their future motivation.

So, in summary of my own verbiage:

Stop.

Step back.

Try and act like an adult human being.

It's amazing how much better the response you get that way tend to be.

--
Matt S Trout - Shadowcat Systems - Perl consulting with a commit bit and a clue

http://shadowcat.co.uk/blog/matt-s-trout/ http://twitter.com/shadowcat_mst/

Email me now on mst (at) shadowcat.co.uk and let's chat about how our Catalyst
commercial support, training and consultancy packages could help your team.
Re: [perl #112792] Perl Major Release Doc Flaws (5.8.0->5.10.0) [ In reply to ]
Matt S Trout via RT wrote:
> We didn't do it again.[sic]---
>
Oh but you did. You acted like an over-the top ass toward someone trying to
document problems as they find them -- this one I as a bit late on. BUT
so were
alot of ther people -- because for YEARS after 5.8, people told me all I
had to do was to have my environment vars set for UTF-8 and perl would
respect that.

So that misinformation was out there for long time. Now, multiple years
later,
I'm getting a different story. It wasn't just the crew of 5.8 / 5.8.1
that messed up. 5.10.0 was the biggest miss. That's where it should
have been well publicized. An opportunity to address it in release
notes in 5.12/5.14 happened, but other incompatibilities were being
introduced there.

Your who over-the-top rant seems to be based on an inability for you to
demonstrate basic reading skill and you apparent unfamiliarity with the
bug system.

The subject said the bug occurred in 5.8->5.10. It didnt' say it
happened in
5.14 -- I never entered 5.14 -- that's done by perlbug and I can't
change the field (I tried)... that field is read only -- BUT if you were
really helping the project like you claim you are, you would know that
-- having filed bugs.


> that anybody who didn't consider your incompetent and offensive bug report
> to be valid was "part of the problem that is killing perl" was, frankly,
> stunningly impressive to me.
>
Another miracle of your inability to read. I never said they had to
consider the bug report valid. I said if they though what I was saying
had no merit, then they are throwing out the baby with the bathwater and
not considering differing views.
>
>> I'm glad at least some people in this project have a >5th grade reading
>> level.
>>
>
> Most of us do. That's why we find it so disappointing that you appear not
> to have such a writing level.

Sorry, I it was a fast write - I tried to dumb it down as best I could, but
from having it justed by a writing program, I couldn't apparently get it
down below a 14th grade reading level... Sorry about that, I know it was
hard for you.


> I had to read your original report twice.
>
I some times have to read dense material 3-5 times. That you only read
it twice -- you must be a bit smarter than average.


> Your brand of unconstructive, overly verbose, entitled bullshit is unfair,
> unkind,
----
Unkind? It's accurate and aimed at the release it happened in --
though I don't think it would be out-of-line to to spread the blaim a
abit since the problem has got worse with causing previous gen programs
to fail.

> I strongly suggest that you stop and think about how your words are
> coming across to these ensthusiastic, unpaid volunteers.
>
----
The intelligent once would realize it wasn't directed at them but at
things that occured in the past and that (I might be a bit miffed
because of all the impompats I've had to deal with of late.
> If you have the slightest shred of empathy
Now that IS a laugh. **if** I **had** the slightest sense -- it most
certainly would have been completely shredded by your gentle display of
empathy.. Great job in representing the perl community. Is this a paid
position for you?

tend to respond to people in the say they respond to me. You say you
represent the perl community as their 'protector'...fine -- so they way
you act toward me is how you want me to treat others? Why do I think
that might cause more problems than anything else? I usually find
acting like a troll isn't the most helpful way to reform someone into
something you want done that is positive. Attempting to tear them down
often hs
> and a functioning theory of
> mind, I hope that by considering things from their point of view you'll
> be able to realise why you've been getting less than ideal reactions.
>
----
You haven't known me for long, apparently, since no matter how I
approach a matter, I get similar reactions, so there is little incentive
to change -- not that I haven't been making suggestions/bug reports for
the pest 10+ years.
> Once you have realised
What a fool you are, I can completely ignore the rest if what you said,
as you are more guilty than I for jumping on me about something I have
no input into
and cannot change. You are just a time sink.


> If you can do that, you'll be far more likely to get the sort of responses
> you were originally looking for, and perhaps your comments can lead to some
> actual positive results for perl - and I choose to believe that you do
> genuinely care about that.
>
----
I will take each issue on it's merits as I would expect any other
professional to do. To do otherwise is not only unprofessional but
harmful to perl as a while.


> Sadly, your current attitude
is the result of the same types of mistakes burning 5-10 years of scripts.

Maye if that stopped happening I could make *positive* progress rather
than always
having to to back and redo old code just to make it work again.


> So, in summary of your own verbiage
>
Stop trolling and going off on people over things they cannot control.
> Try and act like an adult human being.
>
Human being as animals... most of whom don't act well.


> It's amazing how much better the response you get that way tend to be
>
---
What amzing is the many ways I've tried, and have it make no difference.

now since this whole thing started by you not knowing what version i
filed this against (in the title) -- and the fact that the other field
is a read-only
field for submitters.

Maybe you should yell at those who create the data from the forms?
Re: [perl #112792] Perl Major Release Doc Flaws (5.8.0->5.10.0) [ In reply to ]
On Thu, May 10, 2012 at 06:22:25PM -0700, Linda W wrote:
> Oh but you did. You acted like an over-the top ass toward someone trying to
> document problems as they find them -- this one I as a bit late on.
> BUT so were
> alot of ther people -- because for YEARS after 5.8, people told me
> all I had to do was to have my environment vars set for UTF-8 and
> perl would respect that.
>
> So that misinformation was out there for long time. Now, multiple
> years later,
> I'm getting a different story. It wasn't just the crew of 5.8 /
> 5.8.1 that messed up. 5.10.0 was the biggest miss. That's where it
> should have been well publicized. An opportunity to address it in
> release notes in 5.12/5.14 happened, but other incompatibilities
> were being introduced there.

In 5.8.0, a new feature was introduced, whereby a utf8 locale set in the
environment automatically made the STD filehandles utf8. This seemed like
a sensible idea at the time; in practice, it broke a lot of code, and was
quickly removed for 5.8.1, and was *clearly documented* in the 5.8.1
perldelta as the second item in 'Incompatible Changes'.

Due to an oversight, that announcement wasn't copied into the 5.10.0
delta too. Sorry about that that; although we did manage to get 1600 lines
of other stuff into that perldelta, including 20 incompatibility
announcements; so we do take seriously getting such documentation right.

But note that this mis-feature wasn't in 5.8.1, 5.8.2, .. 5.8.9, or for
any of 5.10.x, 5.12.x or 5.14.x; thus the window of its existence was
quite small (especially as people tend to skip the .0 release and wait for
the .1).

Note that in general, adding unicode support is *very* hard to do; around
the time of 5.6.x and 5.8.x no one really know the best way to make it
work, and some of the early ideas turned out to poor. Since then we have
been fixing some of these implementation choices, and steering a
difficult course between fixing things that are clearly wrong, and trying
not to break backwards compatibility; often there is much debate before
such changes.

It regarded by some that perl currently has amongst the most complete and
correct unicode support of comparable programming languages.

--
The Enterprise successfully ferries an alien VIP from one place to another
without serious incident.
-- Things That Never Happen in "Star Trek" #7
Re: [perl #112792] Perl Major Release Doc Flaws (5.8.0->5.10.0) [ In reply to ]
On Fri, May 11, 2012 at 11:47:16AM +0100, Dave Mitchell wrote:
> On Thu, May 10, 2012 at 06:22:25PM -0700, Linda W wrote:
> > Oh but you did. You acted like an over-the top ass toward someone trying to
> > document problems as they find them -- this one I as a bit late on.
> > BUT so were
> > alot of ther people -- because for YEARS after 5.8, people told me
> > all I had to do was to have my environment vars set for UTF-8 and
> > perl would respect that.
> >
> > So that misinformation was out there for long time. Now, multiple
> > years later,
> > I'm getting a different story. It wasn't just the crew of 5.8 /
> > 5.8.1 that messed up. 5.10.0 was the biggest miss. That's where it
> > should have been well publicized. An opportunity to address it in
> > release notes in 5.12/5.14 happened, but other incompatibilities
> > were being introduced there.
>
> In 5.8.0, a new feature was introduced, whereby a utf8 locale set in the
> environment automatically made the STD filehandles utf8. This seemed like
> a sensible idea at the time; in practice, it broke a lot of code, and was
> quickly removed for 5.8.1, and was *clearly documented* in the 5.8.1
> perldelta as the second item in 'Incompatible Changes'.

And if the people offering advice were not aware of that, they weren't
very competent to be offering advice in the first place. The problem is
that too many people over-estimate their abilities, and it's hard to tell
the over-confident (bluffer or genuine) from the genuinely competent.
Particularly as the genuinely competent are aware of their limitations,
and as a result generally downplay their confidence.

> Note that in general, adding unicode support is *very* hard to do; around
> the time of 5.6.x and 5.8.x no one really know the best way to make it
> work, and some of the early ideas turned out to poor. Since then we have
> been fixing some of these implementation choices, and steering a
> difficult course between fixing things that are clearly wrong, and trying
> not to break backwards compatibility; often there is much debate before
> such changes.

Also, IIRC, the idea of enabling things based on a UTF-8 locale was what the
Linux/Unicode folks at the time said was the right thing to do. We were
following the best advice we had, which turned out to be horribly broken in
practice. In particular, some distributions changed to defaulting installs
to give the user a UTF-8 locale, without the user really being aware of
what was being thrust upon them, and even though the rest of the programs
and systems the user was interacting with were unchanged and assumed the
same 8-bit environment as before.

(This stuff is *still* screwed up. Ex-work upgraded a system from really-old
Linux to modern-ish Linux about 18 months ago and something broke. Turned
out that a program written in *Java*, of all things, was "honouring" the
locale setting by using UTF-8 down a network socket that it opened if the
locale was UTF-8. "using UTF-8" in the sense of converting all octets to
UTF-8 on the way out. This is despite the fact that this broke the
content-length and content-type headers that the self same code had just sent
down the same socket. The fix - LC_ALL="C")

(Note also the fun Python 3.0 had by assuming that a UTF-8 locale means that
all aspects of the environment are reliably UTF-8, and not coping well when
that was not true. They fixed Python 3.1 to cope with messy realities)

And that's not the only example of Unicode theory proving to be badly wrong
in practice, when implemented. The Unicode Consortium themselves are having
to rethink their rules on matching inverted character classes, given that
this

/[^ß]/i

doesn't match what people expect, and this

"ß" =~ /(s)(s)/i

is positively unimplementable.

> it regarded by some that perl currently has amongst the most complete and
> correct unicode support of comparable programming languages.

Which is also why we have some of the pain of an early adopter.

Nicholas Clark
Re: [perl #112792] Perl Major Release Doc Flaws (5.8.0->5.10.0) [ In reply to ]
On 2012-05-11, Nicholas Clark <nick@ccl4.org> wrote:
> /[^ß]/i
>
> doesn't match what people expect, and this
>
> "ß" =~ /(s)(s)/i
>
> is positively unimplementable.
>
This has nothing to do with Unicode. You have the same problem in
ISO-8859-1 too.

-- Petr
Re: [perl #112792] Perl Major Release Doc Flaws (5.8.0->5.10.0) [ In reply to ]
On Fri, May 11, 2012 at 1:44 PM, Petr Pisar <ppisar@redhat.com> wrote:
> On 2012-05-11, Nicholas Clark <nick@ccl4.org> wrote:
>>     /[^ß]/i
>>
>> doesn't match what people expect, and this
>>
>>    "ß" =~ /(s)(s)/i
>>
>> is positively unimplementable.
>>
> This has nothing to do with Unicode. You have the same problem in
> ISO-8859-1 too.
>

I don't think ISO ever recommended that "ß" should positively match
/ss/i, the Unicode consortium did.

Leon
Re: [perl #112792] Perl Major Release Doc Flaws (5.8.0->5.10.0) [ In reply to ]
Dave Mitchell wrote:
> In 5.8.0, a new feature was introduced, whereby a utf8 locale set in the
> environment automatically made the STD filehandles utf8. This seemed like
> a sensible idea at the time; in practice, it broke a lot of code, and was
> quickly removed for 5.8.1, and was *clearly documented* in the 5.8.1
> perldelta as the second item in 'Incompatible Changes'.
>
----
Alot of people missed that. When I coudn't long after 5.8.1,
figure out why my prog was no longer working -- others would ask me --
are you sure you have the ENV vars set in the environment -- YEARS after
5.8.1. Only recently did I find out. If I was the only one who missed
the announcement, others would have quickly corrected my error, but many
people missed that, that's what is so frustrating!.... sigh.....
> But note that this mis-feature wasn't in 5.8.1, 5.8.2, .. 5.8.9, or for
> any of 5.10.x, 5.12.x or 5.14.x; thus the window of its existence was
> quite small (especially as people tend to skip the .0 release and wait for
> the .1).
>
Really? That may be true of 50% of the people....but some like the
shiny new
5.8.0.... it had been a long time since there'd been a new release (but
not as long as until the 5.10...be we didn't now at the time).
> Note that in general, adding unicode support is *very* hard to do; around
> the time of 5.6.x and 5.8.x no one really know the best way to make it
> work, and some of the early ideas turned out to poor.
I very much relate to that. I misdesign many times before getting it
right.


> It regarded by some that perl currently has amongst the most complete and
> correct unicode support of comparable programming languages.
>

That's one of the most frustrating things... my public bragging about
perl - and how it was UTF-8 native in a UTF-8 env... what an idiot...
As you might tell,
I tend to be outspoken and perl was one of my hot topics...

Thus that among other incompats since then lead into my writing the
initial report -- I hope you understand, it's nothing against people
who had nothing to do with it. But the communication was awful.
Re: [perl #112792] Perl Major Release Doc Flaws (5.8.0->5.10.0) [ In reply to ]
Nicholas Clark wrote:
>
> And if the people offering advice were not aware of that, they weren't
> very competent to be offering advice in the first place. The problem is
> that too many people over-estimate their abilities, and it's hard to tell
> the over-confident (bluffer or genuine) from the genuinely competent.
> Particularly as the genuinely competent are aware of their limitations,
> and as a result generally downplay their confidence.
---
*bingo*. Why do you think I've almost never contributed code in
projects.
I don't want errors out there. I don't want to be responsible for code
getting into someone's hands that doesn't work -- or worse, that I can't
remotely figure out quickly how to fix. Personal nightmare. The fact
that software goes out today -- shipped as product that is alpha quality
of 10-20 years ago, is astounding.

Anything that I've written that might have been good enough to
'ship' was so special purpose to not be generally useful enough for
CPAN. But it's narrow purpose was a large part of my feeling good about
how well it did its job.
Sigh.
Re: [perl #112792] Perl Major Release Doc Flaws (5.8.0->5.10.0) [ In reply to ]
On 05/11/2012 05:54 AM, Leon Timmermans wrote:
> On Fri, May 11, 2012 at 1:44 PM, Petr Pisar<ppisar@redhat.com> wrote:
>> On 2012-05-11, Nicholas Clark<nick@ccl4.org> wrote:
>>> /[^ß]/i
>>>
>>> doesn't match what people expect, and this
>>>
>>> "ß" =~ /(s)(s)/i
>>>
>>> is positively unimplementable.
>>>
>> This has nothing to do with Unicode. You have the same problem in
>> ISO-8859-1 too.
>>
>
> I don't think ISO ever recommended that "ß" should positively match
> /ss/i, the Unicode consortium did.
>
> Leon
>

ISO-8859-1 does not define any folds, much less multi-character ones.
Unicode did not have fold case until Version 3.

Perl does case insensitive matching under locale (including 8859 ones)
by using lowercase instead. ß is already lower-cased, so the problem
doesn't occur in ISO-8859-1. But that means it doesn't match SS
case-insensitively either, which is an example of why Unicode introduced
fold case.