Mailing List Archive

On Mon, 28 Feb 2011, Daniel Roesen wrote:

> Seriously, I think we operators would be well advised to help reviewing
> and polish up the Happy Eyeballs spec and - if deemed feasible - promote
> implementation to OS/Distro/App vendors ASAP.
>
> http://tools.ietf.org/html/draft-wing-v6ops-happy-eyeballs-ipv6-01
>
> Happy Eyeballs ain't unproblematic. Opportunistic session setup will
> have impact on - if the IPv6 transport of a dual-stacked target host
> works fine, the opportunistic IPv4 session setups will "unnecessarily"
> (as no data traffic transfer follows) clog up resources on LSNAT44
> and the target firewalls, loadbalancers and host (phantom sessions).
> See section 4.2.1

And, would you have noticed the IPv6 related bug if happy-eyeballs was
already implemented or would legacy IP have worked well enough for you
to not notice (read as - what's better?, getting the bug fixed or not
noticing and harming IPv6 for longer)?

/bz

--
Bjoern A. Zeeb You have to have visions!
Stop bit received. Insert coin for new address family.

On Tue, Mar 01, 2011 at 12:07:36AM +0000, Bjoern A. Zeeb wrote:
> And, would you have noticed the IPv6 related bug if happy-eyeballs was
> already implemented or would legacy IP have worked well enough for you
> to not notice (read as - what's better?, getting the bug fixed or not
> noticing and harming IPv6 for longer)?

IPv6 is not customer-driven, but provider driven. So #1 priority must be
that "things keep working" as painless as possible.

But you're right in the sense that with Happy Eyeballs we need methods
to measure problems being masked by HE. How, is one of the things
which seem to be missing in the Happy Eyeballs discussion.

Best regards,
Daniel

--
CLUE-RIPE -- Jabber: dr@cluenet.de -- dr@IRCnet -- PGP: 0xA85C8AA0

Daniel,

On Tue, Mar 1, 2011 at 1:34 AM, Daniel Roesen <dr@cluenet.de> wrote:
> On Tue, Mar 01, 2011 at 12:07:36AM +0000, Bjoern A. Zeeb wrote:
>> And, would you have noticed the IPv6 related bug if happy-eyeballs was
>> already implemented or would legacy IP have worked well enough for you
>> to not notice (read as - what's better?, getting the bug fixed or not
>> noticing and harming IPv6 for longer)?
>
> IPv6 is not customer-driven, but provider driven. So #1 priority must be
> that "things keep working" as painless as possible.
>
> But you're right in the sense that with Happy Eyeballs we need methods
> to measure problems being masked by HE. How, is one of the things
> which seem to be missing in the Happy Eyeballs discussion.

Totally agree. Like I told to Bjoern when we met in @fosdem a few
weeks ago - from the pure engineering point of view I think a good
thing that could happen is IPv4 would suddenly vanish from the face of
earth for 3-4 months. Then we notice all the problems and can fix them
(very fast ;-) (Un)fortunately this is not possible - as it would be a
major catastrophy from the user experience point of view.

Happy Eyeballs is a bit on the other side of the spectrum - by working
hard to make the UX as seamless as possible indeed it masks these
kinds of problems - so with it the chances are high that these
problems will not be noticed. Actually, even more so since the
opportunistic connection establishment that you mentioned in the first
mail might not even happy if the single protocol consistently wins (so
it is not 100% true about the increase in load).

We plan a bar bof @Prague, I will definitely bring this topic up there
too - meantime if you have ideas, feel free to write them up for the
discussion.

Side remark: I noticed this trend overall - the more robust you have a
protocol to external influences (soft failures instead of hard
failures), the "nicer" is the user experience, and the more hell is in
debugging of this protocol for the support/dev folks when the
experience slowly degrades to the point of being unacceptable. It's a
tough choice.

cheers,
andrew

> -----Original Message-----
> From: ipv6-ops-
> bounces+guillaume.leclanche=swisscom.com@lists.cluenet.de [mailto:ipv6-
> ops-bounces+guillaume.leclanche=swisscom.com@lists.cluenet.de] On
> Behalf Of Daniel Roesen
> Sent: Monday, February 28, 2011 11:34 PM
> To: ipv6-ops@lists.cluenet.de
> Subject: From the dualstack-is-fun department...


> DS-Website having one AAAA RR: ~3 minutes time to IPv4 fallback(!)

I analyzed this behavior, the reason is that the browser is calling the connect() API from the OS. This function will then try the complete TCP opening process, including TCP retries. Linux (at least debian flavour) will do 5 retries, with exponentially growing time between them.

You can play with the number of retries; I put it down to 2, and the fallback-to-ipv4 time was roughly 10s.

I opened a feature request on the kernel bug tracker to request the possibility to tune this parameter for dualstack websites, and put it by default to 2, but nothing happened afaik.

The request :
https://bugzilla.kernel.org/show_bug.cgi?id=23242

The sysctl control is tcp_syn_retries (sorry forgot the hierarchy, you'll have to grep :)

Guillaume

On Feb 28, 2011 10:41 PM, "Andrew Yourtchenko" <ayourtch@gmail.com> wrote:
>
> Daniel,
>
> On Tue, Mar 1, 2011 at 1:34 AM, Daniel Roesen <dr@cluenet.de> wrote:
> > On Tue, Mar 01, 2011 at 12:07:36AM +0000, Bjoern A. Zeeb wrote:
> >> And, would you have noticed the IPv6 related bug if happy-eyeballs was
> >> already implemented or would legacy IP have worked well enough for you
> >> to not notice (read as - what's better?, getting the bug fixed or not
> >> noticing and harming IPv6 for longer)?
> >
> > IPv6 is not customer-driven, but provider driven. So #1 priority must be
> > that "things keep working" as painless as possible.
> >
> > But you're right in the sense that with Happy Eyeballs we need methods
> > to measure problems being masked by HE. How, is one of the things
> > which seem to be missing in the Happy Eyeballs discussion.
>
> Totally agree. Like I told to Bjoern when we met in @fosdem a few
> weeks ago - from the pure engineering point of view I think a good
> thing that could happen is IPv4 would suddenly vanish from the face of
> earth for 3-4 months. Then we notice all the problems and can fix them
> (very fast ;-) (Un)fortunately this is not possible - as it would be a
> major catastrophy from the user experience point of view.
>
> Happy Eyeballs is a bit on the other side of the spectrum - by working
> hard to make the UX as seamless as possible indeed it masks these
> kinds of problems - so with it the chances are high that these
> problems will not be noticed. Actually, even more so since the
> opportunistic connection establishment that you mentioned in the first
> mail might not even happy if the single protocol consistently wins (so
> it is not 100% true about the increase in load).
>
> We plan a bar bof @Prague, I will definitely bring this topic up there
> too - meantime if you have ideas, feel free to write them up for the
> discussion.
>
> Side remark: I noticed this trend overall - the more robust you have a
> protocol to external influences (soft failures instead of hard
> failures), the "nicer" is the user experience, and the more hell is in
> debugging of this protocol for the support/dev folks when the
> experience slowly degrades to the point of being unacceptable. It's a
> tough choice.
>

This also creates the ugly situation where customer calls help desk saying
website x is down, support person tries to get to website x, and it works.
Help desk says, nope "works for me" and the broken ipv6 access or dare I say
ipv4 access is broken to the none-HE user but works for the HE user. If the
none he-user cannot easily convince others that there is a problem, that is
bad.

This is a support nightmare as HE masks the issue and will not be uniformly
deployed -- ever.

This is a classic dilemma. Masking the problem ostensibly makes it go away,
but at the same time exacerbates the ability to resolve it. It is kind of
like beer :) and beer is good, especially when I been troubleshooting
connectivity issues all day and my customers keeping telling me websites
are down
... but not all of them ... they all but works for me ....

Cb
> cheers,
> andrew

Well, there's always the much-hated whitelisting approach...

Hi Guillaume,

On Tue, 1 Mar 2011 08:16:53 +0100
<Guillaume.Leclanche@swisscom.com> wrote:

> > -----Original Message-----
> > From: ipv6-ops-
> > bounces+guillaume.leclanche=swisscom.com@lists.cluenet.de [mailto:ipv6-
> > ops-bounces+guillaume.leclanche=swisscom.com@lists.cluenet.de] On
> > Behalf Of Daniel Roesen
> > Sent: Monday, February 28, 2011 11:34 PM
> > To: ipv6-ops@lists.cluenet.de
> > Subject: From the dualstack-is-fun department...
>
>
> > DS-Website having one AAAA RR: ~3 minutes time to IPv4 fallback(!)
>
> I analyzed this behavior, the reason is that the browser is calling the connect() API from the OS. This function will then try the complete TCP opening process, including TCP retries. Linux (at least debian flavour) will do 5 retries, with exponentially growing time between them.
>
> You can play with the number of retries; I put it down to 2, and the fallback-to-ipv4 time was roughly 10s.
>
> I opened a feature request on the kernel bug tracker to request the possibility to tune this parameter for dualstack websites, and put it by default to 2, but nothing happened afaik.
>
> The request :
> https://bugzilla.kernel.org/show_bug.cgi?id=23242
>
> The sysctl control is tcp_syn_retries (sorry forgot the hierarchy, you'll have to grep :)
>

Are you using a Fritzbox 7390? When you enable IPv6, they enable ULAs
by default (although their ULAs aren't (near-)unique because the unique
id is all zeros!), and don't issue ICMPv6 Destination Unreachables then
they don't have an IPv6 default route. So with an IPv4 only WAN link,
this problem occurs on all dual-stack websites. According to RFC4443,
they SHOULD be issuing ICMPv6 Destination Unreachables, No route to
destination.

Regards,
Mark.

Hi,

On Tue, Mar 01, 2011 at 04:32:49PM +0900, Erik Kline wrote:
> Well, there's always the much-hated whitelisting approach...

Which would not have helped Daniel at all... he has working v6, he
has a whitelisted provider, but "something in his personal setup is
not working"

(I assume that the underlying problem here is "bridging from a VM to a
wifi network", since wifi is bad at the "use a different MAC for the VM
thingie" - but it could be any number of things breaking locally, even
if you're nicely whitelisted)

Gert Doering
-- NetMaster
--
did you enable IPv6 on something today...?

SpaceNet AG Vorstand: Sebastian v. Bomhard
Joseph-Dollinger-Bogen 14 Aufsichtsratsvors.: A. Grundner-Culemann
D-80807 Muenchen HRB: 136055 (AG Muenchen)
Tel: +49 (89) 32356-444 USt-IdNr.: DE813185279

On Tue, Mar 1, 2011 at 8:31 AM, Cameron Byrne <cb.list6@gmail.com> wrote:
>
> On Feb 28, 2011 10:41 PM, "Andrew Yourtchenko" <ayourtch@gmail.com> wrote:
>>
>> Daniel,
>>
>> On Tue, Mar 1, 2011 at 1:34 AM, Daniel Roesen <dr@cluenet.de> wrote:
>> > On Tue, Mar 01, 2011 at 12:07:36AM +0000, Bjoern A. Zeeb wrote:
>> >> And, would you have noticed the IPv6 related bug if happy-eyeballs was
>> >> already implemented or would legacy IP have worked well enough for you
>> >> to not notice (read as - what's better?, getting the bug fixed or not
>> >> noticing and harming IPv6 for longer)?
>> >
>> > IPv6 is not customer-driven, but provider driven. So #1 priority must be
>> > that "things keep working" as painless as possible.
>> >
>> > But you're right in the sense that with Happy Eyeballs we need methods
>> > to measure problems being masked by HE. How, is one of the things
>> > which seem to be missing in the Happy Eyeballs discussion.
>>
>> Totally agree. Like I told to Bjoern when we met in @fosdem a few
>> weeks ago - from the pure engineering point of view I think a good
>> thing that could happen is IPv4 would suddenly vanish from the face of
>> earth for 3-4 months. Then we notice all the problems and can fix them
>> (very fast ;-) (Un)fortunately this is not possible - as it would be a
>> major catastrophy from the user experience point of view.
>>
>> Happy Eyeballs is a bit on the other side of the spectrum - by working
>> hard to make the UX as seamless as possible indeed it masks these
>> kinds of problems - so with it the chances are high that these
>> problems will not be noticed. Actually, even more so since the
>> opportunistic connection establishment that you mentioned in the first
>> mail might not even happy if the single protocol consistently wins (so
>> it is not 100% true about the increase in load).
>>
>> We plan a bar bof @Prague, I will definitely bring this topic up there
>> too - meantime if you have ideas, feel free to write them up for the
>> discussion.
>>
>> Side remark: I noticed this trend overall - the more robust you have a
>> protocol to external influences (soft failures instead of hard
>> failures), the "nicer" is the user experience, and the more hell is in
>> debugging of this protocol for the support/dev folks when the
>> experience slowly degrades to the point of being unacceptable. It's a
>> tough choice.
>>
>
> This also creates the ugly situation where customer calls help desk saying
> website x is down, support person tries to get to website x, and it works.
> Help desk says, nope "works for me" and the broken ipv6 access or dare I say
> ipv4 access is broken to the none-HE user but works for the HE user. If the
> none he-user cannot easily convince others that there is a problem, that is
> bad.

Yes, we already have in the latest text:

"Debugging and Troubleshooting

This mechanism is aimed to help the user experience in case of connectivity
problems. However, this precise reason also makes it tougher to use these
applications as a means of the verification that the problems are fixed. To
assist in that regard, the applications implementing the proposal in this
document SHOULD also provide a mechanism to temporarily use only one
address family."

Too weak ? Wrong approach ?

>
> This is a support nightmare as HE masks the issue and will not be uniformly
> deployed -- ever.
>
> This is a classic dilemma. Masking the problem ostensibly makes it go away,
> but at the same time exacerbates the ability to resolve it. It is kind of
> like beer :)
> and beer is good, especially when I been troubleshooting
> connectivity issues all day and my customers keeping telling me  websites
> are down
> ... but not all of them ... they all but works for me ....

...and "no-one changed anything". (That's what everyone says for the
past 15 or so years, I keep asking just in case, to see how often it's
"we changed X and Y has broken". I can count the occasions on fingers
of one hand, vs. the ~mid-4-digits number of the other outcome. So
fundamentally nothing changes - it breaks by itself today too :-)

(but seriously: appreciate all of the comments. I thought the above
blurb about troubleshooting in the draft should be enough, but maybe
it is not too strongly worded.
Maybe there needs to be some way to flag the problem that has been
worked around. To whom ? How ? Is it supposed to be specifit to this
area or maybe should there be something generic ? I think I'm getting
on hyperbolic HE-tangent trajectory with these questions.)

cheers,
andrew

>
> Cb
>> cheers,
>> andrew
>

On Mon, 2011-02-28 at 23:31 -0800, Cameron Byrne wrote:
> This also creates the ugly situation where customer calls help desk
> saying website x is down, support person tries to get to website x,
> and it works. Help desk says, nope "works for me" and the broken ipv6
> access or dare I say ipv4 access is broken to the none-HE user but
> works for the HE user. If the none he-user cannot easily convince
> others that there is a problem, that is bad.

Cameron, the below seems ~obvious to me in the scenario you describe:
"[...]
HD: What browser are you using?
Cust: X
HD: Well I am using Y v. Z [with HE implementation], try installing Y.
You will need this one [to continue the debugging].
[...]"

A "possibility" help-desk side, however rough, "un-scalable" (ie,
ISP-expensive) and sub-optimal. Given the complete problem scope we are
facing I'm not sure what is going to be worse though:
13min latencies, bad customer experience and multiple support calls,
vs. having a procedure to track down the root cause that HE/disabling
ipv6 fixes. Support desks will need updated flow charts - "boo hoo" :)
("I encourage all my competitors to give crappy customer support...")
And should this become widespread enough, eventually news media will
start say that you have to use "Y version >= Z" to use the Internet...

The above doesn't strike me entirely bad in itself.
Pain's going to hit somewhere for sure and there is no realistic
one-stop solution for all problems, so we have to continue to expand the
band-aid kit.

HE connection success/fail statistics / logging and opt-in reporting
somewhere would certainly be a nice addition as well (name the button
"Make the Internet better..." or so :) )

Regards,
Martin

On Tue, Mar 1, 2011 at 3:22 AM, Gert Doering <gert@space.net> wrote:
> Hi,
>
> On Tue, Mar 01, 2011 at 04:32:49PM +0900, Erik Kline wrote:
>> Well, there's always the much-hated whitelisting approach...
>
> Which would not have helped Daniel at all...  he has working v6, he
> has a whitelisted provider, but "something in his personal setup is
> not working"
>
> (I assume that the underlying problem here is "bridging from a VM to a
> wifi network", since wifi is bad at the "use a different MAC for the VM
> thingie" - but it could be any number of things breaking locally, even
> if you're nicely whitelisted)
>

White-listing, as i understand it, is about containment to control the
negative impacts of those who do not know or care about this "ipv6"
you speak of. White-listing just allows the ISP and content provider
to make planful steps towards IPv6, vs Huge-Content-Player hit the
switch on IPv6 now my helpdesk is overloaded, and people use an
alternate huge content player.

On Tue, Mar 1, 2011 at 4:29 AM, Andrew Yourtchenko <ayourtch@gmail.com> wrote:
> On Tue, Mar 1, 2011 at 8:31 AM, Cameron Byrne <cb.list6@gmail.com> wrote:
>>
>> On Feb 28, 2011 10:41 PM, "Andrew Yourtchenko" <ayourtch@gmail.com> wrote:
>>>
>>> Daniel,
>>>
>>> On Tue, Mar 1, 2011 at 1:34 AM, Daniel Roesen <dr@cluenet.de> wrote:
>>> > On Tue, Mar 01, 2011 at 12:07:36AM +0000, Bjoern A. Zeeb wrote:
>>> >> And, would you have noticed the IPv6 related bug if happy-eyeballs was
>>> >> already implemented or would legacy IP have worked well enough for you
>>> >> to not notice (read as - what's better?, getting the bug fixed or not
>>> >> noticing and harming IPv6 for longer)?
>>> >
>>> > IPv6 is not customer-driven, but provider driven. So #1 priority must be
>>> > that "things keep working" as painless as possible.
>>> >
>>> > But you're right in the sense that with Happy Eyeballs we need methods
>>> > to measure problems being masked by HE. How, is one of the things
>>> > which seem to be missing in the Happy Eyeballs discussion.
>>>
>>> Totally agree. Like I told to Bjoern when we met in @fosdem a few
>>> weeks ago - from the pure engineering point of view I think a good
>>> thing that could happen is IPv4 would suddenly vanish from the face of
>>> earth for 3-4 months. Then we notice all the problems and can fix them
>>> (very fast ;-) (Un)fortunately this is not possible - as it would be a
>>> major catastrophy from the user experience point of view.
>>>
>>> Happy Eyeballs is a bit on the other side of the spectrum - by working
>>> hard to make the UX as seamless as possible indeed it masks these
>>> kinds of problems - so with it the chances are high that these
>>> problems will not be noticed. Actually, even more so since the
>>> opportunistic connection establishment that you mentioned in the first
>>> mail might not even happy if the single protocol consistently wins (so
>>> it is not 100% true about the increase in load).
>>>
>>> We plan a bar bof @Prague, I will definitely bring this topic up there
>>> too - meantime if you have ideas, feel free to write them up for the
>>> discussion.
>>>
>>> Side remark: I noticed this trend overall - the more robust you have a
>>> protocol to external influences (soft failures instead of hard
>>> failures), the "nicer" is the user experience, and the more hell is in
>>> debugging of this protocol for the support/dev folks when the
>>> experience slowly degrades to the point of being unacceptable. It's a
>>> tough choice.
>>>
>>
>> This also creates the ugly situation where customer calls help desk saying
>> website x is down, support person tries to get to website x, and it works.
>> Help desk says, nope "works for me" and the broken ipv6 access or dare I say
>> ipv4 access is broken to the none-HE user but works for the HE user. If the
>> none he-user cannot easily convince others that there is a problem, that is
>> bad.
>
> Yes, we already have in the latest text:
>
> "Debugging and Troubleshooting
>
> This mechanism is aimed to help the user experience in case of connectivity
> problems. However, this precise reason also makes it tougher to use these
> applications as a means of the verification that the problems are fixed. To
> assist in that regard, the applications implementing the proposal in this
> document SHOULD also provide a mechanism to temporarily use only one
> address family."
>
> Too weak ? Wrong approach ?
>

I don't there is anything that you could write in an IETF draft that
would make joe-six-pack understand or care about HE.

>>
>> This is a support nightmare as HE masks the issue and will not be uniformly
>> deployed -- ever.
>>
>> This is a classic dilemma. Masking the problem ostensibly makes it go away,
>> but at the same time exacerbates the ability to resolve it. It is kind of
>> like beer :)
>> and beer is good, especially when I been troubleshooting
>> connectivity issues all day and my customers keeping telling me  websites
>> are down
>> ... but not all of them ... they all but works for me ....
>
> ...and "no-one changed anything". (That's what everyone says for the

Most browsers i have update themselves, or windows update, or apt-get
update them and i generally don't care to know what the updating is
that happened... updates are good in my world. In some larger
corporate environments i know, IE6 is mandatory.

> past 15 or so years, I keep asking just in case, to see how often it's
> "we changed X and Y has broken". I can count the occasions on fingers
> of one hand, vs. the ~mid-4-digits number of the other outcome. So
> fundamentally nothing changes - it breaks by itself today too :-)

Yep. And software updates are just like that. They happen by
themselves, literally.

>
> (but seriously: appreciate all of the comments. I thought the above
> blurb about troubleshooting in the draft should be enough, but maybe
> it is not too strongly worded.

When was the last time your read a warning label on a beer bottle?

> Maybe there needs to be some way to flag the problem that has been
> worked around. To whom ? How ? Is it supposed to be specifit to this
> area or maybe should there be something generic ? I think I'm getting
> on hyperbolic HE-tangent trajectory with these questions.)
>

agreed.

> cheers,
> andrew
>
>>
>> Cb
>>> cheers,
>>> andrew
>>
>

On Tue, Mar 1, 2011 at 7:18 AM, Martin Millnert <martin@millnert.se> wrote:
> On Mon, 2011-02-28 at 23:31 -0800, Cameron Byrne wrote:
>> This also creates the ugly situation where customer calls help desk
>> saying website x is down, support person tries to get to website x,
>> and it works. Help desk says, nope "works for me" and the broken ipv6
>> access or dare I say ipv4 access is broken to the none-HE user but
>> works for the HE user. If the none he-user cannot easily convince
>> others that there is a problem, that is bad.
>
> Cameron, the below seems ~obvious to me in the scenario you describe:
> "[...]
> HD: What browser are you using?
> Cust: X
> HD: Well I am using Y v. Z [with HE implementation], try installing Y.
> You will need this one [to continue the debugging].
> [...]"
>

Even DHS/CERT has tried to get people to stop using IE because it is a
national security risk. Yet, people use IE and many corporations
require IE(6!) for many things ... furthermore, many people just like
browser X. I don't think this will be a very popular option. But, i
understand your logic, it just does not fit with .... the reality of
people and browsers IMHO.

> A "possibility" help-desk side, however rough, "un-scalable" (ie,
> ISP-expensive) and sub-optimal. Given the complete problem scope we are
> facing I'm not sure what is going to be worse though:
>  13min latencies, bad customer experience and multiple support calls,
> vs. having a procedure to track down the root cause that HE/disabling
> ipv6 fixes. Support desks will need updated flow charts - "boo hoo" :)

I think our flow chart ends with pulling the battery out of the phone
and putting it back in. And yes, every support call can be
reverse-engineered to help someone with a common problem with
methodical troubleshooting steps, but i do not believe that is how
helpdesks are run. The troubleshooting logic tree is kept pretty
narrow since there are too many rabbit holes to peak in.

> ("I encourage all my competitors to give crappy customer support...")
>  And should this become widespread enough, eventually news media will
> start say that you have to use "Y version >= Z" to use the Internet...
>

I don't think anyone can sell that logic to the VP of Customer Care.
As i said, even the USA Department of Homeland Security (DHS) cannot
get people or companies or even the federal government to stop using
IE6.

> The above doesn't strike me entirely bad in itself.
>  Pain's going to hit somewhere for sure and there is no realistic
> one-stop solution for all problems, so we have to continue to expand the
> band-aid kit.
>
Agreed. The low tolerance for pain is why we don't generally have
ipv6 deployed.

> HE connection success/fail statistics / logging and opt-in reporting
> somewhere would certainly be a nice addition as well (name the button
> "Make the Internet better..." or so :) )
>

Agreed. Maybe its a gadget/widget/app of some sort. My only point is
we need to be cautious about providing different views of the world.
Nobody says HE is a panacea, but it might be an addictive morphine
that we use to numb the pain instead of taking the medicine (fixing v6
instead of treating it like a 2nd class protocol).

Perhaps there should be some wording in the draft that HE is not to be
used by default. HE should be enabled manually by people who know
what it is, like a helpdesk tech trying to solve a customer's problem.
Turning something on by default, including software upgrades, can
results in things like the 6to4 fiasco. Well intentioned, as are all
the steps on the path to hell.

Cameron


> Regards,
> Martin
>
>

On Tue, Mar 1, 2011 at 4:49 PM, Cameron Byrne <cb.list6@gmail.com> wrote:
> On Tue, Mar 1, 2011 at 4:29 AM, Andrew Yourtchenko <ayourtch@gmail.com> wrote:
>> Yes, we already have in the latest text:
>>
>> "Debugging and Troubleshooting
>>
>> This mechanism is aimed to help the user experience in case of connectivity
>> problems. However, this precise reason also makes it tougher to use these
>> applications as a means of the verification that the problems are fixed. To
>> assist in that regard, the applications implementing the proposal in this
>> document SHOULD also provide a mechanism to temporarily use only one
>> address family."
>>
>> Too weak ? Wrong approach ?
>>
>
> I don't there is anything that you could write in an IETF draft that
> would make joe-six-pack understand or care about HE.

The joe-six-pack who is supporting the IE6 corporate person ?

That paragraph is for the folks who write the code, not for the support/user.

The support guy will simply know "to support customers with IE6, flip that
green button into 'off' position before following the script".

>>
>> ...and "no-one changed anything". (That's what everyone says for the
>
> Most browsers i have update themselves, or windows update, or apt-get
> update them and i generally don't care to know what the updating is
> that happened... updates are good in my world.  In some larger
> corporate environments i know, IE6 is mandatory.

I suppose that given the IE6's security posture, they use it only on
the intranets.

>
>> past 15 or so years, I keep asking just in case, to see how often it's
>> "we changed X and Y has broken". I can count the occasions on fingers
>> of one hand, vs. the ~mid-4-digits number of the other outcome. So
>> fundamentally nothing changes - it breaks by itself today too :-)
>
> Yep.  And software updates are just like that.  They happen by
> themselves, literally.

As soon as they do not interrupt anything - that's fine.
And in the vast majority of the scenarios - they don't.
IE6, if anything, is an argument *for* the updates.

>
>>
>> (but seriously: appreciate all of the comments. I thought the above
>> blurb about troubleshooting in the draft should be enough, but maybe
>> it is not too strongly worded.
>
> When was the last time your read a warning label on a beer bottle?

I've just checked a couple of Belgian beers - there are no warnings on them.
Seems like it "just works" - even if the spec is so ill-defined there
are too many
competing implementations of Belgian Beer.

I think the beer example makes a case there's probably no One True Answer.

How about - "lookup in the DNS happyeyeballs-off.<host-domain-name> to
see if this algorithm needs to be turned off" ?

cheers,
andrew

On Tue, Mar 1, 2011 at 9:28 AM, Andrew Yourtchenko <ayourtch@gmail.com> wrote:
> On Tue, Mar 1, 2011 at 4:49 PM, Cameron Byrne <cb.list6@gmail.com> wrote:
>> On Tue, Mar 1, 2011 at 4:29 AM, Andrew Yourtchenko <ayourtch@gmail.com> wrote:
>>> Yes, we already have in the latest text:
>>>
>>> "Debugging and Troubleshooting
>>>
>>> This mechanism is aimed to help the user experience in case of connectivity
>>> problems. However, this precise reason also makes it tougher to use these
>>> applications as a means of the verification that the problems are fixed. To
>>> assist in that regard, the applications implementing the proposal in this
>>> document SHOULD also provide a mechanism to temporarily use only one
>>> address family."
>>>
>>> Too weak ? Wrong approach ?
>>>
>>
>> I don't there is anything that you could write in an IETF draft that
>> would make joe-six-pack understand or care about HE.
>
> The joe-six-pack who is supporting the IE6 corporate person ?
>

does not matter.

> That paragraph is for the folks who write the code, not for the support/user.
>
> The support guy will simply know "to support customers with IE6, flip that
> green button into 'off' position before following the script".
>

Sorry, brief interlude for my favorite web video about help desk
http://www.youtube.com/watch?v=W8_Kfjo3VjU

>>>
>>> ...and "no-one changed anything". (That's what everyone says for the
>>
>> Most browsers i have update themselves, or windows update, or apt-get
>> update them and i generally don't care to know what the updating is
>> that happened... updates are good in my world.  In some larger
>> corporate environments i know, IE6 is mandatory.
>
> I suppose that given the IE6's security posture, they use it only on
> the intranets.
>

Um. No comment. But,
http://arstechnica.com/microsoft/news/2010/07/internet-explorer-gains-market-share-so-does-ie6.ars
Recent survey has IE6 in a non-trivial spot.


>>
>>> past 15 or so years, I keep asking just in case, to see how often it's
>>> "we changed X and Y has broken". I can count the occasions on fingers
>>> of one hand, vs. the ~mid-4-digits number of the other outcome. So
>>> fundamentally nothing changes - it breaks by itself today too :-)
>>
>> Yep.  And software updates are just like that.  They happen by
>> themselves, literally.
>
> As soon as they do not interrupt anything - that's fine.
> And in the vast majority of the scenarios - they don't.
> IE6, if anything, is an argument *for* the updates.
>

Agreed. But, that does not change anything in the real world.

>>
>>>
>>> (but seriously: appreciate all of the comments. I thought the above
>>> blurb about troubleshooting in the draft should be enough, but maybe
>>> it is not too strongly worded.
>>
>> When was the last time your read a warning label on a beer bottle?
>
> I've just checked a couple of Belgian beers - there are no warnings on them.
> Seems like it "just works" - even if the spec is so ill-defined there
> are too many
> competing implementations of Belgian Beer.
>
> I think the beer example makes a case there's probably no One True Answer.
>
> How about - "lookup in the DNS happyeyeballs-off.<host-domain-name> to
> see if this algorithm needs to be turned off" ?
>

I think you overestimate the tolerance and enthusiasm thousands of
network operators will have for this. I may be wrong. This reminds
me of recent v6ops flame war where the co-creator of 6to4 refused to
admit that 6to4 was ever an issue and could possibly in any way harm
v6 deployment.

I am not saying HE is bad in any way, i am just saying we need to go
real slow and be VERY grounded in reality. The only way i can think
to do that is to add MUST NOT be on by default. HE is a good
work-around *NOT A FIX* for broken connections.... and masking issues
is only ok for a short time if we are really going to follow-up and
fix it. That said, lets wait for symptoms before applying the
tourniquet, and yes, HE is a tourniquet... but hopefully only cutting
off circulation on a per destination basis for a short amount of time.

Cameron

> cheers,
> andrew
>

> I am not saying HE is bad in any way, i am just saying we need to go
> real slow and be VERY grounded in reality.  The only way i can think
> to do that is to add MUST NOT be on by default.  HE is a good
> work-around *NOT A FIX* for broken connections.... and masking issues
> is only ok for a short time if we are really going to follow-up and
> fix it.  That said, lets wait for symptoms before applying the
> tourniquet, and yes, HE is a tourniquet... but hopefully only cutting
> off circulation on a per destination basis for a short amount of time.

The cat may be most of the way out of the bag on this one. Browsers
are already/starting to do multi-connect and take the fastest
connection. Multi-protocol multi-connect isn't a huge philosophical
leap. And that's not all that indistinguishable from HE minus P value
learning. No opinion here, just an observation.

On Tue, Mar 1, 2011 at 7:04 PM, Cameron Byrne <cb.list6@gmail.com> wrote:
> On Tue, Mar 1, 2011 at 9:28 AM, Andrew Yourtchenko <ayourtch@gmail.com> wrote:
>> On Tue, Mar 1, 2011 at 4:49 PM, Cameron Byrne <cb.list6@gmail.com> wrote:
>>> On Tue, Mar 1, 2011 at 4:29 AM, Andrew Yourtchenko <ayourtch@gmail.com> wrote:

[. lots of stuff snip, Touche re. IE6. It's actually ~5% now but still
noticeable. Thanks for the vid - loved to see it again. ]

> me of recent v6ops flame war where the co-creator of 6to4 refused to
> admit that 6to4 was ever an issue and could possibly in any way harm
> v6 deployment.

I am sorry if I made myself sound like in a flame war - that was not
the intent at all.

>
> I am not saying HE is bad in any way,
> i am just saying we need to go real slow and be VERY grounded in reality.
> The only way i can think to do that is to add MUST NOT be on by default.
> HE is a good work-around *NOT A FIX* for broken connections.... and masking issues
> is only ok for a short time if we are really going to follow-up and
> fix it.  That said, lets wait for symptoms before applying the
> tourniquet, and yes, HE is a tourniquet... but hopefully only cutting
> off circulation on a per destination basis for a short amount of time.

What do you think about this:

Some sites may wish to be informed when the the hosts adjust
their "P" value,
in order to troubleshoot the underlying cause. To help these sites,
a strawman proposal is to send a syslog message or other notification
to an address that may be configured
by a site administrator in a centralized fashion.
(The exact method TBD - DHCP option, domain name, etc.)
This syslog message should be sent only first N times that the host
expects to prefer IPv6 but has to use IPv4. I.e. the first N
times it decreases
the value of P. The value of N TBD.


This should make HE into something that can *lower* the complexity of
the troubleshooting ?

If you have the HE at the end user = you configure them to send a
notification to your helpdesk proactively;

If you have it at a helpdesk - you convert this into a big flashing
"That site example.com you just tested, does not work over IPv6.
We made it work because we are so kind to quickly fallback - but IE6
will be broken...."

cheers,
andrew

> Cameron
>

On Tue, Mar 1, 2011 at 3:39 PM, Andrew Yourtchenko <ayourtch@gmail.com> wrote:
> On Tue, Mar 1, 2011 at 7:04 PM, Cameron Byrne <cb.list6@gmail.com> wrote:
>> On Tue, Mar 1, 2011 at 9:28 AM, Andrew Yourtchenko <ayourtch@gmail.com> wrote:
>>> On Tue, Mar 1, 2011 at 4:49 PM, Cameron Byrne <cb.list6@gmail.com> wrote:
>>>> On Tue, Mar 1, 2011 at 4:29 AM, Andrew Yourtchenko <ayourtch@gmail.com> wrote:
>
> [. lots of stuff snip, Touche re. IE6. It's actually ~5% now but still
> noticeable. Thanks for the vid - loved to see it again. ]
>

"IE6 usage, meanwhile, continues to decline. It fell to 11.33% in
February, down from 11.43% in January."

This ran in network world today,
http://www.networkworld.com/news/2011/030111-ie-firefox.html?hpg1=bn

Different measurements result in different results, but i cited a source.

>> me of recent v6ops flame war where the co-creator of 6to4 refused to
>> admit that 6to4 was ever an issue and could possibly in any way harm
>> v6 deployment.
>
> I am sorry if I made myself sound like in a flame war - that was not
> the intent at all.
>

No, this has not been a flame war. I am just very scared about
creating another well intentioned patch that makes things worse like
6to4. And, i don't believe any RFC specified helpdesk interactions
will make much of a differences in the real world.

>>
>> I am not saying HE is bad in any way,
>> i am just saying we need to go real slow and be VERY grounded in reality.
>> The only way i can think to do that is to add MUST NOT be on by default.
>> HE is a good work-around *NOT A FIX* for broken connections.... and masking issues
>> is only ok for a short time if we are really going to follow-up and
>> fix it.  That said, lets wait for symptoms before applying the
>> tourniquet, and yes, HE is a tourniquet... but hopefully only cutting
>> off circulation on a per destination basis for a short amount of time.
>
> What do you think about this:
>
>      Some sites may wish to be informed when the the hosts adjust
> their "P" value,
>      in order to troubleshoot the underlying cause. To help these sites,
>      a strawman proposal is to send a syslog message or other notification
>      to an address that may be configured
>      by a site administrator in a centralized fashion.
>      (The exact method TBD - DHCP option, domain name, etc.)
>      This syslog message should be sent only first N times that the host
>      expects to prefer IPv6 but has to use IPv4. I.e. the first N
> times it decreases
>      the value of P. The value of N TBD.
>

I honestly don't believe there is anything that can be said aside from
default off. That's just my opinion.

>
> This should make HE into something that can *lower* the complexity of
> the troubleshooting ?
>
> If you have the HE at the end user = you configure them to send a
> notification to your helpdesk proactively;
>

I don't work at a helpdesk, but this does not seem realistic IMHO.

> If you have it at a helpdesk - you convert this into a big flashing
> "That site example.com you just tested, does not work over IPv6.
> We made it work because we are so kind to quickly fallback - but IE6
> will be broken...."
>
> cheers,
> andrew
>
>> Cameron
>>
>

On 3/1/11 10:04 AM, Cameron Byrne wrote:
> I am not saying HE is bad in any way, i am just saying we need to go
> real slow and be VERY grounded in reality. The only way i can think
> to do that is to add MUST NOT be on by default. HE is a good
> work-around *NOT A FIX* for broken connections.... and masking issues
> is only ok for a short time if we are really going to follow-up and
> fix it. That said, lets wait for symptoms before applying the
> tourniquet, and yes, HE is a tourniquet... but hopefully only cutting
> off circulation on a per destination basis for a short amount of time.

Injecting some reality into this conversation...

The amount of native IPv6 traffic on our network far exceeds the amount
of traffic seen for 6to4, Teredo, statically routed 6in4, and 6in4
tunnels with BGP capability, *combined*.

Further, 6to4 and Teredo traffic *far exceeds* the traffic seen by our
statically routed 6in4 tunnel servers which exceeds the traffic seen by
the designated routers used to terminate 6in4 tunnels with BGP capability.

Anyway, the point was, the amount of IPv6 traffic on our network that
uses BGP IPv6 tunnel routers is miniscule by comparison to other IPv6
traffic sources (native or tunneled). Just saying. As always, we
highly recommend native IPv6. The vast majority of the IPv6 BGP
sessions we have are native.

In other news:

With regards to latency of IPv6 vs IPv4, when testing to dual stacked
reverse DNS servers, in 782 cases out of 1300 IPv6 was faster than IPv4
by more than 1 millisecond ( http://bgp.he.net/ipv6-progress-report.cgi
). I attribute this to the IPv6 network guys not being locked into
whatever suboptimal purchasing policies that are enforced for their IPv4
transit purchases. ;) I'll work on getting latency data for dual
stacked web servers as well, so we can see how widespread the "IPv6
often faster than IPv4" phenomena is.

Mike.

My bad (I think),

I was having a little trouble following this thread and appear to
mistaken the context of "HE tunnels".

So many IPv6 threads!!! It's awesome... I'll go back my corner and lurk.

Mike.
ps. hahahahahaha whoops.

>
>
> On 3/1/11 10:04 AM, Cameron Byrne wrote:
>> I am not saying HE is bad in any way, i am just saying we need to go
>> real slow and be VERY grounded in reality. The only way i can think
>> to do that is to add MUST NOT be on by default. HE is a good
>> work-around *NOT A FIX* for broken connections.... and masking issues
>> is only ok for a short time if we are really going to follow-up and
>> fix it. That said, lets wait for symptoms before applying the
>> tourniquet, and yes, HE is a tourniquet... but hopefully only cutting
>> off circulation on a per destination basis for a short amount of time.
>
> Injecting some reality into this conversation...
>
> The amount of native IPv6 traffic on our network far exceeds the
> amount of traffic seen for 6to4, Teredo, statically routed 6in4, and
> 6in4 tunnels with BGP capability, *combined*.
>
> Further, 6to4 and Teredo traffic *far exceeds* the traffic seen by our
> statically routed 6in4 tunnel servers which exceeds the traffic seen
> by the designated routers used to terminate 6in4 tunnels with BGP
> capability.
>
> Anyway, the point was, the amount of IPv6 traffic on our network that
> uses BGP IPv6 tunnel routers is miniscule by comparison to other IPv6
> traffic sources (native or tunneled). Just saying. As always, we
> highly recommend native IPv6. The vast majority of the IPv6 BGP
> sessions we have are native.
>
> In other news:
>
> With regards to latency of IPv6 vs IPv4, when testing to dual stacked
> reverse DNS servers, in 782 cases out of 1300 IPv6 was faster than
> IPv4 by more than 1 millisecond (
> http://bgp.he.net/ipv6-progress-report.cgi ). I attribute this to the
> IPv6 network guys not being locked into whatever suboptimal purchasing
> policies that are enforced for their IPv4 transit purchases. ;)
> I'll work on getting latency data for dual stacked web servers as
> well, so we can see how widespread the "IPv6 often faster than IPv4"
> phenomena is.
>
> Mike.
>

Yeah, "HE" might not be the best acronym for happy-eyeballs. H-E might be
marginally better...

On Wed, Mar 2, 2011 at 4:36 AM, Mike Leber <mleber@he.net> wrote:

>
> My bad (I think),
>
> I was having a little trouble following this thread and appear to mistaken
> the context of "HE tunnels".
>
> So many IPv6 threads!!! It's awesome... I'll go back my corner and lurk.
>
> Mike.
> ps. hahahahahaha whoops.
>

--
Tim:>

> -----Original Message-----
> From: Mark Smith
> [mailto:nanog@85d5b20a518b8f6864949bd940457dc124746ddc.nosense.org]
> Sent: Tuesday, March 01, 2011 9:49 AM
> > > DS-Website having one AAAA RR: ~3 minutes time to IPv4 fallback(!)
> >
> > I analyzed this behavior, the reason is that the browser is calling
> the connect() API from the OS. This function will then try the complete
> TCP opening process, including TCP retries. Linux (at least debian
> flavour) will do 5 retries, with exponentially growing time between
> them.
> >
> > You can play with the number of retries; I put it down to 2, and the
> fallback-to-ipv4 time was roughly 10s.
> >
> > I opened a feature request on the kernel bug tracker to request the
> possibility to tune this parameter for dualstack websites, and put it
> by default to 2, but nothing happened afaik.
> >
> > The request :
> > https://bugzilla.kernel.org/show_bug.cgi?id=23242
> >
> > The sysctl control is tcp_syn_retries (sorry forgot the hierarchy,
> you'll have to grep :)
> >
>
> Are you using a Fritzbox 7390? When you enable IPv6, they enable ULAs
> by default (although their ULAs aren't (near-)unique because the unique
> id is all zeros!), and don't issue ICMPv6 Destination Unreachables then
> they don't have an IPv6 default route. So with an IPv4 only WAN link,
> this problem occurs on all dual-stack websites. According to RFC4443,
> they SHOULD be issuing ICMPv6 Destination Unreachables, No route to
> destination.

No, I'm not using Fritzbox. The v6 unreachability problem was "in the Internet" (really: Cisco bug on 7600).

Guillaume