Mailing List Archive

QUIC traffic throttled on AT&T residential
I've AT&T fiber (in RTP, NC) (AS7018) and I notice UDP QUIC traffic
from google (esp. youtube) becomes very slow after a time.

This especially occurs with ipv4 connections. I'm not the only one to
notice; a web search for e.g. "Extremely Poor Youtube TV Performance"
notes the issue.

I assume traffic is being throttled on AT&T's side. And it's not done
with their CPE; I'm bypassing their NAT box and connecting my laptop
directly to the ONT.

A quick google search shows people are aware that QUIC can look like
DoS traffic -- but how widely known is this problem? It may become
worse if / when sites transition to HTTP/3

For now I reject UDP 443 on the client firewall. Applications using
QUIC client libraries then fallback to TCP. This works well and I have
no issues with latency or throughput when using TCP.

May I naively ask if Google staff are working with AT&T to address this?

-- Dan
Re: QUIC traffic throttled on AT&T residential [ In reply to ]
On Tue, Feb 18, 2020 at 5:44 PM Daniel Sterling <sterling.daniel@gmail.com>
wrote:

> I've AT&T fiber (in RTP, NC) (AS7018) and I notice UDP QUIC traffic
> from google (esp. youtube) becomes very slow after a time.
>
> This especially occurs with ipv4 connections. I'm not the only one to
> notice; a web search for e.g. "Extremely Poor Youtube TV Performance"
> notes the issue.
>
> I assume traffic is being throttled on AT&T's side. And it's not done
> with their CPE; I'm bypassing their NAT box and connecting my laptop
> directly to the ONT.
>
> A quick google search shows people are aware that QUIC can look like
> DoS traffic -- but how widely known is this problem? It may become
> worse if / when sites transition to HTTP/3
>
> For now I reject UDP 443 on the client firewall. Applications using
> QUIC client libraries then fallback to TCP. This works well and I have
> no issues with latency or throughput when using TCP.
>
> May I naively ask if Google staff are working with AT&T to address this?


Sounds like you found the answer, ATT just needs to scale up your rejection
approach that is proven to work well.

Yes, most ddos traffic is ipv4 udp and yes Google was made aware that they
would be mixing with bad company in the pool of ipv4 udp traffic ... but
they have reasons.

I am not a fan of quic or any udp traffic. My suggestion was that Google
use an new L4 instead of UDP, but that was too hard for the Googlers.

So here we are.

Just say no to udp.



>
> -- Dan
>
Re: QUIC traffic throttled on AT&T residential [ In reply to ]
Are you suggesting that ATT block all QUIC across their network?

On Tue, Feb 18, 2020, 7:02 PM Ca By <cb.list6@gmail.com> wrote:

>
>
> On Tue, Feb 18, 2020 at 5:44 PM Daniel Sterling <sterling.daniel@gmail.com>
> wrote:
>
>> I've AT&T fiber (in RTP, NC) (AS7018) and I notice UDP QUIC traffic
>> from google (esp. youtube) becomes very slow after a time.
>>
>> This especially occurs with ipv4 connections. I'm not the only one to
>> notice; a web search for e.g. "Extremely Poor Youtube TV Performance"
>> notes the issue.
>>
>> I assume traffic is being throttled on AT&T's side. And it's not done
>> with their CPE; I'm bypassing their NAT box and connecting my laptop
>> directly to the ONT.
>>
>> A quick google search shows people are aware that QUIC can look like
>> DoS traffic -- but how widely known is this problem? It may become
>> worse if / when sites transition to HTTP/3
>>
>> For now I reject UDP 443 on the client firewall. Applications using
>> QUIC client libraries then fallback to TCP. This works well and I have
>> no issues with latency or throughput when using TCP.
>>
>> May I naively ask if Google staff are working with AT&T to address this?
>
>
> Sounds like you found the answer, ATT just needs to scale up your
> rejection approach that is proven to work well.
>
> Yes, most ddos traffic is ipv4 udp and yes Google was made aware that they
> would be mixing with bad company in the pool of ipv4 udp traffic ... but
> they have reasons.
>
> I am not a fan of quic or any udp traffic. My suggestion was that Google
> use an new L4 instead of UDP, but that was too hard for the Googlers.
>
> So here we are.
>
> Just say no to udp.
>
>
>
>>
>> -- Dan
>>
>
Re: QUIC traffic throttled on AT&T residential [ In reply to ]
9On Tue, Feb 18, 2020 at 7:07 PM Ross Tajvar <ross@tajvar.io> wrote:
>
> Are you suggesting that ATT block all QUIC across their network?

One might argue they already *are* doing so; QUIC is essentially
unusable on my AT&T ipv4 residential connection (and a web search
suggests I'm not alone).
Re: QUIC traffic throttled on AT&T residential [ In reply to ]
Yes, other than AT&T increasing their permitted incoming UDP traffic -- the easiest thing AT&T can do -- AT&T could ask the vendor of their flow restricting device to use bi-directional UDP traffic on same 5-tuple to indicate "desire to receive", rather than solely examining incoming UDP traffic as they are doing today; this is not easy but also not impossible.

While it is true Youtube/Google could treat AT&T-sourced connections as 'bad' and force TCP, but I am sure Youtube/Google is already gathering those statistics and already aware that AT&T is throttling. For all we know, you and the others noticing the issue have fallen into the pit of A/B testers checking for their current throttling, and others aren't being throttled. Perhaps it's everyone, the tests are not well described or well announced. Youtube/Google is hoping customers complain to AT&T so that AT&T removes or improves the flow restrictor, because otherwise AT&T customers won't be able to get QUIC. Similarly, AT&T could protect their users from AT&T's own rate limiting by blocking QUIC towards major servers that support QUIC but such blocking becomes problematic as QUIC rolls out beyond Google to Cloudflare and elsewhere.

This tussle has similarities to IPv6 vs IPv4.

-d


> On Feb 18, 2020, at 4:00 PM, Ca By <cb.list6@gmail.com> wrote:
>
>
>
> On Tue, Feb 18, 2020 at 5:44 PM Daniel Sterling <sterling.daniel@gmail.com <mailto:sterling.daniel@gmail.com>> wrote:
> I've AT&T fiber (in RTP, NC) (AS7018) and I notice UDP QUIC traffic
> from google (esp. youtube) becomes very slow after a time.
>
> This especially occurs with ipv4 connections. I'm not the only one to
> notice; a web search for e.g. "Extremely Poor Youtube TV Performance"
> notes the issue.
>
> I assume traffic is being throttled on AT&T's side. And it's not done
> with their CPE; I'm bypassing their NAT box and connecting my laptop
> directly to the ONT.
>
> A quick google search shows people are aware that QUIC can look like
> DoS traffic -- but how widely known is this problem? It may become
> worse if / when sites transition to HTTP/3
>
> For now I reject UDP 443 on the client firewall. Applications using
> QUIC client libraries then fallback to TCP. This works well and I have
> no issues with latency or throughput when using TCP.
>
> May I naively ask if Google staff are working with AT&T to address this?
>
> Sounds like you found the answer, ATT just needs to scale up your rejection approach that is proven to work well.
>
> Yes, most ddos traffic is ipv4 udp and yes Google was made aware that they would be mixing with bad company in the pool of ipv4 udp traffic ... but they have reasons.
>
> I am not a fan of quic or any udp traffic. My suggestion was that Google use an new L4 instead of UDP, but that was too hard for the Googlers.
>
> So here we are.
>
> Just say no to udp.
>
>
>
>
> -- Dan
Re: QUIC traffic throttled on AT&T residential [ In reply to ]
The members of the QUIC WG at IETF have not thought this was a problem as they don't observe it across the board.

The cost for payloads with QUIC is much higher on the CPU side vs TCP as well - this is also not considered an issue either.

The explanation I got (which seems fair) from someone was that they only way to roll a new transport was to squat on some existing stuff that would make it through firewalls.

We have an internet that is largely fixed around running NATs and TCP and UDP only these days. I for one find this sad and don't see a good way forward on either side.

Sent from my iCar

> On Feb 18, 2020, at 7:13 PM, Daniel Sterling <sterling.daniel@gmail.com> wrote:
>
> ?9On Tue, Feb 18, 2020 at 7:07 PM Ross Tajvar <ross@tajvar.io> wrote:
>>
>> Are you suggesting that ATT block all QUIC across their network?
>
> One might argue they already *are* doing so; QUIC is essentially
> unusable on my AT&T ipv4 residential connection (and a web search
> suggests I'm not alone).
Re: QUIC traffic throttled on AT&T residential [ In reply to ]
On Tue, Feb 18, 2020 at 7:20 PM Dan Wing <danwing@gmail.com> wrote:
> For all we know, you and the others noticing the issue have fallen into the pit of A/B testers checking for their current throttling, and others aren't being throttled.

Ouch, I hope not -- do A/B tests that result in extreme performance
degradation continue indefinitely ?

> Youtube/Google is hoping customers complain to AT&T

It seems like this is one area where AT&T has no reason to care
(unless management decides to get involved). The default stance from
support would be, I assume: "Google is slow for you? Well obviously it
can't be Google's fault. Have a new NAT box"

I would bet dollars to donuts that if Google took a proper look at
their stats they would see this issue manifesting as users simply not
using services that have been QUIC-holed.

-- Dan
Re: QUIC traffic throttled on AT&T residential [ In reply to ]
On 2020-02-18 7:07 p.m., Ross Tajvar wrote:
> Are you suggesting that ATT block all QUIC across their network?
Blocking a (for you) undesirable option when an established fallback
exists is a much better end user experience than introducing breakage
into that option

When you throttle or subtly break things you get:

On 2020-02-18 7:12 p.m., Daniel Sterling wrote:
> One might argue they already *are* doing so; QUIC is essentially
> unusable on my AT&T ipv4 residential connection (and a web search
> suggests I'm not alone).

Or: I no longer use my ISP's IPv6 access (via 6rd) since it would cause
terrible slowdowns due to packet loss when it broke

Or: some AT&T customers cannot connect to our customers due to IPv6
HTTPS interception: https://meta.discourse.org/t/-/140769/3

Or (probably the same problem):
https://tutanota.com/blog/posts/att-blocks-tutanota/

With blocking in these cases, QUIC falls back to TCP, Happy Eyeballs
falls back to IPv4, everybody's happy.
Re: QUIC traffic throttled on AT&T residential [ In reply to ]
On 2020-02-18 4:25 p.m., Jared Mauch wrote:
> The members of the QUIC WG at IETF have not thought this was a problem as they don't observe it across the board.
>
> The cost for payloads with QUIC is much higher on the CPU side vs TCP as well - this is also not considered an issue either.
There's plenty of room for system call/interface improvements and
hardware acceleration in UDP stacks, both of which should help with CPU
concerns. Now that UDP may represent a significant portion of internet
traffic, it will be easier for the necessary engineering expense to be
justified.
> The explanation I got (which seems fair) from someone was that they only way to roll a new transport was to squat on some existing stuff that would make it through firewalls.
If there's clearly a two-way flow occurring, i.e. as is the case with a
stream of QUIC traffic, that is a clearly distinguishable case from a
DoS where the recipient is going to be dropping traffic. While this may
be considerably harder to implement at scale than simply throttling UDP
indiscriminately, hopefully there can be enough user suffering that AT&T
feels compelled to do the right thing and allow the internet to continue
to develop new protocols. Not everything fits neatly into a
circuit-switched head-of-line-blocking request-response/HTTP paradigm.
> We have an internet that is largely fixed around running NATs and TCP and UDP only these days. I for one find this sad and don't see a good way forward on either side.

Hopefully situations like this where Google swings their Chrome hammer
for good instead of for evil can help get us there... now if we can just
get those 100 gigabyte game downloads to get delivered over UDP too...

---

Daniel Dent

https://www.danieldent.com
Re: QUIC traffic throttled on AT&T residential [ In reply to ]
On 2020-02-18 4:32 p.m., Michael Brown wrote:
> With blocking in these cases, QUIC falls back to TCP, Happy Eyeballs
> falls back to IPv4, everybody's happy.

The IPv6 deployment landscape might look considerably better if browser
developers were instead to work together to co-ordinate an "unhappy
eyeballs" algorithm that was rolled out over time, gradually degrading
performance for IPv4-only connectivity situations. Now that
non-evergreen browsers are pretty much dead, this might even be a
realistic proposal, especially if it were done gradually enough. Google
announcing a ranking penalty for IPv4-only sites wouldn't hurt either.

A while after implementing "unhappy eyeballs", once the performance
penalties are severe enough that IPv6 adoption is high, the algorithm
could be further adjusted to gradually impose penalties on sites that
dual-stack instead of going v6-only. Growing prevalence of IPv6-only
sites is probably the only thing that will get a lot of access networks
to support v6.

The problem with happy eyeballs is that it works too well: excellent
backwards compatibility reduces or eliminates the incentives for progress.

Happy eyeballs was necessary for it be realistic for sites to deploy
IPv6 support, but now it holds us back.

--

Daniel Dent

https://www.danieldent.com
Re: QUIC traffic throttled on AT&T residential [ In reply to ]
On Tue, Feb 18, 2020 at 8:05 PM Michael Brown <michael@supermathie.net> wrote:
> Blocking a (for you) undesirable option when an established fallback
> exists is a much better end user experience than introducing breakage
> into that option

> Or: I no longer use my ISP's IPv6 access (via 6rd) since it would cause
> terrible slowdowns due to packet loss when it broke

All the +1s.


I have one goal for my home internet: it should work better than my cell phone!

In our house, "screen time" is "all the time". Everyone is on their
phone non-stop. So you'd think getting fiber, installing UBNT APs and
swapping out AT&T's CPE for a Core i5 linux box would provide a better
internet experience than a tree-obstructed cell tower a mile or two
down the road.

But you'd be wrong.


Everyone in the house was, on a daily basis, turning off wifi in favor
of AT&T LTE!

Why was everyone switching off wifi? I couldn't blame them -- I was
toggling it on and off myself to get the occasional website or IM
conversation to load. Why was my network broken?? How was it possible
that a fairly high-latency mobile connection could provide a better
experience than 802.11ac to an AP in the same room backed by a gigabit
PON?


I banged against this for *years*. I punted on using my own router and
tried just AT&T's CPE (reset to factory defaults). That does work
decently, but there are some maddening quirks, not least of which is
insanely high jitter.

I tried SoHo devices running vendor stock firmware driving hardware
NAT. Those also work well -- until they inevitably crash.


I tried ddwrt and openwrt. I tried AQM; I tried QoS. I tried NUCs
running upstream kernels; downstream kernels; I tried custom patches.
I tried HFSC and CoDel; I compiled iproute2 so I could have some
tc_cake.


On the link-layer (wifi) side I tried one AP; two APs; three APs. I
tested any number of combinations of SSID name, channel frequency and
width. I tried with ipv6; no ipv6; I put all 2ghz devices on their own
AP; in desperation I even tried dedicating one AP and an entire 5ghz
frequency range for just one phone.

But nothing mattered until I finally figured it out:

It was DNS. Of course it was DNS. It's always DNS.


When DNS was solid and fast, everything else fell into place. Toggling
wifi worked because it was the same as re-querying DNS! And the DNS
service on mobile works well -- better than the average CPE.

*** I cannot stress this enough. No manner of tuning or tweaking to my
home network stopped users from fleeing it, until I had solid DNS. ***


For fast DNS, you of course need fast UDP. And, as we've empirically
discovered, well-behaved UDP is by no means guaranteed on residential
connections.

It turns out dnsmasq has a couple of tunables that can make a huge
difference to home internet DNS performance. First, you need to be
querying the DNS servers AT&T tells you to via DHCP. They're the least
likely to be throttled, unfortunately. But I've found even that alone
isn't enough.

You need to set dnsmasq's "query-port" option. By default it's random
to protect against CVE-2008-1447, but apparently sending a ton of
random-source-port UDP traffic does not impress the AT&T network flow
control systems, and your DNS traffic becomes unbearably slow (or is
simply dropped entirely).


AT&T gives you two DNS servers via DHCP. You can query more --
8.8.8.8, 4.2.2.2, 2606:4700:4700::1111 -- but if you do, you'll want
to enable dnsmasq's "all-servers" option. Packets are cheap -- send a
query to every server on your list and use whatever comes back first.
If you've angered the UDP flow restrictor, no matter -- with luck at
least one of your packets is going to a server that's up and not
throttled or overloaded!


Of course DNS is just the beginning -- you still need a proper gateway
device with a good NAT stack and/or firewall; you still need a strong
wifi signal; you still need tc_cake so everyone can watch Netflix at
the same time.

But DNS is the *core*. Nothing works well until DNS works well. That
means nothing works well unless UDP works well. And if I have learned
anything about AS7018, it's that UDP -- especially its v4 UDP -- Does.
Not. Work. Well.


Enter QUIC. It may be the perfect transport-layer protocol; but by
putting it on top of UDP it's hobbled. It breaks extant v4 internet in
a way that nothing else we've yet seen does -- it takes what would be
your TCP traffic and gives it inconsistent and intermittently poor
performance. Maybe it's sometimes fast. Maybe it is. But I can tell
you, it sometimes Is Very Much Not So.


As much as I would on principle rather not stick to a legacy, TCP-only
home network --

I can say that right now, my home internet, blocking UDP 443, and
making tons of insecure DNS queries -- is the most stable, fastest,
most usable and enjoyable home internet I've ever had. And my users
agree -- they no longer turn off wifi.


May I naively ask if Google staff have considered scrapping using UDP
and instead proposing a new, first-class transport protocol that OSes
can implement on top of IP? UDP certainly helped speed testing and
iteration for QUIC in real-world scenarios, but I fear UDP is too
brittle ground upon which to build the next generation of internet
transport. Committing to UDP now with HTTP/3 may be a mistake.

And if that doesn't convince you, consider that even I was smart
enough to figure out how to block it :)

-- Dan
Re: QUIC traffic throttled on AT&T residential [ In reply to ]
> On 19 Feb 2020, at 15:47, Daniel Sterling <sterling.daniel@gmail.com> wrote:
>
> On Tue, Feb 18, 2020 at 8:05 PM Michael Brown <michael@supermathie.net> wrote:
>> Blocking a (for you) undesirable option when an established fallback
>> exists is a much better end user experience than introducing breakage
>> into that option
>
>> Or: I no longer use my ISP's IPv6 access (via 6rd) since it would cause
>> terrible slowdowns due to packet loss when it broke
>
> All the +1s.
>
>
> I have one goal for my home internet: it should work better than my cell phone!
>
> In our house, "screen time" is "all the time". Everyone is on their
> phone non-stop. So you'd think getting fiber, installing UBNT APs and
> swapping out AT&T's CPE for a Core i5 linux box would provide a better
> internet experience than a tree-obstructed cell tower a mile or two
> down the road.
>
> But you'd be wrong.
>
>
> Everyone in the house was, on a daily basis, turning off wifi in favor
> of AT&T LTE!
>
> Why was everyone switching off wifi? I couldn't blame them -- I was
> toggling it on and off myself to get the occasional website or IM
> conversation to load. Why was my network broken?? How was it possible
> that a fairly high-latency mobile connection could provide a better
> experience than 802.11ac to an AP in the same room backed by a gigabit
> PON?
>
>
> I banged against this for *years*. I punted on using my own router and
> tried just AT&T's CPE (reset to factory defaults). That does work
> decently, but there are some maddening quirks, not least of which is
> insanely high jitter.
>
> I tried SoHo devices running vendor stock firmware driving hardware
> NAT. Those also work well -- until they inevitably crash.
>
>
> I tried ddwrt and openwrt. I tried AQM; I tried QoS. I tried NUCs
> running upstream kernels; downstream kernels; I tried custom patches.
> I tried HFSC and CoDel; I compiled iproute2 so I could have some
> tc_cake.
>
>
> On the link-layer (wifi) side I tried one AP; two APs; three APs. I
> tested any number of combinations of SSID name, channel frequency and
> width. I tried with ipv6; no ipv6; I put all 2ghz devices on their own
> AP; in desperation I even tried dedicating one AP and an entire 5ghz
> frequency range for just one phone.
>
> But nothing mattered until I finally figured it out:
>
> It was DNS. Of course it was DNS. It's always DNS.
>
>
> When DNS was solid and fast, everything else fell into place. Toggling
> wifi worked because it was the same as re-querying DNS! And the DNS
> service on mobile works well -- better than the average CPE.
>
> *** I cannot stress this enough. No manner of tuning or tweaking to my
> home network stopped users from fleeing it, until I had solid DNS. ***
>
>
> For fast DNS, you of course need fast UDP. And, as we've empirically
> discovered, well-behaved UDP is by no means guaranteed on residential
> connections.
>
> It turns out dnsmasq has a couple of tunables that can make a huge
> difference to home internet DNS performance. First, you need to be
> querying the DNS servers AT&T tells you to via DHCP. They're the least
> likely to be throttled, unfortunately. But I've found even that alone
> isn't enough.
>
> You need to set dnsmasq's "query-port" option. By default it's random
> to protect against CVE-2008-1447, but apparently sending a ton of
> random-source-port UDP traffic does not impress the AT&T network flow
> control systems, and your DNS traffic becomes unbearably slow (or is
> simply dropped entirely)

If dnsmasq supports DNS COOKIE turn it on. DNS COOKIE provides protection
against CVE-2008-1447 provides the other end supports DNS COOKIE without
having to play games with ports.

> AT&T gives you two DNS servers via DHCP. You can query more --
> 8.8.8.8, 4.2.2.2, 2606:4700:4700::1111 -- but if you do, you'll want
> to enable dnsmasq's "all-servers" option. Packets are cheap -- send a
> query to every server on your list and use whatever comes back first.
> If you've angered the UDP flow restrictor, no matter -- with luck at
> least one of your packets is going to a server that's up and not
> throttled or overloaded!
>
>
> Of course DNS is just the beginning -- you still need a proper gateway
> device with a good NAT stack and/or firewall; you still need a strong
> wifi signal; you still need tc_cake so everyone can watch Netflix at
> the same time.
>
> But DNS is the *core*. Nothing works well until DNS works well. That
> means nothing works well unless UDP works well. And if I have learned
> anything about AS7018, it's that UDP -- especially its v4 UDP -- Does.
> Not. Work. Well.
>
>
> Enter QUIC. It may be the perfect transport-layer protocol; but by
> putting it on top of UDP it's hobbled. It breaks extant v4 internet in
> a way that nothing else we've yet seen does -- it takes what would be
> your TCP traffic and gives it inconsistent and intermittently poor
> performance. Maybe it's sometimes fast. Maybe it is. But I can tell
> you, it sometimes Is Very Much Not So.
>
>
> As much as I would on principle rather not stick to a legacy, TCP-only
> home network --
>
> I can say that right now, my home internet, blocking UDP 443, and
> making tons of insecure DNS queries -- is the most stable, fastest,
> most usable and enjoyable home internet I've ever had. And my users
> agree -- they no longer turn off wifi.
>
>
> May I naively ask if Google staff have considered scrapping using UDP
> and instead proposing a new, first-class transport protocol that OSes
> can implement on top of IP? UDP certainly helped speed testing and
> iteration for QUIC in real-world scenarios, but I fear UDP is too
> brittle ground upon which to build the next generation of internet
> transport. Committing to UDP now with HTTP/3 may be a mistake.
>
> And if that doesn't convince you, consider that even I was smart
> enough to figure out how to block it :)
>
> -- Dan

--
Mark Andrews, ISC
1 Seymour St., Dundas Valley, NSW 2117, Australia
PHONE: +61 2 9871 4742 INTERNET: marka@isc.org
Re: QUIC traffic throttled on AT&T residential [ In reply to ]
On Tue, Feb 18, 2020 at 8:48 PM Daniel Sterling <sterling.daniel@gmail.com>
wrote:

[snip impressive debugging story]

As much as I would on principle rather not stick to a legacy, TCP-only
> home network --
>
> I can say that right now, my home internet, blocking UDP 443, and
> making tons of insecure DNS queries -- is the most stable, fastest,
> most usable and enjoyable home internet I've ever had. And my users
> agree -- they no longer turn off wifi.
>

Rather than hobble your home network to work around a misbehaving ISP, have
you considered simply getting a new ISP?

Damian
Re: QUIC traffic throttled on AT&T residential [ In reply to ]
On Wed, Feb 19, 2020 at 12:51 AM Damian Menscher <damian@google.com> wrote:
> [snip impressive debugging story]

lol fair. I didn't umm mean to just brag -- my point was that:
generally available SoHo internet is worse than mobile networks esp.
for UDP traffic!

> Rather than hobble your home network to work around a misbehaving ISP, have you considered simply getting a new ISP?

I could get a leased line, sure, or route all my traffic through a
UDP/TCP VPN -- but the average user won't and can't, no.

-- Dan
Re: QUIC traffic throttled on AT&T residential [ In reply to ]
Michael Brown wrote:

> Blocking a (for you) undesirable option when an established fallback
> exists is a much better end user experience than introducing breakage
> into that option

Are you saying AT&T should block UDP entirely?

Damian Menscher via NANOG wrote:

> As much as I would on principle rather not stick to a legacy, TCP-only
> home network --

Never stick to UDP.

Masataka Ohta
Re: QUIC traffic throttled on AT&T residential [ In reply to ]
Peace,

On Wed, Feb 19, 2020 at 7:49 AM Daniel Sterling
<sterling.daniel@gmail.com> wrote:

> May I naively ask if Google staff have considered scrapping using UDP
> and instead proposing a new, first-class transport protocol that OSes
> can implement on top of IP?

The IETF WG did, at some point. The opinion overall I think is that
this would probably bring in even more troubles both to the protocol
(e.g. in plenty of networks which do not allow any IP proto except 1,
6 and 17) and to networks (we have things like RFC 8086 for a reason).
There might've been other reasons.

--
Töma
Re: QUIC traffic throttled on AT&T residential [ In reply to ]
On Wed, Feb 19, 2020 at 1:28 AM Daniel Dent
<nanog-list@contactdaniel.net> wrote:
>
> On 2020-02-18 4:25 p.m., Jared Mauch wrote:
> > The members of the QUIC WG at IETF have not thought this was a problem as they don't observe it across the board.
> >
> > The cost for payloads with QUIC is much higher on the CPU side vs TCP as well - this is also not considered an issue either.

Jared, you mean: "Cost on the server" here not "cost on the router".
I think, and I ask because at least some of Daniel's note implies, to
my reading, that 'on the router' was your concern?

> There's plenty of room for system call/interface improvements and
> hardware acceleration in UDP stacks, both of which should help with CPU
> concerns. Now that UDP may represent a significant portion of internet
> traffic, it will be easier for the necessary engineering expense to be
> justified.
> > The explanation I got (which seems fair) from someone was that they only way to roll a new transport was to squat on some existing stuff that would make it through firewalls.
> If there's clearly a two-way flow occurring, i.e. as is the case with a

2 way flow means something on your home host or home gateway.
It means very little at internet scale... since, in many cases, you ->
server and server -> you are not sharing many of the same links /
routers / etc.

-chris
Re: QUIC traffic throttled on AT&T residential [ In reply to ]
Daniel Dent wrote:

>> The explanation I got (which seems fair) from someone was that they
>> only way to roll a new transport was to squat on some existing stuff
>> that would make it through firewalls.
> If there's clearly a two-way flow occurring,

Clearly, it should be DOS amplification.

Masataka Ohta
Re: QUIC traffic throttled on AT&T residential [ In reply to ]
Christopher Morrow wrote:

> 2 way flow means something on your home host or home gateway.
> It means very little at internet scale... since, in many cases, you ->
> server and server -> you are not sharing many of the same links /
> routers / etc.

Subject suggests it's retail ISP to homes, which are unlikely to
be multihomed.

Masataka Ohta
Re: QUIC traffic throttled on AT&T residential [ In reply to ]
On Wed, 19 Feb 2020 at 08:18, Masataka Ohta <
mohta@necom830.hpcl.titech.ac.jp> wrote:

> Christopher Morrow wrote:
>
> > 2 way flow means something on your home host or home gateway.
> > It means very little at internet scale... since, in many cases, you ->
> > server and server -> you are not sharing many of the same links /
> > routers / etc.
>
> Subject suggests it's retail ISP to homes, which are unlikely to
> be multihomed.
>
>
>
My network has multiple ingress/egress points, and I do my filtering on the
edge.

The Chris's statement applies to my retail home customers too.

Dave
Re: QUIC traffic throttled on AT&T residential [ In reply to ]
On Wed, Feb 19, 2020 at 3:18 AM Masataka Ohta
<mohta@necom830.hpcl.titech.ac.jp> wrote:
>
> Christopher Morrow wrote:
>
> > 2 way flow means something on your home host or home gateway.
> > It means very little at internet scale... since, in many cases, you ->
> > server and server -> you are not sharing many of the same links /
> > routers / etc.
>
> Subject suggests it's retail ISP to homes, which are unlikely to
> be multihomed.

It probably depends on where you want to do such limiting, right?
"At peering/transit edge" - save your core, dont' carry traffic you
"know" you will throw away anyway.
"At the customer edge" - scaling of state management could be
problematic && you'll carry this 'bad' traffic across your network.

Note: I'm not really casting aspersions on either part of the whole
argument here, just attempting to provide some color to the parts
which seem 'unlikely to be successful'

I think for a bunch of this discussion there are N folks with M goals,
and eventually 'the market will dictate' which final direction we
travel.
Re: QUIC traffic throttled on AT&T residential [ In reply to ]
On 2020-02-19 1:06 a.m., Masataka Ohta wrote:
> Are you saying AT&T should block UDP entirely?

No; while I don't presume to have all the answers they should at the
minimum take into account how it affects the end-user (CUSTOMER!)
experience when making decisions like this.

(No they shouldn't block UDP entirely, but if they're having UDP/443
DDOS problems then blocking it while they get a proper scalable solution
in place is better than throttling it).
Re: QUIC traffic throttled on AT&T residential [ In reply to ]
Christopher Morrow wrote:

>>> 2 way flow means something on your home host or home gateway.
>>> It means very little at internet scale... since, in many cases, you ->
>>> server and server -> you are not sharing many of the same links /
>>> routers / etc.
>>
>> Subject suggests it's retail ISP to homes, which are unlikely to
>> be multihomed.
>
> It probably depends on where you want to do such limiting, right?
> "At peering/transit edge" - save your core, dont' carry traffic you
> "know" you will throw away anyway.
> "At the customer edge" - scaling of state management could be
> problematic && you'll carry this 'bad' traffic across your network.

Limiting is not by edges but by ISPs.

In transit ISPs, there are numerous flows generated by customers
of other ISPs that customer wise rate limiting does not scale.

In access ISPs, revenue increases proportional to the number
of customers that you can prepare rate limiting devices proportional
to the number of customers to make customer wise rate limiting scale.

Masataka Ohta
Re: QUIC traffic throttled on AT&T residential [ In reply to ]
On 2/18/2020 6:00 PM, Ca By wrote:
> On Tue, Feb 18, 2020 at 5:44 PM Daniel Sterling
> <sterling.daniel@gmail.com <mailto:sterling.daniel@gmail.com>> wrote:
>
> I've AT&T fiber (in RTP, NC) (AS7018) and I notice UDP QUIC traffic
> from google (esp. youtube) becomes very slow after a time.
>
> This especially occurs with ipv4 connections. I'm not the only one to
> notice; a web search for e.g. "Extremely Poor Youtube TV Performance"
> notes the issue.
>
> I assume traffic is being throttled on AT&T's side. And it's not done
> with their CPE; I'm bypassing their NAT box and connecting my laptop
> directly to the ONT.
>
> A quick google search shows people are aware that QUIC can look like
> DoS traffic -- but how widely known is this problem? It may become
> worse if / when sites transition to HTTP/3
>
> For now I reject UDP 443 on the client firewall. Applications using
> QUIC client libraries then fallback to TCP. This works well and I have
> no issues with latency or throughput when using TCP.
>
> May I naively ask if Google staff are working with AT&T to address
> this?
>
>
> Sounds like you found the answer, ATT just needs to scale up your
> rejection approach that is proven to work well.
>
> Yes, most ddos traffic is ipv4 udp and yes Google was made aware that
> they would be mixing with bad company in the pool of ipv4 udp traffic
> ... but they have reasons.
>
> I am not a fan of quic or any udp traffic. My suggestion was that
> Google use an new L4 instead of UDP, but that was too hard for the
> Googlers.
>
> So here we are.
>
> Just say no to udp.

Isn't this exactly why Net Neutrality is a thing: So that people (or
companies) are free to develop new applications or enhance existing ones
without running into a quagmire of different policies implemented by any
number of different networks between the application developer and the
application's users?

For what it's worth, Cox cable also treats UDP traffic differently -
impacting the performance of VPNs using UDP. Cox provides a list of
blocked ports on their website, but makes no mention that they throttle
or otherwise treat UDP as a special case. I'm guessing ATT doesn't
disclose this policy transparently either.
Re: QUIC traffic throttled on AT&T residential [ In reply to ]
> On Feb 18, 2020, at 4:00 PM, Ca By <cb.list6@gmail.com> wrote:
>
> I am not a fan of quic or any udp traffic. My suggestion was that Google use an new L4 instead of UDP, but that was too hard for the Googlers.

The argument I have heard is that residential firewalls often block anything that is *not* UDP or TCP. The question for the googlers was existential - can it work at all?

1 2 3  View All