Mailing List Archive

"potential new and different architectural approach" to solve the Comcast - L3 dispute
Since it is Friday, maybe some of peering experts have some time to
speculate what this new approach proposed by Comcast might be, as they
assert it would represent "a significant shift of Internet infrastructure."

http://www.lightreading.com/document.asp?doc_id=202121
http://blog.comcast.com/2010/12/comcast-continues-discussions-with-level-3----offers-to-trial-new-solutions.html

Well, their previous proposal was already representing quite a
significant shift, and not in a good way, but I wonder what the new
offer could be so that they figured it would be more acceptable to Level 3.

I hope due to the speculative nature of the question it will not be
considered off-topic.

-Lorand Jakab
Re: "potential new and different architectural approach" to solve the Comcast - L3 dispute [ In reply to ]
On Dec 17, 2010, at 9:57 AM, Loránd Jakab wrote:

> Since it is Friday, maybe some of peering experts have some time to
> speculate what this new approach proposed by Comcast might be, as they
> assert it would represent "a significant shift of Internet infrastructure."
>
> http://www.lightreading.com/document.asp?doc_id=202121
> http://blog.comcast.com/2010/12/comcast-continues-discussions-with-level-3----offers-to-trial-new-solutions.html

I have no direct knowledge of the situation, but my guess: I suspect the proposal was along the lines of longest-path / best-exit routing by Level(3). In other words, if L(3) carries the traffic (most of the way) to the customer, then Comcast has no complaint--the costs can be more fairly distributed. The "modest investment" is probably in tools to evaluate traffic and routing metrics, to make this work. This isn't really *new* to the peering community, but it isn't normal either.

If anybody knows for sure, I'd be interested to hear.

Cheers,
-Benson
Re: "potential new and different architectural approach" to solve the [ In reply to ]
> On Dec 17, 2010, at 9:57 AM, Lor=E1nd Jakab wrote:
> > Since it is Friday, maybe some of peering experts have some time to
> > speculate what this new approach proposed by Comcast might be, as they
> > assert it would represent "a significant shift of Internet =
> infrastructure."
> >=20
> > http://www.lightreading.com/document.asp?doc_id=3D202121
> > =
> http://blog.comcast.com/2010/12/comcast-continues-discussions-with-level-3=
> ----offers-to-trial-new-solutions.html
>
> I have no direct knowledge of the situation, but my guess: I suspect =
> the proposal was along the lines of longest-path / best-exit routing by =
> Level(3). In other words, if L(3) carries the traffic (most of the way) =
> to the customer, then Comcast has no complaint--the costs can be more =
> fairly distributed. The "modest investment" is probably in tools to =
> evaluate traffic and routing metrics, to make this work. This isn't =
> really *new* to the peering community, but it isn't normal either.
>
> If anybody knows for sure, I'd be interested to hear.

How effective have variations on hot potato routing been, historically?
I seem to recall Cogent made lots of noises early on about how they
could do hot potato routing to encourage peering, but over the years
that didn't seem to pan out that way.

... JG
--
Joe Greco - sol.net Network Services - Milwaukee, WI - http://www.sol.net
"We call it the 'one bite at the apple' rule. Give me one chance [and] then I
won't contact you again." - Direct Marketing Ass'n position on e-mail spam(CNN)
With 24 million small businesses in the US alone, that's way too many apples.
Re: "potential new and different architectural approach" to solve the [ In reply to ]
On Dec 17, 2010, at 11:23 AM, Joe Greco wrote:

> How effective have variations on hot potato routing been, historically?
> I seem to recall Cogent made lots of noises early on about how they
> could do hot potato routing to encourage peering, but over the years
> that didn't seem to pan out that way.

I can't comment on Cogent... But, in general: hot-potato reduces network costs but doesn't eliminate them--more capacity is still required to carry more traffic. The goal is to balance out the cost, assuming the traffic is of adequate value (or equal value, ideally) to both networks.

Cheers,
-Benson
Re: "potential new and different architectural approach" to solve the Comcast - L3 dispute [ In reply to ]
On Fri, Dec 17, 2010 at 12:15 PM, Benson Schliesser
<bensons@queuefull.net> wrote:
> I have no direct knowledge of the situation, but my guess:  I suspect the proposal was along the lines of longest-path / best-exit routing by Level(3).  In other words, if L(3) carries the traffic (most of the way) to the customer, then Comcast has no complaint--the costs can be more fairly distributed.  The "modest investment" is probably in tools to evaluate traffic and routing metrics, to make this work.  This isn't really *new* to the peering community, but it isn't normal either.

That is a reasonable guess, but Level3's FCC filing yesterday spells
out with certainty that Level3 did offer to "cold potato" traffic onto
Comcast (it does not mention the technical means e.g. MED honoring,
CDN smarts, or otherwise) and that Comcast refused.

I agree that the proposed Comcast solution may not be truly "new" but
instead unusual, but unless "Backdoor Santa" tells us what they really
have in mind, I suppose we won't know. If I were Comcast, I would
want to move the significant cost of detailed netflow collection and
analysis infrastructure onto backbone providers by wrapping that
accounting mechanism up into my settlement agreements with peers, as
well as the expense of a cost-ineffective network, and demand that
Level3 and Comcast really calculate how much each network spends on
each bit, and share in that cost. In theory, this is what happens
when an ILEC opens a rate case with its state regulator; and it is how
settlements for POTS calls work (at a very basic level.)

Actually, if I were Comcast, I would focus on running my business more
efficiently, as Level3 has thrown down the gauntlet with the FCC and
requested that the FCC dictate to Comcast specifically, and explicitly
all other broadband access providers, how they will interconnect with
peers and transit suppliers. Level3 must think that their business
would be better off with regulatory oversight of peering, or they
would not have taken this action. Comcast should realize that, of the
three potential motives for their recent actions I have previously
outlined, #1 and #3 are not just highly unlikely, but would be
practically impossible in a regulated environment. As such, they
should further realize that their peering committee is driven by
motive #2, ego, and find the best way to change their position without
losing too much credibility.

--
Jeff S Wheeler <jsw@inconcepts.biz>
Sr Network Operator  /  Innovative Network Concepts
Re: "potential new and different architectural approach" to solve the Comcast - L3 dispute [ In reply to ]
On Fri, Dec 17, 2010 at 11:15:14AM -0600, Benson Schliesser wrote:
>
> I have no direct knowledge of the situation, but my guess: I suspect
> the proposal was along the lines of longest-path / best-exit routing
> by Level(3). In other words, if L(3) carries the traffic (most of the
> way) to the customer, then Comcast has no complaint--the costs can be
> more fairly distributed. The "modest investment" is probably in tools
> to evaluate traffic and routing metrics, to make this work. This
> isn't really *new* to the peering community, but it isn't normal
> either.

Nah, you're still thinking about this like it was a classic peering
dispute over ratios, when nothing could be further from the truth. First
off, by the very nature of a CDN, all of the Netflix/etc traffic is
going to be delivered to the best exit on the long-haul network already.
Second, Comcast is a FULL TRANSIT CUSTOMER of Level 3. Typically the
customer gets to dictate the handoff point to the provider, by either
advertising MEDs, or by sending inconsistent routes. The fact that the
existing Level3/Comcast routing DOESN'T make Level 3 haul all of the
bits to the best exit mean it's highly likely that Comcast agreeing to
haul the bits was part of their commercial transit agreement, probably
in exchange for lower transit prices.

--
Richard A Steenbergen <ras@e-gerbil.net> http://www.e-gerbil.net/ras
GPG Key ID: 0xF8B12CBC (7535 7F59 8204 ED1F CC1C 53AF 4C41 5ECA F8B1 2CBC)
Re: "potential new and different architectural approach" to solve the Comcast - L3 dispute [ In reply to ]
On Dec 17, 2010, at 11:35 AM, Jeff Wheeler wrote:

> ... Level3 must think that their business
> would be better off with regulatory oversight of peering, or they
> would not have taken this action.

And they might be correct in thinking that, if we assume the peering ecosystem is changing i.e. such that traditional "backbones" are being bypassed. Regulatory oversight might have the effect of locking-in today's interconnect regime, which would be ideal for Level(3).

Cheers,
-Benson
Re: "potential new and different architectural approach" to solve the Comcast - L3 dispute [ In reply to ]
On Fri, Dec 17, 2010 at 12:48 PM, Richard A Steenbergen
<ras@e-gerbil.net> wrote:
> advertising MEDs, or by sending inconsistent routes. The fact that the
> existing Level3/Comcast routing DOESN'T make Level 3 haul all of the
> bits to the best exit mean it's highly likely that Comcast agreeing to
> haul the bits was part of their commercial transit agreement, probably
> in exchange for lower transit prices.

It's worth asking why Comcast did not accept Level3's suggestion that
they use MED as a face-saving maneuver, which would have allowed both
sides to declare victory.

A) Comcast may already have the contractual right to use MED but
chooses not to. I agree with you that this is unlikely, not for pure
reasons of economics, but because Comcast has some of the same set of
motives not to send MED to their transit provider as every other
network: prefix aggregation, quality control, and ego. I'll discount
geography, marketing, and inability to calculate useful MED values.
For argument's sake, let's say they currently can start sending MEDs
to Level3 whenever they want. This being the case, Level3's "offer"
would have amounted to Level3 telling Comcast upper management that
Comcast's engineering people are leaving a huge amount of money on the
table, that Level3 is far more cost-effective at running its long-haul
network than Comcast, and that they should leave the big networking to
the big boys. Comcast management could either react badly to this, or
go back to their network folks and ask why they can't be as
cost-effective as Level3.

B) Comcast may not be able to use MED today. In this case, management
may be asking themselves why. An essentially similar scenario can
play out; they can either react badly to Level3, or ask their own
staff why they are wasting money.

C) Comcast doesn't care about MED or the actual cost of doing
business. They are boldly moving towards a future that is opposite
the one "net neutrality" folks advocate, one that looks like my
"Comcast Motive #3."

D) Comcast does not think that beginning to use MED (whether currently
enabled or not) is enough to satisfy the federal regulators and
legislators who are now taking interest in this game of
interconnection brinkmanship, involving 17 million households, between
a major IP carrier delivering traffic from everyone including a
household name like Netflix, and a major cable company that is waiting
for government approval to purchase NBC. They feel they must demand
something very concrete to demonstrate that they are looking out for
consumers' best interest, which means they must make Level3 and/or
Netflix look like the bad guy.

E) Comcast thinks that a system of accounting for the cost of bearing
traffic and dividing it among the involved parties will actually be
good for their business, because they can over-build their
infrastructure as much as they like, perhaps even improving quality
for end-users, and only have to pay for about half of it. The cost of
being inefficient, stupid, or committing purchasing or forecasting
errors drops by half. This looks very much like my "Comcast Motive
#1."
E1) Comcast may also know a thing or two about Hollywood Accounting.
If you do not understand this reference, simply look it up on
Wikipedia. It suffices to say that cost/revenue sharing agreements of
this nature can be manipulated in gross ways to the advantage of the
party doing the bulk of the book-keeping.

F) Management has the same case of ego-driven decision-making that
their technical staff have demonstrated. I find this unlikely but
still possible. We all know this has been the case at the CEO level
in some major interconnection disputes of the past.

I believe this outlines the reasonable scenarios for Comcast avoiding
a face-saving maneuver with Level3.

--
Jeff S Wheeler <jsw@inconcepts.biz>
Sr Network Operator  /  Innovative Network Concepts
Re: "potential new and different architectural approach" to solve the Comcast - L3 dispute [ In reply to ]
On Dec 17, 2010, at 12:35 PM, Jeff Wheeler wrote:
> On Fri, Dec 17, 2010 at 12:15 PM, Benson Schliesser
> <bensons@queuefull.net> wrote:
>> I have no direct knowledge of the situation, but my guess: I suspect the proposal was along the lines of longest-path / best-exit routing by Level(3). In other words, if L(3) carries the traffic (most of the way) to the customer, then Comcast has no complaint--the costs can be more fairly distributed. The "modest investment" is probably in tools to evaluate traffic and routing metrics, to make this work. This isn't really *new* to the peering community, but it isn't normal either.
>
> That is a reasonable guess, but Level3's FCC filing yesterday spells
> out with certainty that Level3 did offer to "cold potato" traffic onto
> Comcast (it does not mention the technical means e.g. MED honoring,
> CDN smarts, or otherwise) and that Comcast refused.
> [...]

Comcast's latest:
http://fjallfoss.fcc.gov/ecfs/comment/view?id=6016064677
Re: "potential new and different architectural approach" to solve the Comcast - L3 dispute [ In reply to ]
http://fcc.gov/

NOTICE: The FCC website and related electronic filing systems and documents
(except for NORS) will be unavailable beginning 6:00 p.m. (EST) Friday,
December 17 through 6:00 a.m. (EST) Monday, December 20 for scheduled
maintenance.


:(


On Fri, Dec 17, 2010 at 3:42 PM, Steve Schultze <sjs@princeton.edu> wrote:

>
> > That is a reasonable guess, but Level3's FCC filing yesterday spells
> > out with certainty that Level3 did offer to "cold potato" traffic onto
> > Comcast (it does not mention the technical means e.g. MED honoring,
> > CDN smarts, or otherwise) and that Comcast refused.
> > [...]
>
> Comcast's latest:
> http://fjallfoss.fcc.gov/ecfs/comment/view?id=6016064677
>



--
---------------------------------------------------------------
Joly MacFie 218 565 9365 Skype:punkcast
WWWhatsup NYC - http://wwwhatsup.com
http://pinstand.com - http://punkcast.com
VP (Admin) - ISOC-NY - http://isoc-ny.org
---------------------------------------------------------------
Re: "potential new and different architectural approach" to solve the Comcast - L3 dispute [ In reply to ]
http://blog.comcast.com/2010/12/comcasts-responds-to-level-3s-fcc-filing.html

On Dec 17, 2010, at 10:25 PM, Joly MacFie wrote:
> http://fcc.gov/
>
> NOTICE: The FCC website and related electronic filing systems and documents (except for NORS) will be unavailable beginning 6:00 p.m. (EST) Friday, December 17 through 6:00 a.m. (EST) Monday, December 20 for scheduled maintenance.
>
>
> :(
>
>
> On Fri, Dec 17, 2010 at 3:42 PM, Steve Schultze <sjs@princeton.edu> wrote:
>
> > That is a reasonable guess, but Level3's FCC filing yesterday spells
> > out with certainty that Level3 did offer to "cold potato" traffic onto
> > Comcast (it does not mention the technical means e.g. MED honoring,
> > CDN smarts, or otherwise) and that Comcast refused.
> > [...]
>
> Comcast's latest:
> http://fjallfoss.fcc.gov/ecfs/comment/view?id=6016064677
>
>
>
> --
> ---------------------------------------------------------------
> Joly MacFie 218 565 9365 Skype:punkcast
> WWWhatsup NYC - http://wwwhatsup.com
> http://pinstand.com - http://punkcast.com
> VP (Admin) - ISOC-NY - http://isoc-ny.org
> ---------------------------------------------------------------
Re: "potential new and different architectural approach" to solve the Comcast - L3 dispute [ In reply to ]
On 12/18/2010 12:38 AM, Steve Schultze wrote:
> http://blog.comcast.com/2010/12/comcasts-responds-to-level-3s-fcc-filing.html
>

I very much doubt whether my comment on the blog will survive their
moderation process, so here it is:

===
I am a Comcast residential HSI customer, and have many clients who are
business HSI Comcast customers. At the same time, I do maintain servers
in my own racks at a datacenter.

What is not mentioned in this letter, is that Comcast is already being
paid - by me, and by every other customer, for access to the content.

Note that Comcast has never said that the Level3/Netflix issue is about
users exceeding their allotted bandwidth (currently at about 250GB/month
for residential); presumably, were a Comcast user to use 249GB of
bandwidth downloading cute pictures of cats, Comcast would have no
objection.

It appears to be the specific issue that Netflix is a possible
competitor to Comcast's TV business, that somehow causes Comcast to
decide that there is a problem.

Understand this: every Netflix video to be streamed, is specifically
requested by a Comcast user, operating under the Comcast-advertised
"High Speed Internet" service and presumably within the bandwidth caps
that Comcast's own contract allows.

That Comcast presumes to have the right to limit, modify, or decide for
me which pieces of the Internet I can have access to, removes Comcast's
common carrier protections, calls into question the truth of your
advertisements for the HSI service, and raises the issue of whether
Comcast is dealing in bad faith with each and every Comcast HSI subscriber.

====

--Patrick
Re: "potential new and different architectural approach" to solve the Comcast - L3 dispute [ In reply to ]
On Sat, Dec 18, 2010 at 01:07:15AM -0500, Patrick Giagnocavo wrote:
>
> Note that Comcast has never said that the Level3/Netflix issue is
> about users exceeding their allotted bandwidth (currently at about
> 250GB/month for residential); presumably, were a Comcast user to use
> 249GB of bandwidth downloading cute pictures of cats, Comcast would
> have no objection.

I believe they want the cat people to pay too, it's just easier to go
after Netflix first.

Lets say for a moment that Comcast's overall ratio with its customers is
approximately the same as their ratio in the leaked Tata graphs (yes I
know that this proves nothing, but lets just assume it for a moment),
i.e. 5:1. They then ask that every network who sends them traffic, even
their transit providers (in the case of Level 3) be under 2:1. What is
the point of insisting on a ratio that is not supported by the traffic
their customers actually request? Because it gives them a convenient
excuse to demand payment from nearly everyone on the Internet for being
out of ratio, and to restrict capacity to those who do not pay.

With so many transit ports running hot, and even peering ports running
hot as in the recent example where they intentionally turned down Global
Crossing capacity (which they claim is settlement free) and CAUSED
congestion, the ISP who hosts the cute cat pictures may have little
choice but to pay Comcast for access, or risk losing their cute cat
hosting business to someone else who is willing to do so.

I've also seen Comcast ignore several offers to honor MEDs or accept
more-specifics from networks who DO meet their published peering
requirements in every way except ratios, so I don't think they're
interested in technical solutions a potential transport cost imbalance
either. If it was about anything other than trying to extract a toll
from content providers, one of these technical solutions would clearly
have been better for them then continuing to force the traffic into
their congested transit ports, which they not only pay for, but then
also do the backhaul for across their own network.

BTW, they rejected my very nice comment on their blog asking if they
would be willing to share the graphs of their transit provider
interfaces (which are NOT peering relationships, and not under NDA) to
back up their claims that the published graphs are false, so I'm
positive yours isn't going to get through. :)

--
Richard A Steenbergen <ras@e-gerbil.net> http://www.e-gerbil.net/ras
GPG Key ID: 0xF8B12CBC (7535 7F59 8204 ED1F CC1C 53AF 4C41 5ECA F8B1 2CBC)
Re: "potential new and different architectural approach" to solve the Comcast - L3 dispute [ In reply to ]
If I was Comcast and I got this deal I'd set up scripts to continuously
spoof requests to Netflix, I mean hey I get paid for the traffic..

j

--
---------------------------------------------------------------
Joly MacFie 218 565 9365 Skype:punkcast
WWWhatsup NYC - http://wwwhatsup.com
http://pinstand.com - http://punkcast.com
VP (Admin) - ISOC-NY - http://isoc-ny.org
---------------------------------------------------------------
Re: "potential new and different architectural approach" to solve the Comcast - L3 dispute [ In reply to ]
+1

In fact, I feel that at home, I need fast, reliable internet access. I wish I could
get that from one provider. Unfortunately, instead, I get fast internet service
from Comcast (most of the time) and I get reliable internet service from
Raw Bandwidth (DSL, 1.5mbps/768k).

Owen
(Comcast Business HSI customer)

On Dec 17, 2010, at 10:07 PM, Patrick Giagnocavo wrote:

> On 12/18/2010 12:38 AM, Steve Schultze wrote:
>> http://blog.comcast.com/2010/12/comcasts-responds-to-level-3s-fcc-filing.html
>>
>
> I very much doubt whether my comment on the blog will survive their
> moderation process, so here it is:
>
> ===
> I am a Comcast residential HSI customer, and have many clients who are
> business HSI Comcast customers. At the same time, I do maintain servers
> in my own racks at a datacenter.
>
> What is not mentioned in this letter, is that Comcast is already being
> paid - by me, and by every other customer, for access to the content.
>
> Note that Comcast has never said that the Level3/Netflix issue is about
> users exceeding their allotted bandwidth (currently at about 250GB/month
> for residential); presumably, were a Comcast user to use 249GB of
> bandwidth downloading cute pictures of cats, Comcast would have no
> objection.
>
> It appears to be the specific issue that Netflix is a possible
> competitor to Comcast's TV business, that somehow causes Comcast to
> decide that there is a problem.
>
> Understand this: every Netflix video to be streamed, is specifically
> requested by a Comcast user, operating under the Comcast-advertised
> "High Speed Internet" service and presumably within the bandwidth caps
> that Comcast's own contract allows.
>
> That Comcast presumes to have the right to limit, modify, or decide for
> me which pieces of the Internet I can have access to, removes Comcast's
> common carrier protections, calls into question the truth of your
> advertisements for the HSI service, and raises the issue of whether
> Comcast is dealing in bad faith with each and every Comcast HSI subscriber.
>
> ====
>
> --Patrick
Re: "potential new and different architectural approach" to solve the Comcast - L3 dispute [ In reply to ]
Richard A Steenbergen wrote:
>
> BTW, they rejected my very nice comment on their blog asking if they
> would be willing to share the graphs of their transit provider
> interfaces (which are NOT peering relationships, and not under NDA) to
> back up their claims that the published graphs are false, so I'm
> positive yours isn't going to get through. :)
>
>
Seems as though, in both this case and Steve's case, Comcast is going
out of their way to spin as much FUD as they can against those who dare
speak out and are making a concerted effort to censor their "blog" (ie,
press release machine).

It's not going unnoticed and I hope they realize that instead of
commenting on their blog, "we" will turn to the official FCC comment
process.

-Dave