Mailing List Archive

Prefix hijacking by AS20115
I've got a problem where AS20115 continues to announce prefixes after
BGP neighbors were shutdown. They claim it's a wedged BGP process but
aren't in any hurry to fix it outside of a maintenance window.

I'm at a loss of what else I can do. They admit the problem but won't
take action saying it needs to wait for a maintenance window. Am I out
of line insisting that's an unacceptable response to a problem that
results in prefix/traffic hijacking?

~Seth
Re: Prefix hijacking by AS20115 [ In reply to ]
On Mon, Sep 28, 2015 at 9:01 PM, Seth Mattinen <sethm@rollernet.us> wrote:
> I've got a problem where AS20115 continues to announce prefixes after BGP
> neighbors were shutdown. They claim it's a wedged BGP process but aren't in
> any hurry to fix it outside of a maintenance window.

If they weren't lying to you, they'd fix it now. That's not the kind
of problem that waits.

Thing is: they lied to you. Long ago they "helpfully" programmed their
router to announce your route regardless of whether you sent a route
to them. They want to wait for a maintenance window to remove that
configuration.


> I'm at a loss of what else I can do. They admit the problem but won't take
> action saying it needs to wait for a maintenance window. Am I out of line
> insisting that's an unacceptable response to a problem that results in
> prefix/traffic hijacking?

Try dropping the link entirely. If they still announce your addresses,
bring it back up but report it as emergency down, escalate, and call
back every 10 minutes until the junior tech understands that it's time
to call and wake up the guy who makes the decision to fix it now.

Regards,
Bill Herrin



--
William Herrin ................ herrin@dirtside.com bill@herrin.us
Owner, Dirtside Systems ......... Web: <http://www.dirtside.com/>
Re: Prefix hijacking by AS20115 [ In reply to ]
On 9/28/15 18:30, William Herrin wrote:
> On Mon, Sep 28, 2015 at 9:01 PM, Seth Mattinen <sethm@rollernet.us> wrote:
>> I've got a problem where AS20115 continues to announce prefixes after BGP
>> neighbors were shutdown. They claim it's a wedged BGP process but aren't in
>> any hurry to fix it outside of a maintenance window.
>
> If they weren't lying to you, they'd fix it now. That's not the kind
> of problem that waits.
>
> Thing is: they lied to you. Long ago they "helpfully" programmed their
> router to announce your route regardless of whether you sent a route
> to them. They want to wait for a maintenance window to remove that
> configuration.
>
>
>> I'm at a loss of what else I can do. They admit the problem but won't take
>> action saying it needs to wait for a maintenance window. Am I out of line
>> insisting that's an unacceptable response to a problem that results in
>> prefix/traffic hijacking?
>
> Try dropping the link entirely. If they still announce your addresses,
> bring it back up but report it as emergency down, escalate, and call
> back every 10 minutes until the junior tech understands that it's time
> to call and wake up the guy who makes the decision to fix it now.
>


I'm at the tail end here almost 8 hours later since the hijacking
started. Their NOC is just blowing me off now and they're happy to
continue the hijacking until it's convenient for them to have a
maintenance window. And that's apparently the final decision.

~Seth
Re: Prefix hijacking by AS20115 [ In reply to ]
Start announcing their prefixes?

Josh Luthman
Office: 937-552-2340
Direct: 937-552-2343
1100 Wayne St
Suite 1337
Troy, OH 45373
On Sep 28, 2015 11:09 PM, "Seth Mattinen" <sethm@rollernet.us> wrote:

> On 9/28/15 18:30, William Herrin wrote:
>
>> On Mon, Sep 28, 2015 at 9:01 PM, Seth Mattinen <sethm@rollernet.us>
>> wrote:
>>
>>> I've got a problem where AS20115 continues to announce prefixes after BGP
>>> neighbors were shutdown. They claim it's a wedged BGP process but aren't
>>> in
>>> any hurry to fix it outside of a maintenance window.
>>>
>>
>> If they weren't lying to you, they'd fix it now. That's not the kind
>> of problem that waits.
>>
>> Thing is: they lied to you. Long ago they "helpfully" programmed their
>> router to announce your route regardless of whether you sent a route
>> to them. They want to wait for a maintenance window to remove that
>> configuration.
>>
>>
>> I'm at a loss of what else I can do. They admit the problem but won't take
>>> action saying it needs to wait for a maintenance window. Am I out of line
>>> insisting that's an unacceptable response to a problem that results in
>>> prefix/traffic hijacking?
>>>
>>
>> Try dropping the link entirely. If they still announce your addresses,
>> bring it back up but report it as emergency down, escalate, and call
>> back every 10 minutes until the junior tech understands that it's time
>> to call and wake up the guy who makes the decision to fix it now.
>>
>>
>
> I'm at the tail end here almost 8 hours later since the hijacking started.
> Their NOC is just blowing me off now and they're happy to continue the
> hijacking until it's convenient for them to have a maintenance window. And
> that's apparently the final decision.
>
> ~Seth
>
Re: Prefix hijacking by AS20115 [ In reply to ]
Is this related to 104.73.161.0/24? That's ours. :-)

We'll take a look and get back to you. Thanks for caring!

Best,

Marty

> On Sep 28, 2015, at 23:08, Seth Mattinen <sethm@rollernet.us> wrote:
>
>> On 9/28/15 18:30, William Herrin wrote:
>>> On Mon, Sep 28, 2015 at 9:01 PM, Seth Mattinen <sethm@rollernet.us> wrote:
>>> I've got a problem where AS20115 continues to announce prefixes after BGP
>>> neighbors were shutdown. They claim it's a wedged BGP process but aren't in
>>> any hurry to fix it outside of a maintenance window.
>>
>> If they weren't lying to you, they'd fix it now. That's not the kind
>> of problem that waits.
>>
>> Thing is: they lied to you. Long ago they "helpfully" programmed their
>> router to announce your route regardless of whether you sent a route
>> to them. They want to wait for a maintenance window to remove that
>> configuration.
>>
>>
>>> I'm at a loss of what else I can do. They admit the problem but won't take
>>> action saying it needs to wait for a maintenance window. Am I out of line
>>> insisting that's an unacceptable response to a problem that results in
>>> prefix/traffic hijacking?
>>
>> Try dropping the link entirely. If they still announce your addresses,
>> bring it back up but report it as emergency down, escalate, and call
>> back every 10 minutes until the junior tech understands that it's time
>> to call and wake up the guy who makes the decision to fix it now.
>
>
> I'm at the tail end here almost 8 hours later since the hijacking started. Their NOC is just blowing me off now and they're happy to continue the hijacking until it's convenient for them to have a maintenance window. And that's apparently the final decision.
>
> ~Seth
Re: Prefix hijacking by AS20115 [ In reply to ]
On 9/28/15 20:19, Martin Hannigan wrote:
>
> Is this related to 104.73.161.0/24? That's ours. :-)
>
> We'll take a look and get back to you. Thanks for caring!
>


Yep, that's one of the affected prefixes.

~Seth
Re: Prefix hijacking by AS20115 [ In reply to ]
That's something I would do. Announce announce and keep adding ports until
I hit a 10 Gig port worth of traffic or saw it fixed. Be sure to put in a
blackhole route for the prefixes. Try to pick blocks that are as
geographically located to your peering routers as possible ...IE in Reno
pick the blocks that seem to be near by - like Reno, Tahoe, Sacramento
..... when that batch of customers makes their phones ring all night
someone will listen.

Would be nice if our membership organization ARIN ( that we all pay to
keep us somewhat organized) had an ability to do something for you.... I
never looked into it...i don't know....maybe it does ?

But, in the mean time I am pretty sure you can document this well and
prove your announcements of theirs was due to the fact you couldn't get
proper technical attention and needed to desperately before your customers
cancel after 8 hours of this. Tomorrow call your lawyers and begin to sue
that cable company (did I recognize that ASN as cable TV ? ) for damages
this must be causing you in ill-will amongst your customer base.

I wonder just how you prove the damage...some equation based on customer
calls and complaints together with how many years you have been in
business as well as the number of contracts that are coming up for
renewal. etc etc. Now that would be interesting to see a formula for that
if anyone has been through it.

Thank You
Bob Evans
CTO




> Start announcing their prefixes?
>
> Josh Luthman
> Office: 937-552-2340
> Direct: 937-552-2343
> 1100 Wayne St
> Suite 1337
> Troy, OH 45373
> On Sep 28, 2015 11:09 PM, "Seth Mattinen" <sethm@rollernet.us> wrote:
>
>> On 9/28/15 18:30, William Herrin wrote:
>>
>>> On Mon, Sep 28, 2015 at 9:01 PM, Seth Mattinen <sethm@rollernet.us>
>>> wrote:
>>>
>>>> I've got a problem where AS20115 continues to announce prefixes after
>>>> BGP
>>>> neighbors were shutdown. They claim it's a wedged BGP process but
>>>> aren't
>>>> in
>>>> any hurry to fix it outside of a maintenance window.
>>>>
>>>
>>> If they weren't lying to you, they'd fix it now. That's not the kind
>>> of problem that waits.
>>>
>>> Thing is: they lied to you. Long ago they "helpfully" programmed their
>>> router to announce your route regardless of whether you sent a route
>>> to them. They want to wait for a maintenance window to remove that
>>> configuration.
>>>
>>>
>>> I'm at a loss of what else I can do. They admit the problem but won't
>>> take
>>>> action saying it needs to wait for a maintenance window. Am I out of
>>>> line
>>>> insisting that's an unacceptable response to a problem that results in
>>>> prefix/traffic hijacking?
>>>>
>>>
>>> Try dropping the link entirely. If they still announce your addresses,
>>> bring it back up but report it as emergency down, escalate, and call
>>> back every 10 minutes until the junior tech understands that it's time
>>> to call and wake up the guy who makes the decision to fix it now.
>>>
>>>
>>
>> I'm at the tail end here almost 8 hours later since the hijacking
>> started.
>> Their NOC is just blowing me off now and they're happy to continue the
>> hijacking until it's convenient for them to have a maintenance window.
>> And
>> that's apparently the final decision.
>>
>> ~Seth
>>
>
Re: Prefix hijacking by AS20115 [ In reply to ]
At 23:11 28/09/2015 -0400, Josh Luthman wrote:

>Start announcing their prefixes?

Contact the upstreams of AS20115 - Cogent, Level3, HE and XO.

-Hank


>Josh Luthman
>Office: 937-552-2340
>Direct: 937-552-2343
>1100 Wayne St
>Suite 1337
>Troy, OH 45373
>On Sep 28, 2015 11:09 PM, "Seth Mattinen" <sethm@rollernet.us> wrote:
>
> > On 9/28/15 18:30, William Herrin wrote:
> >
> >> On Mon, Sep 28, 2015 at 9:01 PM, Seth Mattinen <sethm@rollernet.us>
> >> wrote:
> >>
> >>> I've got a problem where AS20115 continues to announce prefixes after BGP
> >>> neighbors were shutdown. They claim it's a wedged BGP process but aren't
> >>> in
> >>> any hurry to fix it outside of a maintenance window.
> >>>
> >>
> >> If they weren't lying to you, they'd fix it now. That's not the kind
> >> of problem that waits.
> >>
> >> Thing is: they lied to you. Long ago they "helpfully" programmed their
> >> router to announce your route regardless of whether you sent a route
> >> to them. They want to wait for a maintenance window to remove that
> >> configuration.
> >>
> >>
> >> I'm at a loss of what else I can do. They admit the problem but won't take
> >>> action saying it needs to wait for a maintenance window. Am I out of line
> >>> insisting that's an unacceptable response to a problem that results in
> >>> prefix/traffic hijacking?
> >>>
> >>
> >> Try dropping the link entirely. If they still announce your addresses,
> >> bring it back up but report it as emergency down, escalate, and call
> >> back every 10 minutes until the junior tech understands that it's time
> >> to call and wake up the guy who makes the decision to fix it now.
> >>
> >>
> >
> > I'm at the tail end here almost 8 hours later since the hijacking started.
> > Their NOC is just blowing me off now and they're happy to continue the
> > hijacking until it's convenient for them to have a maintenance window. And
> > that's apparently the final decision.
> >
> > ~Seth
> >
Re: Prefix hijacking by AS20115 [ In reply to ]
On Mon, Sep 28, 2015 at 11:59 PM, Bob Evans <bob@fiberinternetcenter.com> wrote:
> That's something I would do. Announce announce and keep adding ports until
> I hit a 10 Gig port worth of traffic or saw it fixed. Be sure to put in a
> blackhole route for the prefixes. Try to pick blocks that are as
> geographically located to your peering routers as possible ...IE in Reno
> pick the blocks that seem to be near by - like Reno, Tahoe, Sacramento
> ..... when that batch of customers makes their phones ring all night
> someone will listen.
>

that seems like a pretty poor strategy... guaranteed to get you into
some hot water, I suspect. Keep in mind that the 'noc' at 20115 isn't
the same thing as the customer-service-center. There's likely little
to link the 2 things together there :(

> Would be nice if our membership organization ARIN ( that we all pay to
> keep us somewhat organized) had an ability to do something for you.... I
> never looked into it...i don't know....maybe it does ?

arin does not guarantee 'routability' of netblocks assigned to your org.

> But, in the mean time I am pretty sure you can document this well and
> prove your announcements of theirs was due to the fact you couldn't get
> proper technical attention and needed to desperately before your customers
> cancel after 8 hours of this. Tomorrow call your lawyers and begin to sue
> that cable company (did I recognize that ASN as cable TV ? ) for damages
> this must be causing you in ill-will amongst your customer base.
>
> I wonder just how you prove the damage...some equation based on customer
> calls and complaints together with how many years you have been in
> business as well as the number of contracts that are coming up for
> renewal. etc etc. Now that would be interesting to see a formula for that
> if anyone has been through it.
>

you COULD find a charter person on-list...there are nine names on the
attendees list for the upcoming meeting... I imagine peeringdb likely
has folk listed... gosh it sure does:

<https://www.peeringdb.com/private/participant_view.php?id=2144>

what with their emails and everything.

> Thank You
> Bob Evans
> CTO
>
>
>
>
>> Start announcing their prefixes?
>>
>> Josh Luthman
>> Office: 937-552-2340
>> Direct: 937-552-2343
>> 1100 Wayne St
>> Suite 1337
>> Troy, OH 45373
>> On Sep 28, 2015 11:09 PM, "Seth Mattinen" <sethm@rollernet.us> wrote:
>>
>>> On 9/28/15 18:30, William Herrin wrote:
>>>
>>>> On Mon, Sep 28, 2015 at 9:01 PM, Seth Mattinen <sethm@rollernet.us>
>>>> wrote:
>>>>
>>>>> I've got a problem where AS20115 continues to announce prefixes after
>>>>> BGP
>>>>> neighbors were shutdown. They claim it's a wedged BGP process but
>>>>> aren't
>>>>> in
>>>>> any hurry to fix it outside of a maintenance window.
>>>>>
>>>>
>>>> If they weren't lying to you, they'd fix it now. That's not the kind
>>>> of problem that waits.
>>>>
>>>> Thing is: they lied to you. Long ago they "helpfully" programmed their
>>>> router to announce your route regardless of whether you sent a route
>>>> to them. They want to wait for a maintenance window to remove that
>>>> configuration.
>>>>
>>>>
>>>> I'm at a loss of what else I can do. They admit the problem but won't
>>>> take
>>>>> action saying it needs to wait for a maintenance window. Am I out of
>>>>> line
>>>>> insisting that's an unacceptable response to a problem that results in
>>>>> prefix/traffic hijacking?
>>>>>
>>>>
>>>> Try dropping the link entirely. If they still announce your addresses,
>>>> bring it back up but report it as emergency down, escalate, and call
>>>> back every 10 minutes until the junior tech understands that it's time
>>>> to call and wake up the guy who makes the decision to fix it now.
>>>>
>>>>
>>>
>>> I'm at the tail end here almost 8 hours later since the hijacking
>>> started.
>>> Their NOC is just blowing me off now and they're happy to continue the
>>> hijacking until it's convenient for them to have a maintenance window.
>>> And
>>> that's apparently the final decision.
>>>
>>> ~Seth
>>>
>>
>
>
Re: Prefix hijacking by AS20115 [ In reply to ]
+1, this is the only sensible advice here.

NSPs actually do seem to care about not letting things like these happen.

On 2015/09/29 01:24 PM, Hank Nussbacher wrote:
> At 23:11 28/09/2015 -0400, Josh Luthman wrote:
>
>> Start announcing their prefixes?
>
> Contact the upstreams of AS20115 - Cogent, Level3, HE and XO.
>
> -Hank
>
>
>> Josh Luthman
>> Office: 937-552-2340
>> Direct: 937-552-2343
>> 1100 Wayne St
>> Suite 1337
>> Troy, OH 45373
>> On Sep 28, 2015 11:09 PM, "Seth Mattinen" <sethm@rollernet.us> wrote:
>>
>> > On 9/28/15 18:30, William Herrin wrote:
>> >
>> >> On Mon, Sep 28, 2015 at 9:01 PM, Seth Mattinen <sethm@rollernet.us>
>> >> wrote:
>> >>
>> >>> I've got a problem where AS20115 continues to announce prefixes
>> after BGP
>> >>> neighbors were shutdown. They claim it's a wedged BGP process but
>> aren't
>> >>> in
>> >>> any hurry to fix it outside of a maintenance window.
>> >>>
>> >>
>> >> If they weren't lying to you, they'd fix it now. That's not the kind
>> >> of problem that waits.
>> >>
>> >> Thing is: they lied to you. Long ago they "helpfully" programmed
>> their
>> >> router to announce your route regardless of whether you sent a route
>> >> to them. They want to wait for a maintenance window to remove that
>> >> configuration.
>> >>
>> >>
>> >> I'm at a loss of what else I can do. They admit the problem but
>> won't take
>> >>> action saying it needs to wait for a maintenance window. Am I out
>> of line
>> >>> insisting that's an unacceptable response to a problem that
>> results in
>> >>> prefix/traffic hijacking?
>> >>>
>> >>
>> >> Try dropping the link entirely. If they still announce your
>> addresses,
>> >> bring it back up but report it as emergency down, escalate, and call
>> >> back every 10 minutes until the junior tech understands that it's
>> time
>> >> to call and wake up the guy who makes the decision to fix it now.
>> >>
>> >>
>> >
>> > I'm at the tail end here almost 8 hours later since the hijacking
>> started.
>> > Their NOC is just blowing me off now and they're happy to continue the
>> > hijacking until it's convenient for them to have a maintenance
>> window. And
>> > that's apparently the final decision.
>> >
>> > ~Seth
>> >
>
RE: Prefix hijacking by AS20115 [ In reply to ]
Cogent and Level3 will tell you that you are not their customer ...HE and XO will react.


Jürgen Jaritsch
Head of Network & Infrastructure

ANEXIA Internetdienstleistungs GmbH

Telefon: +43-5-0556-300
Telefax: +43-5-0556-500

E-Mail: jj@anexia.at
Web: http://www.anexia.at

Anschrift Hauptsitz Klagenfurt: Feldkirchnerstraße 140, 9020 Klagenfurt
Geschäftsführer: Alexander Windbichler
Firmenbuch: FN 289918a | Gerichtsstand: Klagenfurt | UID-Nummer: AT U63216601


-----Original Message-----
From: Paul S. [contact@winterei.se]
Received: Dienstag, 29 Sep. 2015, 6:57
To: nanog@nanog.org [nanog@nanog.org]
Subject: Re: Prefix hijacking by AS20115

+1, this is the only sensible advice here.

NSPs actually do seem to care about not letting things like these happen.

On 2015/09/29 01:24 PM, Hank Nussbacher wrote:
> At 23:11 28/09/2015 -0400, Josh Luthman wrote:
>
>> Start announcing their prefixes?
>
> Contact the upstreams of AS20115 - Cogent, Level3, HE and XO.
>
> -Hank
>
>
>> Josh Luthman
>> Office: 937-552-2340
>> Direct: 937-552-2343
>> 1100 Wayne St
>> Suite 1337
>> Troy, OH 45373
>> On Sep 28, 2015 11:09 PM, "Seth Mattinen" <sethm@rollernet.us> wrote:
>>
>> > On 9/28/15 18:30, William Herrin wrote:
>> >
>> >> On Mon, Sep 28, 2015 at 9:01 PM, Seth Mattinen <sethm@rollernet.us>
>> >> wrote:
>> >>
>> >>> I've got a problem where AS20115 continues to announce prefixes
>> after BGP
>> >>> neighbors were shutdown. They claim it's a wedged BGP process but
>> aren't
>> >>> in
>> >>> any hurry to fix it outside of a maintenance window.
>> >>>
>> >>
>> >> If they weren't lying to you, they'd fix it now. That's not the kind
>> >> of problem that waits.
>> >>
>> >> Thing is: they lied to you. Long ago they "helpfully" programmed
>> their
>> >> router to announce your route regardless of whether you sent a route
>> >> to them. They want to wait for a maintenance window to remove that
>> >> configuration.
>> >>
>> >>
>> >> I'm at a loss of what else I can do. They admit the problem but
>> won't take
>> >>> action saying it needs to wait for a maintenance window. Am I out
>> of line
>> >>> insisting that's an unacceptable response to a problem that
>> results in
>> >>> prefix/traffic hijacking?
>> >>>
>> >>
>> >> Try dropping the link entirely. If they still announce your
>> addresses,
>> >> bring it back up but report it as emergency down, escalate, and call
>> >> back every 10 minutes until the junior tech understands that it's
>> time
>> >> to call and wake up the guy who makes the decision to fix it now.
>> >>
>> >>
>> >
>> > I'm at the tail end here almost 8 hours later since the hijacking
>> started.
>> > Their NOC is just blowing me off now and they're happy to continue the
>> > hijacking until it's convenient for them to have a maintenance
>> window. And
>> > that's apparently the final decision.
>> >
>> > ~Seth
>> >
>
Re: Prefix hijacking by AS20115 [ In reply to ]
On Mon, 28 Sep 2015, Seth Mattinen wrote:
> I'm at the tail end here almost 8 hours later since the hijacking started.
> Their NOC is just blowing me off now and they're happy to continue the
> hijacking until it's convenient for them to have a maintenance window. And
> that's apparently the final decision.

Willful negligence. Will only be in your favor when it comes to collect
damages.

-Dan
Re: Prefix hijacking by AS20115 [ In reply to ]
> On Mon, Sep 28, 2015 at 11:59 PM, Bob Evans <bob@fiberinternetcenter.com>
> wrote:
>> That's something I would do. Announce announce and keep adding ports
>> until
>> I hit a 10 Gig port worth of traffic or saw it fixed. Be sure to put in
>> a
>> blackhole route for the prefixes. Try to pick blocks that are as
>> geographically located to your peering routers as possible ...IE in Reno
>> pick the blocks that seem to be near by - like Reno, Tahoe, Sacramento
>> ..... when that batch of customers makes their phones ring all night
>> someone will listen.
>>
>
> that seems like a pretty poor strategy... guaranteed to get you into
> some hot water, I suspect. Keep in mind that the 'noc' at 20115 isn't
> the same thing as the customer-service-center. There's likely little
> to link the 2 things together there :(

You are right - probably creates more problems than good.

>
>> Would be nice if our membership organization ARIN ( that we all pay to
>> keep us somewhat organized) had an ability to do something for you.... I
>> never looked into it...i don't know....maybe it does ?
>
> arin does not guarantee 'routability' of netblocks assigned to your org.

Yep, I was pretty sure of that - but wouldn't it be nice if arin could
have some communication line or at least try. Yes, never any guarantees
really.

bob

>
>> But, in the mean time I am pretty sure you can document this well and
>> prove your announcements of theirs was due to the fact you couldn't get
>> proper technical attention and needed to desperately before your
>> customers
>> cancel after 8 hours of this. Tomorrow call your lawyers and begin to
>> sue
>> that cable company (did I recognize that ASN as cable TV ? ) for damages
>> this must be causing you in ill-will amongst your customer base.
>>
>> I wonder just how you prove the damage...some equation based on customer
>> calls and complaints together with how many years you have been in
>> business as well as the number of contracts that are coming up for
>> renewal. etc etc. Now that would be interesting to see a formula for
>> that
>> if anyone has been through it.
>>
>
> you COULD find a charter person on-list...there are nine names on the
> attendees list for the upcoming meeting... I imagine peeringdb likely
> has folk listed... gosh it sure does:
>
> <https://www.peeringdb.com/private/participant_view.php?id=2144>
>
> what with their emails and everything.
>
>> Thank You
>> Bob Evans
>> CTO
>>
>>
>>
>>
>>> Start announcing their prefixes?
>>>
>>> Josh Luthman
>>> Office: 937-552-2340
>>> Direct: 937-552-2343
>>> 1100 Wayne St
>>> Suite 1337
>>> Troy, OH 45373
>>> On Sep 28, 2015 11:09 PM, "Seth Mattinen" <sethm@rollernet.us> wrote:
>>>
>>>> On 9/28/15 18:30, William Herrin wrote:
>>>>
>>>>> On Mon, Sep 28, 2015 at 9:01 PM, Seth Mattinen <sethm@rollernet.us>
>>>>> wrote:
>>>>>
>>>>>> I've got a problem where AS20115 continues to announce prefixes
>>>>>> after
>>>>>> BGP
>>>>>> neighbors were shutdown. They claim it's a wedged BGP process but
>>>>>> aren't
>>>>>> in
>>>>>> any hurry to fix it outside of a maintenance window.
>>>>>>
>>>>>
>>>>> If they weren't lying to you, they'd fix it now. That's not the kind
>>>>> of problem that waits.
>>>>>
>>>>> Thing is: they lied to you. Long ago they "helpfully" programmed
>>>>> their
>>>>> router to announce your route regardless of whether you sent a route
>>>>> to them. They want to wait for a maintenance window to remove that
>>>>> configuration.
>>>>>
>>>>>
>>>>> I'm at a loss of what else I can do. They admit the problem but won't
>>>>> take
>>>>>> action saying it needs to wait for a maintenance window. Am I out of
>>>>>> line
>>>>>> insisting that's an unacceptable response to a problem that results
>>>>>> in
>>>>>> prefix/traffic hijacking?
>>>>>>
>>>>>
>>>>> Try dropping the link entirely. If they still announce your
>>>>> addresses,
>>>>> bring it back up but report it as emergency down, escalate, and call
>>>>> back every 10 minutes until the junior tech understands that it's
>>>>> time
>>>>> to call and wake up the guy who makes the decision to fix it now.
>>>>>
>>>>>
>>>>
>>>> I'm at the tail end here almost 8 hours later since the hijacking
>>>> started.
>>>> Their NOC is just blowing me off now and they're happy to continue the
>>>> hijacking until it's convenient for them to have a maintenance window.
>>>> And
>>>> that's apparently the final decision.
>>>>
>>>> ~Seth
>>>>
>>>
>>
>>
>
Re: Prefix hijacking by AS20115 [ In reply to ]
On Tue, Sep 29, 2015 at 2:04 AM, Bob Evans <bob@fiberinternetcenter.com> wrote:
>
>
>> On Mon, Sep 28, 2015 at 11:59 PM, Bob Evans <bob@fiberinternetcenter.com>
>> wrote:
>>> That's something I would do. Announce announce and keep adding ports
>>> until
>>> I hit a 10 Gig port worth of traffic or saw it fixed. Be sure to put in
>>> a
>>> blackhole route for the prefixes. Try to pick blocks that are as
>>> geographically located to your peering routers as possible ...IE in Reno
>>> pick the blocks that seem to be near by - like Reno, Tahoe, Sacramento
>>> ..... when that batch of customers makes their phones ring all night
>>> someone will listen.
>>>
>>
>> that seems like a pretty poor strategy... guaranteed to get you into
>> some hot water, I suspect. Keep in mind that the 'noc' at 20115 isn't
>> the same thing as the customer-service-center. There's likely little
>> to link the 2 things together there :(
>
> You are right - probably creates more problems than good.
>
>>
>>> Would be nice if our membership organization ARIN ( that we all pay to
>>> keep us somewhat organized) had an ability to do something for you.... I
>>> never looked into it...i don't know....maybe it does ?
>>
>> arin does not guarantee 'routability' of netblocks assigned to your org.
>
> Yep, I was pretty sure of that - but wouldn't it be nice if arin could
> have some communication line or at least try. Yes, never any guarantees
> really.

I'm fairly sure that the arin (or ripe or apnic or...) answer to your
question is: "read the contact info in whois... call the stated
numbers."

pretty sure that's also not going to be super helpful, email the poc's
in the peering-db.

> bob
>
>>
>>> But, in the mean time I am pretty sure you can document this well and
>>> prove your announcements of theirs was due to the fact you couldn't get
>>> proper technical attention and needed to desperately before your
>>> customers
>>> cancel after 8 hours of this. Tomorrow call your lawyers and begin to
>>> sue
>>> that cable company (did I recognize that ASN as cable TV ? ) for damages
>>> this must be causing you in ill-will amongst your customer base.
>>>
>>> I wonder just how you prove the damage...some equation based on customer
>>> calls and complaints together with how many years you have been in
>>> business as well as the number of contracts that are coming up for
>>> renewal. etc etc. Now that would be interesting to see a formula for
>>> that
>>> if anyone has been through it.
>>>
>>
>> you COULD find a charter person on-list...there are nine names on the
>> attendees list for the upcoming meeting... I imagine peeringdb likely
>> has folk listed... gosh it sure does:
>>
>> <https://www.peeringdb.com/private/participant_view.php?id=2144>
>>
>> what with their emails and everything.
>>
>>> Thank You
>>> Bob Evans
>>> CTO
>>>
>>>
>>>
>>>
>>>> Start announcing their prefixes?
>>>>
>>>> Josh Luthman
>>>> Office: 937-552-2340
>>>> Direct: 937-552-2343
>>>> 1100 Wayne St
>>>> Suite 1337
>>>> Troy, OH 45373
>>>> On Sep 28, 2015 11:09 PM, "Seth Mattinen" <sethm@rollernet.us> wrote:
>>>>
>>>>> On 9/28/15 18:30, William Herrin wrote:
>>>>>
>>>>>> On Mon, Sep 28, 2015 at 9:01 PM, Seth Mattinen <sethm@rollernet.us>
>>>>>> wrote:
>>>>>>
>>>>>>> I've got a problem where AS20115 continues to announce prefixes
>>>>>>> after
>>>>>>> BGP
>>>>>>> neighbors were shutdown. They claim it's a wedged BGP process but
>>>>>>> aren't
>>>>>>> in
>>>>>>> any hurry to fix it outside of a maintenance window.
>>>>>>>
>>>>>>
>>>>>> If they weren't lying to you, they'd fix it now. That's not the kind
>>>>>> of problem that waits.
>>>>>>
>>>>>> Thing is: they lied to you. Long ago they "helpfully" programmed
>>>>>> their
>>>>>> router to announce your route regardless of whether you sent a route
>>>>>> to them. They want to wait for a maintenance window to remove that
>>>>>> configuration.
>>>>>>
>>>>>>
>>>>>> I'm at a loss of what else I can do. They admit the problem but won't
>>>>>> take
>>>>>>> action saying it needs to wait for a maintenance window. Am I out of
>>>>>>> line
>>>>>>> insisting that's an unacceptable response to a problem that results
>>>>>>> in
>>>>>>> prefix/traffic hijacking?
>>>>>>>
>>>>>>
>>>>>> Try dropping the link entirely. If they still announce your
>>>>>> addresses,
>>>>>> bring it back up but report it as emergency down, escalate, and call
>>>>>> back every 10 minutes until the junior tech understands that it's
>>>>>> time
>>>>>> to call and wake up the guy who makes the decision to fix it now.
>>>>>>
>>>>>>
>>>>>
>>>>> I'm at the tail end here almost 8 hours later since the hijacking
>>>>> started.
>>>>> Their NOC is just blowing me off now and they're happy to continue the
>>>>> hijacking until it's convenient for them to have a maintenance window.
>>>>> And
>>>>> that's apparently the final decision.
>>>>>
>>>>> ~Seth
>>>>>
>>>>
>>>
>>>
>>
>
>
Re: Prefix hijacking by AS20115 [ In reply to ]
On 9/28/15, 10:24 PM, "NANOG on behalf of Seth Mattinen"
<nanog-bounces@nanog.org on behalf of sethm@rollernet.us> wrote:

>On 9/28/15 20:19, Martin Hannigan wrote:
>>
>>Is this related to 104.73.161.0/24? That's ours. :-)
>>
>>We'll take a look and get back to you. Thanks for caring!
>>
>
>
>Yep, that's one of the affected prefixes.
>
>~Seth
Hi Seth, which market was this occurring? Was this already removed? I'm
not seeing it this morning. I would like to figure out what went wrong
here. We shouldn't be nailing up any static configuration to have caused
a situation like this.
Re: Prefix hijacking by AS20115 [ In reply to ]
On 29/Sep/15 16:26, Rampley Jr, Jim F wrote:

>
> Hi Seth, which market was this occurring? Was this already removed? I'm
> not seeing it this morning. I would like to figure out what went wrong
> here. We shouldn't be nailing up any static configuration to have caused
> a situation like this.

You'd be surprised how often this happens, especially on the back of a
conference rocking into a city/country and the local provider having
minimal BGP experience. Once the conference is done, folk leave, and the
provider forgets about things - which is not a problem since the
conference would have come with its own IP address space.

The issue goes unnoticed for 12x months when the conference is trying to
route their usual block in some other city/country, and things just seem
"strange". Someone remembers the previous year's event, calls up the
previous provider, and finds out that the tech. who worked the
activation has since left.

It's not easy...

Many other situations closer to home (i.e., paying customers) where
things like this happen, especially if the customer has IP address space
but does not do BGP (until they want to or leave to the competition).

Blackholing operations that go wrong that folk forget about as well, not
to mention other networks that cut themselves off by using public IP
address space for their enterprise network.

It's not easy at all...

Mark.
Re: Prefix hijacking by AS20115 [ In reply to ]
On 9/29/15 7:26 AM, Rampley Jr, Jim F wrote:
>
>
> On 9/28/15, 10:24 PM, "NANOG on behalf of Seth Mattinen"
> <nanog-bounces@nanog.org on behalf of sethm@rollernet.us> wrote:
>
>> On 9/28/15 20:19, Martin Hannigan wrote:
>>>
>>> Is this related to 104.73.161.0/24? That's ours. :-)
>>>
>>> We'll take a look and get back to you. Thanks for caring!
>>>
>>
>>
>> Yep, that's one of the affected prefixes.
>>
>> ~Seth
> Hi Seth, which market was this occurring? Was this already removed? I'm
> not seeing it this morning. I would like to figure out what went wrong
> here. We shouldn't be nailing up any static configuration to have caused
> a situation like this.
>


Reno, NV. I do believe they've finally withdrawn this morning (I just
woke up, it was a long night).

~Seth
Re: Prefix hijacking by AS20115 [ In reply to ]
On 9/29/15, 9:49 AM, "Seth Mattinen" <sethm@rollernet.us> wrote:


>On 9/29/15 7:26 AM, Rampley Jr, Jim F wrote:
>>
>>
>> On 9/28/15, 10:24 PM, "NANOG on behalf of Seth Mattinen"
>> <nanog-bounces@nanog.org on behalf of sethm@rollernet.us> wrote:
>>
>>> On 9/28/15 20:19, Martin Hannigan wrote:
>>>>
>>>> Is this related to 104.73.161.0/24? That's ours. :-)
>>>>
>>>> We'll take a look and get back to you. Thanks for caring!
>>>>
>>>
>>>
>>> Yep, that's one of the affected prefixes.
>>>
>>> ~Seth
>> Hi Seth, which market was this occurring? Was this already removed?
>>I'm
>> not seeing it this morning. I would like to figure out what went wrong
>> here. We shouldn't be nailing up any static configuration to have
>>caused
>> a situation like this.
>>
>
>
>Reno, NV. I do believe they've finally withdrawn this morning (I just
>woke up, it was a long night).
>
>~Seth
This issue was caused by a hung BGP process which was resolved last night.
Nothing nefarious. No static configuration nailed up, no BGP highjacking
purposely done. ;)
Re: Prefix hijacking by AS20115 [ In reply to ]
On Sep 28, 2015, at 11:59 PM, Bob Evans <bob@FiberInternetCenter.com> wrote:

>
> Would be nice if our membership organization ARIN ( that we all pay to
> keep us somewhat organized) had an ability to do something for you.... I
> never looked into it...i don't know....maybe it does ?
>

No one else has said this, so…

RPKI. Which ARIN does do.

—Sandy

P.S. The following has numerous points of weirdness.

about 104.73.161.0/24, RADB says:

route: 104.73.161.0/24
descr: Proxy for Akamai (AS20940) and Roller Networks (AS11170)
origin: AS20115
mnt-by: MAINT-CHTR-WD
changed: tim.weber@charter.com 20150312 #20:32:27Z
source: RADB

route: 104.73.161.0/24
descr: Akamai Technologies
origin: AS20940
mnt-by: AKAM1-RIPE-MNT
changed: unread@ripe.net 20000101
source: RIPE
remarks: ****************************
remarks: * THIS OBJECT IS MODIFIED
remarks: * Please note that all data that is generally regarded as personal
remarks: * data has been removed from this object.
remarks: * To view the original object, please query the RIPE Database at:
remarks: * http://www.ripe.net/whois
remarks: ****************************

route: 104.64.0.0/10
descr: Akamai
origin: AS35994
mnt-by: AKAM1-ALTDB-MNT
changed: ablock@akamai.com 20140518
source: ALTDB
Re: Prefix hijacking by AS20115 [ In reply to ]
On 9/29/15 8:18 AM, Rampley Jr, Jim F wrote:
>
> This issue was caused by a hung BGP process which was resolved last night.
> Nothing nefarious. No static configuration nailed up, no BGP highjacking
> purposely done. ;)
>


Is there a Cisco bug ID?

~Seth
Re: Prefix hijacking by AS20115 [ In reply to ]
If this is anything like what I deal with the aging timer for the bgp
session is set to 180s by default. After 2 years I've been unable to get
the charter noc to enable bfd on my links to address this issue
On Sep 29, 2015 10:59 AM, "Seth Mattinen" <sethm@rollernet.us> wrote:

> On 9/29/15 8:18 AM, Rampley Jr, Jim F wrote:
> >
>
>> This issue was caused by a hung BGP process which was resolved last night.
>> Nothing nefarious. No static configuration nailed up, no BGP
>> highjacking
>> purposely done. ;)
>>
>>
>
> Is there a Cisco bug ID?
>
> ~Seth
>
Re: Prefix hijacking by AS20115 [ In reply to ]
On Tue, Sep 29, 2015 at 1:29 PM, N M <digitallystoned@gmail.com> wrote:
> If this is anything like what I deal with the aging timer for the bgp
> session is set to 180s by default. After 2 years I've been unable to get
> the charter noc to enable bfd on my links to address this issue

because bfd brings it's own special sort of pain...