Mailing List Archive

Connectivity problem with remote site
Starting last week, I've had one of my site go a little bit looney.
Dropping calls, Phones saying UCM down, Pushing the extension, getting
dialtone, dialing, and having a live call already on the line.

It looks like a connectivity issue, but I can't figure out what's going on.
The site has been up for about 13 months with no issues.. Nothing has
changed in the network.

I'm pinging a bunch of the phones. They haven't dropped any packets, and
all my pings are 2ms or less.

here's what's on the phone Display log:
8:24:07a 14: Name=SEP00235E18F284 Load= SCCP41.8-5-3S Last=UCM-closed-TCP
8:24:11a 23: Name=SEP00235E18F284 Load= SCCP41.8-5-3S Last=Reset-Restart
10:23:00a 14: Name=SEP00235E18F284 Load= SCCP41.8-5-3S Last=UCM-closed-TCP
10:23:05a 23: Name=SEP00235E18F284 Load= SCCP41.8-5-3S Last=Reset-Restart
10:57:17a 14: Name=SEP00235E18F284 Load= SCCP41.8-5-3S Last=UCM-closed-TCP
10:57:21a 23: Name=SEP00235E18F284 Load= SCCP41.8-5-3S Last=Reset-Restart
11:11:00a 14: Name=SEP00235E18F284 Load= SCCP41.8-5-3S Last=UCM-closed-TCP
11:11:05a 23: Name=SEP00235E18F284 Load= SCCP41.8-5-3S Last=Reset-Restart
11:19:24a 14: Name=SEP00235E18F284 Load= SCCP41.8-5-3S Last=UCM-closed-TCP
11:19:28a 23: Name=SEP00235E18F284 Load= SCCP41.8-5-3S Last=Reset-Restart
11:21:41a 23: Name=SEP00235E18F284 Load= SCCP41.8-5-3S Last=Reset-Restart
11:46:48a 14: Name=SEP00235E18F284 Load= SCCP41.8-5-3S Last=UCM-closed-TCP
11:46:56a 23: Name=SEP00235E18F284 Load= SCCP41.8-5-3S Last=Reset-Restart
11:56:38a 14: Name=SEP00235E18F284 Load= SCCP41.8-5-3S Last=UCM-closed-TCP
11:57:01a 14: Name=SEP00235E18F284 Load= SCCP41.8-5-3S Last=UCM-closed-TCP
11:57:01a 18: Name=SEP00235E18F284 Load= SCCP41.8-5-3S Last=Failback
12:22:20p 14: Name=SEP00235E18F284 Load= SCCP41.8-5-3S Last=UCM-closed-TCP
12:22:25p 23: Name=SEP00235E18F284 Load= SCCP41.8-5-3S Last=Reset-Restart
12:27:55p 14: Name=SEP00235E18F284 Load= SCCP41.8-5-3S Last=UCM-closed-TCP
12:28:37p 14: Name=SEP00235E18F284 Load= SCCP41.8-5-3S Last=UCM-closed-TCP
12:28:38p 18: Name=SEP00235E18F284 Load= SCCP41.8-5-3S Last=Failback
12:29:02p 14: Name=SEP00235E18F284 Load= SCCP41.8-5-3S Last=UCM-closed-TCP
12:29:06p 23: Name=SEP00235E18F284 Load= SCCP41.8-5-3S Last=Reset-Restart


I've been pinging the phones since 12:00, so I've had at least 3 "events"
while I've been pinging it.

UCM-closed-TCP indicates that a firewall closed the connection.
I have a L2L vpn between sites:
CUCM -> 6509 -> ASA5550 ===IPSEC===> ASA5505 -> Switch -> Phone

Where can I look for more information to diagnose what's going on?

Mike
Re: Connectivity problem with remote site [ In reply to ]
I'd set up a sniffer behind one of the phones (turn on span to pc port) and let it run until the phone resets. If you can get the same capture going at the CUCM server it will go a long way.

Reset-Restart is simply somebody clicking the Restart button in CCMAdmin.

UCM-closed-TCP does mean the TCP session got terminated, but it doesn't have to have been a firewall doing it. Getting the capture at the phone and CUCM will give you a better idea of what's going on, as it is network related.

-Ryan

On Feb 8, 2010, at 12:43 PM, Mike King wrote:

Starting last week, I've had one of my site go a little bit looney. Dropping calls, Phones saying UCM down, Pushing the extension, getting dialtone, dialing, and having a live call already on the line.

It looks like a connectivity issue, but I can't figure out what's going on. The site has been up for about 13 months with no issues.. Nothing has changed in the network.

I'm pinging a bunch of the phones. They haven't dropped any packets, and all my pings are 2ms or less.

here's what's on the phone Display log:
8:24:07a 14: Name=SEP00235E18F284 Load= SCCP41.8-5-3S Last=UCM-closed-TCP
8:24:11a 23: Name=SEP00235E18F284 Load= SCCP41.8-5-3S Last=Reset-Restart
10:23:00a 14: Name=SEP00235E18F284 Load= SCCP41.8-5-3S Last=UCM-closed-TCP
10:23:05a 23: Name=SEP00235E18F284 Load= SCCP41.8-5-3S Last=Reset-Restart
10:57:17a 14: Name=SEP00235E18F284 Load= SCCP41.8-5-3S Last=UCM-closed-TCP
10:57:21a 23: Name=SEP00235E18F284 Load= SCCP41.8-5-3S Last=Reset-Restart
11:11:00a 14: Name=SEP00235E18F284 Load= SCCP41.8-5-3S Last=UCM-closed-TCP
11:11:05a 23: Name=SEP00235E18F284 Load= SCCP41.8-5-3S Last=Reset-Restart
11:19:24a 14: Name=SEP00235E18F284 Load= SCCP41.8-5-3S Last=UCM-closed-TCP
11:19:28a 23: Name=SEP00235E18F284 Load= SCCP41.8-5-3S Last=Reset-Restart
11:21:41a 23: Name=SEP00235E18F284 Load= SCCP41.8-5-3S Last=Reset-Restart
11:46:48a 14: Name=SEP00235E18F284 Load= SCCP41.8-5-3S Last=UCM-closed-TCP
11:46:56a 23: Name=SEP00235E18F284 Load= SCCP41.8-5-3S Last=Reset-Restart
11:56:38a 14: Name=SEP00235E18F284 Load= SCCP41.8-5-3S Last=UCM-closed-TCP
11:57:01a 14: Name=SEP00235E18F284 Load= SCCP41.8-5-3S Last=UCM-closed-TCP
11:57:01a 18: Name=SEP00235E18F284 Load= SCCP41.8-5-3S Last=Failback
12:22:20p 14: Name=SEP00235E18F284 Load= SCCP41.8-5-3S Last=UCM-closed-TCP
12:22:25p 23: Name=SEP00235E18F284 Load= SCCP41.8-5-3S Last=Reset-Restart
12:27:55p 14: Name=SEP00235E18F284 Load= SCCP41.8-5-3S Last=UCM-closed-TCP
12:28:37p 14: Name=SEP00235E18F284 Load= SCCP41.8-5-3S Last=UCM-closed-TCP
12:28:38p 18: Name=SEP00235E18F284 Load= SCCP41.8-5-3S Last=Failback
12:29:02p 14: Name=SEP00235E18F284 Load= SCCP41.8-5-3S Last=UCM-closed-TCP
12:29:06p 23: Name=SEP00235E18F284 Load= SCCP41.8-5-3S Last=Reset-Restart


I've been pinging the phones since 12:00, so I've had at least 3 "events" while I've been pinging it.

UCM-closed-TCP indicates that a firewall closed the connection.
I have a L2L vpn between sites:
CUCM -> 6509 -> ASA5550 ===IPSEC===> ASA5505 -> Switch -> Phone

Where can I look for more information to diagnose what's going on?

Mike


_______________________________________________
cisco-voip mailing list
cisco-voip@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-voip

_______________________________________________
cisco-voip mailing list
cisco-voip@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-voip
Re: Connectivity problem with remote site [ In reply to ]
I forgot to add:

VPN uptime counter is at 6days and change.

Have several line of business applications that are running thru Citrix that
are not being affected.
Nobody is noticing any drops in connectivity (I know that VoIP is very
sensitive)

On Mon, Feb 8, 2010 at 1:04 PM, Matthew Ballard <mballard@otis.edu> wrote:

> I would first check the VPN for issues. You should be able to check
> uptime on the VPN connection, if that is staying low, and therefore
> interrupting communication with UCM, that would cause those problems. Could
> be an issue with the provider connecting the sites.
>
>
>
> Matthew Ballard
>
> Network Manager
>
> Otis College of Art and Design
>
> mballard@otis.edu
>
>
>
> *From:* cisco-voip-bounces@puck.nether.net [mailto:
> cisco-voip-bounces@puck.nether.net] *On Behalf Of *Mike King
> *Sent:* Monday, February 08, 2010 9:43 AM
> *To:* Cisco VoIPoE List
> *Subject:* [cisco-voip] Connectivity problem with remote site
>
>
>
> Starting last week, I've had one of my site go a little bit looney.
> Dropping calls, Phones saying UCM down, Pushing the extension, getting
> dialtone, dialing, and having a live call already on the line.
>
>
>
> It looks like a connectivity issue, but I can't figure out what's going on.
> The site has been up for about 13 months with no issues.. Nothing has
> changed in the network.
>
>
>
> I'm pinging a bunch of the phones. They haven't dropped any packets, and
> all my pings are 2ms or less.
>
>
>
> here's what's on the phone Display log:
>
> 8:24:07a 14: Name=SEP00235E18F284 Load= SCCP41.8-5-3S Last=UCM-closed-TCP
>
> 8:24:11a 23: Name=SEP00235E18F284 Load= SCCP41.8-5-3S Last=Reset-Restart
>
> 10:23:00a 14: Name=SEP00235E18F284 Load= SCCP41.8-5-3S
> Last=UCM-closed-TCP
>
> 10:23:05a 23: Name=SEP00235E18F284 Load= SCCP41.8-5-3S Last=Reset-Restart
>
> 10:57:17a 14: Name=SEP00235E18F284 Load= SCCP41.8-5-3S
> Last=UCM-closed-TCP
>
> 10:57:21a 23: Name=SEP00235E18F284 Load= SCCP41.8-5-3S Last=Reset-Restart
>
> 11:11:00a 14: Name=SEP00235E18F284 Load= SCCP41.8-5-3S
> Last=UCM-closed-TCP
>
> 11:11:05a 23: Name=SEP00235E18F284 Load= SCCP41.8-5-3S Last=Reset-Restart
>
> 11:19:24a 14: Name=SEP00235E18F284 Load= SCCP41.8-5-3S
> Last=UCM-closed-TCP
>
> 11:19:28a 23: Name=SEP00235E18F284 Load= SCCP41.8-5-3S Last=Reset-Restart
>
> 11:21:41a 23: Name=SEP00235E18F284 Load= SCCP41.8-5-3S Last=Reset-Restart
>
> 11:46:48a 14: Name=SEP00235E18F284 Load= SCCP41.8-5-3S
> Last=UCM-closed-TCP
>
> 11:46:56a 23: Name=SEP00235E18F284 Load= SCCP41.8-5-3S Last=Reset-Restart
>
> 11:56:38a 14: Name=SEP00235E18F284 Load= SCCP41.8-5-3S
> Last=UCM-closed-TCP
>
> 11:57:01a 14: Name=SEP00235E18F284 Load= SCCP41.8-5-3S
> Last=UCM-closed-TCP
>
> 11:57:01a 18: Name=SEP00235E18F284 Load= SCCP41.8-5-3S Last=Failback
>
> 12:22:20p 14: Name=SEP00235E18F284 Load= SCCP41.8-5-3S
> Last=UCM-closed-TCP
>
> 12:22:25p 23: Name=SEP00235E18F284 Load= SCCP41.8-5-3S Last=Reset-Restart
>
> 12:27:55p 14: Name=SEP00235E18F284 Load= SCCP41.8-5-3S Last=UCM-closed-TCP
>
> 12:28:37p 14: Name=SEP00235E18F284 Load= SCCP41.8-5-3S
> Last=UCM-closed-TCP
>
> 12:28:38p 18: Name=SEP00235E18F284 Load= SCCP41.8-5-3S Last=Failback
>
> 12:29:02p 14: Name=SEP00235E18F284 Load= SCCP41.8-5-3S
> Last=UCM-closed-TCP
>
> 12:29:06p 23: Name=SEP00235E18F284 Load= SCCP41.8-5-3S Last=Reset-Restart
>
>
>
>
>
> I've been pinging the phones since 12:00, so I've had at least 3 "events"
> while I've been pinging it.
>
>
>
> UCM-closed-TCP indicates that a firewall closed the connection.
>
> I have a L2L vpn between sites:
>
> CUCM -> 6509 -> ASA5550 ===IPSEC===> ASA5505 -> Switch -> Phone
>
>
>
> Where can I look for more information to diagnose what's going on?
>
>
>
> Mike
>
>
>
>
>
Re: Connectivity problem with remote site [ In reply to ]
If you have QoS enabled at each end of the VPN are you seeing drops in the voice signaling queue(s)?

-Ryan

On Feb 8, 2010, at 1:13 PM, Mike King wrote:

I forgot to add:

VPN uptime counter is at 6days and change.

Have several line of business applications that are running thru Citrix that are not being affected.
Nobody is noticing any drops in connectivity (I know that VoIP is very sensitive)

On Mon, Feb 8, 2010 at 1:04 PM, Matthew Ballard <mballard@otis.edu> wrote:
I would first check the VPN for issues. You should be able to check uptime on the VPN connection, if that is staying low, and therefore interrupting communication with UCM, that would cause those problems. Could be an issue with the provider connecting the sites.


Matthew Ballard

Network Manager

Otis College of Art and Design

mballard@otis.edu


From: cisco-voip-bounces@puck.nether.net [mailto:cisco-voip-bounces@puck.nether.net] On Behalf Of Mike King
Sent: Monday, February 08, 2010 9:43 AM
To: Cisco VoIPoE List
Subject: [cisco-voip] Connectivity problem with remote site


Starting last week, I've had one of my site go a little bit looney. Dropping calls, Phones saying UCM down, Pushing the extension, getting dialtone, dialing, and having a live call already on the line.


It looks like a connectivity issue, but I can't figure out what's going on. The site has been up for about 13 months with no issues.. Nothing has changed in the network.


I'm pinging a bunch of the phones. They haven't dropped any packets, and all my pings are 2ms or less.


here's what's on the phone Display log:

8:24:07a 14: Name=SEP00235E18F284 Load= SCCP41.8-5-3S Last=UCM-closed-TCP

8:24:11a 23: Name=SEP00235E18F284 Load= SCCP41.8-5-3S Last=Reset-Restart

10:23:00a 14: Name=SEP00235E18F284 Load= SCCP41.8-5-3S Last=UCM-closed-TCP

10:23:05a 23: Name=SEP00235E18F284 Load= SCCP41.8-5-3S Last=Reset-Restart

10:57:17a 14: Name=SEP00235E18F284 Load= SCCP41.8-5-3S Last=UCM-closed-TCP

10:57:21a 23: Name=SEP00235E18F284 Load= SCCP41.8-5-3S Last=Reset-Restart

11:11:00a 14: Name=SEP00235E18F284 Load= SCCP41.8-5-3S Last=UCM-closed-TCP

11:11:05a 23: Name=SEP00235E18F284 Load= SCCP41.8-5-3S Last=Reset-Restart

11:19:24a 14: Name=SEP00235E18F284 Load= SCCP41.8-5-3S Last=UCM-closed-TCP

11:19:28a 23: Name=SEP00235E18F284 Load= SCCP41.8-5-3S Last=Reset-Restart

11:21:41a 23: Name=SEP00235E18F284 Load= SCCP41.8-5-3S Last=Reset-Restart

11:46:48a 14: Name=SEP00235E18F284 Load= SCCP41.8-5-3S Last=UCM-closed-TCP

11:46:56a 23: Name=SEP00235E18F284 Load= SCCP41.8-5-3S Last=Reset-Restart

11:56:38a 14: Name=SEP00235E18F284 Load= SCCP41.8-5-3S Last=UCM-closed-TCP

11:57:01a 14: Name=SEP00235E18F284 Load= SCCP41.8-5-3S Last=UCM-closed-TCP

11:57:01a 18: Name=SEP00235E18F284 Load= SCCP41.8-5-3S Last=Failback

12:22:20p 14: Name=SEP00235E18F284 Load= SCCP41.8-5-3S Last=UCM-closed-TCP

12:22:25p 23: Name=SEP00235E18F284 Load= SCCP41.8-5-3S Last=Reset-Restart

12:27:55p 14: Name=SEP00235E18F284 Load= SCCP41.8-5-3S Last=UCM-closed-TCP

12:28:37p 14: Name=SEP00235E18F284 Load= SCCP41.8-5-3S Last=UCM-closed-TCP

12:28:38p 18: Name=SEP00235E18F284 Load= SCCP41.8-5-3S Last=Failback

12:29:02p 14: Name=SEP00235E18F284 Load= SCCP41.8-5-3S Last=UCM-closed-TCP

12:29:06p 23: Name=SEP00235E18F284 Load= SCCP41.8-5-3S Last=Reset-Restart



I've been pinging the phones since 12:00, so I've had at least 3 "events" while I've been pinging it.


UCM-closed-TCP indicates that a firewall closed the connection.

I have a L2L vpn between sites:

CUCM -> 6509 -> ASA5550 ===IPSEC===> ASA5505 -> Switch -> Phone


Where can I look for more information to diagnose what's going on?


Mike




_______________________________________________
cisco-voip mailing list
cisco-voip@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-voip
Re: Connectivity problem with remote site [ In reply to ]
VOIP is very sensitive. What model phones are these and what type of
failover do you have configured for the phones?
http://puck.nether.net/pipermail/cisco-voip/2009-May/001155.html

The failover type determines TCP stack behavior on the phones. The
underlying cause is dropped or delayed packets in your network. Other
applications are just more forgiving.

You can try "slow failover" to be more lenient on signaling. However,
if you are running RTP over those links the RTP will still be negatively
impacted.

/Wes

On Monday, February 08, 2010 1:13:09 PM, Mike King <me@mpking.com> wrote:
> I forgot to add:
>
> VPN uptime counter is at 6days and change.
>
> Have several line of business applications that are running thru
> Citrix that are not being affected.
> Nobody is noticing any drops in connectivity (I know that VoIP is very
> sensitive)
>
> On Mon, Feb 8, 2010 at 1:04 PM, Matthew Ballard <mballard@otis.edu
> <mailto:mballard@otis.edu>> wrote:
>
> I would first check the VPN for issues. You should be able to
> check uptime on the VPN connection, if that is staying low, and
> therefore interrupting communication with UCM, that would cause
> those problems. Could be an issue with the provider connecting
> the sites.
>
>
>
> Matthew Ballard
>
> Network Manager
>
> Otis College of Art and Design
>
> mballard@otis.edu <mailto:mballard@otis.edu>
>
>
>
> *From:* cisco-voip-bounces@puck.nether.net
> <mailto:cisco-voip-bounces@puck.nether.net>
> [mailto:cisco-voip-bounces@puck.nether.net
> <mailto:cisco-voip-bounces@puck.nether.net>] *On Behalf Of *Mike King
> *Sent:* Monday, February 08, 2010 9:43 AM
> *To:* Cisco VoIPoE List
> *Subject:* [cisco-voip] Connectivity problem with remote site
>
>
>
> Starting last week, I've had one of my site go a little bit
> looney. Dropping calls, Phones saying UCM down, Pushing the
> extension, getting dialtone, dialing, and having a live call
> already on the line.
>
>
>
> It looks like a connectivity issue, but I can't figure out what's
> going on. The site has been up for about 13 months with no
> issues.. Nothing has changed in the network.
>
>
>
> I'm pinging a bunch of the phones. They haven't dropped any
> packets, and all my pings are 2ms or less.
>
>
>
> here's what's on the phone Display log:
>
> 8:24:07a 14: Name=SEP00235E18F284 Load= SCCP41.8-5-3S
> Last=UCM-closed-TCP
>
> 8:24:11a 23: Name=SEP00235E18F284 Load= SCCP41.8-5-3S
> Last=Reset-Restart
>
> 10:23:00a 14: Name=SEP00235E18F284 Load= SCCP41.8-5-3S
> Last=UCM-closed-TCP
>
> 10:23:05a 23: Name=SEP00235E18F284 Load= SCCP41.8-5-3S
> Last=Reset-Restart
>
> 10:57:17a 14: Name=SEP00235E18F284 Load= SCCP41.8-5-3S
> Last=UCM-closed-TCP
>
> 10:57:21a 23: Name=SEP00235E18F284 Load= SCCP41.8-5-3S
> Last=Reset-Restart
>
> 11:11:00a 14: Name=SEP00235E18F284 Load= SCCP41.8-5-3S
> Last=UCM-closed-TCP
>
> 11:11:05a 23: Name=SEP00235E18F284 Load= SCCP41.8-5-3S
> Last=Reset-Restart
>
> 11:19:24a 14: Name=SEP00235E18F284 Load= SCCP41.8-5-3S
> Last=UCM-closed-TCP
>
> 11:19:28a 23: Name=SEP00235E18F284 Load= SCCP41.8-5-3S
> Last=Reset-Restart
>
> 11:21:41a 23: Name=SEP00235E18F284 Load= SCCP41.8-5-3S
> Last=Reset-Restart
>
> 11:46:48a 14: Name=SEP00235E18F284 Load= SCCP41.8-5-3S
> Last=UCM-closed-TCP
>
> 11:46:56a 23: Name=SEP00235E18F284 Load= SCCP41.8-5-3S
> Last=Reset-Restart
>
> 11:56:38a 14: Name=SEP00235E18F284 Load= SCCP41.8-5-3S
> Last=UCM-closed-TCP
>
> 11:57:01a 14: Name=SEP00235E18F284 Load= SCCP41.8-5-3S
> Last=UCM-closed-TCP
>
> 11:57:01a 18: Name=SEP00235E18F284 Load= SCCP41.8-5-3S Last=Failback
>
> 12:22:20p 14: Name=SEP00235E18F284 Load= SCCP41.8-5-3S
> Last=UCM-closed-TCP
>
> 12:22:25p 23: Name=SEP00235E18F284 Load= SCCP41.8-5-3S
> Last=Reset-Restart
>
> 12:27:55p 14: Name=SEP00235E18F284 Load= SCCP41.8-5-3S
> Last=UCM-closed-TCP
>
> 12:28:37p 14: Name=SEP00235E18F284 Load= SCCP41.8-5-3S
> Last=UCM-closed-TCP
>
> 12:28:38p 18: Name=SEP00235E18F284 Load= SCCP41.8-5-3S Last=Failback
>
> 12:29:02p 14: Name=SEP00235E18F284 Load= SCCP41.8-5-3S
> Last=UCM-closed-TCP
>
> 12:29:06p 23: Name=SEP00235E18F284 Load= SCCP41.8-5-3S
> Last=Reset-Restart
>
>
>
>
>
> I've been pinging the phones since 12:00, so I've had at least 3
> "events" while I've been pinging it.
>
>
>
> UCM-closed-TCP indicates that a firewall closed the connection.
>
> I have a L2L vpn between sites:
>
> CUCM -> 6509 -> ASA5550 ===IPSEC===> ASA5505 -> Switch -> Phone
>
>
>
> Where can I look for more information to diagnose what's going on?
>
>
>
> Mike
>
>
>
>
>
>
> ------------------------------------------------------------------------
>
> _______________________________________________
> cisco-voip mailing list
> cisco-voip@puck.nether.net
> https://puck.nether.net/mailman/listinfo/cisco-voip
>
Re: Connectivity problem with remote site [ In reply to ]
7941G-GE's and 7961G-GE's. I'm not sure what the failover is, but it's most
likely the default in CUCM 7.1.3


For those following along at home, I found the following error messages in
my ASA's logs (both ends of the VPN termination):


Teardown TCP connection 541098 for elan:x.x.x.202/2000 to VOIP:x.x.x.x/38034
duration 0:33:51 bytes 9220 Flow closed by inspection
%ASA-6-106015: Deny TCP (no connection) from x.x.x.202/2000
to x.x.x.160/38034 flags PSH ACK on interface elan
%ASA-6-106015: Deny TCP (no connection) from x.x.x202/2000
to x.x.x.160/38034 flags PSH ACK on interface elan
%ASA-6-106015: Deny TCP (no connection) from x.x.x.202/2000
to x.x.x.160/38034 flags PSH ACK on interface elan
%ASA-6-106015: Deny TCP (no connection) from x.x.x.202/2000
to x.x.x.160/38034 flags PSH ACK on interface elan

I've removed
inspection skinny
from both ends of my ASA

And after a few minutes of insane confusion (calls lost, call on hold that
cannot be picked up), things seem to be better. I'm not logging any error
messages on any phone's I've been tracing.

I can't for the life of my figure out why this only showed up last thursday,
after 13 months of production use. (again, no changes in the network since
Last October, when the system was upgraded from 4.2.3 to 7.1.3)


On Mon, Feb 8, 2010 at 1:31 PM, Wes Sisk <wsisk@cisco.com> wrote:

> VOIP is very sensitive. What model phones are these and what type of
> failover do you have configured for the phones?
> http://puck.nether.net/pipermail/cisco-voip/2009-May/001155.html
>
> The failover type determines TCP stack behavior on the phones. The
> underlying cause is dropped or delayed packets in your network. Other
> applications are just more forgiving.
>
> You can try "slow failover" to be more lenient on signaling. However, if
> you are running RTP over those links the RTP will still be negatively
> impacted.
>
> /Wes
>
>
> On Monday, February 08, 2010 1:13:09 PM, Mike King <me@mpking.com><me@mpking.com>wrote:
>
> I forgot to add:
>
> VPN uptime counter is at 6days and change.
>
> Have several line of business applications that are running thru Citrix
> that are not being affected.
> Nobody is noticing any drops in connectivity (I know that VoIP is very
> sensitive)
>
> On Mon, Feb 8, 2010 at 1:04 PM, Matthew Ballard <mballard@otis.edu> wrote:
>
>> I would first check the VPN for issues. You should be able to check
>> uptime on the VPN connection, if that is staying low, and therefore
>> interrupting communication with UCM, that would cause those problems. Could
>> be an issue with the provider connecting the sites.
>>
>>
>>
>> Matthew Ballard
>>
>> Network Manager
>>
>> Otis College of Art and Design
>>
>> mballard@otis.edu
>>
>>
>>
>> *From:* cisco-voip-bounces@puck.nether.net [mailto:
>> cisco-voip-bounces@puck.nether.net] *On Behalf Of *Mike King
>> *Sent:* Monday, February 08, 2010 9:43 AM
>> *To:* Cisco VoIPoE List
>> *Subject:* [cisco-voip] Connectivity problem with remote site
>>
>>
>>
>> Starting last week, I've had one of my site go a little bit looney.
>> Dropping calls, Phones saying UCM down, Pushing the extension, getting
>> dialtone, dialing, and having a live call already on the line.
>>
>>
>>
>> It looks like a connectivity issue, but I can't figure out what's going
>> on. The site has been up for about 13 months with no issues.. Nothing has
>> changed in the network.
>>
>>
>>
>> I'm pinging a bunch of the phones. They haven't dropped any packets, and
>> all my pings are 2ms or less.
>>
>>
>>
>> here's what's on the phone Display log:
>>
>> 8:24:07a 14: Name=SEP00235E18F284 Load= SCCP41.8-5-3S Last=UCM-closed-TCP
>>
>> 8:24:11a 23: Name=SEP00235E18F284 Load= SCCP41.8-5-3S Last=Reset-Restart
>>
>> 10:23:00a 14: Name=SEP00235E18F284 Load= SCCP41.8-5-3S
>> Last=UCM-closed-TCP
>>
>> 10:23:05a 23: Name=SEP00235E18F284 Load= SCCP41.8-5-3S
>> Last=Reset-Restart
>>
>> 10:57:17a 14: Name=SEP00235E18F284 Load= SCCP41.8-5-3S
>> Last=UCM-closed-TCP
>>
>> 10:57:21a 23: Name=SEP00235E18F284 Load= SCCP41.8-5-3S
>> Last=Reset-Restart
>>
>> 11:11:00a 14: Name=SEP00235E18F284 Load= SCCP41.8-5-3S
>> Last=UCM-closed-TCP
>>
>> 11:11:05a 23: Name=SEP00235E18F284 Load= SCCP41.8-5-3S
>> Last=Reset-Restart
>>
>> 11:19:24a 14: Name=SEP00235E18F284 Load= SCCP41.8-5-3S
>> Last=UCM-closed-TCP
>>
>> 11:19:28a 23: Name=SEP00235E18F284 Load= SCCP41.8-5-3S
>> Last=Reset-Restart
>>
>> 11:21:41a 23: Name=SEP00235E18F284 Load= SCCP41.8-5-3S
>> Last=Reset-Restart
>>
>> 11:46:48a 14: Name=SEP00235E18F284 Load= SCCP41.8-5-3S
>> Last=UCM-closed-TCP
>>
>> 11:46:56a 23: Name=SEP00235E18F284 Load= SCCP41.8-5-3S
>> Last=Reset-Restart
>>
>> 11:56:38a 14: Name=SEP00235E18F284 Load= SCCP41.8-5-3S
>> Last=UCM-closed-TCP
>>
>> 11:57:01a 14: Name=SEP00235E18F284 Load= SCCP41.8-5-3S
>> Last=UCM-closed-TCP
>>
>> 11:57:01a 18: Name=SEP00235E18F284 Load= SCCP41.8-5-3S Last=Failback
>>
>> 12:22:20p 14: Name=SEP00235E18F284 Load= SCCP41.8-5-3S
>> Last=UCM-closed-TCP
>>
>> 12:22:25p 23: Name=SEP00235E18F284 Load= SCCP41.8-5-3S Last=Reset-Restart
>>
>> 12:27:55p 14: Name=SEP00235E18F284 Load= SCCP41.8-5-3S
>> Last=UCM-closed-TCP
>>
>> 12:28:37p 14: Name=SEP00235E18F284 Load= SCCP41.8-5-3S
>> Last=UCM-closed-TCP
>>
>> 12:28:38p 18: Name=SEP00235E18F284 Load= SCCP41.8-5-3S Last=Failback
>>
>> 12:29:02p 14: Name=SEP00235E18F284 Load= SCCP41.8-5-3S
>> Last=UCM-closed-TCP
>>
>> 12:29:06p 23: Name=SEP00235E18F284 Load= SCCP41.8-5-3S
>> Last=Reset-Restart
>>
>>
>>
>>
>>
>> I've been pinging the phones since 12:00, so I've had at least 3 "events"
>> while I've been pinging it.
>>
>>
>>
>> UCM-closed-TCP indicates that a firewall closed the connection.
>>
>> I have a L2L vpn between sites:
>>
>> CUCM -> 6509 -> ASA5550 ===IPSEC===> ASA5505 -> Switch -> Phone
>>
>>
>>
>> Where can I look for more information to diagnose what's going on?
>>
>>
>>
>> Mike
>>
>>
>>
>>
>>
>
> ------------------------------
>
> _______________________________________________
> cisco-voip mailing listcisco-voip@puck.nether.nethttps://puck.nether.net/mailman/listinfo/cisco-voip
>
>
>
Re: Connectivity problem with remote site [ In reply to ]
Did you upgrade the phone firmware last week? Originally the ASAs did not support sccp version 17 (default with CUCM 7 and firmware 8.5) but that was fixed with version 8.4 on the ASA.

-Ryan

On Feb 8, 2010, at 4:44 PM, Mike King wrote:

7941G-GE's and 7961G-GE's. I'm not sure what the failover is, but it's most likely the default in CUCM 7.1.3


For those following along at home, I found the following error messages in my ASA's logs (both ends of the VPN termination):


Teardown TCP connection 541098 for elan:x.x.x.202/2000 to VOIP:x.x.x.x/38034 duration 0:33:51 bytes 9220 Flow closed by inspection
%ASA-6-106015: Deny TCP (no connection) from x.x.x.202/2000 to x.x.x.160/38034 flags PSH ACK on interface elan
%ASA-6-106015: Deny TCP (no connection) from x.x.x202/2000 to x.x.x.160/38034 flags PSH ACK on interface elan
%ASA-6-106015: Deny TCP (no connection) from x.x.x.202/2000 to x.x.x.160/38034 flags PSH ACK on interface elan
%ASA-6-106015: Deny TCP (no connection) from x.x.x.202/2000 to x.x.x.160/38034 flags PSH ACK on interface elan

I've removed
inspection skinny
from both ends of my ASA

And after a few minutes of insane confusion (calls lost, call on hold that cannot be picked up), things seem to be better. I'm not logging any error messages on any phone's I've been tracing.

I can't for the life of my figure out why this only showed up last thursday, after 13 months of production use. (again, no changes in the network since Last October, when the system was upgraded from 4.2.3 to 7.1.3)


On Mon, Feb 8, 2010 at 1:31 PM, Wes Sisk <wsisk@cisco.com> wrote:
VOIP is very sensitive. What model phones are these and what type of failover do you have configured for the phones?
http://puck.nether.net/pipermail/cisco-voip/2009-May/001155.html

The failover type determines TCP stack behavior on the phones. The underlying cause is dropped or delayed packets in your network. Other applications are just more forgiving.

You can try "slow failover" to be more lenient on signaling. However, if you are running RTP over those links the RTP will still be negatively impacted.

/Wes


On Monday, February 08, 2010 1:13:09 PM, Mike King <me@mpking.com> wrote:
> I forgot to add:
>
> VPN uptime counter is at 6days and change.
>
> Have several line of business applications that are running thru Citrix that are not being affected.
> Nobody is noticing any drops in connectivity (I know that VoIP is very sensitive)
>
> On Mon, Feb 8, 2010 at 1:04 PM, Matthew Ballard <mballard@otis.edu> wrote:
> I would first check the VPN for issues. You should be able to check uptime on the VPN connection, if that is staying low, and therefore interrupting communication with UCM, that would cause those problems. Could be an issue with the provider connecting the sites.
>
>
> Matthew Ballard
>
> Network Manager
>
> Otis College of Art and Design
>
> mballard@otis.edu
>
>
> From: cisco-voip-bounces@puck.nether.net [mailto:cisco-voip-bounces@puck.nether.net] On Behalf Of Mike King
> Sent: Monday, February 08, 2010 9:43 AM
> To: Cisco VoIPoE List
> Subject: [cisco-voip] Connectivity problem with remote site
>
>
> Starting last week, I've had one of my site go a little bit looney. Dropping calls, Phones saying UCM down, Pushing the extension, getting dialtone, dialing, and having a live call already on the line.
>
>
> It looks like a connectivity issue, but I can't figure out what's going on. The site has been up for about 13 months with no issues.. Nothing has changed in the network.
>
>
> I'm pinging a bunch of the phones. They haven't dropped any packets, and all my pings are 2ms or less.
>
>
> here's what's on the phone Display log:
>
> 8:24:07a 14: Name=SEP00235E18F284 Load= SCCP41.8-5-3S Last=UCM-closed-TCP
>
> 8:24:11a 23: Name=SEP00235E18F284 Load= SCCP41.8-5-3S Last=Reset-Restart
>
> 10:23:00a 14: Name=SEP00235E18F284 Load= SCCP41.8-5-3S Last=UCM-closed-TCP
>
> 10:23:05a 23: Name=SEP00235E18F284 Load= SCCP41.8-5-3S Last=Reset-Restart
>
> 10:57:17a 14: Name=SEP00235E18F284 Load= SCCP41.8-5-3S Last=UCM-closed-TCP
>
> 10:57:21a 23: Name=SEP00235E18F284 Load= SCCP41.8-5-3S Last=Reset-Restart
>
> 11:11:00a 14: Name=SEP00235E18F284 Load= SCCP41.8-5-3S Last=UCM-closed-TCP
>
> 11:11:05a 23: Name=SEP00235E18F284 Load= SCCP41.8-5-3S Last=Reset-Restart
>
> 11:19:24a 14: Name=SEP00235E18F284 Load= SCCP41.8-5-3S Last=UCM-closed-TCP
>
> 11:19:28a 23: Name=SEP00235E18F284 Load= SCCP41.8-5-3S Last=Reset-Restart
>
> 11:21:41a 23: Name=SEP00235E18F284 Load= SCCP41.8-5-3S Last=Reset-Restart
>
> 11:46:48a 14: Name=SEP00235E18F284 Load= SCCP41.8-5-3S Last=UCM-closed-TCP
>
> 11:46:56a 23: Name=SEP00235E18F284 Load= SCCP41.8-5-3S Last=Reset-Restart
>
> 11:56:38a 14: Name=SEP00235E18F284 Load= SCCP41.8-5-3S Last=UCM-closed-TCP
>
> 11:57:01a 14: Name=SEP00235E18F284 Load= SCCP41.8-5-3S Last=UCM-closed-TCP
>
> 11:57:01a 18: Name=SEP00235E18F284 Load= SCCP41.8-5-3S Last=Failback
>
> 12:22:20p 14: Name=SEP00235E18F284 Load= SCCP41.8-5-3S Last=UCM-closed-TCP
>
> 12:22:25p 23: Name=SEP00235E18F284 Load= SCCP41.8-5-3S Last=Reset-Restart
>
> 12:27:55p 14: Name=SEP00235E18F284 Load= SCCP41.8-5-3S Last=UCM-closed-TCP
>
> 12:28:37p 14: Name=SEP00235E18F284 Load= SCCP41.8-5-3S Last=UCM-closed-TCP
>
> 12:28:38p 18: Name=SEP00235E18F284 Load= SCCP41.8-5-3S Last=Failback
>
> 12:29:02p 14: Name=SEP00235E18F284 Load= SCCP41.8-5-3S Last=UCM-closed-TCP
>
> 12:29:06p 23: Name=SEP00235E18F284 Load= SCCP41.8-5-3S Last=Reset-Restart
>
>
>
> I've been pinging the phones since 12:00, so I've had at least 3 "events" while I've been pinging it.
>
>
> UCM-closed-TCP indicates that a firewall closed the connection.
>
> I have a L2L vpn between sites:
>
> CUCM -> 6509 -> ASA5550 ===IPSEC===> ASA5505 -> Switch -> Phone
>
>
> Where can I look for more information to diagnose what's going on?
>
>
> Mike
>
>
>
>
>
> _______________________________________________
> cisco-voip mailing list
> cisco-voip@puck.nether.net
> https://puck.nether.net/mailman/listinfo/cisco-voip
>


_______________________________________________
cisco-voip mailing list
cisco-voip@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-voip
Re: Connectivity problem with remote site [ In reply to ]
On Mon, Feb 8, 2010 at 4:56 PM, Ryan Ratliff <rratliff@cisco.com> wrote:

> Did you upgrade the phone firmware last week? Originally the ASAs did not
> support sccp version 17 (default with CUCM 7 and firmware 8.5) but that was
> fixed with version 8.4 on the ASA.
>
>
Nope. Phones have been on 8-5-3S since last October.

Our ASA's are on Version 8.02 (trust me, I want to upgrade to 8.4, but there
is a technical/business reason that is preventing us)

Will disabling inspect skinny have any side-effects? (Least till we can get
onto 8.4 which is targeted sometime in the next two months)

Mike
Re: Connectivity problem with remote site [ In reply to ]
If you are not doing NAT then I don't believe you need to inspect skinny traffic. If you aren't on an ASA version that supports skinny 17 and you've been on CUCM 7 with 8.5 phone firmware since October then I'd say you are lucky it worked for this long (or somebody just turned on inspection and didn't tell anyone).

-Ryan

On Feb 8, 2010, at 5:29 PM, Mike King wrote:



On Mon, Feb 8, 2010 at 4:56 PM, Ryan Ratliff <rratliff@cisco.com> wrote:
Did you upgrade the phone firmware last week? Originally the ASAs did not support sccp version 17 (default with CUCM 7 and firmware 8.5) but that was fixed with version 8.4 on the ASA.


Nope. Phones have been on 8-5-3S since last October.

Our ASA's are on Version 8.02 (trust me, I want to upgrade to 8.4, but there is a technical/business reason that is preventing us)

Will disabling inspect skinny have any side-effects? (Least till we can get onto 8.4 which is targeted sometime in the next two months)

Mike
_______________________________________________
cisco-voip mailing list
cisco-voip@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-voip
Re: Connectivity problem with remote site [ In reply to ]
Ryan,

Is the phone firmware 8.5 specifically what's making the problem, or
is it the combination of that firmware + CM 7? I'm planning an upgrade
from 6.1 to 7.1 next month, is it reasonable to just load 8.5 firmware
on some phones now on 6.1 to verify that the ASA's will behave ok?

We have a few ASA's around on our campus network, and my group doesn't
have direct control over them - so trying to get all my info straight
on this issue. The ASA's are running 8.2.X, trying to find out if they
are running skinny inspection.

On Tue, Feb 9, 2010 at 9:23 AM, Ryan Ratliff <rratliff@cisco.com> wrote:
> If you are not doing NAT then I don't believe you need to inspect skinny
> traffic.  If you aren't on an ASA version that supports skinny 17 and you've
> been on CUCM 7 with 8.5 phone firmware since October then I'd say you are
> lucky it worked for this long (or somebody just turned on inspection and
> didn't tell anyone).
> -Ryan
> On Feb 8, 2010, at 5:29 PM, Mike King wrote:
>
>
> On Mon, Feb 8, 2010 at 4:56 PM, Ryan Ratliff <rratliff@cisco.com> wrote:
>>
>> Did you upgrade the phone firmware last week?  Originally the ASAs did not
>> support sccp version 17 (default with CUCM 7 and firmware 8.5) but that was
>> fixed with version 8.4 on the ASA.
>
> Nope.  Phones have been on 8-5-3S since last October.
> Our ASA's are on Version 8.02 (trust me, I want to upgrade to 8.4, but there
> is a technical/business reason that is preventing us)
> Will disabling inspect skinny have any side-effects?  (Least till we can get
> onto 8.4 which is targeted sometime in the next two months)
> Mike



--
Ed Leatherman
_______________________________________________
cisco-voip mailing list
cisco-voip@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-voip
Re: Connectivity problem with remote site [ In reply to ]
The version of skinny used between the phone and CUCM is determined by both sides indicating the highest version they support, and the highest that both support is used. In the ASA's case it didn't support the new messages with version 17 so if the phone had an 8.4 phone load and you are using CUCM 7 then the devices would be using skinny version 17 and you would get odd issues when making calls (as the ASA would drop certain messages it didn't know how to inspect).

In your case if the phones couldn't even stay registered it's likely some other inspection issue was causing the tcp sessions to get terminated.

-Ryan

On Feb 10, 2010, at 9:18 AM, Ed Leatherman wrote:

Ryan,

Is the phone firmware 8.5 specifically what's making the problem, or
is it the combination of that firmware + CM 7? I'm planning an upgrade
from 6.1 to 7.1 next month, is it reasonable to just load 8.5 firmware
on some phones now on 6.1 to verify that the ASA's will behave ok?

We have a few ASA's around on our campus network, and my group doesn't
have direct control over them - so trying to get all my info straight
on this issue. The ASA's are running 8.2.X, trying to find out if they
are running skinny inspection.

On Tue, Feb 9, 2010 at 9:23 AM, Ryan Ratliff <rratliff@cisco.com> wrote:
> If you are not doing NAT then I don't believe you need to inspect skinny
> traffic. If you aren't on an ASA version that supports skinny 17 and you've
> been on CUCM 7 with 8.5 phone firmware since October then I'd say you are
> lucky it worked for this long (or somebody just turned on inspection and
> didn't tell anyone).
> -Ryan
> On Feb 8, 2010, at 5:29 PM, Mike King wrote:
>
>
> On Mon, Feb 8, 2010 at 4:56 PM, Ryan Ratliff <rratliff@cisco.com> wrote:
>>
>> Did you upgrade the phone firmware last week? Originally the ASAs did not
>> support sccp version 17 (default with CUCM 7 and firmware 8.5) but that was
>> fixed with version 8.4 on the ASA.
>
> Nope. Phones have been on 8-5-3S since last October.
> Our ASA's are on Version 8.02 (trust me, I want to upgrade to 8.4, but there
> is a technical/business reason that is preventing us)
> Will disabling inspect skinny have any side-effects? (Least till we can get
> onto 8.4 which is targeted sometime in the next two months)
> Mike



--
Ed Leatherman

_______________________________________________
cisco-voip mailing list
cisco-voip@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-voip