Mailing List Archive

[lvs-users] LVS/DR Packet Loss Problem
Hi list,

Has anyone experienced packet loss issues on real servers in an LVS/DR
configuration? I don't believe this is an LVS issue per se, but I am
curious if anyone else has experienced the same problem.

What I am seeing is packets are being forwarded from the director to the
real server, but once they arrive at the real server there is packets
loss such that it totally breaks communication with the client (at the
TCP level). It seems to be fairly specific to loosing delayed ACKs. I
see this reflected in the /proc/net/netstat statistics on the client
(DelayedACKLost and TCPDSACKOldSent) and in a side-by-side comparison of
tcpdump captures on the real server's receiving interface (bond1) and
transmitting interface (bond0). The dummy0 interface is configured with
the VIP on the real server and I have tried using the loopback interface
with the same result. We have been using this same configuration for
several years, so I am certain that there are no incorrect sysctl values
(e.g., arp_announce, arp_ignore, etc) and there is no interference from
netfilter.

This was first noticed on the SLES11 SP1 kernel, which is based on
2.6.32. In the process of gathering information to open a case with the
vendor, I found that the same problem occurrs on the kernel.org
2.6.32.28 version. Further testing revealed that the issue was
reproducible on all kernel.org versions (2.6.[32-36]) up to stable
2.6.37, when it no longer happens. It also works on the SLES11 SP0
kernel, which is 2.6.27 based. I have looked through the git changelogs
for the kernel across all versions and there do no seem to be any
changes that would indicate why this is happening, but I don't claim to
be an expert here. I have also searched the LVS mailing list archives in
some depth and found nothing.

If anyone has had a similar experience, I would appreciate any
information you may be able to share.


Thanks!!!



_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users@LinuxVirtualServer.org
Send requests to lvs-users-request@LinuxVirtualServer.org
or go to http://lists.graemef.net/mailman/listinfo/lvs-users
Re: [lvs-users] LVS/DR Packet Loss Problem [ In reply to ]
Quoting btinsley <btinsley@gmail.com>:
> Hi list,
>
> Has anyone experienced packet loss issues on real servers in an LVS/DR
> configuration? I don't believe this is an LVS issue per se, but I am
> curious if anyone else has experienced the same problem.
>
> What I am seeing is packets are being forwarded from the director to the
> real server, but once they arrive at the real server there is packets
> loss such that it totally breaks communication with the client (at the
> TCP level). It seems to be fairly specific to loosing delayed ACKs. I
> see this reflected in the /proc/net/netstat statistics on the client
> (DelayedACKLost and TCPDSACKOldSent) and in a side-by-side comparison of
> tcpdump captures on the real server's receiving interface (bond1) and
> transmitting interface (bond0). The dummy0 interface is configured with
> the VIP on the real server and I have tried using the loopback interface
> with the same result. We have been using this same configuration for
> several years, so I am certain that there are no incorrect sysctl values
> (e.g., arp_announce, arp_ignore, etc) and there is no interference from
> netfilter.
>
> This was first noticed on the SLES11 SP1 kernel, which is based on
> 2.6.32. In the process of gathering information to open a case with the
> vendor, I found that the same problem occurrs on the kernel.org
> 2.6.32.28 version. Further testing revealed that the issue was
> reproducible on all kernel.org versions (2.6.[32-36]) up to stable
> 2.6.37, when it no longer happens. It also works on the SLES11 SP0
> kernel, which is 2.6.27 based. I have looked through the git changelogs
> for the kernel across all versions and there do no seem to be any
> changes that would indicate why this is happening, but I don't claim to
> be an expert here. I have also searched the LVS mailing list archives in
> some depth and found nothing.
>
> If anyone has had a similar experience, I would appreciate any
> information you may be able to share.

Yes--I've seen it with the lvs bundled with RHEL4 and 5 DR, and it
seems to particularly anger connections from Windows 7 SSL/TLS
clients. I'll give .37 a shot and let you know what I see!

cc



_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users@LinuxVirtualServer.org
Send requests to lvs-users-request@LinuxVirtualServer.org
or go to http://lists.graemef.net/mailman/listinfo/lvs-users