Mailing List Archive

Odd issue with ARP in different subnet
I have run into an odd issue with ARP on an EX switch that I think is a
bug in JUNOS, but I wanted to see what others thought before I tried
JTAC (maybe I'm missing something).

I have an EX2200 switch that cannot talk to one of my recursive DNS
servers. The switch is in subnet a.b.c.0/27, while the DNS IP is in
x.y.z.0/29. The DNS IP is anycasted, and the primary server serving it
is in the same a.c.b.0/27 subnet as the switch (the DNS IP is a
secondary IP on the same interface).

When the switch tries to reach the DNS IP, it sends the packet to the
default router. The router sends it to the server, and the server sends
an ARP request for the switch's IP. The sending IP address in the ARP
request is the DNS IP. As far as I can tell, JUNOS doesn't send a
response to the ARP request.

I'm guessing that it isn't sending a response because the sending IP is
in a different subnet, but as far as I can tell from reading the ARP RFC
(826), that is not supposed to figure into an ARP response.

The DNS server is Linux, and I can see Linux will respond to
out-of-subnet ARP requests. I also have an old Cisco switch in the same
subnet, and it also responds to out-of-subnet ARP requests.

If I ping the switch from the Linux server, the ARP request goes out
with the IP in the same subnet, the switch responds, the Linux server
gets an ARP cache entry, and communication works both ways for all IPs
until the ARP cache entry expires on the Linux side.

--
Chris Adams <cmadams@hiwaay.net>
Systems and Network Administrator - HiWAAY Internet Services
I don't speak for anybody but myself - that's enough trouble.
_______________________________________________
juniper-nsp mailing list juniper-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/juniper-nsp
Re: Odd issue with ARP in different subnet [ In reply to ]
Check the default router config.

When the server sends the arp request, the router should reply with
it's own MAC address
Does it not have a route back to the switch?


On Wed, 9 Mar 2011 09:43:43 -0600, Chris Adams wrote:
> I have run into an odd issue with ARP on an EX switch that I think is
> a
> bug in JUNOS, but I wanted to see what others thought before I tried
> JTAC (maybe I'm missing something).
>
> I have an EX2200 switch that cannot talk to one of my recursive DNS
> servers. The switch is in subnet a.b.c.0/27, while the DNS IP is in
> x.y.z.0/29. The DNS IP is anycasted, and the primary server serving
> it
> is in the same a.c.b.0/27 subnet as the switch (the DNS IP is a
> secondary IP on the same interface).
>
> When the switch tries to reach the DNS IP, it sends the packet to the
> default router. The router sends it to the server, and the server
> sends
> an ARP request for the switch's IP. The sending IP address in the
> ARP
> request is the DNS IP. As far as I can tell, JUNOS doesn't send a
> response to the ARP request.
>
> I'm guessing that it isn't sending a response because the sending IP
> is
> in a different subnet, but as far as I can tell from reading the ARP
> RFC
> (826), that is not supposed to figure into an ARP response.
>
> The DNS server is Linux, and I can see Linux will respond to
> out-of-subnet ARP requests. I also have an old Cisco switch in the
> same
> subnet, and it also responds to out-of-subnet ARP requests.
>
> If I ping the switch from the Linux server, the ARP request goes out
> with the IP in the same subnet, the switch responds, the Linux server
> gets an ARP cache entry, and communication works both ways for all
> IPs
> until the ARP cache entry expires on the Linux side.

_______________________________________________
juniper-nsp mailing list juniper-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/juniper-nsp
Re: Odd issue with ARP in different subnet [ In reply to ]
There us a keyword like primary or preferred that tells the switch which ip address to use to source traffic on a particular interface. It doesn't quite fit the behavior I remember but it could be part of the problem. What does the routing table say?

Sent from my iPhone

On Mar 9, 2011, at 10:43 AM, Chris Adams <cmadams@hiwaay.net> wrote:

> I have run into an odd issue with ARP on an EX switch that I think is a
> bug in JUNOS, but I wanted to see what others thought before I tried
> JTAC (maybe I'm missing something).
>
> I have an EX2200 switch that cannot talk to one of my recursive DNS
> servers. The switch is in subnet a.b.c.0/27, while the DNS IP is in
> x.y.z.0/29. The DNS IP is anycasted, and the primary server serving it
> is in the same a.c.b.0/27 subnet as the switch (the DNS IP is a
> secondary IP on the same interface).
>
> When the switch tries to reach the DNS IP, it sends the packet to the
> default router. The router sends it to the server, and the server sends
> an ARP request for the switch's IP. The sending IP address in the ARP
> request is the DNS IP. As far as I can tell, JUNOS doesn't send a
> response to the ARP request.
>
> I'm guessing that it isn't sending a response because the sending IP is
> in a different subnet, but as far as I can tell from reading the ARP RFC
> (826), that is not supposed to figure into an ARP response.
>
> The DNS server is Linux, and I can see Linux will respond to
> out-of-subnet ARP requests. I also have an old Cisco switch in the same
> subnet, and it also responds to out-of-subnet ARP requests.
>
> If I ping the switch from the Linux server, the ARP request goes out
> with the IP in the same subnet, the switch responds, the Linux server
> gets an ARP cache entry, and communication works both ways for all IPs
> until the ARP cache entry expires on the Linux side.
>
> --
> Chris Adams <cmadams@hiwaay.net>
> Systems and Network Administrator - HiWAAY Internet Services
> I don't speak for anybody but myself - that's enough trouble.
> _______________________________________________
> juniper-nsp mailing list juniper-nsp@puck.nether.net
> https://puck.nether.net/mailman/listinfo/juniper-nsp
>

_______________________________________________
juniper-nsp mailing list juniper-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/juniper-nsp
Re: Odd issue with ARP in different subnet [ In reply to ]
Once upon a time, Gordon Smith <gordon@gswsystems.com> said:
> Check the default router config.
>
> When the server sends the arp request, the router should reply with
> it's own MAC address
> Does it not have a route back to the switch?

No, the router isn't proxy ARPing. Let me put some IPs to the problem:

EX switch: 10.1.1.5/27
Linux server eth0: 10.1.1.10/27
router (M10i): 10.1.1.30/27
DNS IP: 10.2.2.2/32 (secondary IP on Linux server eth0)

EX wants to reach 10.2.2.2, so it sends the packet to the M10i at
10.1.1.30. Router has route for 10.2.2.2/32 pointing to 10.1.1.10, so
it sends the packet to the Linux server. Linux server realizes it
doesn't need to route back to EX in the same subnet and is going to send
a packet directly from 10.2.2.2 to 10.1.1.5. Linux server doesn't have
an ARP entry for 10.1.1.5, so it sends an ARP request, using a source IP
of 10.2.2.2 (since that's the source of the desired packet).

At this point the EX sees the ARP request for its IP, but doesn't
respond to it. I'm guessing it is ignoring the ARP request because the
source IP is in a different subnet (but that's just a guess).

There's also an old Cisco switch on the same segment, and it replies to
out-of-subnet ARP requests just fine. I also tried a FreeBSD host in a
similar setup with a different Linux server, and it also works okay. I
don't have any other OSes handy to try.

Per another email, I tried setting the Linux server to put the DNS IP on
a loopback interface instead of the ethernet, but it still sent the ARP
request with the DNS IP as the source.

--
Chris Adams <cmadams@hiwaay.net>
Systems and Network Administrator - HiWAAY Internet Services
I don't speak for anybody but myself - that's enough trouble.
_______________________________________________
juniper-nsp mailing list juniper-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/juniper-nsp
Re: Odd issue with ARP in different subnet [ In reply to ]
On Wed, Mar 09, 2011 at 09:43:43AM -0600, Chris Adams wrote:
> I have an EX2200 switch that cannot talk to one of my recursive DNS
> servers. The switch is in subnet a.b.c.0/27, while the DNS IP is in
> x.y.z.0/29. The DNS IP is anycasted, and the primary server serving it
> is in the same a.c.b.0/27 subnet as the switch (the DNS IP is a
> secondary IP on the same interface).

Why don't you set a secondary IP address on the switch on the same
subnet x.y.z.0/29?
_______________________________________________
juniper-nsp mailing list juniper-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/juniper-nsp
Re: Odd issue with ARP in different subnet [ In reply to ]
Put the /32 on a loopback instead of a secondary address.

See
http://www.netlinxinc.com/netlinx-blog/45-dns/119-anycast-dns-part-2-using-static-routes-.html

You could always fire up WireShark and watch exactly what's going on on
the wire



On Wed, 9 Mar 2011 18:56:44 -0600, Chris Adams wrote:
> Once upon a time, Gordon Smith <gordon@gswsystems.com> said:
>> Check the default router config.
>>
>> When the server sends the arp request, the router should reply with
>> it's own MAC address
>> Does it not have a route back to the switch?
>
> No, the router isn't proxy ARPing. Let me put some IPs to the
> problem:
>
> EX switch: 10.1.1.5/27
> Linux server eth0: 10.1.1.10/27
> router (M10i): 10.1.1.30/27
> DNS IP: 10.2.2.2/32 (secondary IP on Linux server eth0)
>
> EX wants to reach 10.2.2.2, so it sends the packet to the M10i at
> 10.1.1.30. Router has route for 10.2.2.2/32 pointing to 10.1.1.10,
> so
> it sends the packet to the Linux server. Linux server realizes it
> doesn't need to route back to EX in the same subnet and is going to
> send
> a packet directly from 10.2.2.2 to 10.1.1.5. Linux server doesn't
> have
> an ARP entry for 10.1.1.5, so it sends an ARP request, using a source
> IP
> of 10.2.2.2 (since that's the source of the desired packet).
>
> At this point the EX sees the ARP request for its IP, but doesn't
> respond to it. I'm guessing it is ignoring the ARP request because
> the
> source IP is in a different subnet (but that's just a guess).
>
> There's also an old Cisco switch on the same segment, and it replies
> to
> out-of-subnet ARP requests just fine. I also tried a FreeBSD host in
> a
> similar setup with a different Linux server, and it also works okay.
> I
> don't have any other OSes handy to try.
>
> Per another email, I tried setting the Linux server to put the DNS IP
> on
> a loopback interface instead of the ethernet, but it still sent the
> ARP
> request with the DNS IP as the source.

_______________________________________________
juniper-nsp mailing list juniper-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/juniper-nsp
Re: Odd issue with ARP in different subnet [ In reply to ]
On Wed, Mar 9, 2011 at 7:56 PM, Chris Adams <cmadams@hiwaay.net> wrote:

> Once upon a time, Gordon Smith <gordon@gswsystems.com> said:
> > Check the default router config.
> >
> > When the server sends the arp request, the router should reply with
> > it's own MAC address
> > Does it not have a route back to the switch?
>
> No, the router isn't proxy ARPing. Let me put some IPs to the problem:
>
> EX switch: 10.1.1.5/27
> Linux server eth0: 10.1.1.10/27
> router (M10i): 10.1.1.30/27
> DNS IP: 10.2.2.2/32 (secondary IP on Linux server eth0)
>
> EX wants to reach 10.2.2.2, so it sends the packet to the M10i at
> 10.1.1.30. Router has route for 10.2.2.2/32 pointing to 10.1.1.10, so
> it sends the packet to the Linux server. Linux server realizes it
> doesn't need to route back to EX in the same subnet and is going to send
> a packet directly from 10.2.2.2 to 10.1.1.5. Linux server doesn't have
> an ARP entry for 10.1.1.5, so it sends an ARP request, using a source IP
> of 10.2.2.2 (since that's the source of the desired packet).
>

I don't think the server should arp for 10.1.1.5 from 10.2.2.2. Devices
don't usually arp (or answer) for things that aren't in the same subnet. If
it has a static route it would arp for the destination of it's static route
and then create a packet with the source IP of 10.2.2.2 and dest ip of
whatever the final IP is and a destination mac of the arp reply from the
next hop in the static route. Does the EX4200 have a route pointing
10.2.2.2 to 10.1.1.10? If so it should do the above. If not it will
continue sending the packets to it's default gateway. This should actually
work if the EX sends packets to it's default gateway and the linux server
replies directly back even though it's asymetric. Do get any ICMP packets
in response?


> At this point the EX sees the ARP request for its IP, but doesn't
> respond to it. I'm guessing it is ignoring the ARP request because the
> source IP is in a different subnet (but that's just a guess).


> There's also an old Cisco switch on the same segment, and it replies to
> out-of-subnet ARP requests just fine. I also tried a FreeBSD host in a
> similar setup with a different Linux server, and it also works okay. I
> don't have any other OSes handy to try.
>
> Per another email, I tried setting the Linux server to put the DNS IP on
> a loopback interface instead of the ethernet, but it still sent the ARP
> request with the DNS IP as the source.
>
> --
> Chris Adams <cmadams@hiwaay.net>
> Systems and Network Administrator - HiWAAY Internet Services
> I don't speak for anybody but myself - that's enough trouble.
> _______________________________________________
> juniper-nsp mailing list juniper-nsp@puck.nether.net
> https://puck.nether.net/mailman/listinfo/juniper-nsp
>
>
_______________________________________________
juniper-nsp mailing list juniper-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/juniper-nsp
Re: Odd issue with ARP in different subnet [ In reply to ]
Once upon a time, Gordon Smith <gordon@gswsystems.com> said:
> Put the /32 on a loopback instead of a secondary address.

Um, I did that (as I said at the end of the email). It didn't change
anything.

> You could always fire up WireShark and watch exactly what's going on on
> the wire

That's how I got all the info; I ran tcpdump on the Linux server and
"monitor traffic interface me0" (which is really tcpdump) on the EX.
Both sides see the same packets; the EX just doesn't respond to the ARP
requests.

--
Chris Adams <cmadams@hiwaay.net>
Systems and Network Administrator - HiWAAY Internet Services
I don't speak for anybody but myself - that's enough trouble.


> On Wed, 9 Mar 2011 18:56:44 -0600, Chris Adams wrote:
> >Once upon a time, Gordon Smith <gordon@gswsystems.com> said:
> >>Check the default router config.
> >>
> >>When the server sends the arp request, the router should reply with
> >>it's own MAC address
> >>Does it not have a route back to the switch?
> >
> >No, the router isn't proxy ARPing. Let me put some IPs to the
> >problem:
> >
> >EX switch: 10.1.1.5/27
> >Linux server eth0: 10.1.1.10/27
> >router (M10i): 10.1.1.30/27
> >DNS IP: 10.2.2.2/32 (secondary IP on Linux server eth0)
> >
> >EX wants to reach 10.2.2.2, so it sends the packet to the M10i at
> >10.1.1.30. Router has route for 10.2.2.2/32 pointing to 10.1.1.10,
> >so
> >it sends the packet to the Linux server. Linux server realizes it
> >doesn't need to route back to EX in the same subnet and is going to
> >send
> >a packet directly from 10.2.2.2 to 10.1.1.5. Linux server doesn't
> >have
> >an ARP entry for 10.1.1.5, so it sends an ARP request, using a source
> >IP
> >of 10.2.2.2 (since that's the source of the desired packet).
> >
> >At this point the EX sees the ARP request for its IP, but doesn't
> >respond to it. I'm guessing it is ignoring the ARP request because
> >the
> >source IP is in a different subnet (but that's just a guess).
> >
> >There's also an old Cisco switch on the same segment, and it replies
> >to
> >out-of-subnet ARP requests just fine. I also tried a FreeBSD host in
> >a
> >similar setup with a different Linux server, and it also works okay.
> >I
> >don't have any other OSes handy to try.
> >
> >Per another email, I tried setting the Linux server to put the DNS IP
> >on
> >a loopback interface instead of the ethernet, but it still sent the
> >ARP
> >request with the DNS IP as the source.
_______________________________________________
juniper-nsp mailing list juniper-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/juniper-nsp
Re: Odd issue with ARP in different subnet [ In reply to ]
Once upon a time, Keegan Holley <keegan.holley@sungard.com> said:
> I don't think the server should arp for 10.1.1.5 from 10.2.2.2. Devices
> don't usually arp (or answer) for things that aren't in the same subnet.

Can you name anything (other than the EX) that behaves as you describe?
Obviously Linux will send such ARPs (which I believe is proper as the
IPs are both local), and Linux, Cisco IOS, and FreeBSD all respond to
such ARPs.

As I said, reading the ARP RFC would tend to indicate it is expected
behavior; the method for replying to an ARP request says nothing about
checking the source IP address. If your IP is in the destination field
of a request, you swap destination and source MAC and IP (putting your
MAC in the new source) and send the reply.

--
Chris Adams <cmadams@hiwaay.net>
Systems and Network Administrator - HiWAAY Internet Services
I don't speak for anybody but myself - that's enough trouble.
_______________________________________________
juniper-nsp mailing list juniper-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/juniper-nsp
Re: Odd issue with ARP in different subnet [ In reply to ]
On 3/9/11 9:04 PM, Chris Adams wrote:
> That's how I got all the info; I ran tcpdump on the Linux server and
> "monitor traffic interface me0" (which is really tcpdump) on the EX.
> Both sides see the same packets; the EX just doesn't respond to the ARP
> requests.
>
Linux does quirky stuff with ARPing by default. Try this in your
/etc/sysctl.conf (then sysctl -p to load it).

net.ipv4.conf.bond0.arp_ignore = 1
net.ipv4.conf.bond0.arp_announce = 2

Replace bond0 with the interface facing your EX.

Docs/explanation are here:

http://kb.linuxvirtualserver.org/wiki/Using_arp_announce/arp_ignore_to_disable_ARP

David
_______________________________________________
juniper-nsp mailing list juniper-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/juniper-nsp
Re: Odd issue with ARP in different subnet [ In reply to ]
Once upon a time, Brandon Ross <bross@torreypoint.com> said:
> *sigh* Since no one else actually seems to be answering the question
> you've actually asked.

Well, I was wondering if I was wording my problem wrong or something.
Basically, there is obviously a problem between Linux and JUNOS on an EX
(haven't tried a router in this setup), and I wanted to know if this is
a JUNOS bug or some switch-specific behavior that I had missed.

> Yes, this appears to be a bug. Juniper could argue that they have simply
> interpreted the RFC differently than Linux and Cisco, and it's probably a
> reasonable argument. I haven't personally studied the RFC in detail, but
> if they don't mention a requirement of checking the source IP to make sure
> it's on-net, then I see no reason why doing so would be a good idea.

Being an older RFC (from 1982), it is much less structured than current
format, but it is pretty straight-forward in how an incoming ARP request
is to be handled:

?Do I have the hardware type in ar$hrd?
Yes: (almost definitely)
[optionally check the hardware length ar$hln]
?Do I speak the protocol in ar$pro?
Yes:
[optionally check the protocol length ar$pln]
Merge_flag := false
If the pair <protocol type, sender protocol address> is
already in my translation table, update the sender
hardware address field of the entry with the new
information in the packet and set Merge_flag to true.
?Am I the target protocol address?
Yes:
If Merge_flag is false, add the triplet <protocol type,
sender protocol address, sender hardware address> to
the translation table.
?Is the opcode ares_op$REQUEST? (NOW look at the opcode!!)
Yes:
Swap hardware and protocol fields, putting the local
hardware and protocol addresses in the sender fields.
Set the ar$op field to ares_op$REPLY
Send the packet to the (new) target hardware address on
the same hardware on which the request was received.

There's nothing in there that says to filter based on the source
protocol address (which is still just my guess as to what JUNOS is
doing).

I guess I'll try JTAC tomorrow.
--
Chris Adams <cmadams@hiwaay.net>
Systems and Network Administrator - HiWAAY Internet Services
I don't speak for anybody but myself - that's enough trouble.
_______________________________________________
juniper-nsp mailing list juniper-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/juniper-nsp
Re: Odd issue with ARP in different subnet [ In reply to ]
On 03/09/2011 03:43 PM, Chris Adams wrote:
> I have run into an odd issue with ARP on an EX switch that I think is a
> bug in JUNOS, but I wanted to see what others thought before I tried
> JTAC (maybe I'm missing something).
>
> I have an EX2200 switch that cannot talk to one of my recursive DNS
> servers. The switch is in subnet a.b.c.0/27, while the DNS IP is in
> x.y.z.0/29. The DNS IP is anycasted, and the primary server serving it
> is in the same a.c.b.0/27 subnet as the switch (the DNS IP is a
> secondary IP on the same interface).
>
> When the switch tries to reach the DNS IP, it sends the packet to the
> default router. The router sends it to the server, and the server sends
> an ARP request for the switch's IP. The sending IP address in the ARP
> request is the DNS IP. As far as I can tell, JUNOS doesn't send a
> response to the ARP request.

Yeah, we see this a lot. You need the following in /etc/sysctl.conf:

# These values make linux be sensible about making and replying
# to ARP requests - specifically they force ARP requests to come
# from an in-subnet IP, and ignore ARP replies for out-of-subnet
# addresses
net.ipv4.conf.all.arp_ignore = 1
net.ipv4.conf.all.arp_announce = 2

...and then you need to move the anycast /32 to the loopback (lo) interface.
_______________________________________________
juniper-nsp mailing list juniper-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/juniper-nsp
Re: Odd issue with ARP in different subnet [ In reply to ]
On 03/10/2011 02:09 AM, Chris Adams wrote:
> Once upon a time, Keegan Holley<keegan.holley@sungard.com> said:
>> I don't think the server should arp for 10.1.1.5 from 10.2.2.2. Devices
>> don't usually arp (or answer) for things that aren't in the same subnet.
>
> Can you name anything (other than the EX) that behaves as you describe?

We've seen this cause problems with other (non-Juniper) things. I think
Windows barfs on it in certain versions. It's probably a bug in JunOS in
theory, but in my experience you'll need to change the ARP sysctl values
to be 100% sure of it working.
_______________________________________________
juniper-nsp mailing list juniper-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/juniper-nsp