cisco/vyatta VRRP incompatibility - MAC address

21 posts / 0 new
Last post
unrouteable
cisco/vyatta VRRP incompatibility - MAC address

The protocol standards for VRRP (and its Cisco-specific predecessor, HSRP) say that the virtual router IP address should be assigned a special MAC address, so that when there is a transition between the primary and backup routers, the other devices on the network do not need to learn a new MAC address to route traffic.

This is how the protocol is implemented on a Cisco router, and it works just great. When there is a router transition, your network switch learns that the MAC address has moved to a new port, and none of your computers on the network notice any problems.

Alas, Linux does not support using more than one MAC address on the same interface, so the Linux implementation of VRRP (used by Vyatta), keepalived, uses a "trick" instead - it issues gratuitous ARPs to all the hosts on the network whenever keepalived has transited to master or won a router election.

This is nowhere near as reliable as its supposed to be according to the standard - when using the virtual MAC address, only your network switch needs to do anything for a router transition. With the Linux keepalived, every host on your network needs to update their ARP tables for the transition to work.

I'm currently supporting a network that is converting from Cisco to Vyatta - the catch is, during the interim they're using the Cisco as the backup failover if something goes wrong with the Vyatta. But thanks to the non-standard VRRP implementation of Linux/keepalived, there are a few problems:

* when there is a transition from Cisco master to Linux master, some traffic gets lost until all the clients have updated their ARP tables, as the Cisco once it has resigned the master role will not accept traffic on the virtual MAC address that was formerly used for the master. Going the other way is OK as long as the Linux router is still up and routing, since out-of-date clients are sending to the same MAC address as the router's regular interface on that subnet, so it receives the traffic and routes it.

* sometimes the ARP tables of the network devices don't get updated fast enough, or don't get updated at all. If the transition was from the Cisco to the Linux router, then those devices are off the air. I'm still tracking this one down - I think the gratuitous ARP messages from the Linux box are either getting lost or ignored. It varies among different OS's - we haven't seen any Linux boxes fail to update their ARP caches, but have had trouble with Solaris and NetBSD.

Any ideas on how to make the Cisco/Vyatta VRRP work smoother?

stig
Re: cisco/vyatta VRRP incompatibility - MAC address

unrouteable wrote:
Any ideas on how to make the Cisco/Vyatta VRRP work smoother?

I wonder if a mac-vlan could be assigned the vmac on state transition?

Anyway, just yesterday one of the keepalived developers mentioned that they're considering adding support for vmac.
http://sourceforge.net/mailarchive/message.php?msg_name=x2jf41ac2851004141123s3aa5cc35jc37b9d612090a870@mail.gmail.com

unrouteable
Re: cisco/vyatta VRRP incompatibility - MAC address

stig wrote:
unrouteable wrote:
Any ideas on how to make the Cisco/Vyatta VRRP work smoother?

I wonder if a mac-vlan could be assigned the vmac on state transition?

Anyway, just yesterday one of the keepalived developers mentioned that they're considering adding support for vmac.
http://sourceforge.net/mailarchive/message.php?msg_name=x2jf41ac2851004141123s3aa5cc35jc37b9d612090a870@mail.gmail.com

That should work, in theory (although in theory, everything works).
And it all seems doable inside transition scripts.

Looks like first do "insmod macvlan" followed by:

# ip link add link eth0 name eth0_macalias type macvlan
# ip link set eth0_macalias address 00:00:5e:00:01:XX

where XX is the hex number of the VRRP group.

Then we need another transition script to destroy the macvlan when the master resigns, which might be this code:

# ip link delete eth0_macalias

The above examples seem to work on the command line, but I haven't built the lab environment yet to test it. Will let you know.

Speaking of transition scripts, it doesn't look like there's a way to put parameters in the transition scripts, or any kind of environment vars that tell you what the virtual IP or interface is? As it stands now I have to write a separate transition script for each VRRP group (which gets old quickly, I've got a dozen or more), or do dirty tricks with the name of the script and symlinks.

stig
Re: cisco/vyatta VRRP incompatibility - MAC address

unrouteable wrote:
Speaking of transition scripts, it doesn't look like there's a way to put parameters in the transition scripts, or any kind of environment vars that tell you what the virtual IP or interface is? As it stands now I have to write a separate transition script for each VRRP group (which gets old quickly, I've got a dozen or more), or do dirty tricks with the name of the script and symlinks.

I haven't actually tested this, but perhaps this quick hack would be useful:

vyatta@R1:/opt/vyatta/sbin$ diff -u vyatta-vrrp-state.pl.bak vyatta-vrrp-state.pl
--- vyatta-vrrp-state.pl.bak    2010-04-15 14:26:26.000000000 -0700
+++ vyatta-vrrp-state.pl        2010-04-15 14:27:44.000000000 -0700
@@ -88,7 +88,7 @@
 }
 
 if (!($vrrp_transitionscript eq 'null')){
-    exec("$vrrp_transitionscript");
+    exec("$vrrp_transitionscript $vrrp_state $vrrp_intf $vrrp_group");
 }
unrouteable
Re: cisco/vyatta VRRP incompatibility - MAC address

stig wrote:
unrouteable wrote:
Speaking of transition scripts, it doesn't look like there's a way to put parameters in the transition scripts, or any kind of environment vars that tell you what the virtual IP or interface is? As it stands now I have to write a separate transition script for each VRRP group (which gets old quickly, I've got a dozen or more), or do dirty tricks with the name of the script and symlinks.

I haven't actually tested this, but perhaps this quick hack would be useful:

vyatta@R1:/opt/vyatta/sbin$ diff -u vyatta-vrrp-state.pl.bak vyatta-vrrp-state.pl
--- vyatta-vrrp-state.pl.bak    2010-04-15 14:26:26.000000000 -0700
+++ vyatta-vrrp-state.pl        2010-04-15 14:27:44.000000000 -0700
@@ -88,7 +88,7 @@
 }
 
 if (!($vrrp_transitionscript eq 'null')){
-    exec("$vrrp_transitionscript");
+    exec("$vrrp_transitionscript $vrrp_state $vrrp_intf $vrrp_group");
 }

Yes, that works very well! Thanks!

unrouteable
Re: cisco/vyatta VRRP incompatibility - MAC address

unrouteable wrote:

Looks like first do "insmod macvlan" followed by:

# ip link add link eth0 name eth0_macalias type macvlan
# ip link set eth0_macalias address 00:00:5e:00:01:XX

where XX is the hex number of the VRRP group.

Then we need another transition script to destroy the macvlan when the master resigns, which might be this code:

# ip link delete eth0_macalias

The above examples seem to work on the command line, but I haven't built the lab environment yet to test it. Will let you know.

Yes, it all seems to work, except I also had to do "ip link set eth0_macalias up". So it should be possible to provide Cisco compatibility by adding or removing the VRRP virtual MAC address in a transition script, where the transition script also does "arping -U -I eth0_macalias -s " for every TARGET-IP on different network switch segments (i.e. you don't need to bop every host on the LAN, just send out enough packets so that all your switches map the VRRP virtual MAC to the correct port).

This is still an awful kludge - hopefully the keepalived folks will make this unnecessary by adding the necessary functionality to keepalived.

unrouteable
Re: cisco/vyatta VRRP incompatibility - MAC address

unrouteable wrote:
stig wrote:
unrouteable wrote:
Speaking of transition scripts, it doesn't look like there's a way to put parameters in the transition scripts, or any kind of environment vars that tell you what the virtual IP or interface is? As it stands now I have to write a separate transition script for each VRRP group (which gets old quickly, I've got a dozen or more), or do dirty tricks with the name of the script and symlinks.

I haven't actually tested this, but perhaps this quick hack would be useful:

vyatta@R1:/opt/vyatta/sbin$ diff -u vyatta-vrrp-state.pl.bak vyatta-vrrp-state.pl
--- vyatta-vrrp-state.pl.bak    2010-04-15 14:26:26.000000000 -0700
+++ vyatta-vrrp-state.pl        2010-04-15 14:27:44.000000000 -0700
@@ -88,7 +88,7 @@
 }
 
 if (!($vrrp_transitionscript eq 'null')){
-    exec("$vrrp_transitionscript");
+    exec("$vrrp_transitionscript $vrrp_state $vrrp_intf $vrrp_group");
 }

Will the very effective and highly desirable patch above go into Larkspur? Wasn't there when I checked a couple of weeks ago.

stig
Re: cisco/vyatta VRRP incompatibility - MAC address

unrouteable wrote:
Will the very effective and highly desirable patch above go into Larkspur? Wasn't there when I checked a couple of weeks ago.

It didn't make it for the first beta, but I just committed it so it'll be in the final 6.1 version.
http://suva.vyatta.com/git/?p=vyatta-cfg-system.git;a=commitdiff;h=1b22d1bbcf8146736d9f69d1edb57a36e55f6fdc

jschillinger
cisco/vyatta VRRP incompatibility - MAC address

N00b here with a n00b question, sorry.

So, has this been fixed in 6.1? The way I read the documentation it appears to be using a virtual MAC address as it should be.

Thanks!

unrouteable
cisco/vyatta VRRP incompatibility - MAC address

jschillinger wrote:
N00b here with a n00b question, sorry.

So, has this been fixed in 6.1? The way I read the documentation it appears to be using a virtual MAC address as it should be.

Thanks!

I don't think so - nothing in the keepalived change log that says they've added MAC address functionality.

And the Vyatta HAref documentation for 6.1 still talks about changing MAC addresses and using gratuitous ARP. VRRP implementations that use the virtual MAC address do not need to do that.

Quote:
The VRRP group has a unique numeric identifier and is assigned a single virtual IP address (called the virtual IP or VIP). The VIP is linked with the MAC address of the master router. If the master router fails, a new master is elected and the new master notifies the network of its MAC address by issuing a gratuitous ARP.
mahomedsh
cisco/vyatta VRRP incompatibility - MAC address

Hi

Sorry to bring up an old thread but we are trying to do the same thing as unrouteable (except it's between 2 vyattas). We have over 50 VLANs and growing. The gratuitous ARP is currently being ignored by some devices.

Does anyone have the transition scripts that that unrouteable used (or similar). I'm also hoping unrouteable sees this :) but didn't want to PM him/her directly out of politeness.

unrouteable
cisco/vyatta VRRP incompatibility - MAC address

Can you recover connectivity with those devices with the "arping -U" or "arping -a" command? If so, that can be automated with a VRRP transition script.

Another path is to try a bleeding edge version of keepalived - version 1.2.0
has the "use_vmac" option according to their developer mailing list.

The scheme I outlined earlier in the thread is still that, just a scheme, I haven't actually implemented that, I just proved to my satisfaction that it is workable. However, I did build a framework for VRRP transition scripts that should make building that scheme possible, I will release that code in a few days after I've had the chance to double-check it.

I think you will need to assign another IP to the Vyatta pair just for the MACVLAN scheme, and point the errant devices to that IP. If the "arping" command works, it will be easier to use that.

Hey Vyatta folks, is keepalived 1.2 in the pipeline? If not, how hard will it be to port it ourselves?

gilligan
cisco/vyatta VRRP incompatibility - MAC address

unrouteable wrote:

Hey Vyatta folks, is keepalived 1.2 in the pipeline? If not, how hard will it be to port it ourselves?

We're looking at the virtual MAC feature in the keepalived development branch. I have it running in the lab. One issue is that it brings up a new interface (a macvlan interface) for the virtual router. This interface gets a name of the form vrrp.. It assigns the virtual IP address to this interface, not the physical interface it is associated with. So, this would require some non-trivial change to the configuration to support this.

I don't see much discussion of this feature on the keepalived-devel list. If anyone here has experimented with this keepalived feature and would be willing to share their experiences, that would be helpful.

gilligan
cisco/vyatta VRRP incompatibility - MAC address

And BTW, see bug 5707 if you'd like to track this issue.

mahomedsh
cisco/vyatta VRRP incompatibility - MAC address

unrouteable wrote:
Can you recover connectivity with those devices with the "arping -U" or "arping -a" command? If so, that can be automated with a VRRP transition script.

Hi

Thanks for responding. My colleague managed to get the other machines to recognise the new mac by using /usr/bin/arping -A -c5 -I bond0.yyy x.x.x.x where yyy=vif and x.x.x.x is the IP on the interface.

But the thing is that he got that out of /opt/vyatta/sbin/vyatta-vrrp-state.pl which already sends that command.

gilligan wrote:
One issue is that it brings up a new interface (a macvlan interface) for the virtual router. This interface gets a name of the form vrrp.. It assigns the virtual IP address to this interface, not the physical interface it is associated with.

. . .

If anyone here has experimented with this keepalived feature and would be willing to share their experiences, that would be helpful.

We have been trying to experiment with this based on the script stig posted here http://www.vyatta.org/node/4605#comment-17823

We are using a different address for the vrrp and another address for the macvlan interface. The two problems we are facing is that once the macvlan interface comes up, the vrrp communication between the two routers stops.

The second issue is if we have a transistion script assigned to a vif and it completes the failover and becomes master, we are unable to make it backup by reducing the priority. e.g.

Router A - Priority 70 - Master
Router B - Priority 60 - Backup (Showing Router A with priority of 70)

Router A - Priority 50 - Master (it doesn't go to backup)
Router B - Priority 60 - Backup (Should be master, but still shows router A with priority of 70). The only way to make this master is to increase Router B's priority to 71 or more.

mahomedsh
cisco/vyatta VRRP incompatibility - MAC address

A further update on testing we have been doing. We have modified stig's script and removed vrrp on all interfaces except 1. We then attached the transition script to one interface that changes the macvlan interface for all the other interfaces. i.e. script attached to eth0.10 that then creates a new interface for eth0.20, eth0.30 etc.

This is working except for 1 major problem. If the macvlan interface and the main interface have overlapping subnets, then you can't ping the address on the macvlan interface.

This is what I mean:

Doesn't work
eth0.20      172.22.6.11/24                    u/u                              
eth0.20-vr20 172.22.6.10/24                    u/u                              

In the above scenario, machine on 172.22.6.100/24 can ping 172.22.6.11 but not 172.22.6.10.

Works
eth0.20      192.168.0.11/24                    u/u                              
eth0.20-vr20 172.22.6.10/24                    u/u                              

In this case, 172.22.6.10 becomes reachable.

Any ideas why and how we can resolve this?

tapczan
cisco/vyatta VRRP incompatibility - MAC address

Maybe you should use /32 (172.22.6.10/32) as a virtual address?

mahomedsh
cisco/vyatta VRRP incompatibility - MAC address

Thanks for the suggestion tapczan, but unfortunately that didn't work. It's as if eth0.20 will intercept all traffic to the /24 and won't pass it to eth0.20-vr20. But if they are in different subnets then it passes the traffic through.

mahomedsh
cisco/vyatta VRRP incompatibility - MAC address

We believe it is to do with arp resolution failing. The client doesn't receive a response from the router.

unrouteable
cisco/vyatta VRRP incompatibility - MAC address

1) Perhaps there's something time-dependent going on, and vyatta-vrrp-state.pl is sending the ARP stuff too early. You can experiment with VRRP transition scripts, have them start a job which sleeps for 5 seconds and then runs arping again. Ugly but it might be reliable enough untili the Vyatta gang get the latest keepalived integrated.

2) My framework for VRRP transition scripts is attached. It works in Mendocino - it might need slight adjustment for Napa, depending on whether the bootup process changed. The issue is that when a Vyatta router with VRRP enabled boots up, there are VRRP transitions at boot, one after the other, and if the transition script tries to run Vyatta config commands during bootup, it could stop the system from coming up properly. So the script tries to detect bootup state and sleeps until it's over.

3) The original intent of these scripts was to do a failover for an IP address that couldn't be shared via VRRP - for example, an IP in a /30 subnet where there are no spare IPs to run the VRRP protocol. But you can pretty much expand it as necessary - whenever there is a VRRP state change, it can be configured to run a sequence of shell commands, Vyatta op-mode commands, and Vyatta config commands.

4) The documentation is in comments in the script. Basically, set up /etc/vrrp-triggers.conf with the descriptions of your interface and your VRRP group, and then the commands you want to run. Typically, you'll create one set of commands for actions to take during "master" state, and another set of commands for actions in "non-master" (backup or fault) state(s).

vrrp-triggers.sh follows below.

#!/bin/sh


# usage:
#
# create /etc/vrrp-triggers.conf with lines like this:
#
# interface eth2.100 group 100
# state master     delete interface ethernet eth1 disable
# state nonmaster  set    interface ethernet eth1 disable
# state fault      shell  logger -t vrrp-eth2.100 "ouch!"
# state backup     run    clear vrrp master interface eth2.100 group 100
#
# interface eth3 group 150
# [...]
#
# link this script to a VRRP group with something like :
#  set interface ethernet eth0 vif 10 vrrp vrrp-group 10 run-transition-scripts backup /usr/local/sbin/vrrp-triggers.sh
#  set interface ethernet eth0 vif 10 vrrp vrrp-group 10 run-transition-scripts fault /usr/local/sbin/vrrp-triggers.sh
#  set interface ethernet eth0 vif 10 vrrp vrrp-group 10 run-transition-scripts master /usr/local/sbin/vrrp-triggers.sh
#
# then this script will run the commands associated with the above states
# whenever a transition occurs.

# documentation for /etc/vrrp-triggers.conf
#
# interface  group 
#  is the name of the interface that VRRP is running on.  Must
# match Linux syntax, i.e. "eth2.100" for Vyatta "eth2 vif 100".
#  is the VRRP group number.  All "state" lines listed after
# this statement, until the next "interface" line, will be run
# for VRRP transitions for this interface and VRRP group.
#
# state   
#  is one of "backup", "fault", "master", or "nonmaster".
# "nonmaster" matches states other than "master".
#
#  is one of "set" (run Vyatta "set" command in config mode), "delete"
# (run Vyatta "delete" command in config mode), "run" (run Vyatta op-mode
# command), and "shell" (run shell command).
#
#  is the rest of the command you want run, as in the examples
# above.


# how many seconds to wait for system startup to complete.  doing config
# changes while /etc/init.d/vyatta-router is running may cause damage 
maxinitwait=30

# how old a lockdir can be (in seconds) - if it is too old remove it and
# proceed anyway
maxlockdelay=60




LOGOPT=

debug=false
if [ ${DEBUG:-no} = YES ]
then
	set -x
	LOGOPT=-s
	debug=true
fi


bomb() {
	logger $LOGOPT -t "${ID}-$PIDSTR" "$1"
	exit 23
}

log() {
	logger $LOGOPT -t "${ID}-$PIDSTR" "$1"
}

vyattacfg() {
CFGWRAP=/opt/vyatta/sbin/vyatta-cfg-cmd-wrapper

$CFGWRAP begin
while read cmdline
do
        eval $CFGWRAP $cmdline
done
$CFGWRAP commit
$CFGWRAP end

}

vyattarun() {
unset _OFR_CONFIGURE
while read cmdline
do
	echo $cmdline | vbash --noediting --noprofile -i
done
}

run_action() {
action=$1
case $action in
set*|delete*) echo $action | vyattacfg ; log "$action" ;;
run*) echo ${action##run}  | vyattarun ; log "$action" ;;
shell*) echo ${action##shell} | sh ;     log "$action" ;;
*) log "unrecognized action in $configfile: $action" ;;
esac
}

# trim leading space/tabs
trim_line() {
line=$1
while [ "${line:0:1}" = " " -o "${line:0:1}" = "	" ]
do
	line=${line## }
	line=${line##	}
done
echo $line
}


set -u

PIDSTR="[$$]"

trap 'logger -t ${ID:-$0}-$PIDSTR "error exit - temp file in ${TMPFILE:-XXX}"' EXIT

ID=vrrp-triggers
CONFIGFILE=/etc/vrrp-triggers.conf

log "args: ${*:-missing, pre VC6.1 systems need vyatta-vrrp-state.pl patch}"
state=${1}; shift
interface=${1}; shift
vrrpgroup=${1}; shift

PIDSTR="$state[$$]"

if [ ! -s $CONFIGFILE ]
then
	bomb "please put desired actions in $CONFIGFILE, read $0 for details"
fi

TMPFILE=$(mktemp /tmp/vrrp-${interface}-${state}-${vrrpgroup}-XXXXXXXXXX)
if [ ! -w $TMPFILE ]
then
	log "mktemp fails - cannot write $TMPFILE - cannot log all errors"
	TMPFILE=""
else
	if $debug
	then
		exec >$TMPFILE
	else
		exec >$TMPFILE 2>&1
	fi
fi


ID="${ID}-${interface}-${vrrpgroup}"
LOCKDIR="/var/run/$ID.lock"

starttime=$(date +%s)

while [ $(expr $(date +%s) - $starttime) -lt $maxinitwait ]
do

	# these patterns may be Mendocino-specific - should generalize
	# with config file
	#
	if ps auxwww | egrep "/etc/init.d/vyatta-router|opt/vyatta/sbin/vyatta-boot-config-loader" | egrep -v egrep >/dev/null
	then
		log "Vyatta startup still running, will sleep a little"
		sleep 2
		continue
	fi

	# prevent re-entrancy
	if mkdir $LOCKDIR
	then
		break # we have the lock
	else
		rc=$?
		if [ ! -d $LOCKDIR ]
		then
			log "mkdir of lockdir failed, status $rc, using plan B"
			# uh oh, mkdir failed for a reason other than
			# directory exists.  something is very worng
			sleep 10
			break    # and proceed without the lock
		fi
		locktime=$(stat --format %Y $LOCKDIR)
		now=$(date +%s)
		age=$(expr $now - $locktime)
		if [ $age -gt $maxlockdelay ]
		then
			log "$LOCKDIR is $age seconds old, over $maxlockdelay seconds, ignoring"
			touch $LOCKDIR # freshen up the lock while we're using it
			break
		else
			log "$LOCKDIR exists, last modified at $locktime ($age seconds ago), will sleep a little"
			sleep 2
		fi
	fi
done

trap 'rm -rf $LOCKDIR; logger -t "${ID:-$0}-$PIDSTR" "error exit - temp file in ${TMPFILE:-XXX}"' EXIT

cfginterface=""

while read cfgline
do
	# skip blank lines
	if [ -z "$cfgline" ]
	then continue
	fi

	cfgline=$(trim_line "$cfgline")

	case $cfgline in
	"#*") continue ;;   # it's a comment
	interface*)
		set $cfgline
		if [ "${3:-XXX}" != "group" ]
		then log "syntax error in $CONFIGFILE: $cfgline"
		fi
		cfginterface=${2:-XXX}
		cfggroup=${4:-XXX}
		;;
	state*)
		if [ "$interface" = "$cfginterface" -a "$vrrpgroup" = "$cfggroup" ]
		then
			stateline=$(trim_line "${cfgline##state}")
			actionline=$(trim_line "${stateline#*[ 	]}")
			cfgstate=$(trim_line "${stateline%%[ 	]*}")
			cfgstate=${cfgstate%[ 	]*}

			case $cfgstate in
			master|backup|fault|nonmaster) ;;
			*) log "unrecognized state in $CONFIGFILE: $cfgstate";;
			esac

			if [ "$cfgstate" = "$state" ]  ||
			[ "$cfgstate" = "nonmaster" -a "$state" = "backup" ] ||
			[ "$cfgstate" = "nonmaster" -a "$state" = "fault" ]
			then
				run_action "$actionline"
			fi
		else
			continue  # ignore, not this interface
		fi
		;;
	*)
	esac
done < $CONFIGFILE

rm -rf $LOCKDIR # don't need to hold it any longer

if [ -s $TMPFILE ]
then
	log "state change actions produced output - appending to syslog"
	logger -t ${ID}-$PIDSTR -f $TMPFILE
	log "state change actions produced output, logged above"
fi

trap 'rm -f $TMPFILE; logger -t ${ID}-$PIDSTR "normal exit"' EXIT

exit 0
shemminger
cisco/vyatta VRRP incompatibility - MAC address

One issue with using macvlan for this is that it conflicts with direct macvlan support in CLI (see pseudo-ethernet).

Log in or register to post comments