[
Guest fencing on a RHEL KVM host
]

4 years, 10 months ago

In my home I have a HP N36L running Red Hat 6 (currently 6.3) that I use as a KVM host to do my experiments and where sometimes I need to create vm clusters using RHEL HA. To ensure data integrity one of RHEL HA requirements is to have a fence device and in my case (a single KVM host) I can use the fence_virtd daemon with libvirt as a backend module.

The instructions below are for configuring the fence device on RHEL 6 hosts and guests, but they can be ported to RHEL 5 as well with very little modifications.

Host configuration:

Let's start Installing the necessary packages on the host:

# yum install -y fence-virt fence-virtd fence-virtd-libvirt fence-virtd-multicast

Then create the key needed for the host to authenticate all the fencing requests from the guest cluster:

# mkdir /etc/cluster
# dd if=/dev/urandom of=/etc/cluster/fence_xvm.key bs=4096 count=1

Fix selinux permissions if needed:

# restorecon -Rv /etc/cluster/

The key should then be copied to /etc/cluster/fence_xvm.key of every guest.

Now run the configuration tool:

# fence_virtd -c

and remember to use libvirt as Backend module:

Backend module [checkpoint]: libvirt

The configuration will be saved to /etc/fence_virt.conf and should look like this one:

backends {
    libvirt {
        uri = "qemu:///system";
    }

}

listeners {
    multicast {
        port = "1229";
        family = "ipv4";
        address = "225.0.0.12";
        key_file = "/etc/cluster/fence_xvm.key";
    }

}

fence_virtd {
    module_path = "/usr/lib64/fence-virt";
    backend = "libvirt";
    listener = "multicast";
}

If you are using iptables you have to add some rules on both host and guests.
On the host add the following lines to your /etc/sysconfig/iptables file:

-A INPUT -p udp -m state --state NEW -m udp --dport 1229 -j ACCEPT 
-A INPUT -d 225.0.0.12/32 -p igmp -j ACCEPT

On the guests:

-A INPUT -p tcp -m state --state NEW -m tcp --dport 1229 -j ACCEPT

Depending on your environment you may also have to allow the all-hosts group multicast address on both host and guests:

-A INPUT -d 224.0.0.1/32 -p igmp -j ACCEPT

Remember to add those rules before the line -A INPUT -j REJECT --reject-with icmp-host-prohibited.
To apply the new rules restart iptables:

# service iptables restart

Now start the fence daemon on the host:

# service fence_virtd start

If you don't have the cluster software installed on the host (and since you are running everything on a single host you probably don't have it) you can automatically start the fence daemon at boot running the following commands:

# chkconfig --add fence_virtd
# chkconfig fence_virtd on

Test the fence daemon from the host and the guests (it may list not clustered vm too):

# fence_xvm -o list
cluster_node1             f2bd9d70-411e-393c-0720-3311985a63bf on
cluster_node2             3fd4a9cd-aa43-11b1-a943-9f8623b790b3 on

You can also force a guest reboot:

# fence_xvm -o reboot -H cluster_node2

If something is not working and you need to debug use the "-d" option (add more "d" to increase debugging level):

# fence_xvm -o reboot -H cluster_node2 -t 5 -ddddddd
Debugging threshold is now 7
-- args @ 0x7fffe1ccdbf0 --
  args->domain = cluster_node2
  args->op = 2
  args->net.key_file = /etc/cluster/fence_xvm.key
  args->net.hash = 2
  args->net.addr = 225.0.0.12
  args->net.auth = 2
  args->net.port = 1229
  args->net.ifindex = 0
  args->net.family = 2
  args->timeout = 5
  args->retr_time = 20
  args->flags = 0
  args->debug = 7
-- end args --
Reading in key file /etc/cluster/fence_xvm.key into 0x7fffe1cccac0 (4096 max size)
Actual key length = 4096 bytes
Build IP address list
Connecting to Netlink...
Sending address dump request
Waiting for response
Received 128 bytes
Closing Netlink connection
Connecting to Netlink...
Sending address dump request
Waiting for response
Received 108 bytes
Adding IP 127.0.0.1 to list (family 2)
Adding IP 192.168.1.61 to list (family 2)
Closing Netlink connection
ipv4_listen: Setting up ipv4 listen socket
ipv4_listen: Success; fd = 3
Setting up ipv4 multicast send (225.0.0.12:1229)
Joining IP Multicast group (pass 1)
Joining IP Multicast group (pass 2)
Setting TTL to 2 for fd4
ipv4_send_sk: success, fd = 4
Opening /dev/urandom
Sending to 225.0.0.12 via 127.0.0.1
Setting up ipv4 multicast send (225.0.0.12:1229)
Joining IP Multicast group (pass 1)
Joining IP Multicast group (pass 2)
Setting TTL to 2 for fd4
ipv4_send_sk: success, fd = 4
Opening /dev/urandom
Sending to 225.0.0.12 via 192.168.1.61
Waiting for connection from XVM host daemon.
Issuing TCP challenge
Responding to TCP challenge
TCP Exchange + Authentication done... 
Waiting for return value from XVM host

If you are running more than one guest cluster you probably want to restrict the possibility for a virtual machine to fence systems that belong to a different cluster.
To accomplish that you should add to the /etc/fence_virt.conf file the groups directive:

groups { 
    group {
        ip = "192.168.1.55"; # Host IP
        ip = "192.168.1.60"; # Cluster VIP
        ip = "192.168.1.61"; # cluster_node1 IP
        ip = "192.168.1.62"; # cluster_node2 IP
        uuid = "cluster_node1";
        uuid = "cluster_node2";
    }

}

where ip are the ip addresses of the virtual machines and uuid is the domain name, not the domain uuid.
Note that in my configuration when I add the groups directive the command fence_xvm -o list doesn't work anymore.

Guests configuration:

Once the host configuration is completed fencing must be set up on the guest cluster and this can be done from the web interface.
Select your cluster, go to the tab Fence Devices, click to Add and select Fence virt (Multicast Mode) as instance type, choose a name (in my case kvm) and click to Submit.
Now go back to the cluster main page and click on the first node, look for the Fence Devices section and click on Add Fence Method. Choose the method name (in my case kvm) and then click to Add Fence Instance and choose the fence instance previously created.
In the Domain form enter the libvirt domain name of the guest (it can be obtained using the virsh list command on the host).

After repeating these steps for all the nodes of the cluster the configuration will look like:

<?xml version="1.0"?>
<cluster config_version="6" name="cluster">
        <clusternodes>
                <clusternode name="cluster_node1" nodeid="1">
                        <fence>
                                <method name="kvm">
                                        <device domain="cluster_node1" name="kvm"/>
                                </method>
                        </fence>
                </clusternode>
                <clusternode name="cluster_node2" nodeid="2">
                        <fence>
                                <method name="kvm">
                                        <device domain="cluster_node2" name="kvm"/>
                                </method>
                        </fence>
                </clusternode>
        </clusternodes>
        <cman expected_votes="1" two_node="1"/>
        <fencedevices>
                <fencedevice agent="fence_xvm" name="kvm"/>
        </fencedevices>
</cluster>

The fence_xvm agent supports the following options:

# ccs -h cluster_node1 --lsfenceopts fence_xvm
fence_xvm - Fence agent for virtual machines
  Required Options:
  Optional Options:
    option: No description available
    debug: Specify (stdin) or increment (command line) debug level
    ip_family: IP Family ([auto], ipv4, ipv6)
    multicast_address: Multicast address (default=225.0.0.12 / ff05::3:1)
    ipport: Multicast or VMChannel IP port (default=1229)
    retrans: Multicast retransmit time (in 1/10sec; default=20)
    auth: Authentication (none, sha1, [sha256], sha512)
    hash: Packet hash strength (none, sha1, [sha256], sha512)
    key_file: Shared key file (default=/etc/cluster/fence_xvm.key)
    port: Virtual Machine (domain name) to fence
    use_uuid: Treat [domain] as UUID instead of domain name. This is provided for compatibility with older fence_xvmd installations.
    action: Fencing action (null, off, on, [reboot], status, list, monitor, metadata)
    timeout: Fencing timeout (in seconds; default=30)
    domain: Virtual Machine (domain name) to fence (deprecated; use port)

They can be added or removed using the ccs command or manually editing the cluster.conf file:

<fencedevice agent="fence_xvm" name="kvm" key_file="/etc/cluster/fence_xvm.key" multicast_address="225.0.0.12" ipport="1229" auth="sha256" hash="sha256"/>

Once edited, validate and propagate the config to the other nodes:

# ccs_config_validate
Configuration validates
# ccs_sync -i -f /etc/cluster/cluster.conf 
You have not authenticated to the ricci daemon on cluster_node1
Password: 
You have not authenticated to the ricci daemon on cluster_node2
Password:

References: Red Hat Enterprise Linux 6 Cluster Administration, RHKB 5936