Red Hat Enterprise Linux 6 High Availability Add-On User Manual

This document includes a new section, Section 10.4, “ Cluster Daemon crashes”

This document provides information on setting debug options in Section 6.14.4, “ Logging”

Section 8.7, “ Configuring Debug Options”

, and Section 10.13, “ Debug Logging for Distributed

Lock Manager (DLM) Needs to be Enabled” .

As of Red Hat Enterprise Linux 6.3, the root user or a user who has been granted lu ci administrator permissions can also use the lu ci interface to add users to the system, as

described in Section 4.3, “ Controlling Access to luci”

As of the Red Hat Enterprise Linux 6.3 release, the ccs command validates the configuration according to the cluster schema at /usr/share/cl uster/cl uster. rng on the node that you specify with the -h option. Previously the ccs command always used the cluster schema that was packaged with the ccs command itself, /usr/share/ccs/cl uster. rng on the local system.

For information on configuration validation, refer to Section 6.1.6, “ Configuration Validation”

The tables describing the fence device parameters in Appendix A, Fence Device Parameters

and the tables describing the HA resource parameters in

Appendix B, HA Resource Parameters now

include the names of those parameters as they appear in the cl uster. co nf file.

In addition, small corrections and clarifications have been made throughout the document.

⁠Chapt er 1 . Red Hat High Availabilit y Add- O n Configurat ion and Management O verview

1.1.4 . New and Changed Feat ures for Red Hat Ent erprise Linux 6.4

Red Hat Enterprise Linux 6.4 includes the following documentation and feature updates and changes.

The Red Hat Enterprise Linux 6.4 release provides support for the Eaton Network Power Controller

(SNMP Interface) fence agent, the HP BladeSystem fence agent, and the IBM iPDU fence agent.

For information on fence device parameters, refer to Appendix A, Fence Device Parameters .

Section 4.3, “ Controlling Access to luci” .

now provides a description of the NFS Server resource agent.

As of Red Hat Enterprise Linux 6.4, the root user or a user who has been granted lu ci administrator permissions can also use the lu ci interface to delete users from the system. This is documented in

This document includes a new section, Section 6.1.5, “ Commands that Overwrite Previous

provides a description of the new nfsrestart parameter for the Filesystem and GFS2 HA resources.

Settings” .

Section 3.3, “ Enabling IP Ports” now includes information on filtering the i ptabl es firewall for

i g mp.

The IPMI LAN fence agent now supports a parameter to configure the privilege level on the IPMI device, as documented in

Appendix A, Fence Device Parameters .

In addition to Ethernet bonding mode 1, bonding modes 0 and 2 are now supported for inter-node communication in a cluster. Troubleshooting advice in this document that suggests you ensure that you are using only supported bonding modes now notes this.

VLAN-tagged network devices are now supported for cluster heartbeat communication.

Troubleshooting advice indicating that this is not supported has been removed from this document.

The Red Hat High Availability Add-On now supports the configuration of redundant ring protocol.

For general information on using this feature and configuring the cl uster. co nf configuration file, refer to

Section 8.6, “ Configuring Redundant Ring Protocol” . For information on configuring

redundant ring protocol with lu ci, refer to

Section 4.5.4, “ Configuring Redundant Ring Protocol” .

For information on configuring redundant ring protocol with the ccs command, refer to

Section 6.14.5, “ Configuring Redundant Ring Protocol” .

In addition, small corrections and clarifications have been made throughout the document.

1.1.5. New and Changed Feat ures for Red Hat Ent erprise Linux 6.5

Red Hat Enterprise Linux 6.5 includes the following documentation and feature updates and changes.

This document includes a new section, Section 8.8, “ Configuring nfsexport and nfsserver

Resources” .

The tables of fence device parameters in Appendix A, Fence Device Parameters have been updated

to reflect small updates to the lu ci interface.

In addition, many small corrections and clarifications have been made throughout the document.

Clust er Administ rat ion

1.1.6. New and Changed Feat ures for Red Hat Ent erprise Linux 6.6

Red Hat Enterprise Linux 6.6 includes the following documentation and feature updates and changes.

The tables of fence device parameters in Appendix A, Fence Device Parameters have been updated

to reflect small updates to the lu ci interface.

The tables of resource agent parameters in

Appendix B, HA Resource Parameters have been

updated to reflect small updates to the lu ci interface.

Table B.3, “ Bind Mount (bi nd -mo unt Resource) (Red Hat Enterprise Linux 6.6 and later)”

documents the parameters for the Bind Mount resource agent.

As of Red Hat Enterprise Linux 6.6 release, you can use the --no enabl e option of the ccs --

startal l command to prevent cluster services from being enabled, as documented in

Section 7.2, “ Starting and Stopping a Cluster”

Table A.12, “ Fence kdump”

documents the parameters for the kdump fence agent.

As of the Red Hat Enterprise Linux 6.6 release, you can sort the columns in a resource list on the

lu ci display by clicking on the header for the sort category, as described in Section 4.9,

“ Configuring Global Cluster Resources” .

In addition, many small corrections and clarifications have been made throughout the document.

1.1.7. New and Changed Feat ures for Red Hat Ent erprise Linux 6.7

Red Hat Enterprise Linux 6.7 includes the following documentation and feature updates and changes.

This document now includes a new chapter, Chapter 2, Getting Started: Overview

, which provides a summary procedure for setting up a basic Red Hat High Availability cluster.

Appendix A, Fence Device Parameters now includes a table listing the parameters for the Emerson

Network Power Switch (SNMP interface).

Appendix A, Fence Device Parameters now includes a table listing the parameters for the

fence_xvm fence agent, titled as "Fence virt (Multicast Mode"). The table listing the parameters for the fence_vi rt fence agent is now titled "Fence virt ((Serial/VMChannel Mode)". Both tables have been updated to reflect the lu ci display.

Appendix A, Fence Device Parameters now includes a table listing the parameters for the

The troubleshooting procedure described in Section 10.10, “ Quorum Disk Does Not Appear as

Cluster Member” has been updated.

In addition, many small corrections and clarifications have been made throughout the document.

1.2. Configurat ion Basics

To set up a cluster, you must connect the nodes to certain cluster hardware and configure the nodes into the cluster environment. Configuring and managing the Red Hat High Availability Add-On consists of the following basic steps:

⁠Chapt er 1 . Red Hat High Availabilit y Add- O n Configurat ion and Management O verview

1. Setting up hardware. Refer to Section 1.3, “ Setting Up Hardware”

2. Installing Red Hat High Availability Add-On software. Refer to Section 1.4, “ Installing Red Hat

High Availability Add-On software” .

3. Configuring Red Hat High Availability Add-On Software. Refer to Section 1.5, “ Configuring

Red Hat High Availability Add-On Software” .

1.3. Set t ing Up Hardware

Setting up hardware consists of connecting cluster nodes to other hardware required to run the Red

Hat High Availability Add-On. The amount and type of hardware varies according to the purpose and availability requirements of the cluster. Typically, an enterprise-level cluster requires the following type of hardware (refer to

Figure 1.1, “ Red Hat High Availability Add-On Hardware Overview” ). For

considerations about hardware and other cluster configuration concerns, refer to Chapter 3, Before

Configuring the Red Hat High Availability Add-On

or check with an authorized Red Hat representative.

Cluster nodes — Computers that are capable of running Red Hat Enterprise Linux 6 software, with at least 1GB of RAM.

Network switches for public network — This is required for client access to the cluster.

Network switches for private network — This is required for communication among the cluster nodes and other cluster hardware such as network power switches and Fibre Channel switches.

Fencing device — A fencing device is required. A network power switch is recommended to perform fencing in an enterprise-level cluster. For information about supported fencing devices, see

Appendix A, Fence Device Parameters .

Storage — Some type of storage is required for a cluster. Figure 1.1, “ Red Hat High Availability

Add-On Hardware Overview” shows shared storage, but shared storage may not be required for

your specific use.

Clust er Administ rat ion

Fig u re 1.1. Red Hat Hig h Availab ilit y Ad d - O n Hard ware O verview

1.4 . Inst alling Red Hat High Availabilit y Add-On soft ware

To install Red Hat High Availability Add-On software, you must have entitlements for the software. If you are using the lu ci configuration GUI, you can let it install the cluster software. If you are using other tools to configure the cluster, secure and install the software as you would with Red Hat

Enterprise Linux software.

You can use the following yum i nstal l command to install the Red Hat High Availability Add-On software packages:

# yum i nstal l rg manag er l vm2-cl uster g fs2-uti l s

Note that installing only the rg manag er will pull in all necessary dependencies to create an HA cluster from the HighAvailability channel. The l vm2-cl uster and g fs2-uti l s packages are part of ResilientStorage channel and may not be needed by your site.

⁠Chapt er 1 . Red Hat High Availabilit y Add- O n Configurat ion and Management O verview

Warning

After you install the Red Hat High Availability Add-On packages, you should ensure that your software update preferences are set so that nothing is installed automatically. Installation on a running cluster can cause unexpected behaviors.

Upgrading Red Hat High Availabilit y Add-On Soft ware

It is possible to upgrade the cluster software on a given major release of Red Hat Enterprise Linux without taking the cluster out of production. Doing so requires disabling the cluster software on one host at a time, upgrading the software, and restarting the cluster software on that host.

1. Shut down all cluster services on a single cluster node. For instructions on stopping cluster software on a node, refer to

Section 9.1.2, “ Stopping Cluster Software” . It may be desirable to

manually relocate cluster-managed services and virtual machines off of the host prior to stopping rg manag er.

2. Execute the yum upd ate command to update installed packages.

3. Reboot the cluster node or restart the cluster services manually. For instructions on starting

cluster software on a node, refer to Section 9.1.1, “ Starting Cluster Software” .

1.5. Configuring Red Hat High Availabilit y Add-On Soft ware

Configuring Red Hat High Availability Add-On software consists of using configuration tools to specify the relationship among the cluster components. The following cluster configuration tools are available with Red Hat High Availability Add-On:

Co n g a — This is a comprehensive user interface for installing, configuring, and managing Red

Hat High Availability Add-On. Refer to Chapter 4, Configuring Red Hat High Availability Add-On With

Conga

and Chapter 5, Managing Red Hat High Availability Add-On With Conga for information about

configuring and managing High Availability Add-On with Co n g a.

The ccs command — This command configures and manages Red Hat High Availability Add-On.

Refer to Chapter 6, Configuring Red Hat High Availability Add-On With the ccs Command

and

Chapter 7, Managing Red Hat High Availability Add-On With ccs

for information about configuring and managing High Availability Add-On with the ccs command.

Command-line tools — This is a set of command-line tools for configuring and managing Red Hat

High Availability Add-On. Refer to Chapter 8, Configuring Red Hat High Availability Manually and

Chapter 9, Managing Red Hat High Availability Add-On With Command Line Tools for information about

configuring and managing a cluster with command-line tools. Refer to Appendix E, Command Line

Tools Summary

for a summary of preferred command-line tools.

Note

system-co nfi g -cl uster is not available in Red Hat Enterprise Linux 6.

Clust er Administ rat ion

Chapter 2. Getting Started: Overview

This chapter provides a summary procedure for setting up a basic Red Hat High Availability cluster consisting of two nodes running Red Hat Enterprise Linux release 6. This procedure uses the lu ci user interface to create the cluster. While this procedure creates a basic cluster, it does not yield a complete supported cluster configuration. Further details on planning and deploying a cluster are provided in the remainder of this document.

2.1. Inst allat ion and Syst em Set up

Before creating a Red Hat High Availability cluster, perform the following setup and installation steps.

1. Ensure that your Red Hat account includes the following support entitlements:

RHEL: Server

Red Hat Applications: High availability

Red Hat Applications: Resilient Storage, if using the Clustered Logical Volume Manager

(CLVM) and GFS2 file systems.

2. Register the cluster systems for software updates, using either Red Hat Subscriptions

Manager (RHSM) or RHN Classic.

3. On each node in the cluster, configure the iptables firewall. The iptables firewall can be disabled, or it can be configured to allow cluster traffic to pass through.

To disable the iptables system firewall, execute the following commands.

# servi ce i ptabl es sto p

# chkco nfi g i ptabl es o ff

For information on configuring the iptables firewall to allow cluster traffic to pass trhough, see

Section 3.3, “ Enabling IP Ports” .

4. On each node in the cluster, configure SELinux. SELinux is supported on Red Hat Enterprise

Linux 6 cluster nodes in Enforcing or Permissive mode with a targeted policy, or it can be disabled. To check the current SELinux state, run the g etenfo rce:

# g etenfo rce

Permissive

For information on enabling and disabling SELinux, see the Security-Enhanced Linux user guide.

5. Install the cluster packages and package groups.

a. On each node in the cluster, install the Hi g h Avai l abi l i ty and R esi l i ant

Sto rag e package groups.

# yum g ro upi nstal l ' Hi g h Avai l abi l i ty' ' R esi l i ent Sto rag e' b. On the the node that will be hosting the web management interface, install the lu ci package.

⁠Chapt er 2 . G et t ing St art ed: O verview

# yum i nstal l l uci

2.2. St art ing Clust er Services

Start the cluster services on the nodes in the cluster using the following procedure.

1. On both nodes in the cluster, start the ri cci service and set a password for user ri cci .

# servi ce ri cci start

Starting ricci: [ OK ]

# passwd ri cci

New password:

Retype new password:

2. On the node that will be hosting the web management interface, start the lu ci service. This will provide the link from which to access lu ci on this node.

# servi ce l uci start

Starting luci: generating https SSL certificates... done

[ OK ]

Please, point your web browser to https://example-01:8084 to access luci

2.3. Creat ing t he Clust er

Use the following procedure to create a cluster.

1. To access the High Availability management web interface, point your browser to the link provided by the lu ci service and log in using the root account on the node hosting lu ci.

Logging in to lu ci displays the lu ci Ho meb ase page. page.

2. To create a cluster, click on Man ag e Clu st ers from the menu on the left navigation pane of the Ho meb ase page. This displays the clu st ers page.

Fig u re 2.1. lu ci Ho meb ase p ag e

3. From the clu st ers page, click the C reate button. This displays the Creat e New Clu st er screen.

Clust er Administ rat ion

Fig u re 2.2. Clu st ers men u

4. On the Creat e New Clu st er screen, enter the parameters for the cluster you are creating.

The Passwo rd field will be the ri cci password you defined for the indicated node.

Fig u re 2.3. Creat e New Clu st er screen

5. After you have completed entering the parameters for the cluster, click the C reate C l uster button. A progress bar is displayed with the cluster is formed. Once cluster creation has completed, lu ci displays the cluster general properties.

6. Verify the cluster status by running the cl ustat command on either node of the cluster.

# cl ustat

Cluster Status for exampleHA @ Thu Sep 29 12:17:39 2011

Member Status: Quorate

Member Name ID

Status

------ ---- --

-- -----node1.example.com 1

Online, Local node2.example.com 2

Online

⁠Chapt er 2 . G et t ing St art ed: O verview

If you cannot create the cluster, double check the firewall settings, as described in

Section 3.3.3, “ Configuring the iptables Firewall to Allow Cluster Components”

If you can create the cluster (there is an /etc/cl uster/cl uster. co nf on each node) but the cluster will not form, you may have multicast issues. To test this, change the transport

mode from UDP multicast to UDP unicast, as described in Section 3.12, “ UDP Unicast

Traffic” . Note, however, that in unicast mode there is a traffic increase compared to multicast

mode, which adds to the processing load of the node.

2.4 . Configuring Fencing

You must configure a fencing device for each node in the cluster. When configuring a fencing device, you should ensure that your fencing device does not share power with the node that it controls. For

information on fence device configuration, see Section 4.6, “ Configuring Fence Devices” . For

information on configuring fencing for cluster nodes, see Section 4.7, “ Configuring Fencing for

Cluster Members” .

After configuring a fence device for a node, it is important to test the fence device, to ensure that the cluster will cut off access to a resource when the cluster loses communication with that node. How you break communication with the node will depend on your system setup and the type of fencing you have configured. You may need to physically disconnect network cables, or force a kernel panic on the node. You can then check whether the node has been fenced as expected.

When creating a two-node cluster, you may need to configure a tie-breaking mechanism for the cluster to avoid split brains and fence races for the cluster, which can occur when the cluster interconnect experiences issues that prevent the nodes from communicating. For information on avoiding fence races, refer to the Red Hat Knowledgebase solution "What are my options for avoiding fence races in RHEL 5, 6, and 7 High Availability clusters with an even number of nodes?" on Red Hat Customer Portal at https://access.redhat.com/solutions/91653 . For information on avoiding fencing loops, refer to the Red Hat Knowledgebase solution "How can I avoid fencing loops with 2 node clusters and Red Hat High Availability clusters?" on Red Hat Customer Portal at https://access.redhat.com/solutions/272913 .

2.5. Configuring a High Availabilit y Applicat ion

After creating a cluster and configuring fencing for the nodes in the cluster, you define and configure the components of the high availability service you will run on the cluster. To complete your cluster setup, perform the following steps.

1. Configure shared storage and file systems required by your application. For information on high availability logical volumes, see

Appendix F, High Availability LVM (HA-LVM)

. For information on the GFS2 clustered file system, see the Global File System 2 manual.

2. Optionally, you can customize your cluster's behavior by configuring a failover domain. A failover domain determines which cluster nodes an application will run on in what circumstances, determined by a set of failover domain configuration options. For information on failover domain options and how they determine a cluster's behavior, see the High

Availability Add-On Overview. For information on configuring failover domains, see Section 4.8,

“ Configuring a Failover Domain” .

3. Configure cluster resources for your system. Cluster resources are the individual components of the applications running on a cluster node. For information on configuring cluster

resources, see Section 4.9, “ Configuring Global Cluster Resources” .

4. Configure the cluster services for your cluster. A cluster service is the collection of cluster resources required by an application running on a cluster node that can fail over to another

Clust er Administ rat ion

node in a high availability cluster. You can configure the startup and recovery policies for a cluster service, and you can configure resource trees for the resources that constitute the service, which determine startup and shutdown order for the resources as well as the relationships between the resources. For information on service policies, resource trees, service operations, and resource actions, see the High Availability Add-On Overview. For

information on configuring cluster services, see Section 4.10, “ Adding a Cluster Service to the

Cluster” .

2.6. T est ing t he Configurat ion

The specifics of a procedure for testing your cluster configuration will depend on the high availability application you are running in the cluster, but in general you can check whether the application is running and whether it fails over as follows.

1. Verify that the service you created is running with the cl ustat command, which you can run on either cluster node. In this example, the service exampl e_apache is running on

no d e1. exampl e. co m.

# cl ustat

Cluster Status for exampleHA @ Thu Sep 29 12:17:39 2011

Member Status: Quorate

Member Name ID Status

------ ---- ---- -----node1.example.com 1 Online, Local node2.example.com 2 Online

Service Name Owner (Last) State

------ ---- ----- ------ ----service:example_apache node-01.example.com started

2. Check whether the service is operational. How you do this depends on the application. For example, if you are running an Apache web server, point a browser to the URL you defined for the server to see if the display is correct.

3. Shut down the cluster software on the node on which the the service is running, which you can determine from the cl ustat display.

a. Click on the cluster name beneath Man ag e Clu st ers from the menu on the left side of the lu ci Ho meb ase page. This displays the nodes the constitute the cluster.

b. Select the node you want to leave the cluster by clicking the checkbox for that node.

c. Select the Leave Clu st er function from the menu at the top of the page. This causes a message to appear at the top of the page indicating that the node is being stopped.

d. Refresh the page to see the updated status of the node.

4. Run the cl ustat command again to see if the service is now running on the other cluster node. If it is, check again to see if the service is still operational. For example, if you are running an apache web server, check whether you can still display the website.

5. Make sure you have the node you disabled rejoin the cluster by returning to the cluster node display, selecting the node, and selecting Jo in Clu st er.

⁠Chapt er 3. Before Configuring t he Red Hat High Availabilit y Add- O n

Chapter 3. Before Configuring the Red Hat High Availability Add-

On

This chapter describes tasks to perform and considerations to make before installing and configuring the Red Hat High Availability Add-On, and consists of the following sections.

Important

Make sure that your deployment of Red Hat High Availability Add-On meets your needs and can be supported. Consult with an authorized Red Hat representative to verify your configuration prior to deployment. In addition, allow time for a configuration burn-in period to test failure modes.

Section 3.1, “ General Configuration Considerations”

Section 3.2, “ Compatible Hardware”

Section 3.3, “ Enabling IP Ports”

Section 3.4, “ Configuring lu ci with /etc/sysco nfi g /l uci ”

Section 3.5, “ Configuring ACPI For Use with Integrated Fence Devices”

Section 3.6, “ Considerations for Configuring HA Services”

Section 3.7, “ Configuration Validation”

Section 3.8, “ Considerations for Net wo rkMan ag er”

Section 3.9, “ Considerations for Using Quorum Disk”

Section 3.10, “ Red Hat High Availability Add-On and SELinux”

Section 3.11, “ Multicast Addresses”

Section 3.12, “ UDP Unicast Traffic”

Section 3.13, “ Considerations for ri cci ”

Section 3.14, “ Configuring Virtual Machines in a Clustered Environment”

3.1. General Configurat ion Considerat ions

You can configure the Red Hat High Availability Add-On in a variety of ways to suit your needs. Take into account the following general considerations when you plan, configure, and implement your deployment.

Nu mb er o f clu st er n o d es su p p o rt ed

The maximum number of cluster nodes supported by the High Availability Add-On is 16.

Sin g le sit e clu st ers

Clust er Administ rat ion

Only single site clusters are fully supported at this time. Clusters spread across multiple physical locations are not formally supported. For more details and to discuss multi-site clusters, please speak to your Red Hat sales or support representative.

G FS2

Although a GFS2 file system can be implemented in a standalone system or as part of a cluster configuration, Red Hat does not support the use of GFS2 as a single-node file system. Red Hat does support a number of high-performance single-node file systems that are optimized for single node, which have generally lower overhead than a cluster file system. Red Hat recommends using those file systems in preference to GFS2 in cases where only a single node needs to mount the file system. Red Hat will continue to support singlenode GFS2 file systems for existing customers.

When you configure a GFS2 file system as a cluster file system, you must ensure that all nodes in the cluster have access to the shared file system. Asymmetric cluster configurations in which some nodes have access to the file system and others do not are not supported.This does not require that all nodes actually mount the GFS2 file system itself.

No - sin g le- p o in t - o f - f ailu re h ard ware co n f ig u rat io n

Clusters can include a dual-controller RAID array, multiple bonded network channels, multiple paths between cluster members and storage, and redundant un-interruptible power supply (UPS) systems to ensure that no single failure results in application down time or loss of data.

Alternatively, a low-cost cluster can be set up to provide less availability than a no-singlepoint-of-failure cluster. For example, you can set up a cluster with a single-controller RAID array and only a single Ethernet channel.

Certain low-cost alternatives, such as host RAID controllers, software RAID without cluster support, and multi-initiator parallel SCSI configurations are not compatible or appropriate for use as shared cluster storage.

Dat a in t eg rit y assu ran ce

To ensure data integrity, only one node can run a cluster service and access clusterservice data at a time. The use of power switches in the cluster hardware configuration enables a node to power-cycle another node before restarting that node's HA services during a failover process. This prevents two nodes from simultaneously accessing the same data and corrupting it. Fence devices (hardware or software solutions that remotely power, shutdown, and reboot cluster nodes) are used to guarantee data integrity under all failure conditions.

Et h ern et ch an n el b o n d in g

Cluster quorum and node health is determined by communication of messages among cluster nodes via Ethernet. In addition, cluster nodes use Ethernet for a variety of other critical cluster functions (for example, fencing). With Ethernet channel bonding, multiple

Ethernet interfaces are configured to behave as one, reducing the risk of a single-point-offailure in the typical switched Ethernet connection among cluster nodes and other cluster hardware.

As of Red Hat Enterprise Linux 6.4, bonding modes 0, 1, and 2 are supported.

IPv4 an d IPv6

⁠Chapt er 3. Before Configuring t he Red Hat High Availabilit y Add- O n

The High Availability Add-On supports both IPv4 and IPv6 Internet Protocols. Support of

IPv6 in the High Availability Add-On is new for Red Hat Enterprise Linux 6.

3.2. Compat ible Hardware

Before configuring Red Hat High Availability Add-On software, make sure that your cluster uses appropriate hardware (for example, supported fence devices, storage devices, and Fibre Channel switches). Refer to the Red Hat Hardware Catalog at https://hardware.redhat.com/ for the most current hardware compatibility information.

3.3. Enabling IP Port s

Before deploying the Red Hat High Availability Add-On, you must enable certain IP ports on the cluster nodes and on computers that run lu ci (the Co n g a user interface server). The following sections identify the IP ports to be enabled:

Section 3.3.1, “ Enabling IP Ports on Cluster Nodes”

Section 3.3.2, “ Enabling the IP Port for lu ci”

The following section provides the i ptabl es rules for enabling IP ports needed by the Red Hat High

Availability Add-On:

Section 3.3.3, “ Configuring the iptables Firewall to Allow Cluster Components”

3.3.1. Enabling IP Port s on Clust er Nodes

To allow the nodes in a cluster to communicate with each other, you must enable the IP ports

assigned to certain Red Hat High Availability Add-On components. Table 3.1, “ Enabled IP Ports on

Red Hat High Availability Add-On Nodes” lists the IP port numbers, their respective protocols, and the

components to which the port numbers are assigned. At each cluster node, enable IP ports for

incoming traffic according to Table 3.1, “ Enabled IP Ports on Red Hat High Availability Add-On

Nodes” . You can use system-co nfi g -fi rewal l to enable the IP ports.

T ab le 3.1. En ab led IP Po rt s o n Red Hat Hig h Availab ilit y Ad d - O n No d es

IP Po rt Nu mb er Pro t o co l

5404, 5405

11111

21064

16851

UDP

TCP

Co mp o n en t

co ro sync/cman (Cluster Manager)

ri cci (propagates updated cluster information)

d l m (Distributed Lock Manager)

mo d cl usterd

3.3.2. Enabling t he IP Port for luci

To allow client computers to communicate with a computer that runs lu ci (the Co n g a user interface server), you must enable the IP port assigned to lu ci. At each computer that runs lu ci, enable the IP

port according to Table 3.2, “ Enabled IP Port on a Computer That Runs luci” .

Note

If a cluster node is running lu ci, port 11111 should already have been enabled.

Clust er Administ rat ion

T ab le 3.2. En ab led IP Po rt o n a Co mp u t er T h at Ru n s lu ci

IP Po rt Nu mb er Pro t o co l

8084 TCP

Co mp o n en t

lu ci (Co n g a user interface server)

As of the Red Hat Enterprise Linux 6.1 release, which enabled configuration by means of the

/etc/sysco nfi g /l uci file, you can specifically configure the only IP address lu ci is being served at. You can use this capability if your server infrastructure incorporates more than one network and you want to access lu ci from the internal network only. To do this, uncomment and edit the line in the file that specifies ho st. For example, to change the ho st setting in the file to

10.10.10.10, edit the ho st line as follows: host = 10.10.10.10

For more information on the /etc/sysco nfi g /l uci file, refer to Section 3.4, “ Configuring lu ci with /etc/sysco nfi g /l uci ” .

3.3.3. Configuring t he ipt ables Firewall t o Allow Clust er Component s

Listed below are example iptable rules for enabling IP ports needed by Red Hat Enterprise Linux 6

(with High Availability Add-on). Please note that these examples use 192.168.1.0/24 as a subnet, but you will need to replace 192.168.1.0/24 with the appropriate subnet if you use these rules.

For cman (Cluster Manager), use the following filtering.

$ i ptabl es -I INP UT -m state --state NEW -m mul ti po rt -p ud p -s

19 2. 16 8. 1. 0 /24 -d 19 2. 16 8. 1. 0 /24 --d po rts 54 0 4 ,54 0 5 -j AC C EP T

$ i ptabl es -I INP UT -m ad d rtype --d st-type MULT IC AST -m state --state

NEW -m mul ti po rt -p ud p -s 19 2. 16 8. 1. 0 /24 --d po rts 54 0 4 ,54 0 5 -j AC C EP T

For d l m (Distributed Lock Manager):

$ i ptabl es -I INP UT -m state --state NEW -p tcp -s 19 2. 16 8. 1. 0 /24 -d

19 2. 16 8. 1. 0 /24 --d po rt 210 6 4 -j AC C EP T

For ri cci (part of Conga remote agent):

$ i ptabl es -I INP UT -m state --state NEW -p tcp -s 19 2. 16 8. 1. 0 /24 -d

19 2. 16 8. 1. 0 /24 --d po rt 11111 -j AC C EP T

For mo d cl usterd (part of Conga remote agent):

$ i ptabl es -I INP UT -m state --state NEW -p tcp -s 19 2. 16 8. 1. 0 /24 -d

19 2. 16 8. 1. 0 /24 --d po rt 16 851 -j AC C EP T

For l uci (Conga User Interface server):

$ i ptabl es -I INP UT -m state --state NEW -p tcp -s 19 2. 16 8. 1. 0 /24 -d

19 2. 16 8. 1. 0 /24 --d po rt 80 84 -j AC C EP T

For i g mp (Internet Group Management Protocol):

$ i ptabl es -I INP UT -p i g mp -j AC C EP T

⁠Chapt er 3. Before Configuring t he Red Hat High Availabilit y Add- O n

After executing these commands, run the following command to save the current configuration for the changes to be persistent during reboot.

$ servi ce i ptabl es save ; servi ce i ptabl es restart

3.4 . Configuring luci wit h

/etc/sysco nfi g /l uci

As of the Red Hat Enterprise Linux 6.1 release, you can configure some aspects of lu ci's behavior by means of the /etc/sysco nfi g /l uci file. The parameters you can change with this file include auxiliary settings of the running environment used by the init script as well as server configuration. In addition, you can edit this file to modify some application configuration parameters. There are instructions within the file itself describing which configuration parameters you can change by editing this file.

In order to protect the intended format, you should not change the non-configuration lines of the

/etc/sysco nfi g /l uci file when you edit the file. Additionally, you should take care to follow the required syntax for this file, particularly for the INIT SC R IP T section which does not allow for white spaces around the equal sign and requires that you use quotation marks to enclose strings containing white spaces.

The following example shows how to change the port at which lu ci is being served by editing the

/etc/sysco nfi g /l uci file.

1. Uncomment the following line in the /etc/sysco nfi g /l uci file:

#port = 4443

2. Replace 4443 with the desired port number, which must be higher than or equal to 1024 (not a privileged port). For example, you can edit that line of the file as follows to set the port at which lu ci is being served to 8084 (commenting the line out again would have the same affect, as this is the default value).

port = 8084

3. Restart the lu ci service for the changes to take effect.

As the Red Hat Enterprise Linux 6.6 release, you can implement a fine-grained control over the ciphers behind the secured connection between lu ci and the web browser with the

ssl _ci pher_l i st configuration parameter in /etc/sysco nfi g /l uci . This parameter can be used to impose restrictions as expressed with OpenSSL cipher notation.

Important

When you modify a configuration parameter in the /etc/sysco nfi g /l uci file to redefine a default value, you should take care to use the new value in place of the documented default value. For example, when you modify the port at which lu ci is being served, make sure that you specify the modified value when you enable an IP port for lu ci, as described in

Section 3.3.2, “ Enabling the IP Port for lu ci”

Modified port and host parameters will automatically be reflected in the URL displayed when the lu ci service starts, as described in

Section 4.2, “ Starting lu ci” . You should use this URL to

access lu ci.

Clust er Administ rat ion

For more complete information on the parameters you can configure with the

/etc/sysco nfi g /l uci file, refer to the documentation within the file itself.

3.5. Configuring ACPI For Use wit h Int egrat ed Fence Devices

If your cluster uses integrated fence devices, you must configure ACPI (Advanced Configuration and

Power Interface) to ensure immediate and complete fencing.

Note

For the most current information about integrated fence devices supported by Red Hat High

Availability Add-On, refer to http://www.redhat.com/cluster_suite/hardware/ .

If a cluster node is configured to be fenced by an integrated fence device, disable ACPI Soft-Off for that node. Disabling ACPI Soft-Off allows an integrated fence device to turn off a node immediately and completely rather than attempting a clean shutdown (for example, shutd o wn -h no w).

Otherwise, if ACPI Soft-Off is enabled, an integrated fence device can take four or more seconds to turn off a node (refer to note that follows). In addition, if ACPI Soft-Off is enabled and a node panics or freezes during shutdown, an integrated fence device may not be able to turn off the node. Under those circumstances, fencing is delayed or unsuccessful. Consequently, when a node is fenced with an integrated fence device and ACPI Soft-Off is enabled, a cluster recovers slowly or requires administrative intervention to recover.

Note

The amount of time required to fence a node depends on the integrated fence device used.

Some integrated fence devices perform the equivalent of pressing and holding the power button; therefore, the fence device turns off the node in four to five seconds. Other integrated fence devices perform the equivalent of pressing the power button momentarily, relying on the operating system to turn off the node; therefore, the fence device turns off the node in a time span much longer than four to five seconds.

To disable ACPI Soft-Off, use chkco nfi g management and verify that the node turns off immediately when fenced. The preferred way to disable ACPI Soft-Off is with chkco nfi g management: however, if that method is not satisfactory for your cluster, you can disable ACPI Soft-Off with one of the following alternate methods:

Changing the BIOS setting to "instant-off" or an equivalent setting that turns off the node without delay

Note

Disabling ACPI Soft-Off with the BIOS may not be possible with some computers.

Appending acpi = o ff to the kernel boot command line of the /bo o t/g rub/g rub. co nf file

⁠Chapt er 3. Before Configuring t he Red Hat High Availabilit y Add- O n

Important

This method completely disables ACPI; some computers do not boot correctly if ACPI is completely disabled. Use this method only if the other methods are not effective for your cluster.

The following sections provide procedures for the preferred method and alternate methods of disabling ACPI Soft-Off:

Section 3.5.1, “ Disabling ACPI Soft-Off with chkco nfi g Management”

— Preferred method

Section 3.5.2, “ Disabling ACPI Soft-Off with the BIOS”

— First alternate method

Section 3.5.3, “ Disabling ACPI Completely in the g rub. co nf File”

— Second alternate method

3.5.1. Disabling ACPI Soft -Off wit h

chkco nfi g

Management

You can use chkco nfi g management to disable ACPI Soft-Off either by removing the ACPI daemon

(acpi d ) from chkco nfi g management or by turning off acpi d .

Note

This is the preferred method of disabling ACPI Soft-Off.

Disable ACPI Soft-Off with chkco nfi g management at each cluster node as follows:

1. Run either of the following commands:

chkco nfi g --d el acpi d — This command removes acpi d from chkco nfi g management.

— OR —

chkco nfi g --l evel 34 5 acpi d o ff — This command turns off acpi d .

2. Reboot the node.

3. When the cluster is configured and running, verify that the node turns off immediately when fenced.

Note

You can fence the node with the fence_no d e command or Co n g a.

3.5.2. Disabling ACPI Soft -Off wit h t he BIOS

The preferred method of disabling ACPI Soft-Off is with chkco nfi g management ( Section 3.5.1,

“ Disabling ACPI Soft-Off with chkco nfi g Management” ). However, if the preferred method is not

effective for your cluster, follow the procedure in this section.

Clust er Administ rat ion

Note

Disabling ACPI Soft-Off with the BIOS may not be possible with some computers.

You can disable ACPI Soft-Off by configuring the BIOS of each cluster node as follows:

1. Reboot the node and start the BIO S C MO S Setup Uti l i ty program.

2. Navigate to the Po wer menu (or equivalent power management menu).

3. At the Po wer menu, set the So f t - O f f b y PWR- BT T N function (or equivalent) to In st an t -

O f f (or the equivalent setting that turns off the node via the power button without delay).

Example 3.1, “ BIO S C MO S Setup Uti l i ty: Soft-Off by PWR-BTTN set to Instant-Off”

shows a Po wer menu with ACPI Fu n ct io n set to En ab led and So f t - O f f b y PWR- BT T N set to In st an t - O f f .

Note

The equivalents to ACPI Fu n ct io n , So f t - O f f b y PWR- BT T N, and In st an t - O f f may vary among computers. However, the objective of this procedure is to configure the

BIOS so that the computer is turned off via the power button without delay.

4. Exit the BIO S C MO S Setup Uti l i ty program, saving the BIOS configuration.

5. When the cluster is configured and running, verify that the node turns off immediately when fenced.

Note

You can fence the node with the fence_no d e command or Co n g a.

Examp le 3.1. BIO S C MO S Setup Uti l i ty: So f t - O f f b y PWR- BT T N set t o In st an t - O f f

+---------------------------------------------|-------------------+

| ACPI Function [Enabled] | Item Help |

| ACPI Suspend Type [S1(POS)] |-------------------|

| x Run VGABIOS if S3 Resume Auto | Menu Level * |

| Suspend Mode [Disabled] | |

| HDD Power Down [Disabled] | |

| Soft-Off by PWR-BTTN [Instant-Off | |

| CPU THRM-Throttling [50.0%] | |

| Wake-Up by PCI card [Enabled] | |

| Power On by Ring [Enabled] | |

| Wake Up On LAN [Enabled] | |

| x USB KB Wake-Up From S3 Disabled | |

| Resume by Alarm [Disabled] | |

| x Date(of Month) Alarm 0 | |

| x Time(hh:mm:ss) Alarm 0 : 0 : | |

| POWER ON Function [BUTTON ONLY | |

⁠Chapt er 3. Before Configuring t he Red Hat High Availabilit y Add- O n

| x KB Power ON Password Enter | |

| x Hot Key Power ON Ctrl-F1 | |

| | |

+---------------------------------------------|-------------------+

This example shows ACPI Fu n ct io n set to En ab led , and So f t - O f f b y PWR- BT T N set to

In st an t - O f f .

3.5.3. Disabling ACPI Complet ely in t he

g rub. co nf

File

The preferred method of disabling ACPI Soft-Off is with chkco nfi g management ( Section 3.5.1,

“ Disabling ACPI Soft-Off with chkco nfi g Management” ). If the preferred method is not effective for your cluster, you can disable ACPI Soft-Off with the BIOS power management ( Section 3.5.2,

“ Disabling ACPI Soft-Off with the BIOS” ). If neither of those methods is effective for your cluster, you

can disable ACPI completely by appending acpi = o ff to the kernel boot command line in the

g rub. co nf file.

Important

This method completely disables ACPI; some computers do not boot correctly if ACPI is completely disabled. Use this method only if the other methods are not effective for your cluster.

You can disable ACPI completely by editing the g rub. co nf file of each cluster node as follows:

1. Open /bo o t/g rub/g rub. co nf with a text editor.

2. Append acpi = o ff to the kernel boot command line in /bo o t/g rub/g rub. co nf (refer to

Example 3.2, “ Kernel Boot Command Line with acpi = o ff Appended to It” ).

3. Reboot the node.

4. When the cluster is configured and running, verify that the node turns off immediately when fenced.

Note

You can fence the node with the fence_no d e command or Co n g a.

Examp le 3.2. Kern el Bo o t Co mman d Lin e wit h acpi = o ff Ap p en d ed t o It

# grub.conf generated by anaconda

# Note that you do not have to rerun grub after making changes to this file

# NOTICE: You have a /boot partition. This means that

# all kernel and initrd paths are relative to /boot/, eg.

# root (hd0,0)

# kernel /vmlinuz-version ro root=/dev/mapper/vg_doc01lv_root

Clust er Administ rat ion

# initrd /initrd-[generic-]version.img

#boot=/dev/hda default=0 timeout=5 serial --unit=0 --speed=115200 terminal --timeout=5 serial console title Red Hat Enterprise Linux Server (2.6.32-193.el6.x86_64)

root (hd0,0)

kernel /vmlinuz-2.6.32-193.el6.x86_64 ro root=/dev/mapper/vg_doc01-lv_root console=ttyS0,115200n8 acpi=off

initrd /initramrs-2.6.32-131.0.15.el6.x86_64.img

In this example, acpi = o ff has been appended to the kernel boot command line — the line starting with "kernel /vmlinuz-2.6.32-193.el6.x86_64.img".

3.6. Considerat ions for Configuring HA Services

You can create a cluster to suit your needs for high availability by configuring HA (high-availability) services. The key component for HA service management in the Red Hat High Availability Add-On,

rg manag er, implements cold failover for off-the-shelf applications. In the Red Hat High Availability

Add-On, an application is configured with other cluster resources to form an HA service that can fail over from one cluster node to another with no apparent interruption to cluster clients. HA-service failover can occur if a cluster node fails or if a cluster system administrator moves the service from one cluster node to another (for example, for a planned outage of a cluster node).

To create an HA service, you must configure it in the cluster configuration file. An HA service comprises cluster resources. Cluster resources are building blocks that you create and manage in the cluster configuration file — for example, an IP address, an application initialization script, or a Red

Hat GFS2 shared partition.

To ensure data integrity, only one node can run a cluster service and access cluster-service data at a time. You can specify failover priority in a failover domain. Specifying failover priority consists of assigning a priority level to each node in a failover domain. The priority level determines the failover order — determining which node that an HA service should fail over to. If you do not specify failover priority, an HA service can fail over to any node in its failover domain. Also, you can specify if an HA service is restricted to run only on nodes of its associated failover domain. When associated with an unrestricted failover domain, an HA service can start on any cluster node in the event no member of the failover domain is available.

Figure 3.1, “ Web Server Cluster Service Example” shows an example of an HA service that is a web

server named "content-webserver". It is running in cluster node B and is in a failover domain that consists of nodes A, B, and D. In addition, the failover domain is configured with a failover priority to fail over to node D before node A and to restrict failover to nodes only in that failover domain. The HA service comprises these cluster resources:

IP address resource — IP address 10.10.10.201.

An application resource named "httpd-content" — a web server application init script

/etc/i ni t. d /httpd (specifying httpd ).

A file system resource — Red Hat GFS2 named "gfs2-content-webserver".

⁠Chapt er 3. Before Configuring t he Red Hat High Availabilit y Add- O n

Fig u re 3.1. Web Server Clu st er Service Examp le

Clients access the HA service through the IP address 10.10.10.201, enabling interaction with the web server application, httpd-content. The httpd-content application uses the gfs2-content-webserver file system. If node B were to fail, the content-webserver HA service would fail over to node D. If node D were not available or also failed, the service would fail over to node A. Failover would occur with minimal service interruption to the cluster clients. For example, in an HTTP service, certain state information may be lost (like session data). The HA service would be accessible from another cluster node via the same IP address as it was before failover.

Note

For more information about HA services and failover domains, refer to the High Availability Add-

On Overview. For information about configuring failover domains, refer to Chapter 4,

Configuring Red Hat High Availability Add-On With Conga

(using Co n g a) or

Chapter 8, Configuring

Red Hat High Availability Manually

(using command line utilities).

An HA service is a group of cluster resources configured into a coherent entity that provides

Clust er Administ rat ion

specialized services to clients. An HA service is represented as a resource tree in the cluster configuration file, /etc/cl uster/cl uster. co nf (in each cluster node). In the cluster configuration file, each resource tree is an XML representation that specifies each resource, its attributes, and its relationship among other resources in the resource tree (parent, child, and sibling relationships).

Note

Because an HA service consists of resources organized into a hierarchical tree, a service is sometimes referred to as a resource tree or resource group. Both phrases are synonymous with

HA service.

At the root of each resource tree is a special type of resource — a service resource. Other types of resources comprise the rest of a service, determining its characteristics. Configuring an HA service consists of creating a service resource, creating subordinate cluster resources, and organizing them into a coherent entity that conforms to hierarchical restrictions of the service.

There are two major considerations to take into account when configuring an HA service:

The types of resources needed to create a service

Parent, child, and sibling relationships among resources

The types of resources and the hierarchy of resources depend on the type of service you are configuring.

The types of cluster resources are listed in Appendix B, HA Resource Parameters

. Information about

parent, child, and sibling relationships among resources is described in Appendix C, HA Resource

Behavior

3.7. Configurat ion Validat ion

The cluster configuration is automatically validated according to the cluster schema at

/usr/share/cl uster/cl uster. rng during startup time and when a configuration is reloaded.

Also, you can validate a cluster configuration any time by using the ccs_co nfi g _val i d ate command. For information on configuration validation when using the ccs command, see

Section 6.1.6, “ Configuration Validation”

An annotated schema is available for viewing at /usr/share/d o c/cman-

X. Y . ZZ/cl uster_co nf. html (for example /usr/share/d o c/cman-

3. 0 . 12/cl uster_co nf. html ).

Configuration validation checks for the following basic errors:

XML validity — Checks that the configuration file is a valid XML file.

Configuration options — Checks to make sure that options (XML elements and attributes) are valid.

Option values — Checks that the options contain valid data (limited).

The following examples show a valid configuration and invalid configurations that illustrate the validation checks:

Valid configuration — Example 3.3, “ cl uster. co nf Sample Configuration: Valid File”

⁠Chapt er 3. Before Configuring t he Red Hat High Availabilit y Add- O n

Invalid XML —

Example 3.4, “ cl uster. co nf Sample Configuration: Invalid XML”

Invalid option —

Example 3.5, “ cl uster. co nf Sample Configuration: Invalid Option”

Invalid option value — Example 3.6, “ cl uster. co nf Sample Configuration: Invalid Option

Value”

Examp le 3.3. cl uster. co nf Samp le Co n f ig u rat io n : Valid File

<fence>

</fence>

</clusternode>

<fence>

</fence>

</clusternode>

<fence>

</fence>

</clusternode>

</clusternodes>

</fencedevices>

<rm>

</rm>

</cluster>

Examp le 3.4 . cl uster. co nf Samp le Co n f ig u rat io n : In valid XML

<fence>

</fence>

</clusternode>

<fence>

</fence>

</clusternode>

<fence>

</fence>

</clusternode>

</clusternodes>

</fencedevices>

Clust er Administ rat ion

<rm>

</rm>

<cluster> <----------------INVALID

In this example, the last line of the configuration (annotated as "INVALID" here) is missing a slash

— it is <cl uster> instead of </cl uster>.

Examp le 3.5. cl uster. co nf Samp le Co n f ig u rat io n : In valid O p t io n

<loging debug="off"/> <----------------INVALID

<fence>

</fence>

</clusternode>

<fence>

</fence>

</clusternode>

<fence>

</fence>

</clusternode>

</clusternodes>

</fencedevices>

<rm>

</rm>

In this example, the second line of the configuration (annotated as "INVALID" here) contains an invalid XML element — it is l o g i ng instead of l o g g i ng .

Examp le 3.6 . cl uster. co nf Samp le Co n f ig u rat io n : In valid O p t io n Valu e

<clusternode name="node-01.example.com" nodeid="-1"> <--------

INVALID

<fence>

</fence>

</clusternode>

<fence>

</fence>

</clusternode>

⁠Chapt er 3. Before Configuring t he Red Hat High Availabilit y Add- O n

<fence>

</fence>

</clusternode>

</clusternodes>

</fencedevices>

<rm>

</rm>

In this example, the fourth line of the configuration (annotated as "INVALID" here) contains an invalid value for the XML attribute, no d ei d in the cl usterno d e line for no d e-

0 1. exampl e. co m. The value is a negative value ("-1") instead of a positive value ("1"). For the

no d ei d attribute, the value must be a positive value.

3.8. Considerat ions for Net workManager

The use of Net wo rkMan ag er is not supported on cluster nodes. If you have installed

Net wo rkMan ag er on your cluster nodes, you should either remove it or disable it.

Note

The cman service will not start if Netwo rkManag er is either running or has been configured to run with the chkco nfi g command.

3.9. Considerat ions for Using Quorum Disk

Quorum Disk is a disk-based quorum daemon, q d i skd , that provides supplemental heuristics to determine node fitness. With heuristics you can determine factors that are important to the operation of the node in the event of a network partition. For example, in a four-node cluster with a 3:1 split, ordinarily, the three nodes automatically "win" because of the three-to-one majority. Under those circumstances, the one node is fenced. With q d i skd however, you can set up heuristics that allow the one node to win based on access to a critical resource (for example, a critical network path). If your cluster requires additional methods of determining node health, then you should configure

q d i skd to meet those needs.

Note

Configuring q d i skd is not required unless you have special requirements for node health. An example of a special requirement is an "all-but-one" configuration. In an all-but-one configuration, q d i skd is configured to provide enough quorum votes to maintain quorum even though only one node is working.

Clust er Administ rat ion

Important

Overall, heuristics and other q d i skd parameters for your deployment depend on the site environment and special requirements needed. To understand the use of heuristics and other

q d i skd parameters, refer to the qdisk(5) man page. If you require assistance understanding and using q d i skd for your site, contact an authorized Red Hat support representative.

If you need to use q d i skd , you should take into account the following considerations:

Clu st er n o d e vo t es

When using Quorum Disk, each cluster node must have one vote.

CMAN memb ersh ip t imeo u t valu e

The q d i skd membership timeout value is automatically configured based on the CMAN membership timeout value (the time a node needs to be unresponsive before CMAN considers that node to be dead, and not a member). q d i skd also performs extra sanity checks to guarantee that it can operate within the CMAN timeout. If you find that you need to reset this value, you must take the following into account:

The CMAN membership timeout value should be at least two times that of the q d i skd membership timeout value. The reason is because the quorum daemon must detect failed nodes on its own, and can take much longer to do so than CMAN. Other site-specific conditions may affect the relationship between the membership timeout values of CMAN and

q d i skd . For assistance with adjusting the CMAN membership timeout value, contact an authorized Red Hat support representative.

Fen cin g

To ensure reliable fencing when using q d i skd , use power fencing. While other types of fencing can be reliable for clusters not configured with q d i skd , they are not reliable for a cluster configured with q d i skd .

Maximu m n o d es

A cluster configured with q d i skd supports a maximum of 16 nodes. The reason for the limit is because of scalability; increasing the node count increases the amount of synchronous I/O contention on the shared quorum disk device.

Q u o ru m d isk d evice

A quorum disk device should be a shared block device with concurrent read/write access by all nodes in a cluster. The minimum size of the block device is 10 Megabytes. Examples of shared block devices that can be used by q d i skd are a multi-port SCSI RAID array, a

Fibre Channel RAID SAN, or a RAID-configured iSCSI target. You can create a quorum disk device with mkq d i sk, the Cluster Quorum Disk Utility. For information about using the utility refer to the mkqdisk(8) man page.

⁠Chapt er 3. Before Configuring t he Red Hat High Availabilit y Add- O n

Note

Using JBOD as a quorum disk is not recommended. A JBOD cannot provide dependable performance and therefore may not allow a node to write to it quickly enough. If a node is unable to write to a quorum disk device quickly enough, the node is falsely evicted from a cluster.

3.10. Red Hat High Availabilit y Add-On and SELinux

The High Availability Add-On for Red Hat Enterprise Linux 6 supports SELinux in the enfo rci ng state with the SELinux policy type set to targ eted .

Note

When using SELinux with the High Availability Add-On in a VM environment, you should ensure that the SELinux boolean fenced _can_netwo rk_co nnect is persistently set to o n.

This allows the fence_xvm fencing agent to work properly, enabling the system to fence virtual machines.

For more information about SELinux, refer to Deployment Guide for Red Hat Enterprise Linux 6.

3.11. Mult icast Addresses

The nodes in a cluster communicate among each other using multicast addresses. Therefore, each network switch and associated networking equipment in the Red Hat High Availability Add-On must be configured to enable multicast addresses and support IGMP (Internet Group Management

Protocol). Ensure that each network switch and associated networking equipment in the Red Hat

High Availability Add-On are capable of supporting multicast addresses and IGMP; if they are, ensure that multicast addressing and IGMP are enabled. Without multicast and IGMP, not all nodes can participate in a cluster, causing the cluster to fail; use UDP unicast in these environments, as described in

Section 3.12, “ UDP Unicast Traffic” .

Note

Procedures for configuring network switches and associated networking equipment vary according each product. Refer to the appropriate vendor documentation or other information about configuring network switches and associated networking equipment to enable multicast addresses and IGMP.

3.12. UDP Unicast T raffic

As of the Red Hat Enterprise Linux 6.2 release, the nodes in a cluster can communicate with each other using the UDP Unicast transport mechanism. It is recommended, however, that you use IP multicasting for the cluster network. UDP unicast is an alternative that can be used when IP multicasting is not available.

Clust er Administ rat ion

You can configure the Red Hat High-Availability Add-On to use UDP unicast by setting the cman

transpo rt= "ud pu" parameter in the cl uster. co nf configuration file. You can also specify

Unicast from the Net wo rk Co n f ig u rat io n page of the Co n g a user interface, as described in

Section 4.5.3, “ Network Configuration”

3.13. Considerat ions for

ri cci

For Red Hat Enterprise Linux 6, ri cci replaces ccsd . Therefore, it is necessary that ri cci is running in each cluster node to be able to propagate updated cluster configuration whether it is via the cman_to o l versi o n -r command, the ccs command, or the lu ci user interface server. You can start ri cci by using servi ce ri cci start or by enabling it to start at boot time via

chkco nfi g . For information on enabling IP ports for ri cci , refer to

Section 3.3.1, “ Enabling IP

Ports on Cluster Nodes” .

For the Red Hat Enterprise Linux 6.1 release and later, using ri cci requires a password the first time you propagate updated cluster configuration from any particular node. You set the ri cci password as root after you install ri cci on your system. To set this password, execute the passwd ri cci command, for user ri cci .

3.14 . Configuring Virt ual Machines in a Clust ered Environment

When you configure your cluster with virtual machine resources, you should use the rg manag er tools to start and stop the virtual machines. Using vi rsh to start the machine can result in the virtual machine running in more than one place, which can cause data corruption in the virtual machine.

To reduce the chances of administrators accidentally "double-starting" virtual machines by using both cluster and non-cluster tools in a clustered environment, you can configure your system by storing the virtual machine configuration files in a non-default location. Storing the virtual machine configuration files somewhere other than their default location makes it more difficult to accidentally start a virtual machine using vi rsh, as the configuration file will be unknown out of the box to

vi rsh.

The non-default location for virtual machine configuration files may be anywhere. The advantage of using an NFS share or a shared GFS2 file system is that the administrator does not need to keep the configuration files in sync across the cluster members. However, it is also permissible to use a local directory as long as the administrator keeps the contents synchronized somehow cluster-wide.

In the cluster configuration, virtual machines may reference this non-default location by using the

path attribute of a virtual machine resource. Note that the path attribute is a directory or set of directories separated by the colon ':' character, not a path to a specific file.

Warning

The l i bvi rt-g uests service should be disabled on all the nodes that are running

rg manag er. If a virtual machine autostarts or resumes, this can result in the virtual machine running in more than one place, which can cause data corruption in the virtual machine.

For more information on the attributes of a virtual machine resources, refer to Table B.26, “ Virtual

Machine (vm Resource)” .

4 0

⁠Chapt er 4 . Configuring Red Hat High Availabilit y Add- O n Wit h Conga

Chapter 4. Configuring Red Hat High Availability Add-On With

Conga

This chapter describes how to configure Red Hat High Availability Add-On software using Co n g a.

For information on using Co n g a to manage a running cluster, see Chapter 5, Managing Red Hat High

Availability Add-On With Conga

Note

Conga is a graphical user interface that you can use to administer the Red Hat High

Availability Add-On. Note, however, that in order to use this interface effectively you need to have a good and clear understanding of the underlying concepts. Learning about cluster configuration by exploring the available features in the user interface is not recommended, as it may result in a system that is not robust enough to keep all services running when components fail.

This chapter consists of the following sections:

Section 4.1, “ Configuration Tasks”

Section 4.2, “ Starting lu ci”

Section 4.3, “ Controlling Access to luci”

Section 4.4, “ Creating a Cluster”

Section 4.5, “ Global Cluster Properties”

Section 4.6, “ Configuring Fence Devices”

Section 4.7, “ Configuring Fencing for Cluster Members”

Section 4.8, “ Configuring a Failover Domain”

Section 4.9, “ Configuring Global Cluster Resources”

Section 4.10, “ Adding a Cluster Service to the Cluster”

4 .1. Configurat ion T asks

Configuring Red Hat High Availability Add-On software with Co n g a consists of the following steps:

1. Configuring and running the Co n g a configuration user interface — the lu ci server. Refer to

Section 4.2, “ Starting lu ci”

2. Creating a cluster. Refer to

Section 4.4, “ Creating a Cluster”

3. Configuring global cluster properties. Refer to Section 4.5, “ Global Cluster Properties” .

4. Configuring fence devices. Refer to Section 4.6, “ Configuring Fence Devices”

5. Configuring fencing for cluster members. Refer to Section 4.7, “ Configuring Fencing for

Cluster Members” .

6. Creating failover domains. Refer to

Section 4.8, “ Configuring a Failover Domain” .

4 1

Clust er Administ rat ion

7. Creating resources. Refer to Section 4.9, “ Configuring Global Cluster Resources”

8. Creating cluster services. Refer to

Section 4.10, “ Adding a Cluster Service to the Cluster”

4 .2. St art ing luci

Installing ricci

Using l uci to configure a cluster requires that ri cci be installed and running on the cluster

nodes, as described in Section 3.13, “ Considerations for ri cci ”

. As noted in that section, using ri cci requires a password which l uci requires you to enter for each cluster node

when you create a cluster, as described in Section 4.4, “ Creating a Cluster” .

Before starting lu ci, ensure that the IP ports on your cluster nodes allow connections to port

11111 from the lu ci server on any nodes that lu ci will be communicating with. For information

on enabling IP ports on cluster nodes, see Section 3.3.1, “ Enabling IP Ports on Cluster

Nodes” .

To administer Red Hat High Availability Add-On with Co n g a, install and run lu ci as follows:

1. Select a computer to host lu ci and install the lu ci software on that computer. For example:

# yum i nstal l l uci

Note

Typically, a computer in a server cage or a data center hosts lu ci; however, a cluster computer can host lu ci.

2. Start lu ci using servi ce l uci start. For example:

# servi ce l uci start

Starting luci: generating https SSL certificates... done

[ OK ]

Please, point your web browser to https://nano-01:8084 to access luci

Note

As of Red Hat Enterprise Linux release 6.1, you can configure some aspects of lu ci's behavior by means of the /etc/sysco nfi g /l uci file, including the port and host

parameters, as described in Section 3.4, “ Configuring lu ci with

/etc/sysco nfi g /l uci ”

. Modified port and host parameters will automatically be

reflected in the URL displayed when the lu ci service starts.

4 2

3. At a Web browser, place the URL of the lu ci server into the URL address box and click G o (or the equivalent). The URL syntax for the lu ci server is

⁠Chapt er 4 . Configuring Red Hat High Availabilit y Add- O n Wit h Conga

https: //luci_server_hostname: luci_server_port. The default value of

luci_server_port is 80 84 .

The first time you access lu ci, a web browser specific prompt regarding the self-signed SSL certificate (of the lu ci server) is displayed. Upon acknowledging the dialog box or boxes, your Web browser displays the lu ci login page.

4. Any user able to authenticate on the system that is hosting lu ci can log in to lu ci. As of Red

Hat Enterprise Linux 6.2 only the root user on the system that is running lu ci can access any of the lu ci components until an administrator (the root user or a user with administrator permission) sets permissions for that user. For information on setting lu ci permissions for

users, refer to Section 4.3, “ Controlling Access to luci” .

Logging in to lu ci displays the lu ci Ho meb ase page, as shown in Figure 4.1, “ luci

Homebase page” .

Fig u re 4 .1. lu ci Ho meb ase p ag e

Note

There is an idle timeout for lu ci that logs you out after 15 minutes of inactivity.

4 .3. Cont rolling Access t o luci

Since the initial release of Red Hat Enterprise Linux 6, the following features have been added to the

Users an d Permisio n s page.

As of Red Hat Enterprise Linux 6.2, the root user or a user who has been granted lu ci administrator permissions on a system running lu ci can control access to the various lu ci components by setting permissions for the individual users on a system.

As of Red Hat Enterprise Linux 6.3, the root user or a user who has been granted lu ci administrator permissions can add users to the lu ci interface and then set the user permissions for that user. You will still need to add that user to the system and set up a password for that user, but this feature allows you to configure permissions for the user before the user has logged in to

4 3

Clust er Administ rat ion

lu ci for the first time.

As of Red Hat Enterprise Linux 6.4, the root user or a user who has been granted lu ci administrator permissions can also use the lu ci interface to delete users from the lu ci interface, which resets any permissions you have configured for that user.

Note

You can modify the way in which lu ci performs authentication by editing the

/etc/pam. d /l uci file on the system. For information on using Linux-PAM, see the pam(8) man page.

To add users, delete users, or set the user permissions, log in to lu ci as ro o t or as a user who has previously been granted administrator permissions and click the Ad min selection in the upper right corner of the lu ci screen. This brings up the Users an d Permissio n s page, which displays the existing users.

To add a user to the lu ci interface, click on Ad d a User and enter the name of the user to add. You can then set permissions for that user, although you will still need to set up a password for that user.

To delete users from the lu ci interface, resetting any permissions you have configured for that user, select the user or users and click on D el ete Sel ected .

To set or change permissions for a user, select the user from the dropdown menu under User

Permissio n s. This allows you to set the following permissions:

Luci Ad mi ni strato r

Grants the user the same permissions as the root user, with full permissions on all clusters and the ability to set or remove permissions on all other users except root, whose permissions cannot be restricted.

C an C reate C l usters

Allows the user to create new clusters, as described in Section 4.4, “ Creating a Cluster” .

C an Impo rt Exi sti ng C l usters

Allows the user to add an existing cluster to the lu ci interface, as described in Section 5.1,

“ Adding an Existing Cluster to the luci Interface” .

For each cluster that has been created or imported to lu ci, you can set the following permissions for the indicated user:

C an Vi ew T hi s C l uster

Allows the user to view the specified cluster.

C an C hang e the C l uster C o nfi g urati o n

Allows the user to modify the configuration for the specified cluster, with the exception of adding and removing cluster nodes.

C an Enabl e, D i sabl e, R el o cate, and Mi g rate Servi ce G ro ups

Allows the user to manage high-availability services, as described in Section 5.5,

“ Managing High-Availability Services” .

4 4

⁠Chapt er 4 . Configuring Red Hat High Availabilit y Add- O n Wit h Conga

C an Sto p, Start, and R ebo o t C l uster No d es

Allows the user to manage the individual nodes of a cluster, as described in Section 5.3,

“ Managing Cluster Nodes” .

C an Ad d and D el ete No d es

Allows the user to add and delete nodes from a cluster, as described in Section 4.4,

“ Creating a Cluster” .

C an R emo ve T hi s C l uster fro m Luci

Allows the user to remove a cluster from the lu ci interface, as described in Section 5.4,

“ Starting, Stopping, Restarting, and Deleting Clusters” .

Click Submi t for the permissions to take affect, or click R eset to return to the initial values.

4 .4 . Creat ing a Clust er

Creating a cluster with lu ci consists of naming a cluster, adding cluster nodes to the cluster, entering the ricci passwords for each node, and submitting the request to create a cluster. If the node information and passwords are correct, Co n g a automatically installs software into the cluster nodes

(if the appropriate software packages are not currently installed) and starts the cluster. Create a cluster as follows:

1. Click Man ag e Clu st ers from the menu on the left side of the lu ci Ho meb ase page. The

Clu st ers screen appears, as shown in

Figure 4.2, “ luci cluster management page” .

Fig u re 4 .2. lu ci clu st er man ag emen t p ag e

2. Click Creat e. The Creat e New Clu st er dialog box appears, as shown in Figure 4.3, “ luci cluster creation dialog box” .

4 5

Clust er Administ rat ion

4 6

Fig u re 4 .3. lu ci clu st er creat io n d ialo g b o x

3. Enter the following parameters on the Creat e New Clu st er dialog box, as necessary:

At the Clu st er Name text box, enter a cluster name. The cluster name cannot exceed 15 characters.

If each node in the cluster has the same ricci password, you can check Use t h e same

p asswo rd f o r all n o d es to autofill the p asswo rd field as you add nodes.

Enter the node name for a node in the cluster in the No d e Name column and enter the

ricci password for the node in the Passwo rd column. A node name can be up to 255 bytes in length.

If your system is configured with a dedicated private network that is used only for cluster traffic, you may want to configure lu ci to communicate with ricci on an address that is different from the address to which the cluster node name resolves. You can do this by entering that address as the Ricci Ho st n ame.

If you are using a different port for the ricci agent than the default of 11111, you can change that parameter.

Click Ad d An o t h er No d e and enter the node name and ricci password for each additional node in the cluster.

If you do not want to upgrade the cluster software packages that are already installed on the nodes when you create the cluster, leave the Use lo cally in st alled p ackag es option selected. If you want to upgrade all cluster software packages, select the

Do wn lo ad Packag es option.

⁠Chapt er 4 . Configuring Red Hat High Availabilit y Add- O n Wit h Conga

Note

Whether you select the Use lo cally in st alled p ackag es or the Do wn lo ad

Packag es option, if any of the base cluster components are missing (cman,

rg manag er, mo d cl uster and all their dependencies), they will be installed. If they cannot be installed, the node creation will fail.

Check Reb o o t n o d es b ef o re jo in in g clu st er if desired.

Select En ab le sh ared st o rag e su p p o rt if clustered storage is required; this downloads the packages that support clustered storage and enables clustered LVM. You should select this only when you have access to the Resilient Storage Add-On or the Scalable

File System Add-On.

4. Click C reate C l uster. Clicking C reate C l uster causes the following actions: a. If you have selected Do wn lo ad Packag es, the cluster software packages are downloaded onto the nodes.

b. Cluster software is installed onto the nodes (or it is verified that the appropriate software packages are installed).

c. The cluster configuration file is updated and propagated to each node in the cluster.

d. The added nodes join the cluster.

A message is displayed indicating that the cluster is being created. When the cluster is ready,

the display shows the status of the newly created cluster, as shown in Figure 4.4, “ Cluster node display” . Note that if ricci is not running on any of the nodes, the cluster creation will

fail.

Fig u re 4 .4 . Clu st er n o d e d isp lay

5. After clicking C reate C l uster, you can add or delete nodes from the cluster by clicking the

Ad d or Delet e function from the menu at the top of the cluster node display page. Unless you are deleting an entire cluster, nodes must be stopped before being deleted. For

4 7

Clust er Administ rat ion

information on deleting a node from an existing cluster that is currently in operation, see

Section 5.3.4, “ Deleting a Member from a Cluster”

Warning

Removing a cluster node from the cluster is a destructive operation that cannot be undone.

4 .5. Global Clust er Propert ies

When you select a cluster to configure, a cluster-specific page is displayed. The page provides an interface for configuring cluster-wide properties. You can configure cluster-wide properties by clicking on Co n f ig u re along the top of the cluster display. This yields a tabbed interface which provides the following tabs: G en eral, Fen ce Daemo n , Net wo rk, Red u n d an t Rin g , Q Disk and

Lo g g in g . To configure the parameters in those tabs, follow the steps in the following sections. If you do not need to configure parameters in a tab, skip the section for that tab.

4 .5.1. Configuring General Propert ies

Clicking on the G en eral tab displays the G en eral Pro p ert ies page, which provides an interface for modifying the configuration version.

The Clu st er Name text box displays the cluster name; it does not accept a cluster name change.

The only way to change the name of a cluster is to create a new cluster configuration with the new name.

The Co n f ig u rat io n Versio n value is set to 1 at the time of cluster creation and is automatically incremented each time you modify your cluster configuration. However, if you need to set it to another value, you can specify it at the Co n f ig u rat io n Versio n text box.

If you have changed the Co n f ig u rat io n Versio n value, click Appl y for this change to take effect.

4 .5.2. Configuring Fence Daemon Propert ies

Clicking on the Fen ce Daemo n tab displays the Fen ce Daemo n Pro p ert ies page, which provides an interface for configuring Po st Fail Delay and Po st Jo in Delay. The values you configure for these parameters are general fencing properties for the cluster. To configure specific fence devices for the nodes of the cluster, use the Fen ce Devices menu item of the cluster display, as described in

Section 4.6, “ Configuring Fence Devices” .

The Po st Fail Delay parameter is the number of seconds the fence daemon (fenced ) waits before fencing a node (a member of the fence domain) after the node has failed. The Po st Fail

Delay default value is 0 . Its value may be varied to suit cluster and network performance.

The Po st Jo in Delay parameter is the number of seconds the fence daemon (fenced ) waits before fencing a node after the node joins the fence domain. lu ci sets the Po st Jo in Delay value to 6 . A typical setting for Po st Jo in Delay is between 20 and 30 seconds, but can vary according to cluster and network performance.

Enter the values required and click Appl y for changes to take effect.

4 8

⁠Chapt er 4 . Configuring Red Hat High Availabilit y Add- O n Wit h Conga

Note

For more information about Po st Jo in Delay and Po st Fail Delay, refer to the fenced(8) man page.

4 .5.3. Net work Configurat ion

Clicking on the Net wo rk tab displays the Net wo rk Co n f ig u rat io n page, which provides an interface for configuring the network transport type.

You can use this tab to select one of the following options:

UDP Mu lt icast an d Let Clu st er Ch o o se t h e Mu lt icast Ad d ress

This is the default setting. With this option selected, the Red Hat High Availability Add-On software creates a multicast address based on the cluster ID. It generates the lower 16 bits of the address and appends them to the upper portion of the address according to whether the IP protocol is

IPv4 or IPv6:

For IPv4 — The address formed is 239.192. plus the lower 16 bits generated by Red Hat High

Availability Add-On software.

For IPv6 — The address formed is FF15:: plus the lower 16 bits generated by Red Hat High

Availability Add-On software.

Note

The cluster ID is a unique identifier that cman generates for each cluster. To view the cluster

ID, run the cman_to o l status command on a cluster node.

UDP Mu lt icast an d Sp ecif y t h e Mu lt icast Ad d ress Man u ally

If you need to use a specific multicast address, select this option enter a multicast address into the

Mu lt icast Ad d ress text box.

If you do specify a multicast address, you should use the 239.192.x.x series (or FF15:: for IPv6) that cman uses. Otherwise, using a multicast address outside that range may cause unpredictable results. For example, using 224.0.0.x (which is "All hosts on the network") may not be routed correctly, or even routed at all by some hardware.

If you specify or modify a multicast address, you must restart the cluster for this to take effect. For

information on starting and stopping a cluster with Co n g a, refer to Section 5.4, “ Starting,

Stopping, Restarting, and Deleting Clusters” .

Note

If you specify a multicast address, make sure that you check the configuration of routers that cluster packets pass through. Some routers may take a long time to learn addresses, seriously impacting cluster performance.

UDP Un icast ( UDPU)

4 9

Clust er Administ rat ion

As of the Red Hat Enterprise Linux 6.2 release, the nodes in a cluster can communicate with each other using the UDP Unicast transport mechanism. It is recommended, however, that you use IP multicasting for the cluster network. UDP Unicast is an alternative that can be used when IP multicasting is not available. For GFS2 deployments using UDP Unicast is not recommended.

Click Appl y. When changing the transport type, a cluster restart is necessary for the changes to take effect.

4 .5.4 . Configuring Redundant Ring Prot ocol

As of Red Hat Enterprise Linux 6.4, the Red Hat High Availability Add-On supports the configuration of redundant ring protocol. When using redundant ring protocol, there are a variety of

considerations you must take into account, as described in Section 8.6, “ Configuring Redundant

Ring Protocol” .

Clicking on the Red u n d an t Rin g tab displays the Red u n d an t Rin g Pro t o co l Co n f ig u rat io n page. This page displays all of the nodes that are currently configured for the cluster. If you are configuring a system to use redundant ring protocol, you must specify the Alt ern at e Name for each node for the second ring.

The Red u n d an t Rin g Pro t o co l Co n f ig u rat io n page optionally allows you to specify the

Alt ern at e Rin g Mu lt icast Ad d ress, the Alt ern at e Rin g CMAN Po rt , and the Alt ern at e Rin g

Mu lt icast Packet T T L for the second ring.

If you specify a multicast address for the second ring, either the alternate multicast address or the alternate port must be different from the multicast address for the first ring. If you specify an alternate port, the port numbers of the first ring and the second ring must differ by at least two, since the system itself uses port and port-1 to perform operations. If you do not specify an alternate multicast address, the system will automatically use a different multicast address for the second ring.

4 .5.5. Quorum Disk Configurat ion

Clicking on the Q Disk tab displays the Q u o ru m Disk Co n f ig u rat io n page, which provides an interface for configuring quorum disk parameters if you need to use a quorum disk.

Important

Quorum disk parameters and heuristics depend on the site environment and the special requirements needed. To understand the use of quorum disk parameters and heuristics, refer to the qdisk(5) man page. If you require assistance understanding and using quorum disk, contact an authorized Red Hat support representative.

The Do No t Use a Q u o ru m Disk parameter is enabled by default. If you need to use a quorum disk, click Use a Q u o ru m Disk, enter the quorum disk parameters, and click Appl y. You must restart the cluster for the changes to take effect.

Table 4.1, “ Quorum-Disk Parameters” describes the quorum disk parameters.

T ab le 4 .1. Q u o ru m- Disk Paramet ers

Paramet er Descrip t io n

Paramet er

Sp ecif y Ph ysical

Device: By Device

Lab el

Heu rist ics

Min imu m T o t al

Sco re

⁠Chapt er 4 . Configuring Red Hat High Availabilit y Add- O n Wit h Conga

Descrip t io n

Specifies the quorum disk label created by the mkq d i sk utility. If this field is used, the quorum daemon reads the /pro c/parti ti o ns file and checks for qdisk signatures on every block device found, comparing the label against the specified label. This is useful in configurations where the quorum device name differs among nodes.

Pat h t o Pro g ram — The program used to determine if this heuristic is available. This can be anything that can be executed by /bi n/sh -c. A return value of 0 indicates success; anything else indicates failure. This field is required.

In t erval — The frequency (in seconds) at which the heuristic is polled.

The default interval for every heuristic is 2 seconds.

Sco re — The weight of this heuristic. The default score for each heuristic is 1.

T KO — The number of consecutive failures required before this heuristic is declared unavailable.

The minimum score for a node to be considered "alive". If omitted or set to

0, the default function, fl o o r((n+ 1)/2), is used, where n is the sum of the heuristics scores. The Min imu m T o t al Sco re value must never exceed the sum of the heuristic scores; otherwise, the quorum disk cannot be available.

Note

Clicking Ap p ly on the Q Disk Co n f ig u rat io n tab propagates changes to the cluster configuration file (/etc/cl uster/cl uster. co nf) in each cluster node. However, for the quorum disk to operate or for any modifications you have made to the quorum disk

parameters to take effect, you must restart the cluster (refer to Section 5.4, “ Starting, Stopping,

Restarting, and Deleting Clusters” ), ensuring that you have restarted the q d i skd daemon on

each node.

4 .5.6. Logging Configurat ion

Clicking on the Lo g g in g tab displays the Lo g g in g Co n f ig u rat io n page, which provides an interface for configuring logging settings.

You can configure the following settings for global logging configuration:

Checking Lo g Deb u g g in g Messag es enables debugging messages in the log file.

Checking Lo g Messag es t o Syslo g enables messages to sysl o g . You can select the Syslo g

Messag e Facilit y and the Syslo g Messag e Prio rit y. The Syslo g Messag e Prio rit y setting indicates that messages at the selected level and higher are sent to sysl o g .

Checking Lo g Messag es t o Lo g File enables messages to the log file. You can specify the Lo g

File Pat h name. The lo g f ile messag e p rio rit y setting indicates that messages at the selected level and higher are written to the log file.

You can override the global logging settings for specific daemons by selecting one of the daemons listed beneath the Daemo n - sp ecif ic Lo g g in g O verrid es heading at the bottom of the Lo g g in g

Clust er Administ rat ion

Co n f ig u rat io n page. After selecting the daemon, you can check whether to log the debugging messages for that particular daemon. You can also specify the sysl o g and log file settings for that daemon.

Click Appl y for the logging configuration changes you have specified to take effect.

4 .6. Configuring Fence Devices

Configuring fence devices consists of creating, updating, and deleting fence devices for the cluster.

You must configure the fence devices in a cluster before you can configure fencing for the nodes in the cluster.

Creating a fence device consists of selecting a fence device type and entering parameters for that fence device (for example, name, IP address, login, and password). Updating a fence device consists of selecting an existing fence device and changing parameters for that fence device. Deleting a fence device consists of selecting an existing fence device and deleting it.

Note

It is recommended that you configure multiple fencing mechanisms for each node. A fencing device can fail due to network split, a power outage, or a problem in the fencing device itself.

Configuring multiple fencing mechanisms can reduce the likelihood that the failure of a fencing device will have fatal results.

This section provides procedures for the following tasks:

Creating fence devices — Refer to Section 4.6.1, “ Creating a Fence Device” . Once you have

created and named a fence device, you can configure the fence devices for each node in the

cluster, as described in Section 4.7, “ Configuring Fencing for Cluster Members” .

Updating fence devices — Refer to

Section 4.6.2, “ Modifying a Fence Device”

Deleting fence devices — Refer to

Section 4.6.3, “ Deleting a Fence Device”

From the cluster-specific page, you can configure fence devices for that cluster by clicking on Fen ce

Devices along the top of the cluster display. This displays the fence devices for the cluster and displays the menu items for fence device configuration: Ad d and Delet e. This is the starting point of each procedure described in the following sections.

Note

If this is an initial cluster configuration, no fence devices have been created, and therefore none are displayed.

Figure 4.5, “ luci fence devices configuration page” shows the fence devices configuration screen

before any fence devices have been created.

⁠Chapt er 4 . Configuring Red Hat High Availabilit y Add- O n Wit h Conga

Fig u re 4 .5. lu ci f en ce d evices co n f ig u rat io n p ag e

4 .6.1. Creat ing a Fence Device

To create a fence device, follow these steps:

1. From the Fen ce Devices configuration page, click Ad d . Clicking Ad d displays the Ad d

Fence D evi ce (Instance) dialog box. From this dialog box, select the type of fence device to configure.

2. Specify the information in the Ad d Fence D evi ce (Instance) dialog box according to the type of fence device. Refer to

Appendix A, Fence Device Parameters

for more information about fence device parameters. In some cases you will need to specify additional nodespecific parameters for the fence device when you configure fencing for the individual nodes,

as described in Section 4.7, “ Configuring Fencing for Cluster Members” .

3. Click Submi t.

After the fence device has been added, it appears on the Fen ce Devices configuration page.

4 .6.2. Modifying a Fence Device

To modify a fence device, follow these steps:

1. From the Fen ce Devices configuration page, click on the name of the fence device to modify.

This displays the dialog box for that fence device, with the values that have been configured for the device.

2. To modify the fence device, enter changes to the parameters displayed. Refer to Appendix A,

Fence Device Parameters

for more information.

3. Click Appl y and wait for the configuration to be updated.

4 .6.3. Delet ing a Fence Device

Clust er Administ rat ion

Note

Fence devices that are in use cannot be deleted. To delete a fence device that a node is currently using, first update the node fence configuration for any node using the device and then delete the device.

To delete a fence device, follow these steps:

1. From the Fen ce Devices configuration page, check the box to the left of the fence device or devices to select the devices to delete.

2. Click D el ete and wait for the configuration to be updated. A message appears indicating which devices are being deleted.

When the configuration has been updated, the deleted fence device no longer appears in the display.

4 .7. Configuring Fencing for Clust er Members

Once you have completed the initial steps of creating a cluster and creating fence devices, you need to configure fencing for the cluster nodes by following the steps in this section. Note that you must configure fencing for each node in the cluster.

The following sections provide procedures for configuring a single fence device for a node, configuring a node with a backup fence device, and configuring a node with redundant power supplies:

Section 4.7.1, “ Configuring a Single Fence Device for a Node”

Section 4.7.2, “ Configuring a Backup Fence Device”

Section 4.7.3, “ Configuring a Node with Redundant Power”

4 .7.1. Configuring a Single Fence Device for a Node

Use the following procedure to configure a node with a single fence device.

1. From the cluster-specific page, you can configure fencing for the nodes in the cluster by clicking on No d es along the top of the cluster display. This displays the nodes that constitute the cluster. This is also the default page that appears when you click on the cluster name beneath Man ag e Clu st ers from the menu on the left side of the lu ci Ho meb ase page.

2. Click on a node name. Clicking a link for a node causes a page to be displayed for that link showing how that node is configured.

The node-specific page displays any services that are currently running on the node, as well as any failover domains of which this node is a member. You can modify an existing failover domain by clicking on its name. For information on configuring failover domains, see

Section 4.8, “ Configuring a Failover Domain” .

3. On the node-specific page, under Fen ce Devices, click Ad d Fence Metho d . This displays the Ad d Fence Metho d to No d e dialog box.

⁠Chapt er 4 . Configuring Red Hat High Availabilit y Add- O n Wit h Conga

4. Enter a Met h o d Name for the fencing method that you are configuring for this node. This is an arbitrary name that will be used by Red Hat High Availability Add-On; it is not the same as the DNS name for the device.

5. Click Submi t. This displays the node-specific screen that now displays the method you have just added under Fen ce Devices.

6. Configure a fence instance for this method by clicking the Ad d Fence Instance button that appears beneath the fence method. This displays the Ad d Fen ce Device ( In st an ce) dropdown menu from which you can select a fence device you have previously configured, as

described in Section 4.6.1, “ Creating a Fence Device”

7. Select a fence device for this method. If this fence device requires that you configure nodespecific parameters, the display shows the parameters to configure. For information on fencing parameters, refer to

Appendix A, Fence Device Parameters

Note

For non-power fence methods (that is, SAN/storage fencing), Un f en cin g is selected by default on the node-specific parameters display. This ensures that a fenced node's access to storage is not re-enabled until the node has been rebooted. When you configure a device that requires unfencing, the cluster must first be stopped and the full configuration including devices and unfencing must be added before the cluster is started. For information on unfencing a node, refer to the fence_no d e(8) man page.

8. Click Submi t. This returns you to the node-specific screen with the fence method and fence instance displayed.

4 .7.2. Configuring a Backup Fence Device

Use the following procedure to configure a backup fence device for a node.

1. Use the procedure provided in

Section 4.7.1, “ Configuring a Single Fence Device for a Node”

to configure the primary fencing method for a node.

2. Beneath the display of the primary method you defined, click Ad d Fence Metho d .

3. Enter a name for the backup fencing method that you are configuring for this node and click

Submi t. This displays the node-specific screen that now displays the method you have just added, below the primary fence method.

4. Configure a fence instance for this method by clicking Ad d Fence Instance. This displays a drop-down menu from which you can select a fence device you have previously configured,

as described in Section 4.6.1, “ Creating a Fence Device”

5. Select a fence device for this method. If this fence device requires that you configure nodespecific parameters, the display shows the parameters to configure. For information on fencing parameters, refer to

Appendix A, Fence Device Parameters

6. Click Submi t. This returns you to the node-specific screen with the fence method and fence instance displayed.

Clust er Administ rat ion

You can continue to add fencing methods as needed. You can rearrange the order of fencing methods that will be used for this node by clicking on Mo ve Up and Mo ve Do wn .

4 .7.3. Configuring a Node wit h Redundant Power

If your cluster is configured with redundant power supplies for your nodes, you must be sure to configure fencing so that your nodes fully shut down when they need to be fenced. If you configure each power supply as a separate fence method, each power supply will be fenced separately; the second power supply will allow the system to continue running when the first power supply is fenced and the system will not be fenced at all. To configure a system with dual power supplies, you must configure your fence devices so that both power supplies are shut off and the system is taken completely down. When configuring your system using Co n g a, this requires that you configure two instances within a single fencing method.

To configure fencing for a node with dual power supplies, follow the steps in this section.

1. Before you can configure fencing for a node with redundant power, you must configure each of the power switches as a fence device for the cluster. For information on configuring fence devices, see

Section 4.6, “ Configuring Fence Devices” .

2. From the cluster-specific page, click on No d es along the top of the cluster display. This displays the nodes that constitute the cluster. This is also the default page that appears when you click on the cluster name beneath Man ag e Clu st ers from the menu on the left side of the lu ci Ho meb ase page.

3. Click on a node name. Clicking a link for a node causes a page to be displayed for that link showing how that node is configured.

4. On the node-specific page, click Ad d Fence Metho d .

5. Enter a name for the fencing method that you are configuring for this node.

6. Click Submi t. This displays the node-specific screen that now displays the method you have just added under Fen ce Devices.

7. Configure the first power supply as a fence instance for this method by clicking Ad d Fence

Instance. This displays a drop-down menu from which you can select one of the power

fencing devices you have previously configured, as described in Section 4.6.1, “ Creating a

Fence Device” .

8. Select one of the power fence devices for this method and enter the appropriate parameters for this device.

9. Click Submi t. This returns you to the node-specific screen with the fence method and fence instance displayed.

10. Under the same fence method for which you have configured the first power fencing device, click Ad d Fence Instance. This displays a drop-down menu from which you can select the second power fencing devices you have previously configured, as described in

Section 4.6.1, “ Creating a Fence Device”

11. Select the second of the power fence devices for this method and enter the appropriate parameters for this device.

12. Click Submi t. This returns you to the node-specific screen with the fence methods and fence instances displayed, showing that each device will power the system off in sequence and

power the system on in sequence. This is shown in Figure 4.6, “ Dual-Power Fencing

Section 4.8.1, “ Adding a Failover Domain”

⁠Chapt er 4 . Configuring Red Hat High Availabilit y Add- O n Wit h Conga

Fig u re 4 .6 . Du al- Po wer Fen cin g Co n f ig u rat io n

4 .7.4 . T est ing t he Fence Configurat ion

As of Red Hat Enterprise Linux Release 6.4, you can test the fence configuration for each node in a cluster with the fence_check utility.

The following example shows the output of a successful execution of this command.

[root@ host-098 ~]# fence_check fence_check run at Wed Jul 23 09:13:57 CDT 2014 pid: 4769

Testing host-098 method 1: success

Testing host-099 method 1: success

Testing host-100 method 1: success

For information on this utility, see the fence_check(8) man page.

4 .8. Configuring a Failover Domain

A failover domain is a named subset of cluster nodes that are eligible to run a cluster service in the event of a node failure. A failover domain can have the following characteristics:

Unrestricted — Allows you to specify that a subset of members are preferred, but that a cluster service assigned to this domain can run on any available member.

Clust er Administ rat ion

Restricted — Allows you to restrict the members that can run a particular cluster service. If none of the members in a restricted failover domain are available, the cluster service cannot be started

(either manually or by the cluster software).

Unordered — When a cluster service is assigned to an unordered failover domain, the member on which the cluster service runs is chosen from the available failover domain members with no priority ordering.

Ordered — Allows you to specify a preference order among the members of a failover domain. The member at the top of the list is the most preferred, followed by the second member in the list, and so on.

Failback — Allows you to specify whether a service in the failover domain should fail back to the node that it was originally running on before that node failed. Configuring this characteristic is useful in circumstances where a node repeatedly fails and is part of an ordered failover domain.

In that circumstance, if a node is the preferred node in a failover domain, it is possible for a service to fail over and fail back repeatedly between the preferred node and another node, causing severe impact on performance.

Note

The failback characteristic is applicable only if ordered failover is configured.

Note

Changing a failover domain configuration has no effect on currently running services.

Note

Failover domains are not required for operation.

By default, failover domains are unrestricted and unordered.

In a cluster with several members, using a restricted failover domain can minimize the work to set up the cluster to run a cluster service (such as httpd ), which requires you to set up the configuration identically on all members that run the cluster service. Instead of setting up the entire cluster to run the cluster service, you can set up only the members in the restricted failover domain that you associate with the cluster service.

Note

To configure a preferred member, you can create an unrestricted failover domain comprising only one cluster member. Doing that causes a cluster service to run on that cluster member primarily (the preferred member), but allows the cluster service to fail over to any of the other members.

The following sections describe adding, modifying, and deleting a failover domain:

⁠Chapt er 4 . Configuring Red Hat High Availabilit y Add- O n Wit h Conga

Section 4.8.2, “ Modifying a Failover Domain”

Section 4.8.3, “ Deleting a Failover Domain”

4 .8.1. Adding a Failover Domain

To add a failover domain, follow the steps in this section.

1. From the cluster-specific page, you can configure failover domains for that cluster by clicking on Failo ver Do main s along the top of the cluster display. This displays the failover domains that have been configured for this cluster.

2. Click Ad d . Clicking Ad d causes the display of the Ad d Fai l o ver D o mai n to C l uster dialog box, as shown in

Figure 4.7, “ luci failover domain configuration dialog box”

Fig u re 4 .7. lu ci f ailo ver d o main co n f ig u rat io n d ialo g b o x

3. In the Ad d Fai l o ver D o mai n to C l uster dialog box, specify a failover domain name at the Name text box.

Note

The name should be descriptive enough to distinguish its purpose relative to other names used in your cluster.

Clust er Administ rat ion

4. To enable setting failover priority of the members in the failover domain, click the Prio rit iz ed checkbox. With Prio rit iz ed checked, you can set the priority value, Prio rit y, for each node selected as members of the failover domain.

5. To restrict failover to members in this failover domain, click the Rest rict ed checkbox. With

Rest rict ed checked, services assigned to this failover domain fail over only to nodes in this failover domain.

6. To specify that a node does not fail back in this failover domain, click the No Failb ack checkbox. With No Failb ack checked, if a service fails over from a preferred node, the service does not fail back to the original node once it has recovered.

7. Configure members for this failover domain. Click the Memb er checkbox for each node that is to be a member of the failover domain. If Prio rit iz ed is checked, set the priority in the

Prio rit y text box for each member of the failover domain.

8. Click C reate. This displays the Fai l o ver D o mai ns page with the newly-created failover domain displayed. A message indicates that the new domain is being created. Refresh the page for an updated status.

4 .8.2. Modifying a Failover Domain

To modify a failover domain, follow the steps in this section.

1. From the cluster-specific page, you can configure Failover Domains for that cluster by clicking on Failo ver Do main s along the top of the cluster display. This displays the failover domains that have been configured for this cluster.

2. Click on the name of a failover domain. This displays the configuration page for that failover domain.

3. To modify the Prio rit iz ed , Rest rict ed , or No Failb ack properties for the failover domain, click or unclick the checkbox next to the property and click Upd ate P ro perti es.

4. To modify the failover domain membership, click or unclick the checkbox next to the cluster member. If the failover domain is prioritized, you can also modify the priority setting for the cluster member. Click Upd ate Setti ng s.

4 .8.3. Delet ing a Failover Domain

To delete a failover domain, follow the steps in this section.

2. Select the checkbox for the failover domain to delete.

3. Click Delet e.

4 .9. Configuring Global Clust er Resources

You can configure global resources that can be used by any service running in the cluster, and you can configure resources that are available only to a specific service.

⁠Chapt er 4 . Configuring Red Hat High Availabilit y Add- O n Wit h Conga

To add a global cluster resource, follow the steps in this section. You can add a resource that is

local to a particular service when you configure the service, as described in Section 4.10, “ Adding a

Cluster Service to the Cluster” .

1. From the cluster-specific page, you can add resources to that cluster by clicking on

Reso u rces along the top of the cluster display. This displays the resources that have been configured for that cluster.

2. Click Ad d . This displays the Ad d Reso u rce t o Clu st er drop-down menu.

3. Click the drop-down box under Ad d Reso u rce t o Clu st er and select the type of resource to configure.

4. Enter the resource parameters for the resource you are adding. Appendix B, HA Resource

Parameters

describes resource parameters.

5. Click Submi t. Clicking Submi t returns to the resources page that displays the display of

R eso urces, which displays the added resource (and other resources).

To modify an existing resource, perform the following steps.

1. From the lu ci Reso u rces page, click on the name of the resource to modify. This displays the parameters for that resource.

2. Edit the resource parameters.

3. Click Appl y.

To delete an existing resource, perform the following steps.

1. From the lu ci Reso u rces page, click the checkbox for any resources to delete.

2. Click Delet e.

As of the Red Hat Enterprise Linux 6.6 release, you can sort the columns in a resource list by clicking on the header for the sort category.

Clicking on the Name/IP header once sorts the resources alphabetically, according to resource name. Clicking on the Name/IP header a second time sourts the resources in reverse alphabetic order, according to resource name.

Clicking on the T yp e header once sorts the resources alphabetically, according to resource type.

Clicking on the T yp e header a second time sourts the resources in reverse alphabetic order, according to resource type.

Clicking on the In Use header once sorts the resources so that they are grouped according to whether they are in use or not.

4 .10. Adding a Clust er Service t o t he Clust er

To add a cluster service to the cluster, follow the steps in this section.

1. From the cluster-specific page, you can add services to that cluster by clicking on Service

G ro u p s along the top of the cluster display. This displays the services that have been configured for that cluster. (From the Service G ro u p s page, you can also start, restart, and

disable a service, as described in Section 5.5, “ Managing High-Availability Services”

2. Click Ad d . This displays the Ad d Service G ro u p t o Clu st er dialog box.

Clust er Administ rat ion

3. On the Ad d Servi ce G ro up to C l uster dialog box, at the Service Name text box, type the name of the service.

Note

Use a descriptive name that clearly distinguishes the service from other services in the cluster.

4. Check the Au t o mat ically St art T h is Service checkbox if you want the service to start automatically when a cluster is started and running. If the checkbox is not checked, the service must be started manually any time the cluster comes up from the stopped state.

5. Check the Ru n Exclu sive checkbox to set a policy wherein the service only runs on nodes that have no other services running on them.

6. If you have configured failover domains for the cluster, you can use the drop-down menu of the Failo ver Do main parameter to select a failover domain for this service. For information on configuring failover domains, see

Section 4.8, “ Configuring a Failover Domain” .

7. Use the Reco very Po licy drop-down box to select a recovery policy for the service. The options are to Relo cat e, Rest art , Rest art - Disab le, or Disab le the service.

Selecting the Rest art option indicates that the system should attempt to restart the failed service before relocating the service. Selecting the Relo cat e option indicates that the system should try to restart the service in a different node. Selecting the Disab le option indicates that the system should disable the resource group if any component fails. Selecting the

Rest art - Disab le option indicates that the system should attempt to restart the service in place if it fails, but if restarting the service fails the service will be disabled instead of being moved to another host in the cluster.

If you select Rest art or Rest art - Disab le as the recovery policy for the service, you can specify the maximum number of restart failures before relocating or disabling the service, and you can specify the length of time in seconds after which to forget a restart.

8. To add a resource to the service, click Ad d R eso urce. Clicking Ad d R eso urce causes the display of the Ad d R eso urce T o Servi ce drop-down box that allows you to add an existing global resource or to add a new resource that is available only to this service.

Note

When configuring a cluster service that includes a floating IP address resource, you must configure the IP resource as the first entry.

To add an existing global resource, click on the name of the existing resource from the

Ad d R eso urce T o Servi ce drop-down box. This displays the resource and its parameters on the Service G ro u p s page for the service you are configuring. For

information on adding or modifying global resources, see Section 4.9, “ Configuring

Global Cluster Resources” ).

To add a new resource that is available only to this service, select the type of resource to configure from the Ad d R eso urce T o Servi ce drop-down box and enter the resource parameters for the resource you are adding.

Section 5.1, “ Adding an Existing Cluster to the luci Interface”

describes resource parameters.

⁠Chapt er 4 . Configuring Red Hat High Availabilit y Add- O n Wit h Conga

When adding a resource to a service, whether it is an existing global resource or a resource available only to this service, you can specify whether the resource is an

In d ep en d en t Su b t ree or a No n - Crit ical Reso u rce.

If you specify that a resource is an independent subtree, then if that resource fails only that resource is restarted (rather than the entire service) before the system attempting normal recovery. You can specify the maximum number of restarts to attempt for that resource on a node before implementing the recovery policy for the service. You can also specify the length of time in seconds after which the system will implement the recovery policy for the service.

If you specify that the resource is a non-critical resource, then if that resource fails only that resource is restarted, and if the resource continues to fail then only that resource is disabled, rather than the entire service. You can specify the maximum number of restarts to attempt for that resource on a node before disabling that resource. You can also specify the length of time in seconds after which the system will disable that resource.

9. If you want to add child resources to the resource you are defining, click Ad d C hi l d

R eso urce. Clicking Ad d C hi l d R eso urce causes the display of the Ad d Reso u rce T o

Service drop-down box, from which you can add an existing global resource or add a new resource that is available only to this service. You can continue adding children resources to the resource to suit your requirements.

Note

If you are adding a Samba-service resource, add it directly to the service, not as a child of another resource.

Note

When configuring a dependency tree for a cluster service that includes a floating IP address resource, you must configure the IP resource as the first entry and not as the child of another resource.

10. When you have completed adding resources to the service, and have completed adding children resources to resources, click Submi t. Clicking Submi t returns to the Service

G ro u p s page displaying the added service (and other services).

Clust er Administ rat ion

Note

To verify the existence of the IP service resource used in a cluster service, you can use the

/sbi n/i p ad d r sho w command on a cluster node (rather than the obsoleted i fco nfi g command). The following output shows the /sbi n/i p ad d r sho w command executed on a node running a cluster service:

1: lo: <LOOPBACK,UP> mtu 16436 qdisc noqueue

link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00

inet 127.0.0.1/8 scope host lo

inet6 ::1/128 scope host

valid_lft forever preferred_lft forever

2: eth0: <BROADCAST,MULTICAST,UP> mtu 1356 qdisc pfifo_fast qlen

1000

link/ether 00:05:5d:9a:d8:91 brd ff:ff:ff:ff:ff:ff

inet 10.11.4.31/22 brd 10.11.7.255 scope global eth0

inet6 fe80::205:5dff:fe9a:d891/64 scope link

inet 10.11.4.240/22 scope global secondary eth0

valid_lft forever preferred_lft forever

To modify an existing service, perform the following steps.

1. From the Service G ro u p s dialog box, click on the name of the service to modify. This displays the parameters and resources that have been configured for that service.

2. Edit the service parameters.

3. Click Submi t.

To delete one or more existing services, perform the following steps.

1. From the lu ci Service G ro u p s page, click the checkbox for any services to delete.

2. Click Delet e.

3. As of Red Hat Enterprise Linux 6.3, before lu ci deletes any services a message appears asking you to confirm that you intend to delete the service groups or groups, which stops the resources that comprise it. Click Can cel to close the dialog box without deleting any services, or click Pro ceed to remove the selected service or services.

⁠Chapt er 5. Managing Red Hat High Availabilit y Add- O n Wit h Conga

Chapter 5. Managing Red Hat High Availability Add-On With

Conga

This chapter describes various administrative tasks for managing Red Hat High Availability Add-On and consists of the following sections:

Section 5.2, “ Removing a Cluster from the luci Interface”

Section 5.3, “ Managing Cluster Nodes”

Section 5.4, “ Starting, Stopping, Restarting, and Deleting Clusters”

Section 5.5, “ Managing High-Availability Services”

Section 5.6, “ Backing Up and Restoring the luci Configuration”

5.1. Adding an Exist ing Clust er t o t he luci Int erface

If you have previously created a High Availability Add-On cluster you can easily add the cluster to the lu ci interface so that you can manage the cluster with Co n g a.

To add an existing cluster to the lu ci interface, follow these steps:

1. Click Man ag e Clu st ers from the menu on the left side of the lu ci Ho meb ase page. The

Clu st ers screen appears.

2. Click Ad d . The Ad d Exist in g Clu st er screen appears.

3. Enter the node hostname and ricci password for any of the nodes in the existing cluster.

Since each node in the cluster contains all of the configuration information for the cluster, this should provide enough information to add the cluster to the lu ci interface.

4. Click C o nnect. The Ad d Exist in g Clu st er screen then displays the cluster name and the remaining nodes in the cluster.

5. Enter the individual ricci passwords for each node in the cluster, or enter one password and select Use same p asswo rd f o r all n o d es.

6. Click Ad d C l uster. The previously-configured cluster now displays on the Man ag e

Clu st ers screen.

5.2. Removing a Clust er from t he luci Int erface

You can remove a cluster from the lu ci management GUI without affecting the cluster services or cluster membership. If you remove a cluster, you can later add the cluster back, or you can add it to

another lu ci instance, as described in Section 5.1, “ Adding an Existing Cluster to the luci Interface”

To remove a cluster from the lu ci management GUI without affecting the cluster services or cluster membership, follow these steps:

1. Click Man ag e Clu st ers from the menu on the left side of the lu ci Ho meb ase page. The

Clu st ers screen appears.

2. Select the cluster or clusters you wish to remove.

Clust er Administ rat ion

3. Click Remo ve. The system will ask you to confirm whether to remove the cluster from the lu ci management GUI.

For information on deleting a cluster entirely, stopping all cluster services removing the cluster

configuration information from the nodes themselves, refer to Section 5.4, “ Starting, Stopping,

Restarting, and Deleting Clusters” .

5.3. Managing Clust er Nodes

This section documents how to perform the following node-management functions through the lu ci server component of Co n g a:

Section 5.3.1, “ Rebooting a Cluster Node”

Section 5.3.2, “ Causing a Node to Leave or Join a Cluster”

Section 5.3.3, “ Adding a Member to a Running Cluster”

Section 5.3.4, “ Deleting a Member from a Cluster”

5.3.1. Reboot ing a Clust er Node

To reboot a node in a cluster, perform the following steps:

1. From the cluster-specific page, click on No d es along the top of the cluster display. This displays the nodes that constitute the cluster. This is also the default page that appears when you click on the cluster name beneath Man ag e Clu st ers from the menu on the left side of the lu ci Ho meb ase page.

2. Select the node to reboot by clicking the checkbox for that node.

3. Select the Reb o o t function from the menu at the top of the page. This causes the selected node to reboot and a message appears at the top of the page indicating that the node is being rebooted.

4. Refresh the page to see the updated status of the node.

It is also possible to reboot more than one node at a time by selecting all of the nodes that you wish to reboot before clicking on Reb o o t .

5.3.2. Causing a Node t o Leave or Join a Clust er

You can use the lu ci server component of Co n g a to cause a node to leave an active cluster by stopping all cluster services on the node. You can also use the lu ci server component of Co n g a to cause a node that has left a cluster to rejoin the cluster.

Causing a node to leave a cluster does not remove the cluster configuration information from that node, and the node still appears in the cluster node display with a status of No t a cl uster

member. For information on deleting the node entirely from the cluster configuration, see

Section 5.3.4, “ Deleting a Member from a Cluster” .

To cause a node to leave a cluster, perform the following steps. This shuts down the cluster software in the node. Making a node leave a cluster prevents the node from automatically joining the cluster when it is rebooted.

⁠Chapt er 5. Managing Red Hat High Availabilit y Add- O n Wit h Conga

2. Select the node you want to leave the cluster by clicking the checkbox for that node.

3. Select the Leave Clu st er function from the menu at the top of the page. This causes a message to appear at the top of the page indicating that the node is being stopped.

4. Refresh the page to see the updated status of the node.

It is also possible to cause more than one node at a time to leave the cluster by selecting all of the nodes to leave the cluster before clicking on Leave Clu st er.

To cause a node to rejoin a cluster, select any nodes you want to have rejoin the cluster by clicking the checkbox for those nodes and selecting Jo in Clu st er. This makes the selected nodes join the cluster, and allows the selected nodes to join the cluster when they are rebooted.

5.3.3. Adding a Member t o a Running Clust er

To add a member to a running cluster, follow the steps in this section.

1. From the cluster-specific page, click No d es along the top of the cluster display. This displays the nodes that constitute the cluster. This is also the default page that appears when you click on the cluster name beneath Man ag e Clu st ers from the menu on the left side of the lu ci Ho meb ase page.

2. Click Ad d . Clicking Ad d causes the display of the Ad d No d es T o C l uster dialog box.

3. Enter the node name in the No d e Ho st n ame text box; enter the ricci password in the

Passwo rd text box. If you are using a different port for the ricci agent than the default of

11111, change this parameter to the port you are using.

4. Check the En ab le Sh ared St o rag e Su p p o rt checkbox if clustered storage is required to download the packages that support clustered storage and enable clustered LVM; you should select this only when you have access to the Resilient Storage Add-On or the

Scalable File System Add-On.

5. If you want to add more nodes, click Ad d Ano ther No d e and enter the node name and password for the each additional node.

6. Click Ad d No d es. Clicking Ad d No d es causes the following actions: a. If you have selected Do wn lo ad Packag es, the cluster software packages are downloaded onto the nodes.

b. Cluster software is installed onto the nodes (or it is verified that the appropriate software packages are installed).

c. The cluster configuration file is updated and propagated to each node in the cluster

— including the added node.

d. The added node joins the cluster.

The No d es page appears with a message indicating that the node is being added to the cluster. Refresh the page to update the status.

Clust er Administ rat ion

7. When the process of adding a node is complete, click on the node name for the newly-added

node to configure fencing for this node, as described in Section 4.6, “ Configuring Fence

Devices” .

5.3.4 . Delet ing a Member from a Clust er

To delete a member from an existing cluster that is currently in operation, follow the steps in this section. Note that nodes must be stopped before being deleted unless you are deleting all of the nodes in the cluster at once.

Note

To allow services running on a node to fail over when the node is deleted, skip the next step.

2. Disable or relocate each service that is running on the node to be deleted. For information on disabling and relocating services, see

Section 5.5, “ Managing High-Availability Services” .

3. Select the node or nodes to delete.

4. Click Delet e. The No d es page indicates that the node is being removed. Refresh the page to see the current status.

Important

Removing a cluster node from the cluster is a destructive operation that cannot be undone.

5.4 . St art ing, St opping, Rest art ing, and Delet ing Clust ers

You can start, stop, and restart a cluster by performing these actions on the individual nodes in the cluster. From the cluster-specific page, click on No d es along the top of the cluster display. This displays the nodes that constitute the cluster.

The start and restart operations for cluster nodes or a whole cluster allow you to create short cluster service outages if a cluster service needs to be moved to another cluster member because it running on a node that is being stopped or restarted.

To stop a cluster, perform the following steps. This shuts down the cluster software in the nodes, but does not remove the cluster configuration information from the nodes and the nodes still appear in the cluster node display with a status of No t a cl uster member.

1. Select all of the nodes in the cluster by clicking on the checkbox next to each node.

2. Select the Leave Clu st er function from the menu at the top of the page. This causes a message to appear at the top of the page indicating that each node is being stopped.

3. Refresh the page to see the updated status of the nodes.

⁠Chapt er 5. Managing Red Hat High Availabilit y Add- O n Wit h Conga

To start a cluster, perform the following steps:

1. Select all of the nodes in the cluster by clicking on the checkbox next to each node.

2. Select the Jo in Clu st er function from the menu at the top of the page.

3. Refresh the page to see the updated status of the nodes.

To restart a running cluster, first stop all of the nodes in cluster, then start all of the nodes in the cluster, as described above.

To delete a cluster entirely, perform the following steps. This causes all cluster services to stop and removes the cluster configuration information from the nodes themselves as well as removing them from the cluster display. If you later try to add an existing cluster using any of the nodes you have deleted, lu ci will indicate that the node is not a member of any cluster.

Important

Deleting a cluster is a destructive operation that cannot be undone. To restore a cluster after you have deleted it requires that you recreate and redefine the cluster from scratch.

1. Select all of the nodes in the cluster by clicking on the checkbox next to each node.

2. Select the Delet e function from the menu at the top of the page.

If you wish to remove a cluster from the lu ci interface without stopping any of the cluster services or changing the cluster membership, you can use the Remo ve option on the Man ag e Clu st ers page, as described in

Section 5.2, “ Removing a Cluster from the luci Interface”

5.5. Managing High-Availabilit y Services

In addition to adding and modifying a service, as described in Section 4.10, “ Adding a Cluster

Service to the Cluster” , you can perform the following management functions for high-availability

services through the lu ci server component of Co n g a:

Start a service

Restart a service

Disable a service

Delete a service

Relocate a service

From the cluster-specific page, you can manage services for that cluster by clicking on Service

G ro u p s along the top of the cluster display. This displays the services that have been configured for that cluster.

St art in g a service — To start any services that are not currently running, select any services you want to start by clicking the checkbox for that service and clicking St art .

Rest art in g a service — To restart any services that are currently running, select any services you want to restart by clicking the checkbox for that service and clicking Rest art .

Disab lin g a service — To disable any service that is currently running, select any services you want to disable by clicking the checkbox for that service and clicking Disab le.

Clust er Administ rat ion

Delet in g a service — To delete any services that are not currently running, select any services you want to disable by clicking the checkbox for that service and clicking Delet e.

Relo cat in g a service — To relocate a running service, click on the name of the service in the services display. This causes the services configuration page for the service to be displayed, with a display indicating on which node the service is currently running.

From the Start o n no d e. . . drop-down box, select the node on which you want to relocate the service, and click on the St art icon. A message appears at the top of the screen indicating that the service is being started. You may need to refresh the screen to see the new display indicating that the service is running on the node you have selected.

Note

If the running service you have selected is a vm service, the drop-down box will show a

mi g rate option instead of a rel o cate option.

Note

You can also start, restart, disable or delete an individual service by clicking on the name of the service on the Services page. This displays the service configuration page. At the top right corner of the service configuration page are the same icons for St art , Rest art , Disab le, and Delet e.

5.6. Backing Up and Rest oring t he luci Configurat ion

As of the Red Hat Enterprise Linux 6.2 release, you can use the following procedure to make a backup of the lu ci database, which is stored in the /var/l i b/l uci /d ata/l uci . d b file. This is not the cluster configuration itself, which is stored in the cl uster. co nf file. Instead, it contains the list of users and clusters and related properties that lu ci maintains. By default, the backup this procedure creates will be written to the same directory as the l uci . d b file.

1. Execute servi ce l uci sto p.

2. Execute servi ce l uci backup-d b.

Optionally, you can specify a file name as a parameter for the backup-d b command, which will write the lu ci database to that file. For example, to write the lu ci database to the file

/ro o t/l uci . d b. backup, you can execute the command servi ce l uci backup-d b

/ro o t/l uci . d b. backup. Note, however, that backup files that are written to places other than /var/l i b/l uci /d ata/ (for backups whose filenames you specify when using

servi ce l uci backup-d b) will not show up in the output of the l i st-backups command.

3. Execute servi ce l uci start.

Use the following procedure to restore a lu ci database.

1. Execute servi ce l uci sto p.

2. Execute servi ce l uci l i st-backups and note the file name to restore.

⁠Chapt er 5. Managing Red Hat High Availabilit y Add- O n Wit h Conga

3. Execute servi ce l uci resto re-d b /var/l i b/l uci /d ata/lucibackupfile where

lucibackupfile is the backup file to restore.

For example, the following command restores the lu ci configuration information that was stored in the backup file l uci -backup20 110 9 230 6 2526 . d b: service luci restore-db /var/lib/luci/data/lucibackup20110923062526.db

4. Execute servi ce l uci start.

If you need to restore a lu ci database but you have lost the ho st. pem file from the machine on which you created the backup because of a complete reinstallation, for example, you will need to add your clusters back to lu ci manually in order to re-authenticate the cluster nodes.

Use the following procedure to restore a lu ci database onto a different machine than the one on which the backup was created. Note that in addition to restoring the database itself, you also need to copy the SSL certificate file to ensure that lu ci has been authenticated to the ricci nodes. In this example, the backup is created on the machine l uci 1 and the backup is restored on the machine

l uci 2.

1. Execute the following sequence of commands to create a lu ci backup on l uci 1 and copy both the SSL certificate file and the lu ci backup onto l uci 2.

[root@ luci1 ~]# servi ce l uci sto p

[root@ luci1 ~]# servi ce l uci backup-d b

[root@ luci1 ~]# servi ce l uci l i st-backups

/var/lib/luci/data/luci-backup20120504134051.db

[root@ luci1 ~]# scp /var/l i b/l uci /certs/ho st. pem

/var/l i b/l uci /d ata/l uci -backup20 120 50 4 134 0 51. d b ro o t@ l uci 2:

2. On the l uci 2 machine, ensure that lu ci has been installed and is not running. Install the package, if is not already installed.

3. Execute the following sequence of commands to ensure that the authentications are in place and to restore the lu ci database from l uci 1 onto l uci 2.

[root@ luci2 ~]# cp ho st. pem /var/l i b/l uci /certs/

[root@ luci2 ~]# cho wn l uci : /var/l i b/l uci /certs/ho st. pem

[root@ luci2 ~]# /etc/i ni t. d /l uci resto re-d b ~ /l uci -

backup20 120 50 4 134 0 51. d b

[root@ luci2 ~]# shred -u ~ /ho st. pem ~ /l uci -

backup20 120 50 4 134 0 51. d b

[root@ luci2 ~]# servi ce l uci start

Clust er Administ rat ion

Chapter 6. Configuring Red Hat High Availability Add-On With the ccs Command

As of the Red Hat Enterprise Linux 6.1 release and later, the Red Hat High Availability Add-On provides support for the ccs cluster configuration command. The ccs command allows an administrator to create, modify and view the cl uster. co nf cluster configuration file. You can use the ccs command to configure a cluster configuration file on a local file system or on a remote node.

Using the ccs command, an administrator can also start and stop the cluster services on one or all of the nodes in a configured cluster.

This chapter describes how to configure the Red Hat High Availability Add-On cluster configuration file using the ccs command. For information on using the ccs command to manage a running

cluster, see Chapter 7, Managing Red Hat High Availability Add-On With ccs .

This chapter consists of the following sections:

Section 6.1, “ Operational Overview”

Section 6.2, “ Configuration Tasks”

Section 6.3, “ Starting ricci”

Section 6.4, “ Creating and Modifying a Cluster”

Section 6.5, “ Configuring Fence Devices”

Section 6.7, “ Configuring Fencing for Cluster Members”

Section 6.8, “ Configuring a Failover Domain”

Section 6.9, “ Configuring Global Cluster Resources”

Section 6.10, “ Adding a Cluster Service to the Cluster”

Section 6.13, “ Configuring a Quorum Disk”

Section 6.14, “ Miscellaneous Cluster Configuration”

Section 6.15, “ Propagating the Configuration File to the Cluster Nodes”

Important

Make sure that your deployment of High Availability Add-On meets your needs and can be supported. Consult with an authorized Red Hat representative to verify your configuration prior to deployment. In addition, allow time for a configuration burn-in period to test failure modes.

⁠Chapt er 6 . Configuring Red Hat High Availabilit y Add- O n Wit h t he ccs Command

Important

This chapter references commonly used cl uster. co nf elements and attributes. For a comprehensive list and description of cl uster. co nf elements and attributes, refer to the cluster schema at /usr/share/cl uster/cl uster. rng , and the annotated schema at

/usr/share/d o c/cman-X. Y . ZZ/cl uster_co nf. html (for example

/usr/share/d o c/cman-3. 0 . 12/cl uster_co nf. html ).

6.1. Operat ional Overview

This section describes the following general operational aspects of using the ccs command to configure a cluster:

Section 6.1.1, “ Creating the Cluster Configuration File on a Local System”

Section 6.1.2, “ Viewing the Current Cluster Configuration”

Section 6.1.3, “ Specifying ricci Passwords with the ccs Command”

Section 6.1.4, “ Modifying Cluster Configuration Components”

6.1.1. Creat ing t he Clust er Configurat ion File on a Local Syst em

Using the ccs command, you can create a cluster configuration file on a cluster node, or you can create a cluster configuration file on a local file system and then send that file to a host in a cluster.

This allows you to work on a file from a local machine, where you can maintain it under version control or otherwise tag the file according to your needs. Using the ccs command does not require root privilege.

When you create and edit a cluster configuration file on a cluster node with the ccs command, you use the -h option to specify the name of the host. This creates and edits the

/etc/cl uster/cl uster. co nf file on the host: ccs -h host [options]

To create and edit a cluster configuration file on a local system, use the -f option of the ccs command to specify the name of the configuration file when you perform a cluster operation. You can name this file anything you want.

ccs -f file [options]

As of Red Hat Enterprise Linux 6.6, if you do not specify the -h or the -f parameter of the ccs command, the ccs will attempt to connect to the localhost. This is the equivalent of specifying -h

localhost.

After you have created the file locally you can send it to a cluster node using the --setco nf option of the ccs command. On a host machine in a cluster, the file you send will be named cl uster. co nf and it will be placed in the /etc/cl uster directory.

ccs -h host -f file --setconf

For information on using the --setco nf option of the ccs command, see Section 6.15,

“ Propagating the Configuration File to the Cluster Nodes” .

Clust er Administ rat ion

6.1.2. Viewing t he Current Clust er Configurat ion

If at any time during the creation of a cluster configuration file you want to print the current file, use the following command, specifying a node in the cluster as the host: ccs -h host --getconf

If you are creating your cluster configuration file on a local system you can specify the -f option

instead of the -h option, as described in Section 6.1.1, “ Creating the Cluster Configuration File on a

Local System” .

6.1.3. Specifying ricci Passwords wit h t he ccs Command

Executing ccs commands that distribute copies of the cl uster. co nf file to the nodes of a cluster

requires that ricci be installed and running on the cluster nodes, as described in Section 3.13,

“ Considerations for ri cci ” . Using ricci requires a password the first time you interact with ricci

from any specific machine.

If you have not entered a password for an instance of ricci on a particular machine from the machine you are using, you will be prompted for that password when the ccs command requires it.

Alternately, you can use the -p option to specify a ricci password on the command line.

ccs -h host -p password --sync --activate

When you propagate the cl uster. co nf file to all of the nodes in the cluster with the --sync option of the ccs command and you specify a ricci password for the command, the ccs command will use that password for each node in the cluster. If you need to set different passwords for ricci on individual nodes, you can use the --setco nf option with the -p option to distribute the configuration file to one node at a time.

6.1.4 . Modifying Clust er Configurat ion Component s

You use the ccs command to configure cluster components and their attributes in the cluster configuration file. After you have added a cluster component to the file, in order to modify the attributes of that component you must remove the component you have defined and add the component again, with the modified attributes. Information on how to do this with each component is provided in the individual sections of this chapter.

The attributes of the cman cluster component provide an exception to this procedure for modifying cluster components. To modify these attributes, you execute the --setcman option of the ccs command, specifying the new attributes. Note that specifying this option resets all values that you do

not explicitly specify to their default values, as described in Section 6.1.5, “ Commands that Overwrite

Previous Settings” .

6.1.5. Commands t hat Overwrit e Previous Set t ings

There are several options of the ccs command that implement overwriting semantics when setting properties. This means that you can issue the ccs command with one of these options without specifying any settings, and it will reset all settings to their default values. These options are as follows:

--setto tem

--setd l m

⁠Chapt er 6 . Configuring Red Hat High Availabilit y Add- O n Wit h t he ccs Command

--setrm

--setcman

--setmul ti cast

--setal tmul ti cast

--setfenced aemo n

--setl o g g i ng

--setq uo rumd

For example, to reset all of the fence deamon properties, you can run the following command.

# ccs -h ho stname --setfenced aemo n

Note, however, that if you use one of these commands to reset a property, then the other properties of the command will be reset to their default values. For example, you can use the following command to set the po st_fai l _d el ay property to 5:

# ccs -h ho stname --setfenced aemo n po st_fai l _d el ay= 5

If, after running that command, you run the following command to reset the po st_jo i n_d el ay property to 10, the po st_fai l _d el ay property will be restored to its default value:

# ccs -h ho stname --setfenced aemo n po st_jo i n_d el ay= 10

To reset both the po st_fai l _d el ay and the po st_jo i n_d el ay properties, you indicate them both on the same command, as in the following example:

# ccs -h ho stname --setfenced aemo n po st_fai l _d el ay= 5

po st_jo i n_d el ay= 10

For more information on configuring fence devices, refer to Section 6.5, “ Configuring Fence

Devices” .

6.1.6. Configurat ion Validat ion

When you use the ccs command to create and edit the cluster configuration file, the configuration is automatically validated according to the cluster schema. As of the Red Hat Enterprise Linux 6.3

release, the ccs command validates the configuration according to the cluster schema at

/usr/share/cl uster/cl uster. rng on the node that you specify with the -h option. Previously the ccs command always used the cluster schema that was packaged with the ccs command itself,

/usr/share/ccs/cl uster. rng on the local system. When you use the -f option to specify the local system, the ccs command still uses the cluster schema /usr/share/ccs/cl uster. rng that was packaged with the ccs command itself on that system.

6.2. Configurat ion T asks

Configuring Red Hat High Availability Add-On software with the ccs consists of the following steps:

1. Ensuring that ricci is running on all nodes in the cluster. Refer to Section 6.3, “ Starting

ricci” .

Clust er Administ rat ion

2. Creating a cluster. Refer to

Section 6.4, “ Creating and Modifying a Cluster”

3. Configuring fence devices. Refer to Section 6.5, “ Configuring Fence Devices” .

4. Configuring fencing for cluster members. Refer to Section 6.7, “ Configuring Fencing for

Cluster Members” .

5. Creating failover domains. Refer to

Section 6.8, “ Configuring a Failover Domain”

6. Creating resources. Refer to Section 6.9, “ Configuring Global Cluster Resources” .

7. Creating cluster services. Refer to

Section 6.10, “ Adding a Cluster Service to the Cluster” .

8. Configuring a quorum disk, if necessary. Refer to

Section 6.13, “ Configuring a Quorum Disk”

9. Configuring global cluster properties. Refer to Section 6.14, “ Miscellaneous Cluster

10. Propagating the cluster configuration file to all of the cluster nodes. Refer to Section 6.15,

“ Propagating the Configuration File to the Cluster Nodes” .

6.3. St art ing ricci

In order to create and distribute cluster configuration files on the nodes of the cluster, the ricci service must be running on each node. Before starting ricci, you should ensure that you have configured your system as follows:

1. The IP ports on your cluster nodes should be enabled for ricci. For information on enabling

IP ports on cluster nodes, see Section 3.3.1, “ Enabling IP Ports on Cluster Nodes”

2. The ricci service is installed on all nodes in the cluster and assigned a ricci password, as

described in Section 3.13, “ Considerations for ri cci ” .

After ricci has been installed and configured on each node, start the ricci service on each node:

# servi ce ri cci start

Starting ricci: [ OK ]

6.4 . Creat ing and Modifying a Clust er

This section describes how to create, modify, and delete a skeleton cluster configuration with the ccs command without fencing, failover domains, and HA services. Subsequent sections describe how to configure those parts of the configuration.

To create a skeleton cluster configuration file, first create and name the cluster and then add the nodes to the cluster, as in the following procedure:

1. Create a cluster configuration file on one of the nodes in the cluster by executing the ccs command using the -h parameter to specify the node on which to create the file and the

createcl uster option to specify a name for the cluster: ccs -h host --createcluster clustername

For example, the following command creates a configuration file on no d e-

0 1. exampl e. co m named mycl uster:

⁠Chapt er 6 . Configuring Red Hat High Availabilit y Add- O n Wit h t he ccs Command

ccs -h node-01.example.com --createcluster mycluster

The cluster name cannot exceed 15 characters.

If a cl uster. co nf file already exists on the host that you specify, executing this command will replace that existing file.

If you want to create a cluster configuration file on your local system you can specify the -f option instead of the -h option. For information on creating the file locally, refer to

Section 6.1.1, “ Creating the Cluster Configuration File on a Local System”

2. To configure the nodes that the cluster contains, execute the following command for each node in the cluster. A node name can be up to 255 bytes in length.

ccs -h host --addnode node

For example, the following three commands add the nodes no d e-0 1. exampl e. co m,

no d e-0 2. exampl e. co m, and no d e-0 3. exampl e. co m to the configuration file on no d e-

0 1. exampl e. co m: ccs -h node-01.example.com --addnode node-01.example.com

ccs -h node-01.example.com --addnode node-02.example.com

ccs -h node-01.example.com --addnode node-03.example.com

To view a list of the nodes that have been configured for a cluster, execute the following command: ccs -h host --lsnodes

Example 6.1, “ cl uster. co nf File After Adding Three Nodes” shows a cl uster. co nf

configuration file after you have created the cluster mycl uster that contains the nodes

no d e-0 1. exampl e. co m, no d e-0 2. exampl e. co m, and no d e-0 3. exampl e. co m.

Examp le 6 .1. cl uster. co nf File Af t er Ad d in g T h ree No d es

<fence>

</fence>

</clusternode>

<fence>

</fence>

</clusternode>

<fence>

</fence>

</clusternode>

</clusternodes>

Clust er Administ rat ion

</fencedevices>

<rm>

</rm>

</cluster>

When you add a node to the cluster, you can specify the number of votes the node contributes to determine whether there is a quorum. To set the number of votes for a cluster node, use the following command: ccs -h host --addnode host --votes votes

When you add a node, the ccs assigns the node a unique integer that is used as the node identifier. If you want to specify the node identifier manually when creating a node, use the following command: ccs -h host --addnode host --nodeid nodeid

To remove a node from a cluster, execute the following command: ccs -h host --rmnode node

When you have finished configuring all of the components of your cluster, you will need to sync the

cluster configuration file to all of the nodes, as described in Section 6.15, “ Propagating the

refer to Section 6.7, “ Configuring Fencing for Cluster Members”

6.5. Configuring Fence Devices

Configuring fence devices consists of creating, updating, and deleting fence devices for the cluster.

You must create and name the fence devices in a cluster before you can configure fencing for the nodes in the cluster. For information on configuring fencing for the individual nodes in the cluster,

Before configuring your fence devices, you may want to modify some of the fence daemon properties for your system from the default values. The values you configure for the fence daemon are general values for the cluster. The general fencing properties for the cluster you may want to modify are summarized as follows:

The po st_fai l _d el ay attribute is the number of seconds the fence daemon (fenced ) waits before fencing a node (a member of the fence domain) after the node has failed. The

po st_fai l _d el ay default value is 0 . Its value may be varied to suit cluster and network performance.

The po st-jo i n_d el ay attribute is the number of seconds the fence daemon (fenced ) waits before fencing a node after the node joins the fence domain. The po st_jo i n_d el ay default value is 6 . A typical setting for po st_jo i n_d el ay is between 20 and 30 seconds, but can vary according to cluster and network performance.

You reset the values of the po st_fai l _d el ay and po st_jo i n_d el ay attributes with the --

setfenced aemo n option of the ccs command. Note, however, that executing the ccs --

setfenced aemo n command overwrites all existing fence daemon properties that have been explicitly set and restores them to their default values.

⁠Chapt er 6 . Configuring Red Hat High Availabilit y Add- O n Wit h t he ccs Command

For example, to configure a value for the po st_fai l _d el ay attribute, execute the following command. This command will overwrite the values of all other existing fence daemon properties that you have set with this command and restore them to their default values.

ccs -h host --setfencedaemon post_fail_delay=value

To configure a value for the po st_jo i n_d el ay attribute, execute the following command. This command will overwrite the values of all other existing fence daemon properties that you have set with this command and restore them to their default values.

ccs -h host --setfencedaemon post_join_delay=value

To configure a value for both the the po st_jo i n_d el ay attribute and the po st_fai l _d el ay attribute, execute the following command: ccs -h host --setfencedaemon post_fail_delay=value post_join_delay=value

Note

For more information about the po st_jo i n_d el ay and po st_fai l _d el ay attributes as well as the additional fence daemon properties you can modify, refer to the fenced(8) man page and refer to the cluster schema at /usr/share/cl uster/cl uster. rng , and the annotated schema at /usr/share/d o c/cman-X. Y . ZZ/cl uster_co nf. html .

To configure a fence device for a cluster, execute the following command: ccs -h host --addfencedev devicename [fencedeviceoptions]

For example, to configure an APC fence device in the configuration file on the cluster node no d e1 named my_apc with an IP address of apc_i p_exampl e, a login of l o g i n_exampl e, and a password of passwo rd _exampl e, execute the following command: ccs -h node1 --addfencedev myfence agent=fence_apc ipaddr=apc_ip_example login=login_example passwd=password_example

The following example shows the fenced evi ces section of the cl uster. co nf configuration file after you have added this APC fence device:

</fencedevices>

When configuring fence devices for a cluster, you may find it useful to see a listing of available devices for your cluster and the options available for each device. You may also find it useful to see a listing of fence devices currently configured for your cluster. For information on using the ccs command to print a list of available fence devices and options or to print a list of fence devices

currently configured for your cluster, refer to Section 6.6, “ Listing Fence Devices and Fence Device

Options” .

To remove a fence device from your cluster configuration, execute the following command:

Clust er Administ rat ion

ccs -h host --rmfencedev fence_device_name

For example, to remove a fence device that you have named myfence from the cluster configuration file on cluster node no d e1, execute the following command: ccs -h node1 --rmfencedev myfence

If you need to modify the attributes of a fence device you have already configured, you must first remove that fence device then add it again with the modified attributes.

Note that when you have finished configuring all of the components of your cluster, you will need to

sync the cluster configuration file to all of the nodes, as described in Section 6.15, “ Propagating the

Section 6.7.1, “ Configuring a Single Power-Based Fence Device for a Node”

6.6. List ing Fence Devices and Fence Device Opt ions

You can use the ccs command to print a list of available fence devices and to print a list of options for each available fence type. You can also use the ccs command to print a list of fence devices currently configured for your cluster.

To print a list of fence devices currently available for your cluster, execute the following command: ccs -h host --lsfenceopts

For example, the following command lists the fence devices available on the cluster node no d e1, showing sample output.

[root@ ask-03 ~]# ccs -h no d e1 --l sfenceo pts fence_rps10 - RPS10 Serial Switch fence_vixel - No description available fence_egenera - No description available fence_xcat - No description available fence_na - Node Assassin fence_apc - Fence agent for APC over telnet/ssh fence_apc_snmp - Fence agent for APC over SNMP fence_bladecenter - Fence agent for IBM BladeCenter fence_bladecenter_snmp - Fence agent for IBM BladeCenter over SNMP fence_cisco_mds - Fence agent for Cisco MDS fence_cisco_ucs - Fence agent for Cisco UCS fence_drac5 - Fence agent for Dell DRAC CMC/5 fence_eps - Fence agent for ePowerSwitch fence_ibmblade - Fence agent for IBM BladeCenter over SNMP fence_ifmib - Fence agent for IF MIB fence_ilo - Fence agent for HP iLO fence_ilo_mp - Fence agent for HP iLO MP fence_intelmodular - Fence agent for Intel Modular fence_ipmilan - Fence agent for IPMI over LAN fence_kdump - Fence agent for use with kdump fence_rhevm - Fence agent for RHEV-M REST API fence_rsa - Fence agent for IBM RSA fence_sanbox2 - Fence agent for QLogic SANBox2 FC switches fence_scsi - fence agent for SCSI-3 persistent reservations fence_virsh - Fence agent for virsh fence_virt - Fence agent for virtual machines

⁠Chapt er 6 . Configuring Red Hat High Availabilit y Add- O n Wit h t he ccs Command

fence_vmware - Fence agent for VMware fence_vmware_soap - Fence agent for VMware over SOAP API fence_wti - Fence agent for WTI fence_xvm - Fence agent for virtual machines

To print a list of the options you can specify for a particular fence type, execute the following command: ccs -h host --lsfenceopts fence_type

For example, the following command lists the fence options for the fence_wti fence agent.

[root@ ask-03 ~]# ccs -h no d e1 --l sfenceo pts fence_wti fence_wti - Fence agent for WTI

Required Options:

Optional Options:

option: No description available

action: Fencing Action

ipaddr: IP Address or Hostname

passwd: Login password or passphrase

passwd_script: Script to retrieve password

cmd_prompt: Force command prompt

secure: SSH connection

identity_file: Identity file for ssh

port: Physical plug number or name of virtual machine

inet4_only: Forces agent to use IPv4 addresses only

inet6_only: Forces agent to use IPv6 addresses only

ipport: TCP port to use for connection with device

verbose: Verbose mode

debug: Write debug information to given file

version: Display version information and exit

help: Display help and exit

separator: Separator for CSV created by operation list

power_timeout: Test X seconds for status change after ON/OFF

shell_timeout: Wait X seconds for cmd prompt after issuing command

login_timeout: Wait X seconds for cmd prompt after login

power_wait: Wait X seconds after issuing ON/OFF

delay: Wait X seconds before fencing is started

retry_on: Count of attempts to retry power on

To print a list of fence devices currently configured for your cluster, execute the following command: ccs -h host --lsfencedev

6.7. Configuring Fencing for Clust er Members

Once you have completed the initial steps of creating a cluster and creating fence devices, you need to configure fencing for the cluster nodes. To configure fencing for the nodes after creating a new cluster and configuring the fencing devices for the cluster, follow the steps in this section. Note that you must configure fencing for each node in the cluster.

Clust er Administ rat ion

Note

It is recommended that you configure multiple fencing mechanisms for each node. A fencing device can fail due to network split, a power outage, or a problem in the fencing device itself.

Configuring multiple fencing mechanisms can reduce the likelihood that the failure of a fencing device will have fatal results.

This section documents the following procedures:

Section 6.7.2, “ Configuring a Single Storage-Based Fence Device for a Node”

Section 6.7.3, “ Configuring a Backup Fence Device”

Section 6.7.4, “ Configuring a Node with Redundant Power”

Section 6.7.6, “ Removing Fence Methods and Fence Instances”

6.7.1. Configuring a Single Power-Based Fence Device for a Node

Use the following procedure to configure a node with a single power-based fence device. The fence device is named my_apc, which uses the fence_apc fencing agent. In this example, the device named my_apc was previously configured with the --ad d fenced ev option, as described in

Section 6.5, “ Configuring Fence Devices”

1. Add a fence method for the node, providing a name for the fence method.

ccs -h host --addmethod method node

For example, to configure a fence method named AP C for the node no d e-0 1. exampl e. co m in the configuration file on the cluster node no d e-0 1. exampl e. co m, execute the following command: ccs -h node01.example.com --addmethod APC node01.example.com

2. Add a fence instance for the method. You must specify the fence device to use for the node, the node this instance applies to, the name of the method, and any options for this method that are specific to this node: ccs -h host --addfenceinst fencedevicename node method [options]

For example, to configure a fence instance in the configuration file on the cluster node no d e-

0 1. exampl e. co m that uses power port 1 on the APC switch for the fence device named

my_apc to fence cluster node no d e-0 1. exampl e. co m using the method named AP C , execute the following command: ccs -h node01.example.com --addfenceinst my_apc node01.example.com

APC port=1

You will need to add a fence method for each node in the cluster. The following commands configure a fence method for each node with the method name AP C . The device for the fence method specifies

my_apc as the device name, which is a device previously configured with the --ad d fenced ev

⁠Chapt er 6 . Configuring Red Hat High Availabilit y Add- O n Wit h t he ccs Command

option, as described in Section 6.5, “ Configuring Fence Devices” . Each node is configured with a

unique APC switch power port number: The port number for no d e-0 1. exampl e. co m is 1, the port number for no d e-0 2. exampl e. co m is 2, and the port number for no d e-0 3. exampl e. co m is 3.

ccs -h node01.example.com --addmethod APC node01.example.com

ccs -h node01.example.com --addmethod APC node02.example.com

ccs -h node01.example.com --addmethod APC node03.example.com

ccs -h node01.example.com --addfenceinst my_apc node01.example.com APC port=1 ccs -h node01.example.com --addfenceinst my_apc node02.example.com APC port=2 ccs -h node01.example.com --addfenceinst my_apc node03.example.com APC port=3

Example 6.2, “ cl uster. co nf After Adding Power-Based Fence Methods ” shows a cl uster. co nf

configuration file after you have added these fencing methods and instances to each node in the cluster.

Examp le 6 .2. cl uster. co nf Af t er Ad d in g Po wer- Based Fen ce Met h o d s

<fence>

</method>

</fence>

</clusternode>

<fence>

</method>

</fence>

</clusternode>

<fence>

</method>

</fence>

</clusternode>

</clusternodes>

</fencedevices>

<rm>

</rm>

</cluster>

Note that when you have finished configuring all of the components of your cluster, you will need to

Clust er Administ rat ion

sync the cluster configuration file to all of the nodes, as described in Section 6.15, “ Propagating the

option, as described in Section 6.5, “ Configuring Fence Devices” . Each node is configured with a

6.7.2. Configuring a Single St orage-Based Fence Device for a Node

When using non-power fencing methods (that is, SAN/storage fencing) to fence a node, you must configure unfencing for the fence device. This ensures that a fenced node is not re-enabled until the node has been rebooted. When you configure a device that requires unfencing, the cluster must first be stopped and the full configuration including devices and unfencing must be added before the cluster is started.

When you configure unfencing for a node, you specify a device that mirrors the corresponding fence device you have configured for the node with the notable addition of the explicit action of o n or

enabl e.

For more information about unfencing a node, refer to the fence_no d e(8) man page.

Use the following procedure to configure a node with a single storage-based fence device that uses a fence device named sanswi tch1, which uses the fence_sanbo x2 fencing agent.

1. Add a fence method for the node, providing a name for the fence method.

ccs -h host --addmethod method node

For example, to configure a fence method named SAN for the node no d e-0 1. exampl e. co m in the configuration file on the cluster node no d e-0 1. exampl e. co m, execute the following command: ccs -h node01.example.com --addmethod SAN node01.example.com

For example, to configure a fence instance in the configuration file on the cluster node no d e-

0 1. exampl e. co m that uses the SAN switch power port 11 on the fence device named

sanswi tch1 to fence cluster node no d e-0 1. exampl e. co m using the method named SAN, execute the following command: ccs -h node01.example.com --addfenceinst sanswitch1 node01.example.com SAN port=11

3. To configure unfencing for the storage-based fence device on this node, execute the following command: ccs -h host --addunfence fencedevicename node action=on|off

You will need to add a fence method for each node in the cluster. The following commands configure a fence method for each node with the method name SAN. The device for the fence method specifies

sanswi tch as the device name, which is a device previously configured with the --addfencedev

⁠Chapt er 6 . Configuring Red Hat High Availabilit y Add- O n Wit h t he ccs Command

unique SAN physical port number: The port number for no d e-0 1. exampl e. co m is 11, the port number for no d e-0 2. exampl e. co m is 12, and the port number for no d e-0 3. exampl e. co m is

13.

ccs -h node01.example.com --addmethod SAN node01.example.com

ccs -h node01.example.com --addmethod SAN node02.example.com

ccs -h node01.example.com --addmethod SAN node03.example.com

ccs -h node01.example.com --addfenceinst sanswitch1 node01.example.com

SAN port=11 ccs -h node01.example.com --addfenceinst sanswitch1 node02.example.com

SAN port=12 ccs -h node01.example.com --addfenceinst sanswitch1 node03.example.com

SAN port=13 ccs -h node01.example.com --addunfence sanswitch1 node01.example.com port=11 action=on ccs -h node01.example.com --addunfence sanswitch1 node02.example.com port=12 action=on ccs -h node01.example.com --addunfence sanswitch1 node03.example.com port=13 action=on

Example 6.3, “ cl uster. co nf After Adding Storage-Based Fence Methods ” shows a

cl uster. co nf configuration file after you have added fencing methods, fencing instances, and unfencing to each node in the cluster.

Examp le 6 .3. cl uster. co nf Af t er Ad d in g St o rag e- Based Fen ce Met h o d s

<fence>

</method>

</fence>

</unfence>

</clusternode>

<fence>

</method>

</fence>

</unfence>

</clusternode>

<fence>

</method>

</fence>

Clust er Administ rat ion

</unfence>

</clusternode>

</clusternodes>

</fencedevices>

<rm>

</rm>

</cluster>

Note that when you have finished configuring all of the components of your cluster, you will need to

sync the cluster configuration file to all of the nodes, as described in Section 6.15, “ Propagating the

Example 6.4, “ cl uster. co nf After Adding Backup Fence Methods ”

6.7.3. Configuring a Backup Fence Device

You can define multiple fencing methods for a node. If fencing fails using the first method, the system will attempt to fence the node using the second method, followed by any additional methods you have configured. To configure a backup fencing method for a node, you configure two methods for a node, configuring a fence instance for each method.

Note

The order in which the system will use the fencing methods you have configured follows their order in the cluster configuration file. The first method you configure with the ccs command is the primary fencing method, and the second method you configure is the backup fencing method. To change the order, you can remove the primary fencing method from the configuration file, then add that method back.

Note that at any time you can print a list of fence methods and instances currently configured for a node by executing the following command. If you do not specify a node, this command will list the fence methods and instances currently configured for all nodes.

ccs -h host --lsfenceinst [node]

Use the following procedure to configure a node with a primary fencing method that uses a fence device named my_apc, which uses the fence_apc fencing agent, and a backup fencing device that uses a fence device named sanswi tch1, which uses the fence_sanbo x2 fencing agent. Since the

sanswi tch1 device is a storage-based fencing agent, you will need to configure unfencing for that device as well.

1. Add a primary fence method for the node, providing a name for the fence method.

ccs -h host --addmethod method node

For example, to configure a fence method named AP C as the primary method for the node

no d e-0 1. exampl e. co m in the configuration file on the cluster node no d e-

0 1. exampl e. co m, execute the following command:

⁠Chapt er 6 . Configuring Red Hat High Availabilit y Add- O n Wit h t he ccs Command

ccs -h node01.example.com --addmethod APC node01.example.com

2. Add a fence instance for the primary method. You must specify the fence device to use for the node, the node this instance applies to, the name of the method, and any options for this method that are specific to this node: ccs -h host --addfenceinst fencedevicename node method [options]

For example, to configure a fence instance in the configuration file on the cluster node no d e-

0 1. exampl e. co m that uses the APC switch power port 1 on the fence device named my_apc to fence cluster node no d e-0 1. exampl e. co m using the method named AP C , execute the following command: ccs -h node01.example.com --addfenceinst my_apc node01.example.com

APC port=1

3. Add a backup fence method for the node, providing a name for the fence method.

ccs -h host --addmethod method node

For example, to configure a backup fence method named SAN for the node no d e-

0 1. exampl e. co m in the configuration file on the cluster node no d e-0 1. exampl e. co m, execute the following command: ccs -h node01.example.com --addmethod SAN node01.example.com

4. Add a fence instance for the backup method. You must specify the fence device to use for the node, the node this instance applies to, the name of the method, and any options for this method that are specific to this node: ccs -h host --addfenceinst fencedevicename node method [options]

For example, to configure a fence instance in the configuration file on the cluster node no d e-

0 1. exampl e. co m that uses the SAN switch power port 11 on the fence device named

5. Since the sanswi tch1 device is a storage-based device, you must configure unfencing for this device.

ccs -h node01.example.com --addunfence sanswitch1 node01.example.com port=11 action=on

You can continue to add fencing methods as needed.

This procedure configures a fence device and a backup fence device for one node in the cluster. You will need to configure fencing for the other nodes in the cluster as well.

Clust er Administ rat ion

shows a cl uster. co nf configuration file after you have added a power-based primary fencing method and a storage-based backup fencing method to each node in the cluster.

Examp le 6 .4 . cl uster. co nf Af t er Ad d in g Backu p Fen ce Met h o d s

<fence>

</method>

</method>

</fence>

</unfence

</clusternode>

<fence>

</method>

</method>

</fence>

</unfence

</clusternode>

<fence>

</method>

</method>

</fence>

</unfence

</clusternode>

</clusternodes>

</fencedevices>

<rm>

</rm>

</cluster>

⁠Chapt er 6 . Configuring Red Hat High Availabilit y Add- O n Wit h t he ccs Command

Note that when you have finished configuring all of the components of your cluster, you will need to

sync the cluster configuration file to all of the nodes, as described in Section 6.15, “ Propagating the

Section 6.5, “ Configuring Fence Devices”

Note

The order in which the system will use the fencing methods you have configured follows their order in the cluster configuration file. The first method you configure is the primary fencing method, and the second method you configure is the backup fencing method. To change the order, you can remove the primary fencing method from the configuration file, then add that method back.

6.7.4 . Configuring a Node wit h Redundant Power

If your cluster is configured with redundant power supplies for your nodes, you must be sure to configure fencing so that your nodes fully shut down when they need to be fenced. If you configure each power supply as a separate fence method, each power supply will be fenced separately; the second power supply will allow the system to continue running when the first power supply is fenced and the system will not be fenced at all. To configure a system with dual power supplies, you must configure your fence devices so that both power supplies are shut off and the system is taken completely down. This requires that you configure two instances within a single fencing method, and that for each instance you configure both fence devices with an acti o n attribute of o ff before configuring each of the devices with an acti o n attribute of o n.

To configure fencing for a node with dual power supplies, follow the steps in this section.

1. Before you can configure fencing for a node with redundant power, you must configure each of the power switches as a fence device for the cluster. For information on configuring fence devices, see

To print a list of fence devices currently configured for your cluster, execute the following command: ccs -h host --lsfencedev

2. Add a fence method for the node, providing a name for the fence method.

ccs -h host --addmethod method node

For example, to configure a fence method named AP C -d ual for the node no d e-

0 1. exampl e. co m in the configuration file on the cluster node no d e-0 1. exampl e. co m, execute the following command: ccs -h node01.example.com --addmethod APC-dual node01.example.com

Clust er Administ rat ion

3. Add a fence instance for the first power supply to the fence method. You must specify the fence device to use for the node, the node this instance applies to, the name of the method, and any options for this method that are specific to this node. At this point you configure the

acti o n attribute as o ff.

ccs -h host --addfenceinst fencedevicename node method [options] action=off

For example, to configure a fence instance in the configuration file on the cluster node no d e-

0 1. exampl e. co m that uses the APC switch power port 1 on the fence device named apc1 to fence cluster node no d e-0 1. exampl e. co m using the method named AP C -d ual , and setting the acti o n attribute to o ff, execute the following command: ccs -h node01.example.com --addfenceinst apc1 node01.example.com

APC-dual port=1 action=off

4. Add a fence instance for the second power supply to the fence method. You must specify the fence device to use for the node, the node this instance applies to, the name of the method, and any options for this method that are specific to this node. At this point you configure the

acti o n attribute as o ff for this instance as well: ccs -h host --addfenceinst fencedevicename node method [options] action=off

For example, to configure a second fence instance in the configuration file on the cluster node no d e-0 1. exampl e. co m that uses the APC switch power port 1 on the fence device named apc2 to fence cluster node no d e-0 1. exampl e. co m using the same method as you specified for the first instance named AP C -d ual , and setting the acti o n attribute to o ff, execute the following command: ccs -h node01.example.com --addfenceinst apc2 node01.example.com

APC-dual port=1 action=off

5. At this point, add another fence instance for the first power supply to the fence method, configuring the acti o n attribute as o n. You must specify the fence device to use for the node, the node this instance applies to, the name of the method, and any options for this method that are specific to this node, and specifying the acti o n attribute as o n: ccs -h host --addfenceinst fencedevicename node method [options] action=on

For example, to configure a fence instance in the configuration file on the cluster node no d e-

0 1. exampl e. co m that uses the APC switch power port 1 on the fence device named apc1 to fence cluster node no d e-0 1. exampl e. co m using the method named AP C -d ual , and setting the acti o n attribute to o n, execute the following command: ccs -h node01.example.com --addfenceinst apc1 node01.example.com

APC-dual port=1 action=on

6. Add another fence instance for second power supply to the fence method, specifying the

acti o n attribute as o n for this instance. You must specify the fence device to use for the node, the node this instance applies to, the name of the method, and any options for this method that are specific to this node as well as the acti o n attribute of o n.

⁠Chapt er 6 . Configuring Red Hat High Availabilit y Add- O n Wit h t he ccs Command

ccs -h host --addfenceinst fencedevicename node method [options] action=on

For example, to configure a second fence instance in the configuration file on the cluster node no d e-0 1. exampl e. co m that uses the APC switch power port 1 on the fence device named apc2 to fence cluster node no d e-0 1. exampl e. co m using the same method as you specified for the first instance named AP C -d ual and setting the acti o n attribute to o n, execute the following command: ccs -h node01.example.com --addfenceinst apc2 node01.example.com

APC-dual port=1 action=on

Example 6.5, “ cl uster. co nf After Adding Dual-Power Fencing ” shows a cl uster. co nf

configuration file after you have added fencing for two power supplies for each node in a cluster.

Examp le 6 .5. cl uster. co nf Af t er Ad d in g Du al- Po wer Fen cin g

<fence>

</method>

</fence>

</clusternode>

<fence>

</method>

</fence>

</clusternode>

<fence>

</method>

</fence>

</clusternode>

</clusternodes>

Clust er Administ rat ion

</fencedevices>

<rm>

</rm>

</cluster>

Note that when you have finished configuring all of the components of your cluster, you will need to

sync the cluster configuration file to all of the nodes, as described in Section 6.15, “ Propagating the

sync the cluster configuration file to all of the nodes, as described in Section 6.15, “ Propagating the

6.7.5. T est ing t he Fence Configurat ion

As of Red Hat Enterprise Linux Release 6.4, you can test the fence configuration for each node in a cluster with the fence_check utility.

The following example shows the output of a successful execution of this command.

[root@ host-098 ~]# fence_check fence_check run at Wed Jul 23 09:13:57 CDT 2014 pid: 4769

Testing host-098 method 1: success

Testing host-099 method 1: success

Testing host-100 method 1: success

For information on this utility, see the fence_check(8) man page.

6.7.6. Removing Fence Met hods and Fence Inst ances

To remove a fence method from your cluster configuration, execute the following command: ccs -h host --rmmethod method node

For example, to remove a fence method that you have named AP C that you have configured for

no d e0 1. exampl e. co m from the cluster configuration file on cluster node

no d e0 1. exampl e. co m, execute the following command: ccs -h node01.example.com --rmmethod APC node01.example.com

To remove all fence instances of a fence device from a fence method, execute the following command: ccs -h host --rmfenceinst fencedevicename node method

For example, to remove all instances of the fence device named apc1 from the method named AP C -

d ual configured for no d e0 1. exampl e. co m from the cluster configuration file on cluster node

no d e0 1. exampl e. co m, execute the following command: ccs -h node01.example.com --rmfenceinst apc1 node01.example.com APC-dual

6.8. Configuring a Failover Domain

⁠Chapt er 6 . Configuring Red Hat High Availabilit y Add- O n Wit h t he ccs Command

A failover domain is a named subset of cluster nodes that are eligible to run a cluster service in the event of a node failure. A failover domain can have the following characteristics:

Unrestricted — Allows you to specify that a subset of members are preferred, but that a cluster service assigned to this domain can run on any available member.

(either manually or by the cluster software).

Note

The failback characteristic is applicable only if ordered failover is configured.

Note

Changing a failover domain configuration has no effect on currently running services.

Note

Failover domains are not required for operation.

By default, failover domains are unrestricted and unordered.

In a cluster with several members, using a restricted failover domain can minimize the work to set up the cluster to run a cluster service (such as httpd ) which requires you to set up the configuration identically on all members that run the cluster service. Instead of setting up the entire cluster to run the cluster service, you can set up only the members in the restricted failover domain that you associate with the cluster service.

Clust er Administ rat ion

Note

To configure a failover domain, perform the following procedure:

1. To add a failover domain, execute the following command: ccs -h host --addfailoverdomain name [restricted] [ordered]

[nofailback]

Note

The name should be descriptive enough to distinguish its purpose relative to other names used in your cluster.

For example, the following command configures a failover domain named exampl e_pri on

no d e-0 1. exampl e. co m that is unrestricted, ordered, and allows failback: ccs -h node-01.example.com --addfailoverdomain example_pri ordered

2. To add a node to a failover domain, execute the following command: ccs -h host --addfailoverdomainnode failoverdomain node priority

For example, to configure the failover domain exampl e_pri in the configuration file on

no d e-0 1. exampl e. co m so that it contains no d e-0 1. exampl e. co m with a priority of 1,

no d e-0 2. exampl e. co m with a priority of 2, and no d e-0 3. exampl e. co m with a priority of 3, execute the following commands: ccs -h node-01.example.com --addfailoverdomainnode example_pri node-01.example.com 1 ccs -h node-01.example.com --addfailoverdomainnode example_pri node-02.example.com 2 ccs -h node-01.example.com --addfailoverdomainnode example_pri node-03.example.com 3

You can list all of the failover domains and failover domain nodes configured in a cluster with the following command: ccs -h host --lsfailoverdomain

To remove a failover domain, execute the following command: ccs -h host --rmfailoverdomain name

To remove a node from a failover domain, execute the following command:

⁠Chapt er 6 . Configuring Red Hat High Availabilit y Add- O n Wit h t he ccs Command

ccs -h host --rmfailoverdomainnode failoverdomain node

Note that when you have finished configuring all of the components of your cluster, you will need to

local to a particular service when you configure the service, as described in Section 6.10, “ Adding a

6.9. Configuring Global Clust er Resources

You can configure two types of resources:

Global — Resources that are available to any service in the cluster.

Service-specific — Resources that are available to only one service.

To see a list of currently configured resources and services in the cluster, execute the following command: ccs -h host --lsservices

To add a global cluster resource, execute the following command. You can add a resource that is

Cluster Service to the Cluster” .

ccs -h host --addresource resourcetype [resource options]

For example, the following command adds a global file system resource to the cluster configuration file on no d e0 1. exampl e. co m. The name of the resource is web_fs, the file system device is

/d ev/sd d 2, the file system mountpoint is /var/www, and the file system type is ext3.

ccs -h node01.example.com --addresource fs name=web_fs device=/dev/sdd2 mountpoint=/var/www fstype=ext3

For information about the available resource types and resource options, see Appendix B, HA

Resource Parameters

To remove a global resource, execute the following command: ccs -h host --rmresource resourcetype [resource options]

If you need to modify the parameters of an existing global resource, you can remove the resource and configure it again.

Note that when you have finished configuring all of the components of your cluster, you will need to

sync the cluster configuration file to all of the nodes, as described in Section 6.15, “ Propagating the

the ccs command to print a list of available services and their options, refer to Section 6.11,

6.10. Adding a Clust er Service t o t he Clust er

To configure a cluster service in a cluster, perform the following steps:

1. Add a service to the cluster with the following command: ccs -h host --addservice servicename [service options]

Clust er Administ rat ion

Note

Use a descriptive name that clearly distinguishes the service from other services in the cluster.

When you add a service to the cluster configuration, you configure the following attributes:

auto start — Specifies whether to autostart the service when the cluster starts. Use "1" to enable and "0" to disable; the default is enabled.

d o mai n — Specifies a failover domain (if required).

excl usi ve — Specifies a policy wherein the service only runs on nodes that have no other services running on them.

reco very — Specifies a recovery policy for the service. The options are to relocate, restart, disable, or restart-disable the service. The restart recovery policy indicates that the system should attempt to restart the failed service before trying to relocate the service to another node. The relocate policy indicates that the system should try to restart the service in a different node. The disable policy indicates that the system should disable the resource group if any component fails. The restart-disable policy indicates that the system should attempt to restart the service in place if it fails, but if restarting the service fails the service will be disabled instead of being moved to another host in the cluster.

For example, to add a service to the configuration file on the cluster node no d e-

0 1. exampl e. co m named exampl e_apache that uses the failover domain exampl e_pri , and that has recovery policy of rel o cate, execute the following command: ccs -h node-01.example.com --addservice example_apache domain=example_pri recovery=relocate

When configuring services for a cluster, you may find it useful to see a listing of available services for your cluster and the options available for each service. For information on using

“ Listing Available Cluster Services and Resources” .

2. Add resources to the service with the following command: ccs -h host --addsubservice servicename subservice [service options]

Depending on the type of resources you want to use, populate the service with global or service-specific resources. To add a global resource, use the --ad d subservi ce option of the ccs to add a resource. For example, to add the global file system resource named

web_fs to the service named exampl e_apache on the cluster configuration file on no d e-

0 1. exampl e. co m, execute the following command: ccs -h node01.example.com --addsubservice example_apache fs ref=web_fs

To add a service-specific resource to the service, you need to specify all of the service

⁠Chapt er 6 . Configuring Red Hat High Availabilit y Add- O n Wit h t he ccs Command

options. For example, if you had not previously defined web_fs as a global service, you could add it as a service-specific resource with the following command: ccs -h node01.example.com --addsubservice example_apache fs name=web_fs device=/dev/sdd2 mountpoint=/var/www fstype=ext3

3. To add a child service to the service, you also use the --ad d subservi ce option of the ccs command, specifying the service options.

If you need to add services within a tree structure of dependencies, use a colon (":") to separate elements and brackets to identify subservices of the same type. The following example adds a third nfscl i ent service as a subservice of an nfscl i ent service which is in itself a subservice of an nfscl i ent service which is a subservice of a service named

servi ce_a: ccs -h node01.example.com --addsubservice service_a nfsclient[1]:nfsclient[2]:nfsclient

Note

If you are adding a Samba-service resource, add it directly to the service, not as a child of another resource.

Note

When configuring a dependency tree for a cluster service that includes a floating IP address resource, you must configure the IP resource as the first entry.

Note

To verify the existence of the IP service resource used in a cluster service, you can use the

1: lo: <LOOPBACK,UP> mtu 16436 qdisc noqueue

link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00

inet 127.0.0.1/8 scope host lo

inet6 ::1/128 scope host

valid_lft forever preferred_lft forever

2: eth0: <BROADCAST,MULTICAST,UP> mtu 1356 qdisc pfifo_fast qlen

1000

link/ether 00:05:5d:9a:d8:91 brd ff:ff:ff:ff:ff:ff

inet 10.11.4.31/22 brd 10.11.7.255 scope global eth0

inet6 fe80::205:5dff:fe9a:d891/64 scope link

inet 10.11.4.240/22 scope global secondary eth0

valid_lft forever preferred_lft forever

Clust er Administ rat ion

To remove a service and all of its subservices, execute the following command: ccs -h host --rmservice servicename

To remove a subservice, execute the following command: ccs -h host --rmsubservice servicename subservice [service options]

Note that when you have finished configuring all of the components of your cluster, you will need to

sync the cluster configuration file to all of the nodes, as described in Section 6.15, “ Propagating the

their default values, as described in Section 6.1.5, “ Commands that Overwrite Previous Settings”

6.11. List ing Available Clust er Services and Resources

You can use the ccs command to print a list of resources and services that are available for a cluster. You can also use the ccs command to print a list of the options you can specify for a particular service or resource type.

To print a list of cluster services currently available for your cluster, execute either of the following commands (--l sreso urceo pts is an alias to --l sservi ceo pts): ccs -h host --lsserviceopts ccs -h host --lsresourceopts

For example, the following command lists the cluster services and resources available on the cluster node no d e1, showing sample output.

[root@ ask-03 ~]# ccs -h no d e1 --l sservi ceo pts service - Defines a service (resource group).

ASEHAagent - Sybase ASE Failover Instance

SAPDatabase - SAP database resource agent

SAPInstance - SAP instance resource agent apache - Defines an Apache web server clusterfs - Defines a cluster file system mount.

fs - Defines a file system mount.

ip - This is an IP address.

lvm - LVM Failover script mysql - Defines a MySQL database server named - Defines an instance of named server netfs - Defines an NFS/CIFS file system mount.

nfsclient - Defines an NFS client.

nfsexport - This defines an NFS export.

nfsserver - This defines an NFS server resource.

openldap - Defines an Open LDAP server oracledb - Oracle 10g Failover Instance orainstance - Oracle 10g Failover Instance oralistener - Oracle 10g Listener Instance postgres-8 - Defines a PostgreSQL server samba - Dynamic smbd/nmbd resource agent script - LSB-compliant init script as a clustered resource.

tomcat-6 - Defines a Tomcat server vm - Defines a Virtual Machine action - Overrides resource action timings for a resource instance.

⁠Chapt er 6 . Configuring Red Hat High Availabilit y Add- O n Wit h t he ccs Command

To print a list of the options you can specify for a particular service type, execute the following command: ccs -h host --lsserviceopts service_type

For example, the following command lists the service options for the vm service.

[root@ ask-03 ~]# ccs -f no d e1 --l sservi ceo pts vm vm - Defines a Virtual Machine

Required Options:

name: Name

Optional Options:

domain: Cluster failover Domain

autostart: Automatic start after quorum formation

exclusive: Exclusive resource group

recovery: Failure recovery policy

migration_mapping: memberhost:targethost,memberhost:targethost ..

use_virsh: If set to 1, vm.sh will use the virsh command to manage virtual machines instead of xm. This is required when using non-Xen virtual machines (e.g. qemu / KVM).

xmlfile: Full path to libvirt XML file describing the domain.

migrate: Migration type (live or pause, default = live).

path: Path to virtual machine configuration files.

snapshot: Path to the snapshot directory where the virtual machine image will be stored.

depend: Top-level service this depends on, in service:name format.

depend_mode: Service dependency mode (soft or hard).

max_restarts: Maximum restarts for this service.

restart_expire_time: Restart expiration time; amount of time before a restart is forgotten.

status_program: Additional status check program

hypervisor: Hypervisor

hypervisor_uri: Hypervisor URI (normally automatic).

migration_uri: Migration URI (normally automatic).

__independent_subtree: Treat this and all children as an independent subtree.

__enforce_timeouts: Consider a timeout for operations as fatal.

__max_failures: Maximum number of failures before returning a failure to a status check.

__failure_expire_time: Amount of time before a failure is forgotten.

__max_restarts: Maximum number restarts for an independent subtree before giving up.

__restart_expire_time: Amount of time before a failure is forgotten for an independent subtree.

6.12. Virt ual Machine Resources

Virtual machine resources are configured differently than other cluster resources. In particular, they are not grouped into service definitions. As of the Red Hat Enterprise Linux 6.2 release, when you configure a virtual machine in a cluster with the ccs command you can use the --ad d vm (rather than the ad d servi ce option). This ensures that the vm resource is defined directly under the rm configuration node in the cluster configuration file.

Clust er Administ rat ion

A virtual machine resource requires at least a name and a path attribute. The name attribute should match the name of the l i bvi rt domain and the path attribute should specify the directory where the shared virtual machine definitions are stored.

Note

The path attribute in the cluster configuration file is a path specification or a directory name, not a path to an individual file.

If virtual machine definitions are stored on a shared directory named /mnt/vm_d efs, the following command will define a virtual machine named g uest1:

# ccs -h no d e1. exampl e. co m --ad d vm g uest1 path= /mnt/vm_d efs

Running this command adds the following line to the rm configuration node in the cl uster. co nf file:

6.13. Configuring a Quorum Disk

Important

Quorum-disk parameters and heuristics depend on the site environment and the special requirements needed. To understand the use of quorum-disk parameters and heuristics, refer to the qdisk(5) man page. If you require assistance understanding and using quorum disk, contact an authorized Red Hat support representative.

Use the following command to configure your system for using a quorum disk: ccs -h host --setquorumd [quorumd options]

Note that this command resets all other properties that you can set with the --setq uo rumd option to

Table 6.1, “ Quorum Disk Options”

summarizes the meaning of the quorum disk options you may need to set. For a complete list of quorum disk parameters, refer to the cluster schema at

/usr/share/cl uster/cl uster. rng , and the annotated schema at /usr/share/d o c/cman-

X. Y . ZZ/cl uster_co nf. html .

T ab le 6 .1. Q u o ru m Disk O p t io n s

Paramet er in t erval vo t es t ko

Descrip t io n

The frequency of read/write cycles, in seconds.

The number of votes the quorum daemon advertises to cman when it has a high enough score.

The number of cycles a node must miss to be declared dead.

100

⁠Chapt er 6 . Configuring Red Hat High Availabilit y Add- O n Wit h t he ccs Command

Paramet er min _sco re d evice lab el

Descrip t io n

The minimum score for a node to be considered "alive". If omitted or set to

0, the default function, fl o o r((n+ 1)/2), is used, where n is the sum of the heuristics scores. The Min imu m Sco re value must never exceed the sum of the heuristic scores; otherwise, the quorum disk cannot be available.

The storage device the quorum daemon uses. The device must be the same on all nodes.

Specifies the quorum disk label created by the mkq d i sk utility. If this field contains an entry, the label overrides the Device field. If this field is used, the quorum daemon reads /pro c/parti ti o ns and checks for qdisk signatures on every block device found, comparing the label against the specified label. This is useful in configurations where the quorum device name differs among nodes.

Use the following command to configure the heuristics for a quorum disk: ccs -h host --addheuristic [heuristic options]

Table 6.2, “ Quorum Disk Heuristics” summarizes the meaning of the quorum disk heuristics you may

need to set.

T ab le 6 .2. Q u o ru m Disk Heu rist ics

Paramet er p ro g ram in t erval sco re t ko

Descrip t io n

The path to the program used to determine if this heuristic is available.

This can be anything that can be executed by /bi n/sh -c. A return value of 0 indicates success; anything else indicates failure. This parameter is required to use a quorum disk.

The frequency (in seconds) at which the heuristic is polled. The default interval for every heuristic is 2 seconds.

The weight of this heuristic. Be careful when determining scores for heuristics. The default score for each heuristic is 1.

The number of consecutive failures required before this heuristic is declared unavailable.

To see a list of the quorum disk options and heuristics that are configured on a system, you can execute the following command: ccs -h host --lsquorum

To remove a heuristic specified by a heuristic option, you can execute the following command: ccs -h host rmheuristic [heuristic options]

Note that when you have finished configuring all of the components of your cluster, you will need to

sync the cluster configuration file to all of the nodes, as described in Section 6.15, “ Propagating the

However, for the quorum disk to operate, you must restart the cluster (refer to Section 7.2,

101

Clust er Administ rat ion

Note

Syncing and activating propagates and activates the updated cluster configuration file.

“ Starting and Stopping a Cluster” ), ensuring that you have restarted the q d i skd daemon on

each node.

6.14 . Miscellaneous Clust er Configurat ion

This section describes using the ccs command to configure the following:

Section 6.14.1, “ Cluster Configuration Version”

Section 6.14.2, “ Multicast Configuration”

Section 6.14.3, “ Configuring a Two-Node Cluster”

Section 6.14.4, “ Logging”

Section 6.14.5, “ Configuring Redundant Ring Protocol”

You can also use the ccs command to set advanced cluster configuration parameters, including

to tem options, d l m options, rm options and cman options. For information on setting these parameters see the ccs(8) man page and the annotated cluster configuration file schema at

/usr/share/d o c/cman-X. Y . ZZ/cl uster_co nf. html .

To view a list of the miscellaneous cluster attributes that have been configured for a cluster, execute the following command: ccs -h host --lsmisc

6.14 .1. Clust er Configurat ion Version

A cluster configuration file includes a cluster configuration version value. The configuration version value is set to 1 by default when you create a cluster configuration file and it is automatically incremented each time you modify your cluster configuration. However, if you need to set it to another value, you can specify it with the following command: ccs -h host --setversion n

You can get the current configuration version value on an existing cluster configuration file with the following command: ccs -h host --getversion

To increment the current configuration version value by 1 in the cluster configuration file on every node in the cluster, execute the following command: ccs -h host --incversion

6.14 .2. Mult icast Configurat ion

102

⁠Chapt er 6 . Configuring Red Hat High Availabilit y Add- O n Wit h t he ccs Command

If you do not specify a multicast address in the cluster configuration file, the Red Hat High Availability

Add-On software creates one based on the cluster ID. It generates the lower 16 bits of the address and appends them to the upper portion of the address according to whether the IP protocol is IPv4 or

IPv6:

For IPv4 — The address formed is 239.192. plus the lower 16 bits generated by Red Hat High

Availability Add-On software.

For IPv6 — The address formed is FF15:: plus the lower 16 bits generated by Red Hat High

Availability Add-On software.

Note

The cluster ID is a unique identifier that cman generates for each cluster. To view the cluster

ID, run the cman_to o l status command on a cluster node.

You can manually specify a multicast address in the cluster configuration file with the following command: ccs -h host --setmulticast multicastaddress

Note that this command resets all other properties that you can set with the --setmul ti cast option

to their default values, as described in Section 6.1.5, “ Commands that Overwrite Previous Settings”

If you specify a multicast address, you should use the 239.192.x.x series (or FF15:: for IPv6) that

cman uses. Otherwise, using a multicast address outside that range may cause unpredictable results. For example, using 224.0.0.x (which is "All hosts on the network") may not be routed correctly, or even routed at all by some hardware.

If you specify or modify a multicast address, you must restart the cluster for this to take effect. For

information on starting and stopping a cluster with the ccs command, refer to Section 7.2, “ Starting and Stopping a Cluster” .

Note

To remove a multicast address from a configuration file, use the --setmul ti cast option of the ccs but do not specify a multicast address: ccs -h host --setmulticast

6.14 .3. Configuring a T wo-Node Clust er

If you are configuring a two-node cluster, you can execute the following command to allow a single node to maintain quorum (for example, if one node fails): ccs -h host --setcman two_node=1 expected_votes=1

103

Clust er Administ rat ion

Note that this command resets all other properties that you can set with the --setcman option to their

default values, as described in Section 6.1.5, “ Commands that Overwrite Previous Settings”

When you use the ccs --setcman command to add, remove, or modify the two _no d e option, you must restart the cluster for this change to take effect. For information on starting and stopping a cluster with the ccs command, refer to

Section 7.2, “ Starting and Stopping a Cluster” .

6.14 .4 . Logging

You can enable debugging for all daemons in a cluster, or you can enable logging for specific cluster processing.

To enable debugging for all daemons, execute the following command. By default, logging is directed to the /var/l o g /cl uster/daemon. l o g file.

ccs -h host --setlogging [logging options]

For example, the following command enables debugging for all daemons.

# ccs -h no d e1. exampl e. co m --setl o g g i ng d ebug = o n

Note that this command resets all other properties that you can set with the --setl o g g i ng option

to their default values, as described in Section 6.1.5, “ Commands that Overwrite Previous Settings”

To enable debugging for an individual cluster process, execute the following command. Per-daemon logging configuration overrides the global settings.

ccs -h host --addlogging [logging daemon options]

For example, the following commands enable debugging for the co ro sync and fenced daemons.

# ccs -h no d e1. exampl e. co m --ad d l o g g i ng name= co ro sync d ebug = o n

# ccs -h no d e1. exampl e. co m --ad d l o g g i ng name= fenced d ebug = o n

To remove the log settings for individual daemons, use the following command.

ccs -h host --rmlogging name=clusterprocess

For example, the following command removes the daemon-specific log settings for the fenced daemaon ccs -h host --rmlogging name=fenced

For a list of the logging daemons for which you can enable logging as well as the additional logging options you can configure for both global and per-daemon logging, refer to the cl uster. co nf(5) man page.

Note that when you have finished configuring all of the components of your cluster, you will need to

sync the cluster configuration file to all of the nodes, as described in Section 6.15, “ Propagating the

considerations you must take into account, as described in Section 8.6, “ Configuring Redundant

6.14 .5. Configuring Redundant Ring Prot ocol

104

⁠Chapt er 6 . Configuring Red Hat High Availabilit y Add- O n Wit h t he ccs Command

As of Red Hat Enterprise Linux 6.4, the Red Hat High Availability Add-On supports the configuration of redundant ring protocol. When using redundant ring protocol, there are a variety of

Ring Protocol” .

To specify a second network interface to use for redundant ring protocol, you add an alternate name for the node using the --ad d al t option of the ccs command: ccs -h host --addalt node_name alt_name

For example, the following command configures the alternate name cl usternet-no d e1-eth2 for the cluster node cl usternet-no d e1-eth1:

# ccs -h cl usternet-no d e1-eth1 --ad d al t cl usternet-no d e1-eth1

cl usternet-no d e1-eth2

Optionally, you can manually specify a multicast address, a port, and a TTL for the second ring. If you specify a multicast for the second ring, either the alternate multicast address or the alternate port must be different from the multicast address for the first ring. If you specify an alternate port, the port numbers of the first ring and the second ring must differ by at least two, since the system itself uses port and port-1 to perform operations. If you do not specify an alternate multicast address, the system will automatically use a different multicast address for the second ring.

To specify an alternate multicast address, port, or TTL for the second ring, you use the --

setal tmul ti cast option of the ccs command: ccs -h host --setaltmulticast [alt_multicast_address]

[alt_multicast_options].

For example, the following command sets an alternate multicast address of 239.192.99.88, a port of

888, and a TTL of 3 for the cluster defined in the cl uster. co nf file on node cl usternet-no d e1-

eth1: ccs -h clusternet-node1-eth1 --setaltmulticast 239.192.99.88 port=888 ttl=3

To remove an alternate multicast address, specify the --setal tmul ti cast option of the ccs command but do not specify a multicast address. Note that executing this command resets all other properties that you can set with the --setal tmul ti cast option to their default values, as described in

Section 6.1.5, “ Commands that Overwrite Previous Settings” .

When you have finished configuring all of the components of your cluster, you will need to sync the

cluster configuration file to all of the nodes, as described in Section 6.15, “ Propagating the

Section 7.1, “ Managing Cluster Nodes”

6.15. Propagat ing t he Configurat ion File t o t he Clust er Nodes

After you have created or edited a cluster configuration file on one of the nodes in the cluster, you need to propagate that same file to all of the cluster nodes and activate the configuration.

Use the following command to propagate and activate a cluster configuration file. When you use the

--acti vate option, you must also specify the --sync option for the activation to take affect.

ccs -h host --sync --activate

105

Clust er Administ rat ion

To verify that all of the nodes specified in the hosts cluster configuration file have the identical cluster configuration file, execute the following command: ccs -h host --checkconf

If you have created or edited a configuration file on a local node, use the following command to send that file to one of the nodes in the cluster: ccs -f file -h host --setconf

To verify that all of the nodes specified in the local file have the identical cluster configuration file, execute the following command: ccs -f file --checkconf

106

⁠Chapt er 7 . Managing Red Hat High Availabilit y Add- O n Wit h ccs

Chapter 7. Managing Red Hat High Availability Add-On With ccs

This chapter describes various administrative tasks for managing the Red Hat High Availability Add-

On by means of the ccs command, which is supported as of the Red Hat Enterprise Linux 6.1 release and later. This chapter consists of the following sections:

Section 7.2, “ Starting and Stopping a Cluster”

Section 7.3, “ Diagnosing and Correcting Problems in a Cluster”

7.1. Managing Clust er Nodes

This section documents how to perform the following node-management functions with the ccs command:

Section 7.1.1, “ Causing a Node to Leave or Join a Cluster”

Section 7.1.2, “ Adding a Member to a Running Cluster”

7.1.1. Causing a Node t o Leave or Join a Clust er

You can use the ccs command to cause a node to leave a cluster by stopping cluster services on that node. Causing a node to leave a cluster does not remove the cluster configuration information from that node. Making a node leave a cluster prevents the node from automatically joining the cluster when it is rebooted.

To cause a node to leave a cluster, execute the following command, which stops cluster services on the node specified with the -h option: ccs -h host --stop

When you stop cluster services on a node, any service that is running on that node will fail over.

To delete a node entirely from the cluster configuration, use the --rmno d e option of the ccs

command, as described in Section 6.4, “ Creating and Modifying a Cluster” .

To cause a node to rejoin a cluster execute the following command, which starts cluster services on the node specified with the -h option: ccs -h host --start

7.1.2. Adding a Member t o a Running Clust er

To add a member to a running cluster, add a node to the cluster as described in Section 6.4,

“ Creating and Modifying a Cluster” . After updating the configuration file, propagate the file to all

nodes in the cluster and be sure to activate the new cluster configuration file, as described in

Section 6.15, “ Propagating the Configuration File to the Cluster Nodes” .

7.2. St art ing and St opping a Clust er

107

Clust er Administ rat ion

You can use the ccs command to stop a cluster by using the following command to stop cluster services on all nodes in the cluster: ccs -h host --stopall

You can use the ccs command to start a cluster that is not running by using the following command to start cluster services on all nodes in the cluster: ccs -h host --startall

When you use the --startal l option of the ccs command to start a cluster, the command automatically enables the cluster resources. For some configurations, such as when services have been intentionally disabled on one node to disable fence loops, you may not want to enable the services on that node. As of Red Hat Enterprise Linux 6.6 release, you can use the --no enabl e option of the ccs --startal l command to prevent the services from being enabled: ccs -h host --startall --noenable

7.3. Diagnosing and Correct ing Problems in a Clust er

For information about diagnosing and correcting problems in a cluster, see Chapter 10, Diagnosing

and Correcting Problems in a Cluster

. There are a few simple checks that you can perform with the ccs

command, however.

To verify that all of the nodes specified in the host's cluster configuration file have identical cluster configuration files, execute the following command: ccs -h host --checkconf

If you have created or edited a configuration file on a local node, you can verify that all of the nodes specified in the local file have identical cluster configuration files with the following command: ccs -f file --checkconf

108

⁠Chapt er 8 . Configuring Red Hat High Availabilit y Manually

Chapter 8. Configuring Red Hat High Availability Manually

This chapter describes how to configure Red Hat High Availability Add-On software by directly editing the cluster configuration file (/etc/cl uster/cl uster. co nf) and using command-line tools. The chapter provides procedures about building a configuration file one section at a time, starting with a sample file provided in the chapter. As an alternative to starting with a sample file provided here, you could copy a skeleton configuration file from the cl uster. co nf man page.

However, doing so would not necessarily align with information provided in subsequent procedures in this chapter. There are other ways to create and configure a cluster configuration file; this chapter provides procedures about building a configuration file one section at a time. Also, keep in mind that this is just a starting point for developing a configuration file to suit your clustering needs.

This chapter consists of the following sections:

Section 8.1, “ Configuration Tasks”

Section 8.2, “ Creating a Basic Cluster Configuration File”

Section 8.3, “ Configuring Fencing”

Section 8.4, “ Configuring Failover Domains”

Section 8.5, “ Configuring HA Services”

Section 8.7, “ Configuring Debug Options”

Section 8.6, “ Configuring Redundant Ring Protocol”

Section 8.9, “ Verifying a Configuration”

Important

/usr/share/d o c/cman-X. Y . ZZ/cl uster_co nf. html (for example

/usr/share/d o c/cman-3. 0 . 12/cl uster_co nf. html ).

109

Clust er Administ rat ion

Important

Certain procedure in this chapter call for using the cman_to o l versi o n -r command to propagate a cluster configuration throughout a cluster. Using that command requires that

ri cci is running. Using ricci requires a password the first time you interact with ricci from

any specific machine. For information on the ri cci service, refer to Section 3.13,

“ Considerations for ri cci ” .

Note

Procedures in this chapter may include specific commands for some of the command-line

tools listed in Appendix E, Command Line Tools Summary

. For more information about all commands and variables, refer to the man page for each command-line tool.

8.1. Configurat ion T asks

Configuring Red Hat High Availability Add-On software with command-line tools consists of the following steps:

1. Creating a cluster. Refer to

Section 8.2, “ Creating a Basic Cluster Configuration File” .

2. Configuring fencing. Refer to Section 8.3, “ Configuring Fencing” .

3. Configuring failover domains. Refer to

Section 8.4, “ Configuring Failover Domains”

4. Configuring HA services. Refer to

Section 8.5, “ Configuring HA Services”

5. Verifying a configuration. Refer to Section 8.9, “ Verifying a Configuration” .

8.2. Creat ing a Basic Clust er Configurat ion File

Provided that cluster hardware, Red Hat Enterprise Linux, and High Availability Add-On software are installed, you can create a cluster configuration file (/etc/cl uster/cl uster. co nf) and start running the High Availability Add-On. As a starting point only, this section describes how to create a skeleton cluster configuration file without fencing, failover domains, and HA services. Subsequent sections describe how to configure those parts of the configuration file.

Important

This is just an interim step to create a cluster configuration file; the resultant file does not have any fencing and is not considered to be a supported configuration.

The following steps describe how to create and configure a skeleton cluster configuration file.

Ultimately, the configuration file for your cluster will vary according to the number of nodes, the type of fencing, the type and number of HA services, and other site-specific requirements.

1. At any node in the cluster, create /etc/cl uster/cl uster. co nf, using the template of the example in

Example 8.1, “ cl uster. co nf Sample: Basic Configuration” .

110

⁠Chapt er 8 . Configuring Red Hat High Availabilit y Manually

2. (O pti o nal ) If you are configuring a two-node cluster, you can add the following line to the configuration file to allow a single node to maintain quorum (for example, if one node fails):

<cman two _no d e= "1"

expected _vo tes= "1"/>

When you add or remove the two _no d e option from the cl uster. co nf file, you must restart the cluster for this change to take effect when you update the configuration. For information

on updating a cluster configuration, refer to Section 9.4, “ Updating a Configuration” . For an

example of specifying the two _no d e option, refer to Example 8.2, “ cl uster. co nf Sample:

Basic Two-Node Configuration” .

3. Specify the cluster name and the configuration version number using the cl uster attributes:

name and co nfi g _versi o n (refer to

Example 8.1, “ cl uster. co nf Sample: Basic

Configuration” or

Example 8.2, “ cl uster. co nf Sample: Basic Two-Node Configuration” ).

4. In the cl usterno d es section, specify the node name and the node ID of each node using the cl usterno d e attributes: name and no d ei d . The node name can be up to 255 bytes in length.

5. Save /etc/cl uster/cl uster. co nf.

6. Validate the file against the cluster schema (cl uster. rng ) by running the

ccs_co nfi g _val i d ate command. For example:

[root@ example-01 ~]# ccs_co nfi g _val i d ate

Configuration validates

7. Propagate the configuration file to /etc/cl uster/ in each cluster node. For example, you could propagate the file to other cluster nodes using the scp command.

Note

Propagating the cluster configuration file this way is necessary the first time a cluster is created. Once a cluster is installed and running, the cluster configuration file can be propagated using the cman_to o l versi o n -r command. It is possible to use the

scp command to propagate an updated configuration file; however, the cluster software must be stopped on all nodes while using the scp command.In addition, you should run the ccs_co nfi g _val i d ate if you propagate an updated configuration file via the scp.

Note

While there are other elements and attributes present in the sample configuration file

(for example, fence and fenced evi ces), there is no need to populate them now.

Subsequent procedures in this chapter provide information about specifying other elements and attributes.

8. Start the cluster. At each cluster node run the following command:

servi ce cman start

For example:

111

Clust er Administ rat ion

[root@ example-01 ~]# servi ce cman start

Starting cluster:

Checking Network Manager... [ OK ]

Global setup... [ OK ]

Loading kernel modules... [ OK ]

Mounting configfs... [ OK ]

Starting cman... [ OK ]

Waiting for quorum... [ OK ]

Starting fenced... [ OK ]

Starting dlm_controld... [ OK ]

Starting gfs_controld... [ OK ]

Unfencing self... [ OK ]

Joining fence domain... [ OK ]

9. At any cluster node, run cman_to o l no d es to verify that the nodes are functioning as members in the cluster (signified as "M" in the status column, "Sts"). For example:

[root@ example-01 ~]# cman_to o l no d es

Node Sts Inc Joined Name

1 M 548 2010-09-28 10:52:21 node-01.example.com

2 M 548 2010-09-28 10:52:21 node-02.example.com

3 M 544 2010-09-28 10:52:21 node-03.example.com

10. If the cluster is running, proceed to

Section 8.3, “ Configuring Fencing”

Basic Configurat ion Examples

Example 8.1, “ cl uster. co nf Sample: Basic Configuration”

and Example 8.2, “ cl uster. co nf

Sample: Basic Two-Node Configuration” (for a two-node cluster) each provide a very basic sample

cluster configuration file as a starting point. Subsequent procedures in this chapter provide information about configuring fencing and HA services.

Examp le 8.1. cl uster. co nf Samp le: Basic Co n f ig u rat io n

<fence>

</fence>

</clusternode>

<fence>

</fence>

</clusternode>

<fence>

</fence>

</clusternode>

</clusternodes>

112

⁠Chapt er 8 . Configuring Red Hat High Availabilit y Manually

</fencedevices>

<rm>

</rm>

</cluster>

Examp le 8.2. cl uster. co nf Samp le: Basic T wo - No d e Co n f ig u rat io n

<fence>

</fence>

</clusternode>

<fence>

</fence>

</clusternode>

</clusternodes>

</fencedevices>

<rm>

</rm>

</cluster>

T he

co nsensus

Value for

to tem

in a T wo-Node Clust er

When you create a two-node cluster and you do not intend to add additional nodes to the cluster at a later time, then you should omit the co nsensus value in the to tem tag in the cl uster. co nf file so that the co nsensus value is calculated automatically. When the co nsensus value is calculated automatically, the following rules are used:

If there are two nodes or fewer, the co nsensus value will be (token * 0.2), with a ceiling of 2000 msec and a floor of 200 msec.

If there are three or more nodes, the co nsensus value will be (token + 2000 msec)

If you let the cman utility configure your consensus timeout in this fashion, then moving at a later time from two to three (or more) nodes will require a cluster restart, since the consensus timeout will need to change to the larger value based on the token timeout.

If you are configuring a two-node cluster and intend to upgrade in the future to more than two nodes, you can override the consensus timeout so that a cluster restart is not required when moving from two to three (or more) nodes. This can be done in the cl uster. co nf as follows:

Note that the configuration parser does not calculate X + 2000 automatically. An integer value must be used rather than an equation.

113

Clust er Administ rat ion

The advantage of using the optimized consensus timeout for two-node clusters is that overall failover time is reduced for the two-node case, since consensus is not a function of the token timeout.

Note that for two-node autodetection in cman, the number of physical nodes is what matters and not the presence of the two _no d e= 1 directive in the cl uster. co nf file.

8.3. Configuring Fencing

Configuring fencing consists of (a) specifying one or more fence devices in a cluster and (b) specifying one or more fence methods for each node (using a fence device or fence devices specified).

Note

It is recommended that you configure multiple fencing mechanisms for each node. A fencing device can fail due to network split, a power outage, or a problem in the fencing device itself.

Configuring multiple fencing mechanisms can reduce the likelihood that the failure of a fencing device will have fatal results.

Based on the type of fence devices and fence methods required for your configuration, configure

cl uster. co nf as follows:

1. In the fenced evi ces section, specify each fence device, using a fenced evi ce element

and fence-device dependent attributes. Example 8.3, “ APC Fence Device Added to

cl uster. co nf ”

shows an example of a configuration file with an APC fence device added

to it.

2. At the cl usterno d es section, within the fence element of each cl usterno d e section, specify each fence method of the node. Specify the fence method name, using the metho d attribute, name. Specify the fence device for each fence method, using the d evi ce element

and its attributes, name and fence-device-specific parameters. Example 8.4, “ Fence Methods

Added to cl uster. co nf ” shows an example of a fence method with one fence device for

each node in the cluster.

3. For non-power fence methods (that is, SAN/storage fencing), at the cl usterno d es section, add an unfence section. This ensures that a fenced node is not re-enabled until the node has been rebooted. When you configure a device that requires unfencing, the cluster must first be stopped and the full configuration including devices and unfencing must be added before the cluster is started. For more information about unfencing a node, refer to the

fence_no d e(8) man page.

The unfence section does not contain metho d sections like the fence section does. It contains d evi ce references directly, which mirror the corresponding device sections for

fence, with the notable addition of the explicit action (acti o n) of "on" or "enable". The same fenced evi ce is referenced by both fence and unfence d evi ce lines, and the same per-node arguments should be repeated.

Specifying the acti o n attribute as "on" or "enable" enables the node when rebooted.

Example 8.4, “ Fence Methods Added to cl uster. co nf ”

and Example 8.5,

“ cl uster. co nf: Multiple Fence Methods per Node” include examples of the unfence

elements and attributed.

For more information about unfence refer to the fence_no d e man page.

114

⁠Chapt er 8 . Configuring Red Hat High Availabilit y Manually

4. Update the co nfi g _versi o n attribute by incrementing its value (for example, changing from co nfi g _versi o n= "2" to co nfi g _versi o n= "3">).

5. Save /etc/cl uster/cl uster. co nf.

6. (O pti o nal ) Validate the updated file against the cluster schema (cl uster. rng ) by running the ccs_co nfi g _val i d ate command. For example:

[root@ example-01 ~]# ccs_co nfi g _val i d ate

Configuration validates

7. Run the cman_to o l versi o n -r command to propagate the configuration to the rest of the cluster nodes. This will also run additional validation. It is necessary that ri cci be running in each cluster node to be able to propagate updated cluster configuration information.

8. Verify that the updated configuration file has been propagated.

9. Proceed to

Section 8.4, “ Configuring Failover Domains”

If required, you can configure complex configurations with multiple fence methods per node and with multiple fence devices per fence method. When specifying multiple fence methods per node, if fencing fails using the first method, fenced , the fence daemon, tries the next method, and continues to cycle through methods until one succeeds.

Sometimes, fencing a node requires disabling two I/O paths or two power ports. This is done by specifying two or more devices within a fence method. fenced runs the fence agent once for each fence-device line; all must succeed for fencing to be considered successful.

More complex configurations are shown in

the section called “ Fencing Configuration Examples”

You can find more information about configuring specific fence devices from a fence-device agent man page (for example, the man page for fence_apc). In addition, you can get more information

about fencing parameters from Appendix A, Fence Device Parameters

, the fence agents in

/usr/sbi n/, the cluster schema at /usr/share/cl uster/cl uster. rng , and the annotated schema at /usr/share/d o c/cman-X. Y . ZZ/cl uster_co nf. html (for example,

/usr/share/d o c/cman-3. 0 . 12/cl uster_co nf. html ).

Note

As of Red Hat Enterprise Linux Release 6.4, you can test the fence configuration for each node in a cluster with the fence_check utility. For information on this utility, see the

fence_check(8) man page.

Fencing Configurat ion Examples

The following examples show a simple configuration with one fence method per node and one fence device per fence method:

Example 8.3, “ APC Fence Device Added to cl uster. co nf ”

Example 8.4, “ Fence Methods Added to cl uster. co nf ”

The following examples show more complex configurations:

Example 8.5, “ cl uster. co nf: Multiple Fence Methods per Node”

115

Clust er Administ rat ion

Example 8.6, “ cl uster. co nf: Fencing, Multipath Multiple Ports”

Example 8.7, “ cl uster. co nf: Fencing Nodes with Dual Power Supplies”

Note

The examples in this section are not exhaustive; that is, there may be other ways to configure fencing depending on your requirements.

Examp le 8.3. APC Fen ce Device Ad d ed t o cl uster. co nf

<fence>

</fence>

</clusternode>

<fence>

</fence>

</clusternode>

<fence>

</fence>

</clusternode>

</clusternodes>

</fencedevices>

<rm>

</rm>

</cluster>

In this example, a fence device (fenced evi ce) has been added to the fenced evi ces element, specifying the fence agent (ag ent) as fence_apc, the IP address (i pad d r) as

apc_i p_exampl e, the login (l o g i n) as l o g i n_exampl e, the name of the fence device (name) as apc, and the password (passwd ) as passwo rd _exampl e.

Examp le 8.4 . Fen ce Met h o d s Ad d ed t o cl uster. co nf

116

<fence>

</method>

</fence>

⁠Chapt er 8 . Configuring Red Hat High Availabilit y Manually

</clusternode>

<fence>

</method>

</fence>

</clusternode>

<fence>

</method>

</fence>

</clusternode>

</clusternodes>

</fencedevices>

<rm>

</rm>

</cluster>

In this example, a fence method (metho d ) has been added to each node. The name of the fence method (name) for each node is AP C . The device (d evi ce) for the fence method in each node specifies the name (name) as apc and a unique APC switch power port number (po rt) for each node. For example, the port number for node-01.example.com is 1 (po rt= "1"). The device name for each node (d evi ce name= "apc") points to the fence device by the name (name) of apc in this line of the fenced evi ces element: fenced evi ce ag ent= "fence_apc"

i pad d r= "apc_i p_exampl e" l o g i n= "l o g i n_exampl e" name= "apc"

passwd = "passwo rd _exampl e".

Examp le 8.5. cl uster. co nf: Mu lt ip le Fen ce Met h o d s p er No d e

<fence>

</method>

</method>

</fence>

</unfence>

</clusternode>

<fence>

117

Clust er Administ rat ion

</method>

</method>

</fence>

</unfence>

</clusternode>

<fence>

</method>

</method>

</fence>

</unfence>

</clusternode>

</clusternodes>

</fencedevices>

<rm>

</rm>

</cluster>

Examp le 8.6 . cl uster. co nf: Fen cin g , Mu lt ip at h Mu lt ip le Po rt s

118

<fence>

</method>

</fence>

</unfence>

</clusternode>

<fence>

⁠Chapt er 8 . Configuring Red Hat High Availabilit y Manually

</method>

</fence>

</unfence>

</clusternode>

<fence>

</method>

</fence>

</unfence>

</clusternode>

</clusternodes>

</fencedevices>

<rm>

</rm>

</cluster>

Examp le 8.7. cl uster. co nf: Fen cin g No d es wit h Du al Po wer Su p p lies

<fence>

</method>

</fence>

</clusternode>

<fence>

</method>

119

Clust er Administ rat ion

</fence>

</clusternode>

<fence>

</method>

</fence>

</clusternode>

</clusternodes>

</fencedevices>

<rm>

</rm>

</cluster>

When using power switches to fence nodes with dual power supplies, the agents must be told to turn off both power ports before restoring power to either port. The default off-on behavior of the agent could result in the power never being fully disabled to the node.

8.4 . Configuring Failover Domains

A failover domain is a named subset of cluster nodes that are eligible to run a cluster service in the event of a node failure. A failover domain can have the following characteristics:

Unrestricted — Allows you to specify that a subset of members are preferred, but that a cluster service assigned to this domain can run on any available member.

(either manually or by the cluster software).

Ordered — Allows you to specify a preference order among the members of a failover domain.

Ordered failover domains select the node with the lowest priority number first. That is, the node in a failover domain with a priority number of "1" specifies the highest priority, and therefore is the most preferred node in a failover domain. After that node, the next preferred node would be the node with the next highest priority number, and so on.

In that circumstance, if a node is the preferred node in a failover domain, it is possible for a

120

⁠Chapt er 8 . Configuring Red Hat High Availabilit y Manually

service to fail over and fail back repeatedly between the preferred node and another node, causing severe impact on performance.

Note

The failback characteristic is applicable only if ordered failover is configured.

Note

Changing a failover domain configuration has no effect on currently running services.

Note

Failover domains are not required for operation.

By default, failover domains are unrestricted and unordered.

Note

To configure a failover domain, use the following procedures:

1. Open /etc/cl uster/cl uster. co nf at any node in the cluster.

2. Add the following skeleton section within the rm element for each failover domain to be used:

</failoverdomain>

</failoverdomains>

121

Clust er Administ rat ion

Note

The number of fai l o verd o mai nno d e attributes depends on the number of nodes in the failover domain. The skeleton fai l o verd o mai n section in preceding text shows three fai l o verd o mai nno d e elements (with no node names specified), signifying that there are three nodes in the failover domain.

3. In the fai l o verd o mai n section, provide the values for the elements and attributes. For descriptions of the elements and attributes, refer to the failoverdomain section of the annotated cluster schema. The annotated cluster schema is available at /usr/share/d o c/cman-

X. Y . ZZ/cl uster_co nf. html (for example /usr/share/d o c/cman-

3. 0 . 12/cl uster_co nf. html ) in any of the cluster nodes. For an example of a

fai l o verd o mai ns section, refer to

Example 8.8, “ A Failover Domain Added to

cl uster. co nf ”

4. Update the co nfi g _versi o n attribute by incrementing its value (for example, changing from co nfi g _versi o n= "2" to co nfi g _versi o n= "3">).

5. Save /etc/cl uster/cl uster. co nf.

6. (O pti o nal ) Validate the file against the cluster schema (cl uster. rng ) by running the

ccs_co nfi g _val i d ate command. For example:

[root@ example-01 ~]# ccs_co nfi g _val i d ate

Configuration validates

7. Run the cman_to o l versi o n -r command to propagate the configuration to the rest of the cluster nodes.

8. Proceed to

Section 8.5, “ Configuring HA Services”

Example 8.8, “ A Failover Domain Added to cl uster. co nf ” shows an example of a configuration

with an ordered, unrestricted failover domain.

Examp le 8.8. A Failo ver Do main Ad d ed t o cl uster. co nf

122

<fence>

</method>

</fence>

</clusternode>

<fence>

</method>

</fence>

</clusternode>

⁠Chapt er 8 . Configuring Red Hat High Availabilit y Manually

<fence>

</method>

</fence>

</clusternode>

</clusternodes>

</fencedevices>

<rm>

</failoverdomain>

</failoverdomains>

</rm>

</cluster>

The fai l o verd o mai ns section contains a fai l o verd o mai n section for each failover domain in the cluster. This example has one failover domain. In the fai l o verd o mai n line, the name

(name) is specified as exampl e_pri . In addition, it specifies that resources using this domain should fail-back to lower-priority-score nodes when possible (no fai l back= "0 "), that failover is ordered (o rd ered = "1"), and that the failover domain is unrestricted (restri cted = "0 ").

8.5. Configuring HA Services

Configuring HA (High Availability) services consists of configuring resources and assigning them to services.

The following sections describe how to edit /etc/cl uster/cl uster. co nf to add resources and services.

Section 8.5.1, “ Adding Cluster Resources”

Section 8.5.2, “ Adding a Cluster Service to the Cluster”

Important

There can be a wide range of configurations possible with High Availability resources and services. For a better understanding about resource parameters and resource behavior, refer to

and Appendix C, HA Resource Behavior

. For optimal performance and to ensure that your configuration can be supported, contact an authorized

Red Hat support representative.

123

Clust er Administ rat ion

8.5.1. Adding Clust er Resources

You can configure two types of resources:

Global — Resources that are available to any service in the cluster. These are configured in the

reso urces section of the configuration file (within the rm element).

Service-specific — Resources that are available to only one service. These are configured in each

servi ce section of the configuration file (within the rm element).

This section describes how to add a global resource. For procedures about configuring service-

specific resources, refer to Section 8.5.2, “ Adding a Cluster Service to the Cluster”

To add a global cluster resource, follow the steps in this section.

1. Open /etc/cl uster/cl uster. co nf at any node in the cluster.

2. Add a reso urces section within the rm element. For example:

<rm>

</resources>

</rm>

3. Populate it with resources according to the services you want to create. For example, here are resources that are to be used in an Apache service. They consist of a file system (fs) resource, an IP (i p) resource, and an Apache (apache) resource.

<rm>

</resources>

</rm>

Example 8.9, “ cl uster. co nf File with Resources Added ”

shows an example of a

cl uster. co nf file with the reso urces section added.

4. Update the co nfi g _versi o n attribute by incrementing its value (for example, changing from co nfi g _versi o n= "2" to co nfi g _versi o n= "3").

5. Save /etc/cl uster/cl uster. co nf.

6. (O pti o nal ) Validate the file against the cluster schema (cl uster. rng ) by running the

ccs_co nfi g _val i d ate command. For example:

[root@ example-01 ~]# ccs_co nfi g _val i d ate

Configuration validates

124

⁠Chapt er 8 . Configuring Red Hat High Availabilit y Manually

7. Run the cman_to o l versi o n -r command to propagate the configuration to the rest of the cluster nodes.

8. Verify that the updated configuration file has been propagated.

9. Proceed to

Section 8.5.2, “ Adding a Cluster Service to the Cluster” .

Examp le 8.9 . cl uster. co nf File wit h Reso u rces Ad d ed

<fence>

</method>

</fence>

</clusternode>

<fence>

</method>

</fence>

</clusternode>

<fence>

</method>

</fence>

</clusternode>

</clusternodes>

</fencedevices>

<rm>

</failoverdomain>

</failoverdomains>

125

Clust er Administ rat ion

</resources>

</rm>

</cluster>

8.5.2. Adding a Clust er Service t o t he Clust er

To add a cluster service to the cluster, follow the steps in this section.

Note

The examples provided in this section show a cluster service in which all of the resources are at the same level. For information on defining a service in which there is a dependency chain in a resource hierarchy, as well as the rules that govern the behavior of parent and child resources, see

Appendix C, HA Resource Behavior

1. Open /etc/cl uster/cl uster. co nf at any node in the cluster.

2. Add a servi ce section within the rm element for each service. For example:

<rm>

</service>

</rm>

3. Configure the following parameters (attributes) in the servi ce element:

auto start — Specifies whether to autostart the service when the cluster starts. Use '1' to enable and '0' to disable; the default is enabled.

d o mai n — Specifies a failover domain (if required).

excl usi ve — Specifies a policy wherein the service only runs on nodes that have no other services running on them.

reco very — Specifies a recovery policy for the service. The options are to relocate, restart, disable, or restart-disable the service.

4. Depending on the type of resources you want to use, populate the service with global or service-specific resources

For example, here is an Apache service that uses global resources:

<rm>

126

⁠Chapt er 8 . Configuring Red Hat High Availabilit y Manually

</resources>

</service>

</rm>

For example, here is an Apache service that uses service-specific resources:

<rm>

</service>

</rm>

Example 8.10, “ cl uster. co nf with Services Added: One Using Global Resources and One

Using Service-Specific Resources ” shows an example of a cl uster. co nf file with two

services:

exampl e_apache — This service uses global resources web_fs, 127. 14 3. 131. 10 0 , and exampl e_server.

exampl e_apache2 — This service uses service-specific resources web_fs2,

127. 14 3. 131. 10 1, and exampl e_server2.

5. Update the co nfi g _versi o n attribute by incrementing its value (for example, changing from co nfi g _versi o n= "2" to co nfi g _versi o n= "3">).

6. Save /etc/cl uster/cl uster. co nf.

7. (O pti o nal ) Validate the updated file against the cluster schema (cl uster. rng ) by running the ccs_co nfi g _val i d ate command. For example:

[root@ example-01 ~]# ccs_co nfi g _val i d ate

Configuration validates

8. Run the cman_to o l versi o n -r command to propagate the configuration to the rest of the cluster nodes.

9. Verify that the updated configuration file has been propagated.

10. Proceed to

Section 8.9, “ Verifying a Configuration”

127

Clust er Administ rat ion

Examp le 8.10. cl uster. co nf wit h Services Ad d ed : O n e Usin g G lo b al Reso u rces an d

O n e Usin g Service- Sp ecif ic Reso u rces

128

<fence>

</method>

</fence>

</clusternode>

<fence>

</method>

</fence>

</clusternode>

<fence>

</method>

</fence>

</clusternode>

</clusternodes>

</fencedevices>

<rm>

</failoverdomain>

</failoverdomains>

</resources>

⁠Chapt er 8 . Configuring Red Hat High Availabilit y Manually

</service>

</service>

</rm>

</cluster>

8.6. Configuring Redundant Ring Prot ocol

As of Red Hat Enterprise Linux 6.4, the Red Hat High Availability Add-On supports the configuration of redundant ring protocol.

When configuring a system to use redundant ring protocol, you must take the following considerations into account:

Do not specify more than two rings.

Each ring must use the same protocol; do not mix IPv4 and IPv6.

If necessary, you can manually specify a multicast address for the second ring. If you specify a multicast address for the second ring, either the alternate multicast address or the alternate port must be different from the multicast address for the first ring. If you do not specify an alternate multicast address, the system will automatically use a different multicast address for the second ring.

If you specify an alternate port, the port numbers of the first ring and the second ring must differ by at least two, since the system itself uses port and port-1 to perform operations.

Do not use two different interfaces on the same subnet.

In general, it is a good practice to configure redundant ring protocol on two different NICs and two different switches, in case one NIC or one switch fails.

Do not use the i fd o wn command or the servi ce netwo rk sto p command to simulate network failure. This destroys the whole cluster and requires that you restart all of the nodes in the cluster to recover.

Do not use Netwo rkManag er, since it will execute the i fd o wn command if the cable is unplugged.

When one node of a NIC fails, the entire ring is marked as failed.

No manual intervention is required to recover a failed ring. To recover, you only need to fix the original reason for the failure, such as a failed NIC or switch.

To specify a second network interface to use for redundant ring protocol, you add an al tname component to the cl usterno d e section of the cl uster. co nf configuration file. When specifying

al tname, you must specify a name attribute to indicate a second host name or IP address for the node.

129

Clust er Administ rat ion

The following example specifies cl usternet-no d e1-eth2 as the alternate name for cluster node

cl usternet-no d e1-eth1.

<fence>

</method>

</fence>

</clusternode>

The al tname section within the cl usterno d e block is not position dependent. It can come before or after the fence section. Do not specify more than one al tname component for a cluster node or the system will fail to start.

Optionally, you can manually specify a multicast address, a port, and a TTL for the second ring by including an al tmul ti cast component in the cman section of the cl uster. co nf configuration file. The al tmul ti cast component accepts an ad d r, a po rt, and a ttl parameter.

The following example shows the cman section of a cluster configuration file that sets a multicast address, port, and TTL for the second ring.

<cman>

</cman>

8.7. Configuring Debug Opt ions

You can enable debugging for all daemons in a cluster, or you can enable logging for specific cluster processing.

To enable debugging for all daemons, add the following to the /etc/cl uster/cl uster. co nf. By default, logging is directed to the /var/l o g /cl uster/daemon. l o g file.

...

</cluster>

To enable debugging for individual cluster processes, add the following lines to the the

/etc/cl uster/cl uster. co nf file. Per-daemon logging configuration overrides the global settings.

...

130

⁠Chapt er 8 . Configuring Red Hat High Availabilit y Manually

<logging_daemon name="corosync" debug="on" />

<logging_daemon name="fenced" debug="on" />

<logging_daemon name="qdiskd" debug="on" />

<logging_daemon name="rgmanager" debug="on" />

<logging_daemon name="dlm_controld" debug="on" />

<logging_daemon name="gfs_controld" debug="on" />

</logging>

...

</cluster>

8.8. Configuring nfsexport and nfsserver Resources

This section describes the issues and considerations to take into account when configuring an

nfsexpo rt or an nfsserver resource.

The nfsexpo rt resource agent works with NFSv2 and NFSv3 clients. When using nfsexpo rt, you must do the following:

Ensure that nfs and nfsl o ck are enabled at boot.

Add R P C NFSD AR G S= "-N 4 " to the /etc/sysco nfi g /nfs file on all cluster nodes. The "-N 4 " option prevents NFSv4 clients from being able to connect to the server.

Add ST AT D AR G = "-H /usr/sbi n/cl unfsl o ck" to the /etc/sysco nfi g /nfs file on all cluster nodes.

Add nfsl o ck= "1" to the servi ce component in the cl uster. co nf file.

Structure your service as follows:

...

</nfsexport>

</fs>

...

</service>

The nfsserver resource agent works with NFSv3 and NFSv4 clients. When using nfsserver, you must do the following:

Ensure that nfs and nfsl o ck are disabled at boot

Ensure that nfsl o ck= "1" is not set for the service.

Structure your service as follows:

131

Clust er Administ rat ion

...

</nfsserver>

</fs>

...

</service>

When configuring a system to use the nfsserver resource agent for use with NFSv3 and NFSv4, you must account for the following limitations:

Configure only one nfsserver resource per cluster. If you require more, you must use restricted failover domains to ensure that the two services in question can never start on the same host.

Do not reference a globally-configured nfsserver resource in more than one service.

Do not mix old-style NFS services with the new nfsserver in the same cluster. Older NFS services required the NFS daemons to be running; nfsserver requires the daemons to be stopped when the service is started.

When using multiple file systems, you will be unable to use inheritance for the exports; thus reuse of nfscl i ent resources in services with multiple file systems is limited. You may, however, explicitly define target and path attributes for as many nfscl i ents as you like.

8.9. Verifying a Configurat ion

Once you have created your cluster configuration file, verify that it is running correctly by performing the following steps:

1. At each node, restart the cluster software. That action ensures that any configuration additions that are checked only at startup time are included in the running configuration. You can restart the cluster software by running servi ce cman restart. For example:

[root@ example-01 ~]# servi ce cman restart

Stopping cluster:

Leaving fence domain... [ OK ]

Stopping gfs_controld... [ OK ]

Stopping dlm_controld... [ OK ]

Stopping fenced... [ OK ]

Stopping cman... [ OK ]

Waiting for corosync to shutdown: [ OK ]

Unloading kernel modules... [ OK ]

Unmounting configfs... [ OK ]

Starting cluster:

Checking Network Manager... [ OK ]

Global setup... [ OK ]

Loading kernel modules... [ OK ]

Mounting configfs... [ OK ]

Starting cman... [ OK ]

Waiting for quorum... [ OK ]

132

⁠Chapt er 8 . Configuring Red Hat High Availabilit y Manually

Starting fenced... [ OK ]

Starting dlm_controld... [ OK ]

Starting gfs_controld... [ OK ]

Unfencing self... [ OK ]

Joining fence domain... [ OK ]

2. Run servi ce cl vmd start, if CLVM is being used to create clustered volumes. For example:

[root@ example-01 ~]# servi ce cl vmd start

Activating VGs: [ OK ]

3. Run servi ce g fs2 start, if you are using Red Hat GFS2. For example:

[root@ example-01 ~]# servi ce g fs2 start

Mounting GFS2 filesystem (/mnt/gfsA): [ OK ]

Mounting GFS2 filesystem (/mnt/gfsB): [ OK ]

4. Run servi ce rg manag er start, if you using high-availability (HA) services. For example:

[root@ example-01 ~]# servi ce rg manag er start

Starting Cluster Service Manager: [ OK ]

5. At any cluster node, run cman_to o l no d es to verify that the nodes are functioning as members in the cluster (signified as "M" in the status column, "Sts"). For example:

[root@ example-01 ~]# cman_to o l no d es

Node Sts Inc Joined Name

1 M 548 2010-09-28 10:52:21 node-01.example.com

2 M 548 2010-09-28 10:52:21 node-02.example.com

3 M 544 2010-09-28 10:52:21 node-03.example.com

6. At any node, using the cl ustat utility, verify that the HA services are running as expected. In addition, cl ustat displays status of the cluster nodes. For example:

[root@ example-01 ~]#cl ustat

Cluster Status for mycluster @ Wed Nov 17 05:40:00 2010

Member Status: Quorate

Member Name ID Status

------ ---- ---- ------

node-03.example.com 3 Online, rgmanager

node-02.example.com 2 Online, rgmanager

node-01.example.com 1 Online, Local, rgmanager

Service Name Owner (Last)

State

------- ---- ----- ------ ---

service:example_apache node-01.example.com started

service:example_apache2 (none) disabled

133

Clust er Administ rat ion

7. If the cluster is running as expected, you are done with creating a configuration file. You can

manage the cluster with command-line tools described in Chapter 9, Managing Red Hat High

Availability Add-On With Command Line Tools

134

⁠Chapt er 9 . Managing Red Hat High Availabilit y Add- O n Wit h Command Line T ools

Chapter 9. Managing Red Hat High Availability Add-On With

Command Line Tools

This chapter describes various administrative tasks for managing Red Hat High Availability Add-On and consists of the following sections:

Section 9.1, “ Starting and Stopping the Cluster Software”

Section 9.2, “ Deleting or Adding a Node”

Section 9.3, “ Managing High-Availability Services”

Section 9.4, “ Updating a Configuration”

Important

/usr/share/d o c/cman-X. Y . ZZ/cl uster_co nf. html (for example

/usr/share/d o c/cman-3. 0 . 12/cl uster_co nf. html ).

Important

Certain procedure in this chapter call for using the cman_to o l versi o n -r command to propagate a cluster configuration throughout a cluster. Using that command requires that

ri cci is running.

Note

Procedures in this chapter, may include specific commands for some of the command-line

tools listed in Appendix E, Command Line Tools Summary

. For more information about all commands and variables, refer to the man page for each command-line tool.

9.1. St art ing and St opping t he Clust er Soft ware

135

Clust er Administ rat ion

You can start or stop cluster software on a node according to Section 9.1.1, “ Starting Cluster

Software” and Section 9.1.2, “ Stopping Cluster Software” . Starting cluster software on a node

causes it to join the cluster; stopping the cluster software on a node causes it to leave the cluster.

9.1.1. St art ing Clust er Soft ware

To start the cluster software on a node, type the following commands in this order:

1. servi ce cman start

2. servi ce cl vmd start, if CLVM has been used to create clustered volumes

3. servi ce g fs2 start, if you are using Red Hat GFS2

4. servi ce rg manag er start, if you using high-availability (HA) services (rg manag er).

For example:

[root@ example-01 ~]# servi ce cman start

Starting cluster:

Checking Network Manager... [ OK ]

Global setup... [ OK ]

Loading kernel modules... [ OK ]

Mounting configfs... [ OK ]

Starting cman... [ OK ]

Waiting for quorum... [ OK ]

Starting fenced... [ OK ]

Starting dlm_controld... [ OK ]

Starting gfs_controld... [ OK ]

Unfencing self... [ OK ]

Joining fence domain... [ OK ]

[root@ example-01 ~]# servi ce cl vmd start

Starting clvmd: [ OK ]

Activating VG(s): 2 logical volume(s) in volume group "vg_example" now active

[ OK ]

[root@ example-01 ~]# servi ce g fs2 start

Mounting GFS2 filesystem (/mnt/gfsA): [ OK ]

Mounting GFS2 filesystem (/mnt/gfsB): [ OK ]

[root@ example-01 ~]# servi ce rg manag er start

Starting Cluster Service Manager: [ OK ]

[root@ example-01 ~]#

9.1.2. St opping Clust er Soft ware

To stop the cluster software on a node, type the following commands in this order:

1. servi ce rg manag er sto p, if you using high-availability (HA) services (rg manag er).

2. servi ce g fs2 sto p, if you are using Red Hat GFS2

3. umo unt -at g fs2, if you are using Red Hat GFS2 in conjunction with rg manag er, to ensure that any GFS2 files mounted during rg manag er startup (but not unmounted during shutdown) were also unmounted.

4. servi ce cl vmd sto p, if CLVM has been used to create clustered volumes

136

⁠Chapt er 9 . Managing Red Hat High Availabilit y Add- O n Wit h Command Line T ools

5. servi ce cman sto p

For example:

[root@ example-01 ~]# servi ce rg manag er sto p

Stopping Cluster Service Manager: [ OK ]

[root@ example-01 ~]# servi ce g fs2 sto p

Unmounting GFS2 filesystem (/mnt/gfsA): [ OK ]

Unmounting GFS2 filesystem (/mnt/gfsB): [ OK ]

[root@ example-01 ~]# umo unt -at g fs2

[root@ example-01 ~]# servi ce cl vmd sto p

Signaling clvmd to exit [ OK ] clvmd terminated [ OK ]

[root@ example-01 ~]# servi ce cman sto p

Stopping cluster:

Leaving fence domain... [ OK ]

Stopping gfs_controld... [ OK ]

Stopping dlm_controld... [ OK ]

Stopping fenced... [ OK ]

Stopping cman... [ OK ]

Waiting for corosync to shutdown: [ OK ]

Unloading kernel modules... [ OK ]

Unmounting configfs... [ OK ]

[root@ example-01 ~]#

Note

Stopping cluster software on a node causes its HA services to fail over to another node. As an alternative to that, consider relocating or migrating HA services to another node before

stopping cluster software. For information about managing HA services, refer to Section 9.3,

“ Managing High-Availability Services” .

9.2. Delet ing or Adding a Node

This section describes how to delete a node from a cluster and add a node to a cluster. You can delete a node from a cluster according to

Section 9.2.1, “ Deleting a Node from a Cluster” ; you can

add a node to a cluster according to

Section 9.2.2, “ Adding a Node to a Cluster” .

9.2.1. Delet ing a Node from a Clust er

Deleting a node from a cluster consists of shutting down the cluster software on the node to be deleted and updating the cluster configuration to reflect the change.

Important

If deleting a node from the cluster causes a transition from greater than two nodes to two nodes, you must restart the cluster software at each node after updating the cluster configuration file.

To delete a node from a cluster, perform the following steps:

137

Clust er Administ rat ion

1. At any node, use the cl usvcad m utility to relocate, migrate, or stop each HA service running on the node that is being deleted from the cluster. For information about using cl usvcad m, refer to

Section 9.3, “ Managing High-Availability Services”

2. At the node to be deleted from the cluster, stop the cluster software according to Section 9.1.2,

“ Stopping Cluster Software” . For example:

[root@ example-01 ~]# servi ce rg manag er sto p

Stopping Cluster Service Manager: [ OK ]

[root@ example-01 ~]# servi ce g fs2 sto p

Unmounting GFS2 filesystem (/mnt/gfsA): [ OK ]

Unmounting GFS2 filesystem (/mnt/gfsB): [ OK ]

[root@ example-01 ~]# servi ce cl vmd sto p

Signaling clvmd to exit [ OK ] clvmd terminated [ OK ]

[root@ example-01 ~]# servi ce cman sto p

Stopping cluster:

Leaving fence domain... [ OK ]

Stopping gfs_controld... [ OK ]

Stopping dlm_controld... [ OK ]

Stopping fenced... [ OK ]

Stopping cman... [ OK ]

Waiting for corosync to shutdown: [ OK ]

Unloading kernel modules... [ OK ]

Unmounting configfs... [ OK ]

[root@ example-01 ~]#

3. At any node in the cluster, edit the /etc/cl uster/cl uster. co nf to remove the

cl usterno d e section of the node that is to be deleted. For example, in

Example 9.1, “ Threenode Cluster Configuration” , if node-03.example.com is supposed to be removed, then delete

the cl usterno d e section for that node. If removing a node (or nodes) causes the cluster to be a two-node cluster, you can add the following line to the configuration file to allow a single node to maintain quorum (for example, if one node fails):

<cman two _no d e= "1"

expected _vo tes= "1"/>

Refer to Section 9.2.3, “ Examples of Three-Node and Two-Node Configurations” for

comparison between a three-node and a two-node configuration.

4. Update the co nfi g _versi o n attribute by incrementing its value (for example, changing from co nfi g _versi o n= "2" to co nfi g _versi o n= "3">).

5. Save /etc/cl uster/cl uster. co nf.

6. (O pti o nal ) Validate the updated file against the cluster schema (cl uster. rng ) by running the ccs_co nfi g _val i d ate command. For example:

[root@ example-01 ~]# ccs_co nfi g _val i d ate

Configuration validates

7. Run the cman_to o l versi o n -r command to propagate the configuration to the rest of the cluster nodes.

8. Verify that the updated configuration file has been propagated.

138

⁠Chapt er 9 . Managing Red Hat High Availabilit y Add- O n Wit h Command Line T ools

9. If the node count of the cluster has transitioned from greater than two nodes to two nodes, you must restart the cluster software as follows:

a. At each node, stop the cluster software according to Section 9.1.2, “ Stopping Cluster

b. At each node, start the cluster software according to Section 9.1.1, “ Starting Cluster

[root@ example-01 ~]# servi ce rg manag er sto p

Stopping Cluster Service Manager: [

OK ]

[root@ example-01 ~]# servi ce g fs2 sto p

Unmounting GFS2 filesystem (/mnt/gfsA): [

OK ]

Unmounting GFS2 filesystem (/mnt/gfsB): [

OK ]

[root@ example-01 ~]# servi ce cl vmd sto p

Signaling clvmd to exit [

OK ] clvmd terminated [

OK ]

[root@ example-01 ~]# servi ce cman sto p

Stopping cluster:

Leaving fence domain... [

OK ]

Stopping gfs_controld... [

OK ]

Stopping dlm_controld... [

OK ]

Stopping fenced... [

OK ]

Stopping cman... [

OK ]

Waiting for corosync to shutdown: [

OK ]

Unloading kernel modules... [

OK ]

Unmounting configfs... [

OK ]

[root@ example-01 ~]#

section for the node that is to be added. For example, in Example 9.2, “ Two-node Cluster

[root@ example-01 ~]# servi ce cman start

Starting cluster:

Checking Network Manager... [

OK ]

Global setup... [

OK ]

Loading kernel modules... [

OK ]

Mounting configfs... [

OK ]

Starting cman... [

OK ]

Waiting for quorum... [

OK ]

139

Clust er Administ rat ion

Starting fenced... [

OK ]

Starting dlm_controld... [

OK ]

Starting gfs_controld... [

OK ]

Unfencing self... [

OK ]

Joining fence domain... [

OK ]

[root@ example-01 ~]# servi ce cl vmd start

Starting clvmd: [

OK ]

Activating VG(s): 2 logical volume(s) in volume group

"vg_example" now active

[

OK ]

[root@ example-01 ~]# servi ce g fs2 start

Mounting GFS2 filesystem (/mnt/gfsA): [

OK ]

Mounting GFS2 filesystem (/mnt/gfsB): [

OK ]

[root@ example-01 ~]# servi ce rg manag er start

Starting Cluster Service Manager: [

OK ]

[root@ example-01 ~]# c. At any cluster node, run cman_to o l no d es to verify that the nodes are functioning as members in the cluster (signified as "M" in the status column, "Sts"). For example:

[root@ example-01 ~]# cman_to o l no d es

Node Sts Inc Joined Name

1 M 548 2010-09-28 10:52:21 node-01.example.com

2 M 548 2010-09-28 10:52:21 node-02.example.com

d. At any node, using the cl ustat utility, verify that the HA services are running as expected. In addition, cl ustat displays status of the cluster nodes. For example:

[root@ example-01 ~]#cl ustat

Cluster Status for mycluster @ Wed Nov 17 05:40:00 2010

Member Status: Quorate

Member Name ID Status

------ ---- ---- ------

node-02.example.com 2 Online, rgmanager

node-01.example.com 1 Online, Local, rgmanager

Service Name Owner (Last)

State

------- ---- ----- ------

-----

14 0

⁠Chapt er 9 . Managing Red Hat High Availabilit y Add- O n Wit h Command Line T ools

service:example_apache node-01.example.com started

service:example_apache2 (none) disabled

9.2.2. Adding a Node t o a Clust er

Adding a node to a cluster consists of updating the cluster configuration, propagating the updated configuration to the node to be added, and starting the cluster software on that node. To add a node to a cluster, perform the following steps:

1. At any node in the cluster, edit the /etc/cl uster/cl uster. co nf to add a cl usterno d e

Configuration” , if node-03.example.com is supposed to be added, then add a cl usterno d e

section for that node. If adding a node (or nodes) causes the cluster to transition from a twonode cluster to a cluster with three or more nodes, remove the following cman attributes from

/etc/cl uster/cl uster. co nf:

cman two _no d e= "1" expected _vo tes= "1"

Refer to Section 9.2.3, “ Examples of Three-Node and Two-Node Configurations” for

comparison between a three-node and a two-node configuration.

2. Update the co nfi g _versi o n attribute by incrementing its value (for example, changing from co nfi g _versi o n= "2" to co nfi g _versi o n= "3">).

3. Save /etc/cl uster/cl uster. co nf.

4. (O pti o nal ) Validate the updated file against the cluster schema (cl uster. rng ) by running the ccs_co nfi g _val i d ate command. For example:

[root@ example-01 ~]# ccs_co nfi g _val i d ate

Configuration validates

5. Run the cman_to o l versi o n -r command to propagate the configuration to the rest of the cluster nodes.

6. Verify that the updated configuration file has been propagated.

7. Propagate the updated configuration file to /etc/cl uster/ in each node to be added to the cluster. For example, use the scp command to send the updated configuration file to each node to be added to the cluster.

8. If the node count of the cluster has transitioned from two nodes to greater than two nodes, you must restart the cluster software in the existing cluster nodes as follows:

a. At each node, stop the cluster software according to Section 9.1.2, “ Stopping Cluster

b. At each node, start the cluster software according to Section 9.1.1, “ Starting Cluster

[root@ example-01 ~]# servi ce rg manag er sto p

Stopping Cluster Service Manager: [

OK ]

[root@ example-01 ~]# servi ce g fs2 sto p

Unmounting GFS2 filesystem (/mnt/gfsA): [

OK ]

14 1

Clust er Administ rat ion

Unmounting GFS2 filesystem (/mnt/gfsB): [

OK ]

[root@ example-01 ~]# servi ce cl vmd sto p

Signaling clvmd to exit [

OK ] clvmd terminated [

OK ]

[root@ example-01 ~]# servi ce cman sto p

Stopping cluster:

Leaving fence domain... [

OK ]

Stopping gfs_controld... [

OK ]

Stopping dlm_controld... [

OK ]

Stopping fenced... [

OK ]

Stopping cman... [

OK ]

Waiting for corosync to shutdown: [

OK ]

Unloading kernel modules... [

OK ]

Unmounting configfs... [

OK ]

[root@ example-01 ~]#

9. At each node to be added to the cluster, start the cluster software according to Section 9.1.1,

[root@ example-01 ~]# servi ce cman start

Starting cluster:

Checking Network Manager... [

OK ]

Global setup... [

OK ]

Loading kernel modules... [

OK ]

Mounting configfs... [

OK ]

Starting cman... [

OK ]

Waiting for quorum... [

OK ]

Starting fenced... [

OK ]

Starting dlm_controld... [

OK ]

Starting gfs_controld... [

OK ]

Unfencing self... [

OK ]

Joining fence domain... [

OK ]

[root@ example-01 ~]# servi ce cl vmd start

Starting clvmd: [

14 2

⁠Chapt er 9 . Managing Red Hat High Availabilit y Add- O n Wit h Command Line T ools

OK ]

Activating VG(s): 2 logical volume(s) in volume group

"vg_example" now active

[

OK ]

[root@ example-01 ~]# servi ce g fs2 start

Mounting GFS2 filesystem (/mnt/gfsA): [

OK ]

Mounting GFS2 filesystem (/mnt/gfsB): [

OK ]

[root@ example-01 ~]# servi ce rg manag er start

Starting Cluster Service Manager: [

OK ]

[root@ example-01 ~]#

“ Starting Cluster Software” . For example:

[root@ example-01 ~]# servi ce cman start

Starting cluster:

Checking Network Manager... [ OK ]

Global setup... [ OK ]

Loading kernel modules... [ OK ]

Mounting configfs... [ OK ]

Starting cman... [ OK ]

Waiting for quorum... [ OK ]

Starting fenced... [ OK ]

Starting dlm_controld... [ OK ]

Starting gfs_controld... [ OK ]

Unfencing self... [ OK ]

Joining fence domain... [ OK ]

[root@ example-01 ~]# servi ce cl vmd start

Starting clvmd: [ OK ]

Activating VG(s): 2 logical volume(s) in volume group

"vg_example" now active

[ OK ]

[root@ example-01 ~]# servi ce g fs2 start

Mounting GFS2 filesystem (/mnt/gfsA): [ OK ]

Mounting GFS2 filesystem (/mnt/gfsB): [ OK ]

[root@ example-01 ~]# servi ce rg manag er start

Starting Cluster Service Manager: [ OK ]

[root@ example-01 ~]#

10. At any node, using the cl ustat utility, verify that each added node is running and part of the cluster. For example:

[root@ example-01 ~]#cl ustat

Cluster Status for mycluster @ Wed Nov 17 05:40:00 2010

Member Status: Quorate

Member Name ID Status

------ ---- ---- ------

node-03.example.com 3 Online, rgmanager

node-02.example.com 2 Online, rgmanager

14 3

Clust er Administ rat ion

node-01.example.com 1 Online, Local, rgmanager

Service Name Owner (Last)

State

------- ---- ----- ------ ---

service:example_apache node-01.example.com started

service:example_apache2 (none) disabled

For information about using cl ustat, refer to Section 9.3, “ Managing High-Availability

Services” .

In addition, you can use cman_to o l status to verify node votes, node count, and quorum count. For example:

[root@ example-01 ~]#cman_to o l status

Version: 6.2.0

Config Version: 19

Cluster Name: mycluster

Cluster Id: 3794

Cluster Member: Yes

Cluster Generation: 548

Membership state: Cluster-Member

Nodes: 3

Expected votes: 3

Total votes: 3

Node votes: 1

Quorum: 2

Active subsystems: 9

Flags:

Ports Bound: 0 11 177

Node name: node-01.example.com

Node ID: 3

Multicast addresses: 239.192.14.224

Node addresses: 10.15.90.58

11. At any node, you can use the cl usvcad m utility to migrate or relocate a running service to the newly joined node. Also, you can enable any disabled services. For information about

using cl usvcad m, refer to Section 9.3, “ Managing High-Availability Services”

9.2.3. Examples of T hree-Node and T wo-Node Configurat ions

Refer to the examples that follow for comparison between a three-node and a two-node configuration.

Examp le 9 .1. T h ree- n o d e Clu st er Co n f ig u rat io n

14 4

<cman/>

⁠Chapt er 9 . Managing Red Hat High Availabilit y Add- O n Wit h Command Line T ools

<fence>

</method>

</fence>

</clusternode>

<fence>

</method>

</fence>

</clusternode>

<fence>

</method>

</fence>

</clusternode>

</clusternodes>

</fencedevices>

<rm>

</failoverdomain>

</failoverdomains>

</fs>

</ip>

</resources>

</service>

<fs name="web_fs2" device="/dev/sdd3" mountpoint="/var/www"

14 5

Clust er Administ rat ion

fstype="ext3"/>

</service>

</rm>

</cluster>

Examp le 9 .2. T wo - n o d e Clu st er Co n f ig u rat io n

14 6

<fence>

</method>

</fence>

</clusternode>

<fence>

</method>

</fence>

</clusternodes>

</fencedevices>

<rm>

</failoverdomain>

</failoverdomains>

</fs>

</ip>

</resources>

⁠Chapt er 9 . Managing Red Hat High Availabilit y Add- O n Wit h Command Line T ools

</service>

</service>

</rm>

</cluster>

9.3. Managing High-Availabilit y Services

You can manage high-availability services using the Clu st er St at u s Ut ilit y, cl ustat, and the

Clu st er User Service Ad min ist rat io n Ut ilit y, cl usvcad m. cl ustat displays the status of a cluster and cl usvcad m provides the means to manage high-availability services.

This section provides basic information about managing HA services using the cl ustat and

cl usvcad m commands. It consists of the following subsections:

Section 9.3.1, “ Displaying HA Service Status with cl ustat”

Section 9.3.2, “ Managing HA Services with cl usvcad m”

9.3.1. Displaying HA Service St at us wit h

cl ustat

cl ustat displays cluster-wide status. It shows membership information, quorum view, the state of all high-availability services, and indicates which node the cl ustat command is being run at (Local).

Table 9.1, “ Services Status”

describes the states that services can be in and are displayed when

running cl ustat. Example 9.3, “ cl ustat Display”

shows an example of a cl ustat display. For more detailed information about running the cl ustat command refer to the cl ustat man page.

T ab le 9 .1. Services St at u s

Services St at u s

St art ed

Reco verin g

Disab led

St o p p ed

Descrip t io n

The service resources are configured and available on the cluster system that owns the service.

The service is pending start on another node.

The service has been disabled, and does not have an assigned owner. A disabled service is never restarted automatically by the cluster.

In the stopped state, the service will be evaluated for starting after the next service or node transition. This is a temporary state. You may disable or enable the service from this state.

14 7

Clust er Administ rat ion

Services St at u s

Failed

Un in it ializ ed

Descrip t io n

The service is presumed dead. A service is placed into this state whenever a resource's stop operation fails. After a service is placed into this state, you must verify that there are no resources allocated (mounted file systems, for example) prior to issuing a d i sabl e request. The only operation that can take place when a service has entered this state is

d i sabl e.

This state can appear in certain cases during startup and running

cl ustat -f.

Examp le 9 .3. cl ustat Disp lay

[root@ example-01 ~]#cl ustat

Cluster Status for mycluster @ Wed Nov 17 05:40:15 2010

Member Status: Quorate

Member Name ID Status

------ ---- ---- ------

node-03.example.com 3 Online, rgmanager

node-02.example.com 2 Online, rgmanager

node-01.example.com 1 Online, Local, rgmanager

Service Name Owner (Last) State

------- ---- ----- ------ -----

service:example_apache node-01.example.com started

service:example_apache2 (none) disabled

9.3.2. Managing HA Services wit h

cl usvcad m

You can manage HA services using the cl usvcad m command. With it you can perform the following operations:

Enable and start a service.

Disable a service.

Stop a service.

Freeze a service

Unfreeze a service

Migrate a service (for virtual machine services only)

Relocate a service.

Restart a service.

Table 9.2, “ Service Operations”

describes the operations in more detail. For a complete description on how to perform those operations, refer to the cl usvcad m utility man page.

T ab le 9 .2. Service O p erat io n s

14 8

⁠Chapt er 9 . Managing Red Hat High Availabilit y Add- O n Wit h Command Line T ools

Service

O p erat io n

Descrip t io n

En ab le

Disab le

Start the service, optionally on a preferred target and optionally according to failover domain rules. In the absence of either a preferred target or failover domain rules, the local host where

cl usvcad m is run will start the service. If the original start fails, the service behaves as though a relocate operation was requested (refer to Relo cat e in this table). If the operation succeeds, the service is placed in the started state.

Stop the service and place into the disabled state. This is the only permissible operation when a service is in the failed state.

Relo cat e

Move the service to another node.

Optionally, you may specify a preferred node to receive the service, but the inability of the service to run on that host

(for example, if the service fails to start or the host is offline) does not prevent relocation, and another node is chosen.

rg manag er attempts to start the service on every permissible node in the cluster.

If no permissible target node in the cluster successfully starts the service, the relocation fails and the service is attempted to be restarted on the original owner. If the original owner cannot restart the service, the service is placed in the

stopped state.

St o p

Freez e

Stop the service and place into the

stopped state.

Freeze a service on the node where it is currently running. This prevents status checks of the service as well as failover in the event the node fails or rgmanager is stopped. This can be used to suspend a service to allow maintenance of

underlying resources. Refer to the section called “ Considerations for Using the

Freeze and Unfreeze Operations” for

important information about using the

freeze and unfreeze operations.

Un f reez e

Unfreeze takes a service out of the freeze state. This re-enables status checks.

Refer to the section called

“ Considerations for Using the Freeze and

Unfreeze Operations” for important

information about using the freeze and

unfreeze operations.

Co mman d Syn t ax cl usvcad m -e

<servi ce_name> or cl usvcad m

-e

<servi ce_name> -m <member>

(Using the -m option specifies the preferred target member on which to start the service.)

cl usvcad m -d

<servi ce_name> cl usvcad m -r

<servi ce_name> or cl usvcad m

-r

<servi ce_name> -m <member>

(Using the -m option specifies the preferred target member on which to start the service.)

cl usvcad m -s

<servi ce_name> cl usvcad m -Z <servi ce_name> cl usvcad m -U

<servi ce_name>

14 9

Clust er Administ rat ion

Service

O p erat io n

Mig rat e

Descrip t io n Co mman d Syn t ax

Migrate a virtual machine to another node. You must specify a target node.

Depending on the failure, a failure to migrate may result with the virtual machine in the failed state or in the started state on the original owner.

cl usvcad m -M

<servi ce_name> -m <member>

Important

For the migrate operation, you

must specify a target node using the -m <member> option.

Rest art

Restart a service on the node where it is currently running.

cl usvcad m -R

<servi ce_name>

Co nside rat io ns fo r Using t he Fre e ze and Unfre e ze Ope rat io ns

Using the freeze operation allows maintenance of parts of rg manag er services. For example, if you have a database and a web server in one rg manag er service, you may freeze the rg manag er service, stop the database, perform maintenance, restart the database, and unfreeze the service.

When a service is frozen, it behaves as follows:

Status checks are disabled.

Start operations are disabled.

Stop operations are disabled.

Failover will not occur (even if you power off the service owner).

Important

Failure to follow these guidelines may result in resources being allocated on multiple hosts:

You must not stop all instances of rgmanager when a service is frozen unless you plan to reboot the hosts prior to restarting rgmanager.

You must not unfreeze a service until the reported owner of the service rejoins the cluster and restarts rgmanager.

9.4 . Updat ing a Configurat ion

Updating the cluster configuration consists of editing the cluster configuration file

(/etc/cl uster/cl uster. co nf) and propagating it to each node in the cluster. You can update the configuration using either of the following procedures:

Section 9.4.1, “ Updating a Configuration Using cman_to o l versi o n -r”

Section 9.4.2, “ Updating a Configuration Using scp”

9.4 .1. Updat ing a Configurat ion Using

cman_to o l versi o n -r

150

⁠Chapt er 9 . Managing Red Hat High Availabilit y Add- O n Wit h Command Line T ools

To update the configuration using the cman_to o l versi o n -r command, perform the following steps:

1. At any node in the cluster, edit the /etc/cl uster/cl uster. co nf file.

2. Update the co nfi g _versi o n attribute by incrementing its value (for example, changing from co nfi g _versi o n= "2" to co nfi g _versi o n= "3").

3. Save /etc/cl uster/cl uster. co nf.

4. Run the cman_to o l versi o n -r command to propagate the configuration to the rest of the cluster nodes. It is necessary that ri cci be running in each cluster node to be able to propagate updated cluster configuration information.

5. Verify that the updated cl uster. co nf configuration file has been propagated. If not, use the

scp command to propagate it to /etc/cl uster/ in each cluster node.

6. You may skip this step (restarting cluster software) if you have made only the following configuration changes:

Deleting a node from the cluster configuration—except where the node count changes from greater than two nodes to two nodes. For information about deleting a node from a

cluster and transitioning from greater than two nodes to two nodes, refer to Section 9.2,

“ Deleting or Adding a Node” .

Adding a node to the cluster configuration—except where the node count changes from two nodes to greater than two nodes. For information about adding a node to a cluster

and transitioning from two nodes to greater than two nodes, refer to Section 9.2.2,

“ Adding a Node to a Cluster” .

Changes to how daemons log information.

HA service/VM maintenance (adding, editing, or deleting).

Resource maintenance (adding, editing, or deleting).

Failover domain maintenance (adding, editing, or deleting).

Otherwise, you must restart the cluster software as follows:

a. At each node, stop the cluster software according to Section 9.1.2, “ Stopping Cluster

b. At each node, start the cluster software according to Section 9.1.1, “ Starting Cluster

cluster and transitioning from greater than two nodes to two nodes, refer to Section 9.2,

Stopping and starting the cluster software ensures that any configuration changes that are checked only at startup time are included in the running configuration.

7. At any cluster node, run cman_to o l no d es to verify that the nodes are functioning as members in the cluster (signified as "M" in the status column, "Sts"). For example:

[root@ example-01 ~]# cman_to o l no d es

Node Sts Inc Joined Name

1 M 548 2010-09-28 10:52:21 node-01.example.com

2 M 548 2010-09-28 10:52:21 node-02.example.com

3 M 544 2010-09-28 10:52:21 node-03.example.com

151

Clust er Administ rat ion

8. At any node, using the cl ustat utility, verify that the HA services are running as expected. In addition, cl ustat displays status of the cluster nodes. For example:

[root@ example-01 ~]#cl ustat

Cluster Status for mycluster @ Wed Nov 17 05:40:00 2010

Member Status: Quorate

Member Name ID Status

------ ---- ---- ------

node-03.example.com 3 Online, rgmanager

node-02.example.com 2 Online, rgmanager

node-01.example.com 1 Online, Local, rgmanager

Service Name Owner (Last)

State

------- ---- ----- ------ ---

service:example_apache node-01.example.com started

service:example_apache2 (none) disabled

9. If the cluster is running as expected, you are done updating the configuration.

9.4 .2. Updat ing a Configurat ion Using

scp

To update the configuration using the scp command, perform the following steps:

1. At any node in the cluster, edit the /etc/cl uster/cl uster. co nf file.

2. Update the co nfi g _versi o n attribute by incrementing its value (for example, changing from co nfi g _versi o n= "2" to co nfi g _versi o n= "3">).

3. Save /etc/cl uster/cl uster. co nf.

4. Validate the updated file against the cluster schema (cl uster. rng ) by running the

ccs_co nfi g _val i d ate command. For example:

[root@ example-01 ~]# ccs_co nfi g _val i d ate

Configuration validates

5. If the updated file is valid, use the scp command to propagate it to /etc/cl uster/ in each cluster node.

6. Verify that the updated configuration file has been propagated.

7. To reload the new configuration, execute the following command on one of the cluster nodes: cman_tool version -r

If you do not have ri cci installed, you can use the following command: cman_tool version -s

152

⁠Chapt er 9 . Managing Red Hat High Availabilit y Add- O n Wit h Command Line T ools

8. You may skip this step (restarting cluster software) if you have made only the following configuration changes:

Deleting a node from the cluster configuration—except where the node count changes from greater than two nodes to two nodes. For information about deleting a node from a

“ Deleting or Adding a Node” .

Adding a node to the cluster configuration—except where the node count changes from two nodes to greater than two nodes. For information about adding a node to a cluster

and transitioning from two nodes to greater than two nodes, refer to Section 9.2.2,

“ Adding a Node to a Cluster” .

Changes to how daemons log information.

HA service/VM maintenance (adding, editing, or deleting).

Resource maintenance (adding, editing, or deleting).

Failover domain maintenance (adding, editing, or deleting).

Otherwise, you must restart the cluster software as follows:

a. At each node, stop the cluster software according to Section 9.1.2, “ Stopping Cluster

b. At each node, start the cluster software according to Section 9.1.1, “ Starting Cluster

Section 6.15, “ Propagating the Configuration File to the Cluster Nodes”

Stopping and starting the cluster software ensures that any configuration changes that are checked only at startup time are included in the running configuration.

9. Verify that that the nodes are functioning as members in the cluster and that the HA services are running as expected.

a. At any cluster node, run cman_to o l no d es to verify that the nodes are functioning as members in the cluster (signified as "M" in the status column, "Sts"). For example:

[root@ example-01 ~]# cman_to o l no d es

Node Sts Inc Joined Name

1 M 548 2010-09-28 10:52:21 node-01.example.com

2 M 548 2010-09-28 10:52:21 node-02.example.com

3 M 544 2010-09-28 10:52:21 node-03.example.com

b. At any node, using the cl ustat utility, verify that the HA services are running as expected. In addition, cl ustat displays status of the cluster nodes. For example:

[root@ example-01 ~]#cl ustat

Cluster Status for mycluster @ Wed Nov 17 05:40:00 2010

Member Status: Quorate

Member Name ID Status

------ ---- ---- ------

node-03.example.com 3 Online, rgmanager

node-02.example.com 2 Online, rgmanager

node-01.example.com 1 Online, Local,

153

Clust er Administ rat ion

rgmanager

Service Name Owner (Last)

State

------- ---- ----- ------

-----

service:example_apache node-01.example.com started

service:example_apache2 (none) disabled

If the cluster is running as expected, you are done updating the configuration.

154

⁠Chapt er 1 0 . Diagnosing and Correct ing Problems in a Clust er

Chapter 10. Diagnosing and Correcting Problems in a Cluster

Clusters problems, by nature, can be difficult to troubleshoot. This is due to the increased complexity that a cluster of systems introduces as opposed to diagnosing issues on a single system. However, there are common issues that system administrators are more likely to encounter when deploying or administering a cluster. Understanding how to tackle those common issues can help make deploying and administering a cluster much easier.

This chapter provides information about some common cluster issues and how to troubleshoot them.

Additional help can be found in our knowledge base and by contacting an authorized Red Hat support representative. If your issue is related to the GFS2 file system specifically, you can find information about troubleshooting common GFS2 issues in the Global File System 2 document.

10.1. Configurat ion Changes Do Not T ake Effect

When you make changes to a cluster configuration, you must propagate those changes to every node in the cluster.

When you configure a cluster using Co n g a, Co n g a propagates the changes automatically when you apply the changes.

For information on propagating changes to cluster configuration with the ccs command, refer to

For information on propagating changes to cluster configuration with command line tools, refer to

Section 9.4, “ Updating a Configuration” .

If you make any of the following configuration changes to your cluster, it is not necessary to restart the cluster after propagating those changes the changes to take effect.

Deleting a node from the cluster configuration—except where the node count changes from greater than two nodes to two nodes.

Adding a node to the cluster configuration—except where the node count changes from two nodes to greater than two nodes.

Changing the logging settings.

Adding, editing, or deleting HA services or VM components.

Adding, editing, or deleting cluster resources.

Adding, editing, or deleting failover domains.

If you make any other configuration changes to your cluster, however, you must restart the cluster to implement those changes. The following cluster configuration changes require a cluster restart to take effect:

Adding or removing the two _no d e option from the cluster configuration file.

Renaming the cluster.

Changing any co ro sync or o penai s timers.

Adding, changing, or deleting heuristics for quorum disk, changing any quorum disk timers, or changing the quorum disk device. For these changes to take effect, a global restart of the

q d i skd daemon is required.

155

Clust er Administ rat ion

Changing the central _pro cessi ng mode for rg manag er. For this change to take effect, a global restart of rg manag er is required.

Changing the multicast address.

Switching the transport mode from UDP multicast to UDP unicast, or switching from UDP unicast to UDP multicast.

You can restart the cluster using Co n g a, the ccs command, or command line tools,

For information on restarting a cluster with Co n g a, refer to Section 5.4, “ Starting, Stopping,

Restarting, and Deleting Clusters” .

For information on restarting a cluster with the ccs command, refer to Section 7.2, “ Starting and

Stopping a Cluster” .

For information on restarting a cluster with command line tools, refer to Section 9.1, “ Starting and

Stopping the Cluster Software” .

10.2. Clust er Does Not Form

If you find you are having trouble getting a new cluster to form, check for the following things:

Make sure you have name resolution set up correctly. The cluster node name in the

cl uster. co nf file should correspond to the name used to resolve that cluster's address over the network that cluster will be using to communicate. For example, if your cluster's node names are no d ea and no d eb make sure both nodes have entries in the

/etc/cl uster/cl uster. co nf file and /etc/ho sts file that match those names.

If the cluster uses multicast for communication between nodes, make sure that multicast traffic is not being blocked, delayed, or otherwise interfered with on the network that the cluster is using to communicate. Note that some Cisco switches have features that may cause delays in multicast traffic.

Use tel net or SSH to verify whether you can reach remote nodes.

Execute the ethto o l eth1 | g rep l i nk command to check whether the ethernet link is up.

Use the tcpd ump command at each node to check the network traffic.

Ensure that you do not have firewall rules blocking communication between your nodes.

Ensure that the interfaces the cluster uses for inter-node communication are not using any bonding mode other than 0, 1, or 2. (Bonding modes 0 and 2 are supported as of Red Hat

Enterprise Linux 6.4.)

10.3. Nodes Unable t o Rejoin Clust er aft er Fence or Reboot

If your nodes do not rejoin the cluster after a fence or reboot, check for the following things:

Clusters that are passing their traffic through a Cisco Catalyst switch may experience this problem.

Ensure that all cluster nodes have the same version of the cl uster. co nf file. If the

cl uster. co nf file is different on any of the nodes, then nodes may be unable to join the cluster post fence.

156

⁠Chapt er 1 0 . Diagnosing and Correct ing Problems in a Clust er

As of Red Hat Enterprise Linux 6.1, you can use the following command to verify that all of the nodes specified in the host's cluster configuration file have the identical cluster configuration file: ccs -h host --checkconf

For information on the ccs command, see Chapter 6, Configuring Red Hat High Availability Add-On

With the ccs Command

and Chapter 7, Managing Red Hat High Availability Add-On With ccs .

Make sure that you have configured chkco nfi g o n for cluster services in the node that is attempting to join the cluster.

Ensure that no firewall rules are blocking the node from communicating with other nodes in the cluster.

10.4 . Clust er Daemon crashes

RGManager has a watchdog process that reboots the host if the main rg manag er process fails unexpectedly. This causes the cluster node to get fenced and rg manag er to recover the service on another host. When the watchdog daemon detects that the main rg manag er process has crashed then it will reboot the cluster node, and the active cluster nodes will detect that the cluster node has left and evict it from the cluster.

The lower number process ID (PID) is the watchdog process that takes action if its child (the process with the higher PID number) crashes. Capturing the core of the process with the higher PID number using g co re can aid in troubleshooting a crashed daemon.

Install the packages that are required to capture and view the core, and ensure that both the

rg manag er and rg manag er-d ebug i nfo are the same version or the captured application core might be unusable.

$ yum -y --enablerepo=rhel-debuginfo install gdb rgmanager-debuginfo

10.4 .1. Capt uring t he

rg manag er

Core at Runt ime

There are two rg manag er processes that are running as it is started. You must capture the core for the rg manag er process with the higher PID.

The following is an example output from the ps command showing two processes for rg manag er.

$ ps aux | grep rgmanager | grep -v grep root 22482 0.0 0.5 23544 5136 ? S<Ls Dec01 0:00 rgmanager root 22483 0.0 0.2 78372 2060 ? S<l Dec01 0:47 rgmanager

In the following example, the pi d o f program is used to automatically determine the highernumbered pid, which is the appropriate pid to create the core. The full command captures the application core for the process 22483 which has the higher pid number.

$ gcore -o /tmp/rgmanager-$(date '+%F_%s').core $(pidof -s rgmanager)

10.4 .2. Capt uring t he Core When t he Daemon Crashes

157

Clust er Administ rat ion

By default, the /etc/i ni t. d /functi o ns script blocks core files from daemons called by

/etc/i ni t. d /rg manag er. For the daemon to create application cores, you must enable that option. This procedure must be done on all cluster nodes that need an application core caught.

For creating a core file when the rgmanager daemon crashes, edit the /etc/sysco nfi g /cl uster file. The DAEMONCOREFILELIMIT parameter allows the daemon to create core files if the process crashes. There is a -w option that prevents the watchdog process from running. The watchdog daemon is responsible for rebooting the cluster node if rg manag er crashes and, in some cases, if the watchdog daemon is running then the core file will not be generated, so it must be disabled to capture core files.

DAEMONCOREFILELIMIT="unlimited"

RGMGR_OPTS="-w"

Restart rgmanager to activate the new configuration options: service rgmanager restart

Note

If cluster services are running on this cluster node, then it could leave the running services in a bad state.

The core file will be written when it is generated from a crash of the rg manag er process.

ls /core*

The output should appear similar to the following:

/core.11926

Move or delete any old cores files under the / directory before restarting rg manag er to capture the application core. The cluster node that experienced the rg manag er crash should be rebooted or fenced after the core is captured to ensure that the watchdog process was not running.

10.4 .3. Recording a

g d b

Backt race Session

Once you have captured the core file, you can view its contents by using g d b, the GNU Debugger.

To record a script session of g d b on the core file from the affected system, run the following:

$ script /tmp/gdb-rgmanager.txt

$ gdb /usr/sbin/rgmanager /tmp/rgmanager-.core.

This will start a g d b session, while scri pt records it to the appropriate text file. While in g d b, run the following commands:

(gdb) thread apply all bt full

(gdb) quit

Press ctrl -D to stop the script session and save it to the text file.

158

⁠Chapt er 1 0 . Diagnosing and Correct ing Problems in a Clust er

10.5. Clust er Services Hang

When the cluster services attempt to fence a node, the cluster services stop until the fence operation has successfully completed. Therefore, if your cluster-controlled storage or services hang and the cluster nodes show different views of cluster membership or if your cluster hangs when you try to fence a node and you need to reboot nodes to recover, check for the following conditions:

The cluster may have attempted to fence a node and the fence operation may have failed.

Look through the /var/l o g /messag es file on all nodes and see if there are any failed fence messages. If so, then reboot the nodes in the cluster and configure fencing correctly.

Verify that a network partition did not occur, as described in Section 10.8, “ Each Node in a Two-

Node Cluster Reports Second Node Down” . and verify that communication between nodes is still

possible and that the network is up.

If nodes leave the cluster the remaining nodes may be inquorate. The cluster needs to be quorate to operate. If nodes are removed such that the cluster is no longer quorate then services and storage will hang. Either adjust the expected votes or return the required amount of nodes to the cluster.

Note

You can fence a node manually with the fence_no d e command or with Co n g a. For

information, see the fence_no d e man page and Section 5.3.2, “ Causing a Node to Leave or

Join a Cluster” .

10.6. Clust er Service Will Not St art

If a cluster-controlled service will not start, check for the following conditions.

There may be a syntax error in the service configuration in the cl uster. co nf file. You can use the rg _test command to validate the syntax in your configuration. If there are any configuration or syntax faults, the rg _test will inform you what the problem is.

$ rg _test test /etc/cl uster/cl uster. co nf start servi ce servicename

For more information on the rg _test command, see Section C.5, “ Debugging and Testing

Services and Resource Ordering” .

If the configuration is valid, then increase the resource group manager's logging and then read the messages logs to determine what is causing the service start to fail. You can increase the log level by adding the l o g l evel = "7" parameter to the rm tag in the cl uster. co nf file. You will then get increased verbosity in your messages logs with regards to starting, stopping, and migrating clustered services.

10.7. Clust er-Cont rolled Services Fails t o Migrat e

If a cluster-controlled service fails to migrate to another node but the service will start on some specific node, check for the following conditions.

159

Clust er Administ rat ion

Ensure that the resources required to run a given service are present on all nodes in the cluster that may be required to run that service. For example, if your clustered service assumes a script file in a specific location or a file system mounted at a specific mount point then you must ensure that those resources are available in the expected places on all nodes in the cluster.

Ensure that failover domains, service dependency, and service exclusivity are not configured in such a way that you are unable to migrate services to nodes as you'd expect.

If the service in question is a virtual machine resource, check the documentation to ensure that all of the correct configuration work has been completed.

Increase the resource group manager's logging, as described in Section 10.6, “ Cluster Service

Will Not Start” , and then read the messages logs to determine what is causing the service start to

fail to migrate.

10.8. Each Node in a T wo-Node Clust er Report s Second Node Down

If your cluster is a two-node cluster and each node reports that it is up but that the other node is down, this indicates that your cluster nodes are unable to communicate with each other via multicast over the cluster heartbeat network. This is known as "split brain" or a "network partition." To address

this, check the conditions outlined in Section 10.2, “ Cluster Does Not Form”

10.9. Nodes are Fenced on LUN Pat h Failure

If a node or nodes in your cluster get fenced whenever you have a LUN path failure, this may be a result of the use of a quorum disk over multipathed storage. If you are using a quorum disk, and your quorum disk is over multipathed storage, ensure that you have all of the correct timings set up to tolerate a path failure.

10.10. Quorum Disk Does Not Appear as Clust er Member

If you have configured your system to use a quorum disk but the quorum disk does not appear as a member of your cluster, you can perform the following steps.

Review the /var/l o g /cl uster/q d i skd . l o g file.

Run ps -ef | g rep q d i sk to determine if the process is running.

Ensure that <q uo rumd . . . > is configured correctly in the /etc/cl uster/cl uster. co nf file.

Enable debugging output for the q d i skd daemon.

For information on enabling debugging in the /etc/cl uster/cl uster. co nf file, see

Section 8.7, “ Configuring Debug Options”

For information on enabling debugging using l uci , see Section 4.5.6, “ Logging

For information on enabling debugging with the ccs command, see Section 6.14.4, “ Logging”

Note that it may take multiple minutes for the quorum disk to register with the cluster. This is normal and expected behavior.

10.11. Unusual Failover Behavior

160

⁠Chapt er 1 0 . Diagnosing and Correct ing Problems in a Clust er

A common problem with cluster servers is unusual failover behavior. Services will stop when other services start or services will refuse to start on failover. This can be due to having complex systems of failover consisting of failover domains, service dependency, and service exclusivity. Try scaling back to a simpler service or failover domain configuration and see if the issue persists. Avoid features like service exclusivity and dependency unless you fully understand how those features may effect failover under all conditions.

10.12. Fencing Occurs at Random

If you find that a node is being fenced at random, check for the following conditions.

The root cause of fences is always a node losing token, meaning that it lost communication with the rest of the cluster and stopped returning heartbeat.

Any situation that results in a system not returning heartbeat within the specified token interval could lead to a fence. By default the token interval is 10 seconds. It can be specified by adding the desired value (in milliseconds) to the token parameter of the totem tag in the cl uster. co nf file (for example, setting to tem to ken= "30 0 0 0 " for 30 seconds).

Ensure that the network is sound and working as expected.

Ensure that the interfaces the cluster uses for inter-node communication are not using any bonding mode other than 0, 1, or 2. (Bonding modes 0 and 2 are supported as of Red Hat

Enterprise Linux 6.4.)

Take measures to determine if the system is "freezing" or kernel panicking. Set up the kd ump utility and see if you get a core during one of these fences.

Make sure some situation is not arising that you are wrongly attributing to a fence, for example the quorum disk ejecting a node due to a storage failure or a third party product like Oracle RAC rebooting a node due to some outside condition. The messages logs are often very helpful in determining such problems. Whenever fences or node reboots occur it should be standard practice to inspect the messages logs of all nodes in the cluster from the time the reboot/fence occurred.

Thoroughly inspect the system for hardware faults that may lead to the system not responding to heartbeat when expected.

10.13. Debug Logging for Dist ribut ed Lock Manager (DLM) Needs t o be

Enabled

There are two debug options for the Distributed Lock Manager (DLM) that you can enable, if necessary: DLM kernel debugging, and POSIX lock debugging.

To enable DLM debugging, edit the /etc/cl uster/cl uster. co nf file to add configuration options to the d l m tag. The l o g _d ebug option enables DLM kernel debugging messages, and the

pl o ck_d ebug option enables POSIX lock debugging messages.

The following example section of a /etc/cl uster/cl uster. co nf file shows the d l m tag that enables both DLM debug options:

...

161

Clust er Administ rat ion

...

</cluster>

After editing the /etc/cl uster/cl uster. co nf file, run the cman_to o l versi o n -r command to propagate the configuration to the rest of the cluster nodes.

162

⁠Chapt er 1 1 . SNMP Configurat ion wit h t he Red Hat High Availabilit y Add- O n

Chapter 11. SNMP Configuration with the Red Hat High

Availability Add-On

As of the Red Hat Enterprise Linux 6.1 release and later, the Red Hat High Availability Add-On provides support for SNMP traps. This chapter describes how to configure your system for SNMP followed by a summary of the traps that the Red Hat High Availability Add-On emits for specific cluster events.

11.1. SNMP and t he Red Hat High Availabilit y Add-On

The Red Hat High Availability Add-On SNMP subagent is fo g ho rn, which emits the SNMP traps. The

fo g ho rn subagent talks to the snmpd daemon by means of the AgentX Protocol. The fo g ho rn subagent only creates SNMP traps; it does not support other SNMP operations such as g et or set.

There are currently no co nfi g options for the fo g ho rn subagent. It cannot be configured to use a specific socket; only the default AgentX socket is currently supported.

11.2. Configuring SNMP wit h t he Red Hat High Availabilit y Add-On

To configure SNMP with the Red Hat High Availability Add-On, perform the following steps on each node in the cluster to ensure that the necessary services are enabled and running.

1. To use SNMP traps with the Red Hat High Availability Add-On, the snmpd service is required and acts as the master agent. Since the fo g ho rn service is the subagent and uses the

AgentX protocol, you must add the following line to the /etc/snmp/snmpd . co nf file to enable AgentX support: master agentx

2. To specify the host where the SNMP trap notifications should be sent, add the following line to the to the /etc/snmp/snmpd . co nf file: trap2sink host

For more information on notification handling, see the snmpd . co nf man page.

3. Make sure that the snmpd daemon is enabled and running by executing the following commands:

# chkco nfi g snmpd o n

# servi ce snmpd start

4. If the messag ebus daemon is not already enabled and running, execute the following commands:

# chkco nfi g messag ebus o n

# servi ce messag ebus start

5. Make sure that the fo g ho rn daemon is enabled and running by executing the following commands:

163

Clust er Administ rat ion

# chkco nfi g fo g ho rn o n

# servi ce fo g ho rn start

6. Execute the following command to configure your system so that the C O R O SY NC -MIB generates SNMP traps and to ensure that the co ro sync-no ti fyd daemon is enabled and running:

# echo "O P T IO NS= \"-d \" " > /etc/sysco nfi g /co ro sync-no ti fyd

# chkco nfi g co ro sync-no ti fyd o n

# servi ce co ro sync-no ti fyd start

After you have configured each node in the cluster for SNMP and ensured that the necessary services are running, D-bus signals will be received by the fo g ho rn service and translated into

SNMPv2 traps. These traps are then passed to the host that you defined with the trapsi nk entry to receive SNMPv2 traps.

11.3. Forwarding SNMP t raps

It is possible to forward SNMP traps to a machine that is not part of the cluster where you can use the

snmptrapd daemon on the external machine and customize how to respond to the notifications.

Perform the following steps to forward SNMP traps in a cluster to a machine that is not one of the cluster nodes:

1. For each node in the cluster, follow the procedure described in Section 11.2, “ Configuring

SNMP with the Red Hat High Availability Add-On” , setting the trap2si nk host entry in the

/etc/snmp/snmpd . co nf file to specify the external host that will be running the snmptrapd daemon.

2. On the external host that will receive the traps, edit the /etc/snmp/snmptrapd . co nf configuration file to specify your community strings. For example, you can use the following entry to allow the snmptrapd daemon to process notifications using the publ i c community string.

authCommunity log,execute,net public

3. On the external host that will receive the traps, make sure that the snmptrapd daemon is enabled and running by executing the following commands:

# chkco nfi g snmptrapd o n

# servi ce snmptrapd start

For further information on processing SNMP notifications, see the snmptrapd . co nf man page.

11.4 . SNMP T raps Produced by Red Hat High Availabilit y Add-On

The fo g ho rn daemon generates the following traps:

fenceNo ti fyFenceNo d e

This trap occurs whenever a fenced node attempts to fence another node. Note that this trap is only generated on one node - the node that attempted to perform the fence operation. The notification includes the following fields:

164

⁠Chapt er 1 1 . SNMP Configurat ion wit h t he Red Hat High Availabilit y Add- O n

fenceNo d eName - name of the fenced node

fenceNo d eID - node id of the fenced node

fenceR esul t - the result of the fence operation (0 for success, -1 for something went wrong, -

2 for no fencing methods defined)

rg manag erServi ceStateC hang e

This trap occurs when the state of a cluster service changes. The notification includes the following fields:

rg manag erServi ceName - the name of the service, which includes the service type (for example, servi ce: fo o or vm: fo o ).

rg manag erServi ceState - the state of the service. This excludes transitional states such as

starti ng and sto ppi ng to reduce clutter in the traps.

rg manag erServi ceFl ag s - the service flags. There are currently two supported flags:

fro zen, indicating a service which has been frozen using cl usvcad m -Z, and parti al , indicating a service in which a failed resource has been flagged as no n-cri ti cal so that the resource may fail and its components manually restarted without the entire service being affected.

rg manag erServi ceC urrentO wner - the service owner. If the service is not running, this will be (no ne).

rg manag erServi ceP revi o usO wner - the last service owner, if known. If the last owner is not known, this may indicate (no ne).

The co ro sync-no d i fyd daemon generates the following traps:

co ro syncNo ti cesNo d eStatus

This trap occurs when a node joins or leaves the cluster. The notification includes the following fields:

co ro syncO bjectsNo d eName - node name

co ro syncO bjectsNo d eID - node id

co ro syncO bjectsNo d eAd d ress - node IP address

co ro syncO bjectsNo d eStatus - node status (jo i ned or l eft)

co ro syncNo ti cesQ uo rumStatus

This trap occurs when the quorum state changes. The notification includes the following fields:

co ro syncO bjectsNo d eName - node name

co ro syncO bjectsNo d eID - node id

co ro syncO bjectsQ uo rumStatus - new state of the quorum (q uo rate or NO T q uo rate)

co ro syncNo ti cesAppStatus

This trap occurs when a client application connects or disconnects from Corosync.

co ro syncO bjectsNo d eName - node name

co ro syncO bjectsNo d eID - node id

165

Clust er Administ rat ion

co ro syncO bjectsAppName - application name

co ro syncO bjectsAppStatus - new state of the application (co nnected or

d i sco nnected )

166

⁠Chapt er 1 2 . Clust ered Samba Configurat ion

Chapter 12. Clustered Samba Configuration

As of the Red Hat Enterprise Linux 6.2 release, the Red Hat High Availability Add-On provides support for running Clustered Samba in an active/active configuration. This requires that you install and configure CTDB on all nodes in a cluster, which you use in conjunction with GFS2 clustered file systems.

Note

Red Hat Enterprise Linux 6 supports a maximum of four nodes running clustered Samba.

This chapter describes the procedure for configuring CTDB by configuring an example system. For information on configuring GFS2 file systems, refer to Global File System 2. For information on configuring logical volumes, refer to Logical Volume Manager Administration.

Note

Simultaneous access to the data in the Samba share from outside of Samba is not supported.

12.1. CT DB Overview

CTDB is a cluster implementation of the TDB database used by Samba. To use CTDB, a clustered file system must be available and shared on all nodes in the cluster. CTDB provides clustered features on top of this clustered file system. As of the Red Hat Enterprise Linux 6.2 release, CTDB also runs a cluster stack in parallel to the one provided by Red Hat Enterprise Linux clustering. CTDB manages node membership, recovery/failover, IP relocation and Samba services.

12.2. Required Packages

In addition to the standard packages required to run the Red Hat High Availability Add-On and the

Red Hat Resilient Storage Add-On, running Samba with Red Hat Enterprise Linux clustering requires the following packages:

ctd b samba samba-co mmo n samba-wi nbi nd -cl i ents

12.3. GFS2 Configurat ion

Configuring Samba with the Red Hat Enterprise Linux clustering requires two GFS2 file systems: One small file system for CTDB, and a second file system for the Samba share. This example shows how to create the two GFS2 file systems.

167

Clust er Administ rat ion

Before creating the GFS2 file systems, first create an LVM logical volume for each of the file systems.

For information on creating LVM logical volumes, refer to Logical Volume Manager Administration. This example uses the following logical volumes:

/d ev/csmb_vg /csmb_l v, which will hold the user data that will be exported via a Samba share and should be sized accordingly. This example creates a logical volume that is 100GB in size.

/d ev/csmb_vg /ctd b_l v, which will store the shared CTDB state information and needs to be

1GB in size.

You create clustered volume groups and logical volumes on one node of the cluster only.

To create a GFS2 file system on a logical volume, run the mkfs. g fs2 command. You run this command on one cluster node only.

To create the file system to host the Samba share on the logical volume /d ev/csmb_vg /csmb_l v, execute the following command:

[root@ clusmb-01 ~]# mkfs. g fs2 -j3 -p l o ck_d l m -t csmb: g fs2

/d ev/csmb_vg /csmb_l v

The meaning of the parameters is as follows:

-j

Specifies the number of journals to create in the filesystem. This example uses a cluster with three nodes, so we create one journal per node.

-p

Specifies the locking protocol. l o ck_d l m is the locking protocol GFS2 uses for inter-node communication.

-t

Specifies the lock table name and is of the format cluster_name:fs_name. In this example, the cluster name as specified in the cl uster. co nf file is csmb, and we use g fs2 as the name for the file system.

The output of this command appears as follows:

This will destroy any data on /dev/csmb_vg/csmb_lv.

It appears to contain a gfs2 filesystem.

Are you sure you want to proceed? [y/n] y

Device:

/dev/csmb_vg/csmb_lv

Blocksize: 4096

Device Size 100.00 GB (26214400 blocks)

Filesystem Size: 100.00 GB (26214398 blocks)

Journals: 3

Resource Groups: 400

Locking Protocol: "lock_dlm"

Lock Table: "csmb:gfs2"

UUID:

94297529-ABG3-7285-4B19-182F4F2DF2D7

168

⁠Chapt er 1 2 . Clust ered Samba Configurat ion

In this example, the /d ev/csmb_vg /csmb_l v file system will be mounted at /mnt/g fs2 on all nodes. This mount point must match the value that you specify as the location of the share directory

with the path = option in the /etc/samba/smb. co nf file, as described in Section 12.5, “ Samba

Section 12.4, “ CTDB Configuration” .

To create the file system to host the CTDB state information on the logical volume

/d ev/csmb_vg /ctd b_l v, execute the following command:

[root@ clusmb-01 ~]# mkfs. g fs2 -j3 -p l o ck_d l m -t csmb: ctd b_state

/d ev/csmb_vg /ctd b_l v

Note that this command specifies a different lock table name than the lock table in the example that created the filesystem on /d ev/csmb_vg /csmb_l v. This distinguishes the lock table names for the different devices used for the file systems.

The output of the mkfs. g fs2 appears as follows:

This will destroy any data on /dev/csmb_vg/ctdb_lv.

It appears to contain a gfs2 filesystem.

Are you sure you want to proceed? [y/n] y

Device:

/dev/csmb_vg/ctdb_lv

Blocksize: 4096

Device Size 1.00 GB (262144 blocks)

Filesystem Size: 1.00 GB (262142 blocks)

Journals: 3

Resource Groups: 4

Locking Protocol: "lock_dlm"

Lock Table: "csmb:ctdb_state"

UUID:

BCDA8025-CAF3-85BB-B062-CC0AB8849A03

In this example, the /d ev/csmb_vg /ctd b_l v file system will be mounted at /mnt/ctd b on all nodes. This mount point must match the value that you specify as the location of the . ctd b. l o ck file with the C T D B_R EC O VER Y _LO C K option in the /etc/sysco nfi g /ctd b file, as described in

12.4 . CT DB Configurat ion

The CTDB configuration file is located at /etc/sysco nfi g /ctd b. The mandatory fields that must be configured for CTDB operation are as follows:

C T D B_NO D ES

C T D B_P UBLIC _AD D R ESSES

C T D B_R EC O VER Y _LO C K

C T D B_MANAG ES_SAMBA (must be enabled)

C T D B_MANAG ES_WINBIND (must be enabled if running on a member server)

The following example shows a configuration file with the mandatory fields for CTDB operation set with example parameters:

169

Clust er Administ rat ion

CTDB_NODES=/etc/ctdb/nodes

CTDB_PUBLIC_ADDRESSES=/etc/ctdb/public_addresses

CTDB_RECOVERY_LOCK="/mnt/ctdb/.ctdb.lock"

CTDB_MANAGES_SAMBA=yes

CTDB_MANAGES_WINBIND=yes

The meaning of these parameters is as follows.

C T D B_NO D ES

Specifies the location of the file which contains the cluster node list.

The /etc/ctd b/no d es file that C T D B_NO D ES references simply lists the IP addresses of the cluster nodes, as in the following example:

192.168.1.151

192.168.1.152

192.168.1.153

In this example, there is only one interface/IP on each node that is used for both cluster/CTDB communication and serving clients. However, it is highly recommended that each cluster node have two network interfaces so that one set of interfaces can be dedicated to cluster/CTDB communication and another set of interfaces can be dedicated to public client access. Use the appropriate IP addresses of the cluster network here and make sure the hostnames/IP addresses used in the cl uster. co nf file are the same.

Similarly, use the appropriate interfaces of the public network for client access in the

publ i c_ad d resses file.

It is critical that the /etc/ctd b/no d es file is identical on all nodes because the ordering is important and CTDB will fail if it finds different information on different nodes.

C T D B_P UBLIC _AD D R ESSES

Specifies the location of the file that lists the IP addresses that can be used to access the

Samba shares exported by this cluster. These are the IP addresses that you should configure in DNS for the name of the clustered Samba server and are the addresses that

CIFS clients will connect to. Configure the name of the clustered Samba server as one DNS type A record with multiple IP addresses and let round-robin DNS distribute the clients across the nodes of the cluster.

For this example, we have configured a round-robin DNS entry csmb-server with all the addresses listed in the /etc/ctd b/publ i c_ad d resses file. DNS will distribute the clients that use this entry across the cluster in a round-robin fashion.

The contents of the /etc/ctd b/publ i c_ad d resses file on each node are as follows:

192.168.1.201/0 eth0

192.168.1.202/0 eth0

192.168.1.203/0 eth0

This example uses three addresses that are currently unused on the network. In your own configuration, choose addresses that can be accessed by the intended clients.

Alternately, this example shows the contents of the /etc/ctd b/publ i c_ad d resses files in a cluster in which there are three nodes but a total of four public addresses. In this example, IP address 198.162.2.1 can be hosted by either node 0 or node 1 and will be available to clients as long as at least one of these nodes is available. Only if both nodes 0

170

⁠Chapt er 1 2 . Clust ered Samba Configurat ion

and 1 fail does this public address become unavailable to clients. All other public addresses can only be served by one single node respectively and will therefore only be available if the respective node is also available.

The /etc/ctd b/publ i c_ad d resses file on node 0 includes the following contents:

198.162.1.1/24 eth0

198.162.2.1/24 eth1

The /etc/ctd b/publ i c_ad d resses file on node 1 includes the following contents:

198.162.2.1/24 eth1

198.162.3.1/24 eth2

The /etc/ctd b/publ i c_ad d resses file on node 2 includes the following contents:

198.162.3.2/24 eth2

C T D B_R EC O VER Y _LO C K

Specifies a lock file that CTDB uses internally for recovery. This file must reside on shared storage such that all the cluster nodes have access to it. The example in this section uses the GFS2 file system that will be mounted at /mnt/ctd b on all nodes. This is different from the GFS2 file system that will host the Samba share that will be exported. This recovery lock file is used to prevent split-brain scenarios. With newer versions of CTDB (1.0.112 and later), specifying this file is optional as long as it is substituted with another split-brain prevention mechanism.

C T D B_MANAG ES_SAMBA

When enabling by setting it to yes, specifies that CTDB is allowed to start and stop the

Samba service as it deems necessary to provide service migration/failover.

When C T D B_MANAG ES_SAMBA is enabled, you should disable automatic i ni t startup of the smb and nmb daemons by executing the following commands:

[root@ clusmb-01 ~]# chkco nfi g snb o ff

[root@ clusmb-01 ~]# chkco nfi g nmb o ff

C T D B_MANAG ES_WINBIND

When enabling by setting it to yes, specifies that CTDB is allowed to start and stop the

wi nbi nd daemon as required. This should be enabled when you are using CTDB in a

Windows domain or in active directory security mode.

When C T D B_MANAG ES_WINBIND is enabled, you should disable automatic i ni t startup of the wi nbi nd daemon by executing the following command:

[root@ clusmb-01 ~]# chkco nfi g wi nd i nbd o ff

12.5. Samba Configurat ion

The Samba configuration file smb. co nf is located at /etc/samba/smb. co nf in this example. It contains the following parameters:

171

Clust er Administ rat ion

[global]

guest ok = yes

clustering = yes

netbios name = csmb-server

[csmb]

comment = Clustered Samba

public = yes

path = /mnt/gfs2/share

writeable = yes

ea support = yes

This example exports a share with name csmb located at /mnt/g fs2/share. This is different from the GFS2 shared filesystem at /mnt/ctd b/. ctd b. l o ck that we specified as the

C T D B_R EC O VER Y _LO C K parameter in the CTDB configuration file at /etc/sysco nfi g /ctd b.

In this example, we will create the share directory in /mnt/g fs2 when we mount it for the first time.

The cl usteri ng = yes entry instructs Samba to use CTDB. The netbi o s name = csmb-

server entry explicitly sets all the nodes to have a common NetBIOS name. The ea suppo rt parameter is required if you plan to use extended attributes.

The smb. co nf configuration file must be identical on all of the cluster nodes.

Samba also offers registry-based configuration using the net co nf command to automatically keep configuration in sync between cluster members without having to manually copy configuration files among the cluster nodes. For information on the net co nf command, refer to the net(8) man page.

12.6. St art ing CT DB and Samba Services

After starting up the cluster, you must mount the GFS2 file systems that you created, as described in

Section 12.3, “ GFS2 Configuration” . The permissions on the Samba share directory and user

accounts on the cluster nodes should be set up for client access.

Execute the following command on all of the nodes to start up the ctd bd daemon. Since this example configured CTDB with C T D B_MANAG ES_SAMBA= yes, CTDB will also start up the Samba service on all nodes and export all configured Samba shares.

[root@ clusmb-01 ~]# servi ce ctd b start

It can take a couple of minutes for CTDB to start Samba, export the shares, and stabilize. Executing

ctd b status shows the status of CTDB, as in the following example:

[root@ clusmb-01 ~]# ctd b status

Number of nodes:3 pnn:0 192.168.1.151 OK (THIS NODE) pnn:1 192.168.1.152 OK pnn:2 192.168.1.153 OK

Generation:1410259202

Size:3 hash:0 lmaster:0 hash:1 lmaster:1 hash:2 lmaster:2

Recovery mode:NORMAL (0)

Recovery master:0

172

⁠Chapt er 1 2 . Clust ered Samba Configurat ion

When you see that all nodes are "OK", it is safe to move on to use the clustered Samba server, as described in

Section 12.7, “ Using the Clustered Samba Server”

12.7. Using t he Clust ered Samba Server

Clients can connect to the Samba share that was exported by connecting to one of the IP addresses specified in the /etc/ctd b/publ i c_ad d resses file, or using the csmb-server DNS entry we configured earlier, as shown below: or

[root@ clusmb-01 ~]# mo unt -t ci fs //csmb-server/csmb /mnt/sambashare -o

user= testmo nkey

[user@ clusmb-01 ~]$ smbcl i ent //csmb-server/csmb

173

Clust er Administ rat ion

Appendix A. Fence Device Parameters

This appendix provides tables with parameter descriptions of fence devices. You can configure the parameters with lu ci, by using the ccs command, or by editing the etc/cl uster/cl uster. co nf file. For a comprehensive list and description of the fence device parameters for each fence agent, refer to the man page for that agent.

Note

The Name parameter for a fence device specifies an arbitrary name for the device that will be used by Red Hat High Availability Add-On. This is not the same as the DNS name for the device.

Note

Certain fence devices have an optional Passwo rd Scrip t parameter. The Passwo rd Scrip t parameter allows you to specify that a fence-device password is supplied from a script rather than from the Passwo rd parameter. Using the Passwo rd Scrip t parameter supersedes the

Passwo rd parameter, allowing passwords to not be visible in the cluster configuration file

(/etc/cl uster/cl uster. co nf).

Table A.1, “ Fence Device Summary” lists the fence devices, the fence device agents associated with

the fence devices, and provides a reference to the table documenting the parameters for the fence devices.

T ab le A.1. Fen ce Device Su mmary

Fen ce Device Fen ce Ag en t

APC Power

Switch

(telnet/SSH)

APC Power

Switch over

SNMP

Brocade

Fabric Switch

Cisco MDS

Cisco UCS

Dell DRAC 5

Dell iDRAC

Eaton Network

Power Switch

(SNMP

Interface) fence_apc fence_apc_snmp fence_brocade fence_cisco_mds fence_cisco_ucs fence_drac5 fence_idrac fence_eaton_snmp

Ref eren ce t o Paramet er

Descrip t io n

Table A.2, “ APC Power Switch

(telnet/SSH)”

Table A.3, “ APC Power Switch over SNMP”

Table A.4, “ Brocade Fabric

Switch”

Table A.5, “ Cisco MDS”

Table A.6, “ Cisco UCS”

Table A.7, “ Dell DRAC 5”

Table A.8, “ Eaton Network

Power Controller (SNMP

Interface) (Red Hat Enterprise

Linux 6.4 and later)”

174

Fen ce Device Fen ce Ag en t

Egenera

BladeFrame

Emerson

Network Power

Switch (SNMP

Interface)

Fujitsu

Siemens

Remoteview

Service Board

(RSB)

BladeSystem fence_egenera fence_emerson ePowerSwitch fence_eps

Fence kdump fence_kdump fence_xvm Fence virt

(Multicast

Mode)

Fence virt

(Serial/VMCha nnel Mode) fence_virt fence_rsb fence_hpblade

HP iLO Device

(Integrated

Lights Out),

HP iLO2 fence_ilo fence_ilo2

HPiLO3 fence_ilo3

HPiLO4 fence_ilo4

HP iLO

(Integrated

Lights Out) MP

IBM

BladeCenter

IBM

BladeCenter

SNMP

IBM Integrated

Management

Module fence_ilo_mp fence_bladecenter fence_ibmblade fence_imm

⁠Appendix A. Fence Device Paramet ers

Ref eren ce t o Paramet er

Descrip t io n

Table A.9, “ Egenera

BladeFrame”

Table A.10, “ Emerson Network

Power Switch (SNMP interface)

(Red Hat Enterprise LInux 6.7

and later) ”

Table A.11, “ ePowerSwitch”

Table A.12, “ Fence kdump”

Table A.13, “ Fence virt

(Multicast Mode) ”

Table A.14, “ Fence virt

(Serial/VMChannel Mode)”

Table A.15, “ Fujitsu Siemens

Remoteview Service Board

(RSB)”

Table A.16, “ HP BladeSystem

(Red Hat Enterprise Linux 6.4

and later)”

Table A.17, “ HP iLO (Integrated

Lights Out) and HP iLO2”

Table A.17, “ HP iLO (Integrated

Lights Out) and HP iLO2”

Table A.18, “ HP iLO (Integrated

Lights Out) MP”

Table A.19, “ IBM BladeCenter”

Table A.20, “ IBM BladeCenter

SNMP”

Table A.21, “ IBM iPDU (Red

175

Clust er Administ rat ion

Fen ce Device Fen ce Ag en t

IBM iPDU

IF MIB

Intel Modular

IPMI

(Intelligent

Platform

Management

Interface) Lan

RHEV-M REST

API

SCSI Fencing

VMware

Fencing

(SOAP

Interface)

WTI Power

Switch fence_ipdu fence_ifmib fence_intelmodular fence_ipmilan fence_rhevm fence_scsi fence_vmware_soap fence_wti

Ref eren ce t o Paramet er

Descrip t io n

Hat Enterprise Linux 6.4 and later)”

Table A.22, “ IF MIB”

Table A.23, “ Intel Modular”

Table A.25, “ RHEV-M REST

API (RHEL 6.2 and later against RHEV 3.0 and later)”

Table A.26, “ SCSI Reservation

Fencing”

Table A.27, “ VMware Fencing

(SOAP Interface) (Red Hat

Enterprise Linux 6.2 and later)”

Table A.28, “ WTI Power

Switch”

Table A.2, “ APC Power Switch (telnet/SSH)” lists the fence device parameters used by fence_apc,

the fence agent for APC over telnet/SSH.

T ab le A.2. APC Po wer Swit ch ( t eln et /SSH) lu ci Field

Name

cl uster. co n f At t rib u t e name i pad d r

Descrip t io n

A name for the APC device connected to the cluster into which the fence daemon logs via telnet/ssh.

The IP address or hostname assigned to the device.

IP Address or

Hostname

IP Port

(optional)

Password

Script

(optional)

Power Wait

(seconds)

Power Timeout

(seconds)

Shell Timeout

(seconds)

i ppo rt l o g i n passwd passwd _scri pt

The TCP port to use to connect to the device. The default port is

23, or 22 if Use SSH is selected.

The login name used to access the device.

The password used to authenticate the connection to the device.

The script that supplies a password for access to the fence device. Using this supersedes the Passwo rd parameter.

po wer_wai t

Number of seconds to wait after issuing a power off or power on command.

po wer_ti meo ut

Number of seconds to wait before testing for a status change after issuing a power on or power on command. The default value is 20.

shel l _ti me o ut l o g i n_ti me o ut

Number of seconds to wait for a command prompt after issuing a command. The default value is 3.

Number of seconds to wait for a command prompt after login.

The default value is 5.

176

⁠Appendix A. Fence Device Paramet ers lu ci Field

Times to Retry

Power On

Operation

Port

Switch

(optional)

Delay

(optional)

Use SSH

SSH Options

Path to SSH

Identity File

cl uster. co n f At t rib u t e retry_o n

Descrip t io n

Number of attempts to retry a power on operation. The default value is 1.

po rt swi tch d el ay secure

The port.

The switch number for the APC switch that connects to the node when you have multiple daisy-chained switches.

The number of seconds to wait before fencing is started. The default value is 0.

Indicates that system will use SSH to access the device. When using SSH, you must specify either a password, a password script, or an identity file.

ssh_o pti o ns SSH options to use. The default value is -1 -c bl o wfi sh.

i d enti ty_fi l e

The identity file for SSH.

Table A.3, “ APC Power Switch over SNMP” lists the fence device parameters used by

fence_apc_snmp, the fence agent for APC that logs into the SNP device via the SNMP protocol.

T ab le A.3. APC Po wer Swit ch o ver SNMP lu ci Field cl uster. co n f At t rib u t e name

Descrip t io n

Name A name for the APC device connected to the cluster into which the fence daemon logs via the SNMP protocol.

The IP address or hostname assigned to the device.

IP Address or

Hostname

UDP/TCP port

i pad d r ud ppo rt

Password

l o g i n passwd

Password

Script

(optional)

passwd _scri pt

SNMP Version

snmp_versi o n

SNMP

Community

co mmuni ty

SNMP Security

Level

SNMP

Authentication

Protocol

snmp_sec_l e vel snmp_auth_p ro t

SNMP Privacy

Protocol

SNMP Privacy

Protocol

Password

SNMP Privacy

Protocol Script

snmp_pri v_p ro t snmp_pri v_p asswd snmp_pri v_p asswd _scri p t

The UDP/TCP port to use for connection with the device; the default value is 161.

The login name used to access the device.

The password used to authenticate the connection to the device.

The script that supplies a password for access to the fence device. Using this supersedes the Passwo rd parameter.

The SNMP version to use (1, 2c, 3); the default value is 1.

The SNMP community string; the default value is pri vate.

The SNMP security level (noAuthNoPriv, authNoPriv, authPriv).

The SNMP authentication protocol (MD5, SHA).

The SNMP privacy protocol (DES, AES).

The SNMP privacy protocol password.

The script that supplies a password for SNMP privacy protocol.

Using this supersedes the SNMP p rivacy p ro t o co l p asswo rd parameter.

177

Clust er Administ rat ion lu ci Field

Power Wait

(seconds)

Power Timeout

(seconds)

Shell Timeout

(seconds)

Times to Retry

Power On

Operation

Port (Outlet)

Number

Delay

(optional)

cl uster. co n f At t rib u t e

Descrip t io n po wer_wai t

Number of seconds to wait after issuing a power off or power on command.

po wer_ti meo ut

Number of seconds to wait before testing for a status change after issuing a power on or power on command. The default value is 20.

shel l _ti me o ut l o g i n_ti me o ut retry_o n

Number of seconds to wait for a command prompt after issuing a command. The default value is 3.

Number of seconds to wait for a command prompt after login.

The default value is 5.

Number of attempts to retry a power on operation. The default value is 1.

po rt d el ay

The port.

The number of seconds to wait before fencing is started. The default value is 0.

Table A.4, “ Brocade Fabric Switch”

lists the fence device parameters used by fence_bro cad e, the fence agent for Brocade FC switches.

T ab le A.4 . Bro cad e Fab ric Swit ch lu ci Field

Name

IP Address or

Hostname

Password

Script

(optional)

Force IP

Family

Force

Command

Prompt

Power Wait

(seconds)

Power Timeout

(seconds)

Shell Timeout

(seconds)

Times to Retry

Power On

Operation

Port

cl uster. co n f At t rib u t e name i pad d r l o g i n passwd passwd _scri pt

Descrip t io n

A name for the Brocade device connected to the cluster.

The IP address assigned to the device.

The login name used to access the device.

The password used to authenticate the connection to the device.

The script that supplies a password for access to the fence device. Using this supersedes the Passwo rd parameter.

i net4 _o nl y

, i net6 _o nl y

Force the agent to use IPv4 or IPv6 addresses only

cmd _pro mpt

The command prompt to use. The default value is ’\$’.

po wer_wai t

Number of seconds to wait after issuing a power off or power on

po wer_ti meo ut shel l _ti me o ut l o g i n_ti me o ut retry_o n po rt

command.

Number of seconds to wait before testing for a status change after issuing a power on or power on command. The default value is 20.

Number of seconds to wait for a command prompt after issuing a command. The default value is 3.

Number of seconds to wait for a command prompt after login.

The default value is 5.

Number of attempts to retry a power on operation. The default value is 1.

The switch outlet number.

178

⁠Appendix A. Fence Device Paramet ers lu ci Field

Delay

(optional)

Use SSH

SSH Options

Path to SSH

Identity File

Unfencing

cl uster. co n f At t rib u t e

Descrip t io n d el ay secure

The number of seconds to wait before fencing is started. The default value is 0.

Indicates that the system will use SSH to access the device.

When using SSH, you must specify either a password, a password script, or an identity file.

ssh_o pti o ns SSH options to use. The default value is -1 -c bl o wfi sh.

i d enti ty_fi l e

The identity file for SSH.

unfence

section of the cluster configuration file

When enabled, this ensures that a fenced node is not re-enabled until the node has been rebooted. This is necessary for nonpower fence methods (that is, SAN/storage fencing). When you configure a device that requires unfencing, the cluster must first be stopped and the full configuration including devices and unfencing must be added before the cluster is started. For more information about unfencing a node, refer to the fence_no d e(8) man page. For information about configuring unfencing in the

cluster configuration file, refer to Section 8.3, “ Configuring

Fencing” . For information about configuring unfencing with the

ccs command, refer to

Section 6.7.2, “ Configuring a Single

Storage-Based Fence Device for a Node” .

Table A.5, “ Cisco MDS” lists the fence device parameters used by fence_ci sco _md s, the fence

agent for Cisco MDS.

T ab le A.5. Cisco MDS lu ci Field cl uster. co n f At t rib u t e name

Descrip t io n

Name

IP Address or

Hostname

UDP/TCP port

(optional)

i pad d r ud ppo rt

Password

l o g i n passwd

Password

Script

(optional)

SNMP Version

snmp_versi o n

SNMP

Community

co mmuni ty

SNMP Security

Level

SNMP

Authentication

Protocol

SNMP Privacy

Protocol

passwd _scri pt snmp_sec_l e vel snmp_auth_p ro t snmp_pri v_p ro t

A name for the Cisco MDS 9000 series device with SNMP enabled.

The IP address or hostname assigned to the device.

The UDP/TCP port to use for connection with the device; the default value is 161.

The login name used to access the device.

The password used to authenticate the connection to the device.

The script that supplies a password for access to the fence device. Using this supersedes the Passwo rd parameter.

The SNMP version to use (1, 2c, 3).

The SNMP community string.

The SNMP security level (noAuthNoPriv, authNoPriv, authPriv).

The SNMP authentication protocol (MD5, SHA).

The SNMP privacy protocol (DES, AES).

179

Clust er Administ rat ion lu ci Field

SNMP Privacy

Protocol

Password

SNMP Privacy

Protocol Script

Power Wait

(seconds)

Power Timeout

(seconds)

Shell Timeout

(seconds)

Times to Retry

Power On

Operation

Port (Outlet)

Number

Delay

(optional)

cl uster. co n f At t rib u t e snmp_pri v_p asswd

Descrip t io n

The SNMP privacy protocol password.

snmp_pri v_p asswd _scri p t

The script that supplies a password for SNMP privacy protocol.

Using this supersedes the SNMP p rivacy p ro t o co l p asswo rd parameter.

po wer_wai t

Number of seconds to wait after issuing a power off or power on command.

po wer_ti meo ut

Number of seconds to wait before testing for a status change after issuing a power on or power on command. The default value is 20.

shel l _ti me o ut l o g i n_ti me o ut retry_o n

Number of seconds to wait for a command prompt after issuing a command. The default value is 3.

Number of seconds to wait for a command prompt after login.

The default value is 5.

Number of attempts to retry a power on operation. The default value is 1.

po rt d el ay

The port.

The number of seconds to wait before fencing is started. The default value is 0.

Table A.6, “ Cisco UCS” lists the fence device parameters used by fence_ci sco _ucs, the fence

agent for Cisco UCS.

T ab le A.6 . Cisco UCS lu ci Field

Name

IP Address or

Hostname

IP port

(optional)

Password

Script

(optional)

Use SSL

Sub-

Organization

Power Wait

(seconds)

Power Timeout

(seconds)

Shell Timeout

(seconds)

cl uster. co n f At t rib u t e name i pad d r i ppo rt l o g i n passwd passwd _scri pt ssl subo rg

Descrip t io n

A name for the Cisco UCS device.

The IP address or hostname assigned to the device.

The TCP port to use to connect to the device.

The login name used to access the device.

The password used to authenticate the connection to the device.

The script that supplies a password for access to the fence device. Using this supersedes the Passwo rd parameter.

Use SSL connections to communicate with the device.

Additional path needed to access suborganization.

po wer_wai t

Number of seconds to wait after issuing a power off or power on command.

po wer_ti meo ut shel l _ti me o ut

Number of seconds to wait before testing for a status change after issuing a power on or power on command. The default value is 20.

Number of seconds to wait for a command prompt after issuing a command. The default value is 3.

180

⁠Appendix A. Fence Device Paramet ers lu ci Field

(seconds)

Times to Retry

Power On

Operation

Port (Outlet)

Number

Delay

(optional)

cl uster. co n f At t rib u t e l o g i n_ti me o ut retry_o n po rt d el ay

Descrip t io n

Number of seconds to wait for a command prompt after login.

The default value is 5.

Number of attempts to retry a power on operation. The default value is 1.

Name of virtual machine.

The number of seconds to wait before fencing is started. The default value is 0.

Table A.7, “ Dell DRAC 5”

lists the fence device parameters used by fence_d rac5, the fence agent for Dell DRAC 5.

T ab le A.7. Dell DRAC 5 lu ci Field cl uster. co n f At t rib u t e

Descrip t io n

Name

IP Address or

Hostname

IP Port

(optional)

name i pad d r i ppo rt

The name assigned to the DRAC.

The IP address or hostname assigned to the DRAC.

The TCP port to use to connect to the device.

Password

l o g i n passwd

The login name used to access the DRAC.

The password used to authenticate the connection to the DRAC.

Password

Script

(optional)

Use SSH

Power Wait

(seconds)

Delay

(seconds)

Power Timeout

(seconds)

Shell Timeout

(seconds)

passwd _scri pt secure

Indicates that the system will use SSH to access the device.

When using SSH, you must specify either a password, a password script, or an identity file.

SSH Options

Path to SSH

Identity File

ssh_o pti o ns SSH options to use. The default value is -1 -c bl o wfi sh.

i d enti ty_fi l e

The identity file for SSH.

Module Name

mo d ul e_nam e

Force

Command

Prompt

(optional) The module name for the DRAC when you have multiple DRAC modules.

cmd _pro mpt

The command prompt to use. The default value is ’\$’.

po wer_wai t d el ay po wer_ti meo ut shel l _ti me o ut l o g i n_ti me o ut

The script that supplies a password for access to the fence device. Using this supersedes the Passwo rd parameter.

Number of seconds to wait after issuing a power off or power on command.

The number of seconds to wait before fencing is started. The default value is 0.

Number of seconds to wait before testing for a status change after issuing a power on or power on command. The default value is 20.

Number of seconds to wait for a command prompt after issuing a command. The default value is 3.

Number of seconds to wait for a command prompt after login.

The default value is 5.

181

Clust er Administ rat ion lu ci Field

Times to Retry

Power On

Operation

cl uster. co n f At t rib u t e retry_o n

Descrip t io n

Number of attempts to retry a power on operation. The default value is 1.

Table A.8, “ Eaton Network Power Controller (SNMP Interface) (Red Hat Enterprise Linux 6.4 and later)” lists the fence device parameters used by fence_eato n_snmp, the fence agent for the Eaton

over SNMP network power switch.

T ab le A.8. Eat o n Net wo rk Po wer Co n t ro ller ( SNMP In t erf ace) ( Red Hat En t erp rise Lin u x

6 .4 an d lat er) lu ci Field cl uster. co n f At t rib u t e

Descrip t io n

Name

name

A name for the Eaton network power switch connected to the cluster.

The IP address or hostname assigned to the device.

IP Address or

Hostname

i pad d r

UDP/TCP Port

(optional)

ud ppo rt

Password

l o g i n passwd

Password

Script

(optional)

passwd _scri pt

SNMP Version

snmp_versi o n

SNMP

Community

co mmuni ty

SNMP Security

Level

SNMP

Authentication

Protocol

SNMP Privacy

Protocol

SNMP Privacy

Protocol

Password

SNMP Privacy

Protocol Script

snmp_sec_l e vel snmp_auth_p ro t snmp_pri v_p ro t snmp_pri v_p asswd

Power wait

(seconds)

Power Timeout

(seconds)

Shell Timeout

(seconds)

The UDP/TCP port to use for connection with the device; the default value is 161.

The login name used to access the device.

The password used to authenticate the connection to the device.

The script that supplies a password for access to the fence device. Using this supersedes the Passwo rd parameter.

The SNMP version to use (1, 2c, 3); the default value is 1.

The SNMP community string; the default value is pri vate.

The SNMP security level (noAuthNoPriv, authNoPriv, authPriv).

The SNMP authentication protocol (MD5, SHA).

The SNMP privacy protocol (DES, AES).

The SNMP privacy protocol password.

snmp_pri v_p asswd _scri p t

The script that supplies a password for SNMP privacy protocol.

Using this supersedes the SNMP p rivacy p ro t o co l p asswo rd parameter.

po wer_wai t

Number of seconds to wait after issuing a power off or power on command.

po wer_ti meo ut

Number of seconds to wait before testing for a status change after issuing a power on or power on command. The default value is 20.

shel l _ti me o ut l o g i n_ti me o ut

Number of seconds to wait for a command prompt after issuing a command. The default value is 3.

Number of seconds to wait for a command prompt after login.

The default value is 5.

182

⁠Appendix A. Fence Device Paramet ers lu ci Field

Times to Retry

Power On

Operation

Port (Outlet)

Number

Delay

(optional)

cl uster. co n f At t rib u t e retry_o n po rt d el ay

Descrip t io n

Number of attempts to retry a power on operation. The default value is 1.

Physical plug number or name of virtual machine. This parameter is always required.

The number of seconds to wait before fencing is started. The default value is 0.

Table A.9, “ Egenera BladeFrame” lists the fence device parameters used by fence_eg enera, the

fence agent for the Egenera BladeFrame.

T ab le A.9 . Eg en era Blad eFrame lu ci Field

Name

CServer

ESH Path

(optional)

Username lpan pserver

Delay

(optional)

Unfencing

cl uster. co n f At t rib u t e name cserver esh user l pan pserver d el ay unfence

section of the cluster configuration file

Descrip t io n

A name for the Egenera BladeFrame device connected to the cluster.

The hostname (and optionally the username in the form of

username@ ho stname) assigned to the device. Refer to the fence_egenera(8) man page for more information.

The path to the esh command on the cserver (default is

/opt/panmgr/bin/esh)

The login name. The default value is ro o t.

The logical process area network (LPAN) of the device.

The processing blade (pserver) name of the device.

The number of seconds to wait before fencing is started. The default value is 0.

cluster configuration file, refer to Section 8.3, “ Configuring

Fencing” . For information about configuring unfencing with the

ccs command, refer to

Section 6.7.2, “ Configuring a Single

Storage-Based Fence Device for a Node” .

Table A.10, “ Emerson Network Power Switch (SNMP interface) (Red Hat Enterprise LInux 6.7 and later) ” lists the fence device parameters used by fence_emerso n, the fence agent for Emerson over

SNMP.

T ab le A.10. Emerso n Net wo rk Po wer Swit ch ( SNMP in t erf ace) ( Red Hat En t erp rise LIn u x

6 .7 an d lat er) lu ci Field

Name

cl uster. co n f At t rib u t e name

Descrip t io n

A name for the Emerson Network Power Switch device connected to the cluster.

183

Clust er Administ rat ion lu ci Field cl uster. co n f At t rib u t e

Descrip t io n

IP Address or

Hostname

UDP/TCP Port

(optional)

i pad d r ud ppo rt

Password

l o g i n passwd

Password

Script

(optional)

passwd _scri pt

SNMP Version

snmp_versi o n

SNMP

Community

co mmuni ty

SNMP Security

Level

SNMP

Authentication

Protocol

SNMP Privacy

Protocol

SNMP privacy protocol password

SNMP Privacy

Protocol Script

snmp_sec_l e vel snmp_auth_p ro t snmp_pri v_p ro t snmp_pri v_p asswd

Power Wait

(seconds)

Power Timeout

(seconds)

Shell Timeout

(seconds)

Times to Retry

Power On

Operation

Port (Outlet)

Number

Delay

(optional)

The IP address or hostname assigned to the device.

UDP/TCP port to use for connections with the device; the default value is 161.

The login name used to access the device.

The password used to authenticate the connection to the device.

The script that supplies a password for access to the fence device. Using this supersedes the Passwo rd parameter.

The SNMP version to use (1, 2c, 3); the default value is 1.

The SNMP community string.

The SNMP security level (noAuthNoPriv, authNoPriv, authPriv).

The SNMP authentication protocol (MD5, SHA).

The SNMP privacy protocol (DES, AES).

The SNMP Privacy Protocol Password.

snmp_pri v_p asswd _scri p t

The script that supplies a password for SNMP privacy protocol.

Using this supersedes the SNMP p rivacy p ro t o co l p asswo rd parameter.

po wer_wai t

Number of seconds to wait after issuing a power off or power on command.

po wer_ti meo ut

Number of seconds to wait before testing for a status change after issuing a power on or power on command. The default value is 20.

shel l _ti me o ut l o g i n_ti me o ut retry_o n

Number of seconds to wait for a command prompt after issuing a command. The default value is 3.

Number of seconds to wait for a command prompt after login.

The default value is 5.

Number of attempts to retry a power on operation. The default value is 1.

po rt d el ay

Physical plug number or name of virtual machine.

The number of seconds to wait before fencing is started. The default value is 0.

Table A.11, “ ePowerSwitch” lists the fence device parameters used by fence_eps, the fence agent

for ePowerSwitch.

T ab le A.11. ePo werSwit ch lu ci Field

Name

cl uster. co n f At t rib u t e name

Descrip t io n

A name for the ePowerSwitch device connected to the cluster.

184

⁠Appendix A. Fence Device Paramet ers lu ci Field

IP Address or

Hostname

Password

Script

(optional)

Name of

Hidden Page

Times to Retry

Power On

Operation

Port (Outlet)

Number

Delay

(optional)

cl uster. co n f At t rib u t e i pad d r l o g i n passwd passwd _scri pt hi d d en_pag e retry_o n po rt

Descrip t io n

The IP address or hostname assigned to the device.

The login name used to access the device.

The password used to authenticate the connection to the device.

The script that supplies a password for access to the fence device. Using this supersedes the Passwo rd parameter.

The name of the hidden page for the device.

Number of attempts to retry a power on operation. The default value is 1.

Physical plug number or name of virtual machine.

d el ay

The number of seconds to wait before fencing is started. The default value is 0.

Table A.12, “ Fence kdump” lists the fence device parameters used by fence_kd ump, the fence agent

for kd ump crash recovery service. Note that fence_kd ump is not a replacement for traditional fencing methods; The fence_kd ump agent can detect only that a node has entered the kd ump crash recovery service. This allows the kd ump crash recovery service to complete without being preempted by traditional power fencing methods.

T ab le A.12. Fen ce kd u mp lu ci Field

Name

IP Family

IP Port

(optional)

Operation

Timeout

(seconds)

(optional)

Node name

cl uster. co n f At t rib u t e name fami l y i ppo rt ti meo ut

Descrip t io n

A name for the fence_kd ump device.

IP network family. The default value is auto .

IP port number that the fence_kd ump agent will use to listen for messages. The default value is 7410.

Number of seconds to wait for message from failed node.

no d ename

Name or IP address of the node to be fenced.

Table A.13, “ Fence virt (Multicast Mode) ” lists the fence device parameters used by fence_xvm, the

fence agent for virtual machines using multicast.

T ab le A.13. Fen ce virt ( Mu lt icast Mo d e) lu ci Field

Name

Timeout

(optional)

Domain

cl uster. co n f At t rib u t e name ti meo ut

po rt (formerly

d o mai n)

Descrip t io n

A name for the Fence virt fence device.

Fencing timeout, in seconds. The default value is 30.

Virtual machine (domain UUID or name) to fence.

185

Clust er Administ rat ion lu ci Field

Delay

(optional)

cl uster. co n f At t rib u t e d el ay

Descrip t io n

Fencing delay, in seconds. The fence agent will wait the specified number of seconds before attempting a fencing operation. The default value is 0.

Table A.14, “ Fence virt (Serial/VMChannel Mode)”

lists the fence device parameters used by

fence_vi rt, the fence agent for virtual machines using VM channel or serial mode .

T ab le A.14 . Fen ce virt ( Serial/VMCh an n el Mo d e) lu ci Field cl uster. co n f At t rib u t e

Name

name

Serial Device

seri al _d ev i ce

Descrip t io n

A name for the Fence virt fence device.

On the host, the serial device must be mapped in each domain's configuration file. For more information, see the fence_vi rt man page. If this field is specified, it causes the fence_vi rt fencing agent to operate in serial mode. Not specifying a value causes the fence_vi rt fencing agent to operate in VM channel mode.

The serial parameters. The default is 115200, 8N1.

Serial

Parameters

VM Channel IP

Address

Timeout

(optional)

Domain

seri al _para ms channel _ad d ress ti meo ut

The channel IP. The default value is 10.0.2.179.

Fencing timeout, in seconds. The default value is 30.

Delay

(optional)

po rt (formerly

d o mai n)

i ppo rt d el ay

Virtual machine (domain UUID or name) to fence.

The channel port. The default value is 1229, which is the value used when configuring this fence device with lu ci.

Fencing delay, in seconds. The fence agent will wait the specified number of seconds before attempting a fencing operation. The default value is 0.

Table A.15, “ Fujitsu Siemens Remoteview Service Board (RSB)” lists the fence device parameters

used by fence_rsb, the fence agent for Fujitsu-Siemens RSB.

T ab le A.15. Fu jit su Siemen s Remo t eview Service Bo ard ( RSB) lu ci Field

Name

IP Address or

Hostname

Password

Script

(optional)

Use SSH

cl uster. co n f At t rib u t e name i pad d r

Descrip t io n

A name for the RSB to use as a fence device.

The hostname assigned to the device.

l o g i n passwd passwd _scri pt secure

The login name used to access the device.

The password used to authenticate the connection to the device.

The script that supplies a password for access to the fence device. Using this supersedes the Passwo rd parameter.

Indicates that the system will use SSH to access the device.

When using SSH, you must specify either a password, a password script, or an identity file.

186

⁠Appendix A. Fence Device Paramet ers lu ci Field

Path to SSH

Identity File

TCP Port

cl uster. co n f At t rib u t e

Descrip t io n i d enti ty_fi l e i ppo rt

The Identity file for SSH.

The port number on which the telnet service listens. The default value is 3172.

cmd _pro mpt

The command prompt to use. The default value is ’\$’.

Force

Command

Prompt

Power Wait

(seconds)

Delay

(seconds)

Power Timeout

(seconds)

Shell Timeout

(seconds)

Times to Retry

Power On

Operation

po wer_wai t

Number of seconds to wait after issuing a power off or power on command.

d el ay po wer_ti meo ut

The number of seconds to wait before fencing is started. The default value is 0.

Number of seconds to wait before testing for a status change after issuing a power on or power on command. The default value is 20.

shel l _ti me o ut l o g i n_ti me o ut retry_o n

Number of seconds to wait for a command prompt after issuing a command. The default value is 3.

Number of seconds to wait for a command prompt after login.

The default value is 5.

Number of attempts to retry a power on operation. The default value is 1.

Table A.16, “ HP BladeSystem (Red Hat Enterprise Linux 6.4 and later)” lists the fence device

parameters used by fence_hpbl ad e, the fence agent for HP BladeSystem.

T ab le A.16 . HP Blad eSyst em ( Red Hat En t erp rise Lin u x 6 .4 an d lat er) lu ci Field

Name

IP Address or

Hostname

IP Port

(optional)

cl uster. co n f At t rib u t e name i pad d r i ppo rt

Descrip t io n

The name assigned to the HP Bladesystem device connected to the cluster.

The IP address or hostname assigned to the HP BladeSystem device.

The TCP port to use to connect to the device.

Password

l o g i n passwd passwd _scri pt

The login name used to access the HP BladeSystem device. This parameter is required.

The password used to authenticate the connection to the fence device.

The script that supplies a password for access to the fence device. Using this supersedes the Passwo rd parameter.

Password

Script

(optional)

Force

Command

Prompt

Missing port returns OFF instead of failure

Power Wait

(seconds)

cmd _pro mpt mi ssi ng _as

_o ff po wer_wai t

The command prompt to use. The default value is ’\$’.

Missing port returns OFF instead of failure.

Number of seconds to wait after issuing a power off or power on command.

187

Clust er Administ rat ion lu ci Field

Power Timeout

(seconds)

Shell Timeout

(seconds)

Times to Retry

Power On

Operation

Use SSH

cl uster. co n f At t rib u t e po wer_ti meo ut shel l _ti me o ut l o g i n_ti me o ut retry_o n

Descrip t io n

Number of seconds to wait before testing for a status change after issuing a power on or power on command. The default value is 20.

Number of seconds to wait for a command prompt after issuing a command. The default value is 3.

Number of seconds to wait for a command prompt after login.

The default value is 5.

Number of attempts to retry a power on operation. The default value is 1.

SSH Options

Path to SSH

Identity File

secure

Indicates that the system will use SSH to access the device.

When using SSH, you must specify either a password, a password script, or an identity file.

ssh_o pti o ns SSH options to use. The default value is -1 -c bl o wfi sh.

i d enti ty_fi l e

The identity file for SSH.

The fence agents for HP iLO devices fence_i l o and HP iLO2 devices fence_i l o 2. share the

same implementation. Table A.17, “ HP iLO (Integrated Lights Out) and HP iLO2”

lists the fence device parameters used by these agents.

T ab le A.17. HP iLO ( In t eg rat ed Lig h t s O u t ) an d HP iLO 2 lu ci Field

Name

IP Address or

Hostname

IP Port

(optional)

Password

Script

(optional)

Power Wait

(seconds)

Delay

(seconds)

Power Timeout

(seconds)

Shell Timeout

(seconds)

Times to Retry

Power On

Operation

cl uster. co n f At t rib u t e name i pad d r i ppo rt l o g i n passwd passwd _scri pt

Descrip t io n

A name for the server with HP iLO support.

The IP address or hostname assigned to the device.

TCP port to use for connection with the device. The default value is 443.

The login name used to access the device.

The password used to authenticate the connection to the device.

The script that supplies a password for access to the fence device. Using this supersedes the Passwo rd parameter.

po wer_wai t

Number of seconds to wait after issuing a power off or power on command.

d el ay po wer_ti meo ut

The number of seconds to wait before fencing is started. The default value is 0.

Number of seconds to wait before testing for a status change after issuing a power on or power on command. The default value is 20.

shel l _ti me o ut l o g i n_ti me o ut retry_o n

Number of seconds to wait for a command prompt after issuing a command. The default value is 3.

Number of seconds to wait for a command prompt after login.

The default value is 5.

Number of attempts to retry a power on operation. The default value is 1.

188

⁠Appendix A. Fence Device Paramet ers

Table A.18, “ HP iLO (Integrated Lights Out) MP” lists the fence device parameters used by

fence_i l o _mp, the fence agent for HP iLO MP devices.

T ab le A.18. HP iLO ( In t eg rat ed Lig h t s O u t ) MP lu ci Field

Name

IP Address or

Hostname

IP Port

(optional)

Password

Script

(optional)

Use SSH

cl uster. co n f At t rib u t e name i pad d r i ppo rt l o g i n passwd passwd _scri pt

Descrip t io n

A name for the server with HP iLO support.

The IP address or hostname assigned to the device.

TCP port to use for connection with the device.

The login name used to access the device.

The password used to authenticate the connection to the device.

The script that supplies a password for access to the fence device. Using this supersedes the Passwo rd parameter.

SSH Options

Path to SSH

Identity File

Force

Command

Prompt

Power Wait

(seconds)

Delay

(seconds)

Power Timeout

(seconds)

Shell Timeout

(seconds)

Times to Retry

Power On

Operation

secure

ssh_o pti o ns SSH options to use. The default value is -1 -c bl o wfi sh.

i d enti ty_fi l e cmd _pro mpt

Indicates that the system will use SSH to access the device.

When using SSH, you must specify either a password, a password script, or an identity file.

The Identity file for SSH.

The command prompt to use. The default value is ’MP>’, ’hpiLO-

>’.

po wer_wai t

Number of seconds to wait after issuing a power off or power on command.

d el ay

The number of seconds to wait before fencing is started. The default value is 0.

po wer_ti meo ut shel l _ti me o ut l o g i n_ti me o ut retry_o n

Number of seconds to wait before testing for a status change after issuing a power on or power on command. The default value is 20.

Number of seconds to wait for a command prompt after issuing a command. The default value is 3.

Number of seconds to wait for a command prompt after login.

The default value is 5.

Number of attempts to retry a power on operation. The default value is 1.

Table A.19, “ IBM BladeCenter”

lists the fence device parameters used by fence_bl ad ecenter, the fence agent for IBM BladeCenter.

T ab le A.19 . IBM Blad eCen t er lu ci Field

Name

IP Address or

Hostname

IP port

(optional)

cl uster. co n f At t rib u t e name i pad d r

Descrip t io n

A name for the IBM BladeCenter device connected to the cluster.

The IP address or hostname assigned to the device.

i ppo rt

TCP port to use for connection with the device.

189

Clust er Administ rat ion lu ci Field

Password

Script

(optional)

Power Wait

(seconds)

Power Timeout

(seconds)

Shell Timeout

(seconds)

Times to Retry

Power On

Operation

Use SSH

SSH Options

Path to SSH

Identity File

cl uster. co n f At t rib u t e l o g i n passwd passwd _scri pt

Descrip t io n

The login name used to access the device.

The password used to authenticate the connection to the device.

The script that supplies a password for access to the fence device. Using this supersedes the Passwo rd parameter.

po wer_wai t

Number of seconds to wait after issuing a power off or power on command.

po wer_ti meo ut

Number of seconds to wait before testing for a status change after issuing a power on or power on command. The default value is 20.

shel l _ti me o ut l o g i n_ti me o ut retry_o n

Number of seconds to wait for a command prompt after issuing a command. The default value is 3.

Number of seconds to wait for a command prompt after login.

The default value is 5.

Number of attempts to retry a power on operation. The default value is 1.

secure

Indicates that system will use SSH to access the device. When using SSH, you must specify either a password, a password script, or an identity file.

ssh_o pti o ns SSH options to use. The default value is -1 -c bl o wfi sh.

i d enti ty_fi l e

The identity file for SSH.

Table A.20, “ IBM BladeCenter SNMP”

lists the fence device parameters used by fence_i bmbl ad e, the fence agent for IBM BladeCenter over SNMP.

T ab le A.20. IBM Blad eCen t er SNMP lu ci Field cl uster. co n f At t rib u t e name

Descrip t io n

Name

IP Address or

Hostname

UDP/TCP Port

(optional)

i pad d r ud ppo rt

Password

l o g i n passwd

Password

Script

(optional)

SNMP Security

Level

passwd _scri pt

SNMP Version

snmp_versi o n

SNMP

Community

co mmuni ty snmp_sec_l e vel

A name for the IBM BladeCenter SNMP device connected to the cluster.

The IP address or hostname assigned to the device.

UDP/TCP port to use for connections with the device; the default value is 161.

The login name used to access the device.

The password used to authenticate the connection to the device.

The script that supplies a password for access to the fence device. Using this supersedes the Passwo rd parameter.

The SNMP version to use (1, 2c, 3); the default value is 1.

The SNMP community string.

The SNMP security level (noAuthNoPriv, authNoPriv, authPriv).

190

⁠Appendix A. Fence Device Paramet ers lu ci Field

SNMP

Authentication

Protocol

SNMP Privacy

Protocol

SNMP privacy protocol password

SNMP Privacy

Protocol Script

Power Wait

(seconds)

Power Timeout

(seconds)

Shell Timeout

(seconds)

Times to Retry

Power On

Operation

Port (Outlet)

Number

Delay

(optional)

cl uster. co n f At t rib u t e snmp_auth_p ro t snmp_pri v_p ro t snmp_pri v_p asswd

Descrip t io n

The SNMP authentication protocol (MD5, SHA).

The SNMP privacy protocol (DES, AES).

The SNMP Privacy Protocol Password.

snmp_pri v_p asswd _scri p t

The script that supplies a password for SNMP privacy protocol.

Using this supersedes the SNMP p rivacy p ro t o co l p asswo rd parameter.

po wer_wai t

Number of seconds to wait after issuing a power off or power on command.

po wer_ti meo ut shel l _ti me o ut l o g i n_ti me o ut retry_o n

Number of seconds to wait before testing for a status change after issuing a power on or power on command. The default value is 20.

Number of seconds to wait for a command prompt after issuing a command. The default value is 3.

Number of seconds to wait for a command prompt after login.

The default value is 5.

Number of attempts to retry a power on operation. The default value is 1.

po rt d el ay

Physical plug number or name of virtual machine.

The number of seconds to wait before fencing is started. The default value is 0.

Table A.21, “ IBM iPDU (Red Hat Enterprise Linux 6.4 and later)”

lists the fence device parameters used by fence_i pd u, the fence agent for iPDU over SNMP devices.

T ab le A.21. IBM iPDU ( Red Hat En t erp rise Lin u x 6 .4 an d lat er) lu ci Field

Name

cl uster. co n f At t rib u t e name

Descrip t io n

A name for the IBM iPDU device connected to the cluster into which the fence daemon logs via the SNMP protocol.

The IP address or hostname assigned to the device.

IP Address or

Hostname

i pad d r

UDP/TCP Port

ud ppo rt

Password

l o g i n passwd

Password

Script

(optional)

passwd _scri pt

SNMP Version

snmp_versi o n

SNMP

Community

co mmuni ty

The UDP/TCP port to use for connection with the device; the default value is 161.

The login name used to access the device.

The password used to authenticate the connection to the device.

The script that supplies a password for access to the fence device. Using this supersedes the Passwo rd parameter.

The SNMP version to use (1, 2c, 3); the default value is 1.

The SNMP community string; the default value is pri vate.

191

Clust er Administ rat ion lu ci Field

SNMP Security

Level

SNMP

Authentication

Protocol

SNMP Privacy

Protocol

SNMP Privacy

Protocol

Password

SNMP Privacy

Protocol Script

Power Wait

(seconds)

Power Timeout

(seconds)

Shell Timeout

(seconds)

Times to Retry

Power On

Operation

Port (Outlet)

Number

Delay

(optional)

cl uster. co n f At t rib u t e snmp_sec_l e vel snmp_auth_p ro t snmp_pri v_p ro t snmp_pri v_p asswd

Descrip t io n

The SNMP security level (noAuthNoPriv, authNoPriv, authPriv).

The SNMP Authentication Protocol (MD5, SHA).

The SNMP privacy protocol (DES, AES).

The SNMP privacy protocol password.

snmp_pri v_p asswd _scri p t

The script that supplies a password for SNMP privacy protocol.

Using this supersedes the SNMP p rivacy p ro t o co l p asswo rd parameter.

po wer_wai t

Number of seconds to wait after issuing a power off or power on command.

po wer_ti meo ut shel l _ti me o ut l o g i n_ti me o ut retry_o n

Number of seconds to wait before testing for a status change after issuing a power on or power on command. The default value is 20.

Number of seconds to wait for a command prompt after issuing a command. The default value is 3.

Number of seconds to wait for a command prompt after login.

The default value is 5.

Number of attempts to retry a power on operation. The default value is 1.

po rt d el ay

Physical plug number or name of virtual machine.

The number of seconds to wait before fencing is started. The default value is 0.

Table A.22, “ IF MIB”

lists the fence device parameters used by fence_i fmi b, the fence agent for IF-

MIB devices.

T ab le A.22. IF MIB lu ci Field cl uster. co n f At t rib u t e

Name

IP Address or

Hostname

UDP/TCP Port

(optional)

Password

Script

(optional)

name i pad d r ud ppo rt l o g i n passwd passwd _scri pt

SNMP Version

snmp_versi o n

SNMP

Community

co mmuni ty

Descrip t io n

A name for the IF MIB device connected to the cluster.

The IP address or hostname assigned to the device.

The UDP/TCP port to use for connection with the device; the default value is 161.

The login name used to access the device.

The password used to authenticate the connection to the device.

The script that supplies a password for access to the fence device. Using this supersedes the Passwo rd parameter.

The SNMP version to use (1, 2c, 3); the default value is 1.

The SNMP community string.

192

⁠Appendix A. Fence Device Paramet ers lu ci Field

SNMP Security

Level

SNMP

Authentication

Protocol

SNMP Privacy

Protocol

SNMP Privacy

Protocol

Password

SNMP Privacy

Protocol Script

Power Wait

(seconds)

Power Timeout

(seconds)

Shell Timeout

(seconds)

Times to Retry

Power On

Operation

Port (Outlet)

Number

Delay

(optional)

cl uster. co n f At t rib u t e snmp_sec_l e vel snmp_auth_p ro t snmp_pri v_p ro t snmp_pri v_p asswd

Descrip t io n

The SNMP security level (noAuthNoPriv, authNoPriv, authPriv).

The SNMP authentication protocol (MD5, SHA).

The SNMP privacy protocol (DES, AES).

The SNMP privacy protocol password.

snmp_pri v_p asswd _scri p t

The script that supplies a password for SNMP privacy protocol.

Using this supersedes the SNMP p rivacy p ro t o co l p asswo rd parameter.

po wer_wai t

Number of seconds to wait after issuing a power off or power on command.

po wer_ti meo ut shel l _ti me o ut l o g i n_ti me o ut retry_o n

Number of seconds to wait before testing for a status change after issuing a power on or power on command. The default value is 20.

Number of seconds to wait for a command prompt after issuing a command. The default value is 3.

Number of seconds to wait for a command prompt after login.

The default value is 5.

Number of attempts to retry a power on operation. The default value is 1.

po rt d el ay

Physical plug number or name of virtual machine.

The number of seconds to wait before fencing is started. The default value is 0.

Table A.23, “ Intel Modular”

lists the fence device parameters used by fence_i ntel mo d ul ar, the fence agent for Intel Modular.

T ab le A.23. In t el Mo d u lar lu ci Field cl uster. co n f At t rib u t e

Name

IP Address or

Hostname

UDP/TCP Port

(optional)

Password

Script

(optional)

name i pad d r ud ppo rt l o g i n passwd passwd _scri pt

SNMP Version

snmp_versi o n

SNMP

Community

co mmuni ty

Descrip t io n

A name for the Intel Modular device connected to the cluster.

The IP address or hostname assigned to the device.

The UDP/TCP port to use for connection with the device; the default value is 161.

The login name used to access the device.

The password used to authenticate the connection to the device.

The script that supplies a password for access to the fence device. Using this supersedes the Passwo rd parameter.

The SNMP version to use (1, 2c, 3); the default value is 1.

The SNMP community string; the default value is pri vate.

193

Clust er Administ rat ion lu ci Field

SNMP Security

Level

SNMP

Authentication

Protocol

SNMP Privacy

Protocol

SNMP Privacy

Protocol

Password

SNMP Privacy

Protocol Script

Power Wait

(seconds)

Power Timeout

(seconds)

Shell Timeout

(seconds)

Times to Retry

Power On

Operation

Port (Outlet)

Number

Delay

(optional)

cl uster. co n f At t rib u t e snmp_sec_l e vel snmp_auth_p ro t snmp_pri v_p ro t snmp_pri v_p asswd

Descrip t io n

The SNMP security level (noAuthNoPriv, authNoPriv, authPriv).

The SNMP authentication protocol (MD5, SHA).

The SNMP privacy protocol (DES, AES).

The SNMP privacy protocol password.

snmp_pri v_p asswd _scri p t

The script that supplies a password for SNMP privacy protocol.

Using this supersedes the SNMP p rivacy p ro t o co l p asswo rd parameter.

po wer_wai t

Number of seconds to wait after issuing a power off or power on command.

po wer_ti meo ut shel l _ti me o ut l o g i n_ti me o ut retry_o n

Number of seconds to wait before testing for a status change after issuing a power on or power on command. The default value is 20.

Number of seconds to wait for a command prompt after issuing a command. The default value is 3.

Number of seconds to wait for a command prompt after login.

The default value is 5.

Number of attempts to retry a power on operation. The default value is 1.

po rt d el ay

Physical plug number or name of virtual machine.

The number of seconds to wait before fencing is started. The default value is 0.

The fence agents for IPMI over LAN (fence_i pmi l an,) Dell iDRAC (fence_i d rac), IBM Integrated

Management Module (fence_i mm), HP iLO3 devices (fence_i l o 3), and HP iLO4 devices

(fence_i l o 4 ) share the same implementation. Table A.24, “ IPMI (Intelligent Platform Management

Interface) LAN, Dell iDrac, IBM Integrated Management Module, HPiLO3, HPiLO4” lists the fence

device parameters used by these agents.

T ab le A.24 . IPMI ( In t ellig en t Plat f o rm Man ag emen t In t erf ace) LAN, Dell iDrac, IBM

In t eg rat ed Man ag emen t Mo d u le, HPiLO 3, HPiLO 4 lu ci Field

Name

IP Address or

Hostname

cl uster. co n f At t rib u t e name i pad d r

Descrip t io n

A name for the fence device connected to the cluster.

The IP address or hostname assigned to the device.

Password

Script

(optional)

l o g i n passwd passwd _scri pt

The login name of a user capable of issuing power on/off commands to the given port.

The password used to authenticate the connection to the port.

The script that supplies a password for access to the fence device. Using this supersedes the Passwo rd parameter.

194

⁠Appendix A. Fence Device Paramet ers lu ci Field cl uster. co n f At t rib u t e

Descrip t io n

Authentication

Type

Use Lanplus

auth

Authentication type: no ne, passwo rd , or MD 5.

Ciphersuite to use

l anpl us ci pher

Privilege level

pri vl vl

IPMI Operation

Timeout

ti meo ut

Power Wait

(seconds)

Delay

(optional)

po wer_wai t d el ay

T rue or 1. If blank, then value is Fal se. It is recommended that you enable Lanplus to improve the security of your connection if your hardware supports it.

The remote server authentication, integrity, and encryption algorithms to use for IPMIv2 lanplus connections.

The privilege level on the device.

Timeout in seconds for IPMI operation.

Number of seconds to wait after issuing a power off or power on command. The default value is 2 seconds for fence_i pmi l an,

fence_i d rac, fence_i mm, and fence_i l o 4 . The default value is 4 seconds for fence_i l o 3.

The number of seconds to wait before fencing is started. The default value is 0.

Table A.25, “ RHEV-M REST API (RHEL 6.2 and later against RHEV 3.0 and later)”

lists the fence device parameters used by fence_rhevm, the fence agent for RHEV-M REST API.

T ab le A.25. RHEV- M REST API ( RHEL 6 .2 an d lat er ag ain st RHEV 3.0 an d lat er) lu ci Field

Name

IP Address or

Hostname

IP Port

(optional)

Password

Script

(optional)

Use SSL

Power Wait

(seconds)

Power Timeout

(seconds)

Shell Timeout

(seconds)

Times to Retry

Power On

Operation

Port (Outlet)

Number

cl uster. co n f At t rib u t e name i pad d r i ppo rt l o g i n passwd passwd _scri pt

Descrip t io n

Name of the RHEV-M REST API fencing device.

The IP address or hostname assigned to the device.

ssl

Use SSL connections to communicate with the device.

po wer_wai t

Number of seconds to wait after issuing a power off or power on command.

po wer_ti meo ut

Number of seconds to wait before testing for a status change after issuing a power on or power on command. The default value is 20.

shel l _ti me o ut l o g i n_ti me o ut retry_o n

Number of seconds to wait for a command prompt after issuing a command. The default value is 3.

Number of seconds to wait for a command prompt after login.

The default value is 5.

Number of attempts to retry a power on operation. The default value is 1.

po rt

The TCP port to use for connection with the device.

The login name used to access the device.

The password used to authenticate the connection to the device.

The script that supplies a password for access to the fence device. Using this supersedes the Passwo rd parameter.

Physical plug number or name of virtual machine.

195

Clust er Administ rat ion lu ci Field

Delay

(optional)

cl uster. co n f At t rib u t e d el ay

Descrip t io n

The number of seconds to wait before fencing is started. The default value is 0.

Table A.26, “ SCSI Reservation Fencing”

lists the fence device parameters used by fence_scsi , the fence agent for SCSI persistent reservations.

Note

Use of SCSI persistent reservations as a fence method is supported with the following limitations:

When using SCSI fencing, all nodes in the cluster must register with the same devices so that each node can remove another node's registration key from all the devices it is registered with.

Devices used for the cluster volumes should be a complete LUN, not partitions. SCSI persistent reservations work on an entire LUN, meaning that access is controlled to each

LUN, not individual partitions.

It is recommended that devices used for the cluster volumes be specified in the format

/d ev/d i sk/by-i d /xxx where possible. Devices specified in this format are consistent among all nodes and will point to the same disk, unlike devices specified in a format such as

/d ev/sd a which can point to different disks from machine to machine and across reboots.

T ab le A.26 . SCSI Reservat io n Fen cin g lu ci Field

Name

Unfencing

Node name

Key for current action

Delay

(optional)

cl uster. co n f At t rib u t e name unfence

section of the cluster configuration file

no d ename key d el ay

Descrip t io n

A name for the SCSI fence device.

cluster configuration file, refer to Section 8.3, “ Configuring

Fencing” . For information about configuring unfencing with the

ccs command, refer to

Section 6.7.2, “ Configuring a Single

Storage-Based Fence Device for a Node” .

The node name is used to generate the key value used for the current operation.

(overrides node name) Key to use for the current operation. This key should be unique to a node. For the "on" action, the key specifies the key use to register the local node. For the "off" action,this key specifies the key to be removed from the device(s).

The number of seconds to wait before fencing is started. The default value is 0.

196

⁠Appendix A. Fence Device Paramet ers

Table A.27, “ VMware Fencing (SOAP Interface) (Red Hat Enterprise Linux 6.2 and later)” lists the

fence device parameters used by fence_vmware_so ap, the fence agent for VMWare over SOAP API.

T ab le A.27. VMware Fen cin g ( SO AP In t erf ace) ( Red Hat En t erp rise Lin u x 6 .2 an d lat er) lu ci Field

Name

IP Address or

Hostname

IP Port

(optional)

Password

Script

(optional)

Power Wait

(seconds)

Power Timeout

(seconds)

Shell Timeout

(seconds)

Times to Retry

Power On

Operation

VM name

VM UUID

Delay

(optional)

Use SSL

cl uster. co n f At t rib u t e name i pad d r i ppo rt l o g i n passwd passwd _scri pt po wer_wai t

Number of seconds to wait after issuing a power off or power on command.

po wer_ti meo ut shel l _ti me o ut

Number of seconds to wait before testing for a status change after issuing a power on or power on command. The default value is 20.

Number of seconds to wait for a command prompt after issuing a command. The default value is 3.

l o g i n_ti me o ut retry_o n

Number of seconds to wait for a command prompt after login.

The default value is 5.

Number of attempts to retry a power on operation. The default value is 1.

po rt uui d d el ay ssl

Descrip t io n

Name of the virtual machine fencing device.

The IP address or hostname assigned to the device.

The TCP port to use for connection with the device. The default port is 80, or 443 if Use SSL is selected.

The login name used to access the device.

The password used to authenticate the connection to the device.

The script that supplies a password for access to the fence device. Using this supersedes the Passwo rd parameter.

Name of virtual machine in inventory path format (e.g.,

/datacenter/vm/Discovered_virtual_machine/myMachine).

The UUID of the virtual machine to fence.

The number of seconds to wait before fencing is started. The default value is 0.

Use SSL connections to communicate with the device.

Table A.28, “ WTI Power Switch”

lists the fence device parameters used by fence_wti , the fence agent for the WTI network power switch.

T ab le A.28. WT I Po wer Swit ch lu ci Field

Name

IP Address or

Hostname

IP Port

(optional)

Password

Script

(optional)

cl uster. co n f At t rib u t e name i pad d r i ppo rt l o g i n passwd passwd _scri pt

Descrip t io n

A name for the WTI power switch connected to the cluster.

The IP or hostname address assigned to the device.

The TCP port to use to connect to the device.

The login name used to access the device.

The password used to authenticate the connection to the device.

The script that supplies a password for access to the fence device. Using this supersedes the Passwo rd parameter.

197

Clust er Administ rat ion lu ci Field cl uster. co n f At t rib u t e

Descrip t io n

Force command prompt

Power Wait

(seconds)

Power Timeout

(seconds)

Shell Timeout

(seconds)

Times to Retry

Power On

Operation

Use SSH

cmd _pro mpt

The command prompt to use. The default value is [’RSM>’,

’>MPC’, ’IPS>’, ’TPS>’, ’NBB>’, ’NPS>’, ’VMR>’]

po wer_wai t

Number of seconds to wait after issuing a power off or power on

po wer_ti meo ut shel l _ti me o ut l o g i n_ti me o ut retry_o n

command.

Number of seconds to wait before testing for a status change after issuing a power on or power on command. The default value is 20.

Number of seconds to wait for a command prompt after issuing a command. The default value is 3.

Number of seconds to wait for a command prompt after login.

The default value is 5.

Number of attempts to retry a power on operation. The default value is 1.

SSH Options

Path to SSH

Identity File

Port

secure

Indicates that system will use SSH to access the device. When using SSH, you must specify either a password, a password script, or an identity file.

ssh_o pti o ns SSH options to use. The default value is -1 -c bl o wfi sh.

i d enti ty_fi l e

The identity file for SSH.

po rt

Physical plug number or name of virtual machine.

198

⁠Appendix B. HA Resource Paramet ers

Appendix B. HA Resource Parameters

This appendix provides descriptions of HA resource parameters. You can configure the parameters with lu ci, by using the ccs command, or by editing the etc/cl uster/cl uster. co nf file.

Table B.1, “ HA Resource Summary”

lists the resources, their corresponding resource agents, and references to other tables containing parameter descriptions. To understand resource agents in more detail you can view them in /usr/share/cl uster of any cluster node.

In addition to the resource agents described in this appendix, the /usr/share/cl uster directory includes a dummy OCF script for a resource group, servi ce. sh. For more information about the parameters included in this script, refer to the servi ce. sh script itself.

For a more comprehensive list and description of cl uster. co nf elements and attributes, refer to the cluster schema at /usr/share/cl uster/cl uster. rng , and the annotated schema at

/usr/share/d o c/cman-X. Y . ZZ/cl uster_co nf. html (for example /usr/share/d o c/cman-

3. 0 . 12/cl uster_co nf. html ).

Note

In addition to the resource agents documented in this appendix, Red Hat supports the DRDB resource agent. This agent is provided by LINBIT, separately from the Red Hat High

Availability Add-On. Information on the DRDB resource agent can be found at http://drbd.linbit.com/docs/ .

T ab le B.1. HA Reso u rce Su mmary

Reso u rce

Apache

Bind Mount

Reso u rce Ag en t

apache.sh

bind-mount.sh

Condor Instance

Filesystem

GFS2

IP Address

HA LVM

MySQL

Named (Bind 9)

Resource

NFS/CIFS Mount

NFS Client

NFS v3 Export

NFS Server

Oracle 10g/11g

Failover Instance condor.sh

fs.sh

clusterfs.sh

ip.sh

lvm.sh

mysql.sh

named.sh

netfs.sh

nfsclient.sh

nfsexport.sh

nfsserver.sh

oracledb.sh

Ref eren ce t o Paramet er Descrip t io n

Table B.2, “ Apache (apache Resource)”

Table B.3, “ Bind Mount (bi nd -mo unt

Resource) (Red Hat Enterprise Linux 6.6 and later)”

Table B.4, “ Condor Instance (co nd o r

Table B.5, “ Filesystem (fs Resource)”

Table B.6, “ GFS2 (cl usterfs Resource)”

Table B.7, “ IP Address (i p Resource)”

Table B.8, “ HA LVM (l vm Resource)”

Table B.9, “ MySQL (mysq l Resource)”

Table B.10, “ Named (Bind 9) (named

Table B.11, “ NFS/CIFS Mount (netfs

Table B.12, “ NFS Client (nfscl i ent

Table B.13, “ NFS v3 Export (nfsexpo rt

Table B.14, “ NFS Server (nfsserver

Table B.16, “ Oracle 10g/11g Failover Instance

(o racl ed b Resource)”

199

Clust er Administ rat ion

Reso u rce

Oracle 10g/11g

Instance

Oracle 10g/11g

Listener

Open LDAP

PostgreSQL 8

SAP Database

SAP Instance

Samba Server

Script

Sybase ASE Failover

Instance

Tomcat 6

Virtual Machine

Reso u rce Ag en t

orainstance.sh

oralistener.sh

openldap.sh

postgres-8.sh

SAPDatabase

SAPInstance samba.sh

script.sh

ASEHAagent.sh

tomcat-6.sh

vm.sh

Ref eren ce t o Paramet er Descrip t io n

Table B.17, “ Oracle 10g/11g Instance

(o rai nstance Resource)”

Table B.18, “ Oracle 10g/11g Listener

(o ral i stener Resource)”

Table B.15, “ Open LDAP (o penl d ap

Table B.19, “ PostgreSQL 8 (po stg res-8

Table B.20, “ SAP Database (SAP D atabase

Table B.21, “ SAP Instance (SAP Instance

Table B.22, “ Samba Server (samba Resource)”

Table B.23, “ Script (scri pt Resource)”

Table B.24, “ Sybase ASE Failover Instance

(ASEHAag ent Resource)”

Table B.25, “ Tomcat 6 (to mcat-6 Resource)”

Table B.26, “ Virtual Machine (vm Resource)”

NOTE: lu ci displays this as a virtual service if the host cluster can support virtual machines.

T ab le B.2. Ap ach e ( apache Reso u rce) lu ci Field cl uster. co n f At t rib u t e

Descrip t io n

Name

Server Root

Config File

co nfi g _fi l e

httpd Options

httpd _o pti o ns

Path to httpd binary

name

The name of the Apache service.

server_ro o t

The default value is /etc/httpd .

httpd

Specifies the Apache configuration file. The default valuer is

co nf/httpd . co nf.

Other command line options for httpd .

Specifies absolute path of the httpd binary to use.

Shutdown

Wait (seconds)

shutd o wn_wa i t

Specifies the number of seconds to wait for correct end of service shutdown.

T ab le B.3. Bin d Mo u n t ( bi nd -mo unt Reso u rce) ( Red Hat En t erp rise Lin u x 6 .6 an d lat er)

Field

Name

Mount Point

Source of the

Bind Mount

Force

Unmount

lu ci Field cl uster. co nf At t rib u t e name

Specifies a name for the Bind Mount resource

mo untpo i nt

Target of this bind mount

so urce

Source of the bind mount

fo rce_unmo u nt

If set, the cluster will kill all processes using this file system when the resource group is stopped. Otherwise, the unmount will fail, and the resource group will be restarted.

T ab le B.4 . Co n d o r In st an ce ( co nd o r Reso u rce)

200

⁠Appendix B. HA Resource Paramet ers

Field lu ci Field

Instance Name name

Condor

Subsystem

Type

type cl uster. co nf At t rib u t e

Specifies a unique name for the Condor instance.

Specifies the type of Condor subsystem for this instance:

sched d , jo b_server, or q uery_server.

T ab le B.5. Filesyst em ( fs Reso u rce) lu ci Field cl uster. co n f At t rib u t e

Descrip t io n

Name

Filesystem

Type

Mount Point

name fstype

Specifies a name for the file system resource.

If not specified, mo unt tries to determine the file system type.

Device, FS

Label, or UUID

mo untpo i nt

Path in file system hierarchy to mount this file system.

d evi ce

Specifies the device associated with the file system resource.

This can be a block device, file system label, or UUID of a file system.

Mount Options o pti o ns Mount options; that is, options used when the file system is mounted. These may be file-system specific. Refer to the

mount(8) man page for supported mount options.

File System ID

(optional)

fsi d

Note

File System ID is used only by NFS services.

When creating a new file system resource, you can leave this field blank. Leaving the field blank causes a file system ID to be assigned automatically after you commit the parameter during configuration. If you need to assign a file system ID explicitly, specify it in this field.

Force

Unmount

Force fsck

Enable NFS daemon and lockd workaround

(Red Hat

Enterprise

Linux 6.4 and later)

Use Quick

Status Checks

fo rce_unmo u nt nfsrestart

If enabled, forces the file system to unmount. The default setting is disabled. Force Unmount kills all processes using the mount point to free up the mount when it tries to unmount.

fo rce_fsck

If enabled, causes fsck to be run on the file system before mounting it. The default setting is disabled.

If your file system is exported via NFS and occasionally fails to unmount (either during shutdown or service relocation), setting this option will drop all file system references prior to the unmount operation. Setting this option requires that you enable the Fo rce unmo unt option and must not be used together with the NFS Server resource. You should set this option as a last resort only, as this is a hard attempt to unmount a file system.

Reboot Host

Node if

Unmount Fails

q ui ck_statu s

If this option is enabled it will cause the fs. sh agent to bypass the read and write tests on all status checks and simply do a mount test.

sel f_fence

If enabled, reboots the node if unmounting this file system fails.

The fi l esystem resource agent accepts a value of 1, yes, o n, or true to enable this parameter, and a value of 0, no , o ff, or

fal se to disable it. The default setting is disabled.

201

Clust er Administ rat ion

T ab le B.6 . G FS2 ( cl usterfs Reso u rce) lu ci Field cl uster. co n f At t rib u t e

Descrip t io n

Name

Mount Point

Device, FS

Label, or UUID

Filesystem

Type

name

The name of the file system resource.

mo untpo i nt

The path to which the file system resource is mounted.

d evi ce fstype

Mount Options o pti o ns

File System ID

(optional)

fsi d

The device file associated with the file system resource.

Set to GFS2 on lu ci

Mount options.

Note

File System ID is used only by NFS services.

When creating a new GFS2 resource, you can leave this field blank. Leaving the field blank causes a file system ID to be assigned automatically after you commit the parameter during configuration. If you need to assign a file system ID explicitly, specify it in this field.

Force

Unmount

Enable NFS daemon and lockd workaround

(Red Hat

Enterprise

Linux 6.4 and later)

Reboot Host

Node if

Unmount Fails

fo rce_unmo u nt nfsrestart

If enabled, forces the file system to unmount. The default setting is disabled. Force Unmount kills all processes using the mount point to free up the mount when it tries to unmount. With

GFS2 resources, the mount point is not unmounted at service tear-down unless Force Unmount is enabled.

sel f_fence

If enabled and unmounting the file system fails, the node will immediately reboot. Generally, this is used in conjunction with force-unmount support, but it is not required. The G FS2 resource agent accepts a value of 1, yes, o n, or true to enable this parameter, and a value of 0, no , o ff, or fal se to disable it.

T ab le B.7. IP Ad d ress ( i p Reso u rce) lu ci Field

IP Address,

Netmask Bits

cl uster. co n f At t rib u t e ad d ress

Descrip t io n

The IP address (and, optionally, netmask bits) for the resource.

Netmask bits, or network prefix length, may come after the address itself with a slash as a separator, complying with CIDR notation (for example, 10.1.1.1/8). This is a virtual IP address.

IPv4 and IPv6 addresses are supported, as is NIC link monitoring for each IP address.

202

⁠Appendix B. HA Resource Paramet ers lu ci Field

Monitor Link

Disable

Updates to

Static Routes

Number of

Seconds to

Sleep After

Removing an

IP Address

cl uster. co n f At t rib u t e mo ni to r_l i nk d i sabl e_rd i sc sl eepti me

Descrip t io n

Enabling this causes the status check to fail if the link on the NIC to which this IP address is bound is not present.

Disable updating of routing using RDISC protocol.

Specifies the amount of time (in seconds) to sleep.

T ab le B.8. HA LVM ( l vm Reso u rce) lu ci Field

Name

Volume Group

Name

Logical

Volume Name

Fence the

Node if It is

Unable to

Clean UP LVM

Directory

Use Simplified

Database

Backend

Other

Command-

Line Options

Shutdown

Wait (seconds)

cl uster. co n f At t rib u t e named _wo rki ng _D i r named _sd b named _o pti o ns shutd o wn_wa i t

Descrip t io n

The working directory for the named resource. The default value is /var/named

Reserved, currently unused

Additional command-line options of the named resource

Specifies the number of seconds to wait for correct end of service shutdown.

T ab le B.11. NFS/CIFS Mo u n t ( netfs Reso u rce) lu ci Field

Name

cl uster. co n f At t rib u t e name

Descrip t io n

Symbolic name for the NFS or CIFS mount.

Note

This resource is required when a cluster service is configured to be an NFS client.

Mount Point

Host

NFS Export

Directory

Name or CIFS share

Filesystem

Type

mo untpo i nt

Path to which the file system resource is mounted.

ho st expo rt

NFS/CIFS server IP address or hostname.

NFS Export directory name or CIFS share name.

fstype

File system type:

NFS — Specifies using the default NFS version. This is the default setting.

NFS v4 — Specifies using NFSv4 protocol.

CIFS — Specifies using CIFS protocol.

Do Not

Unmount the

Filesystem

During a Stop of Relocation

Operation.

Force

Unmount

no _unmo unt

If enabled, specifies that the file system should not be unmounted during a stop or relocation operation.

Self-Fence If

Unmount Fails

Options

fo rce_unmo u nt sel f_fence o pti o ns

If Force Unmount is enabled, the cluster kills all processes using this file system when the service is stopped. Killing all processes using the file system frees up the file system.

Otherwise, the unmount will fail, and the service will be restarted.

If enabled, reboots the node if unmounting this file system fails.

Mount options. Specifies a list of mount options. If none are specified, the file system is mounted -o sync.

204

⁠Appendix B. HA Resource Paramet ers

T ab le B.12. NFS Clien t ( nfscl i ent Reso u rce) lu ci Field

Name

cl uster. co n f At t rib u t e name

Descrip t io n

This is a symbolic name of a client used to reference it in the resource tree. This is not the same thing as the Target option.

Note

An nfscl i ent resource must be configured as a child of a parent nfsexpo rt resource or a parent nfsserver resource.

Target

Hostname,

Wildcard, or

Netgroup

Allow

Recovery of

This NFS

Client

Options

targ et

This is the server from which you are mounting. It can be specified using a hostname, a wildcard (IP address or hostname based), or a netgroup defining a host or hosts to export to.

al l o w_reco ver

Allow recovery.

o pti o ns

Defines a list of options for this client — for example, additional client access rights. For more information, refer to the exports

(5) man page, General Options.

T ab le B.13. NFS v3 Exp o rt ( nfsexpo rt Reso u rce) lu ci Field

Name

cl uster. co n f At t rib u t e name

Descrip t io n

Descriptive name of the resource. The NFS Export resource ensures that NFS daemons are running. It is fully reusable; typically, only one NFS Export resource is needed. For more information on configuring the nfsexpo rt resource, see

Section 8.8, “ Configuring nfsexport and nfsserver Resources” .

Tip

Name the NFS Export resource so it is clearly distinguished from other NFS resources.

T ab le B.14 . NFS Server ( nfsserver Reso u rce) lu ci Field cl uster. co n f At t rib u t e

Descrip t io n

205

Clust er Administ rat ion lu ci Field

Name

cl uster. co n f At t rib u t e name statd po rt

Descrip t io n

Descriptive name of the NFS server resource. The NFS server resource is useful for exporting NFSv4 file systems to clients.

Because of the way NFSv4 works, only one NFSv4 resource may exist on a server at a time. Additionally, it is not possible to use the NFS server resource when also using local instances of NFS on each cluster node. For more information on configuring the

nfsserver resource, see

Section 8.8, “ Configuring nfsexport and nfsserver Resources” .

The port number used for RPC listener sockets NFS statd listening port

(optional)

T ab le B.15. O p en LDAP ( o penl d ap Reso u rce) lu ci Field

Name

Config File

URL List

sl apd

Options

Shutdown

Wait (seconds)

cl uster. co n f At t rib u t e name co nfi g _fi l e url _l i st sl apd _o pti o ns shutd o wn_wa i t

Descrip t io n

Specifies a service name for logging and other purposes.

Specifies an absolute path to a configuration file. The default value is /etc/o penl d ap/sl apd . co nf.

The default value is l d ap: ///.

Other command line options for sl apd .

Specifies the number of seconds to wait for correct end of service shutdown.

T ab le B.16 . O racle 10g /11g Failo ver In st an ce ( o racl ed b Reso u rce) lu ci Field

Instance Name

(SID) of Oracle

Instance

Oracle Listener

Instance Name

cl uster. co n f At t rib u t e name l i stener_na me

Oracle User

Name

Oracle

Application

Home

Directory

user ho me

Descrip t io n

Instance name.

Oracle listener instance name. If you have multiple instances of

Oracle running, it may be necessary to have multiple listeners on the same machine with different names.

This is the user name of the Oracle user that the Oracle AS instance runs as.

This is the Oracle (application, not user) home directory. It is configured when you install Oracle.

206

⁠Appendix B. HA Resource Paramet ers lu ci Field

Oracle

Installation

Type

Virtual

Hostname

(optional)

cl uster. co n f At t rib u t e type

Descrip t io n

The Oracle installation type.

Default: 10 g

base: Database Instance and Listener only

base-11g : Oracle11g Database Instance and Listener Only

base-em (or 10 g ): Database, Listener, Enterprise Manager, and iSQL*Plus

base-em-11g : Database, Listener, Enterprise Manager dbconsole

i as (or 10 g -i as): Internet Application Server

(Infrastructure)

vho st tns_ad mi n

Virtual Hostname matching the installation hostname of Oracle

10g. Note that during the start/stop of an oracledb resource, your hostname is changed temporarily to this hostname. Therefore, you should configure an oracledb resource as part of an exclusive service only.

Path to specific listener configuration file.

TNS_ADMIN

(optional)

T ab le B.17. O racle 10g /11g In st an ce ( o rai nstance Reso u rce) lu ci Field cl uster. co n f At t rib u t e name

Instance name

(SID) of Oracle instance

Oracle User

Name

Oracle

Application

Home

Directory

List of Oracle

Listeners

(optional, separated by spaces)

Path to Lock

File (optional)

TNS_ADMIN

(optional)

user ho me l i steners l o ckfi l e tns_ad mi n

Descrip t io n

Instance name.

This is the user name of the Oracle user that the Oracle instance runs as.

This is the Oracle (application, not user) home directory. It is configured when you install Oracle.

List of Oracle listeners which will be started with the database instance. Listener names are separated by whitespace. Defaults to empty which disables listeners.

Location for lockfile which will be used for checking if the Oracle should be running or not. Defaults to location under /tmp.

Path to specific listener configuration file.

T ab le B.18. O racle 10g /11g List en er ( o ral i stener Reso u rce) lu ci Field cl uster. co n f At t rib u t e

Listener Name

name

Oracle User

user

Name

Descrip t io n

Listener name.

This is the user name of the Oracle user that the Oracle instance runs as.

207

Clust er Administ rat ion lu ci Field

Oracle

Application

Home

Directory

TNS_ADMIN

(optional)

cl uster. co n f At t rib u t e ho me tns_ad mi n

Descrip t io n

This is the Oracle (application, not user) home directory. It is configured when you install Oracle.

Path to specific listener configuration file.

T ab le B.19 . Po st g reSQ L 8 ( po stg res-8 Reso u rce) lu ci Field

Name

Config File

Postmaster

User

Postmaster

Options

Startup Wait

(seconds)

Shutdown

Wait (seconds)

cl uster. co n f At t rib u t e name co nfi g _fi l e po stmaster_ user po stmaster_ o pti o ns startup_wai t shutd o wn_wa i t

Descrip t io n

Specifies a service name for logging and other purposes.

Define absolute path to configuration file. The default value is

/var/l i b/pg sq l /d ata/po stg resq l . co nf.

User who runs the database server because it cannot be run by root. The default value is postgres.

Other command line options for postmaster.

Specifies the number of seconds to wait for correct end of service startup.

Specifies the number of seconds to wait for correct end of service shutdown.

T ab le B.20. SAP Dat ab ase ( SAP D atabase Reso u rce) lu ci Field cl uster. co n f At t rib u t e

SID

SAP Database

Name

SAP

Executable

Directory

Database

Type

Oracle Listener

Name

ABAP Stack is

Not Installed,

Only Java

Stack is

Installed

Application

Level

Monitoring

Automatic

Startup

Recovery

Path to Java

SDK

D IR _EXEC UT

ABLE

D BT Y P E

NET SER VIC E

NAME

D BJ2EE_O NL

ST R IC T _MO N

IT O R ING

AUT O MAT IC _

R EC O VER

JAVE_HO ME

Descrip t io n

Specifies a unique SAP system identifier. For example, P01.

Specifies the fully qualified path to sapstartsrv and

sapco ntro l .

Specifies one of the following database types: Oracle, DB6, or

ADA.

Specifies Oracle TNS listener name.

If you do not have an ABAP stack installed in the SAP database, enable this parameter.

Activates application level monitoring.

Enable or disable automatic startup recovery.

Path to Java SDK.

208

⁠Appendix B. HA Resource Paramet ers lu ci Field

File Name of the JDBC

Driver

Path to a Pre-

Start Script

Path to a Post-

Start Script

Path to a Pre-

Stop Script

Path to a Post-

Stop Script

J2EE Instance

Bootstrap

Directory

J2EE Security

Store Path

cl uster. co n f At t rib u t e

D B_JAR S

P R E_ST AR T _

USER EXIT

P O ST _ST AR T

_USER EXIT

P R E_ST O P _U

SER EXIT

P O ST _ST O P _

USER EXIT

D IR _BO O T ST

R AP

D IR _SEC ST O

R E

Descrip t io n

File name of the JDBC driver.

Path to a pre-start script.

Path to a post-start script.

Path to a pre-stop script

Path to a post-stop script

The fully qualified path the J2EE instance bootstrap directory.

For example,

/usr/sap/P 0 1/J0 0 /j2ee/cl uster/bo o tstrap.

The fully qualified path the J2EE security store directory. For example,

/usr/sap/P 0 1/SY S/g l o bal /securi ty/l i b/to o l s.

T ab le B.21. SAP In st an ce ( SAP Instance Reso u rce) lu ci Field

SAP Instance

Name

SAP

Executable

Directory

Containing the

SAP START

Profile

Name of the

SAP START

Profile

Number of

Seconds to

Wait Before

Checking

Startup Status

Enable

Automatic

Startup

Recovery

Path to a Pre-

Start Script

Path to a Post-

Start Script

Path to a Pre-

Stop Script

Path to a Post-

Stop Script

cl uster. co n f At t rib u t e

InstanceNam e

D IR _EXEC UT

ABLE

D IR _P R O FIL

ST AR T _P R O F

ILE

ST AR T _WAIT

T IME

AUT O MAT IC _

R EC O VER

P R E_ST AR T _

USER EXIT

P O ST _ST AR T

_USER EXIT

P R E_ST O P _U

SER EXIT

P O ST _ST O P _

USER EXIT

Descrip t io n

The fully qualified SAP instance name. For example,

P01_DVEBMGS00_sapp01ci.

The fully qualified path to sapstartsrv and sapco ntro l .

The fully qualified path to the SAP START profile.

Specifies name of the SAP START profile.

Specifies the number of seconds to wait before checking the startup status (do not wait for J2EE-Addin).

Enable or disable automatic startup recovery.

Path to a pre-start script.

Path to a post-start script.

Path to a pre-stop script

Path to a post-stop script

209

Clust er Administ rat ion

Note

Regarding

Table B.22, “ Samba Server (samba Resource)”

, when creating or editing a cluster service, connect a Samba-service resource directly to the service, not to a resource within a service.

T ab le B.22. Samb a Server ( samba Reso u rce) lu ci Field

Name

Config File

Other

Command-

Line Options for smbd

Other

Command-

Line Options for nmbd

Shutdown

Wait (seconds)

cl uster. co n f At t rib u t e name co nfi g _fi l e smbd _o pti o ns nmbd _o pti o ns shutd o wn_wa i t

Descrip t io n

Specifies the name of the Samba server.

The Samba configuration file

Other command-line options for smbd.

Other command-line options for nmbd.

Specifies number of seconds to wait for correct end of service shutdown.

T ab le B.23. Scrip t ( scri pt Reso u rce) lu ci Field

Name

Full Path to

Script File

cl uster. co n f At t rib u t e name fi l e

Descrip t io n

Specifies a name for the custom user script. The script resource allows a standard LSB-compliant init script to be used to start a clustered service.

Enter the path where this custom script is located (for example,

/etc/i ni t. d /userscript).

T ab le B.24 . Syb ase ASE Failo ver In st an ce ( ASEHAag ent Reso u rce) lu ci Field cl uster. co n f At t rib u t e

Descrip t io n

Instance Name name

ASE Server

Name

Specifies the instance name of the Sybase ASE resource.

server_name

The ASE server name that is configured for the HA service.

SYBASE Home directory

sybase_ho me The home directory of Sybase products.

SYBASE_ASE

Directory

Name

l o g i n_fi l e

The full path of login file that contains the login-password pair.

Interfaces File

i nterfaces_ fi l e

The full path of the interfaces file that is used to start/access the

ASE server.

sybase_ase

The directory name under sybase_home where ASE products are installed.

210

⁠Appendix B. HA Resource Paramet ers lu ci Field

SYBASE_OCS

Directory

Name

Sybase User

Start Timeout

(seconds)

Shutdown

Timeout

(seconds)

Deep Probe

Timeout

cl uster. co n f At t rib u t e sybase_o cs

sybase_user The user who can run ASE server.

start_ti meo ut

The start timeout value.

shutd o wn_ti meo ut

The shutdown timeout value.

d eep_pro be_ ti meo ut

Descrip t io n

The directory name under sybase_home where OCS products are installed. For example, ASE-15_0.

The maximum seconds to wait for the response of ASE server before determining that the server had no response while running deep probe.

T ab le B.25. T o mcat 6 ( to mcat-6 Reso u rce) lu ci Field

Name

Config File

Shutdown

Wait (seconds)

cl uster. co n f At t rib u t e name co nfi g _fi l e shutd o wn_wa i t

Descrip t io n

Specifies a service name for logging and other purposes.

Specifies the absolute path to the configuration file. The default value is /etc/to mcat6 /to mcat6 . co nf.

Specifies the number of seconds to wait for correct end of service shutdown. The default value is 30.

Important

Regarding

Table B.26, “ Virtual Machine (vm Resource)” , when you configure your cluster with

virtual machine resources, you should use the rg manag er tools to start and stop the virtual machines. Using vi rsh to start the machine can result in the virtual machine running in more than one place, which can cause data corruption in the virtual machine. For information on configuring your system to reduce the chances of administrators accidentally "double-

starting" virtual machines by using both cluster and non-cluster tools, refer to Section 3.14,

“ Configuring Virtual Machines in a Clustered Environment” .

Note

Virtual machine resources are configured differently than other cluster resources. To configure a virtual machine resource with lu ci, add a service group to the cluster then add a resource to the service, selecting Vi rtual Machi ne as the resource type and entering the virtual machine resource parameters. For information on configuring a virtual machine with the ccs,

refer to Section 6.12, “ Virtual Machine Resources”

T ab le B.26 . Virt u al Mach in e ( vm Reso u rce) lu ci Field cl uster. co n f At t rib u t e

Service Name

name

Descrip t io n

Specifies the name of the virtual machine. When using the lu ci interface, you specify this as a service name.

211

Clust er Administ rat ion lu ci Field

Automatically

Start This

Service

Run Exclusive

Failover

Domain

Recovery

Policy

cl uster. co n f At t rib u t e auto start excl usi ve d o mai n reco very

Descrip t io n

If enabled, this virtual machine is started automatically after the cluster forms a quorum. If this parameter is disabled, this virtual machine is not started automatically after the cluster forms a quorum; the virtual machine is put into the disabled state.

If enabled, this virtual machine can only be relocated to run on another node exclusively; that is, to run on a node that has no other virtual machines running on it. If no nodes are available for a virtual machine to run exclusively, the virtual machine is not restarted after a failure. Additionally, other virtual machines do not automatically relocate to a node running this virtual machine as Run exclusive. You can override this option by manual start or relocate operations.

Defines lists of cluster members to try in the event that a virtual machine fails.

Recovery policy provides the following options:

Disable — Disables the virtual machine if it fails.

Relocate — Tries to restart the virtual machine in another node; that is, it does not try to restart in the current node.

Restart — Tries to restart the virtual machine locally (in the current node) before trying to relocate (default) to virtual machine to another node.

Restart-Disable — The service will be restarted in place if it fails. However, if restarting the service fails the service will be disabled instead of moved to another host in the cluster.

Restart

Options

Migration

Mapping

max_restart

restart_exp i re_ti me

Migration Type mi g rate

mi g rati o n_m appi ng

With Rest art or Rest art - Disab le selected as the recovery policy for a service, specifies the maximum number of restart failures before relocating or disabling the service and specifies the length of time in seconds after which to forget a restart.

Specifies a migration type of live or pause. The default setting is live.

Specifies an alternate interface for migration. You can specify this when, for example, the network address used for virtual machine migration on a node differs from the address of the node used for cluster communication.

Specifying the following indicates that when you migrate a virtual machine from member to member2, you actually migrate to

targ et2. Similarly, when you migrate from member2 to member, you migrate using targ et.

member: targ et,member2: targ et2

212

⁠Appendix B. HA Resource Paramet ers lu ci Field

Status

Program

cl uster. co n f At t rib u t e status_pro g ram

Descrip t io n

Status program to run in addition to the standard check for the presence of a virtual machine. If specified, the status program is executed once per minute. This allows you to ascertain the status of critical services within a virtual machine. For example, if a virtual machine runs a web server, your status program could check to see whether a web server is up and running; if the status check fails (signified by returning a non-zero value), the virtual machine is recovered.

After a virtual machine is started, the virtual machine resource agent will periodically call the status program and wait for a successful return code (zero) prior to returning. This times out after five minutes.

Path to xmlfile

Used to Create the VM

Configuration

File Path

xml fi l e path

Full path to l i bvi rt XML file containing the l i bvi rt domain definition.

A colon-delimited path specification that the Virtual Machine

Resource Agent (vm. sh) searches for the virtual machine configuration file. For example:

/mnt/g uests/co nfi g : /etc/l i bvi rt/q emu.

Important

The path should never directly point to a virtual machine configuration file.

Path to the VM

Snapshot

Directory

Do Not Force

Kill VM During

Stop

snapsho t

Hypervisor URI hypervi so r_

uri

Migration URI

mi g rati o n_u ri

Tunnel data over ssh during migration

tunnel l ed no _ki l l

Path to the snapshot directory where the virtual machine image will be stored.

Hypervisor URI (normally automatic).

Migration URI (normally automatic).

Tunnel data over ssh during migration.

Do not force kill vm during stop; instead. fail after the timeout expires.

213

Clust er Administ rat ion

Appendix C. HA Resource Behavior

This appendix describes common behavior of HA resources. It is meant to provide ancillary information that may be helpful in configuring HA services. You can configure the parameters with

lu ci or by editing /etc/cl uster/cl uster. co nf. For descriptions of HA resource parameters,

refer to Appendix B, HA Resource Parameters . To understand resource agents in more detail you can

view them in /usr/share/cl uster of any cluster node.

Note

To fully comprehend the information in this appendix, you may require detailed understanding of resource agents and the cluster configuration file, /etc/cl uster/cl uster. co nf.

An HA service is a group of cluster resources configured into a coherent entity that provides specialized services to clients. An HA service is represented as a resource tree in the cluster configuration file, /etc/cl uster/cl uster. co nf (in each cluster node). In the cluster configuration file, each resource tree is an XML representation that specifies each resource, its attributes, and its relationship among other resources in the resource tree (parent, child, and sibling relationships).

Note

Because an HA service consists of resources organized into a hierarchical tree, a service is sometimes referred to as a resource tree or resource group. Both phrases are synonymous with

HA service.

This appendix consists of the following sections:

Section C.1, “ Parent, Child, and Sibling Relationships Among Resources”

Section C.2, “ Sibling Start Ordering and Resource Child Ordering”

Section C.3, “ Inheritance, the <resources> Block, and Reusing Resources”

Section C.4, “ Failure Recovery and Independent Subtrees”

Section C.5, “ Debugging and Testing Services and Resource Ordering”

Note

The sections that follow present examples from the cluster configuration file,

/etc/cl uster/cl uster. co nf, for illustration purposes only.

C.1. Parent , Child, and Sibling Relat ionships Among Resources

214

⁠Appendix C. HA Resource Behavior

A cluster service is an integrated entity that runs under the control of rg manag er. All resources in a service run on the same node. From the perspective of rg manag er, a cluster service is one entity that can be started, stopped, or relocated. Within a cluster service, however, the hierarchy of the resources determines the order in which each resource is started and stopped.The hierarchical levels consist of parent, child, and sibling.

Example C.1, “ Resource Hierarchy of Service foo”

shows a sample resource tree of the service foo. In the example, the relationships among the resources are as follows:

fs: myfs (<fs name="myfs" ...>) and i p: 10 . 1. 1. 2 (<ip address="10.1.1.2 .../>) are siblings.

fs: myfs (<fs name="myfs" ...>) is the parent of scri pt: scri pt_chi l d (<script name="script_child"/>).

scri pt: scri pt_chi l d (<script name="script_child"/>) is the child of fs: myfs (<fs name="myfs" ...>).

Examp le C.1. Reso u rce Hierarch y o f Service f o o

</fs>

</service>

The following rules apply to parent/child relationships in a resource tree:

Parents are started before children.

Children must all stop cleanly before a parent may be stopped.

For a resource to be considered in good health, all its children must be in good health.

Note

When configuring a dependency tree for a cluster service that includes a floating IP address resource, you must configure the IP resource as the first entry and not as the child of another resource.

C.2. Sibling St art Ordering and Resource Child Ordering

The Service resource determines the start order and the stop order of a child resource according to whether it designates a child-type attribute for a child resource as follows:

Designates child-type attribute (typed child resource) — If the Service resource designates a childtype attribute for a child resource, the child resource is typed. The child-type attribute explicitly determines the start and the stop order of the child resource.

Does not designate child-type attribute (non-typed child resource) — If the Service resource does not

designate a child-type attribute for a child resource, the child resource is non-typed. The Service resource does not explicitly control the starting order and stopping order of a non-typed child resource. However, a non-typed child resource is started and stopped according to its order in

215

Clust er Administ rat ion

/etc/cl uster/cl uster. co nf. In addition, non-typed child resources are started after all typed child resources have started and are stopped before any typed child resources have stopped.

Note

The only resource to implement defined child resource type ordering is the Service resource.

For more information about typed child resource start and stop ordering, refer to Section C.2.1,

“ Typed Child Resource Start and Stop Ordering” . For more information about non-typed child

resource start and stop ordering, refer to Section C.2.2, “ Non-typed Child Resource Start and Stop

Ordering” .

C.2.1. T yped Child Resource St art and St op Ordering

For a typed child resource, the type attribute for the child resource defines the start order and the stop order of each resource type with a number that can range from 1 to 100; one value for start, and one value for stop. The lower the number, the earlier a resource type starts or stops. For example,

Table C.1, “ Child Resource Type Start and Stop Order” shows the start and stop values for each resource type; Example C.2, “ Resource Start and Stop Values: Excerpt from Service Resource Agent,

servi ce. sh”

shows the start and stop values as they appear in the Service resource agent,

servi ce. sh. For the Service resource, all LVM children are started first, followed by all File System children, followed by all Script children, and so forth.

T ab le C.1. Ch ild Reso u rce T yp e St art an d St o p O rd er

Reso u rce

LVM

File System

GFS2 File System

NFS Mount

NFS Export

NFS Client

IP Address

Samba

Script

Ch ild T yp e

lvm fs clusterfs netfs nfsexport nfsclient ip smb script

St art - o rd er Valu e

St o p - o rd er Valu e

Examp le C.2. Reso u rce St art an d St o p Valu es: Excerp t f ro m Service Reso u rce Ag en t , servi ce. sh

</special>

216

⁠Appendix C. HA Resource Behavior

Ordering within a resource type is preserved as it exists in the cluster configuration file,

/etc/cl uster/cl uster. co nf. For example, consider the starting order and stopping order of the typed child resources in

Example C.3, “ Ordering Within a Resource Type” .

Examp le C.3. O rd erin g Wit h in a Reso u rce T yp e

</service>

T ype d Child Re so urce St art ing Orde r

Example C.3, “ Ordering Within a Resource Type” , the resources are started in the following order:

1. l vm: 1 — This is an LVM resource. All LVM resources are started first. l vm: 1 (<l vm

name= "1" . . . />) is the first LVM resource started among LVM resources because it is the first LVM resource listed in the Service foo portion of /etc/cl uster/cl uster. co nf.

2. l vm: 2 — This is an LVM resource. All LVM resources are started first. l vm: 2 (<l vm

name= "2" . . . />) is started after l vm: 1 because it is listed after l vm: 1 in the Service foo portion of /etc/cl uster/cl uster. co nf.

3. fs: 1 — This is a File System resource. If there were other File System resources in Service

foo, they would start in the order listed in the Service foo portion of

/etc/cl uster/cl uster. co nf.

4. i p: 10 . 1. 1. 1 — This is an IP Address resource. If there were other IP Address resources in

Service foo, they would start in the order listed in the Service foo portion of

/etc/cl uster/cl uster. co nf.

5. scri pt: 1 — This is a Script resource. If there were other Script resources in Service foo, they would start in the order listed in the Service foo portion of /etc/cl uster/cl uster. co nf.

T ype d Child Re so urce St o pping Orde r

Example C.3, “ Ordering Within a Resource Type” , the resources are stopped in the following order:

1. scri pt: 1 — This is a Script resource. If there were other Script resources in Service foo, they would stop in the reverse order listed in the Service foo portion of

/etc/cl uster/cl uster. co nf.

2. i p: 10 . 1. 1. 1 — This is an IP Address resource. If there were other IP Address resources in

Service foo, they would stop in the reverse order listed in the Service foo portion of

/etc/cl uster/cl uster. co nf.

3. fs: 1 — This is a File System resource. If there were other File System resources in Service

foo, they would stop in the reverse order listed in the Service foo portion of

/etc/cl uster/cl uster. co nf.

217

Clust er Administ rat ion

4. l vm: 2 — This is an LVM resource. All LVM resources are stopped last. l vm: 2 (<l vm

name= "2" . . . />) is stopped before l vm: 1; resources within a group of a resource type are stopped in the reverse order listed in the Service foo portion of

/etc/cl uster/cl uster. co nf.

5. l vm: 1 — This is an LVM resource. All LVM resources are stopped last. l vm: 1 (<l vm

name= "1" . . . />) is stopped after l vm: 2; resources within a group of a resource type are stopped in the reverse order listed in the Service foo portion of

/etc/cl uster/cl uster. co nf.

C.2.2. Non-t yped Child Resource St art and St op Ordering

Additional considerations are required for non-typed child resources. For a non-typed child resource, starting order and stopping order are not explicitly specified by the Service resource.

Instead, starting order and stopping order are determined according to the order of the child resource in /etc/cl uster/cl uster. co nf. Additionally, non-typed child resources are started after all typed child resources and stopped before any typed child resources.

For example, consider the starting order and stopping order of the non-typed child resources in

Example C.4, “ Non-typed and Typed Child Resource in a Service” .

Examp le C.4 . No n - t yp ed an d T yp ed Ch ild Reso u rce in a Service

</service>

No n-t ype d Child Re so urce St art ing Orde r

Example C.4, “ Non-typed and Typed Child Resource in a Service”

, the child resources are started in the following order:

1. l vm: 1 — This is an LVM resource. All LVM resources are started first. l vm: 1 (<l vm

name= "1" . . . />) is the first LVM resource started among LVM resources because it is the first LVM resource listed in the Service foo portion of /etc/cl uster/cl uster. co nf.

2. l vm: 2 — This is an LVM resource. All LVM resources are started first. l vm: 2 (<l vm

name= "2" . . . />) is started after l vm: 1 because it is listed after l vm: 1 in the Service foo portion of /etc/cl uster/cl uster. co nf.

3. fs: 1 — This is a File System resource. If there were other File System resources in Service

foo, they would start in the order listed in the Service foo portion of

/etc/cl uster/cl uster. co nf.

4. i p: 10 . 1. 1. 1 — This is an IP Address resource. If there were other IP Address resources in

Service foo, they would start in the order listed in the Service foo portion of

/etc/cl uster/cl uster. co nf.

218

⁠Appendix C. HA Resource Behavior

6. no ntyped reso urce: fo o — This is a non-typed resource. Because it is a non-typed resource, it is started after the typed resources start. In addition, its order in the Service resource is before the other non-typed resource, no ntyped reso urcetwo : bar; therefore, it is started before no ntyped reso urcetwo : bar. (Non-typed resources are started in the order that they appear in the Service resource.)

7. no ntyped reso urcetwo : bar — This is a non-typed resource. Because it is a non-typed resource, it is started after the typed resources start. In addition, its order in the Service resource is after the other non-typed resource, no ntyped reso urce: fo o ; therefore, it is started after no ntyped reso urce: fo o . (Non-typed resources are started in the order that they appear in the Service resource.)

No n-t ype d Child Re so urce St o pping Orde r

Example C.4, “ Non-typed and Typed Child Resource in a Service”

, the child resources are stopped in the following order:

1. no ntyped reso urcetwo : bar — This is a non-typed resource. Because it is a non-typed resource, it is stopped before the typed resources are stopped. In addition, its order in the

Service resource is after the other non-typed resource, no ntyped reso urce: fo o ; therefore, it is stopped before no ntyped reso urce: fo o . (Non-typed resources are stopped in the reverse order that they appear in the Service resource.)

2. no ntyped reso urce: fo o — This is a non-typed resource. Because it is a non-typed resource, it is stopped before the typed resources are stopped. In addition, its order in the

Service resource is before the other non-typed resource, no ntyped reso urcetwo : bar; therefore, it is stopped after no ntyped reso urcetwo : bar. (Non-typed resources are stopped in the reverse order that they appear in the Service resource.)

3. scri pt: 1 — This is a Script resource. If there were other Script resources in Service foo, they would stop in the reverse order listed in the Service foo portion of

/etc/cl uster/cl uster. co nf.

4. i p: 10 . 1. 1. 1 — This is an IP Address resource. If there were other IP Address resources in

Service foo, they would stop in the reverse order listed in the Service foo portion of

/etc/cl uster/cl uster. co nf.

5. fs: 1 — This is a File System resource. If there were other File System resources in Service

foo, they would stop in the reverse order listed in the Service foo portion of

/etc/cl uster/cl uster. co nf.

6. l vm: 2 — This is an LVM resource. All LVM resources are stopped last. l vm: 2 (<l vm

name= "2" . . . />) is stopped before l vm: 1; resources within a group of a resource type are stopped in the reverse order listed in the Service foo portion of

/etc/cl uster/cl uster. co nf.

7. l vm: 1 — This is an LVM resource. All LVM resources are stopped last. l vm: 1 (<l vm

name= "1" . . . />) is stopped after l vm: 2; resources within a group of a resource type are stopped in the reverse order listed in the Service foo portion of

/etc/cl uster/cl uster. co nf.

C.3. Inherit ance, t he <resources> Block, and Reusing Resources

219

Clust er Administ rat ion

Some resources benefit by inheriting values from a parent resource; that is commonly the case in an

NFS service. Example C.5, “ NFS Service Set Up for Resource Reuse and Inheritance”

shows a typical

NFS service configuration, set up for resource reuse and inheritance.

Examp le C.5. NFS Service Set Up f o r Reso u rce Reu se an d In h erit an ce

</resources>

<nfsexport ref="exports"> <!-- nfsexport's path and fsid attributes

are inherited from the mountpoint &

fsid attribute of the parent fs

resource -->

<nfsclient ref="bob"/> <!-- nfsclient's path is inherited from the

mountpoint and the fsid is added to the

options string during export -->

</nfsexport>

</fs>

<nfsclient ref="bob"/> <!-- Because all of the critical data for this

resource is either defined in the

resources block or inherited, we can

reference it again! -->

</nfsexport>

</fs>

</service>

If the service were flat (that is, with no parent/child relationships), it would need to be configured as follows:

The service would need four nfsclient resources — one per file system (a total of two for file systems), and one per target machine (a total of two for target machines).

220

⁠Appendix C. HA Resource Behavior

The service would need to specify export path and file system ID to each nfsclient, which introduces chances for errors in the configuration.

Example C.5, “ NFS Service Set Up for Resource Reuse and Inheritance”

however, the NFS client resources nfsclient:bob and nfsclient:jim are defined once; likewise, the NFS export resource

nfsexport:exports is defined once. All the attributes needed by the resources are inherited from parent resources. Because the inherited attributes are dynamic (and do not conflict with one another), it is possible to reuse those resources — which is why they are defined in the resources block. It may not be practical to configure some resources in multiple places. For example, configuring a file system resource in multiple places can result in mounting one file system on two nodes, therefore causing problems.

C.4 . Failure Recovery and Independent Subt rees

In most enterprise environments, the normal course of action for failure recovery of a service is to

restart the entire service if any component in the service fails. For example, in Example C.6, “ Service

foo Normal Failure Recovery” , if any of the scripts defined in this service fail, the normal course of

action is to restart (or relocate or disable, according to the service recovery policy) the service.

However, in some circumstances certain parts of a service may be considered non-critical; it may be necessary to restart only part of the service in place before attempting normal recovery. To accomplish that, you can use the __independent_subtree attribute. For example, in

Example C.7, “ Service foo Failure Recovery with __independent_subtree Attribute” , the

__independent_subtree attribute is used to accomplish the following actions:

If script:script_one fails, restart script:script_one, script:script_two, and script:script_three.

If script:script_two fails, restart just script:script_two.

If script:script_three fails, restart script:script_one, script:script_two, and script:script_three.

If script:script_four fails, restart the whole service.

Examp le C.6 . Service foo No rmal Failu re Reco very

</script>

</service>

Examp le C.7. Service foo Failu re Reco very wit h __independent_subtree At t rib u t e

</script>

</service>

In some circumstances, if a component of a service fails you may want to disable only that

221

Clust er Administ rat ion

component without disabling the entire service, to avoid affecting other services that use other components of that service. As of the Red Hat Enterprise Linux 6.1 release, you can accomplish that by using the __independent_subtree="2" attribute, which designates the independent subtree as non-critical.

Note

You may only use the non-critical flag on singly-referenced resources. The non-critical flag works with all resources at all levels of the resource tree, but should not be used at the top level when defining services or virtual machines.

As of the Red Hat Enterprise Linux 6.1 release, you can set maximum restart and restart expirations on a per-node basis in the resource tree for independent subtrees. To set these thresholds, you can use the following attributes:

__max_restarts configures the maximum number of tolerated restarts prior to giving up.

__restart_expi re_ti me configures the amount of time, in seconds, after which a restart is no longer attempted.

C.5. Debugging and T est ing Services and Resource Ordering

You can debug and test services and resource ordering with the rg _test utility. rg _test is a command-line utility provided by the rg manag er package that is run from a shell or a terminal (it is

not available in Co n g a). Table C.2, “ rg _test Utility Summary” summarizes the actions and syntax

for the rg _test utility.

T ab le C.2. rg _test Ut ilit y Su mmary

Act io n

Display the resource rules that

rg _test

understand s.

Test a configuratio n (and

/usr/share/c luster) for errors or redundant resource agents.

Display the start and stop ordering of a service.

Syn t ax rg _test rul es rg _test test /etc/cl uster/cl uster. co nf

Display start order:

rg _test no o p /etc/cl uster/cl uster. co nf start servi ce servicename

Display stop order:

rg _test no o p /etc/cl uster/cl uster. co nf sto p servi ce servicename

222

⁠Appendix C. HA Resource Behavior

Act io n

Explicitly start or stop a service.

Syn t ax

Important

Only do this on one node, and always disable the service in rgmanager first.

Start a service:

rg _test test /etc/cl uster/cl uster. co nf start servi ce servicename

Stop a service:

rg _test test /etc/cl uster/cl uster. co nf sto p servi ce servicename

Calculate and display the resource tree delta between two cluster.conf

files.

rg _test d el ta cluster.conf file 1 cluster.conf file 2

For example:

rg _test d el ta /etc/cl uster/cl uster. co nf. bak

/etc/cl uster/cl uster. co nf

223

Clust er Administ rat ion

Appendix D. Cluster Service Resource Check and Failover

Timeout

This appendix describes how rg manag er monitors the status of cluster resources, and how to modify the status check interval. The appendix also describes the __enfo rce_ti meo uts service parameter, which indicates that a timeout for an operation should cause a service to fail.

Note

To fully comprehend the information in this appendix, you may require detailed understanding of resource agents and the cluster configuration file, /etc/cl uster/cl uster. co nf. For a comprehensive list and description of cl uster. co nf elements and attributes, refer to the cluster schema at /usr/share/cl uster/cl uster. rng , and the annotated schema at

/usr/share/d o c/cman-X. Y . ZZ/cl uster_co nf. html (for example

/usr/share/d o c/cman-3. 0 . 12/cl uster_co nf. html ).

D.1. Modifying t he Resource St at us Check Int erval

rg manag er checks the status of individual resources, not whole services. Every 10 seconds, rgmanager scans the resource tree, looking for resources that have passed their "status check" interval.

Each resource agent specifies the amount of time between periodic status checks. Each resource utilizes these timeout values unless explicitly overridden in the cl uster. co nf file using the special

<acti o n> tag:

<acti o n name= "status" d epth= "*" i nterval = "10 " />

This tag is a special child of the resource itself in the cl uster. co nf file. For example, if you had a file system resource for which you wanted to override the status check interval you could specify the file system resource in the cl uster. co nf file as follows:

<nfsexport...>

</nfsexport>

</fs>

Some agents provide multiple "depths" of checking. For example, a normal file system status check

(depth 0) checks whether the file system is mounted in the correct place. A more intensive check is depth 10, which checks whether you can read a file from the file system. A status check of depth 20 checks whether you can write to the file system. In the example given here, the d epth is set to *, which indicates that these values should be used for all depths. The result is that the test file system is checked at the highest-defined depth provided by the resource-agent (in this case, 20) every 10 seconds.

D.2. Enforcing Resource T imeout s

There is no timeout for starting, stopping, or failing over resources. Some resources take an indeterminately long amount of time to start or stop. Unfortunately, a failure to stop (including a

224

⁠Appendix D. Clust er Service Resource Check and Failover T imeout

timeout) renders the service inoperable (failed state). You can, if desired, turn on timeout enforcement on each resource in a service individually by adding __enfo rce_ti meo uts= "1" to the reference in the cl uster. co nf file.

The following example shows a cluster service that has been configured with the

__enfo rce_ti meo uts attribute set for the netfs resource. With this attribute set, then if it takes more than 30 seconds to unmount the NFS file system during a recovery process the operation will time out, causing the service to enter the failed state.

</screen>

<rm>

<netfs export="/nfstest" force_unmount="1" fstype="nfs" host="10.65.48.65"

mountpoint="/data/nfstest" name="nfstest_data" options="rw,sync,soft"/>

</resources>

</service>

</rm>

225

Clust er Administ rat ion

Appendix E. Command Line Tools Summary

Table E.1, “ Command Line Tool Summary”

summarizes preferred command-line tools for configuring and managing the High Availability Add-On. For more information about commands and variables, refer to the man page for each command-line tool.

T ab le E.1. Co mman d Lin e T o o l Su mmary

Co mman d Lin e T o o l

ccs_co nfi g _d ump — Cluster

Configuration Dump Tool

ccs_co nfi g _val i d ate —

Cluster Configuration

Validation Tool

cl ustat — Cluster Status

Utility

cl usvcad m — Cluster User

Service Administration Utility

cman_to o l — Cluster

Management Tool

fence_to o l — Fence Tool

Used Wit h Pu rp o se

Cluster Infrastructure

ccs_co nfi g _d ump generates XML output of running configuration. The running configuration is, sometimes, different from the stored configuration on file because some subsystems store or set some default information into the configuration. Those values are generally not present on the on-disk version of the configuration but are required at runtime for the cluster to work properly. For more information about this tool, refer to the ccs_config_dump(8) man page.

Cluster Infrastructure

ccs_co nfi g _val i d ate validates

cl uster. co nf against the schema,

cl uster. rng (located in

/usr/share/cl uster/cl uster. rng

on each node). For more information about this tool, refer to the ccs_config_validate(8) man page.

High-availability

Service Management

Components

The cl ustat command displays the status of the cluster. It shows membership information, quorum view, and the state of all configured user services. For more information about this tool, refer to the clustat(8) man page.

High-availability

Service Management

Components

The cl usvcad m command allows you to enable, disable, relocate, and restart high-availability services in a cluster. For more information about this tool, refer to the clusvcadm(8) man page.

Cluster Infrastructure

cman_to o l is a program that manages the CMAN cluster manager. It provides the capability to join a cluster, leave a cluster, kill a node, or change the expected quorum votes of a node in a cluster. For more information about this tool, refer to the cman_tool(8) man page.

Cluster Infrastructure

fence_to o l is a program used to join and leave the fence domain. For more information about this tool, refer to the fence_tool(8) man page.

226

⁠Appendix F. High Availabilit y LVM (HA- LVM)

Appendix F. High Availability LVM (HA-LVM)

The Red Hat High Availability Add-On provides support for high availability LVM volumes (HA-LVM) in a failover configuration. This is distinct from active/active configurations enabled by the Clustered

Logical Volume Manager (CLVM), which is a set of clustering extensions to LVM that allow a cluster of computers to manage shared storage.

When to use CLVM or HA-LVM should be based on the needs of the applications or services being deployed.

If the applications are cluster-aware and have been tuned to run simultaneously on multiple machines at a time, then CLVM should be used. Specifically, if more than one node of your cluster will require access to your storage which is then shared among the active nodes, then you must use CLVM. CLVM allows a user to configure logical volumes on shared storage by locking access to physical storage while a logical volume is being configured, and uses clustered locking services to manage the shared storage. For information on CLVM, and on LVM configuration in general, refer to Logical Volume Manager Administration.

If the applications run optimally in active/passive (failover) configurations where only a single node that accesses the storage is active at any one time, you should use High Availability Logical

Volume Management agents (HA-LVM).

Most applications will run better in an active/passive configuration, as they are not designed or optimized to run concurrently with other instances. Choosing to run an application that is not clusteraware on clustered logical volumes may result in degraded performance if the logical volume is mirrored. This is because there is cluster communication overhead for the logical volumes themselves in these instances. A cluster-aware application must be able to achieve performance gains above the performance losses introduced by cluster file systems and cluster-aware logical volumes. This is achievable for some applications and workloads more easily than others.

Determining what the requirements of the cluster are and whether the extra effort toward optimizing for an active/active cluster will pay dividends is the way to choose between the two LVM variants. Most users will achieve the best HA results from using HA-LVM.

HA-LVM and CLVM are similar in the fact that they prevent corruption of LVM metadata and its logical volumes, which could otherwise occur if multiple machines where allowed to make overlapping changes. HA-LVM imposes the restriction that a logical volume can only be activated exclusively; that is, active on only one machine at a time. This means that only local (non-clustered) implementations of the storage drivers are used. Avoiding the cluster coordination overhead in this way increases performance. CLVM does not impose these restrictions - a user is free to activate a logical volume on all machines in a cluster; this forces the use of cluster-aware storage drivers, which allow for clusteraware file systems and applications to be put on top.

HA-LVM can be setup to use one of two methods for achieving its mandate of exclusive logical volume activation.

The preferred method uses CLVM, but it will only ever activate the logical volumes exclusively.

This has the advantage of easier setup and better prevention of administrative mistakes (like removing a logical volume that is in use). In order to use CLVM, the High Availability Add-On and

Resilient Storage Add-On software, including the cl vmd daemon, must be running.

The procedure for configuring HA-LVM using this method is described in Section F.1,

“ Configuring HA-LVM Failover with CLVM (preferred)” .

The second method uses local machine locking and LVM "tags". This method has the advantage of not requiring any LVM cluster packages; however, there are more steps involved in setting it up and it does not prevent an administrator from mistakenly removing a logical volume from a node in the cluster where it is not active. The procedure for configuring HA-LVM using this method is

227

Clust er Administ rat ion

described in Section F.2, “ Configuring HA-LVM Failover with Tagging” .

F.1. Configuring HA-LVM Failover wit h CLVM (preferred)

To set up HA-LVM failover (using the preferred CLVM variant), perform the following steps:

1. Ensure that your system is configured to support CLVM, which requires the following:

The High Availability Add-On and Resilient Storage Add-On are installed, including the the cmi rro r package if the CLVM logical volumes are to be mirrored.

The l o cki ng _type parameter in the global section of the /etc/l vm/l vm. co nf file is set to the value '3'.

The High Availability Add-On and Resilient Storage Add-On software, including the

cl vmd daemon, must be running. For CLVM mirroring, the cmi rro rd service must be started as well.

2. Create the logical volume and file system using standard LVM and file system commands, as in the following example.

# pvcreate /d ev/sd [cd e]1

# vg create -cy shared _vg /d ev/sd [cd e]1

# l vcreate -L 10 G -n ha_l v shared _vg

# mkfs. ext4 /d ev/shared _vg /ha_l v

# l vchang e -an shared _vg /ha_l v

For information on creating LVM logical volumes, refer to Logical Volume Manager

Administration.

3. Edit the /etc/cl uster/cl uster. co nf file to include the newly created logical volume as a resource in one of your services. Alternately, you can use Co n g a or the ccs command to configure LVM and file system resources for the cluster. The following is a sample resource manager section from the /etc/cl uster/cl uster. co nf file that configures a CLVM logical volume as a cluster resource:

<rm>

</failoverdomain>

</failoverdomains>

</resources>

228

⁠Appendix F. High Availabilit y LVM (HA- LVM)

</service>

</rm>

F.2. Configuring HA-LVM Failover wit h T agging

To set up HA-LVM failover by using tags in the /etc/l vm/l vm. co nf file, perform the following steps:

1. Ensure that the l o cki ng _type parameter in the global section of the

/etc/l vm/l vm. co nf file is set to the value '1'.

2. Create the logical volume and file system using standard LVM and file system commands, as in the following example.

# pvcreate /d ev/sd [cd e]1

# vg create shared _vg /d ev/sd [cd e]1

# l vcreate -L 10 G -n ha_l v shared _vg

# mkfs. ext4 /d ev/shared _vg /ha_l v

For information on creating LVM logical volumes, refer to Logical Volume Manager

Administration.

<rm>

</failoverdomain>

</failoverdomains>

</resources>

</service>

</rm>

229

Clust er Administ rat ion

Note

If there are multiple logical volumes in the volume group, then the logical volume name

(l v_name) in the l vm resource should be left blank or unspecified. Also note that in an HA-LVM configuration, a volume group may be used by only a single service.

4. Edit the vo l ume_l i st field in the /etc/l vm/l vm. co nf file. Include the name of your root volume group and your hostname as listed in the /etc/cl uster/cl uster. co nf file preceded by @. The hostname to include here is the machine on which you are editing the

l vm. co nf file, not any remote hostname. Note that this string MUST match the node name given in the cl uster. co nf file. Below is a sample entry from the /etc/l vm/l vm. co nf file: volume_list = [ "VolGroup00", "@ neo-01" ]

This tag will be used to activate shared VGs or LVs. DO NOT include the names of any volume groups that are to be shared using HA-LVM.

5. Update the i ni tramfs device on all your cluster nodes:

# d racut -H -f /bo o t/i ni tramfs-$(uname -r). i mg $(uname -r)

6. Reboot all nodes to ensure the correct i ni trd device is in use.

230

⁠Appendix G . Revision Hist ory

Appendix G. Revision History

Revisio n 7.1- 13 Fri O ct 9 2015

Resolves: #1270326

Clarifies description of nofailback in example

Revisio n 7.1- 12

Version for 6.7 GA release

Wed Ju l 8 2015

St even Levin e

Revisio n 7.1- 11

Resolves: #965173

Mo n Ju l 6 2015 St even Levin e

Adds Emerson Network Power Switch (SNMP interface) to fence device appendix

Revisio n 7.1- 10

Version for 6.7 Beta release

T h u Ap r 16 2015 St even Levin e

Revisio n 7.1- 9 T u e Mar 24 2015

Resolves: #1208789

Notes named_sdb option of Named resource as currently unused

St even Levin e

Resolves: #965173

Adds Fence virt (Serial) table to fence device appendix

Resolves: #1180339

Updates troubleshooting description for quorum disk not appearing as cluster member

Resolves: #1199090, #1199083

Small corrections to resource parameter descriptions

Resolves: 1199111

Adds reference to DRDB resource agent in appendix on HA resource parameters

Revisio n 7.1- 6 T h u Feb 19 2015

Adding new chapter -- Getting Started: Overview

St even Levin e

St even Levin e Revisio n 7.1- 3 T u e Dec 16 2014

Updating to implement sort_order on the RHEL 6 splash page

Revisio n 7.1- 1

Resolves: #1157218

T u e O ct 28 2014

Corrects STATDARG variable typo.

Revisio n 7.0- 13

Version for 6.6 GA release

Wed O ct 8 2014

Revisio n 7.0- 12

Resolves: #11288810

Adds SCSI fence note

Fri Sep 26 2014

Revisio n 7.0- 11

Version for 6.6 Beta release

T h u Au g 7 2014

St even Levin e

231

Clust er Administ rat ion

Revisio n 7.0- 10

Latest draft for 6.6 release

Mo n Ju l 28 2014 St even Levin e

Resolves: #1107601, #1098288, #969087, #987070, #1010049, #1081182, #1099579, #1062178

Small technical and editorial corrections and clarifications.

Resolves: #1092726, #1093131, #1061769, #1081215, #1081222

Documents new resource agent parameters

Resolves: #1092730, #1081143, #1081147

Documents new fence agent parameters

Resolves: #1035013

Clarifies that automatic updates must be disabled for clusters

Resolves: #1124063

Adds parameter table for Bind Mount resource agent

Resolves: #1081260

Documents new ccs start and stop options

Resolves: #856311

References fence_check man page

Resolves: #987587

Adds parameter table for Named (Bind 9) resource agent

Resolves: #1081175

Adds parameter table for kdump fence agent

Resolves: #1081181

Documents addition of sorting capabilities to cluster resource list on luci

Resolves: #1081222

Documents addition of sorting capabilities to cluster resource list on luci

Resolves: #1094508

Clarifies description of nfsexport resource agent configuration

St even Levin e Revisio n 6 .0- 21

Version for 6.5 GA release

Wed No v 13 2013

Revisio n 6 .0- 20

Resolves: #986462

Wed No v 6 2013

Updates oracledb resource table.

Revisio n 6 .0- 16 T u e O ct 29 2013

Resolves: #1021045

Corrects example of iptables rule.

Revisio n 6 .0- 15

Version for 6.5 Beta release

Fri Sep 27 2013

St even Levin e

232

⁠Appendix G . Revision Hist ory

Revisio n 6 .0- 12 T h u Sep 26 2013 St even Levin e

Resolves: #884758, #893575, #969525, #969139, #987151, #987623

Small updates to fence device parameter tables.

Resolves: #901637, #983739, 986462

Small updates to resource parameter tables.

Resolves: #633495

Documents configuration of nfsexpo rt and nfsserver resources.

Resolves: #852966, #975512, #977194, #991297, #874211, #908328, #919600,#955405, #972521,

#986474, #987135, #698454, #967986

Small corrections and clarifications throughout document.

St even Levin e Revisio n 6 .0- 3 T h u Sep 05 2013

Major review revision for all 6.5 BZ s.

Revisio n 6 .0- 2 Fri Ju n 14 2013

Adding section on configuring nfsserver and nfsexport.

St even Levin e

Revisio n 6 .0- 1 T h u Ju n 13 2013

Updating cluster update procedures and qdisk considerations.

St even Levin e

Revisio n 5.0- 25

Version for 6.4 GA release

Mo n Feb 18 2013 St even Levin e

Revisio n 5.0- 23 Wed Jan 30 2013

Resolves: 901641

Corrects and clarifies iptables rules.

St even Levin e

St even Levin e Revisio n 5.0- 22 T u e Jan 29 2013

Resolves: 788636

Documents RRP configuration through ccs command.

Resolves: 789010

Document RRP configuration in the cl uster. co nf file.

Revisio n 5.0- 20

Resolves: 894097

Fri Jan 18 2013

Removes advice to ensure you are not using VLAN tagging.

Resolves: 845365

Indicates that bonding modes 0 and 2 are now supported.

Revisio n 5.0- 19 T h u Jan 17 2013

Resolves: 896234

Clarifies terminology of cluster node references.

Mo n No v 26 2012 Revisio n 5.0- 16

Version for 6.4 Beta release

Revisio n 5.0- 15 Wed No v 20 2012

St even Levin e

233

Clust er Administ rat ion

Resolves: 838988

Documents nfsrestart attribute for file system resource agents.

Resolves: 843169

Documents IBM iPDU fence agent.

Resolves: 846121

Documents Eaton Network Power Controller (SNMP Interface) fence agent.

Resolves: 856834

Documents HP Bladesystem fence agent.

Resolves: 865313

Documents NFS Server resource agent.

Resolves: 862281

Clarifies which ccs commands overwrite previous settings.

Resolves: 846205

Documents i ptabl es firewall filtering for i g mp component.

Resolves: 857172

Documents ability to remove users from luci.

Resolves: 857165

Documents the privilege level parameter of the IPMI fence agent.

Resolves: 840912

Clears up formatting issue with resource parameter table.

Resolves: 849240, 870292

Clarifies installation procedure.

Resolves: 871165

Clarifies description of IP address parameter in description of IP address resource agent.

Resolves: 845333, 869039, 856681

Fixes small typographical errors and clarifies small technical ambiguities.

Revisio n 5.0- 12 T h u No v 1 2012

Added newly-supported fence agents.

Revisio n 5.0- 7 T h u O ct 25 2012

Added section on override semantics.

Revisio n 5.0- 6 T u e O ct 23 2012

Fixed default value of Post Join Delay.

Revisio n 5.0- 4 T u e O ct 16 2012

Added description of NFS server resource.

Revisio n 5.0- 2 T h u O ct 11 2012

St even Levin e

234

⁠Appendix G . Revision Hist ory

Updates to Conga descriptions.

Revisio n 5.0- 1

Clarifying ccs semantics

Mo n O ct 8 2012 St even Levin e

Revisio n 4 .0- 5

Version for 6.3 GA release

Fri Ju n 15 2012

Revisio n 4 .0- 4 T u e Ju n 12 2012

Resolves: 830148

Ensures consistency of port number examples for luci.

St even Levin e

Revisio n 4 .0- 3 T u e May 21 2012 St even Levin e

Resolves: 696897

Adds cluster.conf parameter information to tables of fence device parameters and resource parameters.

Resolves: 811643

Adds procedure for restoring a lu ci database on a separate machine.

Revisio n 4 .0- 2

Resolves: 815619

Wed Ap r 25 2012 St even Levin e

Removes warning about using UDP Unicast with GFS2 file systems.

Revisio n 4 .0- 1 Fri Mar 30 2012 St even Levin e

Resolves: 771447, 800069, 800061

Updates documentation of lu ci to be consistent with Red Hat Enterprise Linux 6.3 version.

Resolves: 712393

Adds information on capturing an application core for RGManager.

Resolves: 800074

Documents co nd o r resource agent.

Resolves: 757904

Documents lu ci configuration backup and restore.

Resolves: 772374

Adds section on managing virtual machines in a cluster.

Resolves: 712378

Adds documentation for HA-LVM configuration.

Resolves: 712400

Documents debug options.

Resolves: 751156

Documents new fence_i pmi l an parameter.

Resolves: 721373

Documents which configuration changes require a cluster restart.

Revisio n 3.0- 5 T h u Dec 1 2011 St even Levin e

235

Clust er Administ rat ion

Release for GA of Red Hat Enterprise Linux 6.2

Resolves: 755849

Corrects monitor_link parameter example.

Revisio n 3.0- 4 Mo n No v 7 2011

Resolves: 749857

Adds documentation for RHEV-M REST API fence device.

St even Levin e

Revisio n 3.0- 3 Fri O ct 21 2011 St even Levin e

Resolves: #747181, #747182, #747184, #747185, #747186, #747187, #747188, #747189, #747190,

#747192

Corrects typographical errors and ambiguities found during documentation QE review for Red Hat

Enterprise Linux 6.2.

Revisio n 3.0- 2 Fri O ct 7 2011 St even Levin e

Resolves: #743757

Corrects reference to supported bonding mode in troubleshooting section.

Revisio n 3.0- 1 Wed Sep 28 2011 St even Levin e

236

⁠Appendix G . Revision Hist ory

Initial revision for Red Hat Enterprise Linux 6.2 Beta release

Resolves: #739613

Documents support for new ccs options to display available fence devices and available services.

Resolves: #707740

Documents updates to the Conga interface and documents support for setting user permissions to administer Conga.

Resolves: #731856

Documents supports for configuring lu ci by means of the /etc/sysco nfi g /l uci file.

Resolves: #736134

Documents support for UDPU transport.

Resolves: #736143

Documents support for clustered Samba.

Resolves: #617634

Documents how to configure the only IP address lu ci is served at.

Resolves: #713259

Documents support for fence_vmware_so ap agent.

Resolves: #721009

Provides link to Support Essentials article.

Resolves: #717006

Provides information on allowing multicast traffic through the i ptabl es firewall.

Resolves: #717008

Provides information about cluster service status check and failover timeout.

Resolves: #711868

Clarifies description of autostart.

Resolves: #728337

Documents procedure for adding vm resources with the ccs command.

Resolves: #725315, #733011, #733074, #733689

Corrects small typographical errors.

Revisio n 2.0- 1 T h u May 19 2011 St even Levin e

237

Clust er Administ rat ion

Initial revision for Red Hat Enterprise Linux 6.1

Resolves: #671250

Documents support for SNMP traps.

Resolves: #659753

Documents ccs command.

Resolves: #665055

Updates Conga documentation to reflect updated display and feature support.

Resolves: #680294

Documents need for password access for ri cci agent.

Resolves: #687871

Adds chapter on troubleshooting.

Resolves: #673217

Fixes typographical error.

Resolves: #675805

Adds reference to cl uster. co nf schema to tables of HA resource parameters.

Resolves: #672697

Updates tables of fence device parameters to include all currently supported fencing devices.

Resolves: #677994

Corrects information for fence_i l o fence agent parameters.

Resolves: #629471

Adds technical note about setting consensus value in a two-node cluster.

Resolves: #579585

Updates section on upgrading Red Hat High Availability Add-On Software.

Resolves: #643216

Clarifies small issues throughout document.

Resolves: #643191

Provides improvements and corrections for the lu ci documentation.

Resolves: #704539

Updates the table of Virtual Machine resource parameters.

Revisio n 1.0- 1 Wed No v 10 2010

Initial release for Red Hat Enterprise Linux 6

Pau l Ken n ed y

Index

ACPI

238

⁠Index

b eh avio r, HA reso u rces, HA Reso u rce Beh avio r

Bro cad e f ab ric swit ch f en ce d evice ,

Configuring ACPI For Use with Integrated Fence Devices

- configuring,

APC p o wer swit ch o ver SNMP f en ce d evice , Fen ce Device Paramet ers

APC p o wer swit ch o ver t eln et /SSH f en ce d evice , Fen ce Device Paramet ers

CISCO MDS f en ce d evice , Fen ce Device Paramet ers

Cisco UCS f en ce d evice , Fen ce Device Paramet ers

clu st er

administration, Before Configuring the Red Hat High Availability Add-On

, Managing

Red Hat High Availability Add-On With Conga , Managing Red Hat High Availability

Add-On With ccs , Managing Red Hat High Availability Add-On With Command Line

Tools

- diagnosing and correcting problems, Diagnosing and Correcting Problems in a

Cluster , Diagnosing and Correcting Problems in a Cluster

starting, stopping, restarting, Starting and Stopping the Cluster Software

clu st er ad min ist rat io n , Bef o re Co n f ig u rin g t h e Red Hat Hig h Availab ilit y Ad d - O n ,

Man ag in g Red Hat Hig h Availab ilit y Ad d - O n Wit h Co n g a

, Man ag in g Red Hat Hig h

Availab ilit y Ad d - O n Wit h ccs , Man ag in g Red Hat Hig h Availab ilit y Ad d - O n Wit h

Co mman d Lin e T o o ls

adding cluster node, Adding a Member to a Running Cluster

, Adding a Member to a

Running Cluster

- compatible hardware,

Compatible Hardware

- configuration validation,

Configuration Validation

configuring ACPI, Configuring ACPI For Use with Integrated Fence Devices

configuring iptables, Enabling IP Ports

- considerations for using qdisk,

Considerations for Using Quorum Disk

considerations for using quorum disk, Considerations for Using Quorum Disk

deleting a cluster, Starting, Stopping, Restarting, and Deleting Clusters

- deleting a node from the configuration; adding a node to the configuration , Deleting or Adding a Node

- diagnosing and correcting problems in a cluster, Diagnosing and Correcting

Problems in a Cluster , Diagnosing and Correcting Problems in a Cluster

- displaying HA services with clustat,

Displaying HA Service Status with clustat

- enabling IP ports,

Enabling IP Ports

- general considerations,

General Configuration Considerations

joining a cluster, Causing a Node to Leave or Join a Cluster

, Causing a Node to

Leave or Join a Cluster

leaving a cluster, Causing a Node to Leave or Join a Cluster

, Causing a Node to

Leave or Join a Cluster

- managing cluster node,

Managing Cluster Nodes

managing high-availability services, Managing High-Availability Services

Managing High-Availability Services

- managing high-availability services, freeze and unfreeze, Managing HA Services with clusvcadm , Considerations for Using the Freeze and Unfreeze Operations

network switches and multicast addresses, Multicast Addresses

NetworkManager, Considerations for NetworkManager

rebooting cluster node, Rebooting a Cluster Node

removing cluster node, Deleting a Member from a Cluster

239

Clust er Administ rat ion

restarting a cluster, Starting, Stopping, Restarting, and Deleting Clusters

- ricci considerations,

Considerations for ricci

SELinux, Red Hat High Availability Add-On and SELinux

- starting a cluster,

Starting, Stopping, Restarting, and Deleting Clusters

, Starting and

Stopping a Cluster

- starting, stopping, restarting a cluster,

Starting and Stopping the Cluster Software

- stopping a cluster,

Starting, Stopping, Restarting, and Deleting Clusters

, Starting and Stopping a Cluster

- updating a cluster configuration using cman_tool version -r, Updating a

Configuration Using cman_tool version -r

- updating a cluster configuration using scp,

Updating a Configuration Using scp

- updating configuration,

Updating a Configuration

virtual machines, Configuring Virtual Machines in a Clustered Environment

clu st er co n f ig u rat io n ,

Co n f ig u rin g Red Hat Hig h Availab ilit y Ad d - O n Wit h Co n g a

Co n f ig u rin g Red Hat Hig h Availab ilit y Ad d - O n Wit h t h e ccs Co mman d

, Co n f ig u rin g

Red Hat Hig h Availab ilit y Man u ally

deleting or adding a node, Deleting or Adding a Node

updating, Updating a Configuration

clu st er reso u rce relat io n sh ip s, Paren t , Ch ild , an d Sib lin g Relat io n sh ip s Amo n g

Reso u rces

clu st er reso u rce st at u s ch eck, Clu st er Service Reso u rce Ch eck an d Failo ver

T imeo u t

clu st er reso u rce t yp es, Co n sid erat io n s f o r Co n f ig u rin g HA Services

clu st er service man ag ers

configuration, Adding a Cluster Service to the Cluster

, Adding a Cluster Service to the Cluster , Adding a Cluster Service to the Cluster

clu st er services, Ad d in g a Clu st er Service t o t h e Clu st er

, Ad d in g a Clu st er Service t o t h e Clu st er , Ad d in g a Clu st er Service t o t h e Clu st er

- (see also adding to the cluster configuration)

clu st er so f t ware

configuration, Configuring Red Hat High Availability Add-On With Conga ,

Configuring Red Hat High Availability Add-On With the ccs Command

, Configuring

Red Hat High Availability Manually

co n f ig u rat io n

HA service, Considerations for Configuring HA Services

Co n f ig u rin g Hig h Availab ilit y LVM, Hig h Availab ilit y LVM ( HA- LVM)

Co n g a

- accessing,

Configuring Red Hat High Availability Add-On Software

co n sen su s valu e, T h e co n sen su s Valu e f o r t o t em in a T wo - No d e Clu st er

Dell DRAC 5 f en ce d evice , Fen ce Device Paramet ers

Dell iDRAC f en ce d evice , Fen ce Device Paramet ers

Eat o n n et wo rk p o wer swit ch , Fen ce Device Paramet ers

24 0

Eg en era Blad eFrame f en ce d evice ,

Emerso n n et wo rk p o wer swit ch f en ce d evice , Fen ce Device Paramet ers

ePo werSwit ch f en ce d evice ,

f ailo ver t imeo u t , Clu st er Service Reso u rce Ch eck an d Failo ver T imeo u t

f eat u res, n ew an d ch an g ed , New an d Ch an g ed Feat u res

f eed b ack,

Feed b ack

f en ce ag en t

- fence_apc,

fence_apc_snmp, Fence Device Parameters

fence_bladecenter, Fence Device Parameters

fence_brocade, Fence Device Parameters

fence_cisco_mds, Fence Device Parameters

fence_cisco_ucs, Fence Device Parameters

fence_drac5, Fence Device Parameters

fence_eaton_snmp, Fence Device Parameters

- fence_egenera,

- fence_emerson,

fence_eps, Fence Device Parameters

fence_hpblade, Fence Device Parameters

fence_ibmblade, Fence Device Parameters

- fence_idrac,

- fence_ifmib,

- fence_ilo,

- fence_ilo2,

- fence_ilo3,

fence_ilo4, Fence Device Parameters

fence_ilo_mp, Fence Device Parameters

fence_imm, Fence Device Parameters

- fence_intelmodular,

fence_ipdu, Fence Device Parameters

fence_ipmilan, Fence Device Parameters

- fence_kdump,

- fence_rhevm,

fence_rsb, Fence Device Parameters

fence_scsi, Fence Device Parameters

- fence_virt,

- fence_vmware_soap,

fence_wti, Fence Device Parameters

fence_xvm, Fence Device Parameters

f en ce d evice

APC power switch over SNMP, Fence Device Parameters

APC power switch over telnet/SSH, Fence Device Parameters

Brocade fabric switch, Fence Device Parameters

- Cisco MDS,

- Cisco UCS,

- Dell DRAC 5,

- Dell iDRAC,

Eaton network power switch, Fence Device Parameters

- Egenera BladeFrame,

- Emerson network power switch,

ePowerSwitch, Fence Device Parameters

⁠Index

24 1

Clust er Administ rat ion

- Fence virt (Multicast Mode),

Fence virt (Serial/VMChannel Mode), Fence Device Parameters

Fujitsu Siemens Remoteview Service Board (RSB), Fence Device Parameters

- HP BladeSystem,

HP iLO, Fence Device Parameters

- HP iLO MP,

HP iLO2, Fence Device Parameters

HP iLO3, Fence Device Parameters

- HP iLO4,

- IBM BladeCenter,

IBM BladeCenter SNMP, Fence Device Parameters

IBM Integrated Management Module, Fence Device Parameters

IBM iPDU, Fence Device Parameters

IF MIB, Fence Device Parameters

Intel Modular, Fence Device Parameters

- IPMI LAN,

- RHEV-M REST API,

SCSI fencing, Fence Device Parameters

- VMware (SOAP interface),

- WTI power switch,

Fen ce virt f en ce d evice , Fen ce Device Paramet ers f en ce_ap c f en ce ag en t , Fen ce Device Paramet ers

f en ce_ap c_sn mp f en ce ag en t ,

f en ce_b lad ecen t er f en ce ag en t ,

Fen ce Device Paramet ers f en ce_cisco _md s f en ce ag en t , Fen ce Device Paramet ers

f en ce_b ro cad e f en ce ag en t ,

f en ce_cisco _u cs f en ce ag en t ,

Fen ce Device Paramet ers f en ce_eat o n _sn mp f en ce ag en t , Fen ce Device Paramet ers f en ce_eg en era f en ce ag en t , Fen ce Device Paramet ers

f en ce_d rac5 f en ce ag en t ,

f en ce_emerso n f en ce ag en t ,

Fen ce Device Paramet ers f en ce_ep s f en ce ag en t , Fen ce Device Paramet ers

f en ce_h p b lad e f en ce ag en t ,

Fen ce Device Paramet ers f en ce_ib mb lad e f en ce ag en t , Fen ce Device Paramet ers f en ce_id rac f en ce ag en t , Fen ce Device Paramet ers f en ce_if mib f en ce ag en t , Fen ce Device Paramet ers

f en ce_ilo f en ce ag en t ,

f en ce_ilo 2 f en ce ag en t ,

Fen ce Device Paramet ers f en ce_ilo 4 f en ce ag en t , Fen ce Device Paramet ers f en ce_ilo _mp f en ce ag en t , Fen ce Device Paramet ers f en ce_imm f en ce ag en t , Fen ce Device Paramet ers f en ce_in t elmo d u lar f en ce ag en t , Fen ce Device Paramet ers f en ce_ip d u f en ce ag en t , Fen ce Device Paramet ers

f en ce_ilo 3 f en ce ag en t ,

f en ce_ip milan f en ce ag en t ,

f en ce_kd u mp f en ce ag en t ,

24 2

f en ce_rh evm f en ce ag en t ,

f en ce_rsb f en ce ag en t ,

f en ce_scsi f en ce ag en t ,

Fen ce Device Paramet ers f en ce_vmware_so ap f en ce ag en t , Fen ce Device Paramet ers

f en ce_virt f en ce ag en t ,

f en ce_wt i f en ce ag en t ,

Fen ce Device Paramet ers f en ce_xvm f en ce ag en t , Fen ce Device Paramet ers

Fu jit su Siemen s Remo t eview Service Bo ard ( RSB) f en ce d evice, Fen ce Device

Paramet ers

G g en eral

- considerations for cluster administration,

General Configuration Considerations

HA service co n f ig u rat io n

overview, Considerations for Configuring HA Services

h ard ware

compatible, Compatible Hardware

HP Blad esyst em f en ce d evice , Fen ce Device Paramet ers

HP iLO f en ce d evice, Fen ce Device Paramet ers

HP iLO MP f en ce d evice ,

HP iLO 2 f en ce d evice, Fen ce Device Paramet ers

HP iLO 3 f en ce d evice, Fen ce Device Paramet ers

HP iLO 4 f en ce d evice,

IBM Blad eCen t er f en ce d evice , Fen ce Device Paramet ers

IBM Blad eCen t er SNMP f en ce d evice ,

IBM In t eg rat ed Man ag emen t Mo d u le f en ce d evice , Fen ce Device Paramet ers

IBM iPDU f en ce d evice ,

IF MIB f en ce d evice ,

configuring ACPI, Configuring ACPI For Use with Integrated Fence Devices

in t eg rat ed f en ce d evices

In t el Mo d u lar f en ce d evice ,

other Red Hat Enterprise Linux documents, Introduction

in t ro d u ct io n ,

In t ro d u ct io n

IP p o rt s

- enabling,

Enabling IP Ports

IPMI LAN f en ce d evice ,

ip t ab les f irewall, Co n f ig u rin g t h e ip t ab les Firewall t o Allo w Clu st er Co mp o n en t s

ip t ab les

- configuring,

Enabling IP Ports

⁠Index

24 3

Clust er Administ rat ion

LVM, Hig h Availab ilit y, Hig h Availab ilit y LVM ( HA- LVM)

M mu lt icast ad d resses

- considerations for using with network switches and multicast addresses, Multicast

Addresses

mu lt icast t raf f ic, en ab lin g , Co n f ig u rin g t h e ip t ab les Firewall t o Allo w Clu st er

Co mp o n en t s

Net wo rkMan ag er

disable for use with cluster, Considerations for NetworkManager

n f sexp o rt reso u rce, co n f ig u rin g , Co n f ig u rin g n f sexp o rt an d n f sserver Reso u rces

n f sserver reso u rce, co n f ig u rin g ,

Co n f ig u rin g n f sexp o rt an d n f sserver Reso u rces

O o verview

- features, new and changed,

New and Changed Features

P p aramet ers, f en ce d evice,

Considerations for Using Quorum Disk

p aramet ers, HA reso u rces,

HA Reso u rce Paramet ers

Q q d isk

- considerations for using,

q u o ru m d isk

- considerations for using,

Considerations for Using Quorum Disk

R relat io n sh ip s

- cluster resource,

Parent, Child, and Sibling Relationships Among Resources

RHEV- M REST API f en ce d evice , Fen ce Device Paramet ers

ricci

- considerations for cluster administration,

Considerations for ricci

24 4

SCSI f en cin g , Fen ce Device Paramet ers

SELin u x

- configuring,

Red Hat High Availability Add-On and SELinux

st at u s ch eck, clu st er reso u rce, Clu st er Service Reso u rce Ch eck an d Failo ver

T imeo u t

t ab les

- fence devices, parameters,