High Availability with Pacemaker and Corosync on CentOS/RHEL

Tyler Maginnis | February 20, 2024

high availabilityPacemakerCorosyncclusteringCentOSRHELHA

Need Professional CentOS/RHEL Support?

Get expert assistance with your centos/rhel support implementation and management. Tyler on Tech Louisville provides priority support for Louisville businesses.

Call Now: (202) 948-8888 Email for Quote

Same-day service available for Louisville area

High Availability with Pacemaker and Corosync on CentOS/RHEL

High Availability (HA) clusters ensure continuous service availability by eliminating single points of failure. This guide covers building HA clusters using Pacemaker resource manager and Corosync cluster engine on CentOS/RHEL systems.

High Availability Fundamentals

HA Architecture Overview

Understanding cluster components:

# Cluster stack components:
# - Corosync: Cluster communication layer
# - Pacemaker: Resource management layer
# - Resource Agents: Service control scripts
# - Fencing Agents: Node isolation mechanisms

# Check cluster software versions
rpm -qa | grep -E "corosync|pacemaker|pcs"
corosync -v
pacemakerd --version

Prerequisites and Planning

Prepare cluster environment:

# Network requirements
# - Dedicated cluster network (recommended)
# - Multicast or unicast support
# - Low latency (<2ms) between nodes
# - Redundant network paths

# Time synchronization
yum install chrony
systemctl enable --now chronyd
chronyc sources

# Hostname resolution
cat >> /etc/hosts <<EOF
192.168.100.10 node1.example.com node1
192.168.100.11 node2.example.com node2
192.168.100.12 node3.example.com node3
EOF

# SELinux configuration
setsebool -P daemons_enable_cluster_mode on

Cluster Installation and Setup

Installing Cluster Software

Install required packages:

# Enable HA repository (RHEL)
subscription-manager repos --enable=rhel-8-for-x86_64-highavailability-rpms

# Install cluster packages
yum install -y pcs pacemaker corosync fence-agents-all resource-agents

# Enable cluster services
systemctl enable pcsd
systemctl start pcsd

# Set hacluster password (on all nodes)
echo "StrongPassword123!" | passwd --stdin hacluster

Initial Cluster Configuration

Create the cluster:

# Authenticate nodes (from node1)
pcs cluster auth node1 node2 node3 -u hacluster -p StrongPassword123!

# Create cluster
pcs cluster setup --name production_cluster node1 node2 node3

# Start cluster services
pcs cluster start --all
pcs cluster enable --all

# Verify cluster status
pcs cluster status
pcs status

# Check corosync ring status
corosync-cfgtool -s

Corosync Configuration

Advanced Corosync Setup

Configure cluster communication:

# Edit corosync configuration
cat > /etc/corosync/corosync.conf <<EOF
totem {
    version: 2
    cluster_name: production_cluster
    transport: knet
    crypto_cipher: aes256
    crypto_hash: sha256

    interface {
        ringnumber: 0
        bindnetaddr: 192.168.100.0
        mcastaddr: 239.255.1.1
        mcastport: 5405
        ttl: 1
    }

    interface {
        ringnumber: 1
        bindnetaddr: 192.168.200.0
        mcastaddr: 239.255.2.1
        mcastport: 5405
        ttl: 1
    }
}

logging {
    fileline: off
    to_stderr: no
    to_logfile: yes
    logfile: /var/log/cluster/corosync.log
    to_syslog: yes
    debug: off
    timestamp: on
    logger_subsys {
        subsys: QUORUM
        debug: off
    }
}

quorum {
    provider: corosync_votequorum
    expected_votes: 3
    two_node: 0
    wait_for_all: 1
    last_man_standing: 1
    auto_tie_breaker: 0
}
EOF

# Reload corosync configuration
pcs cluster reload corosync

Redundant Ring Configuration

Configure multiple communication paths:

# Configure redundant rings
pcs cluster setup --name production_cluster \
    node1 addr=192.168.100.10 addr=192.168.200.10 \
    node2 addr=192.168.100.11 addr=192.168.200.11 \
    node3 addr=192.168.100.12 addr=192.168.200.12 \
    --transport knet

# Verify ring status
corosync-cfgtool -s
pcs status corosync

Pacemaker Resource Management

Basic Resource Configuration

Create cluster resources:

# Create VIP resource
pcs resource create cluster_vip ocf:heartbeat:IPaddr2 \
    ip=192.168.100.100 \
    cidr_netmask=24 \
    op monitor interval=30s

# Create Apache resource
pcs resource create webserver ocf:heartbeat:apache \
    configfile=/etc/httpd/conf/httpd.conf \
    statusurl="http://localhost/server-status" \
    op monitor interval=30s timeout=20s

# Create filesystem resource
pcs resource create shared_storage ocf:heartbeat:Filesystem \
    device=/dev/sdb1 \
    directory=/var/www/html \
    fstype=xfs \
    op monitor interval=30s

Resource Groups and Constraints

Manage resource relationships:

# Create resource group
pcs resource group add webservice_group cluster_vip shared_storage webserver

# Resource colocation constraints
pcs constraint colocation add webserver with cluster_vip INFINITY

# Resource ordering constraints
pcs constraint order shared_storage then webserver

# Location constraints (preferred node)
pcs constraint location webserver prefers node1=100

# Anti-affinity constraints
pcs constraint colocation add resource1 with resource2 -INFINITY

# View constraints
pcs constraint list

Advanced Resource Types

Configure complex resources:

# Clone resources (active on all nodes)
pcs resource create dlm ocf:pacemaker:controld \
    op monitor interval=30s on-fail=restart \
    clone interleave=true ordered=true

# Master/Slave resources
pcs resource create drbd_data ocf:linbit:drbd \
    drbd_resource=data \
    op monitor interval=10s role=Master \
    op monitor interval=20s role=Slave \
    promotable promoted-max=1 promoted-node-max=1 \
    clone-max=2 clone-node-max=1 notify=true

# Multi-state resources
pcs resource create postgresql ocf:heartbeat:pgsql \
    pgctl="/usr/pgsql-13/bin/pg_ctl" \
    psql="/usr/pgsql-13/bin/psql" \
    pgdata="/var/lib/pgsql/13/data" \
    rep_mode="sync" \
    node_list="node1 node2 node3" \
    master-max=1 \
    op monitor interval=30s \
    promotable notify=true

Cluster Fencing (STONITH)

Configure Fencing Devices

Implement node isolation:

# Enable STONITH
pcs property set stonith-enabled=true

# Configure IPMI fencing
pcs stonith create ipmi-fencing fence_ipmilan \
    pcmk_host_list="node1,node2,node3" \
    ipaddr="192.168.100.200" \
    login="admin" \
    passwd="password" \
    lanplus=true \
    power_timeout=20

# Configure APC PDU fencing
pcs stonith create apc-fencing fence_apc \
    ipaddr="192.168.100.201" \
    login="apc" \
    passwd="apcpassword" \
    pcmk_host_map="node1:1;node2:2;node3:3" \
    power_timeout=20

# Configure SBD fencing (shared storage)
pcs stonith create sbd-fencing fence_sbd \
    devices="/dev/disk/by-id/scsi-xxxxx" \
    meta provides=unfencing

# Test fencing
pcs stonith fence node2

Fencing Strategies

Advanced fencing configurations:

# Fencing levels (try IPMI first, then PDU)
pcs stonith level add 1 node1 ipmi-fencing
pcs stonith level add 2 node1 apc-fencing

# Fencing topology
pcs stonith level add 1 node2 ipmi-fencing
pcs stonith level add 1 node3 ipmi-fencing

# Verify fencing configuration
pcs stonith show
pcs stonith fence --off node2

Cluster Properties and Behavior

Cluster-Wide Settings

Configure cluster behavior:

# Set cluster properties
pcs property set cluster-infrastructure=corosync
pcs property set no-quorum-policy=stop
pcs property set symmetric-cluster=true
pcs property set start-failure-is-fatal=false
pcs property set pe-warn-series-max=1000
pcs property set pe-input-series-max=1000
pcs property set pe-error-series-max=1000

# Resource defaults
pcs resource defaults update resource-stickiness=100
pcs resource defaults update migration-threshold=3
pcs resource defaults update failure-timeout=600

# Operation defaults
pcs resource op defaults update timeout=60s
pcs resource op defaults update on-fail=restart

Quorum Configuration

Manage cluster quorum:

# Configure quorum options
pcs quorum config
pcs quorum status

# Set expected votes
pcs quorum expected-votes 3

# Configure quorum device (QDevice)
yum install corosync-qdevice
pcs qdevice setup model net \
    --enable \
    --cluster-name production_cluster \
    algorithm=ffsplit \
    host=qdevice.example.com

# Add quorum device to cluster
pcs quorum device add model net \
    host=qdevice.example.com \
    algorithm=ffsplit

Monitoring and Maintenance

Cluster Monitoring

Monitor cluster health:

# Real-time monitoring
watch -n 1 'pcs status'
crm_mon -rfA

# Resource monitoring
pcs resource show
pcs resource failcount show

# Check cluster configuration
crm_verify -L -V

# View cluster logs
tail -f /var/log/cluster/corosync.log
journalctl -u pacemaker -f
journalctl -u corosync -f

Maintenance Operations

Perform cluster maintenance:

# Put node in maintenance mode
pcs node maintenance node2

# Put cluster in maintenance mode
pcs property set maintenance-mode=true

# Move resources manually
pcs resource move webserver node3

# Clear resource failures
pcs resource cleanup webserver

# Restart resource
pcs resource restart webserver

# Unmanage resource (disable cluster control)
pcs resource unmanage webserver

# Re-manage resource
pcs resource manage webserver

# Remove maintenance mode
pcs property set maintenance-mode=false
pcs node unstandby node2

Advanced Configurations

Split-Brain Prevention

Configure split-brain handling:

# Configure fencing delay
pcs property set stonith-timeout=120s
pcs property set startup-fencing=true

# Configure cluster communication timeout
pcs property set token=10000
pcs property set consensus=12000

# Auto-recovery from split-brain
pcs resource create split_brain_recovery ocf:pacemaker:Dummy \
    meta target-role=Started \
    op monitor interval=10s

Resource Migration Policies

Configure automated failover:

# Configure migration threshold
pcs resource update webserver meta migration-threshold=3

# Configure failure timeout
pcs resource update webserver meta failure-timeout=600

# Resource utilization and placement
pcs node utilization node1 cpu=8 memory=16384
pcs resource utilization webserver cpu=2 memory=4096

# Placement strategy
pcs property set placement-strategy=balanced

Disaster Recovery

Cluster Backup and Restore

Backup cluster configuration:

# Backup cluster configuration
pcs config backup backup-$(date +%Y%m%d)

# Export CIB
cibadmin -Q > cluster-cib-backup.xml

# Backup corosync config
cp /etc/corosync/corosync.conf /backup/corosync.conf.backup

# Restore cluster configuration
pcs config restore backup-20240220

# Import CIB
cibadmin --replace --xml-file cluster-cib-backup.xml

Geo-Clustering

Configure multi-site clusters:

# Install booth for ticket management
yum install booth-site booth-arbitrator

# Configure booth
cat > /etc/booth/booth.conf <<EOF
transport = UDP
port = 9929
arbitrator = 192.168.100.250
site = 192.168.100.10
site = 192.168.200.10
ticket = "service-ticket"
    expire = 600
    timeout = 10
    weight = 1000
EOF

# Create ticket constraint
pcs constraint ticket add service-ticket webservice_group

Troubleshooting

Common Issues

Debug cluster problems:

# Check cluster health
pcs status --full
crm_mon -1 -rf

# Verify configuration
crm_verify -LV
pcs cluster verify

# Resource debugging
pcs resource debug-start webserver

# Enable debug logging
pcs property set debug-mode=true

# Analyze transitions
crm_simulate -sL

# Check IPC communication
ipcs -a

Recovery Procedures

Recover from failures:

# Recover from complete cluster failure
pcs cluster stop --all
pcs cluster start --all

# Force resource recovery
pcs resource cleanup --force

# Clear fencing history
stonith_admin -H node2 -C

# Rejoin node to cluster
pcs cluster node remove node3
pcs cluster node add node3

Best Practices

Design Considerations

  1. Use dedicated cluster network: Separate cluster traffic from service traffic
  2. Implement proper fencing: Never disable STONITH in production
  3. Test failover scenarios: Regular disaster recovery drills
  4. Monitor cluster health: Implement alerting for cluster events
  5. Document procedures: Maintain runbooks for common operations

Health Check Script

#!/bin/bash
# Cluster health monitoring script

echo "=== Cluster Health Check ==="

# Check if cluster is running
if ! pcs cluster status &>/dev/null; then
    echo "ERROR: Cluster is not running"
    exit 1
fi

# Check quorum
if ! pcs quorum status | grep -q "Quorate:.*Yes"; then
    echo "WARNING: Cluster is not quorate"
fi

# Check for failed resources
failed_count=$(pcs resource failcount show | grep -c "Failures:")
if [ $failed_count -gt 0 ]; then
    echo "WARNING: $failed_count resources have failures"
fi

# Check STONITH status
if ! pcs property show stonith-enabled | grep -q "true"; then
    echo "CRITICAL: STONITH is disabled"
fi

# Check node status
offline_nodes=$(pcs status nodes | grep -c "Offline")
if [ $offline_nodes -gt 0 ]; then
    echo "WARNING: $offline_nodes nodes are offline"
fi

echo "Cluster health check complete"

Conclusion

Pacemaker and Corosync provide enterprise-grade high availability for CentOS/RHEL systems. Proper configuration, testing, and monitoring ensure reliable failover and continuous service availability. Regular maintenance and updates keep clusters running optimally in production environments.