Performance Tuning and Optimization for CentOS/RHEL Systems

Tyler Maginnis | February 19, 2024

performancetuningoptimizationCentOSRHELsystem performance

Need Professional CentOS/RHEL Support?

Get expert assistance with your centos/rhel support implementation and management. Tyler on Tech Louisville provides priority support for Louisville businesses.

Call Now: (202) 948-8888 Email for Quote

Same-day service available for Louisville area

Performance Tuning and Optimization for CentOS/RHEL Systems

System performance optimization is crucial for enterprise CentOS/RHEL deployments. This guide covers comprehensive tuning techniques for CPU, memory, disk I/O, network performance, and kernel parameters.

Performance Analysis Tools

System Monitoring Basics

Essential performance monitoring tools:

# Real-time system statistics
top -c
htop

# CPU statistics
mpstat -P ALL 1
sar -u 1 10

# Memory statistics
free -h
vmstat 1
sar -r 1 10

# Disk I/O statistics
iostat -xz 1
iotop -o
sar -d 1 10

# Network statistics
netstat -i
ss -s
sar -n DEV 1 10

# Process statistics
pidstat 1
ps aux --sort=-%cpu | head

Advanced Performance Analysis

Deep system analysis tools:

# Performance monitoring with perf
perf top
perf record -a -g sleep 10
perf report

# System call tracing
strace -c -p $(pidof nginx)
strace -ttT -p $(pidof httpd)

# CPU profiling
perf stat -a sleep 10
perf stat -p $(pidof mysqld)

# Cache analysis
perf stat -e cache-references,cache-misses,instructions,cycles sleep 10

# Flame graphs
perf record -F 99 -a -g -- sleep 30
perf script | stackcollapse-perf.pl | flamegraph.pl > perf.svg

CPU Performance Tuning

CPU Scheduling Optimization

Configure CPU scheduler parameters:

# Check current scheduler
cat /sys/block/sda/queue/scheduler

# Set scheduler (deadline for databases)
echo deadline > /sys/block/sda/queue/scheduler

# Configure CPU governor
cpupower frequency-info
cpupower frequency-set -g performance

# Set CPU affinity
taskset -c 0-3 nginx
taskset -pc 0,1 $(pidof mysqld)

# Configure IRQ affinity
echo 2 > /proc/irq/24/smp_affinity
echo 4 > /proc/irq/25/smp_affinity

Process Priority Tuning

Optimize process scheduling:

# Set nice values
nice -n -5 /usr/bin/important-app
renice -n -10 -p $(pidof critical-service)

# Real-time scheduling
chrt -f -p 50 $(pidof audio-app)
chrt -r -p 80 $(pidof realtime-app)

# CPU sets for isolation
yum install libcgroup-tools
cgcreate -g cpu:/isolated
echo 4-7 > /sys/fs/cgroup/cpu/isolated/cpuset.cpus
cgexec -g cpu:isolated /usr/bin/app

Memory Optimization

Virtual Memory Tuning

Configure memory management:

# Swappiness configuration
echo "vm.swappiness = 10" >> /etc/sysctl.d/99-memory.conf
sysctl -w vm.swappiness=10

# Dirty page management
cat >> /etc/sysctl.d/99-memory.conf <<EOF
vm.dirty_background_ratio = 5
vm.dirty_ratio = 10
vm.dirty_expire_centisecs = 3000
vm.dirty_writeback_centisecs = 500
EOF

# Page cache pressure
echo "vm.vfs_cache_pressure = 50" >> /etc/sysctl.d/99-memory.conf

# OOM killer tuning
echo "vm.overcommit_memory = 1" >> /etc/sysctl.d/99-memory.conf
echo "vm.panic_on_oom = 0" >> /etc/sysctl.d/99-memory.conf

Huge Pages Configuration

Enable transparent huge pages:

# Check huge pages status
cat /sys/kernel/mm/transparent_hugepage/enabled

# Configure huge pages
echo "vm.nr_hugepages = 1024" >> /etc/sysctl.d/99-hugepages.conf
echo "vm.hugetlb_shm_group = 1001" >> /etc/sysctl.d/99-hugepages.conf

# Disable transparent huge pages for databases
echo never > /sys/kernel/mm/transparent_hugepage/enabled
echo never > /sys/kernel/mm/transparent_hugepage/defrag

# Persistent configuration
cat >> /etc/rc.local <<EOF
echo never > /sys/kernel/mm/transparent_hugepage/enabled
echo never > /sys/kernel/mm/transparent_hugepage/defrag
EOF

Disk I/O Optimization

Storage Performance Tuning

Optimize disk performance:

# I/O scheduler tuning
echo noop > /sys/block/nvme0n1/queue/scheduler  # For NVMe
echo deadline > /sys/block/sdb/queue/scheduler  # For SSD
echo cfq > /sys/block/sdc/queue/scheduler       # For HDD

# Read-ahead configuration
blockdev --setra 256 /dev/sda
echo 256 > /sys/block/sda/queue/read_ahead_kb

# Queue depth tuning
echo 256 > /sys/block/sda/queue/nr_requests

# Filesystem mount options
mount -o noatime,nodiratime,nobarrier /dev/sdb1 /data

Filesystem Optimization

Tune filesystem parameters:

# XFS optimization
xfs_admin -L "DATA" /dev/sdb1
mount -o noatime,nodiratime,logbufs=8,logbsize=256k /dev/sdb1 /data

# EXT4 optimization
tune2fs -o journal_data_writeback /dev/sdc1
tune2fs -O ^has_journal /dev/sdc1  # Disable journal for read-heavy
mount -o noatime,nodiratime,data=writeback /dev/sdc1 /data

# Create optimized filesystem
mkfs.xfs -f -d agcount=32 -l size=512m,lazy-count=1 /dev/sdb1
mkfs.ext4 -E stride=16,stripe-width=64 /dev/sdc1

Network Performance Tuning

TCP/IP Stack Optimization

Configure network parameters:

# TCP tuning
cat >> /etc/sysctl.d/99-network.conf <<EOF
# TCP memory
net.ipv4.tcp_mem = 786432 1048576 26777216
net.ipv4.tcp_rmem = 4096 87380 134217728
net.ipv4.tcp_wmem = 4096 65536 134217728

# TCP behavior
net.ipv4.tcp_congestion_control = bbr
net.ipv4.tcp_fastopen = 3
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_fin_timeout = 30
net.ipv4.tcp_keepalive_time = 1200
net.ipv4.tcp_keepalive_probes = 3
net.ipv4.tcp_keepalive_intvl = 30

# Connection handling
net.ipv4.ip_local_port_range = 1024 65535
net.ipv4.tcp_max_syn_backlog = 8192
net.core.somaxconn = 8192
net.core.netdev_max_backlog = 5000
EOF

sysctl -p /etc/sysctl.d/99-network.conf

Network Interface Tuning

Optimize NIC performance:

# Enable offloading features
ethtool -K eth0 rx on tx on sg on tso on gso on gro on lro on

# Increase ring buffers
ethtool -G eth0 rx 4096 tx 4096

# Configure interrupt coalescing
ethtool -C eth0 rx-usecs 10 tx-usecs 10

# Multi-queue configuration
ethtool -L eth0 combined 8

# RSS configuration
ethtool -X eth0 equal 8

Kernel Parameter Tuning

Core Kernel Parameters

Essential kernel optimizations:

# Create comprehensive sysctl configuration
cat > /etc/sysctl.d/99-performance.conf <<EOF
# File system
fs.file-max = 2097152
fs.nr_open = 2097152

# Process limits
kernel.pid_max = 4194304
kernel.threads-max = 4194304

# Shared memory
kernel.shmmax = 68719476736
kernel.shmall = 4294967296
kernel.shmmni = 4096

# Semaphores
kernel.sem = 250 32000 100 128

# Core dumps
kernel.core_uses_pid = 1
kernel.core_pattern = /var/core/core.%e.%p.%t

# NUMA
kernel.numa_balancing = 0

# Scheduler
kernel.sched_migration_cost_ns = 5000000
kernel.sched_autogroup_enabled = 0
EOF

sysctl -p /etc/sysctl.d/99-performance.conf

Security vs Performance Trade-offs

Balance security and performance:

# Disable unnecessary security features for performance
echo 0 > /proc/sys/kernel/randomize_va_space  # Disable ASLR
setenforce 0  # Disable SELinux (not recommended for production)

# Optimize for performance with some security
cat >> /etc/sysctl.d/99-balanced.conf <<EOF
# Moderate security with performance
kernel.dmesg_restrict = 1
kernel.kptr_restrict = 1
kernel.yama.ptrace_scope = 1
net.ipv4.tcp_syncookies = 1
net.ipv4.conf.all.rp_filter = 1
EOF

Application-Specific Tuning

Database Server Optimization

Tune for database workloads:

# MySQL/MariaDB optimization
cat >> /etc/sysctl.d/99-mysql.conf <<EOF
# InnoDB optimization
vm.swappiness = 1
vm.dirty_background_ratio = 3
vm.dirty_ratio = 15

# Disable NUMA for databases
kernel.numa_balancing = 0
EOF

# Set limits for mysql user
cat >> /etc/security/limits.d/mysql.conf <<EOF
mysql soft nofile 65535
mysql hard nofile 65535
mysql soft nproc 65535
mysql hard nproc 65535
EOF

# Optimize storage for database
mount -o noatime,nodiratime,nobarrier /dev/sdb1 /var/lib/mysql

Web Server Optimization

Configure for web workloads:

# Nginx optimization
cat >> /etc/sysctl.d/99-nginx.conf <<EOF
# Connection optimization
net.core.somaxconn = 65535
net.ipv4.tcp_max_tw_buckets = 1440000
net.ipv4.ip_local_port_range = 1024 65535
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_timestamps = 0
EOF

# Apache optimization
cat >> /etc/security/limits.d/apache.conf <<EOF
apache soft nofile 65535
apache hard nofile 65535
EOF

Performance Testing

Benchmark Tools

System performance testing:

# CPU benchmark
sysbench cpu --cpu-max-prime=20000 run
stress-ng --cpu 4 --timeout 60s --metrics

# Memory benchmark
sysbench memory --memory-total-size=10G run
stress-ng --vm 2 --vm-bytes 1G --timeout 60s

# Disk I/O benchmark
fio --name=randwrite --ioengine=libaio --rw=randwrite --bs=4k --direct=1 --size=1G --numjobs=4 --runtime=60
dd if=/dev/zero of=/tmp/test bs=1G count=1 oflag=direct

# Network benchmark
iperf3 -s  # Server
iperf3 -c server_ip -t 30  # Client

Load Testing

Application load testing:

# Web server load testing
ab -n 10000 -c 100 http://localhost/
siege -c 100 -t 60s http://localhost/

# Database load testing
sysbench oltp_read_write --mysql-host=localhost --mysql-user=root --mysql-password=pass --mysql-db=test --tables=10 --table-size=100000 prepare
sysbench oltp_read_write --mysql-host=localhost --mysql-user=root --mysql-password=pass --mysql-db=test --tables=10 --table-size=100000 --threads=16 --time=60 run

Monitoring and Automation

Performance Monitoring Setup

Automated monitoring configuration:

# Install monitoring tools
yum install sysstat iotop htop dstat

# Configure sysstat collection
sed -i 's/ENABLED="false"/ENABLED="true"/' /etc/default/sysstat
systemctl enable --now sysstat

# Create monitoring script
cat > /usr/local/bin/performance-monitor.sh <<'EOF'
#!/bin/bash

LOG_DIR="/var/log/performance"
mkdir -p $LOG_DIR

# Collect system metrics
date >> $LOG_DIR/metrics.log
echo "=== CPU Usage ===" >> $LOG_DIR/metrics.log
mpstat 1 5 >> $LOG_DIR/metrics.log

echo "=== Memory Usage ===" >> $LOG_DIR/metrics.log
free -h >> $LOG_DIR/metrics.log

echo "=== Disk I/O ===" >> $LOG_DIR/metrics.log
iostat -xz 1 5 >> $LOG_DIR/metrics.log

echo "=== Network Stats ===" >> $LOG_DIR/metrics.log
ss -s >> $LOG_DIR/metrics.log

# Check for performance issues
CPU_USAGE=$(top -bn1 | grep "Cpu(s)" | awk '{print $2}' | cut -d'%' -f1)
if (( $(echo "$CPU_USAGE > 80" | bc -l) )); then
    echo "WARNING: High CPU usage detected: $CPU_USAGE%" | mail -s "Performance Alert" admin@example.com
fi
EOF

chmod +x /usr/local/bin/performance-monitor.sh

# Schedule monitoring
echo "*/5 * * * * /usr/local/bin/performance-monitor.sh" | crontab -

Tuning Profiles

Use tuned for automatic optimization:

# Install tuned
yum install tuned

# List available profiles
tuned-adm list

# Apply performance profile
tuned-adm profile throughput-performance

# Create custom profile
mkdir /etc/tuned/custom-web
cat > /etc/tuned/custom-web/tuned.conf <<EOF
[main]
include=throughput-performance

[sysctl]
net.core.somaxconn=65535
net.ipv4.tcp_max_syn_backlog=8192
vm.swappiness=10

[vm]
transparent_hugepages=never
EOF

tuned-adm profile custom-web

Best Practices

Performance Tuning Methodology

  1. Baseline Measurement: Establish performance baseline
  2. Identify Bottlenecks: Use monitoring tools
  3. Make Single Changes: Test one change at a time
  4. Measure Impact: Compare with baseline
  5. Document Changes: Keep tuning log
  6. Monitor Long-term: Watch for degradation

Performance Checklist

#!/bin/bash
# Performance audit script

echo "=== System Performance Audit ==="

# Check CPU governor
echo "CPU Governor: $(cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor)"

# Check swappiness
echo "Swappiness: $(cat /proc/sys/vm/swappiness)"

# Check transparent huge pages
echo "THP: $(cat /sys/kernel/mm/transparent_hugepage/enabled)"

# Check I/O scheduler
echo "I/O Schedulers:"
for disk in /sys/block/sd*; do
    echo "  $(basename $disk): $(cat $disk/queue/scheduler)"
done

# Check network optimizations
echo "TCP Congestion Control: $(sysctl net.ipv4.tcp_congestion_control | awk '{print $3}')"

# Check limits
echo "Open Files Limit: $(ulimit -n)"
echo "Process Limit: $(ulimit -u)"

Conclusion

Performance tuning is an iterative process requiring careful measurement and testing. Apply optimizations gradually, monitor their impact, and maintain documentation. Regular performance audits ensure sustained optimal system operation in production environments.