Complete Backup and Disaster Recovery Guide for Ubuntu Server 22.04
A comprehensive backup and disaster recovery strategy is essential for business continuity. This guide covers various backup solutions, from simple scripts to enterprise-grade systems, ensuring your Ubuntu Server data remains safe and recoverable.
Prerequisites
- Ubuntu Server 22.04 LTS
- Root or sudo access
- Sufficient storage for backups
- Basic understanding of Linux file systems
Backup Strategy Planning
Types of Backups
- Full Backup: Complete copy of all data
- Incremental Backup: Only changes since last backup
- Differential Backup: Changes since last full backup
- Snapshot Backup: Point-in-time copy of system state
3-2-1 Rule
- 3 copies of important data
- 2 different storage media types
- 1 offsite backup
Built-in Backup Tools
Using tar for Archive Backups
Basic tar Backup
# Full system backup (excluding certain directories)
sudo tar -czpf /backup/system-$(date +%Y%m%d).tar.gz \
--exclude=/backup \
--exclude=/proc \
--exclude=/tmp \
--exclude=/mnt \
--exclude=/dev \
--exclude=/sys \
--exclude=/run \
--exclude=/media \
--exclude=/var/log \
--exclude=/var/cache \
--exclude=/home/*/.cache \
--exclude=/var/lib/lxcfs \
--one-file-system /
Incremental tar Backup
# Create snapshot file for incremental backups
sudo tar -czpf /backup/full-backup.tar.gz \
--listed-incremental=/backup/snapshot.snar \
/home /etc /var/www
# Subsequent incremental backups
sudo tar -czpf /backup/incremental-$(date +%Y%m%d).tar.gz \
--listed-incremental=/backup/snapshot.snar \
/home /etc /var/www
Restore from tar
# Full restore
sudo tar -xzpf /backup/system-20240120.tar.gz -C /
# Restore specific directory
sudo tar -xzpf /backup/system-20240120.tar.gz -C / home/user/documents
# List contents without extracting
tar -tzf /backup/system-20240120.tar.gz
Rsync for Efficient Backups
Local Rsync Backup
# Basic rsync backup
sudo rsync -avzh --progress \
--exclude={"/dev/*","/proc/*","/sys/*","/tmp/*","/run/*","/mnt/*","/media/*","/lost+found"} \
/ /backup/system/
# With deletion of removed files
sudo rsync -avzh --delete --progress \
/home/ /backup/home/
Remote Rsync Backup
# Backup to remote server
rsync -avzhe ssh --progress \
/home/user/documents/ \
user@backup-server:/backups/documents/
# Pull backup from remote
rsync -avzhe ssh --progress \
user@remote-server:/var/www/ \
/backup/www/
Rsync Backup Script
sudo nano /usr/local/bin/rsync-backup.sh
#!/bin/bash
# Rsync backup script with rotation
# Configuration
SOURCE_DIRS="/home /etc /var/www /var/lib/mysql"
BACKUP_DIR="/backup"
REMOTE_HOST="backup@192.168.1.200"
REMOTE_DIR="/volume1/backups/$(hostname)"
RETENTION_DAYS=30
LOG_FILE="/var/log/rsync-backup.log"
# Create backup directory
TODAY=$(date +%Y%m%d)
BACKUP_PATH="$BACKUP_DIR/$TODAY"
mkdir -p "$BACKUP_PATH"
# Function to log messages
log_message() {
echo "[$(date +'%Y-%m-%d %H:%M:%S')] $1" | tee -a "$LOG_FILE"
}
# Start backup
log_message "Starting backup for $TODAY"
# Backup each directory
for DIR in $SOURCE_DIRS; do
if [ -d "$DIR" ]; then
log_message "Backing up $DIR"
rsync -avzh --delete \
"$DIR" "$BACKUP_PATH/" \
2>&1 | tee -a "$LOG_FILE"
fi
done
# Sync to remote server
log_message "Syncing to remote server"
rsync -avzhe ssh --delete \
"$BACKUP_PATH/" \
"$REMOTE_HOST:$REMOTE_DIR/$TODAY/" \
2>&1 | tee -a "$LOG_FILE"
# Clean old backups
log_message "Cleaning old backups"
find "$BACKUP_DIR" -maxdepth 1 -type d -mtime +$RETENTION_DAYS -exec rm -rf {} \;
# Remote cleanup
ssh "$REMOTE_HOST" "find $REMOTE_DIR -maxdepth 1 -type d -mtime +$RETENTION_DAYS -exec rm -rf {} \;"
log_message "Backup completed"
# Make executable and schedule
sudo chmod +x /usr/local/bin/rsync-backup.sh
sudo crontab -e
# Add: 0 2 * * * /usr/local/bin/rsync-backup.sh
Database Backups
MySQL/MariaDB Backup
Automated MySQL Backup Script
sudo nano /usr/local/bin/mysql-backup.sh
#!/bin/bash
# MySQL backup script with compression and rotation
# Configuration
MYSQL_USER="backup"
MYSQL_PASS="BackupPass123!"
BACKUP_DIR="/backup/mysql"
RETENTION_DAYS=7
MYSQL_HOST="localhost"
LOG_FILE="/var/log/mysql-backup.log"
# Create backup directory
mkdir -p "$BACKUP_DIR"
DATE=$(date +%Y%m%d_%H%M%S)
# Function to log messages
log_message() {
echo "[$(date +'%Y-%m-%d %H:%M:%S')] $1" >> "$LOG_FILE"
}
# Get list of databases
DATABASES=$(mysql -u"$MYSQL_USER" -p"$MYSQL_PASS" -h"$MYSQL_HOST" -e "SHOW DATABASES;" | grep -Ev "(Database|information_schema|performance_schema|mysql|sys)")
# Backup all databases
log_message "Starting MySQL backup"
for DB in $DATABASES; do
log_message "Backing up database: $DB"
# Dump database with compression
mysqldump -u"$MYSQL_USER" -p"$MYSQL_PASS" -h"$MYSQL_HOST" \
--single-transaction \
--routines \
--triggers \
--events \
--add-drop-database \
--databases "$DB" | gzip > "$BACKUP_DIR/${DB}_${DATE}.sql.gz"
# Check if backup was successful
if [ $? -eq 0 ]; then
log_message "Successfully backed up $DB"
else
log_message "ERROR: Failed to backup $DB"
fi
done
# Backup all databases in one file
mysqldump -u"$MYSQL_USER" -p"$MYSQL_PASS" -h"$MYSQL_HOST" \
--all-databases \
--single-transaction \
--routines \
--triggers \
--events | gzip > "$BACKUP_DIR/all_databases_${DATE}.sql.gz"
# Clean old backups
find "$BACKUP_DIR" -name "*.sql.gz" -mtime +$RETENTION_DAYS -delete
log_message "MySQL backup completed"
MySQL Point-in-Time Recovery Setup
# Enable binary logging
sudo nano /etc/mysql/mysql.conf.d/mysqld.cnf
# Add:
log_bin = /var/log/mysql/mysql-bin
binlog_format = ROW
expire_logs_days = 7
max_binlog_size = 100M
# Restart MySQL
sudo systemctl restart mysql
PostgreSQL Backup
PostgreSQL Backup Script
sudo nano /usr/local/bin/postgres-backup.sh
#!/bin/bash
# PostgreSQL backup script
# Configuration
BACKUP_DIR="/backup/postgresql"
RETENTION_DAYS=7
PG_USER="postgres"
LOG_FILE="/var/log/postgres-backup.log"
# Create backup directory
mkdir -p "$BACKUP_DIR"
DATE=$(date +%Y%m%d_%H%M%S)
# Function to log messages
log_message() {
echo "[$(date +'%Y-%m-%d %H:%M:%S')] $1" >> "$LOG_FILE"
}
# Get list of databases
DATABASES=$(sudo -u $PG_USER psql -t -c "SELECT datname FROM pg_database WHERE datistemplate = false AND datname != 'postgres';")
log_message "Starting PostgreSQL backup"
# Backup each database
for DB in $DATABASES; do
DB=$(echo $DB | tr -d ' ')
log_message "Backing up database: $DB"
# Dump database
sudo -u $PG_USER pg_dump -Fc "$DB" > "$BACKUP_DIR/${DB}_${DATE}.dump"
if [ $? -eq 0 ]; then
log_message "Successfully backed up $DB"
else
log_message "ERROR: Failed to backup $DB"
fi
done
# Backup all databases and globals
sudo -u $PG_USER pg_dumpall | gzip > "$BACKUP_DIR/all_databases_${DATE}.sql.gz"
# Clean old backups
find "$BACKUP_DIR" -name "*.dump" -o -name "*.sql.gz" -mtime +$RETENTION_DAYS -delete
log_message "PostgreSQL backup completed"
Enterprise Backup Solutions
Bacula Installation and Configuration
Install Bacula
# Install Bacula components
sudo apt update
sudo apt install bacula-server bacula-client bacula-console-qt -y
# For PostgreSQL catalog
sudo apt install bacula-director-pgsql -y
Configure Bacula Director
sudo nano /etc/bacula/bacula-dir.conf
Director {
Name = bacula-dir
DIRport = 9101
QueryFile = "/etc/bacula/scripts/query.sql"
WorkingDirectory = "/var/lib/bacula"
PidDirectory = "/run/bacula"
Maximum Concurrent Jobs = 20
Password = "DirectorPassword"
Messages = Daemon
}
# Job Definitions
Job {
Name = "BackupServer"
Type = Backup
Level = Incremental
Client = server-fd
FileSet = "Full Set"
Schedule = "WeeklyCycle"
Storage = File
Messages = Standard
Pool = Default
Priority = 10
Write Bootstrap = "/var/lib/bacula/%c.bsr"
}
# FileSet Definition
FileSet {
Name = "Full Set"
Include {
Options {
signature = MD5
compression = GZIP
}
File = /home
File = /etc
File = /var/www
}
Exclude {
File = /proc
File = /tmp
File = /sys
File = /var/lib/bacula
}
}
# Schedule Definition
Schedule {
Name = "WeeklyCycle"
Run = Full 1st sun at 23:05
Run = Differential 2nd-5th sun at 23:05
Run = Incremental mon-sat at 23:05
}
# Storage Definition
Storage {
Name = File
Address = localhost
SDPort = 9103
Password = "StoragePassword"
Device = FileStorage
Media Type = File
}
# Pool Definition
Pool {
Name = Default
Pool Type = Backup
Recycle = yes
AutoPrune = yes
Volume Retention = 365 days
Maximum Volume Bytes = 50G
Maximum Volumes = 100
Label Format = "Vol-"
}
Configure Storage Daemon
sudo nano /etc/bacula/bacula-sd.conf
Storage {
Name = bacula-sd
SDPort = 9103
WorkingDirectory = "/var/lib/bacula"
Pid Directory = "/run/bacula"
Maximum Concurrent Jobs = 20
}
Director {
Name = bacula-dir
Password = "StoragePassword"
}
Device {
Name = FileStorage
Media Type = File
Archive Device = /backup/bacula
LabelMedia = yes
Random Access = yes
AutomaticMount = yes
RemovableMedia = no
AlwaysOpen = no
}
Restic Modern Backup Solution
Install Restic
# Install from repository
sudo apt install restic -y
# Or download latest version
wget https://github.com/restic/restic/releases/download/v0.16.2/restic_0.16.2_linux_amd64.bz2
bunzip2 restic_0.16.2_linux_amd64.bz2
sudo mv restic_0.16.2_linux_amd64 /usr/local/bin/restic
sudo chmod +x /usr/local/bin/restic
Initialize Repository
# Local repository
restic init --repo /backup/restic
# S3 repository
export AWS_ACCESS_KEY_ID="your-key-id"
export AWS_SECRET_ACCESS_KEY="your-secret-key"
restic init --repo s3:s3.amazonaws.com/bucket-name
# SFTP repository
restic init --repo sftp:user@host:/backup/restic
Restic Backup Script
sudo nano /usr/local/bin/restic-backup.sh
#!/bin/bash
# Restic backup script
# Configuration
export RESTIC_REPOSITORY="/backup/restic"
export RESTIC_PASSWORD="YourResticPassword"
BACKUP_PATHS="/home /etc /var/www"
EXCLUDE_FILE="/etc/restic/excludes.txt"
LOG_FILE="/var/log/restic-backup.log"
RETENTION_DAYS=30
RETENTION_WEEKS=12
RETENTION_MONTHS=12
RETENTION_YEARS=3
# Function to log messages
log_message() {
echo "[$(date +'%Y-%m-%d %H:%M:%S')] $1" | tee -a "$LOG_FILE"
}
# Start backup
log_message "Starting Restic backup"
# Create exclude file if not exists
if [ ! -f "$EXCLUDE_FILE" ]; then
cat > "$EXCLUDE_FILE" << EOF
/proc
/sys
/dev
/run
/tmp
/var/tmp
/var/cache
*.tmp
*.temp
*.swp
.cache/
EOF
fi
# Run backup
restic backup $BACKUP_PATHS \
--exclude-file="$EXCLUDE_FILE" \
--tag="$(date +%Y-%m-%d)" \
--verbose \
2>&1 | tee -a "$LOG_FILE"
# Check backup integrity
log_message "Checking backup integrity"
restic check 2>&1 | tee -a "$LOG_FILE"
# Prune old snapshots
log_message "Pruning old snapshots"
restic forget \
--keep-daily $RETENTION_DAYS \
--keep-weekly $RETENTION_WEEKS \
--keep-monthly $RETENTION_MONTHS \
--keep-yearly $RETENTION_YEARS \
--prune \
2>&1 | tee -a "$LOG_FILE"
log_message "Restic backup completed"
Disaster Recovery Procedures
System Recovery Plan
Create Recovery Documentation
sudo mkdir -p /root/disaster-recovery
sudo nano /root/disaster-recovery/recovery-plan.md
# Disaster Recovery Plan
## Emergency Contacts
- System Administrator: Name (phone)
- Network Administrator: Name (phone)
- Management: Name (phone)
## Priority Systems
1. Database servers
2. Web servers
3. Application servers
4. File servers
## Recovery Time Objectives
- Critical systems: 4 hours
- Important systems: 8 hours
- Standard systems: 24 hours
## Recovery Procedures
1. Assess damage
2. Activate DR site (if applicable)
3. Restore from backups
4. Verify functionality
5. Redirect traffic
6. Document incident
Bare Metal Recovery
Create System Recovery Image
sudo nano /usr/local/bin/create-recovery-image.sh
#!/bin/bash
# Create bootable system recovery image
# Configuration
OUTPUT_DIR="/backup/recovery"
ISO_NAME="ubuntu-recovery-$(date +%Y%m%d).iso"
# Install required packages
sudo apt install squashfs-tools genisoimage -y
# Create working directory
WORK_DIR=$(mktemp -d)
cd "$WORK_DIR"
# Copy system files
sudo rsync -av --one-file-system \
--exclude=/proc \
--exclude=/sys \
--exclude=/dev \
--exclude=/run \
--exclude=/tmp \
--exclude=/mnt \
--exclude=/media \
--exclude=/backup \
/ "$WORK_DIR/filesystem/"
# Create squashfs
sudo mksquashfs filesystem filesystem.squashfs -comp xz
# Create ISO
# This is a simplified example - full bootable ISO creation requires more steps
mkdir -p iso/casper
mv filesystem.squashfs iso/casper/
# Create recovery scripts
cat > iso/recovery.sh << 'EOF'
#!/bin/bash
echo "Ubuntu Server Recovery Tool"
echo "1. Restore full system"
echo "2. Restore specific files"
echo "3. Repair boot loader"
echo "4. Check file system"
read -p "Select option: " option
case $option in
1) echo "Restoring full system..."
# Add restoration commands
;;
2) echo "Restore specific files..."
# Add file restoration
;;
3) echo "Repairing boot loader..."
# Add grub repair
;;
4) echo "Checking file system..."
fsck /dev/sda1
;;
esac
EOF
chmod +x iso/recovery.sh
# Clean up
cd /
rm -rf "$WORK_DIR"
echo "Recovery image created: $OUTPUT_DIR/$ISO_NAME"
Automated Recovery Testing
Recovery Test Script
sudo nano /usr/local/bin/test-recovery.sh
#!/bin/bash
# Automated recovery testing script
# Configuration
TEST_DIR="/tmp/recovery-test"
BACKUP_SOURCE="/backup/latest"
LOG_FILE="/var/log/recovery-test.log"
# Function to log messages
log_message() {
echo "[$(date +'%Y-%m-%d %H:%M:%S')] $1" >> "$LOG_FILE"
}
# Test file restoration
test_file_restore() {
log_message "Testing file restoration"
mkdir -p "$TEST_DIR"
# Restore sample files
tar -xzf "$BACKUP_SOURCE/home.tar.gz" -C "$TEST_DIR" home/testuser/.bashrc
if [ -f "$TEST_DIR/home/testuser/.bashrc" ]; then
log_message "File restoration: SUCCESS"
return 0
else
log_message "File restoration: FAILED"
return 1
fi
}
# Test database restoration
test_database_restore() {
log_message "Testing database restoration"
# Create test database
mysql -u root -e "CREATE DATABASE IF NOT EXISTS recovery_test;"
# Restore database
gunzip < "$BACKUP_SOURCE/mysql/testdb_latest.sql.gz" | mysql -u root recovery_test
# Verify restoration
TABLES=$(mysql -u root recovery_test -e "SHOW TABLES;" | wc -l)
if [ $TABLES -gt 1 ]; then
log_message "Database restoration: SUCCESS"
mysql -u root -e "DROP DATABASE recovery_test;"
return 0
else
log_message "Database restoration: FAILED"
return 1
fi
}
# Test application restoration
test_application_restore() {
log_message "Testing application restoration"
# Restore application files
mkdir -p "$TEST_DIR/www"
rsync -av "$BACKUP_SOURCE/www/" "$TEST_DIR/www/"
# Verify critical files
if [ -f "$TEST_DIR/www/index.html" ]; then
log_message "Application restoration: SUCCESS"
return 0
else
log_message "Application restoration: FAILED"
return 1
fi
}
# Run tests
log_message "Starting recovery tests"
test_file_restore
FILE_RESULT=$?
test_database_restore
DB_RESULT=$?
test_application_restore
APP_RESULT=$?
# Clean up
rm -rf "$TEST_DIR"
# Report results
if [ $FILE_RESULT -eq 0 ] && [ $DB_RESULT -eq 0 ] && [ $APP_RESULT -eq 0 ]; then
log_message "All recovery tests passed"
exit 0
else
log_message "Some recovery tests failed"
exit 1
fi
Cloud Backup Integration
AWS S3 Backup
Install AWS CLI
# Install AWS CLI
sudo apt install awscli -y
# Configure AWS credentials
aws configure
S3 Backup Script
sudo nano /usr/local/bin/s3-backup.sh
#!/bin/bash
# S3 backup script with lifecycle management
# Configuration
LOCAL_BACKUP_DIR="/backup"
S3_BUCKET="s3://company-backups"
S3_PATH="$(hostname)/$(date +%Y/%m/%d)"
LOG_FILE="/var/log/s3-backup.log"
BANDWIDTH_LIMIT="10M" # Limit upload bandwidth
# Function to log messages
log_message() {
echo "[$(date +'%Y-%m-%d %H:%M:%S')] $1" >> "$LOG_FILE"
}
# Sync to S3
log_message "Starting S3 sync"
aws s3 sync "$LOCAL_BACKUP_DIR" "$S3_BUCKET/$S3_PATH" \
--exclude "*.tmp" \
--exclude "*.log" \
--storage-class STANDARD_IA \
--bandwidth-limit="$BANDWIDTH_LIMIT" \
2>&1 | tee -a "$LOG_FILE"
# Verify sync
aws s3 ls "$S3_BUCKET/$S3_PATH" --recursive --summarize | tail -2 >> "$LOG_FILE"
log_message "S3 sync completed"
# Apply lifecycle policy for cost optimization
cat > /tmp/lifecycle-policy.json << EOF
{
"Rules": [
{
"ID": "Archive old backups",
"Status": "Enabled",
"Transitions": [
{
"Days": 30,
"StorageClass": "GLACIER"
},
{
"Days": 90,
"StorageClass": "DEEP_ARCHIVE"
}
],
"Expiration": {
"Days": 365
}
}
]
}
EOF
aws s3api put-bucket-lifecycle-configuration \
--bucket company-backups \
--lifecycle-configuration file:///tmp/lifecycle-policy.json
Backup Monitoring and Alerting
Backup Monitor Script
sudo nano /usr/local/bin/backup-monitor.sh
#!/bin/bash
# Monitor backup status and send alerts
# Configuration
BACKUP_DIRS=("/backup/daily" "/backup/mysql" "/backup/postgresql")
MAX_AGE_HOURS=26 # Alert if backup older than 26 hours
MIN_SIZE_MB=100 # Alert if backup smaller than 100MB
EMAIL="admin@example.com"
SLACK_WEBHOOK="https://hooks.slack.com/services/YOUR/WEBHOOK/URL"
# Function to send alert
send_alert() {
local subject="$1"
local message="$2"
# Send email
echo "$message" | mail -s "$subject" "$EMAIL"
# Send to Slack
curl -X POST -H 'Content-type: application/json' \
--data "{\"text\":\"$subject\n$message\"}" \
"$SLACK_WEBHOOK"
}
# Check each backup directory
for dir in "${BACKUP_DIRS[@]}"; do
if [ ! -d "$dir" ]; then
send_alert "Backup Directory Missing" "Directory $dir does not exist on $(hostname)"
continue
fi
# Find latest backup
latest_backup=$(find "$dir" -type f -name "*.gz" -o -name "*.tar" | sort -r | head -1)
if [ -z "$latest_backup" ]; then
send_alert "No Backups Found" "No backup files found in $dir on $(hostname)"
continue
fi
# Check age
file_age=$((($(date +%s) - $(stat -c %Y "$latest_backup")) / 3600))
if [ $file_age -gt $MAX_AGE_HOURS ]; then
send_alert "Backup Too Old" "Latest backup in $dir is $file_age hours old on $(hostname)"
fi
# Check size
file_size=$(($(stat -c %s "$latest_backup") / 1024 / 1024))
if [ $file_size -lt $MIN_SIZE_MB ]; then
send_alert "Backup Too Small" "Backup $latest_backup is only ${file_size}MB on $(hostname)"
fi
done
# Check backup processes
for process in "rsync" "mysqldump" "pg_dump"; do
if pgrep -x "$process" > /dev/null; then
runtime=$(ps -o etime= -p $(pgrep -x "$process" | head -1) | tr -d ' ')
send_alert "Long Running Backup" "$process has been running for $runtime on $(hostname)"
fi
done
Best Practices
- Test Restores: Regularly test backup restoration
- Documentation: Keep detailed recovery procedures
- Encryption: Encrypt sensitive backups
- Versioning: Maintain multiple backup versions
- Monitoring: Alert on backup failures
- Offsite Storage: Keep copies in different locations
- Automation: Automate backup processes
Conclusion
This comprehensive guide provides robust backup and disaster recovery strategies for Ubuntu Server 22.04. From simple scripts to enterprise solutions, implementing these practices ensures business continuity and data protection. Remember that a backup strategy is only as good as its last successful restore test.