Complete Backup and Disaster Recovery Guide for Ubuntu Server 22.04

Tyler Maginnis | January 20, 2024

UbuntuBackupDisaster RecoveryRsyncBaculaBusiness Continuity

Need Professional Ubuntu Server Support?

Get expert assistance with your ubuntu server support implementation and management. Tyler on Tech Louisville provides priority support for Louisville businesses.

Same-day service available for Louisville area

Complete Backup and Disaster Recovery Guide for Ubuntu Server 22.04

A comprehensive backup and disaster recovery strategy is essential for business continuity. This guide covers various backup solutions, from simple scripts to enterprise-grade systems, ensuring your Ubuntu Server data remains safe and recoverable.

Prerequisites

  • Ubuntu Server 22.04 LTS
  • Root or sudo access
  • Sufficient storage for backups
  • Basic understanding of Linux file systems

Backup Strategy Planning

Types of Backups

  1. Full Backup: Complete copy of all data
  2. Incremental Backup: Only changes since last backup
  3. Differential Backup: Changes since last full backup
  4. Snapshot Backup: Point-in-time copy of system state

3-2-1 Rule

  • 3 copies of important data
  • 2 different storage media types
  • 1 offsite backup

Built-in Backup Tools

Using tar for Archive Backups

Basic tar Backup

# Full system backup (excluding certain directories)
sudo tar -czpf /backup/system-$(date +%Y%m%d).tar.gz \
  --exclude=/backup \
  --exclude=/proc \
  --exclude=/tmp \
  --exclude=/mnt \
  --exclude=/dev \
  --exclude=/sys \
  --exclude=/run \
  --exclude=/media \
  --exclude=/var/log \
  --exclude=/var/cache \
  --exclude=/home/*/.cache \
  --exclude=/var/lib/lxcfs \
  --one-file-system /

Incremental tar Backup

# Create snapshot file for incremental backups
sudo tar -czpf /backup/full-backup.tar.gz \
  --listed-incremental=/backup/snapshot.snar \
  /home /etc /var/www

# Subsequent incremental backups
sudo tar -czpf /backup/incremental-$(date +%Y%m%d).tar.gz \
  --listed-incremental=/backup/snapshot.snar \
  /home /etc /var/www

Restore from tar

# Full restore
sudo tar -xzpf /backup/system-20240120.tar.gz -C /

# Restore specific directory
sudo tar -xzpf /backup/system-20240120.tar.gz -C / home/user/documents

# List contents without extracting
tar -tzf /backup/system-20240120.tar.gz

Rsync for Efficient Backups

Local Rsync Backup

# Basic rsync backup
sudo rsync -avzh --progress \
  --exclude={"/dev/*","/proc/*","/sys/*","/tmp/*","/run/*","/mnt/*","/media/*","/lost+found"} \
  / /backup/system/

# With deletion of removed files
sudo rsync -avzh --delete --progress \
  /home/ /backup/home/

Remote Rsync Backup

# Backup to remote server
rsync -avzhe ssh --progress \
  /home/user/documents/ \
  user@backup-server:/backups/documents/

# Pull backup from remote
rsync -avzhe ssh --progress \
  user@remote-server:/var/www/ \
  /backup/www/

Rsync Backup Script

sudo nano /usr/local/bin/rsync-backup.sh
#!/bin/bash
# Rsync backup script with rotation

# Configuration
SOURCE_DIRS="/home /etc /var/www /var/lib/mysql"
BACKUP_DIR="/backup"
REMOTE_HOST="backup@192.168.1.200"
REMOTE_DIR="/volume1/backups/$(hostname)"
RETENTION_DAYS=30
LOG_FILE="/var/log/rsync-backup.log"

# Create backup directory
TODAY=$(date +%Y%m%d)
BACKUP_PATH="$BACKUP_DIR/$TODAY"
mkdir -p "$BACKUP_PATH"

# Function to log messages
log_message() {
    echo "[$(date +'%Y-%m-%d %H:%M:%S')] $1" | tee -a "$LOG_FILE"
}

# Start backup
log_message "Starting backup for $TODAY"

# Backup each directory
for DIR in $SOURCE_DIRS; do
    if [ -d "$DIR" ]; then
        log_message "Backing up $DIR"
        rsync -avzh --delete \
          "$DIR" "$BACKUP_PATH/" \
          2>&1 | tee -a "$LOG_FILE"
    fi
done

# Sync to remote server
log_message "Syncing to remote server"
rsync -avzhe ssh --delete \
  "$BACKUP_PATH/" \
  "$REMOTE_HOST:$REMOTE_DIR/$TODAY/" \
  2>&1 | tee -a "$LOG_FILE"

# Clean old backups
log_message "Cleaning old backups"
find "$BACKUP_DIR" -maxdepth 1 -type d -mtime +$RETENTION_DAYS -exec rm -rf {} \;

# Remote cleanup
ssh "$REMOTE_HOST" "find $REMOTE_DIR -maxdepth 1 -type d -mtime +$RETENTION_DAYS -exec rm -rf {} \;"

log_message "Backup completed"
# Make executable and schedule
sudo chmod +x /usr/local/bin/rsync-backup.sh
sudo crontab -e
# Add: 0 2 * * * /usr/local/bin/rsync-backup.sh

Database Backups

MySQL/MariaDB Backup

Automated MySQL Backup Script

sudo nano /usr/local/bin/mysql-backup.sh
#!/bin/bash
# MySQL backup script with compression and rotation

# Configuration
MYSQL_USER="backup"
MYSQL_PASS="BackupPass123!"
BACKUP_DIR="/backup/mysql"
RETENTION_DAYS=7
MYSQL_HOST="localhost"
LOG_FILE="/var/log/mysql-backup.log"

# Create backup directory
mkdir -p "$BACKUP_DIR"
DATE=$(date +%Y%m%d_%H%M%S)

# Function to log messages
log_message() {
    echo "[$(date +'%Y-%m-%d %H:%M:%S')] $1" >> "$LOG_FILE"
}

# Get list of databases
DATABASES=$(mysql -u"$MYSQL_USER" -p"$MYSQL_PASS" -h"$MYSQL_HOST" -e "SHOW DATABASES;" | grep -Ev "(Database|information_schema|performance_schema|mysql|sys)")

# Backup all databases
log_message "Starting MySQL backup"

for DB in $DATABASES; do
    log_message "Backing up database: $DB"

    # Dump database with compression
    mysqldump -u"$MYSQL_USER" -p"$MYSQL_PASS" -h"$MYSQL_HOST" \
        --single-transaction \
        --routines \
        --triggers \
        --events \
        --add-drop-database \
        --databases "$DB" | gzip > "$BACKUP_DIR/${DB}_${DATE}.sql.gz"

    # Check if backup was successful
    if [ $? -eq 0 ]; then
        log_message "Successfully backed up $DB"
    else
        log_message "ERROR: Failed to backup $DB"
    fi
done

# Backup all databases in one file
mysqldump -u"$MYSQL_USER" -p"$MYSQL_PASS" -h"$MYSQL_HOST" \
    --all-databases \
    --single-transaction \
    --routines \
    --triggers \
    --events | gzip > "$BACKUP_DIR/all_databases_${DATE}.sql.gz"

# Clean old backups
find "$BACKUP_DIR" -name "*.sql.gz" -mtime +$RETENTION_DAYS -delete

log_message "MySQL backup completed"

MySQL Point-in-Time Recovery Setup

# Enable binary logging
sudo nano /etc/mysql/mysql.conf.d/mysqld.cnf

# Add:
log_bin = /var/log/mysql/mysql-bin
binlog_format = ROW
expire_logs_days = 7
max_binlog_size = 100M

# Restart MySQL
sudo systemctl restart mysql

PostgreSQL Backup

PostgreSQL Backup Script

sudo nano /usr/local/bin/postgres-backup.sh
#!/bin/bash
# PostgreSQL backup script

# Configuration
BACKUP_DIR="/backup/postgresql"
RETENTION_DAYS=7
PG_USER="postgres"
LOG_FILE="/var/log/postgres-backup.log"

# Create backup directory
mkdir -p "$BACKUP_DIR"
DATE=$(date +%Y%m%d_%H%M%S)

# Function to log messages
log_message() {
    echo "[$(date +'%Y-%m-%d %H:%M:%S')] $1" >> "$LOG_FILE"
}

# Get list of databases
DATABASES=$(sudo -u $PG_USER psql -t -c "SELECT datname FROM pg_database WHERE datistemplate = false AND datname != 'postgres';")

log_message "Starting PostgreSQL backup"

# Backup each database
for DB in $DATABASES; do
    DB=$(echo $DB | tr -d ' ')
    log_message "Backing up database: $DB"

    # Dump database
    sudo -u $PG_USER pg_dump -Fc "$DB" > "$BACKUP_DIR/${DB}_${DATE}.dump"

    if [ $? -eq 0 ]; then
        log_message "Successfully backed up $DB"
    else
        log_message "ERROR: Failed to backup $DB"
    fi
done

# Backup all databases and globals
sudo -u $PG_USER pg_dumpall | gzip > "$BACKUP_DIR/all_databases_${DATE}.sql.gz"

# Clean old backups
find "$BACKUP_DIR" -name "*.dump" -o -name "*.sql.gz" -mtime +$RETENTION_DAYS -delete

log_message "PostgreSQL backup completed"

Enterprise Backup Solutions

Bacula Installation and Configuration

Install Bacula

# Install Bacula components
sudo apt update
sudo apt install bacula-server bacula-client bacula-console-qt -y

# For PostgreSQL catalog
sudo apt install bacula-director-pgsql -y

Configure Bacula Director

sudo nano /etc/bacula/bacula-dir.conf
Director {
  Name = bacula-dir
  DIRport = 9101
  QueryFile = "/etc/bacula/scripts/query.sql"
  WorkingDirectory = "/var/lib/bacula"
  PidDirectory = "/run/bacula"
  Maximum Concurrent Jobs = 20
  Password = "DirectorPassword"
  Messages = Daemon
}

# Job Definitions
Job {
  Name = "BackupServer"
  Type = Backup
  Level = Incremental
  Client = server-fd
  FileSet = "Full Set"
  Schedule = "WeeklyCycle"
  Storage = File
  Messages = Standard
  Pool = Default
  Priority = 10
  Write Bootstrap = "/var/lib/bacula/%c.bsr"
}

# FileSet Definition
FileSet {
  Name = "Full Set"
  Include {
    Options {
      signature = MD5
      compression = GZIP
    }
    File = /home
    File = /etc
    File = /var/www
  }
  Exclude {
    File = /proc
    File = /tmp
    File = /sys
    File = /var/lib/bacula
  }
}

# Schedule Definition
Schedule {
  Name = "WeeklyCycle"
  Run = Full 1st sun at 23:05
  Run = Differential 2nd-5th sun at 23:05
  Run = Incremental mon-sat at 23:05
}

# Storage Definition
Storage {
  Name = File
  Address = localhost
  SDPort = 9103
  Password = "StoragePassword"
  Device = FileStorage
  Media Type = File
}

# Pool Definition
Pool {
  Name = Default
  Pool Type = Backup
  Recycle = yes
  AutoPrune = yes
  Volume Retention = 365 days
  Maximum Volume Bytes = 50G
  Maximum Volumes = 100
  Label Format = "Vol-"
}

Configure Storage Daemon

sudo nano /etc/bacula/bacula-sd.conf
Storage {
  Name = bacula-sd
  SDPort = 9103
  WorkingDirectory = "/var/lib/bacula"
  Pid Directory = "/run/bacula"
  Maximum Concurrent Jobs = 20
}

Director {
  Name = bacula-dir
  Password = "StoragePassword"
}

Device {
  Name = FileStorage
  Media Type = File
  Archive Device = /backup/bacula
  LabelMedia = yes
  Random Access = yes
  AutomaticMount = yes
  RemovableMedia = no
  AlwaysOpen = no
}

Restic Modern Backup Solution

Install Restic

# Install from repository
sudo apt install restic -y

# Or download latest version
wget https://github.com/restic/restic/releases/download/v0.16.2/restic_0.16.2_linux_amd64.bz2
bunzip2 restic_0.16.2_linux_amd64.bz2
sudo mv restic_0.16.2_linux_amd64 /usr/local/bin/restic
sudo chmod +x /usr/local/bin/restic

Initialize Repository

# Local repository
restic init --repo /backup/restic

# S3 repository
export AWS_ACCESS_KEY_ID="your-key-id"
export AWS_SECRET_ACCESS_KEY="your-secret-key"
restic init --repo s3:s3.amazonaws.com/bucket-name

# SFTP repository
restic init --repo sftp:user@host:/backup/restic

Restic Backup Script

sudo nano /usr/local/bin/restic-backup.sh
#!/bin/bash
# Restic backup script

# Configuration
export RESTIC_REPOSITORY="/backup/restic"
export RESTIC_PASSWORD="YourResticPassword"
BACKUP_PATHS="/home /etc /var/www"
EXCLUDE_FILE="/etc/restic/excludes.txt"
LOG_FILE="/var/log/restic-backup.log"
RETENTION_DAYS=30
RETENTION_WEEKS=12
RETENTION_MONTHS=12
RETENTION_YEARS=3

# Function to log messages
log_message() {
    echo "[$(date +'%Y-%m-%d %H:%M:%S')] $1" | tee -a "$LOG_FILE"
}

# Start backup
log_message "Starting Restic backup"

# Create exclude file if not exists
if [ ! -f "$EXCLUDE_FILE" ]; then
    cat > "$EXCLUDE_FILE" << EOF
/proc
/sys
/dev
/run
/tmp
/var/tmp
/var/cache
*.tmp
*.temp
*.swp
.cache/
EOF
fi

# Run backup
restic backup $BACKUP_PATHS \
    --exclude-file="$EXCLUDE_FILE" \
    --tag="$(date +%Y-%m-%d)" \
    --verbose \
    2>&1 | tee -a "$LOG_FILE"

# Check backup integrity
log_message "Checking backup integrity"
restic check 2>&1 | tee -a "$LOG_FILE"

# Prune old snapshots
log_message "Pruning old snapshots"
restic forget \
    --keep-daily $RETENTION_DAYS \
    --keep-weekly $RETENTION_WEEKS \
    --keep-monthly $RETENTION_MONTHS \
    --keep-yearly $RETENTION_YEARS \
    --prune \
    2>&1 | tee -a "$LOG_FILE"

log_message "Restic backup completed"

Disaster Recovery Procedures

System Recovery Plan

Create Recovery Documentation

sudo mkdir -p /root/disaster-recovery
sudo nano /root/disaster-recovery/recovery-plan.md
# Disaster Recovery Plan

## Emergency Contacts
- System Administrator: Name (phone)
- Network Administrator: Name (phone)
- Management: Name (phone)

## Priority Systems
1. Database servers
2. Web servers
3. Application servers
4. File servers

## Recovery Time Objectives
- Critical systems: 4 hours
- Important systems: 8 hours
- Standard systems: 24 hours

## Recovery Procedures
1. Assess damage
2. Activate DR site (if applicable)
3. Restore from backups
4. Verify functionality
5. Redirect traffic
6. Document incident

Bare Metal Recovery

Create System Recovery Image

sudo nano /usr/local/bin/create-recovery-image.sh
#!/bin/bash
# Create bootable system recovery image

# Configuration
OUTPUT_DIR="/backup/recovery"
ISO_NAME="ubuntu-recovery-$(date +%Y%m%d).iso"

# Install required packages
sudo apt install squashfs-tools genisoimage -y

# Create working directory
WORK_DIR=$(mktemp -d)
cd "$WORK_DIR"

# Copy system files
sudo rsync -av --one-file-system \
    --exclude=/proc \
    --exclude=/sys \
    --exclude=/dev \
    --exclude=/run \
    --exclude=/tmp \
    --exclude=/mnt \
    --exclude=/media \
    --exclude=/backup \
    / "$WORK_DIR/filesystem/"

# Create squashfs
sudo mksquashfs filesystem filesystem.squashfs -comp xz

# Create ISO
# This is a simplified example - full bootable ISO creation requires more steps
mkdir -p iso/casper
mv filesystem.squashfs iso/casper/

# Create recovery scripts
cat > iso/recovery.sh << 'EOF'
#!/bin/bash
echo "Ubuntu Server Recovery Tool"
echo "1. Restore full system"
echo "2. Restore specific files"
echo "3. Repair boot loader"
echo "4. Check file system"
read -p "Select option: " option

case $option in
    1) echo "Restoring full system..."
       # Add restoration commands
       ;;
    2) echo "Restore specific files..."
       # Add file restoration
       ;;
    3) echo "Repairing boot loader..."
       # Add grub repair
       ;;
    4) echo "Checking file system..."
       fsck /dev/sda1
       ;;
esac
EOF

chmod +x iso/recovery.sh

# Clean up
cd /
rm -rf "$WORK_DIR"

echo "Recovery image created: $OUTPUT_DIR/$ISO_NAME"

Automated Recovery Testing

Recovery Test Script

sudo nano /usr/local/bin/test-recovery.sh
#!/bin/bash
# Automated recovery testing script

# Configuration
TEST_DIR="/tmp/recovery-test"
BACKUP_SOURCE="/backup/latest"
LOG_FILE="/var/log/recovery-test.log"

# Function to log messages
log_message() {
    echo "[$(date +'%Y-%m-%d %H:%M:%S')] $1" >> "$LOG_FILE"
}

# Test file restoration
test_file_restore() {
    log_message "Testing file restoration"
    mkdir -p "$TEST_DIR"

    # Restore sample files
    tar -xzf "$BACKUP_SOURCE/home.tar.gz" -C "$TEST_DIR" home/testuser/.bashrc

    if [ -f "$TEST_DIR/home/testuser/.bashrc" ]; then
        log_message "File restoration: SUCCESS"
        return 0
    else
        log_message "File restoration: FAILED"
        return 1
    fi
}

# Test database restoration
test_database_restore() {
    log_message "Testing database restoration"

    # Create test database
    mysql -u root -e "CREATE DATABASE IF NOT EXISTS recovery_test;"

    # Restore database
    gunzip < "$BACKUP_SOURCE/mysql/testdb_latest.sql.gz" | mysql -u root recovery_test

    # Verify restoration
    TABLES=$(mysql -u root recovery_test -e "SHOW TABLES;" | wc -l)

    if [ $TABLES -gt 1 ]; then
        log_message "Database restoration: SUCCESS"
        mysql -u root -e "DROP DATABASE recovery_test;"
        return 0
    else
        log_message "Database restoration: FAILED"
        return 1
    fi
}

# Test application restoration
test_application_restore() {
    log_message "Testing application restoration"

    # Restore application files
    mkdir -p "$TEST_DIR/www"
    rsync -av "$BACKUP_SOURCE/www/" "$TEST_DIR/www/"

    # Verify critical files
    if [ -f "$TEST_DIR/www/index.html" ]; then
        log_message "Application restoration: SUCCESS"
        return 0
    else
        log_message "Application restoration: FAILED"
        return 1
    fi
}

# Run tests
log_message "Starting recovery tests"

test_file_restore
FILE_RESULT=$?

test_database_restore
DB_RESULT=$?

test_application_restore
APP_RESULT=$?

# Clean up
rm -rf "$TEST_DIR"

# Report results
if [ $FILE_RESULT -eq 0 ] && [ $DB_RESULT -eq 0 ] && [ $APP_RESULT -eq 0 ]; then
    log_message "All recovery tests passed"
    exit 0
else
    log_message "Some recovery tests failed"
    exit 1
fi

Cloud Backup Integration

AWS S3 Backup

Install AWS CLI

# Install AWS CLI
sudo apt install awscli -y

# Configure AWS credentials
aws configure

S3 Backup Script

sudo nano /usr/local/bin/s3-backup.sh
#!/bin/bash
# S3 backup script with lifecycle management

# Configuration
LOCAL_BACKUP_DIR="/backup"
S3_BUCKET="s3://company-backups"
S3_PATH="$(hostname)/$(date +%Y/%m/%d)"
LOG_FILE="/var/log/s3-backup.log"
BANDWIDTH_LIMIT="10M"  # Limit upload bandwidth

# Function to log messages
log_message() {
    echo "[$(date +'%Y-%m-%d %H:%M:%S')] $1" >> "$LOG_FILE"
}

# Sync to S3
log_message "Starting S3 sync"

aws s3 sync "$LOCAL_BACKUP_DIR" "$S3_BUCKET/$S3_PATH" \
    --exclude "*.tmp" \
    --exclude "*.log" \
    --storage-class STANDARD_IA \
    --bandwidth-limit="$BANDWIDTH_LIMIT" \
    2>&1 | tee -a "$LOG_FILE"

# Verify sync
aws s3 ls "$S3_BUCKET/$S3_PATH" --recursive --summarize | tail -2 >> "$LOG_FILE"

log_message "S3 sync completed"

# Apply lifecycle policy for cost optimization
cat > /tmp/lifecycle-policy.json << EOF
{
    "Rules": [
        {
            "ID": "Archive old backups",
            "Status": "Enabled",
            "Transitions": [
                {
                    "Days": 30,
                    "StorageClass": "GLACIER"
                },
                {
                    "Days": 90,
                    "StorageClass": "DEEP_ARCHIVE"
                }
            ],
            "Expiration": {
                "Days": 365
            }
        }
    ]
}
EOF

aws s3api put-bucket-lifecycle-configuration \
    --bucket company-backups \
    --lifecycle-configuration file:///tmp/lifecycle-policy.json

Backup Monitoring and Alerting

Backup Monitor Script

sudo nano /usr/local/bin/backup-monitor.sh
#!/bin/bash
# Monitor backup status and send alerts

# Configuration
BACKUP_DIRS=("/backup/daily" "/backup/mysql" "/backup/postgresql")
MAX_AGE_HOURS=26  # Alert if backup older than 26 hours
MIN_SIZE_MB=100   # Alert if backup smaller than 100MB
EMAIL="admin@example.com"
SLACK_WEBHOOK="https://hooks.slack.com/services/YOUR/WEBHOOK/URL"

# Function to send alert
send_alert() {
    local subject="$1"
    local message="$2"

    # Send email
    echo "$message" | mail -s "$subject" "$EMAIL"

    # Send to Slack
    curl -X POST -H 'Content-type: application/json' \
        --data "{\"text\":\"$subject\n$message\"}" \
        "$SLACK_WEBHOOK"
}

# Check each backup directory
for dir in "${BACKUP_DIRS[@]}"; do
    if [ ! -d "$dir" ]; then
        send_alert "Backup Directory Missing" "Directory $dir does not exist on $(hostname)"
        continue
    fi

    # Find latest backup
    latest_backup=$(find "$dir" -type f -name "*.gz" -o -name "*.tar" | sort -r | head -1)

    if [ -z "$latest_backup" ]; then
        send_alert "No Backups Found" "No backup files found in $dir on $(hostname)"
        continue
    fi

    # Check age
    file_age=$((($(date +%s) - $(stat -c %Y "$latest_backup")) / 3600))
    if [ $file_age -gt $MAX_AGE_HOURS ]; then
        send_alert "Backup Too Old" "Latest backup in $dir is $file_age hours old on $(hostname)"
    fi

    # Check size
    file_size=$(($(stat -c %s "$latest_backup") / 1024 / 1024))
    if [ $file_size -lt $MIN_SIZE_MB ]; then
        send_alert "Backup Too Small" "Backup $latest_backup is only ${file_size}MB on $(hostname)"
    fi
done

# Check backup processes
for process in "rsync" "mysqldump" "pg_dump"; do
    if pgrep -x "$process" > /dev/null; then
        runtime=$(ps -o etime= -p $(pgrep -x "$process" | head -1) | tr -d ' ')
        send_alert "Long Running Backup" "$process has been running for $runtime on $(hostname)"
    fi
done

Best Practices

  1. Test Restores: Regularly test backup restoration
  2. Documentation: Keep detailed recovery procedures
  3. Encryption: Encrypt sensitive backups
  4. Versioning: Maintain multiple backup versions
  5. Monitoring: Alert on backup failures
  6. Offsite Storage: Keep copies in different locations
  7. Automation: Automate backup processes

Conclusion

This comprehensive guide provides robust backup and disaster recovery strategies for Ubuntu Server 22.04. From simple scripts to enterprise solutions, implementing these practices ensures business continuity and data protection. Remember that a backup strategy is only as good as its last successful restore test.