diff --git a/example_usage.md b/example_usage.md new file mode 100644 index 0000000..b96c5f0 --- /dev/null +++ b/example_usage.md @@ -0,0 +1,110 @@ +# Docker2PBS - Usage Examples + +## Installation + +```bash +pip install -r requirements.txt +chmod +x docker2pbs.py +``` + +## Basic Usage + +```bash +./docker2pbs.py docker-compose.yml web --pbs-repository user@pbs.example.com:backup-datastore +``` + +## Dry-run Mode (Testing) + +Use `--dry-run` to see what would be executed without actually performing operations: + +```bash +./docker2pbs.py docker-compose.yml web \ + --pbs-repository user@pbs.example.com:backup-datastore \ + --dry-run +``` + +## Usage with Authentication + +```bash +./docker2pbs.py docker-compose.yml database \ + --pbs-repository user@pbs.example.com:backup-datastore \ + --pbs-username backup-user \ + --pbs-password secret123 \ + --pbs-fingerprint aa:bb:cc:dd:ee:ff... +``` + +## Dry-run with Full Configuration + +```bash +./docker2pbs.py docker-compose.yml database \ + --pbs-repository user@pbs.example.com:backup-datastore \ + --pbs-username backup-user \ + --pbs-password secret123 \ + --pbs-fingerprint aa:bb:cc:dd:ee:ff... \ + --dry-run +``` + +## Example docker-compose.yml + +```yaml +version: '3.8' +services: + web: + image: nginx:latest + volumes: + - ./html:/var/www/html + - logs_volume:/var/log/nginx + networks: + - frontend + - backend + + database: + image: postgres:13 + volumes: + - db_data:/var/lib/postgresql/data + - ./backups:/backups + networks: + - backend + environment: + POSTGRES_PASSWORD: secret + +volumes: + logs_volume: + db_data: + external: true + +networks: + frontend: + backend: + external: true +``` + +## What the Script Does + +1. **Parses docker-compose.yml** - reads service configuration +2. **Identifies volumes** - both local and external volumes +3. **Identifies networks** - including external networks +4. **Stops the service** - using `docker-compose stop` +5. **Creates PBS backup** - of all service volumes +6. **Restarts the service** - using `docker-compose up -d` + +## Dry-run Mode + +In `--dry-run` mode, the script: +- Shows detailed configuration summary +- Displays all commands that would be executed +- **DOES NOT** stop or start services +- **DOES NOT** create actual backups +- **DOES NOT** perform any destructive operations + +Perfect for testing configuration before running actual backup. + +## PBS Configuration + +Make sure you have `proxmox-backup-client` installed and configured access to PBS. + +## Supported Volume Types + +- **Bind mounts** - local paths (e.g., `./data:/app/data`) +- **Named volumes** - Docker volumes (e.g., `db_data:/var/lib/postgresql/data`) +- **External volumes** - externally managed volumes \ No newline at end of file diff --git a/network_backup_analysis.md b/network_backup_analysis.md new file mode 100644 index 0000000..052f4ec --- /dev/null +++ b/network_backup_analysis.md @@ -0,0 +1,169 @@ +# Network Configuration Backup Analysis + +## Current State - Networks NOT Backed Up + +The current script only **parses and displays** network information but does **NOT backup or restore** network configurations. + +### What the Script Currently Does with Networks: +1. ✅ Parses network configuration from `docker-compose.yml` +2. ✅ Identifies external vs internal networks +3. ✅ Displays network info in dry-run mode +4. ❌ **DOES NOT backup** network configuration +5. ❌ **DOES NOT restore** networks during restore process + +## What Should Be Backed Up for Networks + +### Network Configuration Elements: +1. **Network Definition** - from `docker-compose.yml` +2. **Network Driver** - bridge, overlay, macvlan, etc. +3. **Network Options** - IPAM, subnets, gateways +4. **External Network References** - which networks are external +5. **Service Network Attachments** - which services use which networks + +### Example Network Configurations: + +```yaml +# docker-compose.yml +networks: + frontend: + driver: bridge + ipam: + driver: default + config: + - subnet: 172.20.0.0/16 + gateway: 172.20.0.1 + + backend: + driver: overlay + attachable: true + + external_net: + external: true + name: existing_network +``` + +## Proposed Network Backup Strategy + +### Option 1: Backup Network Configuration as Metadata +- Create `network_config.json` file with network definitions +- Include in backup as separate archive +- Restore by recreating networks before starting services + +### Option 2: Export Full Network State +- Export actual Docker network inspect data +- Include network-specific files/configs if any +- More complex but complete solution + +### Option 3: Documentation Only +- Create human-readable network documentation +- Include required manual steps for network recreation +- Simplest but requires manual intervention + +## Implementation Recommendation + +**Recommended: Option 1 - Network Configuration Metadata** + +### Backup Process: +1. Extract network config from `docker-compose.yml` +2. Get current Docker network state with `docker network inspect` +3. Create `network_config.json` with combined information +4. Add as `network_config.pxar` archive + +### Restore Process: +1. Extract `network_config.pxar` +2. Read network configuration +3. Create missing networks with `docker network create` +4. Skip external networks (assume they exist) +5. Start services (they'll connect to recreated networks) + +## Network Restore Challenges + +### External Networks +- Cannot be recreated by backup script +- Must exist before restore +- Should be documented in restore instructions + +### Network Conflicts +- Network names might conflict with existing networks +- Subnet conflicts possible +- Need conflict resolution strategy + +### Service Dependencies +- Some services might depend on specific network configurations +- Multi-host overlay networks require swarm mode +- Custom network drivers might not be available + +## Required Script Modifications + +### 1. Network Backup Function +```python +def backup_network_config(self) -> str: + """Create network configuration backup""" + network_config = { + 'compose_networks': {}, # From docker-compose.yml + 'docker_networks': {}, # From docker network inspect + 'service_attachments': {} # Which networks this service uses + } + # Implementation needed + return config_file_path +``` + +### 2. Network Restore Function +```python +def restore_networks(self, config_path: str) -> None: + """Restore network configuration""" + # Read network config + # Create missing networks + # Skip external networks + # Log what was created +``` + +### 3. Integration Points +- Add network backup to main backup process +- Add network restore to restore documentation +- Handle network creation before service start + +## Current Workaround + +Until network backup is implemented, users must: + +1. **Document networks manually**: + ```bash + # Export current network config + docker network ls --format "table {{.Name}}\t{{.Driver}}\t{{.Scope}}" + + # Export network details + docker network inspect network_name > network_name_config.json + ``` + +2. **Recreate networks manually**: + ```bash + # Create bridge network + docker network create --driver bridge \ + --subnet=172.20.0.0/16 \ + --gateway=172.20.0.1 \ + frontend + + # Create overlay network + docker network create --driver overlay \ + --attachable \ + backend + ``` + +3. **Use external networks where possible**: + ```yaml + networks: + my_network: + external: true + name: existing_network + ``` + +## Conclusion + +The current script has a **significant gap** - it doesn't backup or restore network configurations. This means: + +- Services might fail to start after restore if networks don't exist +- Network-specific configurations (subnets, drivers, options) are lost +- Manual network recreation is required for complete restore + +This should be addressed in a future version of the script. \ No newline at end of file diff --git a/restore_example.md b/restore_example.md new file mode 100644 index 0000000..5bc6ae4 --- /dev/null +++ b/restore_example.md @@ -0,0 +1,279 @@ +# Docker2PBS Backup Restore Guide + +## Backup Format + +The script creates **separate .pxar archives for each volume** of the service: + +- `volume_db_data.pxar` - for named volumes +- `bind_dot_html.pxar` - for bind mounts (paths converted to safe names) +- `bind_dot_backups.pxar` - for other bind mounts + +## How to Restore Backups + +### 1. List Available Backups +```bash +proxmox-backup-client list --repository user@pbs.example.com:backup-datastore +``` + +### 2. Inspect Specific Backup Contents +```bash +proxmox-backup-client catalog \ + --repository user@pbs.example.com:backup-datastore \ + host/service_name/2024-01-15_10-30-45 +``` + +### 3. Selective Volume Restoration + +#### Restore Docker Volume +```bash +proxmox-backup-client restore \ + --repository user@pbs.example.com:backup-datastore \ + host/service_name/2024-01-15_10-30-45 \ + volume_db_data.pxar /var/lib/docker/volumes/db_data/_data +``` + +#### Restore Bind Mount +```bash +proxmox-backup-client restore \ + --repository user@pbs.example.com:backup-datastore \ + host/service_name/2024-01-15_10-30-45 \ + bind_dot_html.pxar ./html +``` + +### 4. Restore All Archives at Once +```bash +# Example script to restore all volumes +for archive in volume_db_data.pxar bind_dot_html.pxar bind_dot_backups.pxar; do + echo "Restoring $archive..." + proxmox-backup-client restore \ + --repository user@pbs.example.com:backup-datastore \ + host/service_name/2024-01-15_10-30-45 \ + "$archive" "/tmp/restore/$archive/" +done +``` + +## Archive Naming Convention + +The script automatically generates archive names: + +| Volume Type | Source Example | Archive Name | +|-------------|----------------|--------------| +| Named volume | `db_data` | `volume_db_data.pxar` | +| Bind mount | `./html` | `bind_dot_html.pxar` | +| Bind mount | `/app/data` | `bind__app_data.pxar` | +| Bind mount | `../config` | `bind_dotdot_config.pxar` | + +## Benefits of This Approach + +1. **Selective Restoration** - restore only specific volumes +2. **Clear Structure** - each volume has its own descriptive archive name +3. **Easy Management** - inspect contents of specific volumes +4. **Safer Operations** - lower risk of overwriting unwanted files +5. **Flexible Restore** - restore data to different locations + +## Inspect Archive Contents +```bash +proxmox-backup-client catalog \ + --repository user@pbs.example.com:backup-datastore \ + host/service_name/2024-01-15_10-30-45 \ + volume_db_data.pxar +``` + +## Complete Restore Workflow + +### 1. Stop the Service +```bash +docker-compose stop service_name +``` + +### 2. List Available Backups +```bash +proxmox-backup-client list --repository user@pbs.example.com:backup-datastore +``` + +### 3. Inspect Backup Contents +```bash +proxmox-backup-client catalog \ + --repository user@pbs.example.com:backup-datastore \ + host/service_name/2024-01-15_10-30-45 +``` + +### 4. Restore Specific Volumes +```bash +# Restore Docker volume +proxmox-backup-client restore \ + --repository user@pbs.example.com:backup-datastore \ + host/service_name/2024-01-15_10-30-45 \ + volume_db_data.pxar /var/lib/docker/volumes/db_data/_data + +# Restore bind mounts +proxmox-backup-client restore \ + --repository user@pbs.example.com:backup-datastore \ + host/service_name/2024-01-15_10-30-45 \ + bind_dot_html.pxar ./html +``` + +### 5. Start the Service +```bash +docker-compose up -d service_name +``` + +## Advanced Restore Scenarios + +### Restore to Different Location +```bash +# Restore to temporary location for inspection +proxmox-backup-client restore \ + --repository user@pbs.example.com:backup-datastore \ + host/service_name/2024-01-15_10-30-45 \ + volume_db_data.pxar /tmp/inspect_db_data +``` + +### Restore with Different Ownership +```bash +# Restore and fix permissions +proxmox-backup-client restore \ + --repository user@pbs.example.com:backup-datastore \ + host/service_name/2024-01-15_10-30-45 \ + bind_dot_html.pxar ./html + +# Fix ownership if needed +sudo chown -R 1000:1000 ./html +``` + +### Partial Restore (Specific Files) +```bash +# Mount archive and copy specific files +mkdir /tmp/mount_point +proxmox-backup-client mount \ + --repository user@pbs.example.com:backup-datastore \ + host/service_name/2024-01-15_10-30-45 \ + /tmp/mount_point + +# Copy specific files +cp /tmp/mount_point/volume_db_data.pxar/postgresql.conf ./ +cp /tmp/mount_point/bind_dot_html.pxar/index.html ./ + +# Unmount +fusermount -u /tmp/mount_point +``` + +## Automation Scripts + +### Backup Script +```bash +#!/bin/bash +# backup.sh - Automated backup script + +SERVICE_NAME="$1" +COMPOSE_FILE="docker-compose.yml" +PBS_REPO="user@pbs.example.com:datastore" + +if [ -z "$SERVICE_NAME" ]; then + echo "Usage: $0 " + exit 1 +fi + +echo "Starting backup of service: $SERVICE_NAME" +./docker2pbs.py "$COMPOSE_FILE" "$SERVICE_NAME" --pbs-repository "$PBS_REPO" + +if [ $? -eq 0 ]; then + echo "Backup completed successfully" +else + echo "Backup failed" + exit 1 +fi +``` + +### Restore Script +```bash +#!/bin/bash +# restore.sh - Automated restore script + +SERVICE_NAME="$1" +BACKUP_ID="$2" +PBS_REPO="user@pbs.example.com:datastore" + +if [ -z "$SERVICE_NAME" ] || [ -z "$BACKUP_ID" ]; then + echo "Usage: $0 " + echo "Example: $0 database 2024-01-15_10-30-45" + exit 1 +fi + +echo "Stopping service: $SERVICE_NAME" +docker-compose stop "$SERVICE_NAME" + +echo "Listing archives in backup: $BACKUP_ID" +ARCHIVES=$(proxmox-backup-client catalog --repository "$PBS_REPO" "host/$SERVICE_NAME/$BACKUP_ID" | grep "\.pxar$" | awk '{print $1}') + +for ARCHIVE in $ARCHIVES; do + echo "Restoring archive: $ARCHIVE" + # Determine restore path based on archive type + if [[ $ARCHIVE == volume_* ]]; then + VOLUME_NAME=${ARCHIVE#volume_} + VOLUME_NAME=${VOLUME_NAME%.pxar} + RESTORE_PATH="/var/lib/docker/volumes/${VOLUME_NAME}/_data" + else + # For bind mounts, restore to current directory + RESTORE_PATH="./" + fi + + proxmox-backup-client restore \ + --repository "$PBS_REPO" \ + "host/$SERVICE_NAME/$BACKUP_ID" \ + "$ARCHIVE" "$RESTORE_PATH" +done + +echo "Starting service: $SERVICE_NAME" +docker-compose up -d "$SERVICE_NAME" +``` + +## Troubleshooting + +### Common Issues + +1. **Permission Denied** + ```bash + # Check PBS client permissions + ls -la ~/.config/proxmox-backup/ + + # Run with sudo if needed + sudo proxmox-backup-client restore ... + ``` + +2. **Archive Not Found** + ```bash + # List all archives in backup + proxmox-backup-client catalog \ + --repository user@pbs.example.com:datastore \ + host/service_name/backup_id + ``` + +3. **Network/Connection Issues** + ```bash + # Test PBS connectivity + proxmox-backup-client list --repository user@pbs.example.com:datastore + + # Check PBS server status + ping pbs.example.com + ``` + +4. **Disk Space Issues** + ```bash + # Check available space + df -h /var/lib/docker/volumes/ + df -h ./ + + # Clean up if needed + docker volume prune + docker system prune + ``` + +### Best Practices + +- Always stop services before restoring data +- Test restore procedures in non-production environments +- Keep restore documentation up to date +- Monitor disk space during restore operations +- Verify data integrity after restoration +- Create restore scripts for frequently restored services \ No newline at end of file