radziel/docker2pbs
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Add additional documentation and examples

Browse Source 
- Usage examples for different scenarios
- Network backup analysis documentation
- Restore operation examples and procedures
This commit is contained in:
Radosław Gierwiało
2024-08-24 18:00:00 +02:00
parent b63de7d682
commit 7bad771a59
3 changed files with 558 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

110
example_usage.md Normal file
View File

@@ -0,0 +1,110 @@
# Docker2PBS - Usage Examples
## Installation
```bash
pip install -r requirements.txt
chmod +x docker2pbs.py
```
## Basic Usage
```bash
./docker2pbs.py docker-compose.yml web --pbs-repository user@pbs.example.com:backup-datastore
```
## Dry-run Mode (Testing)
Use `--dry-run` to see what would be executed without actually performing operations:
```bash
./docker2pbs.py docker-compose.yml web \
--pbs-repository user@pbs.example.com:backup-datastore \
--dry-run
```
## Usage with Authentication
```bash
./docker2pbs.py docker-compose.yml database \
--pbs-repository user@pbs.example.com:backup-datastore \
--pbs-username backup-user \
--pbs-password secret123 \
--pbs-fingerprint aa:bb:cc:dd:ee:ff...
```
## Dry-run with Full Configuration
```bash
./docker2pbs.py docker-compose.yml database \
--pbs-repository user@pbs.example.com:backup-datastore \
--pbs-username backup-user \
--pbs-password secret123 \
--pbs-fingerprint aa:bb:cc:dd:ee:ff... \
--dry-run
```
## Example docker-compose.yml
```yaml
version: '3.8'
services:
web:
image: nginx:latest
volumes:
- ./html:/var/www/html
- logs_volume:/var/log/nginx
networks:
- frontend
- backend
database:
image: postgres:13
volumes:
- db_data:/var/lib/postgresql/data
- ./backups:/backups
networks:
- backend
environment:
POSTGRES_PASSWORD: secret
volumes:
logs_volume:
db_data:
external: true
networks:
frontend:
backend:
external: true
```
## What the Script Does
1. **Parses docker-compose.yml** - reads service configuration
2. **Identifies volumes** - both local and external volumes
3. **Identifies networks** - including external networks
4. **Stops the service** - using `docker-compose stop`
5. **Creates PBS backup** - of all service volumes
6. **Restarts the service** - using `docker-compose up -d`
## Dry-run Mode
In `--dry-run` mode, the script:
- Shows detailed configuration summary
- Displays all commands that would be executed
- **DOES NOT** stop or start services
- **DOES NOT** create actual backups
- **DOES NOT** perform any destructive operations
Perfect for testing configuration before running actual backup.
## PBS Configuration
Make sure you have `proxmox-backup-client` installed and configured access to PBS.
## Supported Volume Types
- **Bind mounts** - local paths (e.g., `./data:/app/data`)
- **Named volumes** - Docker volumes (e.g., `db_data:/var/lib/postgresql/data`)
- **External volumes** - externally managed volumes

169
network_backup_analysis.md Normal file
View File

@@ -0,0 +1,169 @@
# Network Configuration Backup Analysis
## Current State - Networks NOT Backed Up
The current script only **parses and displays** network information but does **NOT backup or restore** network configurations.
### What the Script Currently Does with Networks:
1. ✅ Parses network configuration from `docker-compose.yml`
2. ✅ Identifies external vs internal networks
3. ✅ Displays network info in dry-run mode
4. ❌ **DOES NOT backup** network configuration
5. ❌ **DOES NOT restore** networks during restore process
## What Should Be Backed Up for Networks
### Network Configuration Elements:
1. **Network Definition** - from `docker-compose.yml`
2. **Network Driver** - bridge, overlay, macvlan, etc.
3. **Network Options** - IPAM, subnets, gateways
4. **External Network References** - which networks are external
5. **Service Network Attachments** - which services use which networks
### Example Network Configurations:
```yaml
# docker-compose.yml
networks:
frontend:
driver: bridge
ipam:
driver: default
config:
- subnet: 172.20.0.0/16
gateway: 172.20.0.1
backend:
driver: overlay
attachable: true
external_net:
external: true
name: existing_network
```
## Proposed Network Backup Strategy
### Option 1: Backup Network Configuration as Metadata
- Create `network_config.json` file with network definitions
- Include in backup as separate archive
- Restore by recreating networks before starting services
### Option 2: Export Full Network State
- Export actual Docker network inspect data
- Include network-specific files/configs if any
- More complex but complete solution
### Option 3: Documentation Only
- Create human-readable network documentation
- Include required manual steps for network recreation
- Simplest but requires manual intervention
## Implementation Recommendation
**Recommended: Option 1 - Network Configuration Metadata**
### Backup Process:
1. Extract network config from `docker-compose.yml`
2. Get current Docker network state with `docker network inspect`
3. Create `network_config.json` with combined information
4. Add as `network_config.pxar` archive
### Restore Process:
1. Extract `network_config.pxar`
2. Read network configuration
3. Create missing networks with `docker network create`
4. Skip external networks (assume they exist)
5. Start services (they'll connect to recreated networks)
## Network Restore Challenges
### External Networks
- Cannot be recreated by backup script
- Must exist before restore
- Should be documented in restore instructions
### Network Conflicts
- Network names might conflict with existing networks
- Subnet conflicts possible
- Need conflict resolution strategy
### Service Dependencies
- Some services might depend on specific network configurations
- Multi-host overlay networks require swarm mode
- Custom network drivers might not be available
## Required Script Modifications
### 1. Network Backup Function
```python
def backup_network_config(self) -> str:
"""Create network configuration backup"""
network_config = {
'compose_networks': {}, # From docker-compose.yml
'docker_networks': {}, # From docker network inspect
'service_attachments': {} # Which networks this service uses
}
# Implementation needed
return config_file_path
```
### 2. Network Restore Function
```python
def restore_networks(self, config_path: str) -> None:
"""Restore network configuration"""
# Read network config
# Create missing networks
# Skip external networks
# Log what was created
```
### 3. Integration Points
- Add network backup to main backup process
- Add network restore to restore documentation
- Handle network creation before service start
## Current Workaround
Until network backup is implemented, users must:
1. **Document networks manually**:
```bash
# Export current network config
docker network ls --format "table {{.Name}}\t{{.Driver}}\t{{.Scope}}"
# Export network details
docker network inspect network_name > network_name_config.json
```
2. **Recreate networks manually**:
```bash
# Create bridge network
docker network create --driver bridge \
--subnet=172.20.0.0/16 \
--gateway=172.20.0.1 \
frontend
# Create overlay network
docker network create --driver overlay \
--attachable \
backend
```
3. **Use external networks where possible**:
```yaml
networks:
my_network:
external: true
name: existing_network
```
## Conclusion
The current script has a **significant gap** - it doesn't backup or restore network configurations. This means:
- Services might fail to start after restore if networks don't exist
- Network-specific configurations (subnets, drivers, options) are lost
- Manual network recreation is required for complete restore
This should be addressed in a future version of the script.

279
restore_example.md Normal file
View File

@@ -0,0 +1,279 @@
# Docker2PBS Backup Restore Guide
## Backup Format
The script creates **separate .pxar archives for each volume** of the service:
- `volume_db_data.pxar` - for named volumes
- `bind_dot_html.pxar` - for bind mounts (paths converted to safe names)
- `bind_dot_backups.pxar` - for other bind mounts
## How to Restore Backups
### 1. List Available Backups
```bash
proxmox-backup-client list --repository user@pbs.example.com:backup-datastore
```
### 2. Inspect Specific Backup Contents
```bash
proxmox-backup-client catalog \
--repository user@pbs.example.com:backup-datastore \
host/service_name/2024-01-15_10-30-45
```
### 3. Selective Volume Restoration
#### Restore Docker Volume
```bash
proxmox-backup-client restore \
--repository user@pbs.example.com:backup-datastore \
host/service_name/2024-01-15_10-30-45 \
volume_db_data.pxar /var/lib/docker/volumes/db_data/_data
```
#### Restore Bind Mount
```bash
proxmox-backup-client restore \
--repository user@pbs.example.com:backup-datastore \
host/service_name/2024-01-15_10-30-45 \
bind_dot_html.pxar ./html
```
### 4. Restore All Archives at Once
```bash
# Example script to restore all volumes
for archive in volume_db_data.pxar bind_dot_html.pxar bind_dot_backups.pxar; do
echo "Restoring $archive..."
proxmox-backup-client restore \
--repository user@pbs.example.com:backup-datastore \
host/service_name/2024-01-15_10-30-45 \
"$archive" "/tmp/restore/$archive/"
done
```
## Archive Naming Convention
The script automatically generates archive names:
| Volume Type | Source Example | Archive Name |
|-------------|----------------|--------------|
| Named volume | `db_data` | `volume_db_data.pxar` |
| Bind mount | `./html` | `bind_dot_html.pxar` |
| Bind mount | `/app/data` | `bind__app_data.pxar` |
| Bind mount | `../config` | `bind_dotdot_config.pxar` |
## Benefits of This Approach
1. **Selective Restoration** - restore only specific volumes
2. **Clear Structure** - each volume has its own descriptive archive name
3. **Easy Management** - inspect contents of specific volumes
4. **Safer Operations** - lower risk of overwriting unwanted files
5. **Flexible Restore** - restore data to different locations
## Inspect Archive Contents
```bash
proxmox-backup-client catalog \
--repository user@pbs.example.com:backup-datastore \
host/service_name/2024-01-15_10-30-45 \
volume_db_data.pxar
```
## Complete Restore Workflow
### 1. Stop the Service
```bash
docker-compose stop service_name
```
### 2. List Available Backups
```bash
proxmox-backup-client list --repository user@pbs.example.com:backup-datastore
```
### 3. Inspect Backup Contents
```bash
proxmox-backup-client catalog \
--repository user@pbs.example.com:backup-datastore \
host/service_name/2024-01-15_10-30-45
```
### 4. Restore Specific Volumes
```bash
# Restore Docker volume
proxmox-backup-client restore \
--repository user@pbs.example.com:backup-datastore \
host/service_name/2024-01-15_10-30-45 \
volume_db_data.pxar /var/lib/docker/volumes/db_data/_data
# Restore bind mounts
proxmox-backup-client restore \
--repository user@pbs.example.com:backup-datastore \
host/service_name/2024-01-15_10-30-45 \
bind_dot_html.pxar ./html
```
### 5. Start the Service
```bash
docker-compose up -d service_name
```
## Advanced Restore Scenarios
### Restore to Different Location
```bash
# Restore to temporary location for inspection
proxmox-backup-client restore \
--repository user@pbs.example.com:backup-datastore \
host/service_name/2024-01-15_10-30-45 \
volume_db_data.pxar /tmp/inspect_db_data
```
### Restore with Different Ownership
```bash
# Restore and fix permissions
proxmox-backup-client restore \
--repository user@pbs.example.com:backup-datastore \
host/service_name/2024-01-15_10-30-45 \
bind_dot_html.pxar ./html
# Fix ownership if needed
sudo chown -R 1000:1000 ./html
```
### Partial Restore (Specific Files)
```bash
# Mount archive and copy specific files
mkdir /tmp/mount_point
proxmox-backup-client mount \
--repository user@pbs.example.com:backup-datastore \
host/service_name/2024-01-15_10-30-45 \
/tmp/mount_point
# Copy specific files
cp /tmp/mount_point/volume_db_data.pxar/postgresql.conf ./
cp /tmp/mount_point/bind_dot_html.pxar/index.html ./
# Unmount
fusermount -u /tmp/mount_point
```
## Automation Scripts
### Backup Script
```bash
#!/bin/bash
# backup.sh - Automated backup script
SERVICE_NAME="$1"
COMPOSE_FILE="docker-compose.yml"
PBS_REPO="user@pbs.example.com:datastore"
if [ -z "$SERVICE_NAME" ]; then
echo "Usage: $0 <service_name>"
exit 1
fi
echo "Starting backup of service: $SERVICE_NAME"
./docker2pbs.py "$COMPOSE_FILE" "$SERVICE_NAME" --pbs-repository "$PBS_REPO"
if [ $? -eq 0 ]; then
echo "Backup completed successfully"
else
echo "Backup failed"
exit 1
fi
```
### Restore Script
```bash
#!/bin/bash
# restore.sh - Automated restore script
SERVICE_NAME="$1"
BACKUP_ID="$2"
PBS_REPO="user@pbs.example.com:datastore"
if [ -z "$SERVICE_NAME" ] || [ -z "$BACKUP_ID" ]; then
echo "Usage: $0 <service_name> <backup_id>"
echo "Example: $0 database 2024-01-15_10-30-45"
exit 1
fi
echo "Stopping service: $SERVICE_NAME"
docker-compose stop "$SERVICE_NAME"
echo "Listing archives in backup: $BACKUP_ID"
ARCHIVES=$(proxmox-backup-client catalog --repository "$PBS_REPO" "host/$SERVICE_NAME/$BACKUP_ID" | grep "\.pxar$" | awk '{print $1}')
for ARCHIVE in $ARCHIVES; do
echo "Restoring archive: $ARCHIVE"
# Determine restore path based on archive type
if [[ $ARCHIVE == volume_* ]]; then
VOLUME_NAME=${ARCHIVE#volume_}
VOLUME_NAME=${VOLUME_NAME%.pxar}
RESTORE_PATH="/var/lib/docker/volumes/${VOLUME_NAME}/_data"
else
# For bind mounts, restore to current directory
RESTORE_PATH="./"
fi
proxmox-backup-client restore \
--repository "$PBS_REPO" \
"host/$SERVICE_NAME/$BACKUP_ID" \
"$ARCHIVE" "$RESTORE_PATH"
done
echo "Starting service: $SERVICE_NAME"
docker-compose up -d "$SERVICE_NAME"
```
## Troubleshooting
### Common Issues
1. **Permission Denied**
```bash
# Check PBS client permissions
ls -la ~/.config/proxmox-backup/
# Run with sudo if needed
sudo proxmox-backup-client restore ...
```
2. **Archive Not Found**
```bash
# List all archives in backup
proxmox-backup-client catalog \
--repository user@pbs.example.com:datastore \
host/service_name/backup_id
```
3. **Network/Connection Issues**
```bash
# Test PBS connectivity
proxmox-backup-client list --repository user@pbs.example.com:datastore
# Check PBS server status
ping pbs.example.com
```
4. **Disk Space Issues**
```bash
# Check available space
df -h /var/lib/docker/volumes/
df -h ./
# Clean up if needed
docker volume prune
docker system prune
```
### Best Practices
- Always stop services before restoring data
- Test restore procedures in non-production environments
- Keep restore documentation up to date
- Monitor disk space during restore operations
- Verify data integrity after restoration
- Create restore scripts for frequently restored services