From b63de7d682e44d7a3a84873f93bb87cae9416b26 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Rados=C5=82aw=20Gierwia=C5=82o?= Date: Thu, 22 Aug 2024 17:30:00 +0200 Subject: [PATCH] Add comprehensive project documentation - Complete README with features, installation, and usage - Docker compose integration examples - PBS authentication and configuration guide - Troubleshooting section with common issues - Archive naming convention documentation --- README.md | 298 ++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 298 insertions(+) create mode 100644 README.md diff --git a/README.md b/README.md new file mode 100644 index 0000000..16de2ef --- /dev/null +++ b/README.md @@ -0,0 +1,298 @@ +# Docker2PBS + +Docker Compose to Proxmox Backup Server integration tool. Automatically backup Docker services with their volumes and configuration to Proxmox Backup Server. + +## Features + +- ✅ **Docker Compose Integration** - Reads service configuration directly from `docker-compose.yml` +- ✅ **Volume Detection** - Automatically identifies bind mounts, named volumes, and external volumes +- ✅ **Network Analysis** - Parses network configuration including external networks +- ✅ **Safe Operations** - Stops service before backup, restarts after completion +- ✅ **Separate Archives** - Creates individual `.pxar` archives for each volume +- ✅ **Selective Restore** - Restore individual volumes or complete service +- ✅ **Dry-run Mode** - Test configuration without executing operations +- ✅ **PBS Integration** - Direct integration with Proxmox Backup Server +- ✅ **Authentication Support** - Supports PBS username/password/fingerprint authentication + +## Requirements + +- Python 3.6+ +- PyYAML +- Docker and docker-compose +- `proxmox-backup-client` installed and configured + +## Installation + +```bash +# Clone or download the script +git clone +cd docker2pbs + +# Install dependencies +pip install -r requirements.txt + +# Make script executable +chmod +x docker2pbs.py +``` + +## Quick Start + +```bash +# Basic backup +./docker2pbs.py docker-compose.yml web --pbs-repository user@pbs.example.com:datastore + +# Test configuration first +./docker2pbs.py docker-compose.yml web --pbs-repository user@pbs.example.com:datastore --dry-run + +# With authentication +./docker2pbs.py docker-compose.yml database \ + --pbs-repository user@pbs.example.com:datastore \ + --pbs-username backup-user \ + --pbs-password secret123 +``` + +## Usage + +``` +./docker2pbs.py --pbs-repository [options] + +Arguments: + compose_file Path to docker-compose.yaml file + service_name Name of the service to backup + +Required Options: + --pbs-repository PBS repository (format: user@host:datastore) + +Optional: + --pbs-username PBS username for authentication + --pbs-password PBS password for authentication + --pbs-fingerprint PBS server fingerprint for verification + --dry-run Show what would be done without executing + --help Show help message +``` + +## How It Works + +### Backup Process + +1. **Configuration Parsing** + - Reads `docker-compose.yml` file + - Extracts service configuration (volumes, networks) + - Identifies volume types (bind mounts vs named volumes) + - Detects external volumes and networks + +2. **Service Management** + - Stops the target service using `docker-compose stop` + - Ensures data consistency during backup + +3. **Backup Creation** + - Creates separate `.pxar` archive for each volume + - Uses descriptive archive names (e.g., `volume_db_data.pxar`, `bind_dot_html.pxar`) + - Uploads to Proxmox Backup Server + +4. **Service Recovery** + - Restarts the service using `docker-compose up -d` + - Ensures service availability is restored + +### Archive Naming Convention + +| Volume Type | Source Example | Archive Name | +|-------------|----------------|--------------| +| Named volume | `db_data` | `volume_db_data.pxar` | +| Bind mount | `./html` | `bind_dot_html.pxar` | +| Bind mount | `/app/data` | `bind__app_data.pxar` | +| Bind mount | `../config` | `bind_dotdot_config.pxar` | + +### Dry-run Mode + +Use `--dry-run` to preview operations: +- Shows detailed service configuration +- Lists all volumes that would be backed up +- Displays archive names that would be created +- Shows exact commands that would be executed +- **No actual operations performed** + +## Examples + +### Basic Web Service Backup + +```yaml +# docker-compose.yml +services: + web: + image: nginx:latest + volumes: + - ./html:/var/www/html + - ./config:/etc/nginx/conf.d + networks: + - webnet +``` + +```bash +# Backup the web service +./docker2pbs.py docker-compose.yml web --pbs-repository backup@pbs.local:webapps + +# Creates these archives: +# - bind_dot_html.pxar (contains ./html content) +# - bind_dot_config.pxar (contains ./config content) +``` + +### Database Service with Named Volumes + +```yaml +# docker-compose.yml +services: + database: + image: postgres:13 + volumes: + - db_data:/var/lib/postgresql/data + - db_backups:/backups + environment: + POSTGRES_PASSWORD: secret + +volumes: + db_data: + db_backups: + external: true +``` + +```bash +# Backup database service +./docker2pbs.py docker-compose.yml database \ + --pbs-repository dba@pbs.local:databases \ + --pbs-username backup-user + +# Creates these archives: +# - volume_db_data.pxar (PostgreSQL data directory) +# - volume_db_backups.pxar (backup directory) +``` + +## Restore Operations + +See [restore_example.md](restore_example.md) for detailed restore procedures. + +### Quick Restore Example + +```bash +# List available backups +proxmox-backup-client list --repository user@pbs.example.com:datastore + +# Restore specific volume +proxmox-backup-client restore \ + --repository user@pbs.example.com:datastore \ + host/web/2024-01-15_10-30-45 \ + volume_db_data.pxar /var/lib/docker/volumes/db_data/_data +``` + +## Configuration + +### PBS Client Setup + +Ensure `proxmox-backup-client` is installed and configured: + +```bash +# Install PBS client (Debian/Ubuntu) +apt install proxmox-backup-client + +# Configure repository (optional) +export PBS_REPOSITORY=user@pbs.example.com:datastore +export PBS_PASSWORD=your_password +``` + +### Supported Volume Types + +- ✅ **Bind Mounts** - Local directories mounted into containers +- ✅ **Named Volumes** - Docker-managed volumes +- ✅ **External Volumes** - Pre-existing volumes managed outside compose +- ❌ **Tmpfs Volumes** - Memory-based volumes (not backed up) +- ❌ **Anonymous Volumes** - Unnamed volumes (not backed up) + +### Supported Network Types + +- ✅ **Default Networks** - Docker compose default networks +- ✅ **Named Networks** - Custom networks defined in compose +- ✅ **External Networks** - Pre-existing networks +- ✅ **Bridge Networks** - Standard bridge networks +- ✅ **Overlay Networks** - Multi-host networks + +## Security Considerations + +- PBS credentials can be provided via command line or environment variables +- Avoid hardcoding passwords in scripts +- Use PBS fingerprint verification for secure connections +- Consider using PBS tokens instead of passwords +- Ensure backup user has minimal required permissions + +## Troubleshooting + +### Common Issues + +1. **Service fails to stop** + ```bash + # Check if service is running + docker-compose ps + + # Manually stop if needed + docker-compose stop service_name + ``` + +2. **Volume path not found** + ```bash + # Check volume exists + docker volume ls + docker volume inspect volume_name + + # For bind mounts, verify path exists + ls -la /path/to/bind/mount + ``` + +3. **PBS connection fails** + ```bash + # Test PBS connectivity + proxmox-backup-client list --repository user@pbs.example.com:datastore + + # Verify credentials + echo $PBS_PASSWORD + ``` + +4. **Permission issues** + ```bash + # Ensure script is executable + chmod +x docker2pbs.py + + # Check Docker permissions + docker ps + ``` + +### Debug Mode + +For verbose output, modify the script to enable debug logging or run with: + +```bash +# Show all executed commands +bash -x docker2pbs.py docker-compose.yml service --pbs-repository repo --dry-run +``` + +## Contributing + +1. Fork the repository +2. Create feature branch (`git checkout -b feature/amazing-feature`) +3. Commit changes (`git commit -m 'Add amazing feature'`) +4. Push to branch (`git push origin feature/amazing-feature`) +5. Open Pull Request + +## License + +This project is licensed under the MIT License - see the LICENSE file for details. + +## Support + +- Create issues for bugs or feature requests +- Check existing documentation in `/docs` directory +- Review example configurations in `/examples` directory + +## Related Projects + +- [Proxmox Backup Server](https://www.proxmox.com/en/proxmox-backup-server) +- [Docker Compose](https://docs.docker.com/compose/) +- [proxmox-backup-client](https://pbs.proxmox.com/docs/backup-client.html) \ No newline at end of file