Bash File Sync (rsync)

Common rsync Commands

rsync -av source/ dest/

rsync -av dir/ user@server:/path/

rsync -av --delete src/ dst/

rsync -av --progress src/ dst/

🔹 Basic Local Sync

rsync excels at synchronizing files between directories on the same system with efficiency and precision. The core command rsync -av source/ destination/ uses archive mode (-a) to preserve permissions, ownership, timestamps, and symlinks, combined with verbose output (-v). A critical detail is the trailing slash on the source path: source/ copies the *contents* of source into destination, while source (no slash) copies the source *directory itself* into destination. This local copy is fast because it doesn't use a network, relying on the efficient delta-transfer algorithm.

# Sync directory contents
rsync -av /source/folder/ /destination/folder/

# Sync directory itself
rsync -av /source/folder /destination/

# Sync with progress
rsync -av --progress /source/ /backup/

🔹 Remote Sync via SSH

rsync's true power is secure, efficient file synchronization over a network using SSH as its default transport. To copy to a remote server, use the syntax rsync -av local_dir/ user@remote_host:/path/to/dest/. The SSH connection is encrypted, and rsync's algorithm sends only the differences (deltas) between files, minimizing bandwidth usage. This makes it perfect for secure backups to remote machines, deploying application code to web servers, or mirroring directories across locations. It's far more efficient and feature-rich than a simple scp copy for repeated synchronization tasks.

# Upload to remote server
rsync -av /local/folder/ [email protected]:/remote/path/

# Download from remote server
rsync -av [email protected]:/remote/folder/ /local/backup/

# Sync with custom SSH port
rsync -av -e "ssh -p 2222" /local/ user@server:/remote/

🔹 Mirror Directories with Delete

The --delete flag makes the destination an exact mirror of the source by removing files that exist only at the destination. This is crucial for creating true backups or staging areas where you want no stray files. However, it is a powerful and dangerous option—files deleted are gone. **Always** perform a dry run first with --dry-run (or -n) to see what would be changed or deleted without making any modifications. For example: rsync -av --dry-run --delete source/ dest/. This safety check prevents catastrophic data loss.

# Mirror directory (delete extra files)
rsync -av --delete /source/ /destination/

# Dry run to preview changes
rsync -av --delete --dry-run /source/ /destination/

# Mirror to remote server
rsync -av --delete /local/ user@server:/backup/

🔹 Exclude Files and Patterns

To prevent certain files from being synchronized, use the --exclude=PATTERN option. You can exclude multiple items by repeating the flag: --exclude='*.log' --exclude='temp/' --exclude='.git'. For complex exclusions, you can also list patterns in a file and use --exclude-from=FILE. This is essential for backups, allowing you to skip temporary files, cache directories, large media files, version control metadata, or any other content that doesn't need to be copied, saving time, bandwidth, and storage space.

# Exclude specific file types
rsync -av --exclude '*.log' /source/ /destination/

# Exclude multiple patterns
rsync -av --exclude '*.tmp' --exclude 'cache/' /source/ /dest/

# Exclude from file list
rsync -av --exclude-from='exclude-list.txt' /source/ /dest/

🔹 Show Transfer Progress

The --progress flag provides a detailed, real-time view of rsync's file transfer activity. For each file, it shows the filename, percentage complete, current transfer speed, and an estimate of the remaining time for that file. When combined with -h (human-readable file sizes), it gives a clear picture of the sync operation's progress, which is especially useful for large transfers. While -v (verbose) lists files, --progress adds the dynamic progress indicators, helping you gauge the speed and estimate completion time during long-running syncs.

# Show progress for each file
rsync -av --progress /source/ /destination/

# Human-readable sizes with progress
rsync -avh --progress /source/ /backup/

# Show overall statistics
rsync -av --stats /source/ /destination/

🔹 Bandwidth Limiting

Prevent rsync from saturating your network link by using the --bwlimit=RATE option, where RATE is in kilobytes per second (KB/s). For example, --bwlimit=1000 limits the transfer to about 1 MB/s (1000 KB/s). This is critical in shared environments or during business hours when you need to perform a large backup or sync without impacting other users' network performance for video calls, browsing, or other latency-sensitive applications. It allows you to schedule data transfers without causing disruptive network congestion.

# Limit to 1000 KB/s (1 MB/s)
rsync -av --bwlimit=1000 /source/ user@server:/backup/

# Limit to 500 KB/s
rsync -av --bwlimit=500 /large-files/ /destination/

# Combine with progress
rsync -av --bwlimit=2000 --progress /source/ /dest/

🔹 Incremental Backups

Create efficient, space-saving backup snapshots using rsync's --link-dest=DIR option, the core of tools like rsnapshot. This technique creates hard links for unchanged files from a previous backup snapshot, while copying only new or modified files. The result is a full, browsable backup that only uses additional disk space for the files that have actually changed since the last snapshot. A typical command might look like: rsync -av --link-dest=/backups/yesterday /source/ /backups/today/. This enables daily or hourly backups with minimal storage overhead.

# Create incremental backup
rsync -av --link-dest=/backup/previous/ /source/ /backup/current/

# Daily backup script example
TODAY=$(date +%Y-%m-%d)
YESTERDAY=$(date -d "yesterday" +%Y-%m-%d)
rsync -av --link-dest=/backup/$YESTERDAY/ /source/ /backup/$TODAY/