Backup & Sync Automation

By the end of this lesson you can:

• Detect changed files reliably by hashing their contents.
• Do incremental backups — copy only what changed.
• Keep versioned, dated snapshots and rotate old ones (retention).
• One-way sync a source folder to a destination (mirror), including deletes.

Backup wisdom in one line: 3 copies of your data, on 2 different media, with 1 off-site. We're building the engine that makes the extra copies — automatically and efficiently.

Hashing a file

import hashlib
from pathlib import Path

def file_hash(path: Path, algo: str = "sha256") -> str:
    h = hashlib.new(algo)
    with open(path, "rb") as f:
        for chunk in iter(lambda: f.read(65536), b""):   # read in 64KB blocks
            h.update(chunk)
    return h.hexdigest()

Reading in chunks lets you hash huge files without loading them into memory. The hex digest is a fingerprint: same content → same hash. Comparing hashes (not timestamps) is the reliable change-detector — timestamps can lie, content can't.

A manifest of the last backup

import json

def load_manifest(path: Path) -> dict:
    return json.loads(path.read_text()) if path.exists() else {}

def save_manifest(path: Path, manifest: dict) -> None:
    path.write_text(json.dumps(manifest, indent=2))

Store {relative_path: hash} for the last backup. Next run, re-hash the source and compare to the manifest to find what's new or changed — that's the "incremental" magic.

Incremental copy

import shutil

def incremental_backup(src: Path, dest: Path, manifest_path: Path) -> dict:
    old = load_manifest(manifest_path)
    new, changed = {}, []
    for f in src.rglob("*"):
        if not f.is_file():
            continue
        rel = str(f.relative_to(src))
        digest = file_hash(f)
        new[rel] = digest
        if old.get(rel) != digest:           # new or modified
            target = dest / rel
            target.parent.mkdir(parents=True, exist_ok=True)
            shutil.copy2(f, target)          # copy2 keeps timestamps
            changed.append(rel)
    save_manifest(manifest_path, new)
    return {"total": len(new), "copied": len(changed), "files": changed}

Only files whose hash differs from the manifest get copied. The first run copies everything; later runs copy just the deltas — often a handful of files instead of thousands.

Versioned snapshots + retention

from datetime import datetime

def snapshot(src: Path, backup_root: Path, keep: int = 7) -> Path:
    stamp = datetime.now().strftime("%Y%m%d-%H%M%S")
    dest = backup_root / stamp
    shutil.copytree(src, dest)
    # retention: keep only the newest <keep> snapshots
    snaps = sorted([d for d in backup_root.iterdir() if d.is_dir()],
                   reverse=True)
    for old in snaps[keep:]:
        shutil.rmtree(old)
    return dest

Dated snapshot folders give you point-in-time recovery ("restore yesterday"); the retention loop keeps disk bounded by deleting all but the newest N — the rotation pattern from Lesson 7.

Goal: a CLI that does a fast incremental backup, reports exactly what changed, and offers a verify mode that re-hashes the backup to confirm integrity.

import argparse, hashlib, json, shutil, logging
from pathlib import Path
from datetime import datetime

logging.basicConfig(level=logging.INFO, format="%(levelname)s %(message)s")
log = logging.getLogger("backup")

def file_hash(path: Path) -> str:
    h = hashlib.sha256()
    with open(path, "rb") as f:
        for chunk in iter(lambda: f.read(65536), b""):
            h.update(chunk)
    return h.hexdigest()

def backup(src: str, dest: str) -> None:
    src_p, dest_p = Path(src), Path(dest)
    dest_p.mkdir(parents=True, exist_ok=True)
    manifest_p = dest_p / ".manifest.json"
    old = json.loads(manifest_p.read_text()) if manifest_p.exists() else {}

    new, copied, unchanged = {}, 0, 0
    for f in src_p.rglob("*"):
        if not f.is_file():
            continue
        rel = str(f.relative_to(src_p))
        digest = file_hash(f)
        new[rel] = digest
        if old.get(rel) == digest:
            unchanged += 1
            continue
        target = dest_p / rel
        target.parent.mkdir(parents=True, exist_ok=True)
        shutil.copy2(f, target)
        copied += 1
        log.info("copied %s", rel)

    manifest_p.write_text(json.dumps(new, indent=2))
    log.info("backup done: %d copied, %d unchanged, %d total",
             copied, unchanged, len(new))

def verify(dest: str) -> None:
    dest_p = Path(dest)
    manifest = json.loads((dest_p / ".manifest.json").read_text())
    bad = 0
    for rel, expected in manifest.items():
        f = dest_p / rel
        if not f.exists() or file_hash(f) != expected:
            log.error("CORRUPT/MISSING: %s", rel); bad += 1
    log.info("verify: %d files, %d problems", len(manifest), bad)

if __name__ == "__main__":
    p = argparse.ArgumentParser(description="Incremental backup.")
    sub = p.add_subparsers(dest="cmd", required=True)
    b = sub.add_parser("run"); b.add_argument("src"); b.add_argument("dest")
    v = sub.add_parser("verify"); v.add_argument("dest")
    a = p.parse_args()
    backup(a.src, a.dest) if a.cmd == "run" else verify(a.dest)

$ python backup.py run documents backups/docs
INFO copied report.docx
INFO copied notes/todo.txt
INFO backup done: 2 copied, 148 unchanged, 150 total

$ python backup.py verify backups/docs
INFO verify: 150 files, 0 problems

Read the code

The manifest of hashes makes the second run almost instant: 148 unchanged files are skipped, only the 2 edited ones copy. The verify command re-hashes the backup against the manifest to catch silent corruption (bit-rot, bad disk) — a real backup tool you can trust must prove the copy is intact, not just assume it. Schedule backup.py run with cron (Lesson 36) and your documents back themselves up incrementally every night.

Smart backups copy only what changed, and the reliable change-detector is a content hash (compare hashes, not timestamps). Store a manifest of path → hash, re-hash the source each run, and copy just the deltas — instant after the first run. Keep dated snapshots for point-in-time recovery and a retention loop to bound disk. Verify backups by re-hashing against the manifest. Know the difference between a versioned backup (keeps history) and a mirror sync (deletes to match, dangerous — default to dry-run). These primitives underpin every real backup system, including the database backups of Lesson 39.