DevOps Mindset: Idempotence, Logging, Alerts

By the end of this lesson you can:

• Explain and apply idempotence as a design default, not an afterthought.
• Build observability: logs, metrics, and a heartbeat so silence means failure.
• Design alerting that pages on real problems without crying wolf.
• Adopt the supporting habits: config over hard-coding, secrets in env, dry-run by default.

A SCRIPT                       A SYSTEM
runs when you run it       →   runs unattended, on schedule
works on your machine      →   works anywhere (config + env)
prints to your screen      →   logs to files you can read later
breaks silently            →   alerts you before users notice
re-running may double-act  →   re-running is safe (idempotent)
"it worked once"           →   "it keeps working"

Pillar 1 · Idempotence — design for re-runs

Schedulers retry. Humans re-run. Crashes resume. So every operation should be safe to repeat. Make idempotence the default, not a patch:

# create-if-missing, not create (which errors on the 2nd run)
Path("output").mkdir(parents=True, exist_ok=True)

# upsert, not insert (no duplicate rows on re-run)
db.execute("INSERT OR REPLACE INTO t VALUES (?, ?)", (id, value))

# check-then-act with a key (no double email/charge)
if id not in already_sent:
    send(id); already_sent.add(id)

Ask of every step: "what happens if this runs twice?" If the answer is "a problem," redesign it. Idempotence is what makes retries and resumes (Lesson 45) safe.

Pillar 2 · Observability — see what happened

# 1) structured logs (Lessons 13-14): what, when, how serious
log.info("processed %d records in %.1fs", n, elapsed)

# 2) metrics: numbers you can track over time
metrics = {"records": n, "duration_s": elapsed, "errors": errs}
Path("metrics.jsonl").open("a").write(json.dumps(metrics) + "\n")

# 3) heartbeat: prove "it ran", so silence == failure
write_heartbeat(success=True, at=datetime.now())

• Logs tell the story of one run (and persist via rotation).
• Metrics let you spot trends — "the job is getting slower," "errors are creeping up."
• Heartbeat turns absence into a signal: if the expected success ping doesn't arrive, something is wrong even if nothing crashed loudly.

Pillar 3 · Alerting — find out first

The goal: you learn about a problem before your users do. But over-alerting is its own failure — if everything pages, people ignore everything.

Good alert       actionable, urgent, rare        → page someone
Noise            informational, frequent, "FYI"   → log it, don't page
Alert fatigue    too many alerts → all ignored    → the real danger

# alert on what's actionable, at the right severity, debounced
if overall == "critical":
    raise_alert("health", "critical", details)   # pages on-call (Lesson 34)
elif overall == "warning":
    raise_alert("health", "warning", details)    # Slack only
# everything else → just logs

The supporting habits

• Config over hard-coding — paths, thresholds, recipients in config/env (Lesson 8), so the same code runs in dev and prod.
• Secrets in the environment — never in code or git (Lessons 8, 25); rotate on exposure.
• Dry-run by default for destructive actions — show before you act (Lessons 38, 44).
• Fail loud, recover gracefully — exit non-zero, alert, but checkpoint so recovery is cheap (Lesson 45).

Goal: take a naive script and harden it through the three pillars — the same transformation you'd apply to anything before trusting it in production.

# BEFORE — a "works on my machine" script
import requests, csv

data = requests.get("https://api.example.com/sales?key=sk-live-abc123").json()
with open("/Users/me/report.csv", "w") as f:        # hard-coded path
    w = csv.writer(f)
    for row in data:                                 # crashes on one bad row
        w.writerow([row["id"], row["amount"]])
print("done")                                        # no record, no alert

# AFTER — production-grade
import os, csv, json, logging, sys
from pathlib import Path
from datetime import datetime
from dotenv import load_dotenv
import requests
from requests.adapters import HTTPAdapter
from urllib3.util.retry import Retry

HERE = Path(__file__).resolve().parent
load_dotenv(HERE / ".env")                           # secrets from env
logging.basicConfig(level=logging.INFO,              # observability: logs
    format="%(asctime)s %(levelname)s %(message)s")
log = logging.getLogger("sales")

def session():
    retry = Retry(total=5, backoff_factor=1,
                  status_forcelist=[429, 500, 502, 503, 504])
    s = requests.Session(); s.mount("https://", HTTPAdapter(max_retries=retry))
    return s

def run() -> int:
    out = Path(os.getenv("OUTPUT_DIR", HERE / "out"))  # config over hard-coding
    out.mkdir(parents=True, exist_ok=True)             # idempotent
    started = datetime.now()
    try:
        r = session().get("https://api.example.com/sales",
                          headers={"Authorization": f"Bearer {os.environ['API_KEY']}"},
                          timeout=15)                  # secret in env, resilient
        r.raise_for_status()
        rows, errors = r.json(), 0
        target = out / "report.csv"
        with open(target, "w", newline="", encoding="utf-8") as f:
            w = csv.writer(f); w.writerow(["id", "amount"])
            for row in rows:
                try:
                    w.writerow([row["id"], float(row["amount"])])
                except (KeyError, ValueError):         # per-item isolation
                    errors += 1; log.warning("bad row skipped: %s", row)
        elapsed = (datetime.now() - started).total_seconds()
        log.info("wrote %d rows (%d skipped) in %.1fs", len(rows), errors, elapsed)
        # metrics + heartbeat
        (out / "metrics.jsonl").open("a").write(json.dumps(
            {"at": started.isoformat(), "rows": len(rows),
             "errors": errors, "duration_s": elapsed}) + "\n")
        return 0
    except Exception as e:
        log.exception("run failed")
        # alert(f"sales export FAILED: {e}", "critical")   # Lesson 34
        return 1

if __name__ == "__main__":
    sys.exit(run())                                    # fail loud (exit code)

Read the diff

Same job, transformed: the secret moved to the environment, the path became config with a safe default, mkdir(exist_ok=True) made it idempotent, a resilient session handles transient errors, per-row try/except isolates bad data, logs + a metrics line + (commented) alert give observability, and sys.exit fails loud for the scheduler. None of it is new — it's the habits applied. That checklist is what "production-ready" means.

The DevOps mindset turns scripts into systems via three pillars: idempotence (every step safe to re-run — the default, not a patch), observability (logs + metrics + heartbeat, so you can always see what happened and silence means failure), and alerting (page only on actionable, urgent, rare problems — protect attention from alert fatigue). Supporting habits: config over hard-coding, secrets in env, dry-run for destructive actions, fail loud but recover cheaply. None of this is new code — it's the disciplined application of everything in Level 7. Carry this checklist into the capstone.