Cleaning Messy Text Data

• Strip whitespace with strip, lstrip, rstrip.
• Normalise case with lower, upper, title.
• Replace substrings with replace; collapse with split + join.
• Detect missing or empty values defensively.

Here's a tiny customer list. Find five problems.

name,email,city
Ahmad ,ahmad@EMAIL.com,KL
mei,  mei@email.com  ,kuala lumpur
SURESH,suresh@email.com,Kuala Lumpur
Aisyah,,K.L.
   Devi   ,devi@email.com,kl

Whitespace

s = "   hello world   "
print(repr(s.strip()))         # → 'hello world'
print(repr(s.lstrip()))        # → 'hello world   '
print(repr(s.rstrip()))        # → '   hello world'

Case

print("KuALa LumPuR".lower())      # → kuala lumpur
print("kuala lumpur".upper())      # → KUALA LUMPUR
print("kuala lumpur".title())      # → Kuala Lumpur

Replace and collapse

s = "K.L."
print(s.replace(".", ""))          # → KL

# Collapse runs of whitespace using split + join
s = "kuala       lumpur"
print(" ".join(s.split()))         # → kuala lumpur

"a b".split() with no argument splits on any run of whitespace AND drops empties — handy for collapsing.

Detecting missing values

val = ""
if not val.strip():
    print("missing")

An empty string is falsy. val.strip() turns "all-whitespace" values into "", which makes them falsy too. not val.strip() means "blank or only whitespace".

A clean function

Bundle the work into one function and call it for every value:

def clean(s):
    s = s.strip().lower()
    s = " ".join(s.split())
    return s.replace(".", "")

print(clean("   K.L.  "))          # → kl
print(clean("kuala  lumpur"))      # → kuala lumpur
print(clean("KuAlA   LuMpUr "))    # → kuala lumpur

Save the dirty data from the warm-up as customers.csv. Now clean it.

# clean_customers.py — normalise messy strings
import csv
from pathlib import Path

CITY_MAP = {
    "kl": "Kuala Lumpur",
    "kuala lumpur": "Kuala Lumpur",
}

def clean_text(s):
    if s is None:
        return ""
    return " ".join(s.strip().split())

def clean_city(s):
    key = clean_text(s).lower().replace(".", "")
    return CITY_MAP.get(key, clean_text(s).title())

with open("customers.csv", newline="") as f:
    rows = list(csv.DictReader(f))

cleaned = []
for r in rows:
    cleaned.append({
        "name":  clean_text(r["name"]).title(),
        "email": clean_text(r["email"]).lower(),
        "city":  clean_city(r["city"]),
    })

# Drop rows missing an email
final = [r for r in cleaned if r["email"]]

print(f"Loaded {len(rows)} rows, kept {len(final)} after cleaning.\n")
for r in final:
    print(f"  {r['name']:<10} {r['email']:<25} {r['city']}")

with open("customers_clean.csv", "w", newline="") as f:
    w = csv.DictWriter(f, fieldnames=["name", "email", "city"])
    w.writeheader()
    w.writerows(final)

Sample output

Loaded 5 rows, kept 4 after cleaning.

  Ahmad      ahmad@email.com           Kuala Lumpur
  Mei        mei@email.com             Kuala Lumpur
  Suresh     suresh@email.com          Kuala Lumpur
  Devi       devi@email.com            Kuala Lumpur

Read the diff

Three helpers (clean_text, clean_city, and the missing-email filter) together fix all five problems. The cleanup is in one place — easy to test, easy to extend. That's the "clean once, near the loader" rule.

Six tools handle 90% of dirty data: strip, lower/upper/title, replace, split + join, a falsy check for missing values, and a small lookup-map for spelling variants. Wrap them in one clean_* function per column and apply at load time. Downstream code can then trust the data.