Training a Neural Net — Loss & Optimisers

• Picture training as rolling downhill on a loss landscape.
• Explain gradient, learning rate, and a training step.
• Know what Adam / SGD do and why Adam is the default.
• Diagnose a learning rate that's too high or too low from the loss curve.

loss
 │   *                         ← bad weights, high error
 │    \
 │     \__                     gradient points UP the slope;
 │        \___                 we step the OPPOSITE way (downhill)
 │            \_____ * ←       good weights, low error
 └────────────────────── weight value

The loss is "how wrong are we". The gradient tells which way the loss increases. Training repeatedly steps the weights a little bit downhill, lowering the loss. That's gradient descent.

One step, in words

1. forward pass:  predict, compute loss
2. backward pass: compute gradient of loss wrt every weight (backprop)
3. update:        weight = weight − learning_rate × gradient
4. repeat for every batch, every epoch

Learning rate — the most important dial

too high   → overshoots, loss bounces or explodes (NaN)
too low    → crawls, takes forever, may get stuck
just right → loss drops smoothly to a low plateau

from tensorflow.keras.optimizers import Adam
model.compile(optimizer=Adam(learning_rate=0.001),  # the default-ish
              loss="sparse_categorical_crossentropy",
              metrics=["accuracy"])

SGD vs Adam

SGD   plain steps downhill; needs careful tuning + momentum
Adam  adapts the step size per-weight automatically;
      the sensible default — start here, almost always works

Gradient descent by hand (1 variable)

# minimise f(w) = (w - 3)**2 ; the minimum is at w = 3
def f(w):  return (w - 3) ** 2
def grad(w): return 2 * (w - 3)

w = 0.0
lr = 0.1
for step in range(20):
    w = w - lr * grad(w)
print(round(w, 3))     # → ~3.0  (rolled to the minimum)

That loop IS training, in miniature — just one weight and a toy loss. Keras does this for millions of weights using calculus you don't have to write.

# lr_demo.py — same net, three learning rates
import numpy as np
from sklearn.datasets import load_iris
from sklearn.preprocessing import StandardScaler
from sklearn.model_selection import train_test_split
from tensorflow import keras
from tensorflow.keras import layers
from tensorflow.keras.optimizers import Adam
import matplotlib.pyplot as plt

X, y = load_iris(return_X_y=True)
X = StandardScaler().fit_transform(X)
Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=0.2, random_state=0)

def make():
    m = keras.Sequential([
        layers.Input(shape=(4,)),
        layers.Dense(16, activation="relu"),
        layers.Dense(3, activation="softmax"),
    ])
    return m

plt.figure(figsize=(8, 5))
for lr in [0.0001, 0.01, 1.0]:
    m = make()
    m.compile(optimizer=Adam(lr),
              loss="sparse_categorical_crossentropy", metrics=["accuracy"])
    h = m.fit(Xtr, ytr, epochs=60, batch_size=8, verbose=0)
    plt.plot(h.history["loss"], label=f"lr={lr}")

plt.xlabel("epoch"); plt.ylabel("training loss"); plt.legend()
plt.title("Effect of learning rate")
plt.savefig("lr_demo.png", dpi=150)

What you'll see

lr=0.0001  loss drops painfully slowly — undershooting
lr=0.01    loss drops fast and smoothly — just right
lr=1.0     loss is erratic / stuck high — overshooting

Read the diff

The loss curve is your training dashboard. Smooth steady decline = healthy. Crawling = LR too low. Spiky or rising = LR too high. When a net "won't learn", the learning rate is the first thing to check — and Adam at ~0.001 is the safe starting point.

Training minimises loss by gradient descent: forward (loss), backward (gradients via backprop), update (step downhill). Learning rate is the step size — too high diverges, too low crawls. Adam adapts per-weight and is the default. The loss curve is your dashboard; read it to diagnose problems. Next: the universal enemy — overfitting.