Enleashed

We now have almost all the pieces needed for a machine to learn.

We have:

TRAINING EXAMPLES

↓

MODEL

↓

PREDICTIONS

↓

LOSS FUNCTION

↓

A NUMBER TELLING US HOW WRONG THE MODEL IS

But one crucial question remains:

How should the model change its parameters to make the loss smaller?

For many machine-learning models, the answer is gradient descent.

At its simplest:

Gradient descent is a method for repeatedly adjusting model parameters in a direction that reduces the loss.

Conceptually:

CURRENT PARAMETERS

↓

CALCULATE LOSS

↓

ASK WHICH WAY MAKES LOSS FALL

↓

MOVE PARAMETERS A LITTLE

↓

CALCULATE LOSS AGAIN

↓

REPEAT

This simple idea sits at the heart of much of modern machine learning.

Imagine standing on a hill

Imagine you are standing somewhere on a landscape.

Your goal is to reach a low point.

But there is a problem:

you cannot see the entire landscape.

You can only determine:

Which way is downhill from where I am standing?

So you:

determine the local slope,
take a small step downhill,
determine the slope again,
take another step,
repeat.

That is the basic intuition behind gradient descent.

The landscape is the loss function

In machine learning:

your position

represents:

the current parameter values.

The:

height of the landscape

represents:

the loss.

So:

HIGHER = WORSE

LOWER = BETTER.

Training means moving through parameter space towards regions of lower loss.

Start with one parameter

Suppose our model contains one parameter:

Different values of w produce different predictions.

Those predictions produce different losses.

Imagine:

Parameter w	Loss
-3	30
-2	18
-1	9
0	4
1	1
2	3
3	10

The best value in this simple example is around:

w = 1.

But imagine we do not know that beforehand.

We need a method for finding it.

Start somewhere

Suppose we begin at:

w = -2.

The loss is:

18.

We ask:

If I increase w slightly, does the loss rise or fall?

Suppose it falls.

That tells us:

move w upwards.

So we might update:

-2 → -1.5.

Then evaluate again.

Follow the slope

At:

w = -1.5

the loss is lower.

The slope still tells us:

Move towards larger w.

So:

-1.5 → -1.0

then:

-1.0 → -0.5

then:

-0.5 → 0

and so on.

Eventually we approach the bottom of the valley.

The derivative tells us the slope

For a function with one parameter, calculus gives us a tool for measuring local slope:

the derivative.

Conceptually:

derivative > 0

means:

Increasing the parameter increases the loss.

So we should move the parameter:

downwards.

And:

derivative < 0

means:

Increasing the parameter decreases the loss.

So we should move the parameter:

upwards.

Move in the opposite direction

This is why gradient descent moves in the opposite direction to the derivative.

If the slope points uphill:

go the other way.

Conceptually:

NEW PARAMETER

OLD PARAMETER

−

STEP SIZE × SLOPE.

In mathematical notation:

θ_new = θ_old - η∇L(θ)

where:

θ represents the model parameters,
L is the loss,
∇L is the gradient of the loss,
η is the learning rate.

Do not worry about the notation.

The idea is much simpler:

Measure which way is uphill, then take a step in the opposite direction.

Why is it called descent?

Because the objective is to:

descend the loss landscape.

Conceptually:

HIGH LOSS

↓

LOWER LOSS

↓

LOWER LOSS

↓

LOW LOSS.

Each parameter update attempts to move downhill.

From derivative to gradient

A model with one parameter needs:

one derivative.

But real machine-learning models often contain many parameters.

Suppose we have:

θ₁, θ₂, θ₃, θ₄.

We want to know:

What happens to the loss if each parameter changes slightly?

We calculate a derivative for each parameter.

Together, these derivatives form the:

gradient.

The gradient is a vector

Conceptually:

GRADIENT

[slope for parameter 1, slope for parameter 2, slope for parameter 3, ...].

For example:

∇L = [2.0, -0.5, 3.1].

This tells us:

parameter 1 should move downward,
parameter 2 should move upward,
parameter 3 should move downward.

The gradient tells us how the loss responds locally to changes in every parameter.

Gradient descent updates them together

Suppose our parameters are:

θ = [10, 5, -2].

The gradient is:

∇L = [2, -1, 4].

The optimiser uses this information to adjust all three parameters.

Conceptually:

θ₁ ↓

θ₂ ↑

θ₃ ↓.

After the update, the model makes new predictions.

Then we calculate the loss again.

The learning loop becomes clearer

We can now describe training more precisely:

TRAINING EXAMPLES

↓

MAKE PREDICTIONS

↓

CALCULATE LOSS

↓

CALCULATE GRADIENT

↓

UPDATE PARAMETERS

↓

MAKE NEW PREDICTIONS

↓

CALCULATE NEW LOSS

↓

REPEAT.

This is the basic machinery behind training many machine-learning models.

A simple regression example

Suppose our model is:

Predicted demand = a + b × temperature.

The parameters are:

and:

Initially:

a = 20

b = 0.

The predictions are poor.

The loss is high.

Gradient descent calculates:

How would the loss change if a changed slightly?

and:

How would the loss change if b changed slightly?

Then it adjusts both.

After many updates

Perhaps the parameters evolve:

a = 20, b = 0

↓

a = 25, b = -0.1

↓

a = 31, b = -0.3

↓

a = 38, b = -0.5

↓

a = 44, b = -0.7.

The model gradually moves towards parameters that better fit the training data.

The learning rate

How large should each step be?

This is controlled by the learning rate.

Often represented as:

η.

Conceptually:

PARAMETER UPDATE

LEARNING RATE × GRADIENT.

The learning rate determines how aggressively the optimiser responds to the slope.

A very small learning rate

Suppose we take tiny steps.

The optimiser may move:

0.0001

at a time.

This can be stable.

But learning may be extremely slow.

Imagine descending a mountain by moving:

one centimetre per step.

You are heading in the right direction.

You may simply take forever to arrive.

A very large learning rate

Now imagine taking enormous steps.

You might approach the valley:

LEFT SIDE

↓

jump

↓

RIGHT SIDE

↓

jump

↓

LEFT SIDE AGAIN.

Instead of settling near the bottom, you repeatedly overshoot it.

Too large can become unstable

Suppose the optimum parameter is near:

Gradient descent might move:

-2

↓

-5

↓

-20.

The updates become increasingly unstable.

Loss may increase rather than decrease.

The learning rate creates a trade-off

Too small

slow learning,
lots of computation.

Too large

overshooting,
oscillation,
instability.

Appropriate

meaningful progress,
reasonably stable convergence.

Choosing the learning rate is therefore an important part of model training.

The learning rate is a hyperparameter

Recall the distinction:

PARAMETERS

are learned from data.

HYPERPARAMETERS

control how learning happens.

The learning rate is a hyperparameter.

The optimiser uses it to determine how quickly parameters should change.

Learning rates can change during training

The learning rate does not have to remain constant.

We might begin with:

larger steps

when far from a good solution.

Then use:

smaller steps

later.

Conceptually:

EARLY TRAINING

large steps

↓

LATER TRAINING

smaller steps.

This is called a learning-rate schedule.

Why reduce the learning rate?

Imagine approaching the bottom of a valley.

Large steps helped you travel quickly when you were far away.

Near the bottom, those same steps may cause you to bounce around.

Smaller steps allow finer adjustment.

Gradient descent needs a starting point

Before training begins, the model needs initial parameter values.

For a simple regression, the starting point may not matter much.

For a neural network with millions or billions of parameters, initialisation matters considerably.

Parameters are commonly initialised using carefully chosen random values.

Why random?

Suppose every neuron in a neural network begins with identical parameters.

They may receive identical signals and make identical updates.

They would continue behaving identically.

Random initialisation helps break this symmetry.

Different parts of the network can begin learning different patterns.

Different starts can produce different models

Suppose two models have:

identical architecture,
identical training data,
identical loss function.

But their parameters begin at different random values.

They may follow different paths through the loss landscape.

Eventually they may reach different parameter configurations.

Both may perform well.

There may not be one unique answer

In a complex neural network, many different parameter configurations may produce similar performance.

So training is not necessarily trying to discover:

THE ONE TRUE SET OF PARAMETERS.

It is trying to find:

a useful low-loss configuration.

The loss landscape can be complicated

Our simple valley analogy is useful.

But real neural-network loss landscapes can contain:

valleys,
ridges,
flat regions,
saddle points,
many dimensions.

And instead of:

two or three parameters,

there may be:

billions.

We cannot visualise this directly.

A billion-dimensional landscape

Suppose a model has:

100 billion parameters.

Each parameter is one dimension of parameter space.

Training occurs in a:

100-billion-dimensional optimisation problem.

That sounds almost impossible.

Yet gradient-based optimisation works remarkably well in practice.

This is one of the extraordinary practical discoveries behind modern deep learning.

We do not search every possibility

Imagine trying every possible combination of:

100 billion parameter values.

That is completely infeasible.

Gradient descent avoids exhaustive search.

Instead, it uses local information:

Given where I am now, which direction appears to reduce loss?

That makes optimisation tractable.

Local information can guide enormous systems

This is a powerful general principle.

The optimiser does not need a complete map of parameter space.

It repeatedly uses:

local gradient information

to improve the current solution.

Conceptually:

CURRENT STATE

↓

LOCAL INFORMATION

↓

UPDATE

↓

NEW STATE.

This should feel familiar from our earlier discussion of feedback systems.

Gradient descent is iterative

Training does not usually solve the entire optimisation problem in one calculation.

Instead:

STEP 1

↓

STEP 2

↓

STEP 3

↓

...

↓

STEP 1,000,000.

Each step changes the model slightly.

Learning emerges through repeated updates.

The model is a dynamical system during training

At training step:

the model has parameters:

θₜ.

After an update:

θₜ₊₁.

Conceptually:

θₜ

GRADIENT INFORMATION

↓

UPDATE RULE

↓

θₜ₊₁.

The training process evolves through a sequence of states.

Gradient descent is feedback

There is a useful connection to control systems.

A feedback controller might:

OBSERVE STATE

↓

COMPARE WITH DESIRED STATE

↓

CALCULATE ERROR

↓

TAKE ACTION

↓

NEW STATE.

Gradient descent does something structurally similar:

OBSERVE MODEL PERFORMANCE

↓

CALCULATE LOSS

↓

CALCULATE GRADIENT

↓

UPDATE PARAMETERS

↓

NEW MODEL.

The model is repeatedly corrected using feedback.

The target is not necessarily zero loss

Suppose real-world data contains:

measurement noise,
randomness,
missing variables.

Perfect prediction may be impossible.

So the goal is not necessarily:

LOSS = 0.

It is:

Find parameters that produce useful generalisation and sufficiently low loss.

Zero training loss can actually be suspicious

Suppose a highly flexible model achieves:

zero training error.

That may mean it learned the underlying relationship extremely well.

Or it may mean:

it memorised the training data.

We still need validation and test data to determine whether the learned parameters generalise.

Full-batch gradient descent

One way to calculate the gradient is to use:

every training example.

Suppose we have:

1 million examples.

We calculate predictions for all of them.

Then:

overall loss

↓

gradient

↓

one parameter update.

This is batch gradient descent.

The problem with using everything

For enormous datasets, calculating the gradient across every example before each update can be expensive.

Imagine having:

1 trillion training tokens.

Waiting to process the entire dataset before making one update would be impractical.

So modern machine learning usually uses smaller batches.

Stochastic gradient descent

At the other extreme, we could use:

one example

to estimate the gradient.

Then:

EXAMPLE

↓

LOSS

↓

GRADIENT

↓

UPDATE.

This is stochastic gradient descent, or SGD.

Why "stochastic"?

Because the update now depends on whichever example happens to be selected.

Different examples may suggest slightly different directions.

So the gradient estimate contains randomness.

Mini-batch gradient descent

In practice, modern machine learning commonly uses a compromise:

mini-batches.

For example:

256 training examples

↓

PREDICTIONS

↓

AVERAGE LOSS

↓

GRADIENT

↓

PARAMETER UPDATE.

Then the next 256 examples.

Mini-batches provide an estimate

The gradient calculated from a mini-batch is not exactly the same as the gradient over the entire dataset.

It is an estimate.

But if the batch is reasonably representative, it can provide a useful direction.

This makes training much more computationally practical.

Noisy gradients can actually help

The randomness introduced by mini-batches is not necessarily bad.

A perfectly smooth descent might become stuck in an awkward region.

Some noise can help the optimiser move through complex landscapes.

Again:

Imperfection is not automatically a defect.

The important question is whether the uncertainty is manageable and useful.

Batch size matters

Suppose:

batch size = 1.

Updates can be very noisy.

Suppose:

batch size = entire dataset.

Updates become much more stable but computationally expensive.

Mini-batch size creates another trade-off between:

computational efficiency,
memory requirements,
gradient noise,
training behaviour.

Hardware affects batch size

Large batches require more memory.

Modern GPUs and AI accelerators are designed to process many operations in parallel.

So batch size is not merely a statistical choice.

It also depends on:

physical hardware.

This is another connection between machine-learning mathematics and the machines that actually run it.

An epoch revisited

Suppose we have:

10,000 training examples

and:

batch size = 100.

Then one epoch contains approximately:

100 parameter updates.

After those 100 updates, the model has processed every training example once.

Then the next epoch begins.

Training can involve enormous numbers of updates

A modern model may perform:

thousands

or:

millions

of optimisation steps.

Each step may involve:

many examples,
billions of parameters,
huge numbers of arithmetic operations.

This is why training modern AI requires enormous computation.

How do we calculate all those gradients?

Here we encounter another crucial idea:

backpropagation.

A neural network may contain many layers.

The final loss depends indirectly on parameters throughout the network.

We need to calculate:

How did each parameter contribute to the final loss?

Backpropagation provides an efficient way to calculate these gradients.

Forward pass

First, data moves through the model.

Conceptually:

INPUT

↓

LAYER 1

↓

LAYER 2

↓

LAYER 3

↓

OUTPUT

↓

LOSS.

This is called the forward pass.

Backward pass

Then information about the loss moves backwards through the computational graph.

Conceptually:

LOSS

↓

HOW MUCH DID LAYER 3 CONTRIBUTE?

↓

HOW MUCH DID LAYER 2 CONTRIBUTE?

↓

HOW MUCH DID LAYER 1 CONTRIBUTE?

↓

GRADIENT FOR EACH PARAMETER.

This is the backward pass.

The chain rule makes this possible

Backpropagation relies heavily on the chain rule from calculus.

If:

A affects B

and:

B affects C

and:

C affects loss,

the chain rule allows us to calculate:

How does changing A ultimately affect the loss?

This allows gradients to flow through many layers of computation.

Backpropagation and gradient descent are different

These terms are often confused.

Backpropagation

calculates:

the gradients.

Gradient descent

uses those gradients to:

update the parameters.

So:

BACKPROPAGATION

↓

GRADIENT

↓

GRADIENT DESCENT / OPTIMISER

↓

PARAMETER UPDATE.

They work together, but they are not the same thing.

A neural-network training step

A single training step might look like:

1. Select mini-batch

↓

2. Forward pass

↓

3. Calculate loss

↓

4. Backpropagate

↓

5. Calculate gradients

↓

6. Update parameters

↓

7. Repeat.

This loop may execute millions of times.

Optimisers improve on basic gradient descent

Basic gradient descent says:

Move opposite the gradient.

Modern machine learning often uses more sophisticated optimisers.

Examples include:

SGD with momentum,
RMSProp,
Adam,
AdamW.

These still use gradients.

But they modify how parameter updates are calculated.

Momentum

Imagine rolling a ball downhill.

It does not respond only to the slope at this exact instant.

It also has:

momentum from previous movement.

Optimisation algorithms can use a similar idea.

Previous gradients influence the current update.

This can help:

smooth noisy movement,
accelerate progress,
reduce oscillation.

Adam

Adam is one of the most widely used optimisers in deep learning.

Conceptually, it adapts parameter updates using information about:

recent gradients,
recent squared gradients.

Different parameters can effectively receive differently scaled updates.

We do not need the detailed equations here.

The important point is:

Modern optimisers build on the basic gradient-descent idea rather than replacing it entirely.

Different parameters may need different step sizes

Suppose one parameter has gradients around:

0.00001.

Another has gradients around:

1,000.

Using exactly the same effective update scale for both may be inefficient.

Adaptive optimisers attempt to handle these differences.

Flat regions

Suppose the loss landscape becomes nearly flat.

The gradient may become:

very small.

Then parameter updates become tiny.

Training can slow dramatically.

This is one optimisation challenge.

Steep regions

Elsewhere, gradients may become extremely large.

Then parameter updates can become enormous.

Training may become unstable.

This is related to the problem of:

exploding gradients.

Vanishing gradients

In deep networks, gradients can sometimes become extremely small as they propagate backwards through many layers.

Early layers then receive almost no useful learning signal.

This is called the:

vanishing-gradient problem.

It historically made very deep networks difficult to train.

Exploding gradients

The opposite can also occur.

Gradients become enormous as they move backwards.

Parameter updates can become unstable.

This is called:

exploding gradients.

Techniques such as gradient clipping can help control it.

Architecture changed partly because optimisation mattered

Modern deep-learning architectures were not designed only around:

what functions can they represent?

They were also influenced by:

Can we actually train them effectively using gradient-based optimisation?

This led to innovations such as:

improved activation functions,
residual connections,
normalisation methods.

A theoretically powerful model is not useful if we cannot train it.

ReLU

One influential activation function is:

Rectified Linear Unit, or ReLU.

Conceptually:

negative input → 0

positive input → pass it through.

ReLU helped make deep networks easier to optimise compared with some earlier activation functions.

Residual connections

Very deep networks can make learning difficult.

Residual connections allow information and gradients to bypass some layers.

Conceptually:

INPUT

↓

TRANSFORMATION

SHORTCUT CONNECTION

↓

OUTPUT.

These became central to many deep architectures.

Transformers still use gradient descent

Modern language models may appear radically different from simple regression.

They contain:

embeddings,
attention,
many transformer layers,
billions of parameters.

But the basic training loop remains recognisable:

TEXT

↓

PREDICT TOKENS

↓

CALCULATE LOSS

↓

BACKPROPAGATE

↓

CALCULATE GRADIENTS

↓

UPDATE BILLIONS OF PARAMETERS.

The scale changed enormously.

The core learning principle remained surprisingly simple.

One token contributes a learning signal

Suppose the text is:

The sky is blue.

The model receives:

The sky is

and predicts:

blue → 20%

The actual token is:

blue.

The loss says:

You should have assigned more probability to "blue".

Backpropagation determines how parameters throughout the network contributed to that probability.

The optimiser changes them slightly.

Repeat across enormous datasets

Now repeat that process across:

billions or trillions of tokens.

Each token contributes a tiny learning signal.

Across enormous numbers of updates, the model's parameters gradually encode increasingly rich statistical structure.

Intelligence emerges from accumulated updates

No single training example teaches a language model:

language.

No single gradient update creates:

reasoning.

Capabilities emerge from enormous numbers of small parameter changes interacting across a huge network.

Conceptually:

TINY UPDATE

...

↓

LARGE-SCALE LEARNED BEHAVIOUR.

Gradient descent does not understand the objective

This is important.

Gradient descent does not know:

what a cat is,
what electricity is,
what truth is,
what fairness is.

It simply receives:

a mathematical loss

and attempts to find parameter changes that reduce it.

The optimiser is indifferent to meaning

Suppose our loss rewards the wrong behaviour.

Gradient descent will happily optimise it.

This returns us to the central lesson from loss functions:

Powerful optimisation makes defining the right objective more important, not less.

Optimisation pressure

The more effectively a system can reduce its loss, the more strongly it can exploit whatever structure exists in that objective.

If the objective is well designed:

excellent optimisation can be extremely useful.

If the objective is badly specified:

excellent optimisation can amplify the mistake.

Gradient descent finds correlations useful for the objective

Suppose snow strongly predicts:

wolf

in the training data.

If using snow reduces classification loss, gradient descent has no inherent reason to reject it.

From the optimiser's perspective:

snow is useful information.

Humans may know it is a shortcut.

The optimiser knows only that it reduces loss.

Better data can change the gradient

Now add:

dogs in snow,
wolves without snow.

Suddenly relying on snow produces more errors.

The gradients change.

The optimiser begins adjusting parameters away from that shortcut.

So:

DATA

shapes:

LOSS

which shapes:

GRADIENTS

which shape:

PARAMETERS.

The entire learning chain

We can now write the process more completely:

TRAINING DATA

↓

MODEL PARAMETERS

↓

PREDICTIONS

↓

LOSS FUNCTION

↓

LOSS

↓

GRADIENT

↓

OPTIMISER

↓

PARAMETER UPDATE

↓

NEW MODEL PARAMETERS.

Then repeat.

Gradient descent learns locally

At each step, gradient descent asks something like:

What small change would improve things from where I am now?

It does not directly ask:

What is the globally perfect model?

This distinction matters.

Gradient descent is fundamentally a:

local iterative process.

Local improvement can produce global capability

This is one of the striking features of machine learning.

Each update may be tiny and local.

Yet after enormous numbers of updates, the resulting model can display sophisticated global behaviour.

Simple repeated rules can create complex systems.

There is a connection to evolution

The analogy should not be pushed too far, but there is an interesting similarity.

Evolution does not begin with a complete blueprint of the final organism.

Instead:

variation

↓

selection pressure

↓

repeated adaptation.

Gradient-based learning similarly uses repeated local adjustments under an objective.

But the mechanisms are fundamentally different.

There is also a connection to feedback control

Gradient descent is closer mathematically to optimisation than biological evolution.

But structurally:

CURRENT CONDITION

↓

ERROR SIGNAL

↓

CORRECTIVE ACTION

↓

NEW CONDITION

is a recurring pattern across:

control,
learning,
adaptation.

This feedback structure will appear throughout the course.

Gradient descent and time

Training itself takes place through time.

At step:

we have:

θₜ.

Then:

θₜ₊₁ = θₜ - update.

So the parameters form a sequence:

θ₀ → θ₁ → θ₂ → θ₃ → ...

The model has a learning trajectory.

The past shapes the current model

Every previous gradient update contributed to the current parameters.

So:

CURRENT PARAMETERS

are partly the accumulated consequence of:

PAST TRAINING EXAMPLES.

This gives us another connection to the course's central timeline.

The model carries history forward

Conceptually:

PAST DATA

↓

PAST LOSSES

↓

PAST GRADIENTS

↓

PAST PARAMETER UPDATES

↓

CURRENT PARAMETERS

↓

CURRENT PREDICTION.

The current model is shaped by its training history.

But old examples are not stored as individual gradients

After training, we do not usually keep a list saying:

Example 1 changed parameter 7 by this amount

Example 2 changed parameter 7 by that amount.

The effects become entangled through repeated updates.

The final parameters reflect the accumulated training process.

This makes attribution difficult

Suppose a language model produces a particular sentence.

Which training example caused it?

Usually there is no simple answer.

The behaviour may emerge from patterns distributed across:

many examples,
many gradients,
many parameters.

This is one reason interpreting large models is difficult.

Gradient descent is not guaranteed to find truth

It finds parameter values that reduce the specified loss on the available data.

Those are not the same thing.

Conceptually:

LOW TRAINING LOSS

does not automatically imply:

TRUE MODEL OF REALITY.

The model might:

overfit,
exploit shortcuts,
learn spurious correlations,
inherit biased data.

Optimisation quality and epistemic quality are different.

Gradient descent is not guaranteed to find the global optimum

For complicated models, the optimiser may not find the absolute lowest possible loss.

But in practice, that may not matter.

A solution can be extremely useful without being mathematically globally optimal.

What matters is often:

good enough loss

good generalisation.

Sometimes exact optimisation is unnecessary

Suppose two parameter configurations achieve:

99.1%

and:

99.2%

performance.

Finding the second might require ten times more computation.

Depending on the application, the first may be entirely sufficient.

Optimisation itself has costs.

Compute is a scarce resource

Every gradient step requires computation.

That computation requires:

processors,
memory,
electricity,
cooling,
time.

So the abstract mathematical instruction:

take another optimisation step

has a physical cost.

Training has an economic stopping point

Suppose additional training costs:

€1 million

and improves model performance by:

0.001%.

Is that worthwhile?

Perhaps.

Perhaps not.

The answer depends on the value of the improvement.

So even model training becomes an optimisation problem involving:

performance versus resources.

Distributed gradient descent

Large models may be too large to train on one processor.

Training can be distributed across:

many GPUs,
many servers,
entire data centres.

Different machines process parts of the training workload.

Their results must then be coordinated.

Data parallelism

One approach is:

GPU 1 → batch A

GPU 2 → batch B

GPU 3 → batch C

GPU 4 → batch D.

Each calculates gradients.

Those gradients are combined.

Then the model parameters are updated.

This is data parallelism.

Model parallelism

Sometimes the model itself is too large to fit on one accelerator.

Different parts of the model are placed on different devices.

Now:

MODEL

is physically distributed across machines.

Training requires enormous communication between them.

Gradient descent becomes an infrastructure problem

At small scale:

gradient descent

looks like a calculus problem.

At enormous scale, it becomes:

a networking problem,
a memory problem,
a semiconductor problem,
an electricity problem,
a cooling problem.

This is why the physical infrastructure of AI matters.

A tiny equation can require a giant machine

The update rule may be conceptually simple:

parameters ← parameters - learning rate × gradient.

But applying it to:

hundreds of billions of parameters

across:

trillions of training examples

can require enormous industrial infrastructure.

The mathematics is simple.

The scale is not.

Gradient descent can be automated

Humans do not manually inspect billions of parameters.

The entire process is automated.

The computer:

makes predictions,
calculates loss,
calculates gradients,
updates parameters.

Again and again.

This automation is what makes large-scale machine learning possible.

But humans design the learning system

Humans still make choices about:

training data,
architecture,
loss function,
optimiser,
learning rate,
batch size,
stopping criteria.

The optimiser automates parameter search.

It does not remove system design.

Gradient descent is a search process

A useful way to think about machine learning is:

Training is searching parameter space for a model that performs well.

Gradient descent makes that search efficient by using local slope information.

Instead of:

TRY RANDOM MODEL

↓

TRY ANOTHER RANDOM MODEL

↓

TRY ANOTHER

it uses feedback from the loss to guide the next move.

The gradient contains directional information

Loss tells us:

How bad is the current model?

Gradient tells us:

Which direction should we change it?

This distinction is important.

LOSS = evaluation

GRADIENT = direction for improvement.

Loss without gradient

Suppose someone tells you:

Your score is 37.

That tells you how well you did.

But not:

What should you change to improve it?

The gradient adds information about how changes affect the score.

It turns evaluation into a direction for learning.

Gradient descent closes the learning loop

We began with:

EXAMPLES

↓

PREDICTIONS

↓

ERROR.

Now we can complete it:

EXAMPLES

↓

PREDICTIONS

↓

LOSS

↓

GRADIENT

↓

PARAMETER UPDATE

↓

NEW MODEL

↓

NEW PREDICTIONS.

This is a genuine feedback loop.

A useful gradient-descent checklist

When examining a learning system, ask:

What parameters are being optimised?
What loss function produces the learning signal?
How are gradients calculated?
What optimiser is used?
What is the learning rate?
Does the learning rate change?
What is the batch size?
How noisy are the gradient estimates?
How are parameters initialised?
How many optimisation steps are performed?
How is training stopped?
Is validation loss being monitored?
Are gradients vanishing or exploding?
Is the optimiser finding useful solutions?
How much computation does optimisation require?
Is lower training loss producing better generalisation?
Is the model optimising the objective we actually care about?

Gradient descent is simple in principle.

Its behaviour depends on the entire learning system around it.

The central idea

A loss function tells the machine:

How wrong am I?

Gradient descent asks:

Which way should I change to become less wrong?

Then:

CURRENT PARAMETERS

↓

CALCULATE LOSS

↓

CALCULATE GRADIENT

↓

MOVE DOWNHILL

↓

NEW PARAMETERS

↓

REPEAT.

At enormous scale, this process can adjust billions of parameters using trillions of examples.

That repeated feedback loop is one of the central mechanisms behind modern machine learning.

But reducing training loss is not the ultimate objective.

A model can become extraordinarily good at the examples it has seen while becoming worse at the examples it has not.

That brings us to one of the most important problems in machine learning:

When has the model learned the underlying pattern, and when has it simply learned the training data too well?

That is the problem of overfitting.