Enleashed

Model complexity
So far, we have asked:
What should a model learn?
We have looked at:
inputs and outputs,
regression,
classification,
training data,
parameters,
loss functions,
gradient descent,
overfitting,
underfitting,
generalisation,
feature engineering,
feature importance.
But there is another fundamental question:
How complicated should the model itself be?
A model needs enough complexity to capture the important structure in the data.
But too much complexity can allow it to learn:
noise,
accidents,
irrelevant details.
This creates one of the central tensions in machine learning:
A model should be complex enough to learn the pattern, but not so complex that it memorises the past.
What do we mean by complexity?

Model complexity describes roughly:
How flexible a model is in the relationships it can represent.
A very simple model can represent only a small range of relationships.
A highly complex model can represent many different relationships.
For example, suppose we want to predict:
from:
A simple model might assume:
a straight line.
A more complex model might allow:
a curve.
An even more complex model might allow:
a highly irregular function with many bends.
Increasing complexity gives the model more freedom.
A simple model

Suppose we fit:
house price
from:
floor area.
A simple linear model might say:
price = a + b × floor area

This assumes:

> **Each additional square metre changes predicted price by roughly the same amount.**

That is a strong assumption.

The model has limited flexibility.

---

### A more complex model

We might instead allow:

```text
price =
a
+ b₁ × floor area
+ b₂ × floor area²

Now the relationship can curve.

Perhaps additional floor area becomes more valuable for larger houses.

The model can represent more possibilities.

---

### More complexity again

We could add:

- floor area³,
- floor area⁴,
- interactions with location,
- interactions with building type,
- many other features.

Eventually the model could represent extremely complicated relationships.

Its **capacity** has increased.

---

### Model capacity

Model complexity is closely related to **model capacity**.

Capacity describes:

> **How rich a family of functions the model can represent.**

A low-capacity model can represent:

**fewer possible relationships**.

A high-capacity model can represent:

**many more possible relationships**.

Conceptually:

LOW CAPACITY

↓

FEWER POSSIBLE FUNCTIONS

↓

STRONGER ASSUMPTIONS

whereas:

HIGH CAPACITY

↓

MORE POSSIBLE FUNCTIONS

↓

GREATER FLEXIBILITY

The more capacity a model has, the more shapes and relationships it can potentially learn from the data.

---

### Complexity is not simply the number of parameters

The number of parameters is one useful indicator of model complexity.

For example:

**Model A → 2 parameters**

**Model B → 200 parameters**

**Model C → 2 billion parameters**.

But parameter count is not the complete definition of complexity.

Complexity also depends on:

- architecture,
- constraints,
- regularisation,
- feature representation,
- training procedure.

Two models with the same number of parameters can behave very differently.

---

### Complexity gives a model freedom

Imagine fitting observations on a graph.

A straight line has limited freedom.

It cannot bend around every observation.

A highly flexible curve can.

That sounds useful.

If the real relationship is complicated, we want enough flexibility to represent it.

But there is a danger.

The model may begin fitting:

**the particular observations we happened to collect**

rather than:

**the underlying relationship that generated them**.

---

### The simplest model may underfit

Suppose the true relationship is curved.

We fit:

**a straight line**.

The model cannot capture the pattern.

It performs badly on:

- training data,
- new data.

This is:

**underfitting**.

The model does not have enough capacity to represent the underlying structure.

---

### Increasing complexity can help

Now allow the model to fit a curve.

Training error falls.

Validation error also falls.

Excellent.

The additional complexity has allowed the model to capture more of the useful structure.

Conceptually:

TOO SIMPLE

↓

INCREASE COMPLEXITY

↓

CAPTURE MORE OF THE REAL PATTERN

↓

BETTER GENERALISATION

But this improvement does not necessarily continue forever.

---

### Too much complexity can overfit

Keep increasing model flexibility.

Eventually the model may begin fitting:

- random noise,
- measurement errors,
- unusual observations,
- accidental relationships in the training sample.

Training performance can continue improving.

But performance on unseen data may begin getting worse.

This is:

**overfitting**.

---

### Training error and validation error behave differently

As model complexity increases:

**training error**

will often continue to fall.

After all, a more flexible model has more ways to fit the training observations.

But:

**validation error**

may behave differently.

At first it falls as the model learns useful structure.

Eventually it may rise as the model begins learning details that do not generalise.

Conceptually:

```text
Validation
error

high  \            /
       \          /
        \________/
             ↑
       useful complexity

       Model complexity →

The best model is therefore not necessarily the model with the lowest training error.

---

### Training error alone cannot choose complexity

Suppose:

**Model A**

has training error:

**10**.

Model B:

**5**.

Model C:

**0.001**.

It might seem obvious that Model C is best.

But now look at validation error:

```text
Model      Training error      Validation error

A               10                   12
B                5                    7
C                0.001               25
**the particular examples it happened to see**.

High variance is associated with:

**overfitting**.

---

### Stable but wrong versus unstable

A high-bias model can be:

**stable but consistently wrong**.

A high-variance model can be:

**accurate on one training sample but unstable across samples**.

Ideally, we want a model that captures the important structure while remaining reasonably stable.

---

### The trade-off

Conceptually:

SIMPLE MODEL

↓

HIGHER BIAS

LOWER VARIANCE

whereas:

COMPLEX MODEL

↓

LOWER BIAS

HIGHER POTENTIAL VARIANCE

The challenge is to find enough flexibility to capture useful structure without becoming excessively sensitive to noise.

---

### Complexity interacts with the amount of data

A model that is far too complex for:

**1,000 observations**

may work very well with:

**100 million observations**.

Why?

Because more data provides more evidence with which to constrain the model.

With very little data, a flexible model may find many different relationships that fit the observations.

With much more data, many of those accidental explanations become harder to sustain.

So:

> **Appropriate model complexity depends partly on how much evidence we have.**

---

### More data can support more complex models

Conceptually:

SMALL DATASET

↓

LIMITED EVIDENCE

↓

COMPLEX MODEL CAN EASILY FIT ACCIDENTAL PATTERNS

whereas:

LARGE DATASET

↓

MORE EVIDENCE

↓

MORE COMPLEX RELATIONSHIPS CAN BE LEARNED RELIABLY

This is one reason the growth of very large datasets helped make modern deep learning possible.

---

### But more data is not automatically better

Suppose we have:

**one billion observations**.

But they contain:

- systematic measurement errors,
- biased sampling,
- incorrect labels.

A highly capable model may simply become extremely good at learning those problems.

Complexity increases our ability to discover patterns.

It does not guarantee that the patterns are useful or meaningful.

---

### Complex problems may genuinely require complex models

Consider image recognition.

An image might contain:

**millions of pixel values**.

The relationship between those raw pixels and concepts such as:

- dog,
- pedestrian,
- tumour,
- traffic light

can be extraordinarily complicated.

A simple linear model may not have enough capacity.

Deep neural networks can learn increasingly sophisticated representations such as:

PIXELS

↓

EDGES

↓

TEXTURES

↓

SHAPES

↓

OBJECTS

Here, complexity is useful because the underlying prediction problem itself is complex.

---

### But complex models can also learn complex nonsense

The same flexibility that allows a model to discover subtle useful patterns allows it to discover subtle useless ones.

Suppose most images of wolves in the training dataset contain:

**snow**.

The model might learn:

> **Snow is strong evidence that the image contains a wolf.**

That relationship genuinely exists in the training data.

But it is not the relationship we intended the model to learn.

---

### Complexity increases the number of possible solutions

A simple model has relatively few ways to fit the data.

A highly complex model may have enormous numbers of possible solutions.

Many different parameter combinations may achieve:

**very low training loss**.

This creates another important question:

> **Which of all those possible solutions will the learning algorithm actually find?**

The answer depends on:

- architecture,
- initialisation,
- optimisation,
- regularisation,
- training data.

---

### Regularisation

One way to control effective model complexity is:

**regularisation**.

Regularisation introduces a preference for solutions that are less extreme or less complex.

Conceptually:

FIT THE DATA WELL

+

AVOID UNNECESSARY COMPLEXITY

Rather than simply asking:

> Which model minimises prediction error on the training data?

we add another consideration:

> Can we achieve similar predictive performance with a less extreme solution?

---

### Adding complexity to the loss function

Recall that the loss function tells the model:

> **What counts as being wrong?**

We can modify the objective so that complexity itself carries a cost.

Conceptually:

```text
Total loss
=
prediction loss
+
complexity penalty

Now the model has two pressures:

**fit the observations**

and:

**avoid unnecessary complexity**.

---

### Regularisation strength

Suppose:

**λ**

controls the strength of the complexity penalty.

If:

**λ = 0**

there is no regularisation penalty.

As λ increases:

**complexity is penalised more strongly**.

Too little regularisation may allow:

**overfitting**.

Too much regularisation may produce:

**underfitting**.

Again, we have a trade-off.

---

### L1 and L2 regularisation

Two common approaches are:

**L1 regularisation**

and:

**L2 regularisation**.

The mathematical details are not essential here.

Conceptually:

**L2 regularisation**

tends to encourage parameters to remain smaller.

**L1 regularisation**

can encourage some parameters to become exactly:

**zero**.

That can effectively remove some features from the model.

---

### Regularisation can therefore affect feature importance

Suppose a model contains:

**1,000 features**.

Without regularisation, it may assign small amounts of predictive weight to hundreds of them.

With stronger regularisation, it may concentrate more strongly on a smaller number of useful features.

So:

**feature selection**

and:

**model complexity**

are closely connected.

---

### Early stopping is also complexity control

Recall what happens during training.

At first:

**training loss falls**

and:

**validation loss falls**.

Eventually:

**training loss continues falling**

while:

**validation loss begins rising**.

We can stop training near the point where validation performance is best.

This is called:

**early stopping**.

---

### Why early stopping can work

A model may first learn:

**large, persistent patterns**.

With continued training it may begin learning:

**smaller peculiarities of the training dataset**.

Stopping earlier limits how far that process goes.

So effective model complexity depends not only on:

**the architecture**

but also on:

**how it is trained**.

---

### Data augmentation

Another way to discourage overfitting is to expose the model to meaningful variations of the training examples.

For images, we might:

- crop,
- rotate,
- flip,
- slightly alter brightness.

Instead of seeing one exact image repeatedly, the model sees many plausible versions.

This encourages it to learn:

**stable underlying structure**

rather than:

**one exact arrangement of pixels**.

---

### Data augmentation contains assumptions

Suppose we horizontally flip a photograph of a dog.

It is still a dog.

That transformation probably preserves the label.

But not every transformation preserves meaning.

Rotate a handwritten:

**6**

far enough and it may resemble:

**9**.

So augmentation itself encodes assumptions about:

> **Which changes should not alter the underlying meaning?**

Once again, human judgement enters the learning system.

---

### Simpler does not automatically mean better

There is a long tradition in science of preferring simpler explanations.

This is related to **Occam's razor**:

> **When competing explanations account for the evidence equally well, prefer the simpler one.**

But this does not mean:

> Always choose the simplest model available.

Reality itself can be complicated.

A model that is simpler than the phenomenon it is trying to represent may simply be wrong.

---

### Complexity should earn its place

Suppose:

**Model A**

achieves:

**91.0% accuracy**.

It is:

- small,
- fast,
- interpretable.

Model B achieves:

**91.1% accuracy**.

But it requires:

- 1,000 times more computation,
- much more memory,
- a complicated deployment pipeline.

Is Model B really better?

Perhaps.

But the additional complexity needs to justify its cost.

---

### Prediction accuracy is not the only objective

Increasing model complexity can affect:

- training time,
- inference time,
- memory,
- electricity consumption,
- engineering effort,
- interpretability,
- reliability,
- monitoring difficulty.

The best predictive model is not automatically the best overall system.

---

### Complexity and interpretability

Consider:

**linear regression with five features**.

A human can inspect:

- coefficients,
- relationships,
- feature contributions.

Now consider:

**a neural network containing billions of parameters**.

Its internal representation is far harder to inspect directly.

Greater flexibility can come at the cost of:

**greater opacity**.

---

### Interpretability can itself have value

Suppose two medical models have almost identical predictive performance.

One allows clinicians to understand reasonably well:

**why predictions are being made**.

The other is almost completely opaque.

Depending on the application, the first may be preferable even if its raw predictive accuracy is slightly lower.

So:

> **The best model depends on what the service needs from the model.**

---

### Model complexity is not system complexity

There is another important distinction.

Suppose the final predictive model is:

**simple linear regression**.

But before the data reaches it, the system uses:

- 20 databases,
- 500 engineered features,
- several neural networks,
- external APIs.

The final mathematical model may be simple.

The overall system is not.

So we should distinguish:

**MODEL COMPLEXITY**

from:

**SYSTEM COMPLEXITY**.

---

### Complexity can move around the system

Suppose we use:

**linear regression**

but one of its inputs is:

**a 1,024-dimensional embedding produced by a giant neural network**.

Calling the whole system:

**simple**

would be misleading.

The complexity has simply moved into the representation stage.

---

### Better representations can allow simpler models

Recall feature engineering.

Suppose the true relationship is:

**y = x²**.

A linear model using only:

**x**

cannot represent it.

But create a feature:

**x²**.

Now a linear model can represent the relationship easily.

The model itself remains simple.

The useful complexity has moved into:

**the feature representation**.

---

### Deep learning often learns the representation

Traditional machine learning often relies heavily on humans designing useful features.

Deep learning increasingly asks the model to learn:

**the representation itself**.

For example:

RAW PIXELS

↓

LEARNED FEATURES

↓

HIGHER-LEVEL REPRESENTATIONS

↓

PREDICTION

That requires substantial model capacity.

---

### Hyperparameters control complexity

Many decisions affecting complexity are called:

**hyperparameters**.

Examples include:

- polynomial degree,
- tree depth,
- number of trees,
- number of neural-network layers,
- number of hidden units,
- regularisation strength.

These differ from ordinary model parameters.

---

### Parameters versus hyperparameters

Recall:

**parameters**

are learned from the training data.

Examples:

- regression coefficients,
- neural-network weights.

**Hyperparameters**

control aspects of the model structure or learning process.

Examples:

- learning rate,
- maximum tree depth,
- regularisation strength.

We often choose hyperparameters by comparing performance on validation data.

---

### Validation helps us choose complexity

Suppose we try polynomial models with degrees:

**1, 2, 3, 4, 5, 6**.

We train each on the training set.

Then compare validation error:

```text
Degree     Validation error

1              12.0
2               8.1
3               6.4
4               6.2
5               7.0
6               9.5
Degree 4 performs best on the validation data.

That gives us evidence that degree 4 provides an appropriate level of complexity for this particular problem.

---

### But we can overfit the validation set too

Suppose we try:

**10,000 different model configurations**

and repeatedly choose whichever performs best on the same validation dataset.

Eventually we may begin exploiting:

**peculiarities of the validation set**.

The validation set has effectively become part of the optimisation process.

This is why we preserve a final:

**test set**.

---

### The full workflow

Conceptually:

TRAINING SET

↓

LEARN PARAMETERS

↓

VALIDATION SET

↓

CHOOSE MODEL COMPLEXITY AND HYPERPARAMETERS

↓

TEST SET

↓

FINAL INDEPENDENT EVALUATION

The test set should ideally remain untouched until the important modelling choices have been made.

---

### Modern deep learning complicates the classical story

The traditional picture suggests:

TOO SIMPLE

↓

UNDERFITTING

↓

GOOD COMPLEXITY

↓

GOOD GENERALISATION

↓

TOO COMPLEX

↓

OVERFITTING

This remains a very useful intuition.

But modern deep learning has revealed that the relationship between complexity and generalisation can be more complicated.

---

### Overparameterisation

A model is **overparameterised** when it has more than enough parameters to fit the training data.

Modern neural networks can contain:

- millions,
- billions,
- even more parameters.

They may have enough capacity to memorise enormous datasets.

Yet they can still generalise surprisingly well.

This challenged some traditional intuitions about model complexity.

---

### Parameter count is not the whole story

Why can enormous models generalise?

There is no single answer.

Their behaviour depends on interactions between:

- dataset size,
- architecture,
- optimisation,
- regularisation,
- representation learning.

Even the optimisation algorithm can favour certain types of solutions over others.

So:

> **Model size alone does not determine generalisation.**

---

### Double descent

In some modern machine-learning systems, test error behaves in a surprising way.

As complexity increases:

**test error falls**

then:

**rises**

and then:

**falls again**.

This phenomenon is called:

**double descent**.

It shows that the classical simple U-shaped relationship between complexity and validation error is not universal.

---

### The deeper lesson

The lesson is not:

> Complexity no longer matters.

It is:

> **Complexity is more subtle than simply counting parameters.**

Generalisation emerges from the interaction between:

**MODEL**

+

**DATA**

+

**REPRESENTATION**

+

**OPTIMISATION**

+

**REGULARISATION**.

---

### Foundation models push complexity much further

Modern foundation models can contain:

**billions of parameters**.

They are trained using:

- enormous datasets,
- specialised hardware,
- huge amounts of computation.

Their scale allows them to learn extremely rich representations of:

- language,
- images,
- sound,
- code.

But this introduces another dimension of complexity.

---

### Complexity has a physical cost

A model is not merely:

**mathematics floating in the cloud**.

Training and running complex models requires physical infrastructure:

- processors,
- memory,
- networks,
- electricity,
- cooling,
- data centres.

So increasing model complexity can increase real-world resource consumption.

We will return to this later when we examine:

**the physical machine behind artificial intelligence**.

---

### Complexity becomes a resource-allocation question

Suppose increasing a model from:

**10 billion parameters**

to:

**100 billion parameters**

produces a modest improvement in performance.

But it requires vastly more:

- compute,
- electricity,
- hardware,
- capital.

Is the improvement worth it?

That is no longer purely a machine-learning question.

It is also:

**an optimisation problem**

and:

**a resource-allocation problem**.

---

### Model complexity should match the problem

There is little reason to use:

**a giant neural network**

for a relationship adequately captured by:

**linear regression**.

Likewise, forcing a tiny linear model onto:

**natural language understanding**

may be hopeless.

The appropriate complexity depends on the structure of the task.

---

### A useful principle

Instead of asking:

> **What is the most powerful model available?**

ask:

> **What level of complexity is required to solve this problem sufficiently well?**

This shifts the goal from:

**maximum model capability**

to:

**appropriate model capability**.

---

### "Sufficiently well" depends on consequences

Suppose a simple model predicts ordinary electricity demand almost as well as a vastly more complicated one.

The simple model may be sufficient.

But suppose we are detecting:

**pedestrians in front of an autonomous vehicle**.

A small improvement in rare-event detection could have enormous value.

The appropriate complexity therefore depends partly on:

**the consequences of error**.

---

### Model selection is a design problem

We are balancing:

- predictive performance,
- generalisation,
- interpretability,
- computational cost,
- latency,
- reliability,
- resource consumption,
- governance.

There is rarely one universally best model.

There is a model that is more appropriate for:

**a particular problem, service and set of objectives**.

---

### The central idea

Model complexity determines how much freedom a model has to represent patterns.

Too little complexity:

UNDERFITTING

↓

MODEL CANNOT CAPTURE IMPORTANT STRUCTURE

Increasing complexity:

BETTER REPRESENTATION

↓

BETTER GENERALISATION

But unconstrained complexity can produce:

OVERFITTING

↓

MODEL LEARNS ACCIDENTAL STRUCTURE

↓

POOR GENERALISATION

Modern machine learning makes the story more nuanced, because complexity cannot be understood from parameter count alone.

Instead, generalisation emerges from the interaction of:

**model + data + representation + optimisation + regularisation**.

The goal is therefore not:

> **Build the most complicated model possible.**

Nor is it:

> **Always choose the simplest model.**

It is:

> **Use enough complexity to capture the structure that matters, while controlling the freedom to learn structure that does not generalise.**

And there is a broader systems lesson:

> **Complexity should earn its place.**

A more complex model may make a better prediction.

But the improvement must ultimately be weighed against the additional:

- computation,
- resources,
- opacity,
- operational difficulty

required to achieve it.