Module 4 — Prediction: learning patterns from the past
Lesson 11 of 16
Underfitting
In the previous lesson, we saw what happens when a model learns the training data too closely.
That is overfitting.
But the opposite problem can also occur.
A model may fail to learn enough of the underlying structure in the data.
This is underfitting.
Underfitting happens when a model is too limited, too constrained, or insufficiently trained to capture important patterns in the data.
At its simplest:
TRAINING PERFORMANCE = POOR
and:
TEST PERFORMANCE = POOR
The model has not learned the problem well enough in the first place.
A simple analogy
Imagine a student preparing for a mathematics exam.
Instead of understanding the material, they learn only one rule:
Whenever you see numbers, add them together.
For:
2 + 3
this works.
For:
5 + 7
it works.
But ask:
7 × 4
and the student answers:
11.
Ask:
12 - 5
and they answer:
17.
The rule is simply too limited to represent the variety of problems being asked.
That is the intuition behind underfitting.
A model can be too simple for the world
Suppose the real relationship between two variables looks like:
a curve.
But we insist on fitting:
a straight line.
The line may capture the broad direction.
But it cannot capture the actual shape.
Conceptually:
REAL RELATIONSHIP
*
* *
* *
* *
* *
But our model can represent only:
*
*
*
*
*
No matter how much we adjust the line, it cannot reproduce the curved relationship.
The problem is not necessarily:
- insufficient optimisation,
- bad gradient descent,
- too little training.
The model itself may simply be incapable of representing the pattern.
### Model capacity
This introduces the idea of **model capacity**.
Model capacity describes, roughly, how rich or flexible a set of relationships a model can represent.
A model with low capacity can represent:
**a relatively limited range of functions**.
A model with high capacity can represent:
**a much richer range of functions**.
---
### Capacity is not the same as quality
A higher-capacity model is not automatically better.
More capacity means:
**more possibilities**.
That can help capture complicated relationships.
But it can also create more opportunities to:
**overfit**.
So we face a balance:
**TOO LITTLE CAPACITY**
↓
**UNDERFITTING**
**APPROPRIATE CAPACITY**
↓
**USEFUL GENERALISATION**
**TOO MUCH EFFECTIVE FLEXIBILITY**
↓
**RISK OF OVERFITTING**.
---
### A regression example
Suppose house price depends on:
- floor area,
- location,
- number of bedrooms,
- age,
- condition,
- energy efficiency,
- local amenities.
Now suppose our model uses only:
**floor area**.
It might learn:
> Larger houses tend to cost more.
That is useful.
But it ignores much of the structure determining price.
A:
**100 m² house in central Dublin**
and:
**100 m² house in a rural location**
may have very different prices.
If the model cannot account for that, its predictions will remain limited.
---
### Underfitting can come from missing features
Sometimes the model architecture is perfectly capable.
The problem is that we have not given it enough useful information.
Suppose we predict electricity demand using only:
**time of day**.
Demand also depends on:
- temperature,
- day of week,
- season,
- holidays,
- economic activity,
- behaviour.
The model may capture:
**morning and evening peaks**,
but miss substantial variation.
The problem may be the input representation rather than the learning algorithm.
---
### Missing information creates limits
Suppose two situations look identical according to the features available to the model.
But in reality they differ in an important way that was not measured.
The model cannot distinguish them.
From its perspective:
**INPUT A = INPUT B**.
If the correct outputs differ, the model has no information telling it why.
No amount of optimisation can recover information that is simply absent from the inputs.
---
### This connects back to measurement
Recall:
> **Data is created through systems of measurement.**
What we choose to measure determines what information is available to the model.
So underfitting can sometimes begin long before model training.
Conceptually:
**REALITY**
↓
**MEASUREMENT**
↓
**AVAILABLE FEATURES**
↓
**MODEL**.
If important aspects of reality never enter the dataset, the model may be fundamentally limited.
---
### Underfitting can come from the wrong model
Suppose the relationship is:
**non-linear**.
But we choose a model capable only of representing:
**linear relationships**.
Even with:
- perfect data,
- excellent optimisation,
- unlimited training time,
the model cannot represent the true relationship.
This is a problem of:
**model specification**.
---
### Underfitting can come from too much regularisation
Recall that regularisation helps prevent overfitting.
We might penalise complexity:
**TOTAL LOSS**
=
**PREDICTION LOSS**
+
**COMPLEXITY PENALTY**.
But suppose the complexity penalty is extremely strong.
The model may become unwilling to learn relationships that are genuinely present.
We have prevented overfitting so aggressively that we create:
**underfitting**.
---
### Regularisation is therefore a balance
#### Too little regularisation
The model may learn noise.
**Risk: overfitting.**
#### Too much regularisation
The model may suppress useful structure.
**Risk: underfitting.**
The correct amount depends on:
- model,
- data,
- task.
---
### Underfitting can come from insufficient training
Suppose the model is capable of learning the relationship.
But we stop gradient descent after:
**10 updates**.
Perhaps the loss is still falling rapidly.
The model simply has not had enough time to learn.
This is another form of underfitting.
---
### Training curves can reveal this
Suppose:
**training loss is high**
and:
**validation loss is high**.
But both are still falling.
Conceptually:
```text
Loss
|
|\
| \
| \
| \
| \
| \
+---------------- Training time
The model may simply need:
**more training**.
---
### Compare this with overfitting
With overfitting:
**training loss becomes very low**
while:
**validation loss begins increasing**.
With underfitting:
**both remain relatively poor**.
A simplified diagnostic is:
| Situation | Training performance | Validation performance |
|---|---|---|
| Underfitting | Poor | Poor |
| Good fit | Good | Good |
| Overfitting | Very good | Worse |
Reality is more complicated, but this is a useful starting point.
---
### Underfitting is sometimes called high bias
Recall the **bias-variance trade-off**.
Underfitting is often associated with:
**high bias**.
Here, bias means that the model makes strong simplifying assumptions that prevent it from representing the underlying relationship.
For example:
> Assume the relationship is always linear.
If reality is strongly non-linear, that assumption creates systematic error.
---
### Systematic error
Suppose an electricity-demand model consistently:
- underestimates winter peaks,
- overestimates summer nights.
These are not random mistakes.
There is structure in the error.
That suggests the model may be missing something important.
A useful model should ideally leave errors that look increasingly like:
**unpredictable residual variation**
rather than:
**obvious patterns it failed to learn**.
---
### Residuals
The difference between:
**actual value**
and:
**predicted value**
is called a **residual**.
Conceptually:
**RESIDUAL = ACTUAL - PREDICTED**.
Residuals can tell us a lot about what the model has failed to capture.
---
### Residuals should not contain obvious patterns
Suppose we predict electricity demand.
Then plot residual error against temperature.
We discover:
**cold temperatures → consistently positive residuals**.
That means:
**actual demand > predicted demand**
whenever it becomes cold.
The model has failed to capture some relationship between:
**temperature**
and:
**demand**.
That is useful diagnostic information.
---
### Errors contain information
A prediction error is not merely:
**something to minimise**.
It can also tell us:
> **What structure is the model still missing?**
We can examine errors across:
- time,
- location,
- groups,
- features,
- operating conditions.
Patterns in those errors can reveal underfitting.
---
### Time can reveal underfitting
Suppose a demand model predicts the same basic daily shape every day.
But real demand differs systematically between:
- weekdays,
- weekends,
- holidays.
Plotting residuals over time may reveal:
**SATURDAY → repeated error**
**SUNDAY → repeated error**
**HOLIDAYS → large repeated error**.
The model has not learned important temporal structure.
---
### Space can reveal underfitting
Suppose a national model predicts housing prices.
Its average error appears reasonable.
But map the residuals.
We discover:
**urban centres → systematically underestimated**
and:
**rural areas → systematically overestimated**.
The model has failed to capture spatial structure.
Again:
> **Aggregation can hide information.**
---
### Groups can reveal underfitting
Suppose a health model performs well overall.
But performance is poor for:
**older patients**.
Perhaps the relationship between features and outcomes differs with age.
The aggregate metric may hide this.
Underfitting can therefore occur:
**locally**
even when global performance appears acceptable.
---
### One model may not fit every regime
Suppose electricity demand behaves differently during:
- normal operation,
- extreme weather,
- public holidays,
- major events.
A single simple relationship may not capture all regimes.
The system may need:
- richer features,
- interactions,
- separate sub-models,
- more flexible architectures.
---
### Interactions matter
Suppose electricity demand depends on:
**temperature**
and:
**time of day**.
Perhaps temperature has little effect at:
**03:00**
but a large effect at:
**18:00**.
Then the effect of temperature depends on another variable.
This is an **interaction**.
A model that considers each feature independently may miss it.
---
### Reality is often conditional
The effect of:
**X**
may depend on:
**Y**.
For example:
**rain**
may affect traffic differently depending on:
**time of day**.
**temperature**
may affect electricity demand differently depending on:
**heating technology**.
**income**
may affect purchasing differently depending on:
**location**.
Models need sufficient capacity to represent these conditional relationships.
---
### Non-linearity matters
Many real-world systems are not linear.
Doubling an input does not necessarily:
**double the output**.
Examples include:
- biological systems,
- traffic congestion,
- electricity networks,
- financial markets,
- human behaviour.
A model that assumes simple proportional relationships may systematically underfit such systems.
---
### Thresholds matter
Suppose a machine operates normally below:
**90°C**.
Above:
**90°C**
failure probability rises dramatically.
A simple linear model may smooth over this threshold.
But the threshold may be the most important part of the problem.
---
### Saturation matters
Suppose advertising spending increases sales.
At first:
**more advertising → much more sales**.
Eventually:
**more advertising → little additional effect**.
This is saturation.
A linear model might incorrectly predict that sales continue increasing indefinitely.
---
### Cycles matter
Many systems contain:
- daily cycles,
- weekly cycles,
- annual cycles.
A model unable to represent periodicity may underfit time-series data.
This is why representation matters.
---
### Feature engineering can help
Suppose we give the model:
**hour = 23**
and:
**hour = 0**.
Numerically, these values appear far apart.
In reality, they are:
**one hour apart**.
We might represent time cyclically using features that preserve this structure.
Good feature engineering can make important patterns easier to learn.
---
### Underfitting can therefore be a representation problem
The model may have enough theoretical capacity.
But the way information is represented makes the relationship difficult to learn.
So we should distinguish:
**MODEL TOO SIMPLE**
from:
**REPRESENTATION TOO POOR**.
Both can produce underfitting.
---
### Neural networks reduce some feature-engineering requirements
Traditional machine learning often relied heavily on humans designing useful features.
Deep learning can learn many representations automatically.
For images, instead of manually specifying:
- edges,
- corners,
- shapes,
a neural network can learn useful internal representations from pixels.
This is one reason deep learning became so powerful.
---
### But representation has not disappeared
Even modern AI systems still depend on choices about:
- tokenisation,
- context,
- architecture,
- input resolution,
- data preprocessing.
There is no completely representation-free learning system.
The machine always receives some representation of reality.
---
### Underfitting can result from poor optimisation
Suppose a neural network has enough capacity.
But gradient descent struggles to find useful parameters.
Perhaps:
- learning rate is inappropriate,
- gradients vanish,
- training is unstable.
The model may perform poorly even though a good solution exists somewhere in parameter space.
This is **optimisation failure** rather than insufficient model capacity.
---
### Capacity and optimisation are different questions
Ask:
#### Question 1
> Can this model represent a good solution?
This is about:
**capacity**.
#### Question 2
> Can our training process actually find that solution?
This is about:
**optimisation**.
A model can have enormous capacity and still underfit if optimisation fails.
---
### Data can also make learning difficult
Suppose the training dataset contains:
- severe noise,
- inconsistent labels,
- poor measurements.
The underlying pattern may be difficult to identify.
The model's poor performance may look like underfitting.
But the real limitation may be:
**information quality**.
---
### More complexity does not fix bad information
Suppose the input data contains almost no information about the target.
Adding:
- more layers,
- more parameters,
- more computation
cannot magically create the missing information.
This is a crucial principle:
> **Model complexity cannot substitute for informative data.**
---
### The irreducible error
Some uncertainty cannot be removed with the available information.
Suppose two apparently identical situations can genuinely produce different outcomes.
No model can predict perfectly.
There may be an:
**irreducible component of error**.
We should not mistake unavoidable uncertainty for underfitting.
---
### Example: a coin
Suppose we try to predict a fair coin toss using:
- day,
- time,
- weather,
- colour of the table.
A huge neural network does not solve the problem.
The available features do not determine the outcome.
Poor predictive accuracy is not necessarily underfitting.
The event may simply be unpredictable from the information available.
---
### This is an important diagnostic distinction
When a model performs poorly, possible explanations include:
- model too simple,
- poor features,
- insufficient training,
- optimisation failure,
- noisy labels,
- missing information,
- genuine randomness.
Simply saying:
> **The model needs to be bigger**
may be completely wrong.
---
### More data may or may not help
Suppose the model underfits because:
**it is too simple**.
Adding more data will not necessarily solve the problem.
If the model can represent only a straight line, giving it:
**one billion examples of a curved relationship**
does not make the line curved.
---
### More model capacity may help
In that case, we might move from:
**linear model**
to:
**polynomial model**
or:
**decision tree**
or:
**neural network**.
The appropriate choice depends on the task.
---
### But capacity should be justified
Suppose a slightly more flexible model solves the problem.
There may be little reason to immediately deploy:
**a 100-billion-parameter model**.
More complexity can mean:
- more computation,
- more memory,
- harder interpretation,
- greater overfitting risk.
Model complexity has costs.
---
### Learning curves can help diagnose the problem
Suppose we plot performance as the amount of training data increases.
#### Case A
Training and validation performance are both poor and remain close together.
This can suggest:
**underfitting**.
#### Case B
Training performance is excellent but validation performance remains much worse.
This suggests:
**overfitting**.
Learning curves help distinguish the two.
---
### What should we do about underfitting?
Possible responses include:
- use a more expressive model,
- add useful features,
- improve feature representation,
- reduce excessive regularisation,
- train for longer,
- improve optimisation,
- improve data quality,
- model important interactions,
- represent time and space more appropriately.
The correct response depends on the cause.
---
### Sometimes separate models are useful
Suppose one relationship governs:
**summer**
and another:
**winter**.
Instead of forcing one simple model to represent both, we might use:
**different models for different regimes**.
Alternatively, a sufficiently flexible model may learn the regimes itself.
---
### Mixture models
A more general idea is:
> Different parts of the data may be best explained by different models.
A **mixture model** combines multiple components.
Conceptually:
**INPUT**
↓
**WHICH REGIME?**
↓
**APPROPRIATE MODEL**
↓
**PREDICTION**.
Later, we will encounter a related idea in modern AI:
**mixture-of-experts models**.
---
### Underfitting and classification
Suppose we classify animals using only:
**weight**.
Perhaps:
**cats tend to be lighter than dogs**.
But there is substantial overlap.
Using weight alone may never achieve strong classification.
Add:
- height,
- facial features,
- image information.
Now the classes become easier to distinguish.
The original model underfit partly because its representation was too limited.
---
### Decision boundaries can be too simple
Imagine two classes arranged in a circle:
**Class A in the centre**
and:
**Class B around the outside**.
A classifier capable only of drawing:
**a straight line**
cannot separate them well.
The correct decision boundary is:
**curved**.
The model needs sufficient flexibility to represent that geometry.
---
### Training accuracy reveals the problem
If the model cannot even classify the:
**training examples**
well, that is a strong indication of underfitting.
Overfitting models usually have:
**excellent training performance**.
Underfitting models often struggle even with what they have already seen.
---
### Underfitting in language models
Imagine training a language model with:
- tiny model capacity,
- limited context,
- insufficient training.
It might learn:
- common words,
- simple grammar,
- frequent phrases.
But fail at:
- long-range dependencies,
- complex reasoning,
- rare concepts.
Its representation is not rich enough to capture the structure of language.
---
### Model scale can reduce underfitting
One reason larger language models became more capable is that increased model capacity allowed them to represent richer patterns.
Combined with:
- more data,
- more compute,
this led to substantial improvements.
This is part of what AI researchers observed in **scaling laws**.
---
### Scaling laws
Empirically, researchers found that model performance often improves predictably as we increase combinations of:
- model size,
- training data,
- computation.
Very roughly:
**MORE PARAMETERS**
+
**MORE DATA**
+
**MORE COMPUTE**
↓
**LOWER LOSS**.
This helped drive the development of increasingly large foundation models.
---
### But scaling requires balance
A huge model trained on too little data may not use its capacity effectively.
A huge dataset used with a tiny model may exceed what the model can learn.
Enormous compute cannot compensate indefinitely for poor design.
Useful scaling involves balancing:
**MODEL**
**DATA**
**COMPUTE**.
---
### Bigger is not automatically smarter
Suppose we double model size but:
- use poor data,
- choose the wrong objective,
- fail to train properly.
The larger model may not meaningfully improve.
Scale is one ingredient.
It is not magic.
---
### Underfitting and domain knowledge
Sometimes we already know important structure about the problem.
For example:
- energy must be conserved,
- objects obey physical constraints,
- probabilities must sum to one.
Incorporating such knowledge can help models learn useful relationships more efficiently.
---
### Physics-informed models
In some scientific applications, known physical relationships can be incorporated into model design or training objectives.
Instead of asking the model to rediscover every physical law from data, we provide some structure.
This can reduce the amount of data required.
---
### But assumptions can also cause underfitting
Suppose we impose a physical assumption that is:
**wrong for the actual system**.
Now the model is prevented from representing reality accurately.
Prior knowledge helps only when it is appropriate.
Again:
> **Every modelling assumption includes some possibilities and excludes others.**
---
### This returns us to models as representations
Recall from Module 1:
> **Models are representations, not reality.**
Every model chooses:
- what to include,
- what to ignore,
- what relationships are possible.
Underfitting occurs when those modelling choices exclude important structure.
---
### Abstraction always loses something
A useful model deliberately simplifies reality.
A map does not show every:
- tree,
- stone,
- person.
That is why it is useful.
But simplify too far and the map becomes useless.
Imagine a map containing only:
**IRELAND**
with no:
- roads,
- towns,
- rivers.
Technically it represents something.
It cannot help you navigate.
That is underfitting as an analogy.
---
### The right abstraction depends on the task
A motorway map does not need:
**every footpath**.
A hiking map might.
Similarly, the appropriate model depends on:
**what prediction we need to make**.
There is no universally correct level of complexity.
---
### Prediction defines what information matters
Suppose we predict:
> Will it rain somewhere in Ireland tomorrow?
We may need relatively coarse spatial information.
Suppose instead:
> Will it rain on this particular farm between 14:00 and 15:00?
We need much finer information.
The same model may be adequate for one question and underfit the other.
---
### Resolution matters
Underfitting can occur because we represent:
**time too coarsely**
or:
**space too coarsely**.
Suppose electricity demand is represented only as:
**annual national consumption**.
That might be sufficient for some long-term questions.
It is useless for predicting:
**local network congestion at 18:00 tomorrow**.
---
### Aggregation can create apparent simplicity
Recall:
> **Spatial and temporal aggregation can destroy information.**
If the dataset has already averaged away important variation, the model cannot recover it.
Again:
**DATA REPRESENTATION**
can impose a ceiling on:
**MODEL PERFORMANCE**.
---
### Underfitting and state
Later we will introduce the idea of:
**state**.
Suppose the future depends on:
- current temperature,
- stored energy,
- velocity,
- previous actions.
But our model sees only:
**current temperature**.
Its representation of the current system is incomplete.
Poor prediction may result because the model lacks enough information about:
**where the system currently is**.
---
### Memory can matter
Suppose we predict:
**what happens next**
using only:
**what is happening now**.
But the system depends on its history.
For example:
- battery state of charge,
- building temperature,
- financial momentum,
- user behaviour.
A memoryless model may underfit a stateful process.
---
### Past observations can contain state information
Suppose a building is currently:
**20°C**.
That alone does not tell us whether its heating demand is about to rise.
But knowing:
- it was 24°C two hours ago,
- outdoor temperature is falling,
- heating has been off
provides much more information.
Historical context can help infer hidden state.
---
### Sequence models address this
Models designed for sequential data can use information across time.
Examples include:
- recurrent neural networks,
- transformers,
- state-space models.
These can represent temporal dependencies that a simple static model cannot.
---
### Underfitting and context windows
A language model has a finite **context window**.
Suppose information needed to answer a question occurred far outside that window.
The model cannot directly use it.
The limitation is not necessarily intelligence.
It is:
**available context**.
Again, prediction is bounded by what information enters the model.
---
### There is a hierarchy of possible problems
When performance is poor, ask in order:
**Do we have the necessary information?**
↓
**Is it represented appropriately?**
↓
**Can the model represent the relationship?**
↓
**Can the optimiser find useful parameters?**
↓
**Have we trained sufficiently?**
↓
**Does the model generalise?**
Jumping immediately to:
**bigger model**
skips several important questions.
---
### Underfitting versus overfitting
We can now compare the two directly.
#### Underfitting
The model learns:
**too little structure**.
Typical pattern:
**training error high**
**validation error high**.
Possible causes:
- insufficient capacity,
- poor features,
- too much regularisation,
- insufficient training,
- poor optimisation.
#### Overfitting
The model learns:
**too much training-specific structure**.
Typical pattern:
**training error very low**
**validation error significantly higher**.
Possible causes:
- excessive effective flexibility,
- limited data,
- noisy data,
- memorisation,
- shortcut learning.
---
### The goal lies between them
We are trying to find a model that is:
**complex enough**
to capture the real structure,
but:
**constrained enough**
to ignore accidental detail.
Conceptually:
**UNDERFIT**
↓
**INCREASE USEFUL CAPACITY**
↓
**GOOD GENERALISATION**
↓
**EXCESSIVE TRAINING-SPECIFIC FIT**
↓
**OVERFIT**.
The optimum is not necessarily exactly in the mathematical middle.
It depends on the problem.
---
### Bias-variance trade-off
This tension is traditionally described through:
**bias**
and:
**variance**.
Very roughly:
#### High bias
The model imposes assumptions that are too restrictive.
Result:
**underfitting**.
#### High variance
The model responds too strongly to the particular sample used for training.
Result:
**overfitting**.
Good modelling tries to balance these sources of error.
---
### More complex is not always less biased in a useful way
Suppose increasing complexity reduces bias.
That sounds good.
But if variance rises dramatically, test performance may get worse.
We care about:
**generalisation error**,
not complexity for its own sake.
---
### The model should be as complex as the evidence justifies
This gives us a useful principle:
> **Complexity should be earned by evidence.**
If a complicated relationship consistently improves performance on unseen data, the complexity is useful.
If it only improves training fit, it may be learning noise.
Validation provides the evidence.
---
### Underfitting can be useful initially
A deliberately simple model can be a very good starting point.
Suppose a linear regression achieves:
**85% of the useful performance**
with:
- easy interpretation,
- tiny compute requirements.
It provides a baseline.
Then we can ask:
> Does a more complex model provide enough additional value to justify itself?
---
### Baselines reveal whether complexity is necessary
Suppose:
**Simple model → 92% accuracy**
**Huge neural network → 92.3% accuracy**.
The larger model may still be worthwhile in some applications.
But the comparison forces us to justify:
- additional compute,
- latency,
- energy,
- complexity.
Without a baseline, we do not know whether the sophisticated model is actually adding much.
---
### Simplicity can improve service design
A simpler model may be:
- faster,
- cheaper,
- easier to explain,
- easier to audit.
If predictive performance is sufficient, these benefits can matter.
The objective is not:
> Build the most sophisticated AI possible.
It is:
> **Build a model appropriate to the service.**
---
### Underfitting can sometimes be intentional
Suppose a financial institution deliberately uses a constrained model.
Why?
Perhaps:
- regulation requires interpretability,
- robustness matters more than tiny predictive gains.
The model may sacrifice some predictive performance intentionally.
That is not necessarily poor design.
Trade-offs matter.
---
### Prediction quality is not the only objective
Recall:
**prediction is not decision-making**.
A slightly underfit but:
- interpretable,
- stable,
- cheap
model might support better overall decisions than an extremely complex model with marginally lower error.
Model choice sits inside a wider service architecture.
---
### Model capacity has a physical cost
A higher-capacity model may require:
- more parameters,
- more memory,
- more compute,
- more electricity,
- larger accelerators.
So model capacity is not purely abstract.
It eventually becomes:
**physical infrastructure**.
This will become especially important when we study modern AI and data centres.
---
### Underfitting and AI scaling
The history of modern AI can partly be seen as discovering that many models were:
**capacity-constrained**
or:
**data-constrained**
or:
**compute-constrained**.
Increasing:
- model size,
- data,
- compute
allowed models to learn increasingly rich relationships.
This produced major advances in:
- vision,
- speech,
- language.
---
### But every scale increase has diminishing value
Eventually, additional:
- parameters,
- examples,
- computation
may produce smaller improvements.
We should therefore ask:
> **What is the marginal value of additional model capacity?**
This is both:
- a machine-learning question,
- an economic question.
---
### Capacity allocation becomes a resource problem
Suppose several AI services share limited compute.
Should we allocate enormous capacity to:
**one model**
for a tiny performance improvement?
Or use that compute for:
- more users,
- lower latency,
- another service?
Even model capacity eventually becomes a shared-resource allocation problem.
---
### The central philosophical point
Underfitting teaches us something deeper about modelling.
Reality contains more detail than any model.
The purpose of modelling is to choose:
**which detail matters for the question we are asking**.
Too much simplification:
**important structure disappears**.
Too little simplification:
**noise and complexity overwhelm generalisation**.
The art is finding the appropriate abstraction.
---
### Ask underfitting questions
When a model performs poorly, ask:
- Is training performance also poor?
- Is the model capable of representing the relationship?
- Are important features missing?
- Is the representation appropriate?
- Does the problem contain nonlinear relationships?
- Are interactions being ignored?
- Is temporal structure missing?
- Is spatial structure missing?
- Does the model need memory?
- Is regularisation too strong?
- Has the model trained for long enough?
- Is optimisation working?
- Is the data informative enough?
- Is the remaining error actually irreducible?
- Would additional complexity improve performance on unseen data?
These questions help distinguish underfitting from other sources of poor performance.
---
### The central idea
Underfitting occurs when the model does not capture enough of the structure needed for useful prediction.
Conceptually:
**REAL WORLD**
↓
**RICH STRUCTURE**
but:
**MODEL**
↓
**OVERLY SIMPLE REPRESENTATION**
↓
**SYSTEMATIC ERROR**.
The model performs poorly on:
**training data**
and:
**new data**.
The solution may involve:
- better information,
- better representation,
- greater model capacity,
- better optimisation,
- more training.
But the objective is not simply:
**make the model bigger**.
It is:
> **Give the model enough capacity to represent the useful structure supported by the evidence.**
We can now see the two opposite failures:
**UNDERFITTING**
→ the model learns too little.
**OVERFITTING**
→ the model learns too much of the wrong detail.
Between them lies the central goal of machine learning:
**GENERALISATION**.
The next lesson therefore asks:
> **What does it actually mean for a model to generalise, and how can something learned from the past remain useful when it encounters something new?**