Enleashed

The purpose of machine learning is not to become good at the past.

It is to learn something from the past that remains useful when the model encounters something new.

That ability is called generalisation.

Generalisation is the ability of a model to perform well on observations it did not see during training.

This is the central goal of machine learning.

A model that performs perfectly on its training data but fails on new data has not really solved the prediction problem.

At its simplest:

PAST EXAMPLES

↓

LEARN PATTERN

↓

NEW EXAMPLE

↓

USEFUL PREDICTION

That final step is what matters.

Training is not the destination

During training, the model repeatedly sees examples where the answer is already known.

It adjusts itself to reduce error on those examples.

But once deployed, the situation changes.

The model receives:

new inputs

without already knowing:

the correct output.

So the real question is:

Did the model learn a transferable relationship, or did it merely learn the examples it was shown?

Generalisation is what separates learning from memorisation

Suppose we train a model on these examples:

2 → 4

3 → 6

4 → 8

Now give it:

If it predicts:

10,

it appears to have learned the underlying relationship:

y = 2x.

It has generalised.

If it can answer only for:

2, 3 and 4,

it has memorised examples rather than learned the pattern.

New does not mean completely unrelated

Generalisation does not mean:

A model should perform perfectly on absolutely anything.

Suppose a model is trained only on:

house prices in Dublin.

Then we ask it to predict:

the mass of a distant galaxy.

That is not a reasonable generalisation test.

Generalisation means performing well on new examples that are sufficiently related to the learning problem.

Generalisation always has a scope

Whenever we say:

The model generalises well,

we should ask:

Across what kind of difference?

For example:

new observations from the same population,
new users,
new locations,
future time periods,
new devices,
slightly different operating conditions.

These are different kinds of generalisation.

Same-distribution generalisation

The simplest case is when new examples are drawn from roughly the same distribution as the training data.

Suppose we randomly sample houses from the same city and same period.

The training set contains some houses.

The test set contains others.

The model is asked:

Can you predict new examples from a familiar environment?

This is often the easiest form of generalisation.

Generalisation through time

Suppose we train an electricity-demand model using:

2020–2025.

Then test it on:

2026.

Now the model must generalise forward through time.

This is harder because:

behaviour may change,
technology may change,
weather patterns may differ.

The future is similar to the past, but not identical.

Generalisation across space

Suppose we train a model in:

Dublin

and test it in:

Cork.

Now the question is:

Did the model learn a general relationship, or something specific to Dublin?

Spatial generalisation is crucial for models intended to operate across:

regions,
networks,
countries.

Generalisation across people

Suppose a health model is trained on thousands of patients.

If the same patients appear in both training and test data, the model may partly learn patient-specific patterns.

A stronger test is:

TRAIN ON SOME PATIENTS

↓

TEST ON COMPLETELY NEW PATIENTS.

This asks whether the model can generalise across individuals.

Generalisation across devices

Imagine an IoT fault-detection model.

It is trained on:

Sensors A–Z.

Then deployed on:

Sensor AA.

Can it still work?

The model may have learned:

genuine fault patterns,
quirks specific to individual sensors.

Holding out entire devices helps reveal the difference.

Generalisation across conditions

Suppose an autonomous vehicle is trained mostly on:

dry roads,
daylight.

Then encounters:

heavy rain,
darkness.

This tests whether its learned representation is robust across environmental conditions.

A model can generalise well in ordinary conditions while failing badly in unusual ones.

Interpolation

When new examples lie within the range of familiar training conditions, the model is often interpolating.

Suppose training temperatures range from:

0°C to 30°C.

Predicting at:

18°C

is interpolation.

The model has seen many similar situations.

Extrapolation

Suppose we ask it to predict at:

60°C.

That is far outside the training range.

The model is extrapolating.

Extrapolation is usually much more uncertain.

Generalisation inside familiar territory is much easier than generalisation beyond it.

Generalisation is not magic

A model does not somehow know how the world works beyond its evidence.

It generalises because training examples contain structure that persists.

For example:

past temperatures and demand

may reveal a relationship that continues tomorrow.

The model assumes, implicitly or explicitly:

Some relationships learned from the past remain valid beyond the training examples.

Prediction is possible because the world has persistence

If every moment were completely unrelated to everything before it, prediction would be impossible.

If everything repeated exactly, prediction would be trivial.

Machine learning is useful because reality contains:

patterns that persist imperfectly.

Generalisation exploits that middle ground.

The challenge is separating structure from accident

The training dataset contains:

persistent structure

plus:

temporary or accidental detail.

A model must determine which relationships are likely to carry forward.

Conceptually:

PAST DATA

↓

SIGNAL

NOISE

↓

LEARNING

↓

TRANSFERABLE STRUCTURE.

That is the generalisation problem.

Overfitting is failed generalisation

Recall:

overfitting

means the model learns too much of the training-specific detail.

Training performance becomes excellent.

New-data performance worsens.

So overfitting is fundamentally:

a failure to generalise.

Underfitting is also failed generalisation

Underfitting fails for the opposite reason.

The model has not captured enough of the underlying structure.

So it performs poorly on:

training data,
new data.

Generalisation lies between these two failures.

The balance

Conceptually:

TOO SIMPLE

↓

UNDERFITTING

↓

ENOUGH STRUCTURE

↓

GOOD GENERALISATION

↓

TOO MUCH TRAINING-SPECIFIC DETAIL

↓

OVERFITTING.

This is one of the central balances in machine learning.

Generalisation is measured using unseen data

Because the model has already seen the training data, we cannot use it alone to judge generalisation.

Instead we evaluate on:

validation data

and:

test data.

These provide evidence about how the model behaves beyond the examples used to fit its parameters.

Test performance is an estimate

Suppose a model achieves:

92% test accuracy.

That does not mean:

It will be exactly 92% accurate forever.

The test set is itself:

a sample.

A different sample may produce:

91%

or:

93%.

So test performance is an estimate of future generalisation.

The test set must represent the deployment challenge

Suppose we intend to deploy in:

rural Ireland.

But the test set contains only:

urban Dublin.

Excellent test performance does not demonstrate relevant generalisation.

The test distribution should resemble the situations in which the model will actually be used.

Generalisation can be local

A model may generalise very well for:

most of the input space

but poorly in particular regions.

For example:

temperature 5–25°C → excellent

temperature below -5°C → poor.

Average performance may hide these local weaknesses.

Generalisation can vary across groups

Suppose:

Overall accuracy = 94%.

But:

Group A = 97%

Group B = 78%.

The model's ability to generalise differs across populations.

This can happen when one group is:

underrepresented,
measured differently,
structurally different.

Generalisation therefore connects directly to fairness.

Training coverage matters

Suppose Group A provides:

1 million training examples.

Group B provides:

1,000.

The model has much more evidence about Group A.

It should not surprise us if performance differs.

This is why:

representative data

matters.

But equal quantities may not be enough

Suppose we collect the same number of examples from both groups.

Perhaps Group B contains:

more diverse conditions.

Then the same sample size may not provide the same coverage.

Generalisation depends on the complexity of the underlying distribution, not merely equal row counts.

Rare events challenge generalisation

Suppose equipment failure occurs:

once in 100,000 observations.

The model may see almost no failures during training.

How can it generalise to future failures?

This is difficult.

The cases that matter most may be the ones with the least historical evidence.

Generalising into the tail

A model may perform well around:

ordinary conditions.

But its safety may depend on:

rare conditions.

Examples include:

extreme weather,
unusual pedestrian behaviour,
rare diseases.

Generalisation in the tails deserves separate evaluation.

The most common case is not necessarily the most important case

Training objectives often optimise average performance.

If 99.9% of examples are ordinary, the model can achieve excellent average performance while remaining poor on the critical 0.1%.

So generalisation should be evaluated relative to:

importance

as well as:

frequency.

Feature quality affects generalisation

Suppose a model predicts wolves using:

snow in the background.

This feature works in the training dataset.

But it is not stable across environments.

A model using:

actual animal morphology

may generalise better.

Generalisation improves when the learned features capture relationships that persist.

Stable features

A feature is valuable for generalisation when its relationship to the target remains reasonably stable across deployment conditions.

For example:

shape of an animal

may be more stable than:

background colour.

But which features are stable depends on the task.

Shortcut learning hurts generalisation

A shortcut is a feature that:

predicts well in training,
does not represent the intended general relationship.

Examples include:

camera watermark → class

postcode → outcome

snow → wolf.

Shortcuts reduce training loss.

They often fail when conditions change.

Generalisation therefore depends on representation

Suppose two models receive the same raw data.

One learns:

superficial correlations.

The other learns:

deeper structural representations.

The second may transfer better to new settings.

This is one reason representation learning is so important.

Invariance

A useful concept is invariance.

Suppose an image is shifted slightly.

It is still:

a cat.

Suppose lighting changes.

Still:

a cat.

A good representation should ideally preserve important identity across changes that should not matter.

What should remain invariant?

This is a design question.

For image recognition:

small translation may not matter.

For reading road signs:

orientation may matter.

For medical imaging:

subtle spatial differences may matter enormously.

Generalisation depends partly on learning the right invariances.

Data augmentation can teach invariance

Suppose we show the model:

many rotated versions of the same cat.

All retain label:

CAT.

The model learns:

Small rotations should not change the class.

Data augmentation can therefore encourage generalisation by showing which variations should be ignored.

Domain knowledge can support generalisation

Suppose we know:

energy must be conserved.

A model that respects this physical constraint may generalise better than one that treats every relationship as unconstrained.

Domain knowledge can reduce the number of possible models the learner must consider.

Inductive bias

Any learning system contains assumptions about:

which kinds of solutions are easier or more plausible.

These assumptions are called inductive biases.

For example:

linear regression favours linear relationships,
convolutional networks exploit spatial locality,
transformers use attention across tokens.

Inductive bias helps make generalisation possible.

Why bias is necessary

Suppose we observe:

2 → 4

3 → 6

4 → 8.

What should happen at:

You might say:

10.

But infinitely many mathematical functions fit the first three examples and give completely different answers at 5.

To generalise, we need assumptions about what kinds of patterns are plausible.

Generalisation always involves assumptions

This is profound.

Data alone cannot uniquely determine behaviour everywhere we have not observed.

A learner needs some bias toward:

simplicity,
smoothness,
locality,
particular architectures.

Generalisation is therefore:

DATA + INDUCTIVE BIAS.

Occam's razor is one form of inductive bias

If two models explain the observed data equally well, we may prefer:

the simpler one.

Why?

Because the simpler relationship may be less likely to be exploiting accidental detail.

This is related to:

Occam's razor.

But simplicity is only one possible bias.

Smoothness

Many models assume something like:

Similar inputs should often produce similar outputs.

This is a smoothness assumption.

For example:

house 100 m²

and:

house 101 m²

probably do not differ in value by:

€10 million

solely because of the extra square metre.

Smoothness allows interpolation.

Locality

Some models assume:

Nearby things are more related than distant things.

This can apply in:

images,
geography,
time series.

Spatial and temporal locality provide powerful inductive structure.

Architecture creates inductive bias

A convolutional network treats:

local patterns

as important.

A recurrent model represents:

sequential history.

A transformer enables:

attention across context.

Architecture is therefore not merely compute machinery.

It expresses assumptions about structure.

Generalisation and model complexity

A larger model can represent more possible functions.

This can improve generalisation if the true relationship requires richness.

But it also means more potential solutions fit the training data.

Why the optimiser selects one that generalises well becomes important.

This remains an active area of research.

Modern deep learning challenged older intuitions

Classical machine-learning intuition suggested that models with too many parameters should overfit heavily.

Yet very large neural networks often generalise surprisingly well.

They may contain:

far more parameters than seem necessary to fit the training set.

This has forced researchers to rethink simple relationships between:

size,
complexity,
generalisation.

Implicit regularisation

The training algorithm itself may favour certain kinds of solutions.

Gradient descent does not explore all possible parameter configurations equally.

It follows a particular trajectory.

This can create an implicit bias toward solutions that generalise better than arbitrary training-perfect solutions.

Data scale changes generalisation

Suppose a model has:

1 billion parameters.

With:

100 examples,

generalisation would be extraordinarily difficult.

With:

trillions of diverse training tokens,

the situation changes dramatically.

Large datasets constrain the space of useful solutions.

Diversity may matter as much as quantity

A language model trained on:

one trillion copies of the same paragraph

would not become broadly intelligent.

Data must contain:

varied structure.

Generalisation benefits when examples cover:

many concepts,
contexts,
styles,
tasks.

Foundation models rely on broad generalisation

A foundation model is trained on broad data.

Then it can be used for many tasks it was not explicitly trained for as isolated supervised problems.

For example:

translation,
summarisation,
coding,
question answering.

This is an extraordinary form of generalisation.

Transfer learning

Suppose a model learns general visual representations from:

millions of images.

Then we fine-tune it using:

a small medical-imaging dataset.

Knowledge learned in the first task transfers to the second.

This is transfer learning.

Transfer works when structure is shared

A pre-trained image model may already understand:

edges,
textures,
shapes.

Those representations can be useful for medical images too.

The new task does not start from zero.

This makes learning much more data-efficient.

Negative transfer

But transfer can also fail.

Suppose a model's prior representations are poorly suited to the new environment.

The existing knowledge may actually interfere.

This is negative transfer.

Prior knowledge is valuable only when it is relevant.

Few-shot generalisation

Suppose a model is given only a few examples of a new task.

For example:

Input A → Output A

Input B → Output B.

Then it correctly handles:

Input C.

This is few-shot generalisation.

Modern foundation models can sometimes do this through context without updating their parameters.

Zero-shot generalisation

Even more strikingly, a model may perform a task without receiving explicit examples of that exact task in the prompt.

This is often called zero-shot generalisation.

For example:

Translate this sentence into French.

The model may infer what is required from the instruction.

In-context learning

Large language models can sometimes infer patterns from examples included in the prompt.

For example:

A → 1

B → 2

C → 3

then:

D → ?

The model adapts its behaviour based on the current context.

Its underlying parameters remain fixed.

This is often called in-context learning.

Generalisation can happen without parameter updates

This is important.

Traditional learning often means:

change parameters.

But a pre-trained model can use its existing parameters to generalise to a new task based on:

current context.

So:

PARAMETRIC LEARNING

and:

IN-CONTEXT ADAPTATION

are different mechanisms.

Generalisation and reasoning

Suppose a model has never seen the exact problem:

If every blue object is heavy and object X is blue, is X heavy?

It can still infer:

yes.

This suggests a form of generalisation over:

relational structure.

The model applies learned patterns to a novel arrangement.

Compositional generalisation

Humans can combine known concepts in new ways.

We know:

red,
bicycle.

So we can understand:

red bicycle

even if we have never seen that exact bicycle before.

This ability to combine familiar components into unfamiliar configurations is called compositional generalisation.

Compositionality is powerful

If every possible combination required a separate training example, learning would be impossibly inefficient.

Generalisation allows systems to reuse learned components.

For example:

known concept A

known concept B

↓

new combination A+B.

Language relies heavily on this.

Systematic generalisation

A stronger idea is systematic generalisation.

Suppose a model learns a rule in one context.

Can it apply the same rule reliably in another?

For example, if it learns:

A is larger than B

and:

B is larger than C,

can it infer:

A is larger than C?

Such reasoning involves generalising structure rather than memorising surface patterns.

Generalisation and abstraction

Abstraction can help transfer.

Suppose a robot learns:

Objects fall when unsupported.

That abstract relationship may apply to:

ball,
cup,
book.

It does not need separate physical laws for every object.

Abstraction allows knowledge to generalise.

Too much abstraction can lose important detail

But abstraction can also underfit.

Suppose we represent every vehicle merely as:

OBJECT.

We lose information about:

bicycle,
truck,
pedestrian.

The correct abstraction level depends on the decision.

Again:

models are representations, not reality.

Generalisation and causality

A model based on superficial correlations may fail when the environment changes.

Causal relationships can sometimes be more stable.

Suppose:

ice cream sales correlate with sunburn.

Predictively useful under familiar weather patterns.

But the causal factor:

temperature / sunlight

may generalise better if circumstances change.

Causality can support intervention

Prediction asks:

What is likely given what I observe?

Causal reasoning asks:

What happens if I change something?

When models are used for decisions, causal structure can sometimes enable stronger generalisation under intervention.

Distribution shift

Generalisation becomes especially difficult when deployment data differs from training data.

This is distribution shift.

Conceptually:

TRAINING WORLD

↓

MODEL

↓

NEW WORLD

If the relationship changes too much, performance can degrade.

Covariate shift

One form of shift occurs when the distribution of inputs changes.

For example:

Training population:

mostly age 20–50.

Deployment population:

mostly age 70–90.

Even if the underlying relationship between inputs and outputs is similar, the model is now operating in a different region of input space.

Concept drift

A deeper shift occurs when:

the relationship between inputs and outputs changes.

Suppose historical fraud patterns become known.

Fraudsters adapt.

Now:

P(Fraud | Features)

changes.

This is often called concept drift.

Generalisation can decay through time

A model may be excellent at deployment.

Six months later:

users change,
competitors change,
technology changes.

Performance deteriorates.

So generalisation is not necessarily a permanent property.

It may need to be monitored.

Continuous evaluation

A deployed system can compare:

predictions made yesterday

with:

outcomes observed today.

Conceptually:

PREDICT

↓

WAIT

↓

OBSERVE OUTCOME

↓

MEASURE ERROR

↓

UPDATE OUR BELIEF ABOUT MODEL QUALITY.

The future continuously tests the model.

Generalisation is stateful too

Suppose a model's environment changes gradually.

A static evaluation from two years ago becomes less relevant.

The system should maintain an evolving understanding of:

how well the model is currently performing.

Model quality itself can be treated as a state to monitor.

Retraining can restore generalisation

If new relationships emerge, we can add recent data and retrain.

Conceptually:

OLD MODEL

↓

NEW DATA

↓

UPDATED PARAMETERS

↓

NEW MODEL.

But retraining creates new risks.

The model might:

forget older patterns,
overreact to recent noise.

Again, adaptation requires balance.

Memory and forgetting

A model trained equally on:

20-year-old data

and:

last week's data

assumes both are equally relevant.

That may be inappropriate in a changing system.

We may need:

temporal weighting

or:

rolling training windows.

Generalisation through time depends partly on how the past is weighted.

Too much forgetting

Suppose a rare event occurs only every ten years.

A model trained only on:

the last six months

may forget the evidence that such events exist.

Short memory can damage tail generalisation.

Too much memory

Suppose consumer behaviour changed permanently five years ago.

Keeping decades of old data at equal weight may drag the model toward an obsolete relationship.

Long memory can also hurt.

Multi-timescale learning

A robust system may need:

short-term adaptation,
medium-term patterns,
long-term structural knowledge.

Different information changes at different rates.

Generalisation improves when the learning system respects those timescales.

Generalisation and uncertainty should be connected

Suppose the model sees an input very similar to its training examples.

It may be reasonably confident.

Now it sees something very unfamiliar.

Ideally:

confidence should fall.

A system should distinguish:

I can generalise confidently here

from:

I am extrapolating far beyond my evidence.

Out-of-distribution detection

One goal is to identify inputs that are very unlike the training distribution.

For example:

This medical image looks unlike anything the model was trained on.

The system might:

flag it,
lower confidence,
request human review.

This can improve reliability.

Generalisation and calibration

A model may be:

well calibrated on familiar data

but:

overconfident under distribution shift.

This is dangerous.

The probabilities still look precise even though the evidence supporting them has weakened.

Generalisation and calibration therefore need to be evaluated together.

A model should sometimes say "I don't know"

A mature prediction system should not be forced to produce confident answers everywhere.

If the input is far outside training experience, a useful response may be:

This prediction is highly uncertain.

This can be better than false precision.

Generalisation and robustness

Generalisation asks:

Does the model work on new examples?

Robustness asks something slightly stronger:

Does it continue working when inputs are perturbed or conditions become difficult?

The concepts overlap but are not identical.

Small perturbations

Suppose an image changes slightly because of:

lighting,
noise,
compression.

A robust classifier should ideally maintain the same prediction if those changes should not affect the underlying class.

This is local robustness.

Adversarial perturbations

Some small changes can be deliberately designed to fool a model.

These are adversarial examples.

A model that generalises well on ordinary test data may still be vulnerable to adversarial inputs.

So ordinary test accuracy is not the whole story.

Robust generalisation

For high-stakes systems, we may want performance across:

natural variation,
distribution shift,
adversarial conditions.

This is a stronger requirement than conventional in-distribution generalisation.

Generalisation and fairness

Suppose a model generalises well on average but poorly for a minority group.

Then its benefits are not distributed evenly.

So we should ask:

Generalisation for whom?

This echoes our earlier question:

Risk for whom?

Aggregate performance can hide unequal service quality.

Historical bias can generalise perfectly

There is an important complication.

Suppose historical hiring data contains discriminatory patterns.

A model learns those patterns accurately.

It then reproduces them successfully on new data.

Statistically, it may:

generalise extremely well.

Socially, that may be undesirable.

So:

Good generalisation does not imply a good objective or a fair system.

A model can generalise the wrong pattern

This is crucial.

Imagine historical data contains:

postcode → lower approval rate.

The pattern persists.

A model predicts it accurately.

Its test accuracy is excellent.

But perhaps the historical relationship reflects:

discrimination,
unequal access,
structural disadvantage.

The model can generalise a socially problematic relationship perfectly.

Generalisation is therefore not enough

A good AI system needs more than:

Does the pattern transfer?

We must also ask:

Is it causal?
Is it legitimate?
Is it fair?
Should the system use it?

Statistical success is only one layer of system quality.

Prediction versus service outcome

Suppose a model generalises beautifully when predicting:

who is likely to default.

But the service then denies credit in a way that:

reinforces inequality,
creates feedback loops.

The predictive model can be excellent.

The overall service can still be poor.

Again:

prediction quality ≠ service quality.

Generalisation to decisions

There is another important question.

Suppose a model's predictions generalise.

Do the decisions based on them generalise too?

A threshold chosen under one environment may become inappropriate in another.

So the full pipeline must be evaluated:

INPUT

↓

PREDICTION

↓

DECISION RULE

↓

ACTION

↓

OUTCOME.

Decision policies can overfit too

Suppose an optimisation rule is tuned precisely to:

historical demand patterns.

The predictor may still be accurate.

But the decision policy may fail when constraints change.

Generalisation applies not only to prediction models, but to whole decision systems.

Generalisation in shared-resource systems

Suppose an allocation algorithm works well when:

demand < capacity.

But fails when:

demand > capacity.

If scarcity is rare in historical data, conventional evaluation may barely test the condition where allocation rules matter most.

Service generalisation needs to include:

stress conditions.

Prediction can alter generalisation

Suppose a traffic model predicts congestion.

Drivers respond.

The resulting traffic distribution changes.

Now the environment differs from the one used to train the model.

The model's own deployment creates distribution shift.

This is a reflexive system.

Recommendation systems do this constantly

A recommendation model predicts:

what users will click.

It shows those items.

Users click partly because:

those were the items shown.

The next training dataset reflects the previous model.

Generalisation is occurring inside a feedback loop.

The data-generating process can depend on the model

Conceptually:

MODEL

↓

DECISION

↓

CHANGES WORLD

↓

NEW DATA

↓

MODEL TRAINING.

The model is not merely learning from a fixed distribution.

It helps create the distribution.

This makes long-term generalisation much harder.

Static test sets cannot capture every feedback effect

A held-out dataset assumes:

If we had deployed this model, these outcomes would still have occurred.

But when the model influences decisions, that may not be true.

We may need:

experiments,
simulations,
online evaluation.

The evaluation problem becomes causal and dynamic.

Generalisation and counterfactuals

Suppose a bank rejects a loan.

We do not observe:

Would the customer have repaid if approved?

The missing outcome is counterfactual.

Training data contains only outcomes under actions that actually happened.

Generalising to different policies therefore requires more than standard supervised learning.

Generalisation under intervention

A model might predict:

Y from X

very well under historical behaviour.

But if we intervene and change:

the relationship may differ.

This is why causal models can sometimes generalise better to new policies.

The world can respond strategically

Suppose an AI system detects fraud.

Fraudsters adapt.

Suppose an admissions system uses a particular metric.

Applicants optimise for that metric.

Once a model affects incentives, behaviour changes.

The future becomes endogenous to the model.

Goodhart's Law returns

Recall:

When a measure becomes a target, it can stop being a good measure.

Suppose a model uses:

test score

to predict academic potential.

If admissions depend heavily on the score, people invest heavily in:

optimising the test itself.

The historical relationship between score and deeper ability may change.

Generalisation can fail because the metric changed behaviour.

Generalisation is therefore partly about system stability

A model works when the relationships it relies on remain sufficiently stable.

But deployment can:

change incentives,
change behaviour,
create new technologies.

So long-term generalisation depends on the surrounding system as well as the learner.

World models

An intelligent agent may need to generalise not merely from:

input → output,

but from:

state + action → next state.

Conceptually:

CURRENT STATE

ACTION

↓

MODEL

↓

PREDICTED NEXT STATE.

A world model should generalise across situations the agent has not encountered exactly before.

Generalisation in robotics

A robot cannot be trained on every possible arrangement of:

furniture,
people,
objects.

It needs to generalise.

For example:

This is a chair, even though I have never seen this exact chair.

If that object is moving toward me, collision risk is increasing.

Robotics depends heavily on structural generalisation.

Sim-to-real generalisation

Robots are often trained partly in simulation.

Then deployed in:

the real world.

The challenge is:

Does learning from simulated experience transfer to physical reality?

This is called:

sim-to-real transfer.

The reality gap

A simulator may differ from reality in:

friction,
lighting,
sensor noise,
object behaviour.

A policy performing perfectly in simulation may fail physically.

This difference is sometimes called the:

reality gap.

Domain randomisation

One approach is to deliberately vary simulation conditions.

For example:

different lighting,
different friction,
different textures.

Instead of learning one precise simulated world, the robot learns across many.

The hope is that reality lies somewhere within that broader range.

This can improve generalisation.

Generalisation and AI agents

An AI agent using tools may face:

websites it has never seen,
new tasks,
changing environments.

It needs to apply previously learned concepts to novel situations.

Agentic AI therefore pushes generalisation beyond static benchmark prediction.

Autonomous systems require stronger generalisation

A chatbot making a poor prediction may produce:

a bad answer.

A robot making a poor generalisation may produce:

a physical action.

As prediction becomes:

agency

and then:

autonomy,

generalisation failures become more consequential.

Generalisation and safety

A safety-critical model must not merely perform well:

on average.

We need evidence about:

unusual conditions,
edge cases,
distribution shift,
tail events.

Safety requires understanding where generalisation stops.

Knowing the boundary is part of intelligence

A system that knows:

I am outside familiar territory

may be safer than one with slightly better average accuracy but no ability to recognise its limits.

Good generalisation includes some understanding of:

where confidence is justified.

Generalisation and model uncertainty

Suppose several plausible models fit historical data equally well.

But they make very different predictions in a new region.

This indicates high:

epistemic uncertainty.

The training data does not strongly constrain what happens there.

Generalisation should therefore be accompanied by uncertainty.

Agreement in training does not imply agreement outside training

Imagine two curves that fit all historical points almost identically.

Inside the training range:

same predictions.

Beyond the range:

one rises

and:

one falls.

Training data cannot tell us which extrapolation is correct.

This is why out-of-range predictions should often carry more uncertainty.

Generalisation can improve with better priors

Suppose we know a physical quantity:

cannot become negative.

Encoding that constraint prevents absurd extrapolations.

Prior knowledge can improve generalisation by excluding impossible functions.

But wrong priors damage generalisation

If we impose:

Demand can never exceed 50 GW

and future demand reaches:

60 GW,

our prior assumption prevents the model from adapting correctly.

Again:

prior knowledge helps only when valid.

No Free Lunch

There is a famous idea in optimisation and machine learning known as the No Free Lunch principle.

Very loosely:

No learning algorithm is best for every possible problem.

Why?

Because every successful generalisation method relies on assumptions about the structure of the problem.

A model that works brilliantly in one world may be inappropriate in another.

There is no universal best model

Linear regression may be perfect for one problem.

A neural network for another.

A state-space model for another.

The correct choice depends on:

structure,
data,
objective,
deployment environment.

This is why understanding the problem matters more than simply choosing the newest algorithm.

Generalisation is the real measure of learning

A model can:

reduce training loss,
memorise data,
increase parameter count.

None of these alone proves useful learning.

The evidence appears when it encounters:

something new.

So:

Generalisation is where learning is tested against reality.

The future is the ultimate generalisation test

Historical test data can simulate unseen examples.

But eventually the model is deployed.

Tomorrow arrives.

New:

users,
events,
locations,
conditions

appear.

That is the real test.

Conceptually:

PAST

↓

TRAINING

↓

HISTORICAL TEST

↓

DEPLOYMENT

↓

FUTURE

↓

ACTUAL GENERALISATION.

The future becomes new training data

Once tomorrow occurs:

future

becomes:

past.

Its outcomes can be measured.

Now we can ask:

Did the model generalise?

Those outcomes become evidence for the next version of the model.

So:

PAST

↓

MODEL

↓

FUTURE PREDICTION

↓

FUTURE HAPPENS

↓

NEW DATA

↓

UPDATED MODEL.

The learning loop continues.

Generalisation is never finally solved

A model may generalise well today.

That does not guarantee:

permanent validity.

The world continues changing.

Reliable AI therefore often requires:

monitoring,
feedback,
updating,
recalibration.

Generalisation is an ongoing relationship between:

model

and:

world.

Ask generalisation questions

Whenever someone presents a model, ask:

What data was it trained on?
What counts as a genuinely new example?
Does it generalise across time?
Across space?
Across users?
Across devices?
Across groups?
Is it interpolating or extrapolating?
Are rare events represented?
Does it rely on shortcuts?
What inductive biases does the model have?
Is confidence lower outside familiar regions?
What happens under distribution shift?
Does deployment change the data-generating process?
Is the model being monitored after deployment?
How quickly does the environment change?
When should the model be retrained?

These questions define what "works" really means.

The central idea

Generalisation is the reason we train models in the first place.

The goal is not:

memorise historical examples.

It is:

extract structure from historical examples that remains useful elsewhere.

Conceptually:

PAST DATA

↓

LEARNING

↓

MODEL

↓

NEW SITUATION

↓

USEFUL PREDICTION.

This requires a balance.

The model must be:

complex enough

to capture important structure,

but:

not so specialised

that it learns only historical accidents.

Generalisation therefore sits between:

UNDERFITTING

and:

OVERFITTING.

But there is an even deeper lesson:

Every prediction about something new relies on an assumption that some structure learned from the past continues to hold.

That assumption may be strong.

It may be weak.

It may eventually fail.

Good machine learning therefore does not merely ask:

Can I predict this?

It also asks:

How far beyond my experience should I expect this prediction to remain reliable?

The next lesson turns to one of the main ways we help models discover relationships that generalise:

feature engineering — how we choose and construct representations that make useful structure easier for a model to learn.