Training data and test data

A machine-learning model learns from examples.

But there is an immediate problem:

How do we know whether the model has actually learned something useful, rather than simply becoming good at the examples it has already seen?

This is why we separate data into:

TRAINING DATA

and:

TEST DATA.

The training data is used to teach the model.

The test data is held back and used to see whether the model can make good predictions on observations it has never seen before.

At its simplest:

DATASET

↓

TRAINING DATA → LEARN

TEST DATA → EVALUATE

This separation is one of the most important ideas in machine learning.

Learning from examples

Suppose we want to predict house prices.

Our dataset contains:

floor area,
location,
number of bedrooms,
building age,
sale price.

Each house provides an example:

INPUTS

↓

KNOWN OUTPUT

For example:

120 m² + Dublin + 3 bedrooms + 20 years old

↓

€520,000

If we have thousands of historical sales, the model can learn patterns connecting the inputs to the sale price.

The training dataset

The examples used to learn those patterns form the training dataset.

Conceptually:

TRAINING EXAMPLES

↓

LEARNING ALGORITHM

↓

TRAINED MODEL

The model adjusts its parameters to reduce prediction error on these examples.

The model has seen the answers

This creates an important issue.

During training, the model effectively has access to both:

the question

and:

the answer.

For example:

House characteristics → €520,000

It makes a prediction.

It compares that prediction with:

€520,000.

It then adjusts itself to reduce the error.

The training data therefore cannot provide an independent assessment of how well the final model works.

An exam analogy

Imagine teaching a student using:

100 questions.

For each question, you show them:

the question,
the correct answer.

They practise those questions repeatedly.

Eventually they answer all 100 correctly.

Have they learned the subject?

Maybe.

But perhaps they simply memorised the answers.

To find out, we give them:

new questions they have never seen before.

That is the role of the test dataset.

The test dataset

The test dataset contains examples that were not used to train the model.

The model receives:

the inputs

and produces:

predictions.

We then compare those predictions with the known outputs.

Conceptually:

UNSEEN INPUT

↓

TRAINED MODEL

↓

PREDICTION

↓

COMPARE WITH ACTUAL OUTCOME

This gives us evidence about how well the model generalises.

Generalisation is the real objective

Machine learning is not primarily about predicting data we already know.

We want to predict:

new observations.

So the real objective is:

Learn patterns from historical examples that remain useful for examples not seen during training.

This is called generalisation.

Memorisation is not enough

Suppose we have:

10,000 training examples.

A sufficiently powerful model might effectively memorise them.

Training accuracy:

100%.

But now we present:

1,000 new examples.

Accuracy:

60%.

The model has learned the training dataset very well.

But it has not learned a relationship that generalises well.

Training performance and test performance

We therefore distinguish between:

training performance

and:

test performance.

For example:

Model	Training accuracy	Test accuracy
A	99%	72%
B	91%	89%

If we looked only at training accuracy, Model A would appear better.

But Model B performs much better on unseen data.

In most applications, Model B is the more useful model.

Why not simply train on all the data?

It may seem wasteful to hold data back.

If more data improves learning, why not use everything?

Because then we lose our independent test.

Suppose the model achieves:

95% accuracy

on the data used to train it.

We do not know whether it has learned:

general patterns

or:

peculiarities of those particular examples.

Holding back data gives us an independent check.

A simple train-test split

Suppose we have:

10,000 observations.

We might use:

8,000 for training

and:

2,000 for testing.

Conceptually:

10,000 OBSERVATIONS

↓

80% TRAINING

20% TEST

The exact proportions depend on the problem.

There is no universal rule that 80/20 is always correct.

Random splitting

For many datasets, observations can be randomly assigned to:

training

or:

test.

This helps make both datasets representative of the same underlying population.

For example:

DATASET

↓

SHUFFLE

↓

RANDOM SPLIT

↓

TRAIN / TEST

But random splitting is not always appropriate.

Time changes the problem

Suppose we are forecasting electricity demand.

Our dataset covers:

2020–2025.

If we randomly split observations, the training dataset might contain:

December 2025,

while the test dataset contains:

June 2022.

That may not reflect the actual forecasting problem.

In reality, we use:

the past

to predict:

the future.

Time-series splitting

For time-series prediction, a more realistic split might be:

2020–2024 → TRAINING

2025 → TEST

Now the evaluation asks:

If we had trained the model using information available up to the end of 2024, how well would it have predicted 2025?

This better resembles deployment.

Respect the arrow of time

This is a fundamental principle.

When predicting the future:

Training should not allow information from the future to leak into the past.

Conceptually:

PAST

↓

TRAIN

↓

FUTURE

↓

TEST

not:

PAST + FUTURE MIXED TOGETHER

↓

RANDOM SPLIT

if that mixing gives the model information it would never have had in reality.

Data leakage

Data leakage occurs when information enters the training process that would not genuinely be available when the model is deployed.

Leakage can make a model appear astonishingly accurate.

But the performance is an illusion.

A simple leakage example

Suppose we want to predict:

Will a patient be admitted to hospital?

Our inputs include:

age,
symptoms,
blood pressure.

But someone accidentally includes:

hospital discharge date.

That variable contains information about whether admission occurred.

The model may achieve extraordinary accuracy.

But it has effectively been given part of the answer.

Future information can leak subtly

Suppose we predict tomorrow's electricity demand.

The dataset contains:

actual temperature tomorrow.

Historically, we know that value.

But at prediction time, we would only have:

forecast temperature tomorrow.

Using actual future temperature during training or testing gives the model information unavailable in deployment.

The evaluation becomes unrealistically optimistic.

Preprocessing can also leak information

Suppose we normalise a variable using:

the mean of the entire dataset.

But the entire dataset includes the test observations.

Information from the test set has now influenced the training pipeline.

The leakage may be small.

But conceptually, the test data is no longer completely independent.

Fit preprocessing on the training data

The safer structure is:

TRAINING DATA

↓

LEARN PREPROCESSING

↓

LEARN MODEL

Then apply those already-learned transformations to:

TEST DATA.

The test data should behave like genuinely new data arriving after the model has already been constructed.

The test set is supposed to represent the unknown

A useful way to think about the test dataset is:

It is a simulation of the future.

We already know the answers because we are evaluating retrospectively.

But the model must behave as though it does not.

The test set therefore represents the conditions under which the model will eventually operate.

The test set must be representative

Suppose we train a medical model using:

adults aged 18–65.

Then test it using more adults aged 18–65.

Performance may be excellent.

But we cannot automatically conclude that it works for:

children

or:

people over 80.

The test dataset only provides evidence about the population it represents.

A model is not simply "90% accurate"

Suppose someone says:

Our model is 90% accurate.

The immediate question should be:

90% accurate on what data?

Performance depends on:

population,
time period,
location,
measurement system,
class balance,
operating conditions.

A test score has context.

Geography matters

Suppose a model predicts house prices.

It is trained and tested using:

Dublin.

It achieves excellent performance.

That does not demonstrate that it will work in:

Cork

or:

London.

The relationship between features and prices may differ spatially.

Testing across locations

A stronger evaluation might be:

TRAIN

Dublin

↓

TEST

Cork

This asks a different question:

Can the model generalise geographically?

That may be much harder than randomly splitting houses within Dublin.

Testing across time

Likewise:

TRAIN

2015–2024

↓

TEST

2025

asks:

Can the model generalise through time?

This is often much closer to the real prediction problem.

Random test sets can sometimes be too easy

Suppose we have sensor measurements every second.

Adjacent observations are extremely similar.

If we randomly split individual observations:

12:00:01 → training

12:00:02 → test

the model may effectively have seen almost the same situation.

The test score can therefore exaggerate generalisation.

Grouped data needs care

Suppose we have medical records containing many observations from each patient.

If records from the same patient appear in both:

training

and:

test,

the model may partly recognise patient-specific patterns.

A better split may keep:

entire patients

in either training or testing.

The unit of independence matters

Imagine an IoT system

Suppose we have:

1,000 sensors.

Each sensor produces thousands of observations.

If observations from every sensor appear in both training and test data, we are asking:

Can the model predict new measurements from sensors it already knows?

That may be useful.

But suppose deployment requires predicting for:

completely new sensors.

Then we should hold out entire sensors.

Different splits answer different questions

Consider:

Random observation split

Can the model predict another observation drawn from a familiar population?

Time split

Can the model predict the future?

Geographic split

Can the model work somewhere new?

User split

Can the model work for people it has never seen?

Device split

Can the model work on new devices?

These are different forms of generalisation.

Testing should resemble deployment

The best test design depends on:

How will the model actually be used?

If the model will predict:

tomorrow,

test through time.

If it will operate in:

new locations,

test across locations.

If it will serve:

new users,

hold out users.

Evaluation design should mirror the real deployment challenge.

Training error normally falls

As a model learns, its performance on the training data generally improves.

Conceptually:

MORE TRAINING

↓

LOWER TRAINING ERROR

This is expected.

But training error alone tells us very little about generalisation.

Test error tells a different story

Initially, as the model learns useful patterns:

training error falls

and:

test error also falls.

But if the model becomes too specialised to the training data:

training error continues falling

while:

test error begins rising.

This is one of the classic signs of overfitting.

Overfitting

Overfitting occurs when the model learns patterns that are specific to the training data rather than relationships that generalise.

Conceptually:

TRAINING DATA

↓

SIGNAL + NOISE

A good model learns mostly:

SIGNAL.

An overfit model begins learning:

NOISE.

A memorised dataset is not intelligence

Imagine a model that stores every training example perfectly.

When shown an identical example:

perfect answer.

When shown something slightly different:

poor answer.

It has memory.

But it has not necessarily learned a useful general representation.

Machine learning is valuable because it can generalise beyond exact memorisation.

Underfitting

The opposite problem is underfitting.

Suppose the relationship is complicated.

We use a model that is far too simple.

Performance is poor on:

training data

and:

test data.

The model has failed to capture enough of the underlying structure.

Training and test performance reveal different failures

Conceptually:

Underfitting

Training performance: poor

Test performance: poor

Good generalisation

Training performance: good

Test performance: good

Overfitting

Training performance: excellent

Test performance: much worse

This comparison helps diagnose model behaviour.

But there is another problem

Suppose we train several models:

Model A

Model B

Model C

Model D

We test all four on the test dataset.

Then choose the model with the best test performance.

Have we now used the test dataset to influence the model?

Yes.

Repeatedly looking at the test set contaminates it

Imagine a student taking the same exam repeatedly.

After each attempt, they see:

their score,
which answers were wrong.

They modify their study strategy specifically for that exam.

Eventually, the exam is no longer a genuine test of unseen knowledge.

The same thing can happen in machine learning.

The validation dataset

To solve this problem, we often introduce a third dataset:

training data

validation data

test data.

Each has a different role.

Training, validation and testing

The structure becomes:

TRAINING DATA

↓

LEARN MODEL PARAMETERS

VALIDATION DATA

↓

CHOOSE MODEL / SETTINGS

TEST DATA

↓

FINAL INDEPENDENT EVALUATION

The test dataset should ideally be touched only after the main model-development decisions have been made.

What is validation data for?

Validation data helps us decide things such as:

which model to use,
how complex it should be,
which features to include,
learning rate,
regularisation strength,
classification threshold.

These choices are often called:

hyperparameters or model-selection decisions.

The test set is the final exam

A useful analogy is:

TRAINING SET = lessons and exercises

VALIDATION SET = practice exam

TEST SET = final exam

You can use the practice exam to improve your approach.

But if you repeatedly redesign everything around the final exam, it stops being an independent evaluation.

A typical split

For example:

70% training

15% validation

15% test.

Or:

80% training

10% validation

10% test.

These are examples, not universal rules.

The appropriate split depends on:

dataset size,
model complexity,
task,
deployment setting.

Large datasets change the proportions

Suppose we have:

100 million observations.

Even:

gives:

1 million test examples.

We may not need to reserve 20%.

Conversely, with only:

500 observations,

holding back large fractions can significantly reduce the data available for learning.

Cross-validation

When data is limited, we can use cross-validation.

Instead of making one fixed training-validation split, we repeatedly divide the data.

For example, with five folds:

Fold 1 → validation

Folds 2–5 → training

Then:

Fold 2 → validation

Folds 1, 3, 4, 5 → training

and so on.

Each observation gets a turn in the validation set.

Why cross-validation helps

Suppose one particular validation split happens to contain unusually difficult examples.

The score may make the model look worse than it really is.

Another split may be unusually easy.

Cross-validation evaluates the model across several partitions.

This gives a more robust estimate of performance.

Cross-validation does not solve everything

We still need to respect the structure of the data.

For time series, ordinary random cross-validation may allow future information to leak backwards.

Instead we may use something like:

TRAIN: Years 1–2 → VALIDATE: Year 3

TRAIN: Years 1–3 → VALIDATE: Year 4

TRAIN: Years 1–4 → VALIDATE: Year 5

This is sometimes called:

rolling or walk-forward validation.

Walk-forward validation

Conceptually:

PAST → NEXT PERIOD

then:

MORE PAST → NEXT PERIOD

then:

MORE PAST → NEXT PERIOD.

This mirrors how a forecasting system actually operates.

At every point:

Only information available at that time is allowed.

The future is the real test set

There is an important philosophical point here.

No matter how carefully we construct historical test datasets:

The real test dataset does not exist yet.

It is the future data the model will encounter after deployment.

Historical testing gives us evidence about likely future performance.

It cannot guarantee it.

Tomorrow may differ from yesterday

Suppose a model performs brilliantly on:

2018–2025.

Then something changes in 2026:

economic conditions,
user behaviour,
technology,
regulation,
climate,
infrastructure.

The relationship learned from historical data may weaken.

This is distribution shift.

Training and test data can come from different distributions

Ideally:

TRAINING DATA

and:

DEPLOYMENT DATA

are sufficiently similar.

But real systems change.

Conceptually:

P_train(X, Y)

may differ from:

P_future(X, Y).

When that happens, historical test performance becomes less informative.

A model can pass the test and still fail later

This is not necessarily because the original evaluation was dishonest.

The world may simply have changed.

For example:

fraud detection.

Fraudsters observe the system and adapt.

Or:

recommendation systems.

User behaviour changes because of the recommendations themselves.

The deployment environment is dynamic.

Models should therefore be monitored

Testing should not end when the model is deployed.

A real machine-learning system may need to monitor:

prediction error,
calibration,
input distributions,
missing data,
unusual observations,
group performance.

The evaluation process becomes continuous.

Deployment creates new test data

Suppose a model makes predictions today.

Tomorrow, the outcomes become known.

Today's predictions can then be compared with:

what actually happened.

So:

PREDICTION

↓

TIME PASSES

↓

OUTCOME OBSERVED

↓

ERROR MEASURED

↓

NEW EVIDENCE ABOUT MODEL PERFORMANCE

The future becomes data.

This reconnects to Module 1

Recall the course's original timeline:

PAST → DATA → MODEL → PREDICTION → DECISION → ACTION → FUTURE

After time passes:

FUTURE → NEW PAST

and therefore:

NEW DATA.

Machine learning operates inside this loop.

Test data is artificial uncertainty

When constructing a test set, we deliberately hide information we actually possess.

We know the answers.

But we pretend we do not.

Why?

Because we are trying to recreate the information boundary that will exist when the model faces the future.

This is a powerful idea:

Testing creates a controlled simulation of not knowing yet.

Do not peek

Suppose the test labels are:

hidden.

The model makes predictions.

Only afterwards do we reveal the answers.

That is a clean evaluation.

If we inspect the answers first and redesign the model around them, we have weakened the test.

Benchmark competitions use this idea

Machine-learning competitions often provide:

training data with labels

and:

test inputs without labels.

Participants submit predictions.

The organiser holds the real test labels privately.

This prevents participants from directly fitting models to the test answers.

Public leaderboards can still create overfitting

Suppose participants submit:

thousands of models.

Each time, they receive a leaderboard score.

They gradually optimise decisions based on that score.

Even without seeing the test labels directly, information about the test set leaks through the scores.

Repeated evaluation can create:

test-set overfitting.

Evaluation itself is a measurement system

Recall Module 2:

Data does not simply exist. It is produced through measurement systems.

The same is true of model evaluation.

A test score depends on:

which examples were selected,
which metric was chosen,
which population was represented,
which time period was tested.

Evaluation is designed.

One number can hide a lot

Suppose:

Test accuracy = 92%.

That sounds impressive.

But perhaps:

Daytime accuracy = 98%

Night-time accuracy = 65%.

Or:

Location A = 97%

Location B = 70%.

Or:

Group A = 96%

Group B = 72%.

Aggregate test performance can hide important structure.

Test across relevant conditions

A good evaluation may therefore examine performance by:

time,
location,
demographic group,
weather,
device type,
operating condition.

This helps reveal where the model works and where it does not.

Extreme conditions matter

Suppose an electricity-demand model performs extremely well:

99% of the time.

But it fails during:

extreme cold.

If extreme cold is precisely when prediction accuracy matters most, average performance may be misleading.

The test set should contain the conditions relevant to the service objective.

Rare events are difficult to test

Suppose catastrophic equipment failure occurs:

once every 100,000 hours.

A test dataset containing:

10,000 hours

may contain no failures at all.

The absence of errors does not demonstrate that the model can predict the rare event.

We need enough relevant examples to evaluate the behaviour we care about.

Test data has uncertainty too

Suppose Model A scores:

91%

and Model B scores:

90%.

Is Model A definitely better?

Not necessarily.

The test dataset is itself a sample.

Another test sample might reverse the result.

Performance estimates have uncertainty.

Larger test sets reduce evaluation uncertainty

Suppose we test on:

10 observations.

One additional mistake changes accuracy by:

10 percentage points.

Test on:

100,000 observations.

One additional mistake barely changes the result.

Larger test datasets generally provide more precise estimates.

But diversity can matter more than sheer size

A dataset containing:

1 million nearly identical observations

may provide less useful evidence than:

100,000 diverse observations

covering:

seasons,
locations,
populations,
operating conditions.

Good testing requires coverage, not merely volume.

External validation

A particularly strong test is sometimes performed using data collected independently.

For example:

TRAIN

Hospital A

↓

TEST

Hospital B.

Or:

TRAIN

Country A

↓

TEST

Country B.

This helps reveal whether the model learned general patterns or environment-specific shortcuts.

Out-of-distribution testing

We may deliberately test the model on conditions unlike its training data.

For example:

unusual weather,
new sensor types,
different populations.

This asks:

How does the model behave when the world moves outside its experience?

That is different from ordinary test accuracy.

Models should know when they are outside their experience

Ideally, a system encountering unfamiliar conditions should not confidently pretend everything is normal.

It may:

increase uncertainty,
flag the observation,
request human review,
fall back to a safer system.

Recognising the limits of training data is an important part of reliable AI.

Training data defines experience

A useful way to think about training data is:

The training dataset is the model's experience of the world.

If something never appears there, the model has limited evidence about it.

If something appears frequently, the model has much more opportunity to learn it.

Test data probes that experience

The test set asks:

Given what you learned from your experience, what can you do with something new?

This is why test performance is fundamentally about:

generalisation.

Data quantity and data coverage are different

Suppose we collect:

10 million images of daytime roads

and:

100 images of roads at night.

The dataset is enormous.

But night-time coverage is poor.

A model trained on this data may perform badly at night.

More data does not automatically mean more complete data.

Training data shapes what can be learned

If training data systematically excludes:

certain groups,
locations,
situations,

the model may perform poorly there.

This connects directly to:

sampling,
selection,
bias,
fairness.

The training dataset is not neutral.

It defines the evidence available to the learning algorithm.

Testing can reveal unfairness

Suppose overall classification accuracy is:

94%.

Now separate the test results:

Group A: 97%

Group B: 79%.

The aggregate score concealed a substantial difference.

Careful test design is therefore essential for evaluating fairness.

Suppose both training and test datasets systematically underrepresent:

Group C.

The model may appear excellent on the test set.

But the evaluation never seriously tested Group C.

A biased test set can validate a biased model.

Independent does not automatically mean representative

A test dataset can be completely separate from training and still be a poor test.

It must also represent:

the conditions in which we care about performance.

So two requirements matter:

INDEPENDENCE

and:

RELEVANCE.

Train on the past, test against the question you actually care about

This gives us a useful rule:

Design the test set around the deployment question, not merely around what is convenient to split.

If the question is:

Will this work next year?

test through time.

If the question is:

Will this work in a new city?

test geographically.

If the question is:

Will this work for new users?

hold out users.

The evaluation should represent the uncertainty the deployed model will actually face.

Training data can grow over time

Suppose the system operates continuously.

Each day:

new observations arrive.

These observations can eventually become new training examples.

So:

TRAIN

↓

DEPLOY

↓

OBSERVE

↓

COLLECT NEW DATA

↓

RETRAIN

↓

DEPLOY UPDATED MODEL.

The training dataset evolves.

But retraining changes the model

Suppose:

Model version 1

was tested carefully.

Then new data is added and the model is retrained.

Now we have:

Model version 2.

Its behaviour may differ.

It should be evaluated again.

Testing belongs to the model-development lifecycle, not just the initial experiment.

Test data should not become training data accidentally

Once test examples have repeatedly influenced model development, they are effectively no longer completely unseen.

It may be necessary to create:

a new test set.

This is especially important in long-running research programmes where the same benchmark is used repeatedly.

Benchmark saturation

Suppose an entire research community optimises models for the same benchmark for ten years.

Even if researchers never directly train on the test labels, thousands of decisions are influenced by benchmark performance.

Eventually, progress on the benchmark may not translate perfectly into progress in the real world.

The benchmark itself becomes part of the optimisation process.

Goodhart's Law appears again

A useful principle is:

When a measure becomes a target, it can stop being a good measure.

If the entire goal becomes:

maximise test benchmark score,

researchers may optimise specifically for the benchmark rather than the underlying real-world objective.

Evaluation metrics should remain connected to the service we actually care about.

Testing AI is not the same as testing software

Traditional software often has deterministic rules.

Given:

input X

we may know exactly what output should occur.

Machine-learning systems are different.

Their behaviour is learned statistically.

We often evaluate:

average performance,
distributions of errors,
failure rates.

Testing therefore becomes partly statistical.

A model can be mostly right and still be unsafe

Suppose an autonomous system is correct:

99.99% of the time.

That sounds excellent.

But at:

1 billion decisions,

0.01% corresponds to:

100,000 errors.

Scale changes the meaning of performance.

Their real-world value is not.

This is why Module 5 will ask:

When is a prediction good?

Testing is not merely about one metric

A serious evaluation may ask:

How accurate is the model?
How uncertain are its predictions?
Is it calibrated?
How does it perform on rare events?
How does performance vary across groups?
How does it perform across time?
How does it perform across space?
How does it behave under distribution shift?
How costly are its errors?

A single test score cannot answer all of these questions.

Training data teaches

The simplest distinction remains:

TRAINING DATA

is used to:

change the model.

The model learns from it.

Validation data guides

VALIDATION DATA

is used to:

change our choices about the model.

We use it to choose:

architecture,
complexity,
settings,
features.

Test data judges

TEST DATA

is used to:

estimate how the finished system performs on unseen data.

It should not repeatedly drive model-development decisions.

So:

TRAIN → LEARN

VALIDATE → CHOOSE

TEST → JUDGE

A useful checklist

When someone reports machine-learning performance, ask:

What data was used for training?
What data was held out?
Was there a validation set?
Was the test set genuinely unseen?
How were observations split?
Was the split random, temporal, spatial or grouped?
Could information leak between training and test?
Does the test set resemble deployment?
Are the same users, devices or locations present in both?
Does the test set contain important rare events?
Is performance reported across relevant groups?
How large is the test set?
How uncertain is the reported score?
Has the model been repeatedly tuned against the test set?
What happens when the future differs from the test data?

These questions are often more important than the headline model architecture.

The central idea

Machine learning is not about explaining examples the model has already seen.

It is about learning something from those examples that remains useful elsewhere.

That requires a separation between:

LEARNING

and:

EVALUATION.

Conceptually:

PAST DATA

↓

TRAINING SET

↓

LEARN MODEL

↓

UNSEEN TEST DATA

↓

EVALUATE GENERALISATION

But the deepest version of the idea is:

PAST

↓

TRAIN

↓

HISTORICAL UNSEEN DATA

↓

TEST

↓

DEPLOY

↓

ACTUAL FUTURE

The historical test set is only our rehearsal.

The future is the real test set.

And because the future does not exist yet, no test score can prove that a model will always work.

It can only give us evidence about how well the model has learned relationships that may continue to hold.

The next question is therefore:

What exactly is the model learning when it learns from these examples?

To answer that, we need to look at parameters.