Enleashed

Regression is one of the simplest and most important forms of machine learning.

The basic question is:

Given some information, can we predict a numerical value?

Examples include predicting:

tomorrow's temperature,
the price of a house,
electricity demand,
journey time,
annual income,
energy consumption,
the remaining lifetime of a machine.

In each case, the output is a number.

This distinguishes regression from classification, which we will examine in the next lesson.

At its simplest:

INPUTS → MODEL → NUMBER

For example:

house characteristics

↓

regression model

↓

predicted price = €450,000

Regression learns relationships between variables

Suppose we observe the outside temperature and electricity demand:

Temperature	Electricity demand
2°C	44 GW
5°C	42 GW
8°C	39 GW
12°C	36 GW
16°C	33 GW

There appears to be a pattern:

As temperature increases, electricity demand tends to decrease.

A regression model attempts to represent that relationship mathematically.

Once we have learned the relationship from historical observations, we can use it to make predictions for new inputs.

From observations to a line

Imagine plotting the data.

Each observation becomes a point:

x-axis = temperature

y-axis = electricity demand

The points will probably not form a perfect line.

Real data contains:

variation,
noise,
measurement error,
other influences.

Regression tries to find a relationship that captures the underlying pattern.

For a simple problem, that relationship might be a straight line.

Linear regression

The simplest form is linear regression.

We assume that the relationship between input x and output y can be approximated by a straight line.

Conceptually:

predicted output = intercept + slope × input

The model learns:

where the line starts,
how steeply it rises or falls.

::contentReference[oaicite:0]{index=0}

What does the slope mean?

Suppose our model learns that electricity demand falls by approximately:

0.7 GW for every 1°C increase in temperature.

The slope is therefore negative.

Conceptually:

higher temperature → lower predicted demand

If the slope were positive:

higher x → higher predicted y

If the slope were approximately zero:

x provides little linear information about y.

The slope describes how the prediction changes as the input changes.

The intercept

The intercept tells us where the fitted line crosses the y-axis.

Suppose the fitted relationship predicts:

45 GW at 0°C.

Then 45 GW is the intercept.

The intercept can be useful mathematically even when the value:

x = 0

has little practical meaning.

For example, in a house-price model:

floor area = 0

may not describe a meaningful house.

So model parameters need to be interpreted in context.

Making a prediction

Suppose our fitted model approximately describes:

electricity demand = 45 - 0.7 × temperature

At:

10°C

the model predicts:

45 - 7 = 38 GW.

So:

temperature = 10°C

↓

regression model

↓

predicted demand = 38 GW

That is regression being used for prediction.

The prediction will rarely be exact

Suppose actual demand turns out to be:

39.5 GW.

Our prediction was:

38 GW.

The model was wrong by:

1.5 GW.

That is normal.

Regression is not trying to reproduce every observation perfectly.

It is trying to learn a relationship that generalises.

Residuals

The difference between an observed value and the value predicted by the model is called a residual.

Conceptually:

RESIDUAL = ACTUAL VALUE - PREDICTED VALUE

Suppose:

actual demand = 39.5 GW

and:

predicted demand = 38 GW.

Then:

residual = +1.5 GW.

The observation lies above the fitted prediction.

Residuals contain information

Residuals are not merely mistakes to be discarded.

They can tell us something about:

model quality,
missing variables,
unusual observations,
changing relationships.

If residuals appear completely random around zero, the model may be capturing much of the systematic structure.

If residuals show patterns, the model may be missing something important.

A pattern in the errors is a clue

Suppose residuals are consistently positive:

every weekday morning.

That suggests the model systematically underpredicts morning demand.

Perhaps it is missing:

time of day.

Or suppose errors are larger:

during very cold weather.

Perhaps the relationship between temperature and demand is nonlinear.

Residuals can reveal structure that the model failed to learn.

Regression does not require one input

Our electricity example used:

temperature → demand.

But electricity demand depends on much more than temperature.

We might use:

temperature,
hour of day,
day of week,
previous demand,
wind speed,
season,
holiday status.

Now the model has multiple inputs.

This is multiple regression.

Multiple regression

Conceptually:

temperature

time of day

day of week

previous demand

↓

REGRESSION MODEL

↓

predicted electricity demand

The model learns how these inputs collectively relate to the output.

A regression model can have many features

Suppose:

x₁ = temperature

x₂ = previous demand

x₃ = hour of day

x₄ = weekday indicator

The model might learn something conceptually like:

prediction = baseline + contribution from x₁ + contribution from x₂ + contribution from x₃ + contribution from x₄

Each input contributes information.

Later we will ask a very interesting question:

How much did each feature contribute to a particular prediction?

That will lead us toward feature importance and Shapley values.

Regression is not just drawing a line

Linear regression is a useful starting point because it is easy to understand.

But regression means more generally:

Predicting a numerical output from one or more inputs.

The relationship does not need to be linear.

A regression model might use:

curves,
decision trees,
random forests,
neural networks.

The output can still be numerical.

Nonlinear relationships

Suppose electricity demand behaves like this:

high when extremely cold,
lower at moderate temperatures,
high again when extremely hot because of air conditioning.

A straight line cannot represent that relationship well.

The relationship might be closer to:

HIGH DEMAND

↓

cold

↓

moderate temperature → LOW DEMAND

↓

hot

↓

HIGH DEMAND

This is nonlinear.

The world is full of nonlinear relationships

Consider speed and aerodynamic drag.

Doubling speed does not simply double drag.

Or consider battery performance.

Temperature effects may become much stronger near extreme conditions.

Or consider income and spending.

An additional €1,000 may have different effects depending on existing income.

Real systems frequently contain nonlinear relationships.

Linear models can still be useful

A model does not have to reproduce every detail of reality to be useful.

Suppose the true relationship is slightly curved.

A straight line may still provide:

reasonable predictions,
easy interpretation,
fast computation.

The appropriate model depends on the problem.

More complexity is not automatically better.

Regression learns from examples

Suppose we have historical observations:

x₁ → y₁

x₂ → y₂

x₃ → y₃

and so on.

The learning algorithm searches for model parameters that make:

predicted values

close to:

observed values.

This is the essence of supervised learning.

Training examples

For house prices, one training example might be:

INPUTS

area = 120 m²
bedrooms = 3
distance from centre = 4 km
age = 20 years

OUTPUT

sale price = €520,000

The model sees many such examples.

It attempts to learn the relationship between:

characteristics

and:

price.

Prediction on a new example

Now a new house appears:

area = 125 m²
bedrooms = 3
distance from centre = 3.8 km
age = 18 years.

Its sale price is not yet known.

The model might predict:

€545,000.

The important point is:

The model has never seen this exact house before.

It must generalise from previous examples.

Regression is interpolation

Often, regression predicts cases that lie within the range of situations seen during training.

Suppose training temperatures range from:

0°C to 25°C.

Predicting demand at:

12°C

is largely an interpolation problem.

The model has seen many comparable conditions.

Regression can also extrapolate

Now suppose we ask the model to predict demand at:

45°C.

But the training data never contained temperatures above:

25°C.

The model must extrapolate beyond its experience.

This is much more dangerous.

Extrapolation can fail badly

Suppose the historical relationship is:

higher temperature → lower demand.

A linear model continues that trend indefinitely.

At sufficiently high temperature it might predict:

negative electricity demand.

Mathematically consistent.

Physically absurd.

The model does not automatically understand the limits of the relationship.

Models learn within the world they observe

This gives us an important principle:

A model is usually most trustworthy in regions of the input space represented by its training data.

Outside that region, uncertainty should generally increase.

This will become important when we discuss:

generalisation,
distribution shift,
model uncertainty.

Regression does not imply causation

Suppose we find:

temperature predicts ice cream sales.

That does not mean:

ice cream sales cause temperature.

Regression can identify predictive relationships.

It does not automatically establish causal relationships.

Recall:

CORRELATION ≠ CAUSATION

and similarly:

REGRESSION ≠ CAUSATION.

A regression coefficient is not automatically a causal effect

Suppose we regress:

income

against:

years of education.

We find that people with more education tend to have higher incomes.

That does not immediately prove:

One additional year of education causes exactly this increase in income.

Other factors may influence both:

family background,
location,
opportunity,
ability,
economic conditions.

Causal claims require stronger assumptions or research designs.

Prediction may not need causation

Suppose our only objective is:

Predict tomorrow's electricity demand accurately.

If temperature strongly predicts demand, it can be useful even if the model does not explicitly represent every causal mechanism.

Prediction asks:

Does this information help estimate the output?

Causal inference asks:

What happens if we intervene and change something?

Those are different questions.

Regression towards the mean

Regression also gives its name to an important statistical phenomenon:

regression to the mean.

Suppose someone performs exceptionally well on one test.

Their next performance may be closer to their usual average.

Likewise, an unusually bad observation may be followed by a more typical one.

Extreme outcomes often contain some element of random variation.

Why this matters

Suppose a company intervenes whenever a machine has an unusually bad performance day.

The machine improves the next day.

It is tempting to conclude:

The intervention caused the improvement.

Perhaps it did.

But some improvement may simply reflect regression to the mean.

This is another reason prediction and causation must be separated.

How does the model choose the line?

Suppose we have many possible lines.

One line is:

too high.

Another:

too low.

Another:

too steep.

Another:

too shallow.

How should the model decide which is best?

We need to define:

What counts as a good fit?

Measuring prediction error

For each observation, we can calculate:

actual value - predicted value.

Suppose:

Actual	Predicted	Error
40	39	+1
35	37	-2
50	47	+3

We need some way of combining these errors.

That requires a loss function.

Squared error

One common approach is to square each error.

So:

error = 2

becomes:

squared error = 4.

And:

error = -2

also becomes:

squared error = 4.

Squaring prevents positive and negative errors from cancelling each other out.

It also penalises large errors more heavily.

Least squares

Traditional linear regression commonly chooses the line that minimises:

the sum of squared residuals.

This is called ordinary least squares.

Conceptually:

TRY MODEL

↓

MAKE PREDICTIONS

↓

CALCULATE RESIDUALS

↓

SQUARE THEM

↓

ADD THEM

↓

ADJUST MODEL TO REDUCE TOTAL

The best-fitting line is the one with the smallest total squared error.

Why square the errors?

Suppose two models make errors:

Model A

1, 1, 1, 1

Model B

0, 0, 0, 4

Both have the same total absolute error:

But squared errors are:

Model A

1 + 1 + 1 + 1 = 4

Model B

0 + 0 + 0 + 16 = 16

Squared error strongly penalises the one large mistake.

Whether that is desirable depends on the application.

Loss functions encode what we care about

This is an important idea that we will return to later.

Before the machine can learn, someone must define:

What counts as being wrong?

If large mistakes are particularly costly, squared error may make sense.

If every unit of error has roughly equal consequence, another loss function may be preferable.

The mathematics of learning already contains a value judgement about errors.

Different errors can have different consequences

Suppose we predict electricity demand.

Overprediction

We schedule too much generation.

Possible consequence:

additional cost.

Underprediction

We schedule too little.

Possible consequence:

shortage or emergency action.

These errors may not have equal consequences.

A symmetric regression loss may therefore not always reflect the actual service objective.

Prediction error and decision cost are different

Suppose:

actual demand = 40 GW.

Model A predicts:

39 GW.

Model B predicts:

41 GW.

Both are wrong by:

1 GW.

Statistically, the errors may appear identical.

Operationally, they may not be.

This distinction becomes important when prediction feeds into decision-making.

The best statistical model may not create the best service

Suppose Model A has:

lower average prediction error.

Model B is slightly less accurate overall but much better at predicting extreme demand peaks.

If the service objective is:

Avoid shortages during peaks,

Model B may be more useful.

Prediction quality depends on context.

Regression can produce uncertainty

A regression model does not need to output only:

predicted value = 40 GW.

It might instead produce:

expected demand = 40 GW

with:

likely range = 37–44 GW.

Or even:

a full probability distribution over possible demand.

This connects regression directly to the probability concepts from Module 3.

Point prediction versus predictive distribution

A point prediction says:

40 GW

A probabilistic prediction says something richer:

Demand is centred around 40 GW, but values between 37 and 44 GW remain plausible.

For many real decisions, the second is much more useful.

Two identical predictions can have different uncertainty

Suppose:

Model A

expected demand = 40 GW

uncertainty = small

Model B

expected demand = 40 GW

uncertainty = large

The point predictions are identical.

But the appropriate decisions may differ.

A system might hold more reserve under Model B.

Regression connects expected values to inputs

Recall expected value from Module 3.

A regression model can often be interpreted as trying to estimate something like:

What is the expected value of Y given X?

Conceptually:

INPUT X

↓

CONDITIONAL DISTRIBUTION OF Y

↓

EXPECTED VALUE

So regression connects naturally to conditional probability.

The same input can produce different real outcomes

Suppose:

temperature = 10°C.

Historical electricity demand under those conditions might have been:

36 GW

38 GW

39 GW

41 GW

43 GW.

The model might predict:

39.4 GW.

That does not mean demand must be 39.4 GW.

It represents a central estimate given the available information.

Why can't the model predict perfectly?

Because temperature is not the only thing affecting demand.

Other influences include:

human behaviour,
industrial activity,
weather conditions,
random events,
measurement noise.

Some information may not be available to the model.

So:

same observed inputs

can correspond to:

different outputs.

Some uncertainty may be irreducible

Even with an excellent model, the world may contain randomness or unobserved information.

We can improve:

sensors,
features,
data,
models.

But prediction error may never reach zero.

A good regression model should therefore help us understand:

what we know

and:

what remains uncertain.

Regression can have many dimensions

With one feature, we can imagine:

a line.

With two features, we might imagine:

a plane.

With hundreds or millions of features, the geometry becomes impossible to visualise directly.

But the underlying idea remains:

INPUT VECTOR

↓

LEARNED FUNCTION

↓

NUMERICAL OUTPUT.

High-dimensional regression

Modern models can use enormous numbers of inputs.

For example, predicting a property value might involve:

structured property data,
satellite imagery,
maps,
local economic data.

Predicting machine failure might use:

hundreds of sensor streams.

The regression problem remains conceptually the same even when the input space becomes enormous.

Regression with neural networks

A neural network can also perform regression.

For example:

camera image

↓

neural network

↓

estimated distance to pedestrian = 12.4 metres

The input is extremely complex.

The output is still numerical.

Therefore the task is still regression.

Regression can predict several numbers at once

Suppose a model receives an image and predicts:

x-position

y-position

width

height

of an object.

The output is a vector of numerical values.

This is still regression.

Regression does not require a single scalar output.

Regression in robotics

A robot may predict:

future x-position

future y-position

future velocity.

For example:

CURRENT STATE

↓

DYNAMICAL MODEL

↓

PREDICTED NEXT STATE

This is regression embedded inside a dynamic system.

Regression through time

Suppose:

x(t)

is the current system state.

We want to estimate:

x(t+1).

A model learns:

current state → next state.

Repeated prediction gives:

x(t)

↓

x(t+1)

↓

x(t+2)

↓

x(t+3).

This begins to connect machine learning with state-space models and control.

Errors can accumulate through time

Suppose each one-step prediction is slightly wrong.

If we repeatedly feed predictions back into the model:

small error

↓

next prediction

↓

larger error

↓

next prediction

↓

possibly larger error

Long-horizon prediction can therefore become much harder than short-horizon prediction.

Regression in space

Suppose sensors measure air pollution at several locations.

We want pollution concentration everywhere else.

Inputs might include:

latitude,
longitude,
nearby sensor readings,
weather.

Output:

predicted pollution concentration.

This is spatial regression.

Space and time can appear together

Suppose we want:

Air pollution at every location in the city one hour from now.

Now the inputs include:

current spatial measurements,
historical measurements,
weather.

The output is a spatial field in the future.

Many real prediction problems are both:

spatial

and:

temporal.

Regression models can learn shortcuts

Suppose a model predicts house prices.

It discovers postcode is extremely predictive.

Perhaps it relies almost entirely on postcode.

The model may perform well.

But we should still ask:

What relationship has it actually learned?

Predictive performance alone does not reveal the model's reasoning.

Feature importance

Once a model uses many features, we naturally want to ask:

Which features matter most?

For example:

House price prediction

location?
floor area?
age?
garden?
nearby transport?

There are many ways to measure feature importance.

Later we will examine this directly.

Feature importance can be global

A global question asks:

Across the whole dataset, which features does the model rely on most?

Perhaps:

location is the strongest predictor overall.

That tells us something about the model's general behaviour.

Feature importance can be local

A local question asks:

Why did the model predict €600,000 for this particular house?

Perhaps:

baseline = €400,000

then:

location = +€120,000

size = +€70,000

age = -€20,000

garden = +€30,000.

Now we are explaining one specific prediction.

Features interact

Suppose:

large garden

adds substantial value in a suburban location.

But in another location, garden size may matter much less.

The contribution of one feature can depend on:

which other features are present.

This makes assigning credit difficult.

Who gets credit for the prediction?

Imagine two features:

location

and:

floor area.

Together they increase the prediction by:

€200,000.

How much of that value belongs to location?

How much belongs to floor area?

The answer may depend on the order in which we consider them.

This creates a credit-allocation problem.

From regression to Shapley values

This gives us an important bridge to a concept we will encounter later.

We can imagine the model's features as cooperating to produce a prediction.

Then ask:

How should the difference between the baseline prediction and the actual prediction be fairly allocated among the features?

One powerful answer comes from cooperative game theory:

Shapley values.

The same mathematical idea used to allocate value among cooperating participants can be used to allocate predictive contribution among features.

We will return to this after feature engineering.

But feature contribution is not causal contribution

Suppose postcode contributes:

+€100,000

to a house-price prediction.

That does not necessarily mean:

Changing the postcode while changing nothing else would causally increase the house's value by €100,000.

Feature attribution explains the model's prediction.

Causal inference asks a different question.

This distinction is essential.

Regression can be simple and interpretable

One advantage of linear regression is that its structure is relatively transparent.

We can inspect:

coefficients,
residuals,
predictions.

This makes it useful not only for prediction but also for understanding relationships.

More powerful models can be less transparent

A neural network may produce much better predictions.

But the mapping from:

thousands of inputs

to:

one output

may be difficult to interpret directly.

This creates a trade-off between:

predictive performance,
interpretability.

Not always, but often.

Simple models make useful baselines

Suppose we develop an enormous neural network for electricity demand forecasting.

Before celebrating, compare it with:

linear regression.

If the neural network improves accuracy by only:

0.1%,

the extra complexity may not be worthwhile.

Simple models provide valuable baselines.

Always ask: better than what?

A model should not be evaluated in isolation.

Compare it with:

historical average,
previous value,
simple linear regression,
existing forecasting method.

A sophisticated model that cannot outperform a simple baseline is not particularly useful.

Regression can overfit

Suppose we allow the model to become extremely complex.

It may bend itself around every historical observation.

Training error becomes almost zero.

But on new data, performance deteriorates.

The model has learned:

noise

rather than:

general structure.

This is overfitting.

Regression can underfit

The opposite can happen.

Suppose the true relationship is strongly curved.

We insist on fitting a straight line.

The model cannot capture the pattern.

This is underfitting.

So we face a balance:

TOO SIMPLE

↓

UNDERFITTING

APPROPRIATE COMPLEXITY

↓

GENERALISATION

TOO COMPLEX

↓

OVERFITTING

We will study this carefully later.

Training error is not enough

Suppose:

Model A

Training error = 1%

Test error = 20%

Model B

Training error = 5%

Test error = 6%

Which is better?

Usually:

Model B.

The objective is not to explain the training dataset perfectly.

It is to perform well on new examples.

Regression is fundamentally about generalisation

The real task is:

PAST EXAMPLES

↓

LEARN RELATIONSHIP

↓

NEW INPUT

↓

PREDICT UNKNOWN OUTPUT

A regression model succeeds when the relationship learned from the past remains useful outside the examples used to train it.

Distribution shift can break regression

Suppose an electricity-demand model was trained between:

2010 and 2020.

Then:

millions of heat pumps appear,
millions of electric vehicles appear,
working patterns change.

The historical relationship between:

temperature

and:

demand

may change.

The regression model may become less accurate.

Tomorrow may not resemble yesterday

Regression relies on some persistence in relationships.

If:

P(Y | X)

changes substantially between training and deployment, prediction deteriorates.

This is one form of distribution shift.

No amount of historical fitting guarantees that the future will behave the same way.

Models need updating

In changing systems, models may need:

new observations,
retraining,
recalibration.

The prediction system becomes a loop:

DATA

↓

MODEL

↓

PREDICTION

↓

OUTCOME

↓

ERROR

↓

NEW DATA

↓

UPDATED MODEL.

Learning can continue after deployment.

Regression can affect the thing being predicted

Suppose an electricity model predicts:

high demand tomorrow.

The prediction triggers:

higher prices,
demand-response actions.

Demand falls.

The prediction helped change the outcome.

Now prediction exists inside a feedback system.

A perfect forecast can become wrong because people use it

Imagine a traffic model predicts:

Road A: 60 minutes

Road B: 30 minutes.

Everyone chooses Road B.

Now Road B becomes congested.

The prediction altered the system.

This is a reflexive prediction problem.

Regression inside intelligent services

A regression model rarely exists alone.

A real service might look like:

SENSORS

↓

INPUTS

↓

REGRESSION MODEL

↓

PREDICTED DEMAND

↓

OPTIMISATION

↓

RESOURCE ALLOCATION

↓

ACTION

↓

NEW DEMAND

The prediction is one component of a larger decision-making system.

Better regression does not guarantee better outcomes

Suppose demand forecasting becomes:

5% more accurate.

That sounds good.

But if the downstream optimisation:

has the wrong objective,
ignores important constraints,
allocates resources unfairly,

the overall service may still be poor.

This reinforces the distinction:

prediction quality ≠ decision quality.

Regression can reduce uncertainty

Suppose before observing temperature we believe tomorrow's demand could plausibly be:

30–50 GW.

Then we learn:

temperature will be 5°C.

The plausible range narrows to:

40–45 GW.

The feature has provided information.

Regression uses that information to reduce uncertainty about the target.

This gives us another way to understand regression

Regression is not merely:

draw a line through points.

It is:

Use information contained in X to reduce uncertainty about numerical Y.

That interpretation connects regression directly to:

probability,
information,
prediction.

Inputs have value because they reduce uncertainty

Suppose knowing:

shoe colour

does nothing to improve electricity-demand prediction.

It contributes little useful information.

Knowing:

temperature

may reduce uncertainty substantially.

Knowing:

previous demand

may reduce it further.

This creates the idea of:

marginal value of information.

Information can overlap

Suppose we already know:

outside temperature.

Then adding:

heating degree days

may provide little additional information because it is derived largely from temperature.

But if temperature were absent, heating degree days might be highly informative.

So:

The value of a feature depends on what other information is already available.

This is an important idea.

Feature value is conditional

Suppose:

Feature A alone gives a large improvement.

Feature B alone also gives a large improvement.

But together they provide only slightly more information than either alone.

How should we divide the credit?

We cannot simply add their individual contributions.

Their information overlaps.

This is exactly the kind of problem that motivates Shapley-based feature attribution.

Regression therefore leads naturally to several deeper questions

Once we can predict numerical values, we need to ask:

How should error be measured?
How do we learn the model parameters?
How complex should the model be?
Which features matter?
How much does each feature contribute?
How uncertain is the prediction?
Will the relationship generalise?
What happens when the world changes?
What decision uses the prediction?

These questions take us from basic regression toward modern machine learning.

A useful regression checklist

When someone presents a regression model, ask:

What is the numerical target?
What are the inputs?
When are those inputs available?
How was the target measured?
What range of values appeared in training?
Are we interpolating or extrapolating?
What loss function was used?
What does a large error mean in practice?
How large are the residuals?
Do residuals contain patterns?
How uncertain are the predictions?
Does the model generalise to new data?
Has the underlying system changed?
Which features drive the prediction?
Are those contributions predictive or causal?
What decision will use the output?

Regression is simple enough to understand clearly, but rich enough to expose many of the central problems of machine learning.

The central idea

Regression predicts:

numerical quantities.

At its simplest:

INPUT x

↓

REGRESSION MODEL

↓

PREDICTED NUMBER ŷ

The model learns from historical examples:

(x₁, y₁)

(x₂, y₂)

(x₃, y₃)

and attempts to learn a relationship that generalises to new observations.

But regression is more than fitting a line.

It introduces several of the deepest ideas in machine learning:

DATA

↓

LEARN RELATIONSHIP

↓

PREDICT

↓

MEASURE ERROR

↓

ADJUST MODEL

↓

GENERALISE

And it reminds us of an important distinction:

A model can predict a relationship without explaining why that relationship exists.

Regression gives us one major family of prediction problems:

What numerical value should we expect?

The next family asks a different question:

Which category does this observation belong to?

That is classification.