Enleashed

Conditional probability tells us that probabilities change when information changes.

Bayesian reasoning turns that into a systematic process for learning.

The basic idea is simple:

Start with what you believed before, observe new evidence, and update what you believe now.

This gives us a powerful framework:

PRIOR BELIEF

↓

NEW EVIDENCE

↓

UPDATED BELIEF

In Bayesian language:

PRIOR

↓

LIKELIHOOD

↓

POSTERIOR

Bayesian reasoning is therefore a formal way of learning from evidence under uncertainty.

Why Bayesian reasoning matters

In the real world, we rarely begin with no information.

A doctor already knows how common a disease is before seeing a test result.

A robot already has an estimate of its position before receiving a new sensor measurement.

A weather model already has a forecast before new satellite data arrives.

An AI system may already have a belief about what comes next before another observation is added to the context.

Bayesian reasoning tells us how to combine:

what we believed before

with:

what the new evidence tells us.

The prior

The prior represents what we believe before seeing the new evidence.

Suppose a disease affects:

1% of the population.

Before seeing any test result, we might write:

P(Disease) = 0.01

This is the prior probability.

It does not mean the patient is 1% diseased.

The patient either has the disease or does not.

The 1% represents our uncertainty before seeing additional evidence.

The likelihood

The likelihood describes how probable the observed evidence would be under a particular hypothesis.

Suppose a medical test is positive.

We might know:

P(Positive test | Disease)

This asks:

If the patient really has the disease, how likely is a positive test?

Perhaps the answer is:

95%.

The likelihood tells us how compatible the evidence is with the hypothesis.

The posterior

After observing the positive test, we want:

P(Disease | Positive test)

This is the posterior probability.

It represents our updated belief after incorporating the evidence.

The posterior becomes the new state of knowledge.

If more evidence arrives later, today's posterior can become tomorrow's prior.

Bayes' theorem

Bayes' theorem links these quantities:

P(A | B) = P(B | A) × P(A) / P(B)

In words:

Posterior = likelihood × prior, adjusted by how common the evidence is overall.

The formula matters.

But the deeper idea matters more:

Evidence should change our beliefs in proportion to how strongly that evidence favours one explanation over another.

A medical example

Suppose:

1% of people have a disease.

A test has:

90% sensitivity

meaning:

P(Positive | Disease) = 0.90

Suppose it also has a:

5% false-positive rate

meaning:

P(Positive | No disease) = 0.05

A patient receives a positive result.

What is:

P(Disease | Positive)?

Many people instinctively answer:

90%.

But that confuses:

P(Positive | Disease)

with:

P(Disease | Positive).

They are different.

Think in populations

Imagine:

10,000 people.

If 1% have the disease:

100 people have it.

Of those, with 90% sensitivity:

90 test positive.

The remaining:

9,900 people do not have the disease.

With a 5% false-positive rate:

495 of them test positive.

So total positive tests are:

90 + 495 = 585.

Of those 585 positive tests:

90 are true positives.

So:

P(Disease | Positive) = 90 / 585

which is only about:

15%.

The positive test raises the probability dramatically from:

to about:

15%.

But it does not make the disease almost certain.

The base rate matters

The reason is the base rate.

The disease was rare to begin with.

Even a fairly accurate test can produce many false positives when applied to a large population where the condition is uncommon.

This is one of the central lessons of Bayesian reasoning:

Evidence should be interpreted relative to what was plausible before the evidence arrived.

Evidence does not exist in isolation

Suppose a smoke alarm activates.

How worried should you be?

That depends on context.

If you are:

cooking,
standing beside a smoking frying pan,

the alarm may provide limited evidence of a house fire.

If you are:

asleep at 03:00,
smell smoke,
hear multiple alarms,

the same alarm signal carries very different implications.

Evidence changes belief relative to prior context.

Prior beliefs are not automatically subjective guesses

The word prior can sound like:

Whatever someone happens to believe.

But priors can come from strong evidence.

For example:

historical failure rates,
population statistics,
previous measurements,
yesterday's state estimate,
known physical constraints.

A robot's prior position estimate may come from its motion model.

A weather model's prior may come from the previous forecast.

A medical prior may come from disease prevalence and patient characteristics.

The prior can be data-driven.

Priors can also be weak

Sometimes we know very little.

In that case, we may use a broad prior that assigns substantial probability across many possibilities.

A broad prior expresses:

We are uncertain.

This is often preferable to pretending we have no assumptions at all.

Every model makes assumptions.

Bayesian methods make some of them explicit.

Strong evidence can overwhelm a weak prior

Suppose we initially think two possibilities are roughly equally plausible.

Then we observe very strong evidence favouring one.

The posterior can move dramatically.

Conceptually:

WEAK PRIOR

STRONG EVIDENCE

↓

POSTERIOR DOMINATED BY EVIDENCE

This is what we would hope from a learning system.

Strong priors require stronger evidence to change

Suppose we are almost certain that a sensor is functioning normally.

One slightly unusual reading may not convince us that it is broken.

But repeated extreme readings may.

The stronger the prior belief, the more evidence is generally required to move the posterior substantially.

This matches everyday reasoning.

Extraordinary claims and prior probability

Suppose someone says:

I saw a dog outside.

That claim has a relatively high prior probability.

Now suppose they say:

I saw a dragon outside.

The sensory report may be equally confident.

But the prior plausibility of the two explanations is very different.

Bayesian reasoning naturally captures this.

Unusual claims require stronger evidence because they begin with much lower prior probability.

Evidence is informative when alternatives predict it differently

Suppose two hypotheses are:

H1: machine is healthy

H2: machine is failing

We observe:

temperature = 90°C.

If:

90°C is common when failing

but:

extremely rare when healthy,

then the observation strongly favours H2.

But suppose both healthy and failing machines commonly operate at 90°C.

Then the observation provides little useful information.

Evidence is valuable when competing hypotheses predict it differently.

Likelihood ratios

One way to think about evidence is to compare:

How likely is this observation if H1 is true?

with:

How likely is it if H2 is true?

If the evidence is:

100 times more likely under H2

than under H1, it strongly shifts belief towards H2.

This comparison is sometimes expressed through a likelihood ratio.

The exact mathematics can wait.

The conceptual idea is powerful.

Bayesian reasoning is sequential

Suppose a robot is trying to locate itself.

At time 1:

Prior position belief

Then GPS arrives:

Update

Then camera data arrives:

Update again

Then wheel movement occurs:

Predict forward

Then another GPS reading arrives:

Update again

The process is:

BELIEF

↓

EVIDENCE

↓

UPDATED BELIEF

↓

NEW EVIDENCE

↓

UPDATED BELIEF

Bayesian reasoning is naturally suited to systems that continuously receive information.

Posterior becomes prior

This is one of the most elegant ideas.

Today's posterior becomes tomorrow's prior.

For example:

Prior at t

Observation at t

↓

Posterior at t

Then:

Posterior at t

becomes:

Prior at t+1

This creates a continuous learning loop.

Bayesian state estimation

Suppose a robot's position is uncertain.

Before a new sensor reading, it has a distribution over possible positions.

That is its prior.

A sensor reading arrives.

The robot asks:

How likely would I be to observe this measurement if I were at each possible position?

That is the likelihood.

It combines prior and likelihood to produce:

posterior position distribution.

The robot now has an updated estimate of where it is.

Prediction and correction

State estimation often alternates between two steps.

Prediction

Use:

previous state,
actions,
system dynamics

to predict where the system should be now.

Correction

Use:

new measurements

to update that prediction.

So:

PRIOR STATE ESTIMATE

↓

DYNAMICS

↓

PREDICTED STATE

↓

OBSERVATION

↓

BAYESIAN UPDATE

↓

POSTERIOR STATE ESTIMATE

This idea underlies many robotics and control methods.

The Kalman filter is Bayesian in spirit

The Kalman filter is a famous state-estimation method.

Under particular assumptions, it provides an elegant way of combining:

model prediction

with:

noisy measurements.

If the sensor is very accurate, the update gives it more weight.

If the sensor is noisy, the system relies more heavily on the prior model estimate.

Conceptually:

Trust evidence according to how reliable it is.

That is deeply Bayesian.

A noisy sensor should cause a smaller update

Suppose a robot thinks it is at:

x = 10

with high confidence.

A very noisy GPS measurement says:

x = 20.

Should the robot immediately jump its estimated position to 20?

Probably not.

The measurement is weak evidence.

Now suppose an extremely accurate positioning system says:

x = 20.

The same value provides much stronger evidence.

The posterior depends on both:

the observation,
the uncertainty associated with that observation.

Bayesian reasoning preserves uncertainty

The result of an update is not necessarily:

state = exactly x.

It may be:

most likely state = x

with:

reduced but non-zero uncertainty.

This is important.

Observations do not automatically create certainty.

They refine a distribution.

Multiple observations can narrow uncertainty

Suppose several independent sensors all suggest approximately the same position.

The posterior distribution may become narrower.

We become more confident.

Conceptually:

BROAD PRIOR

↓

OBSERVATION 1

↓

NARROWER POSTERIOR

↓

OBSERVATION 2

↓

NARROWER AGAIN

Evidence accumulates.

Conflicting evidence can preserve uncertainty

Suppose one sensor says:

Position A

and another equally reliable sensor says:

Position B.

The posterior may remain broad or multimodal.

A good system should not force certainty where evidence is contradictory.

It may represent:

A and B are both plausible.

This is a better response than arbitrarily selecting one.

Bayesian reasoning and missing data

Suppose a sensor goes offline.

No new evidence arrives.

The system can still predict forward using:

its prior state,
system dynamics.

But uncertainty grows.

When the sensor returns, new evidence updates the belief.

This is exactly the framework we developed earlier:

NO OBSERVATION

→ prediction continues

→ uncertainty increases

NEW OBSERVATION

→ posterior updates

→ uncertainty may decrease

Bayesian reasoning and hidden states

Many important states cannot be observed directly.

For example:

health condition,
machine degradation,
user preference,
economic regime.

We instead observe signals.

Bayesian reasoning allows us to infer:

P(Hidden state | Observations).

This is central to many intelligent systems.

A user's preference is hidden

Suppose a recommendation system wants to know whether a user likes science fiction.

It cannot directly observe:

preference = 0.83.

Instead it observes:

clicks,
watch time,
ratings,
skipped content.

Each observation provides evidence.

The system updates its belief about the user's hidden preferences.

But observed behaviour is conditioned by what was shown

This requires care.

Suppose the user only receives science-fiction recommendations.

They click some of them.

The system may conclude:

The user strongly prefers science fiction.

But it never tested alternatives.

The evidence was generated by the recommendation policy itself.

Bayesian updating is only as good as the model of how evidence was generated.

The likelihood model matters

Suppose the system asks:

P(click | user likes science fiction).

But perhaps clicks also depend on:

thumbnail,
time of day,
placement,
price.

If the likelihood model ignores these factors, the posterior may be misleading.

Bayes' theorem does not fix a badly specified model.

Good updating still requires a good model of how observations arise.

Bayesian reasoning and causality are different

Bayesian reasoning tells us how to update beliefs given evidence.

It does not automatically tell us whether one variable causes another.

Suppose:

P(Disease | Symptom)

is high.

That does not necessarily mean:

Symptom causes Disease.

Conditional probability and causal reasoning remain distinct.

Bayes helps with inference.

Causality helps with intervention.

Bayesian inference

Suppose we have an unknown parameter:

Perhaps θ represents:

failure probability,
average demand,
effectiveness of a treatment.

We begin with a prior distribution:

P(θ).

Then observe data:

We update to:

P(θ | D).

This is Bayesian inference.

Instead of producing one fixed estimate of θ, we obtain a probability distribution describing what values remain plausible.

Parameters can be uncertain too

Traditional explanations sometimes talk as though model parameters are fixed numbers we simply need to discover.

Bayesian reasoning instead allows us to represent uncertainty about them.

For example:

Failure probability is probably around 2%, but values between 1% and 4% remain plausible.

This can be more informative than:

failure probability = 2.137%.

But only if the data is informative

Suppose we want to learn tomorrow's weather.

We collect one million observations of:

shoe colour.

More data does not help if the evidence is unrelated to the uncertain quantity.

The value of information depends on how strongly the evidence changes the posterior.

Bayesian reasoning and machine learning

Machine learning can often be interpreted through a Bayesian lens.

We have:

training data

and want to infer:

model parameters,
predictions,
hidden representations.

A Bayesian approach may represent uncertainty over:

possible models or parameters

rather than selecting only one.

Predictions then average over that uncertainty.

Bayesian model averaging

Suppose:

Model A

and:

Model B

both plausibly explain the data.

Instead of selecting one model with certainty, we can assign probabilities to them.

Then predictions account for:

P(Model A | Data)

and:

P(Model B | Data).

This is Bayesian model averaging.

It allows model uncertainty to enter predictions.

Deep learning is usually not fully Bayesian

Modern neural networks are often trained to produce one set of parameters.

They are not automatically maintaining a full probability distribution over every possible model.

But Bayesian ideas still influence approaches to:

uncertainty estimation,
probabilistic neural networks,
ensembles,
Bayesian optimisation.

The conceptual framework remains useful even when the implementation differs.

Bayesian reasoning and language models

Consider:

The capital of France is ...

The context acts as evidence.

The model has learned prior statistical structure from training data.

Given the current context, it produces a distribution over next tokens.

Conceptually, this resembles:

PRIOR LANGUAGE STRUCTURE

CURRENT CONTEXT

↓

UPDATED DISTRIBUTION OVER NEXT TOKEN

This is not exactly classical Bayesian inference internally.

But the high-level pattern of:

prior knowledge + new context → updated prediction

is closely related.

Context continually updates prediction

Suppose we begin:

The bank...

Many continuations are plausible.

Now:

The bank approved my...

Financial meaning becomes more likely.

Then:

The bank approved my mortgage...

The possible continuations narrow further.

Each token changes the conditional distribution.

Evidence accumulates through context.

Bayesian reasoning and AI agents

An AI agent operating in the world may maintain beliefs about:

where it is,
what other agents intend,
whether a resource is available,
whether an action will succeed.

As observations arrive, it updates those beliefs.

The agent then chooses actions based on its posterior state estimate.

So:

OBSERVE

↓

UPDATE BELIEF

↓

PREDICT

↓

DECIDE

↓

ACT

↓

OBSERVE AGAIN

Bayesian reasoning naturally fits into autonomous systems.

Belief states

In a partially observable environment, the agent may not know the true state.

Instead, it maintains a belief state:

A probability distribution over possible states.

For example:

State A: 70%

State B: 20%

State C: 10%

The belief state summarises what the agent currently thinks is plausible.

New observations update the belief.

Belief can become the state used for decisions

This creates an important conceptual shift.

If the true physical state is hidden, the decision-making system cannot condition directly on:

true state.

Instead, it acts based on:

belief about state.

So:

PHYSICAL STATE

↓

PARTIAL OBSERVATIONS

↓

BELIEF STATE

↓

DECISION

The internal state of the intelligent system is therefore probabilistic.

Bayes and diagnosis

Medical diagnosis is a classic Bayesian process.

Start with:

prior probability of disease.

Observe:

symptoms,
blood tests,
scans.

Each piece of evidence updates the probability.

The clinician may end with:

P(Disease | all available evidence).

Then comes a separate question:

What treatment should be given?

Inference and decision remain distinct.

Bayes and fault detection

Suppose an industrial machine normally has a:

1% probability of imminent failure.

Then:

vibration rises,
temperature rises,
power consumption changes.

Each signal may increase failure probability.

Eventually:

P(Failure soon | observations)

may exceed a threshold requiring maintenance.

This is Bayesian reasoning embedded in a service decision.

Thresholds turn posterior belief into action

Suppose:

P(Failure) = 20%.

Should we stop the machine?

There is no universally correct probability threshold.

It depends on:

cost of shutdown,
cost of failure,
safety consequences,
alternatives.

The Bayesian posterior provides information.

The decision rule determines what to do with it.

Again:

prediction ≠ decision.

Bayes and false positives

Suppose a security system observes unusual behaviour.

The evidence may increase the probability of fraud.

But fraud may be rare.

If the system ignores the prior base rate, it may generate enormous numbers of false alarms.

Bayesian reasoning helps prevent:

rare condition + noisy detector → excessive false positives.

This matters in:

medicine,
fraud detection,
cybersecurity,
safety monitoring.

Base-rate neglect

Humans often focus on vivid new evidence and ignore the prior probability.

This is called base-rate neglect.

For example:

The test is 99% accurate, so a positive result means 99% chance of disease.

Not necessarily.

The prevalence matters.

Or:

This behaviour looks suspicious, therefore this person is probably fraudulent.

But if fraud is extremely rare, the base rate still matters.

Bayesian reasoning forces us to combine both.

The opposite problem: ignoring new evidence

We can also over-weight priors.

Suppose a system has historically been extremely reliable.

Then several strong warning signals appear.

Saying:

It almost never fails, so everything is fine

would ignore important new evidence.

Good Bayesian reasoning balances:

prior belief

with:

strength of evidence.

Priors should be updateable

A prior should not be treated as dogma.

It is simply the belief before the current evidence.

If strong evidence arrives, the posterior should move.

This is an important philosophical principle:

Beliefs should be revisable in response to evidence.

Bayesian reasoning formalises that principle.

But evidence must be interpreted through a model

Suppose a sensor reports:

100.

Does that strongly indicate failure?

Only if we understand:

P(sensor reports 100 | failure)

and:

P(sensor reports 100 | healthy).

Evidence has meaning only relative to hypotheses about how the world generates observations.

Data is evidence, not reality

This returns us to Module 2.

A measurement is not the physical state itself.

It is evidence about the state.

Bayesian reasoning formalises the transformation:

OBSERVATION

↓

LIKELIHOOD

↓

BELIEF ABOUT HIDDEN REALITY

So:

data → inference

rather than:

data = truth.

Bayesian reasoning and uncertainty reduction

Suppose our prior distribution is wide.

New evidence arrives.

If the evidence is informative, the posterior may become narrower.

Our epistemic uncertainty has decreased.

But the posterior rarely becomes infinitely narrow.

We still preserve uncertainty.

This is a strength rather than a weakness.

Evidence can increase uncertainty too

New information does not always make us more certain.

Suppose two equally reliable sensors disagree dramatically.

Before seeing them, we had one plausible state.

After seeing conflicting evidence, several explanations become possible:

Sensor A failed,
Sensor B failed,
our model is wrong.

The posterior may become more complicated or even wider.

Learning can reveal that we knew less than we thought.

Bayesian surprise

If an observation was extremely unlikely under our current beliefs, it can be described as surprising.

Suppose a model expects:

P(observation) ≈ 0.0001.

Then the observation occurs.

This suggests something important:

a rare event happened,
our state estimate was wrong,
our model may be wrong.

Surprise can trigger learning or investigation.

Anomalies are evidence

Suppose an AI system encounters a situation unlike its training data.

Instead of confidently forcing it into an existing category, a better system might recognise:

This observation has low probability under my current model.

That is a useful form of uncertainty awareness.

Anomalies can tell us where the model needs updating.

Bayes and model revision

Bayesian reasoning usually assumes a set of possible hypotheses.

But what if every existing hypothesis explains the observation badly?

Perhaps we need a new model.

This connects to:

unknown unknowns.

Probability can compare possibilities we represented.

It cannot assign meaningful probability to possibilities we never considered.

The model class creates a boundary

Suppose our hypotheses are:

Sensor failure,
Normal operation.

But the actual event is:

cyberattack.

If cyberattack was not represented, Bayesian updating can only redistribute belief between the available explanations.

This is a limitation of every model.

Bayesian reasoning is powerful within the possibility space we define. It does not guarantee that the possibility space is complete.

Bayes and scientific reasoning

Science often follows a Bayesian-like pattern.

We have:

existing theories

Then collect:

new observations.

We ask:

How compatible is this evidence with each explanation?

Some theories become more plausible.

Others become less plausible.

New hypotheses may emerge.

Scientific knowledge evolves through repeated updating.

Different priors can converge

Suppose two researchers begin with different priors.

If they observe enough strong and relevant evidence, their posterior beliefs may move closer together.

This is another attractive feature of Bayesian reasoning.

Evidence can overcome initial disagreement.

But convergence depends on:

evidence quality,
model assumptions,
whether both consider the true explanation possible.

Bayesian reasoning does not remove judgement

Someone still has to decide:

which hypotheses to consider,
which prior to use,
how the likelihood is modelled,
what evidence is trustworthy.

The mathematics makes assumptions explicit.

It does not eliminate them.

This is a recurring theme throughout the course.

Bayes and fairness

Suppose an AI estimates:

P(Default | individual information).

Should group-level historical default rates enter the prior?

Perhaps that improves prediction.

But it may also reproduce historical inequalities.

The Bayesian calculation can be mathematically valid while the service-design choice remains ethically contested.

This is another example of:

Mathematics can tell us the consequences of a modelling choice. It cannot automatically tell us whether that choice is fair.

Historical priors can carry historical inequality

Suppose one group historically had higher recorded default rates because of:

poorer access to credit,
unstable employment opportunities,
discriminatory practices.

Using those historical rates as priors may perpetuate the effects of that history.

Again:

historical data contains history.

Bayesian reasoning does not wash that history away.

Bayesian reasoning should not become destiny

A prior is not a fate.

Suppose someone belongs to a group with historically higher risk.

Strong individual evidence may substantially change the posterior.

A well-designed system should update rather than simply assign people the average characteristics of a group.

But which variables should matter remains a service-design question.

Bayes and continuous learning

Imagine a system that continually observes its environment.

At every moment:

PRIOR

OBSERVATION

↓

POSTERIOR

Then:

POSTERIOR → NEXT PRIOR

This allows the model to adapt over time.

The system becomes stateful.

Its beliefs today depend on:

what it believed yesterday,
what happened since.

This connects strongly to the broader course theme of time.

A belief trajectory

Instead of only tracking the physical system through time:

x(1) → x(2) → x(3)

we can also track the system's belief:

P(x₁) → P(x₂ | evidence) → P(x₃ | new evidence).

The intelligent system has its own evolving informational state.

The world and the model evolve together

This gives us two parallel processes:

Physical world

STATE → ACTION → NEW STATE

Model belief

PRIOR → EVIDENCE → POSTERIOR

These interact.

The system acts based on its beliefs.

Its actions change the world.

The changed world produces new evidence.

So:

BELIEF

↓

DECISION

↓

ACTION

↓

WORLD CHANGES

↓

NEW OBSERVATION

↓

BELIEF UPDATE

This is the architecture of many intelligent agents.

Bayesian reasoning and the course framework

We can now enrich our central diagram:

PAST

↓

DATA

↓

PRIOR BELIEF ABOUT PRESENT STATE

↓

NEW OBSERVATION

↓

BAYESIAN UPDATE

↓

POSTERIOR BELIEF

↓

PREDICTION OF POSSIBLE FUTURES

↓

DECISION

↓

ACTION

↓

NEW REALITY

↓

NEW OBSERVATION

↓

UPDATE AGAIN

The system is continuously learning.

Ask Bayesian questions

Whenever new evidence changes a prediction, ask:

What did we believe before?
What is the prior probability?
What evidence arrived?
How likely is that evidence under each hypothesis?
What is the base rate?
How reliable is the evidence?
Are observations independent?
Could we be double-counting evidence?
How should the posterior change?
How uncertain is the posterior?
Does the model include all plausible explanations?
Would more information materially change the decision?

These questions encourage disciplined updating.

The central idea

Bayesian reasoning is a formal framework for learning from evidence.

It begins with:

what we currently believe

then incorporates:

what we have just observed

to produce:

what we should believe now.

In shorthand:

PRIOR

↓

EVIDENCE

↓

POSTERIOR

And then:

POSTERIOR → NEXT PRIOR.

This makes Bayesian reasoning naturally suited to a changing world.

Intelligence is not having one permanently correct model of reality. It is maintaining beliefs that can change when evidence changes.

Bayesian reasoning therefore connects:

past knowledge

to:

present evidence

to:

updated beliefs about the future.

In the next lesson, we will focus more closely on the first part of that process:

prior knowledge and new evidence — where priors come from, when they are useful, when they can become dangerous, and how intelligent systems should balance history against what is happening now.