Risk versus uncertainty

We often use the words risk and uncertainty as though they mean the same thing.

They are related.

But it is useful to distinguish them.

A simple way to think about the difference is:

Risk is uncertainty we can describe reasonably well with probabilities.

Uncertainty is broader and includes situations where even the probabilities themselves are unclear.

This distinction matters because different kinds of unknowns require different kinds of decisions.

Sometimes we know the possible outcomes and can estimate how likely they are.

Sometimes we do not.

A simple example of risk

Suppose a machine has been operating for many years.

Historical data suggests:

1% probability of failure during the next month.

We understand:

what failure means,
how often it has happened,
roughly what conditions influence it.

This is a risk.

The future is uncertain.

But the uncertainty can be quantified reasonably well.

A different kind of uncertainty

Now suppose we introduce an entirely new type of machine.

It uses:

a new material,
a new operating regime,
a new control system.

There is almost no relevant historical data.

Someone asks:

What is the probability it fails during the next month?

We could produce a number.

But how meaningful would it be?

The deeper problem is that we are uncertain about:

the failure mechanisms,
the appropriate model,
whether historical analogies apply.

This is a more fundamental form of uncertainty.

Known probabilities versus uncertain probabilities

Consider two situations.

Situation A

We estimate:

P(failure) = 5%

and have strong evidence supporting that estimate.

Situation B

One plausible model says:

P(failure) = 1%

another says:

10%

and another says:

40%.

In Situation B, the outcome is uncertain.

But so is the probability model itself.

That distinction matters enormously.

Risk lives inside a model

When we describe something as a risk, we usually have:

a set of possible outcomes,
some probability distribution over them,
consequences attached to those outcomes.

Conceptually:

POSSIBLE OUTCOMES

PROBABILITIES

CONSEQUENCES

↓

RISK

For example:

1% chance of a €1 million loss

is a risk statement.

Uncertainty can sit outside the model

But what if we do not know:

all possible outcomes,
the correct probabilities,
whether the model itself is appropriate?

Then uncertainty exists at a deeper level.

We might ask:

Are there failure modes we have not considered?

Could the environment change in a way the model does not represent?

Is the historical probability still relevant?

These questions cannot always be answered by simply calculating a larger probability table.

Frank Knight and the classic distinction

Economist Frank Knight famously distinguished between:

measurable uncertainty

and:

unmeasurable uncertainty.

The first became associated with risk.

The second with uncertainty.

The terminology is not used identically everywhere.

But the distinction is useful:

RISK → probability model reasonably known

UNCERTAINTY → probability model itself uncertain

A coin flip is mostly risk

Suppose we flip a fair coin.

We know the possible outcomes:

Heads

or:

Tails.

We assign:

P(Heads) = 0.5

P(Tails) = 0.5.

We do not know which outcome will happen.

But the uncertainty is well characterised.

This is a classic risk problem.

A new pandemic is much more uncertain

Now imagine the first weeks of a newly emerging disease.

We may not know:

transmission rate,
mortality,
long-term effects,
mutation rate,
population susceptibility.

Assigning precise probabilities is much harder.

New observations arrive continuously.

Models disagree.

The uncertainty is not only:

Which outcome will occur?

It is also:

What model of the process should we believe?

Risk is often aleatoric

Recall the earlier distinction between:

aleatoric uncertainty

and:

epistemic uncertainty.

Risk often corresponds closely to aleatoric uncertainty:

We understand the process reasonably well, but outcomes remain variable.

For example:

equipment failure,
insurance claims,
customer arrivals.

We know the distribution reasonably well.

Individual outcomes remain unpredictable.

Deeper uncertainty is often epistemic

Uncertainty may instead arise because:

we do not know enough.

Perhaps:

data is scarce,
models disagree,
the environment is novel.

This is closer to epistemic uncertainty.

Additional:

data,
experiments,
research

may reduce it.

But sometimes even the right questions are not yet obvious.

Risk can be priced

When probabilities are well understood, risk can often be incorporated into economic decisions.

Insurance is an obvious example.

Suppose an insurer estimates:

expected annual loss = €500 per customer.

It can use:

expected loss,
variance,
tail risk

to help determine a premium.

The uncertainty has been translated into a price.

Deep uncertainty is harder to price

Now imagine insuring a completely new technology for which:

failure rates are unknown,
losses could be extremely large,
historical evidence is weak.

The insurer may:

charge a large premium,
limit coverage,
refuse to insure.

Why?

Because it is not simply facing known risk.

It is uncertain about the risk model itself.

This distinction matters for investment

Suppose an investor considers two projects.

Project A

Historical data allows a reasonably good estimate of:

revenue,
costs,
failure probabilities.

Project B

Depends on a completely new technology and unknown regulation.

Project B may have an attractive expected value.

But that calculation rests on much weaker assumptions.

Investors often demand higher returns for uncertainty that is difficult to quantify.

Expected value works best when probabilities are meaningful

Suppose we calculate:

Expected loss = probability × consequence.

That only helps if the probability estimate has some credibility.

If:

P(loss)

is itself wildly uncertain, the expected value can create an illusion of precision.

For example:

Expected loss = €2.73 million

may look very precise.

But perhaps the underlying failure probability could plausibly vary by a factor of ten.

The decimal places are meaningless.

Model risk

This leads to model risk.

Suppose our calculations are internally correct.

But the model assumptions are wrong.

For example:

wrong distribution,
wrong correlations,
wrong causal relationships.

Then the model may produce highly precise but misleading risk estimates.

The risk calculation is only as good as the model underneath it.

Financial models learned this painfully

Financial systems may estimate:

probability of default,
portfolio losses,
market volatility.

These calculations can work well during ordinary conditions.

But correlations can change sharply during crises.

Assets assumed to be diversified may suddenly fall together.

The historical risk model can fail exactly when it matters most.

This is uncertainty about the stability of the model.

Correlations are not fixed laws

Suppose historically two assets have weak correlation.

A portfolio model assumes:

They provide diversification.

Then a crisis occurs.

Investors sell both simultaneously.

Correlation rises sharply.

The historical relationship was real.

It was not permanent.

This is another form of distribution shift.

Tail risk versus deep uncertainty

Tail risk means:

We understand the distribution reasonably well, but extreme outcomes exist in the tail.

Deep uncertainty means:

We may not even understand the tail correctly.

These are different problems.

1-in-100 event

is a risk if we can estimate that frequency credibly.

But if the true frequency could be:

1-in-20

or:

1-in-10,000,

we are much less certain about the risk itself.

Rare events are difficult to estimate

Suppose a catastrophic event has happened:

twice in 100 years.

What is its true probability?

Perhaps approximately:

2% per year.

But the sample is tiny.

And perhaps:

climate,
technology,
infrastructure

have changed during those 100 years.

Historical frequency may provide weak evidence about the future.

Rare-event probabilities often contain substantial epistemic uncertainty.

Absence of evidence is not evidence of impossibility

Suppose a system has operated for:

10 years

without catastrophic failure.

Can we conclude:

Catastrophic failure is impossible?

No.

Perhaps it is:

extremely rare,
triggered only by conditions not yet experienced.

Historical survival provides evidence.

It does not prove safety.

Unknown unknowns

Some uncertainties are not represented in the model at all.

Imagine designing a system with failure scenarios:

The actual failure occurs through:

Scenario D

which nobody considered.

No probability was assigned to D.

This is an unknown unknown.

The problem is not that D had low probability in the model.

It was outside the model.

Probability cannot represent what we never considered

A probability distribution must be defined over some set of possibilities.

If the possibility space excludes an important event, the probability model can still sum perfectly to:

100%.

It may still be incomplete.

This is an important limitation:

A complete probability distribution is only complete relative to the modelled possibility space.

The map can be probabilistically precise and still incomplete

Suppose a model says:

State A: 60%

State B: 30%

State C: 10%

The probabilities sum to:

100%.

That looks complete.

But if the true state is:

State D

then the entire model was misspecified.

Again:

the map is not the territory.

Scenarios can help with deep uncertainty

When probabilities are unreliable, one alternative is to use scenario analysis.

Instead of saying:

Scenario A has exactly 63.7% probability,

we might consider:

low-demand future,
medium-demand future,
high-demand future,
structural-break future.

Then ask:

How does our decision perform under each?

This can be more honest when precise probabilities are not justified.

Scenario analysis separates possibility from probability

A scenario does not necessarily claim:

This future has probability X.

It says:

This is a plausible future worth considering.

This is useful when we know enough to imagine possible states but not enough to estimate their probabilities confidently.

Stress testing

Another technique is stress testing.

Instead of asking:

What is the most likely future?

we ask:

What happens if conditions become much worse than expected?

For example:

demand increases by 30%,
a major generator fails,
interest rates jump,
several supply chains fail simultaneously.

Stress tests reveal how systems behave under difficult scenarios.

Stress testing is not forecasting

A stress test does not necessarily mean:

We think this scenario will happen.

It asks:

If it did happen, would the system survive?

This is a different question from prediction.

It is about resilience.

Robustness

A robust decision performs reasonably well across many plausible conditions.

Suppose Action A is optimal if one particular forecast is exactly correct but terrible otherwise.

Action B is slightly less efficient under the central forecast but works acceptably across many scenarios.

Under deep uncertainty, Action B may be preferable.

This is robust decision-making.

Optimisation can become fragile

Suppose we optimise a system perfectly for:

expected demand = 100 units.

We remove every piece of spare capacity.

The system becomes extremely efficient.

Then demand becomes:

110.

The system fails.

Optimising tightly around one assumed future can create fragility.

Resilience is different from efficiency

Efficiency asks:

How well do we use resources under expected conditions?

Resilience asks:

How well do we cope when conditions differ from expectations?

A system can be:

highly efficient

and:

extremely fragile.

Deep uncertainty makes resilience valuable.

Safety margins

Engineering often responds to uncertainty using safety margins.

Suppose calculations suggest a structure needs to withstand:

100 units of load.

Designers may build it for significantly more.

Why?

Because:

loads are uncertain,
materials vary,
models are imperfect,
unexpected conditions occur.

Safety margins acknowledge that our calculations are not reality.

Redundancy

Another response is redundancy.

Instead of relying on one component, use:

two,
three,
backup systems.

If one fails, another can take over.

Redundancy may appear inefficient under normal conditions.

Under uncertainty, it can be highly valuable.

Optionality

Another useful concept is optionality.

Suppose we are uncertain about future demand.

Instead of building one enormous fixed system immediately, we might design something that can expand later.

This preserves options.

Under deep uncertainty, flexibility can have value even if it looks more expensive initially.

Reversibility

Suppose we are uncertain whether a policy will work.

A reversible decision allows us to:

try,
observe,
update,
change course.

An irreversible decision requires much greater confidence.

So uncertainty should influence not only:

what action we take

but also:

how reversible that action is.

Explore before committing

Suppose two technologies might solve a problem.

Evidence is weak.

Instead of committing all resources immediately, we might:

run pilots,
collect evidence,
update beliefs.

This is another response to epistemic uncertainty.

The decision buys information.

The value of information increases under uncertainty

Suppose two possible actions have almost identical expected value.

But we are highly uncertain about which is actually better.

A new experiment could resolve much of the uncertainty.

That information may be extremely valuable.

Under risk with well-known probabilities, additional information may add less.

Uncertainty can be reduced through learning

Epistemic uncertainty can sometimes be reduced by:

collecting data,
improving sensors,
conducting experiments,
building better models.

This gives us an active strategy:

UNCERTAINTY

↓

WHAT DO WE NEED TO LEARN?

↓

COLLECT INFORMATION

↓

UPDATE

↓

REDUCED UNCERTAINTY

But some uncertainty is irreducible

Suppose human behaviour tomorrow depends on millions of individual choices.

Even with excellent models, some uncertainty remains.

This is aleatoric uncertainty.

More information may reduce uncertainty somewhat.

But not eliminate it.

This is why intelligent systems must learn to operate under uncertainty rather than waiting for perfect knowledge.

Social systems can contain especially deep uncertainty.

Predictions can change behaviour.

Policies can change incentives.

People respond strategically.

New information spreads.

So the system being forecast may react to the forecast itself.

This makes probability distributions less stable.

Reflexivity creates model uncertainty

Suppose a model predicts:

Asset price will rise.

Many traders act on the prediction.

The price rises.

Or perhaps everyone expects others to buy and buys first.

The forecast changes the system.

The data-generating process is now partly endogenous to the prediction.

This makes historical risk estimates harder to interpret.

AI can create new uncertainty

AI does not only reduce uncertainty through prediction.

It can create new uncertainties.

For example:

new forms of cyberattack,
unexpected interactions between agents,
labour-market changes,
new social behaviours.

A new technology changes the system we are trying to model.

The model may therefore make its own historical training data less relevant.

Autonomous agents create strategic uncertainty

Suppose many AI agents:

trade,
negotiate,
allocate resources.

Each agent's behaviour depends on what it expects other agents to do.

Now uncertainty concerns not only:

physical randomness

but:

strategic responses.

This is closer to game theory than simple probability.

Risk and uncertainty in robotics

Suppose an autonomous vehicle detects a pedestrian.

Risk

It estimates:

5% probability the pedestrian steps into the road.

Deeper uncertainty

The model has never encountered:

this environment,
this type of behaviour.

It may not trust the 5% estimate.

The correct response should account for both.

A confidence interval around risk

Instead of:

P(event) = 5%

we might effectively believe:

The probability is probably somewhere between 2% and 20%.

Now we have uncertainty about the risk estimate itself.

This is often a more realistic representation.

A distribution over probabilities

Bayesian approaches can represent uncertainty about parameters.

Suppose:

p = probability of failure.

Instead of assuming:

p = 0.05

we may have a distribution over p.

Perhaps values around:

0.05

are most plausible, but:

0.02–0.10

remain possible.

This represents epistemic uncertainty about the risk.

Confidence should reflect model uncertainty

Suppose a system says:

Failure probability = 4.7%.

That number can appear authoritative.

But if the model is uncertain, the interface should not hide that.

We might instead communicate:

Estimated failure probability around 5%, but evidence is limited.

This is more informative.

Precision should match knowledge

Under deep uncertainty, reporting:

6.374%

may be absurd.

Perhaps the evidence supports only:

low

moderate

or:

somewhere between 2% and 20%.

False precision creates false confidence.

Risk thresholds depend on confidence in the risk estimate

Suppose a safety system acts when:

risk > 10%.

The model estimates:

8%.

If that estimate is very precise, perhaps no action is needed.

But if the plausible range is:

2–30%,

the decision is much harder.

Decision systems should consider uncertainty around risk estimates, not merely point probabilities.

Precaution

When:

consequences are severe,
uncertainty is high,

decision-makers may act cautiously even without high estimated probability.

This is the intuition behind a precautionary approach.

The logic is:

Lack of certainty does not necessarily justify inaction when downside could be catastrophic.

But excessive precaution can also create enormous costs.

Balance is required.

Different domains tolerate uncertainty differently

Consider:

Nuclear safety

Some outcomes are catastrophic.

The same level of model uncertainty may be acceptable in one domain and unacceptable in another.

Risk appetite

Organisations often define a risk appetite.

This describes the amount or type of risk they are willing to accept.

For example:

a start-up may accept large financial uncertainty,
an airline has extremely low tolerance for safety risk.

Risk appetite turns probability and consequence into governance.

Risk tolerance is a design choice

There is no universal threshold saying:

1% risk is acceptable.

One percent risk of:

slightly delayed delivery

is very different from:

fatal system failure.

Acceptability depends on:

consequence,
frequency,
who bears the risk,
alternatives.

Who bears the uncertainty?

Suppose an AI service improves expected efficiency.

But uncertainty is transferred onto users.

For example:

company saves capacity,
customers experience unreliable access.

The organisation receives the upside.

Users bear the tail risk.

This becomes a fairness issue.

Expected efficiency can hide unequal risk

Suppose a shared resource system has:

excellent average performance.

But shortages consistently fall on:

the same users.

Aggregate risk may appear low.

Individual risk may be concentrated.

So we should ask:

Risk for whom?

A policy might have:

positive expected social value.

But perhaps one vulnerable group faces:

large downside risk.

The average can hide this.

Decision-making should consider:

expected outcomes,
uncertainty,
distribution across people.

Insurance redistributes risk

Insurance provides an interesting example.

An individual faces uncertain loss.

Many individuals pool risk.

The insurer absorbs variability and charges a predictable premium.

So insurance converts:

uncertain large individual loss

into:

more predictable small payment.

Risk can therefore be redistributed across a system.

Shared resources can pool uncertainty too

Suppose one household's electricity demand is highly uncertain.

Across millions of households, some variation cancels.

The shared grid pools demand uncertainty.

Likewise:

shared transport,
cloud computing,
healthcare systems

can pool uncertain demand.

This is one reason shared infrastructure can be efficient.

But common shocks remain

Pooling works best when individual risks are not perfectly correlated.

If everyone needs the resource simultaneously, pooling helps much less.

Examples include:

heating during extreme cold,
hospital demand during a pandemic,
computing demand during a major event.

Common shocks create systemic risk.

Systemic risk

Systemic risk describes the possibility that problems spread across or affect the whole system.

Examples include:

financial contagion,
widespread grid failure,
supply-chain collapse.

These risks are difficult because components are connected.

The failure of one part can alter the probability of failure elsewhere.

Networks create dependencies

Suppose each component has low individual failure risk.

But components depend on one another.

A failure in one node creates overload elsewhere.

Now system risk cannot be estimated by examining components independently.

We need the network structure.

This connects probability back to:

space

and:

connectivity.

The distribution itself can depend on system state

Suppose an electricity line becomes congested.

Failure probabilities elsewhere may increase.

Risk is not fixed.

It changes with:

state.

So we might need:

P(Failure | Current state)

rather than one universal failure probability.

Risk becomes dynamic.

Risk can be controlled

Actions can change the distribution of outcomes.

For example:

add reserve capacity

→ lower shortage risk.

slow vehicle

→ lower collision risk.

diversify portfolio

→ reduce financial risk.

So risk is not only something we observe.

It is something we can influence through decisions.

Risk as a control objective

Suppose a service aims to maintain:

P(shortage) < 1%.

Now risk becomes a constraint.

The optimisation problem becomes:

Minimise cost while keeping shortage probability below the acceptable threshold.

This is a chance constraint.

It combines probability with optimisation.

Chance constraints

A chance constraint might conceptually say:

P(demand ≤ capacity) ≥ 99.9%.

Rather than requiring demand to be below capacity under every imaginable scenario, we set a reliability requirement.

This can provide a practical balance between:

cost,
uncertainty,
reliability.

But chance constraints need trustworthy probabilities

If the probability model is wrong, a:

99.9% reliability constraint

may not actually produce 99.9% reliability.

This brings us back to calibration.

Risk-based design requires:

calibrated probabilities,
stable distributions.

Under deep uncertainty, additional robustness may be needed.

Robust optimisation

Instead of assuming one precise probability distribution, robust optimisation may consider a set of plausible possibilities.

The system chooses a decision that performs acceptably across that set.

Conceptually:

NOT ONE PERFECT MODEL

but:

MANY PLAUSIBLE MODELS

↓

FIND A DECISION THAT WORKS ACROSS THEM

This can protect against model uncertainty.

Distributionally robust optimisation

A related idea is distributionally robust optimisation.

Instead of assuming one exact probability distribution, we allow:

a family of plausible distributions.

We then optimise against uncertainty about the distribution itself.

This sits naturally between:

risk

and:

deep uncertainty.

Minimax decisions

One extreme approach is:

Choose the action whose worst-case outcome is best.

This is a minimax idea.

It can be appropriate for catastrophic-risk situations.

But it may be overly conservative when worst cases are extremely implausible.

Again, there is no universal decision rule.

Regret

Another useful idea is regret.

Suppose we choose Action A.

Later, after the future becomes known, we discover Action B would have been better.

Regret measures the difference.

Under uncertainty, we may choose decisions that minimise:

maximum regret

rather than maximise expected value.

This can be useful when probabilities are unreliable.

Adaptive decisions

Another strategy is not to make every decision now.

Suppose we can:

make a small initial decision,
observe what happens,
update,
make the next decision.

This is adaptive decision-making.

It is especially valuable under uncertainty.

The future reveals information

Suppose we are planning infrastructure for:

30 years.

Demand is highly uncertain.

Instead of committing all capacity today, we can design expansion stages.

As demand becomes observable:

UPDATE

and:

BUILD MORE IF NEEDED.

The ability to adapt has value.

Real options

Finance and investment theory sometimes describe this as real options.

The option to:

wait,
expand,
abandon,
switch

can have economic value under uncertainty.

Flexibility itself becomes an asset.

AI systems should preserve optionality too

Suppose an AI agent is uncertain about the environment.

A decision that leaves future options open may be preferable to one that commits irreversibly.

For example:

slow down

rather than:

make an aggressive manoeuvre.

The agent trades a small immediate cost for future flexibility.

Irreversibility raises the required evidence

Suppose a decision is easy to undo.

We may act with moderate confidence.

Suppose it is irreversible.

We may require stronger evidence.

This is a useful general principle:

The harder an action is to reverse, the more uncertainty matters.

Risk versus uncertainty in AI development

Suppose we deploy a well-tested image classifier in a familiar environment.

We may have measurable risks:

false positives,
false negatives.

Now consider deploying a highly autonomous general agent capable of interacting with complex systems in ways we have never observed.

The problem contains much deeper uncertainty.

We may not know:

all possible behaviours,
emergent interactions,
long-term consequences.

These are not merely known error rates.

AGI discussions often concern uncertainty more than risk

People sometimes ask:

What is the probability that advanced AI causes catastrophic harm?

Different experts give very different numbers.

The disagreement itself reveals how uncertain the underlying model is.

We do not have repeated historical examples of:

human-level or superhuman general AI deployment.

So these questions involve profound epistemic uncertainty.

A number can hide disagreement

Suppose someone says:

10% probability of catastrophic AI outcome.

That looks like a risk estimate.

But where did 10% come from?

Perhaps:

one expert thinks 1%,
another thinks 50%,
evidence is sparse.

Writing one probability can hide the deeper model uncertainty.

This is why uncertainty about probabilities should sometimes be communicated explicitly.

The same is true of long-term forecasts

Forecasts about:

climate,
technology,
geopolitics,
economic growth

often involve multiple layers of uncertainty.

We may have:

uncertainty inside each model

plus:

uncertainty about which model is appropriate.

Long horizons amplify both.

Time expands uncertainty

The further ahead we predict:

more disturbances can occur,
behaviour can change,
technology can change,
institutions can change.

So:

near-term uncertainty

may resemble measurable risk.

long-term uncertainty

may increasingly involve structural uncertainty.

This is why very distant point forecasts should be treated cautiously.

Risk, uncertainty and prediction machines

AI is often described as reducing the cost of prediction.

Better prediction can reduce risk by narrowing distributions.

But no prediction machine can eliminate:

unknown unknowns,
structural change,
model misspecification.

So AI may convert some uncertainty into measurable risk.

That is enormously valuable.

But it does not abolish uncertainty.

Better AI can reveal that uncertainty is larger

Sometimes a better model does not give a narrower forecast.

Instead, it discovers:

We were previously overconfident.

The improved distribution may become wider.

That can look like worse prediction.

In reality, the system is representing uncertainty more honestly.

"I don't know the probability" is a valid state

People often feel pressured to produce a numerical probability.

Sometimes that is useful.

But sometimes the correct answer is:

We do not have enough evidence to estimate this probability reliably.

That is information.

It distinguishes:

quantifiable risk

from:

poorly characterised uncertainty.

Uncertainty should change behaviour

If uncertainty is low and risk is quantified:

optimise confidently.

If uncertainty is high:

preserve margins,
collect more information,
keep options open,
monitor closely,
avoid irreversible actions.

Different epistemic conditions justify different strategies.

A hierarchy of responses

We might think about decisions like this:

Poorly understood, reversible decision

Experiment and learn.

Poorly understood, irreversible high-consequence decision

Use greater caution and robustness.

This is not a universal rule.

But it captures the role uncertainty should play in decision design.

Risk and uncertainty in shared-resource systems

Suppose expected demand for a service is well understood.

Then capacity can be designed using:

demand distribution,
reliability target.

That is largely a risk problem.

Now suppose a new technology causes demand patterns to change unpredictably.

Historical distributions become unreliable.

The service faces deeper uncertainty.

It may need:

more flexibility,
adaptive pricing,
spare capacity,
continuous updating.

Static designs struggle with uncertain futures

Suppose infrastructure is designed once for:

forecast demand in 2040.

If that forecast is wrong, the system may be:

overbuilt,
underbuilt.

A more adaptive design might respond continuously to new information.

This is one reason flexibility has value.

Intelligence can make services adaptive

An intelligent service can:

FORECAST

↓

OBSERVE ACTUAL DEMAND

↓

UPDATE

↓

REALLOCATE

↓

LEARN

This allows the system to cope with uncertainty better than a rigid plan.

Prediction helps.

Feedback helps more.

Risk is not just something to minimise

Sometimes taking risk creates value.

Entrepreneurship involves risk.

Exploration involves risk.

Innovation involves uncertainty.

An intelligent system should not always choose:

lowest variance possible.

The question is:

Which risks are worth taking for which potential benefits?

Exploration deliberately accepts uncertainty

Suppose a recommendation system always selects the option it believes is best.

It may never learn about alternatives.

Exploration deliberately chooses uncertain actions.

That creates short-term risk.

But it generates information.

This can improve future decisions.

Risk and learning are connected

Choosing an uncertain action can reveal:

how the system behaves,
whether assumptions are correct.

So some risk may have informational value.

This is another reason expected immediate value is not the whole decision problem.

Society chooses which risks are acceptable

A mathematical model can estimate:

probabilities,
losses.

It cannot independently decide:

This risk is morally acceptable.

Society must determine:

safety standards,
rights,
acceptable failure rates,
who is allowed to bear risk.

These are governance choices.

Who chooses the risk threshold?

Suppose an autonomous service is designed for:

99% reliability.

Why 99%?

Why not:

95%

or:

99.999%?

The choice has consequences for:

cost,
access,
safety.

There is no purely mathematical answer.

Someone defines the service standard.

Who bears the residual risk?

Even after mitigation, some risk remains.

Suppose a service is:

99.9% reliable.

Who experiences the:

0.1% failures?

If failures are distributed randomly, that is one service.

If the same people repeatedly experience them, that is another.

Risk distribution matters.

Risk can accumulate over time

Suppose a failure probability is:

1% per use.

One use may seem safe.

But a user relies on the system:

1,000 times.

The cumulative chance of experiencing at least one failure can become substantial.

Per-event risk and lifetime risk are different.

Small AI error rates can become large at scale

Suppose an AI has:

0.1% serious error rate.

At:

1,000 predictions

that may mean roughly one serious error.

At:

one billion predictions

it could mean around:

one million errors,

if conditions are comparable.

Scale transforms small probabilities into large numbers of events.

System risk depends on exposure

So a risk statement should ask:

probability per what?
per prediction?
per user?
per year?
across how many opportunities?

Probability without exposure can be misleading.

The central framework

We can now separate several layers:

POSSIBLE OUTCOMES

↓

PROBABILITY DISTRIBUTION

↓

RISK

But also:

UNCERTAINTY ABOUT DATA

UNCERTAINTY ABOUT PARAMETERS

UNCERTAINTY ABOUT MODEL

UNKNOWN POSSIBILITIES

↓

DEEPER UNCERTAINTY

Decision-making then has to respond to both.

Ask what kind of unknown you face

Whenever someone presents a risk estimate, ask:

What are the possible outcomes?
How were their probabilities estimated?
How much data supports those probabilities?
Are the probabilities calibrated?
Could the distribution shift?
How uncertain are the model parameters?
Do plausible models disagree?
Could important outcomes be missing from the model?
What happens in the tails?
Who experiences the downside?
Can the action be reversed?
Can we gather more information first?
Would a robust decision perform better across plausible futures?

These questions help distinguish measurable risk from deeper uncertainty.

The central idea

Risk and uncertainty are related but not identical.

Risk means we do not know which outcome will occur, but we have a reasonably meaningful model of the possibilities and their probabilities.

Uncertainty includes the harder situations where:

probabilities are poorly known,
models disagree,
the system may change,
important possibilities may not even have been represented.

So:

RISK

can often be managed using:

probabilities,
expected values,
variance,
insurance,
optimisation.

DEEP UNCERTAINTY

may require:

robustness,
flexibility,
safety margins,
scenarios,
experimentation,
adaptation.

The harder it is to know the probability distribution, the less sensible it becomes to optimise as though that distribution were certain.

This is an important principle for AI.

A model may always produce a number.

That does not mean the number captures everything we do not know.

Good intelligent systems should therefore distinguish:

I know the risk

from:

I am uncertain about the risk itself.

In the next lesson, we will look more closely at the part of probability distributions where many important risks hide:

rare events and tails — why low-probability outcomes can dominate real-world decisions, why extreme events are difficult to learn from data, and why scale makes rare failures much less rare in practice.