Enleashed

Real-world datasets are rarely complete.

A sensor goes offline.

A customer does not answer a question.

A satellite cannot see through cloud.

A measurement is taken every five minutes rather than continuously.

A hospital test is only performed when a clinician thinks it is necessary.

A device loses its network connection.

A database begins recording a variable only halfway through its history.

Something happens in a place where nobody was measuring it.

The result is missing data.

It is tempting to think of missing data as evidence that something has gone wrong.

Sometimes it is.

But often it is simply a normal consequence of trying to observe a complex world using finite measurement systems.

We do not need perfect knowledge of every variable at every point in time and space in order to make useful predictions and decisions.

What matters is understanding what we have observed, what we have not observed, why information may be missing, and how uncertain that makes us about the state of the world.

What is missing data?

Suppose we have temperature readings from four sensors:

Time	Sensor A	Sensor B	Sensor C	Sensor D
10:00	18.2	18.5	18.1	18.4
10:05	18.3	18.6	—	18.5
10:10	18.4	18.7	—	18.6
10:15	18.5	18.8	18.4	18.7

Sensor C has no recorded values at 10:05 or 10:10.

Those observations are missing.

But notice what we do still know.

We know:

the sensor reported 18.1°C at 10:00,
it reported 18.4°C at 10:15,
nearby sensors continued reporting,
the missing period lasted only ten minutes.

We have incomplete information.

We do not have no information.

That distinction is important.

Missing does not mean zero

One of the simplest but most important rules is:

Missing is not the same as zero.

Suppose an electricity meter has no reading for 14:00.

That does not mean:

electricity demand = 0

It means:

electricity demand = unknown

Likewise:

missing temperature ≠ 0°C

missing income ≠ €0

missing traffic count ≠ no traffic

Zero is an observation.

Missing means we do not have an observation.

These are fundamentally different states.

Missing does not necessarily mean broken

Missing data can arise because something failed.

But it can also arise because of how the measurement system was designed.

Perhaps:

measurements are intentionally taken only once an hour,
a medical test is performed only for some patients,
a satellite passes over a location only periodically,
a survey contains optional questions,
a sensor conserves battery by transmitting intermittently.

The absence of a measurement may therefore be entirely expected.

This matters because real-world intelligent systems must operate under incomplete observation.

Perfect information is usually unavailable.

The world continues when the sensor goes offline

Imagine an Internet of Things temperature sensor installed in a building.

At 13:00 it reports:

21.2°C

At 13:05:

21.4°C

Then its wireless connection drops.

No measurements arrive for fifteen minutes.

At 13:25 it reconnects and reports:

21.9°C

The temperature did not cease to exist between 13:05 and 13:25.

The physical system continued evolving.

What disappeared was our ability to observe it directly.

This gives us an important distinction:

STATE OF THE WORLD

is not the same as:

OBSERVATION OF THE WORLD

A missing observation does not imply a missing state.

Observation is not state

Suppose the true temperature at time t is part of the state of the system:

x(t)

A sensor produces an observation:

y(t)

If the sensor goes offline, we may lose:

y(t)

But the physical state:

x(t)

still exists.

Conceptually:

WORLD STATE → SENSOR → OBSERVATION

If the sensor fails:

WORLD STATE → ? → NO OBSERVATION

The state has not vanished.

Our knowledge of it has become less certain.

This idea will become extremely important later when we study state estimation, robotics and partially observable systems.

Missing data creates uncertainty

Suppose a temperature sensor last reported:

20.0°C

one second ago.

We may still be fairly confident that the current temperature is close to 20°C.

Now suppose the sensor has been offline for:

one hour.

We may be less confident.

After:

one day

our uncertainty may be much greater.

So instead of pretending we know the missing value exactly, we can represent our knowledge as:

estimated state + uncertainty

For example:

estimated temperature = 20.3°C ± 0.2°C

Later:

estimated temperature = 21°C ± 2°C

The longer we go without observation, the wider our uncertainty may become.

This is often a much better way of thinking about missing data.

We do not always need to fill the gap perfectly

A common instinct is:

We must somehow reconstruct the exact missing value.

But that is not always necessary.

Suppose a building controller needs to decide whether to turn on cooling.

The precise missing temperature might have been:

22.31°C

But perhaps the controller only needs to know whether the building is likely to be:

comfortably below 25°C

If the available evidence makes that overwhelmingly likely, the system may still make a perfectly reasonable decision.

The goal is not always:

recover every missing number perfectly

The goal may instead be:

make a good decision while acknowledging uncertainty.

Different decisions require different certainty

Suppose a sensor has gone offline.

If the decision is:

Should we slightly adjust the ventilation?

a rough estimate may be sufficient.

If the decision is:

Should we shut down a nuclear reactor?

the required level of confidence will be very different.

The acceptable amount of missing information depends on:

the decision,
the consequences of error,
the available alternatives,
the uncertainty.

This connects missing data directly to decision-making.

Why data goes missing

Missing data can arise for many reasons.

Sensor failure

A physical sensor may:

fail,
lose power,
become damaged,
require maintenance.

Communication failure

The sensor may still work while its network connection fails.

Sampling design

The system may deliberately measure only at particular times.

Limited coverage

There may simply be no sensor at a particular location.

Human behaviour

People may:

skip survey questions,
miss appointments,
decline tests,
choose not to provide information.

Operational processes

Some information may only be collected under particular conditions.

Data integration

Two databases may contain different fields or cover different time periods.

Privacy

Some information may intentionally not be collected or retained.

Missingness can therefore be:

accidental,
intentional,
physical,
technical,
behavioural,
institutional.

Missing across time

Time-series data frequently contains gaps.

Suppose a sensor reports:

Time	Temperature
10:00	18.0
10:05	18.2
10:10	—
10:15	—
10:20	18.8

We know something about what happened before and after the gap.

If temperature normally changes gradually, we might infer that the missing values were somewhere between:

18.2°C

and:

18.8°C

We do not know the exact path.

But the temporal structure provides evidence.

Interpolation

One simple technique for estimating missing values between known observations is interpolation.

Suppose:

10:00 → 20°C

10:10 → missing

10:20 → 22°C

A simple linear interpolation might estimate:

10:10 → 21°C

Conceptually, we assume the value changed smoothly between the two observations.

This may be reasonable for some variables.

It may be terrible for others.

Interpolation contains an assumption

Suppose the true values were:

20 → 28 → 22

The interpolated value:

would miss the temporary spike completely.

Interpolation is therefore not recovering a hidden fact.

It is making an inference based on an assumption about how the system behaves between observations.

This is a general principle:

Estimated missing values are predictions, not observations.

They should not automatically be treated as though they were measured directly.

Forward filling

Another simple technique is to carry the previous value forward.

Suppose:

Time	Operating mode
10:00	Normal
10:05	—
10:10	—
10:15	Standby

We might assume:

10:05 → Normal

10:10 → Normal

until evidence tells us otherwise.

This is sometimes called forward filling.

It can make sense when states tend to persist.

But again, it relies on an assumption.

The state could have changed during the missing interval.

Using neighbouring observations

Spatial data provides another source of information.

Suppose one weather station goes offline.

Nearby stations report:

17.8°C

18.1°C

18.0°C

We might infer that the missing station is probably somewhere around:

18°C

The closer and more similar the neighbouring locations are, the more informative they may be.

So:

SPATIAL NEIGHBOURS → ESTIMATE MISSING LOCATION

This is a form of spatial interpolation.

Space and time can be combined

Suppose a sensor goes offline.

We may have:

its own previous readings,
its own later readings,
nearby sensors,
weather conditions,
a physical model.

Instead of relying on one source, we can combine them.

Conceptually:

PAST OBSERVATIONS

NEIGHBOURING OBSERVATIONS

SYSTEM MODEL

↓

ESTIMATE OF MISSING STATE

This is a much richer way to handle missing information.

Redundancy makes systems resilient

Real engineering systems often deliberately contain multiple sources of information.

A robot may estimate its position using:

GPS,
cameras,
wheel encoders,
inertial sensors,
maps.

If GPS temporarily disappears, the robot does not necessarily stop functioning.

It may continue estimating its position from the other sensors.

Its uncertainty may increase.

When GPS returns, the estimate can be corrected.

This is sensor fusion.

Sensor fusion

Suppose a vehicle wants to know its position.

GPS says:

position ≈ A

Wheel movement suggests:

position ≈ B

Camera observations suggest:

position ≈ C

Each source has different uncertainty.

A state-estimation system can combine these observations.

If one disappears temporarily, the others still provide information.

This is a fundamental idea in autonomous systems:

Robust intelligence does not require every sensor to work perfectly all the time.

It requires the system to understand what information is available and how reliable it is.

Graceful degradation

A well-designed system should often be able to continue operating when information becomes incomplete.

Perhaps:

all sensors available → high confidence

one sensor unavailable → moderate confidence

several sensors unavailable → low confidence

critical uncertainty → safe fallback

This is sometimes described as graceful degradation.

Instead of moving instantly from:

working

to:

not working

the quality of the system's knowledge can degrade gradually.

Uncertainty should increase when observations disappear

Suppose a robot knows its position accurately at time t.

It moves forward.

Without any new observations, it can estimate its new position from:

previous position,
speed,
direction.

But small errors accumulate.

After one second:

small uncertainty

After ten seconds:

larger uncertainty

After one minute:

larger still

When a reliable observation arrives, uncertainty may shrink again.

Conceptually:

PREDICT → UNCERTAINTY GROWS

then:

OBSERVE → UNCERTAINTY SHRINKS

This is a central idea in state estimation.

Missingness can itself be data

Sometimes the fact that a value is missing tells us something.

Suppose a medical test is recorded only when a doctor suspects a particular condition.

Then:

test result missing

may imply:

doctor did not request test

That fact may itself contain information.

Or suppose customers with very high incomes are less likely to answer an income question.

The absence of the answer may not be random.

The pattern of missingness may reveal something about the underlying process.

Missing completely at random

Statisticians distinguish several different kinds of missingness.

The simplest case is when data is missing completely at random.

Imagine a sensor occasionally loses a packet because of random radio interference.

Whether a value disappears has nothing to do with:

the value itself,
the state of the system,
other important variables.

In that case, the missing observations may behave approximately like a random subset of the data.

This is the easiest situation to handle.

Missing at random

Despite the name, missing at random has a more specific statistical meaning.

Suppose survey responses about income are more likely to be missing among younger participants.

If we know the participant's age, the pattern of missingness may be explainable using observed variables.

The missingness is related to information we already have.

This can sometimes be modelled.

Missing not at random

The hardest case occurs when missingness is related to the missing value itself or to information we do not observe.

Suppose people with extremely high incomes are particularly unlikely to report their income.

Then the missing values may systematically differ from the observed values.

Simply ignoring them could create a biased picture of the population.

This is often described as missing not at random.

Why the cause of missingness matters

Imagine a dataset containing household electricity consumption.

Ten percent of readings are missing.

There are several possible explanations.

Scenario A

Packets were randomly lost during transmission.

Scenario B

Meters lose communication more frequently in rural areas.

Scenario C

Meters fail specifically during periods of extremely high demand.

The same percentage is missing in each case.

But the implications are completely different.

We therefore need to ask:

Why is the data missing?

not merely:

How much data is missing?

Deleting missing rows

One simple response is to remove every observation containing a missing value.

Suppose we begin with:

100,000 rows

and only:

500 rows

contain missing values.

Deleting those rows may be perfectly reasonable.

But suppose:

40,000 rows

contain some missing information.

Deleting them could throw away a huge amount of useful evidence.

Worse, if the missingness is systematic, the remaining dataset may no longer represent the original population.

Complete-case analysis can create bias

Suppose a health survey asks people to report their income.

Higher-income respondents are more likely to skip the question.

If we delete everyone without an income value, the remaining dataset may disproportionately represent lower-income people.

The dataset becomes:

complete

but:

less representative.

This is an important lesson:

A perfectly complete dataset can be worse than an incomplete but properly understood dataset.

Completeness is not the same as quality.

Imputation

Another approach is to replace missing values with estimates.

This is called imputation.

Simple approaches include replacing missing values with:

the mean,
the median,
the previous value,
a nearby value.

More sophisticated approaches may use:

regression,
nearest neighbours,
probabilistic models,
machine-learning models,
state-space models.

The appropriate technique depends on the structure of the data.

Mean imputation

Suppose we observe:

18, 20, missing, 22, 20

The mean of the observed values is:

We might replace the missing value with:

This produces a complete dataset.

But it also introduces artificial certainty.

We did not observe 20.

We inserted it.

If we perform this many times, the dataset may appear less variable than reality.

Imputation should not hide uncertainty

Suppose the missing value could plausibly lie between:

15 and 25.

Replacing it with:

creates the appearance that we know the value exactly.

A better approach may represent:

best estimate = 20

together with:

uncertainty around estimate

The key principle is:

Filling a blank should not make uncertainty disappear.

Multiple possible values

Instead of pretending there is one correct replacement, we can consider several plausible possibilities.

Perhaps:

missing value → 18, 20 or 22

with different probabilities.

We can then examine whether our conclusion changes depending on which plausible value is used.

This treats the missing information as uncertainty rather than as a defect that must be hidden.

Probabilistic imputation

A probabilistic model might estimate:

P(x | observed data)

In words:

Given everything we have observed, what values are plausible for the missing quantity?

Instead of saying:

the missing temperature was 20.4°C

we might say:

Values near 20.4°C are most likely, but nearby values are also plausible.

This is a more faithful representation of what we actually know.

Confidence can depend on distance from observation

Suppose a weather station goes offline.

Five minutes later, we may estimate its temperature with high confidence.

Five hours later, our confidence may be lower.

Likewise in space.

Estimating temperature:

100 metres from a sensor

may be easier than:

100 kilometres from the nearest sensor.

So uncertainty can depend on both:

distance through time

and:

distance through space.

This gives us a powerful spatiotemporal idea:

Our uncertainty often grows as we move further from what we have actually observed.

Missing data across space

Imagine an environmental-monitoring network.

Sensors exist in:

cities,
towns,
major industrial areas.

But there are few sensors in remote regions.

The resulting dataset contains geographic gaps.

A national map may still be created using spatial models.

But predictions in poorly observed areas should generally carry more uncertainty.

A good map should communicate not only:

estimated value

but also:

confidence in that estimate.

A map of uncertainty can be as important as a map of prediction

Suppose an AI system predicts air pollution across a country.

It produces:

MAP 1 → predicted pollution

But we might also want:

MAP 2 → uncertainty

Some locations may have:

many nearby sensors,
recent observations,
stable conditions.

Others may have:

few sensors,
old observations,
unusual conditions.

The predictions should not necessarily be trusted equally.

The uncertainty map tells us where our knowledge is weak.

Missing data and machine learning

Machine-learning models often expect inputs in a particular form.

Suppose a model expects:

age,
income,
temperature,
location.

What happens if income is missing?

Different models handle this differently.

Possible approaches include:

impute a value,
include a missingness indicator,
use a model capable of handling missing values directly,
estimate the variable from other information,
abstain from making a prediction.

There is no universal solution.

Missingness indicators

Sometimes we explicitly tell the model that a value was missing.

For example:

income = estimated value
income_missing = 1

versus:

income = observed value
income_missing = 0

This allows the model to distinguish between:

measured information

and:

estimated or absent information.

That can be useful when the fact that something is missing carries information.

Models can learn from incomplete observations

Modern models can often operate with partial information.

Suppose an AI system normally receives ten features.

One is missing.

The remaining nine may still contain enough information for a useful prediction.

The correct response does not necessarily have to be:

Cannot compute.

It might instead be:

Prediction remains possible, but confidence is lower.

This is often a much more realistic approach to intelligence.

Humans do this constantly

Humans rarely possess complete information.

Suppose you look outside and see:

wet roads,
people carrying umbrellas,
dark clouds.

You did not directly observe the rain.

But you may infer:

It probably rained recently.

You are reasoning from incomplete observations.

Or suppose someone is late for a meeting.

You do not know why.

You may consider several possibilities rather than insisting that one explanation is certainly correct.

Human reasoning routinely operates under partial information.

AI systems must do the same.

Perfect observation is impossible

Consider the ambition:

Measure everything about the world continuously.

Immediately we encounter limits.

There are finite:

sensors,
storage,
bandwidth,
energy,
money,
computing resources.

Measurement itself has costs.

So we must choose:

what to measure,
where,
how frequently,
at what resolution.

Missing information is therefore not merely an accident.

It is partly a consequence of finite resources.

Measurement is itself a resource-allocation problem

Suppose we can afford:

1,000 sensors

but there are:

1,000,000 possible locations.

Where should the sensors go?

Perhaps we place more sensors where:

conditions change quickly,
consequences are important,
uncertainty is high,
populations are large.

Now data collection itself becomes an optimisation problem.

We are allocating scarce measurement resources.

This connects directly to later parts of the course on shared resources and service design.

The value of information

This introduces an important idea:

How valuable would an additional observation actually be?

Suppose our current uncertainty is already very low.

Another sensor may add almost no useful information.

Elsewhere, uncertainty may be high and the consequences of error severe.

An additional measurement there could be extremely valuable.

So measurement should sometimes be targeted according to the value of information.

We do not necessarily want maximum data.

We want useful data.

Active sensing

An intelligent system can sometimes decide where to measure next.

Imagine a robot exploring a building.

It is highly certain about one corridor.

Another area is poorly mapped.

Rather than collecting more observations of the known corridor, the robot may move towards the uncertain area.

Conceptually:

CURRENT BELIEF

↓

WHERE AM I MOST UNCERTAIN?

↓

TAKE NEW OBSERVATION

↓

UPDATE BELIEF

This is sometimes called active sensing.

The machine is not passively receiving data.

It is deciding which information would be most useful.

Missing data and partially observable systems

Many intelligent systems operate in environments where the true state cannot be fully observed.

A robot cannot see through walls.

A doctor cannot directly observe every process inside a patient.

An electricity operator cannot perfectly observe every appliance.

A financial model cannot know every future decision made by every participant.

These are partially observable systems.

The challenge is not to eliminate partial observability.

It is to reason intelligently despite it.

Belief about state

When the true state is uncertain, a system can maintain a belief about possible states.

Instead of:

state = x

we may have:

possible state A → 60%

possible state B → 30%

possible state C → 10%

The machine represents what it believes about the world given the available evidence.

New observations update those beliefs.

This connects missing data directly to probability and Bayesian reasoning.

Prediction can bridge missing observations

Suppose we know the state at time:

and have a model describing how the system evolves.

We can predict:

state at t+1

even if no measurement arrives.

Then:

state at t+2

But uncertainty grows.

When a new observation arrives, the model can compare:

prediction

with:

measurement

and update its estimate.

This gives us the recurring loop:

PREDICT → OBSERVE → UPDATE → PREDICT

That loop sits at the heart of many intelligent systems.

The Kalman filter idea

Later in the course we will encounter methods such as the Kalman filter.

The mathematical details can wait.

The conceptual idea is simple.

The system has:

a model of how the world changes

and:

measurements of the world

Neither is perfect.

So it combines them.

Conceptually:

MODEL PREDICTION

NEW OBSERVATION

↓

UPDATED STATE ESTIMATE

If the measurement is unreliable, the system may trust the model more.

If the model is uncertain and the measurement is highly reliable, it may trust the observation more.

This is uncertainty-aware reasoning.

Missing data does not have to stop the loop

Suppose no observation arrives.

The system can still:

predict forward

using its model.

Its uncertainty increases.

When observations resume, it updates again.

So:

OBSERVATION AVAILABLE

→ predict + correct

OBSERVATION MISSING

→ predict + increase uncertainty

This is a much more useful way of thinking about real-world intelligence than demanding a complete dataset at every moment.

Autonomous vehicles cannot demand perfect data

Imagine an autonomous vehicle that refused to operate whenever:

GPS briefly disappeared,
one camera was obscured,
a road marking was unclear,
a sensor returned an uncertain measurement.

It would be practically useless.

Real autonomous systems need to:

combine sensors,
estimate hidden state,
represent uncertainty,
recognise when uncertainty becomes unsafe,
adapt their behaviour accordingly.

Missing information is therefore part of the operating environment.

The right action may change with uncertainty

Suppose a robot estimates:

obstacle distance = approximately 20 metres

with very low uncertainty.

It may continue normally.

Now suppose:

obstacle distance = somewhere between 5 and 25 metres

The expected value might still be around 15 metres.

But the uncertainty is much greater.

A safe decision may therefore be:

slow down

until better information becomes available.

This demonstrates why uncertainty itself can influence action.

Expected value is not enough

Imagine two estimates.

Estimate A

distance = 20 m ± 0.2 m

Estimate B

distance = 20 m ± 15 m

Both have the same central estimate:

20 m

But they represent radically different knowledge.

A decision system that receives only the estimated value loses important information.

A better system receives:

estimate + uncertainty.

AI systems should sometimes say "I don't know"

Missing data connects to a broader principle.

An intelligent system should not always be forced to produce a confident answer.

Sometimes the correct output is:

insufficient information,
low confidence,
additional observation required,
several outcomes remain plausible.

This is especially important in high-consequence applications.

Uncertainty is information.

Hiding it does not make the system more intelligent.

This connects to AI hallucinations

Later we will examine why generative AI systems sometimes produce confident-sounding statements that are incorrect.

One way to understand the problem is that systems may be encouraged to always produce an answer even when the available evidence is weak.

A more useful intelligent system would ideally distinguish:

I have strong evidence

from:

I am inferring

from:

I do not know.

Handling missing information well therefore connects directly to trustworthy AI.

Missingness can reveal where we need better data

Suppose a model performs poorly in a particular region.

We discover that the region has very few sensors.

The problem may not primarily be the model.

It may be the observation system.

This suggests an important distinction:

MODEL UNCERTAINTY

versus:

DATA UNCERTAINTY

Sometimes better algorithms help.

Sometimes the answer is simply:

We need another measurement.

Data quality is not the same as data completeness

A dataset can be:

100% complete

and contain terrible measurements.

Another can be:

90% complete

with extremely accurate, well-understood observations.

Which is better?

There is no automatic answer.

Data quality includes many dimensions:

accuracy,
relevance,
timeliness,
coverage,
consistency,
uncertainty,
representativeness.

Completeness is only one of them.

More data can create false confidence

Suppose we have measurements everywhere except one remote region.

A model fills the region with predictions.

The final dataset now appears complete.

Every location has a number.

But some values were:

measured

while others were:

estimated.

If we remove that distinction, the completed dataset may look more certain than the evidence justifies.

A good information system should preserve provenance:

Where did this value come from?

Observed versus estimated

It can be useful to distinguish explicitly between:

OBSERVED

and:

ESTIMATED

For example:

Time	Temperature	Source	Uncertainty
10:00	18.2°C	Measured	Low
10:05	18.4°C	Estimated	Moderate
10:10	18.6°C	Estimated	Moderate
10:15	18.8°C	Measured	Low

Now the dataset contains not only values.

It also contains information about how much we know about those values.

This is much richer.

Provenance matters

A value might originate from:

direct measurement,
human entry,
interpolation,
another model,
historical average,
simulation.

Those sources are not equivalent.

Knowing the provenance of data helps us judge how much confidence to place in it.

As AI systems increasingly generate information that becomes input to other AI systems, provenance will become even more important.

A prediction can become someone else's data

Suppose Sensor A fails.

Model A estimates its missing value.

That estimated value is stored in a database.

Later, Model B is trained using that database.

Unless the provenance is preserved, Model B may treat the estimate as though it were a direct observation.

The pipeline becomes:

REAL OBSERVATIONS

↓

MODEL ESTIMATE

↓

DATABASE

↓

TRAINING DATA

↓

NEW MODEL

The distinction between observation and inference can disappear.

This is an important issue for increasingly AI-generated datasets.

Missing data can propagate through systems

Imagine several systems connected together.

System A has incomplete observations.

It produces an estimate.

System B uses that estimate to make a prediction.

System C uses the prediction to allocate resources.

Uncertainty can propagate through the chain.

So intelligent system design should ask not only:

What is our prediction?

but:

How certain are the inputs that produced it?

We should propagate uncertainty

Suppose:

input uncertainty is high

but the final model reports:

prediction = 83.472%

The precision of the number may be misleading.

A better system should allow uncertainty to propagate from:

measurement

through:

inference

through:

prediction

into:

decision-making.

Conceptually:

UNCERTAIN OBSERVATION

↓

UNCERTAIN STATE

↓

UNCERTAIN PREDICTION

↓

DECISION UNDER UNCERTAINTY

This will become a central theme in the next module.

The goal is not a perfectly observed world

It is tempting to imagine that better technology will eventually give us:

complete data about everything

But that is unlikely.

The world is:

too large,
too dynamic,
too complex.

Measurement has costs.

Sensors fail.

People make choices.

Events happen between observations.

Some things cannot be directly observed at all.

Intelligence therefore cannot depend on perfect information.

The challenge is not to eliminate uncertainty. It is to represent and reason with it.

A better mental model

Instead of thinking:

DATASET

as a perfect table containing reality, think:

REALITY

↓

PARTIAL OBSERVATIONS

↓

MEASURED DATA

MISSING INFORMATION

↓

INFERENCE

↓

ESTIMATED STATE + UNCERTAINTY

This is much closer to how real intelligent systems operate.

From missing data to uncertainty

Suppose an IoT network contains:

10,000 sensors.

At any moment:

9,850 are reporting normally,
100 have delayed packets,
30 are temporarily offline,
20 are undergoing maintenance.

The correct response is not necessarily:

The dataset is broken.

This may simply be the normal operating condition of a large distributed system.

A robust system should be designed around that reality.

It should know:

what it has observed,
what it has inferred,
how old the information is,
how uncertain the estimate is,
when uncertainty becomes unacceptable.

That is a much stronger design principle than demanding perfect data.

Perfect data can be unnecessarily expensive

Suppose achieving:

99% coverage

costs €1 million.

Achieving:

99.9% coverage

costs €10 million.

Achieving:

99.999% coverage

costs €100 million.

If good inference allows the service to operate safely and effectively at 99% coverage, pursuing perfect observation may waste enormous resources.

The right question is not:

How do we eliminate every missing value?

It is:

What quality of observation is required to make the decisions we need to make?

This turns data collection into a service-design problem.

Information has value relative to decisions

An observation is valuable when it can change what we should do.

Suppose we are already almost certain that:

no action is required.

Another measurement may have little value.

But if we are close to a critical decision boundary, another observation may be extremely valuable.

So the value of data depends on:

how it changes decisions.

This idea will return when we study:

optimisation,
resource allocation,
active learning,
intelligent services.

Ask questions about missing data

Whenever you encounter missing information, ask:

What exactly is missing?
Why is it missing?
Is the missingness expected?
Is it random or systematic?
Does missingness itself contain information?
Can nearby observations help?
Can previous observations help?
Can a model estimate the missing state?
How uncertain is that estimate?
Does uncertainty grow with time?
Does uncertainty vary across space?
Do we actually need the missing value to make the decision?
Would another measurement materially improve the decision?
Are estimated values being distinguished from observed values?
Is uncertainty being propagated through the system?
At what point should the system stop and request more information?

These questions are usually more useful than simply asking:

How do we fill the blank?

The central idea

Missing data is not necessarily a failure.

It is often simply a consequence of observing a complex world using finite sensors, finite resources and imperfect communication systems.

The world continues to exist when we stop observing it.

So the challenge is not always to reconstruct every missing value perfectly.

It is to maintain the best possible understanding of the system given the information available.

That means distinguishing:

observation

from:

state

and:

measurement

from:

inference.

It also means preserving uncertainty.

We do not need perfect data at every point in time and space. We need enough information to make useful decisions, together with an honest representation of what we do not know.

This gives us a richer view of intelligent systems:

OBSERVE WHAT WE CAN

↓

INFER WHAT WE CANNOT DIRECTLY OBSERVE

↓

REPRESENT UNCERTAINTY

↓

DECIDE WHETHER WE KNOW ENOUGH TO ACT

↓

TAKE ANOTHER MEASUREMENT IF WE DO NOT

Missing data therefore leads naturally into probability.

Once our observations are incomplete, noisy or uncertain, we need a language for describing what we believe about the things we cannot know exactly.

That language is probability.

But before we reach the probability module, we need to consider another fundamental question about datasets:

Who or what gets observed in the first place?

That is the subject of the next lesson:

sampling and selection.