Module 4 — Prediction: learning patterns from the past
Lesson 14 of 16
Feature importance
A machine-learning model may use:
- 5 features,
- 50 features,
- 5,000 features,
- or millions of learned representations.
Once the model has been trained, a natural question is:
Which pieces of information actually matter to its predictions?
This is the problem of feature importance.
Feature importance attempts to quantify how much different inputs contribute to the predictive behaviour of a model.
At first this sounds simple.
It is not.
Why do we care which features matter?
Suppose a model predicts electricity demand using:
- temperature,
- time of day,
- day of week,
- previous demand,
- electricity price,
- wind speed.
The model performs well.
But we might still want to know:
Which features are doing most of the work?
This can help us:
- understand the model,
- identify redundant inputs,
- detect suspicious relationships,
- improve feature engineering,
- reduce data collection,
- diagnose errors,
- explain individual predictions.
Prediction alone does not provide explanation
A model might tell us:
Predicted electricity demand = 42 GW.
That answers:
What does the model predict?
It does not answer:
Why did the model predict 42 GW?
Feature importance attempts to open that box.
There are several different questions called "feature importance"
When someone says:
Feature A is important,
they might mean:
- The model uses Feature A frequently.
- Removing Feature A makes predictions worse.
- Changing Feature A changes the prediction.
- Feature A explains a large part of one particular prediction.
- Feature A contains a lot of predictive information.
- Feature A contributes something that other features cannot replace.
These are related.
They are not identical.
So we should always ask:
Important according to what definition?
Global versus local importance
One of the most useful distinctions is between:
GLOBAL IMPORTANCE
and:
LOCAL IMPORTANCE.
Global feature importance
Global importance asks:
Which features matter most to the model overall?
Suppose a house-price model produces:
Location 0.42
Floor area 0.31
Number of bedrooms 0.12
Building age 0.09
Garden 0.06
This suggests that across the dataset:
**location**
may be more influential than:
**floor area**
and:
**building age**.
That is a **global feature-importance statement**.
It tells us something about the model's behaviour across many predictions.
It does not yet explain any one particular prediction.
---
### Local feature importance
Now consider one specific house.
Suppose the model predicts:
**€600,000**.
A local explanation might say:
**Baseline prediction: €400,000**
**Location: +€120,000**
**Floor area: +€70,000**
**Garden: +€30,000**
**Building age: -€20,000**
This tells us how the model moved from:
**a baseline prediction**
to:
**this particular prediction**.
That is **local feature importance**.
---
### Global and local importance answer different questions
#### Global importance
> Which features matter most across the dataset?
#### Local importance
> Which features mattered most for this prediction?
These are not the same thing.
A feature can have:
**low global importance**
but:
**very high local importance**
for a particular case.
---
### A rare feature can matter enormously
Suppose an equipment-failure model uses:
**abnormal vibration**.
Most machines operate normally.
So abnormal vibration appears in only:
**0.1% of observations**.
Across the whole dataset, its average importance may appear small.
But when that vibration appears:
**failure risk may rise dramatically**.
The feature is rare.
Its local importance can still be enormous.
Again:
> **Frequency is not the same as importance.**
---
### Importance depends on the model
Suppose we train:
**Model A**
and:
**Model B**
on the same data.
Model A relies heavily on:
**location**.
Model B relies more on:
**floor area**.
Both may achieve similar test accuracy.
Feature importance therefore describes:
> **how this model uses the available information**
not:
> **the one true importance of each variable in reality**.
---
### A feature can be important because the model chose to use it
Suppose:
**postcode**
strongly predicts house price.
The model may rely heavily on it.
Another model might instead use:
- local transport,
- local income,
- school access.
Both encode overlapping spatial information.
So feature importance can depend on:
**which alternative features are available**.
---
### Redundant features complicate importance
Suppose we include:
**temperature in Celsius**
and:
**temperature in Fahrenheit**.
They contain essentially the same information.
If both are present, the model may divide importance between them.
Perhaps:
**Celsius = 40%**
**Fahrenheit = 40%**.
Remove Fahrenheit and suddenly:
**Celsius = 80%**.
Did Celsius become more important in reality?
No.
The attribution changed because the feature set changed.
---
### Feature importance is relational
This gives us a crucial idea:
> **A feature's predictive value depends partly on what other information is available.**
Suppose Feature A is:
**location**.
If we know nothing else, location may be extremely informative.
But if we already know:
- postcode,
- local income,
- nearby house prices,
location may add much less new information.
Its value is **conditional**.
---
### Marginal contribution
A useful way to ask about feature value is:
> How much does this feature improve the prediction when it is added to what we already know?
This is its **marginal contribution**.
Conceptually:
**MODEL WITHOUT FEATURE A**
↓
**prediction**
then:
**ADD FEATURE A**
↓
**new prediction**
The difference tells us something about A's contribution.
---
### But added to what?
This is where things become difficult.
Suppose we have two features:
**A = location**
**B = floor area**.
Start with neither.
Add A first.
Then add B.
We may get one allocation of importance.
Reverse the order.
We may get another.
So marginal contribution depends on:
**which coalition of features already exists**.
---
### A two-feature example
Suppose:
**No features → €400,000**
**Location only → €520,000**
**Floor area only → €530,000**
**Both → €600,000**.
If we add location first:
**Location contributes +€120,000**.
Then floor area contributes:
**+€80,000**.
But if we add floor area first:
**Floor area contributes +€130,000**.
Then location contributes:
**+€70,000**.
Which allocation is correct?
That is the attribution problem.
---
### Features interact
The issue is not merely redundancy.
Features can genuinely interact.
Suppose:
**large garden**
adds substantial value only in:
**suburban houses**.
Then:
**garden value**
depends on:
**location**.
The contribution of one feature cannot always be understood independently of the others.
---
### Interaction effects
An **interaction effect** occurs when the contribution of one feature changes depending on another feature.
For example:
**temperature**
might strongly affect electricity demand during:
**18:00–21:00**
but hardly matter at:
**03:00**.
So:
**temperature importance**
depends on:
**time of day**.
---
### Feature importance can therefore change with state
Suppose an electricity system is unconstrained.
A local demand feature may have little impact on a prediction of:
**system risk**.
Now the same location approaches a network limit.
That feature may become extremely important.
So:
> **Feature importance can be state-dependent.**
This is especially important in dynamic systems.
---
### Time can change importance
Suppose yesterday's demand is a very strong predictor on:
**ordinary weekdays**.
But on:
**Christmas Day**
it becomes much less useful.
A holiday indicator may suddenly become much more important.
Feature importance is not necessarily fixed through time.
---
### Space can change importance
Suppose temperature is a strong predictor of heating demand in:
**northern Europe**.
It may be far less important in:
**a warm climate**.
A global importance score can therefore hide geographic variation.
---
### Group-specific importance
A feature may also matter differently across populations.
Suppose:
**age**
strongly affects one medical prediction in older patients but very little in younger patients.
A single global importance ranking may conceal this.
We may need to examine importance by:
- subgroup,
- location,
- time,
- state.
---
### Feature importance is not causality
This is one of the most important warnings.
Suppose a model finds:
**umbrella ownership**
highly important for predicting:
**whether it is raining**.
That does not mean:
**umbrellas cause rain**.
The feature is predictive.
It is not necessarily causal.
---
### Predictive importance
Feature importance asks something like:
> **How much does this feature contribute to the model's prediction?**
Causal importance asks:
> **What would happen to the real-world outcome if we intervened and changed this variable?**
These are different questions.
---
### A postcode example
Suppose postcode has high importance in a house-price model.
This tells us:
> The model uses postcode strongly.
It does not tell us:
> Moving the same physical house to a different postcode while changing nothing else would causally change its value by exactly the attributed amount.
The model is describing statistical structure.
Not a physical intervention.
---
### Why the distinction matters
Suppose a medical model finds:
**hospital ID**
to be an important feature.
Perhaps different hospitals treat different types of patients.
The model may use hospital identity as a strong predictor.
But changing the hospital ID field in the database does not itself change patient health.
Feature importance can reveal correlations that require interpretation.
---
### Permutation importance
One common way to estimate feature importance is **permutation importance**.
The idea is simple.
First:
**measure model performance normally**.
Then randomly shuffle one feature.
This destroys the relationship between that feature and the target while leaving the other features unchanged.
Then measure performance again.
---
### The permutation idea
Suppose accuracy is:
**92%**.
Shuffle:
**location**.
Accuracy falls to:
**70%**.
Location appears important.
Now shuffle:
**garden size**.
Accuracy falls to:
**91.8%**.
Garden size appears much less important globally.
Conceptually:
**MORE PERFORMANCE LOSS AFTER SHUFFLING**
↓
**GREATER FEATURE IMPORTANCE**.
---
### Why shuffling works
If a feature contains useful information, destroying its alignment with the observations should make the model perform worse.
If performance barely changes, the feature may not be adding much unique information.
This makes permutation importance intuitive.
---
### But correlated features create problems
Suppose:
**Celsius**
and:
**Fahrenheit**
are both included.
Shuffle Celsius.
The model still has Fahrenheit.
Performance barely changes.
Permutation importance may conclude:
**Celsius is unimportant**.
Shuffle Fahrenheit.
The model still has Celsius.
Again:
**Fahrenheit seems unimportant**.
Yet temperature itself is crucial.
Redundancy hides importance.
---
### Importance can be distributed across substitutes
If two features contain overlapping information, removing either one alone may have little effect.
This does not mean the information is unimportant.
It means the model has:
**alternative routes to the same information**.
Feature importance must therefore be interpreted in the context of correlation and redundancy.
---
### Drop-column importance
Another approach is:
1. train the full model,
2. remove one feature,
3. retrain,
4. compare performance.
If performance falls substantially, the feature was useful.
This is sometimes called **drop-column importance**.
---
### Retraining changes the question
Permutation asks:
> How much does the existing trained model rely on this feature?
Drop-column analysis asks something closer to:
> How much worse is the best model we can build without this feature?
These are subtly different questions.
---
### Model-specific importance
Some models provide their own importance measures.
For example:
**linear regression coefficients**
can sometimes indicate how features influence predictions.
Decision trees can measure how often or how effectively features are used to split the data.
These can be useful.
But they also have limitations.
---
### Coefficients can be misleading without scaling
Suppose:
**Feature A** ranges from:
**0 to 1**.
Feature B ranges from:
**0 to 1,000,000**.
Their coefficients are not directly comparable without considering:
- feature scale,
- distribution.
A small coefficient on a large-scale variable can still have a huge practical effect.
---
### Tree importance can favour certain features
Some tree-based importance measures can favour variables with:
- many possible split points,
- many categories.
So a high built-in importance score should not automatically be treated as absolute truth.
Different methods answer slightly different questions.
---
### Explainability needs a reference point
For local feature importance, we usually need some notion of:
**baseline**.
Suppose the model predicts:
**€600,000**.
Relative to what are we explaining the extra value?
Perhaps:
**average model prediction = €400,000**.
Then feature contributions explain the movement:
**€400,000 → €600,000**.
---
### The baseline changes the interpretation
Suppose we choose a different baseline:
**€500,000**.
Now only:
**€100,000**
needs to be explained.
Feature contributions will change.
So local explanations are always relative to some reference.
---
### Baselines are modelling choices
Possible baselines might represent:
- average prediction,
- expected prediction over a reference population,
- prediction with missing features.
Different baselines can answer different questions.
There is no universally correct reference for every explanation.
---
### Shapley values solve a credit-allocation problem
We now return to the idea introduced in the previous lesson.
Suppose multiple features jointly produce a prediction.
We want to allocate the difference between:
**baseline**
and:
**final prediction**
among those features.
But feature contributions depend on the order in which information is introduced.
Shapley values provide a principled solution.
---
### The coalition idea
Imagine each feature as a **player** in a cooperative game.
The players form coalitions.
For each coalition, we ask:
> How much predictive value does this combination of features create?
Then for each feature:
> How much additional value does it contribute when joining different coalitions?
The Shapley value averages those marginal contributions appropriately.
---
### A simple two-feature Shapley example
Suppose:
**Baseline = €400k**
**Location only = €520k**
**Floor area only = €530k**
**Location + floor area = €600k**.
There are two possible orders.
#### Order 1
**Location → Floor area**
Location contributes:
**€520k - €400k = €120k**.
Floor area contributes:
**€600k - €520k = €80k**.
#### Order 2
**Floor area → Location**
Floor area contributes:
**€530k - €400k = €130k**.
Location contributes:
**€600k - €530k = €70k**.
---
### Average across the possible orders
For location:
**(€120k + €70k) / 2 = €95k**.
For floor area:
**(€80k + €130k) / 2 = €105k**.
Together:
**€95k + €105k = €200k**.
And:
**€400k + €200k = €600k**.
The feature contributions exactly explain the difference between baseline and final prediction.
---
### Why this is useful
Neither ordering had a privileged claim to being:
**the correct order**.
The Shapley approach considers both.
With many features, it considers the feature's marginal contribution across all relevant coalitions.
This creates an allocation that satisfies several useful mathematical properties.
---
### Efficiency
The contributions add up to the total amount being explained.
Conceptually:
**BASELINE**
+
**ALL FEATURE CONTRIBUTIONS**
=
**FINAL PREDICTION**.
Nothing is left unexplained within the attribution system.
---
### Symmetry
Suppose two features always make exactly the same marginal contribution.
A Shapley allocation gives them the same value.
Identical contributors are treated identically.
---
### Dummy feature
Suppose a feature never changes the prediction regardless of which coalition it joins.
Its Shapley value is:
**zero**.
A feature that contributes nothing receives no credit.
---
### Additivity
If we combine value functions, Shapley allocations combine consistently.
This property becomes particularly important in cooperative game theory.
For feature attribution, it contributes to the mathematical consistency of the explanation.
---
### Mathematical fairness is not social fairness
The word **fair** needs care here.
Shapley values are considered fair in the sense that they satisfy a particular set of axioms about allocating value.
That does not mean:
> A model explained using Shapley values is socially fair.
A discriminatory model can still have a mathematically perfect Shapley explanation.
---
### SHAP
In machine learning, one widely used family of Shapley-based explanation methods is:
**SHAP**
which stands for:
**SHapley Additive exPlanations**.
SHAP methods estimate Shapley-style contributions for model predictions.
---
### SHAP can explain a single prediction
Suppose a loan-risk model gives:
**baseline default risk = 10%**
and predicts:
**35%** for one applicant.
A SHAP-style explanation might indicate that:
- debt level increased model score,
- income reduced it,
- repayment history reduced it,
- another feature increased it.
The explanation describes how the model arrived at the prediction relative to its baseline.
---
### SHAP values may not be probability-point changes directly
Depending on the model and implementation, the additive values may operate on:
- raw model score,
- log-odds,
- another internal scale.
So an explanation should always specify:
> **What quantity is being decomposed?**
This matters for correct interpretation.
---
### Exact Shapley calculation becomes expensive
Suppose there are:
**2 features**.
Very easy.
Now:
**20 features**.
There are enormously more possible feature coalitions.
For:
**n features**,
the number of subsets grows exponentially.
Exact computation can therefore become expensive.
---
### Practical SHAP methods use shortcuts
Different SHAP methods exploit:
- approximations,
- particular model structures.
For example, there are methods tailored to:
- tree models,
- deep networks.
The conceptual goal remains the same:
**estimate Shapley-style marginal contributions efficiently**.
---
### Approximation introduces uncertainty too
If feature attribution is estimated approximately, the explanation itself may contain numerical uncertainty.
We should resist presenting every attribution value as infinitely precise.
An explanation is another model-derived quantity.
---
### Shapley values depend on what "missing a feature" means
This is an important technical issue.
To ask:
> What would the model predict without Feature A?
we need some definition of:
**without Feature A**.
Do we:
- replace it with an average?
- integrate over plausible values?
- condition on the remaining features?
Different choices can produce different explanations.
---
### Feature dependence matters
Suppose:
**age**
and:
**years of employment**
are highly correlated.
If age is "removed", what values of employment history should remain plausible?
Treating features as completely independent may create unrealistic combinations.
For example:
**age 18**
with:
**40 years employment history**.
Feature attribution needs to consider these dependencies carefully.
---
### Explanations can leave the real data manifold
If an explanation evaluates combinations of features that could never occur in reality, the resulting importance values may be misleading.
This is sometimes described in terms of moving outside the:
**data manifold**.
In plain language:
> The explanation is asking the model about impossible or highly unrealistic cases.
---
### Feature attribution is therefore model-dependent and assumption-dependent
A Shapley value is mathematically defined once we specify the game.
But constructing the game for a machine-learning model involves choices about:
- baseline,
- missing features,
- dependence.
The mathematics does not remove those modelling choices.
---
### Global importance from local explanations
Suppose we calculate local Shapley values for:
**100,000 observations**.
We can aggregate them.
For example:
**average absolute Shapley value by feature**.
This gives a global importance measure.
So:
**LOCAL EXPLANATIONS**
↓
**AGGREGATE**
↓
**GLOBAL MODEL BEHAVIOUR**.
---
### Why use absolute values?
Suppose temperature sometimes:
**raises**
the prediction
and sometimes:
**lowers**
it.
If we simply average signed contributions, they may cancel.
Average absolute contribution instead asks:
> How strongly does this feature usually move the prediction, regardless of direction?
This is one common global importance measure.
---
### Direction still matters
Absolute importance tells us:
**how much**.
It does not tell us:
**which way**.
We should separately examine whether:
- higher feature values tend to raise predictions,
- lower them,
- behave nonlinearly.
---
### Dependence plots
A feature-dependence plot can show:
**feature value**
against:
**feature contribution**.
For example:
**temperature**
might show:
- near-zero contribution at mild temperatures,
- large positive contribution during extreme cold.
This reveals nonlinear and state-dependent behaviour.
---
### Feature interactions can also be attributed
Shapley theory can be extended to look at:
**interaction contributions**.
For example:
> How much of the prediction comes specifically from the interaction between temperature and time of day?
This can reveal relationships that individual importance scores miss.
---
### Importance can be unstable
Suppose we retrain the same model on a slightly different sample.
Feature rankings change dramatically.
That suggests importance estimates may be unstable.
The model may have several alternative ways to explain the data.
---
### Stability is useful information
If:
**location**
is the top feature across:
- different samples,
- different model versions,
we have stronger evidence that it is structurally important to the model.
If rankings change constantly, we should be cautious.
---
### Importance can change after retraining
Suppose user behaviour changes.
The model is retrained.
A feature that used to dominate becomes less useful.
Feature importance is therefore not a permanent property.
It should be monitored through time in changing systems.
---
### Model monitoring can include feature attribution
A deployed system might monitor:
- prediction accuracy,
- input distributions,
- calibration,
- feature importance.
Suppose a previously irrelevant feature suddenly becomes dominant.
That may signal:
- distribution shift,
- data pipeline problems,
- shortcut learning.
Interpretability can help detect system change.
---
### A broken sensor can become "important"
Imagine an IoT model.
A temperature sensor begins reporting:
**999** whenever it disconnects.
If disconnections correlate with machine failure, the model may learn:
**999 → failure**.
Feature importance shows temperature becoming extremely influential.
The model may actually be learning:
**sensor fault state**.
This may be useful or dangerous depending on deployment.
---
### Explanation can reveal data quality problems
Suppose a model consistently relies on:
**record ID**
or:
**file name**.
Those features should probably not contain meaningful predictive information.
High importance can reveal:
- leakage,
- data artefacts,
- unintended shortcuts.
Feature importance is therefore a debugging tool.
---
### Explainability can also create false confidence
A neat chart showing:
**Feature A: 37%**
**Feature B: 24%**
can look authoritative.
But we should ask:
- Which importance method?
- Relative to what baseline?
- Across what population?
- Are features correlated?
- Is this local or global?
Numbers need context.
---
### A feature is not "37% responsible" for reality
This language should be avoided.
Feature importance usually tells us something about:
**the model's predictive behaviour**.
It does not mean the feature caused:
**37% of the real-world outcome**.
The distinction between:
**model attribution**
and:
**causal attribution**
must remain clear.
---
### Feature importance can help compare models
Suppose two models have similar accuracy.
#### Model A relies mostly on:
- temperature,
- previous demand.
#### Model B relies heavily on:
- strange identifier fields.
Model A may be more trustworthy.
Understanding what models use can help us choose between similarly performing systems.
---
### Interpretability can support domain validation
A domain expert may inspect feature importance and say:
> These relationships make physical sense.
Or:
> This feature should have no relationship whatsoever with the target.
That does not prove the model is correct.
But it can identify obvious problems.
---
### Unexpected importance can be discovery
Not every surprise is an error.
Suppose a machine-failure model finds:
**a particular vibration frequency**
far more informative than engineers expected.
Further investigation may reveal a previously unknown failure mechanism.
Machine learning can therefore generate hypotheses.
---
### Prediction can lead to science
The process can become:
**MODEL FINDS PREDICTIVE PATTERN**
↓
**HUMANS INVESTIGATE**
↓
**POSSIBLE MECHANISM IDENTIFIED**
↓
**EXPERIMENT**
↓
**CAUSAL KNOWLEDGE**.
Feature importance can support discovery.
But the causal step requires additional evidence.
---
### Feature importance and fairness
Suppose a lending model heavily relies on:
**postcode**.
This should prompt questions.
Postcode may proxy:
- income,
- ethnicity,
- historical disadvantage.
Feature importance helps us see that the model is using this information.
It does not by itself tell us whether using it is legitimate.
---
### Removing the feature may not solve the issue
Remove postcode.
But the model retains:
- school,
- income,
- commuting pattern.
These may reconstruct similar information.
So fairness cannot be reduced to:
> Remove the important sensitive-looking feature.
The information structure is more complicated.
---
### Fairness auditing can use attribution
We can compare feature contributions across groups.
For example:
> Does postcode systematically push predictions downward for one group?
This can reveal patterns worth investigating.
But fairness ultimately depends on:
- outcomes,
- objectives,
- rights,
- context.
Feature attribution is one diagnostic among many.
---
### Importance and privacy
Suppose a highly sensitive feature contributes almost nothing to prediction.
Why collect it?
Removing it could reduce:
- privacy risk,
- storage,
- governance burden
with little predictive cost.
Feature importance can therefore support **data minimisation**.
---
### Value versus cost
Suppose:
**Feature A**
improves prediction substantially and costs almost nothing to collect.
Feature B:
- adds tiny predictive value,
- requires invasive monitoring.
A rational system might remove B.
Feature importance can be combined with:
**feature acquisition cost**.
---
### Value of information
This leads to the concept of **value of information**.
A feature is useful when knowing it changes our ability to:
- predict,
- decide.
But its value is not merely:
**how correlated it is with the target**.
It depends on:
- what other information we have,
- what decision follows,
- how costly it is to acquire.
---
### Predictive value versus decision value
Suppose Feature A improves forecast accuracy slightly.
But it changes no downstream decisions.
Its practical value may be:
**almost zero**.
Feature B improves accuracy by the same amount but prevents critical shortages.
Its decision value may be enormous.
So:
**feature importance to prediction**
and:
**feature importance to the service**
are different.
---
### Decision-focused feature importance
We can ask a deeper question:
> If this feature were unavailable, how much worse would our actual decisions become?
This moves beyond pure prediction.
Now the value of information depends on:
- objectives,
- constraints,
- system state.
This will become important in later modules.
---
### Information value can be state-dependent
Suppose electricity supply is abundant.
Knowing demand with extreme precision may have limited value.
During scarcity, the same information may be crucial.
So:
**VALUE OF FEATURE**
can depend on:
**CURRENT STATE**.
This links feature importance to dynamic service design.
---
### Information can have option value
Suppose a sensor rarely changes a decision.
But during one rare emergency it becomes essential.
Its average predictive value may be low.
Its resilience value may be high.
Again:
> **Average importance can hide tail value.**
---
### Shared information can have overlapping value
Suppose three sensors measure almost the same physical quantity.
Any one sensor may be sufficient.
Individually, their marginal values depend on whether the others are available.
This is another cooperative allocation problem.
Shapley theory becomes naturally relevant.
---
### From features to resources
There is a broader conceptual bridge here.
Imagine replacing:
**features**
with:
**resources**.
The same question arises:
> How much value does Resource A contribute when combined with different coalitions of other resources?
This is why Shapley theory appears in:
- cooperative games,
- cost allocation,
- infrastructure,
- energy systems,
- machine-learning explainability.
The underlying mathematical problem is:
**allocating joint value among interacting contributors**.
---
### This is a recurring theme in the course
A feature's value is not necessarily intrinsic.
It depends on:
**what else is already available**.
The same is true for:
- sensors,
- generators,
- hospital capacity,
- compute resources.
Value is often:
**marginal**
and:
**relational**.
---
### Feature importance can help simplify models
Suppose we discover:
**20 features**
contribute almost all predictive value.
Another:
**500 features**
contribute almost nothing.
Removing low-value features may produce:
- faster models,
- cheaper data pipelines,
- easier auditing.
Simplification can improve the whole service.
---
### But removing low-importance features needs care
A feature may matter only:
- for rare events,
- for one subgroup,
- in one location.
Global average importance can miss these cases.
Before removing a feature, examine:
**where and when it matters**.
---
### Feature importance can support sensor design
Suppose an industrial system has:
**50 sensors**.
Some contribute very little predictive information.
Perhaps:
- they can be sampled less often,
- removed.
Others become critical during particular states.
This can inform:
**sensor placement and sensing strategy**.
---
### Dynamic feature acquisition
An intelligent service might not collect every feature continuously.
Instead:
**START WITH CHEAP FEATURES**
↓
**ESTIMATE UNCERTAINTY**
↓
if uncertainty high:
**ACQUIRE ADDITIONAL INFORMATION**.
This is dynamic feature acquisition.
---
### A medical example
Start with:
- age,
- symptoms.
If risk is clearly low:
**stop**.
If uncertainty remains high:
**order inexpensive test**.
If still uncertain:
**order expensive scan**.
The service progressively buys information where it has marginal value.
---
### Feature value becomes sequential
The value of the next feature depends on:
**what we already observed**.
So:
**Feature B value after A**
may differ from:
**Feature B value before A**.
This is exactly the marginal-contribution logic we have been developing.
---
### Feature importance and active sensing
A robot can perform something similar.
Perhaps:
**camera**
is enough under normal conditions.
In fog:
**camera uncertainty rises**.
The system activates:
**radar**.
Information acquisition adapts to state.
---
### Static importance versus dynamic importance
Traditional feature importance asks:
> Which features matter?
A more advanced intelligent system asks:
> **Which information matters now?**
That distinction becomes increasingly important as systems become:
- interactive,
- stateful,
- resource-constrained.
---
### Feature importance and AI agents
An AI agent may have access to:
- search,
- calculator,
- database,
- sensors.
Each tool provides new information.
The agent must decide:
> Which information source should I use next?
This is a feature-acquisition problem at a higher level.
---
### Tool use has cost
Calling a tool may require:
- time,
- compute,
- money.
An intelligent agent should ideally acquire information when:
**expected value of information > cost of obtaining it**.
Now feature importance becomes part of sequential decision-making.
---
### A useful hierarchy
We can distinguish:
**FEATURE EXISTS**
↓
**FEATURE IS PREDICTIVE**
↓
**FEATURE CONTRIBUTES UNIQUE INFORMATION**
↓
**FEATURE CHANGES THE DECISION**
↓
**FEATURE CREATES ENOUGH VALUE TO JUSTIFY ITS COST**.
Each question is progressively stronger.
---
### Correlation is the weakest level
A feature may correlate with the target.
That does not mean:
- the model needs it,
- it adds unique information,
- it changes decisions,
- it is worth collecting.
Feature importance moves us beyond simple correlation.
---
### Predictive importance is not utility
A feature may strongly affect the model prediction.
But if the output has little consequence, its service value may still be low.
Ultimately, information has value because of:
**what we can do differently after receiving it**.
---
### Feature importance is therefore one layer of value
Conceptually:
**FEATURE**
↓
**PREDICTIVE CONTRIBUTION**
↓
**IMPROVED PREDICTION**
↓
**IMPROVED DECISION**
↓
**REAL-WORLD VALUE**.
A large contribution at the first stage does not guarantee large value at the final stage.
---
### Explainability should match the question
If a user asks:
> Why did I receive this prediction?
Use:
**local explanation**.
If an engineer asks:
> Which sensors does the model rely on overall?
Use:
**global importance**.
If a policymaker asks:
> Does the system systematically disadvantage one group?
Feature importance alone is insufficient.
The explanation method should match the decision question.
---
### A useful feature-importance checklist
When someone presents a feature-importance result, ask:
- Is this local or global importance?
- Which model is being explained?
- Which importance method was used?
- What is the baseline?
- Are features correlated?
- Are some features substitutes?
- Are there interactions?
- Does importance change across time?
- Across space?
- Across groups?
- Does the feature matter only in rare events?
- Is the importance stable across retraining?
- Does the feature reflect leakage?
- Is it a proxy for sensitive information?
- Is this predictive or causal importance?
- Does the feature change actual downstream decisions?
- What does the feature cost to collect?
These questions turn an importance score into something meaningful.
---
### The central idea
A model makes a prediction using many pieces of information.
Feature importance asks:
> **Which pieces of information drove that prediction, and how much did they matter?**
At a basic level:
**FEATURES**
↓
**MODEL**
↓
**PREDICTION**
↓
**ATTRIBUTION OF PREDICTIVE CONTRIBUTION**.
But the problem is difficult because features:
- overlap,
- correlate,
- interact,
- change value by context.
There is rarely one intrinsic importance attached permanently to a feature.
Its value depends on:
**what other information is available**
and:
**which prediction we are trying to explain**.
That leads naturally to the next lesson.
If predictive value is created jointly by a coalition of features, how should we share that value among them?
A powerful answer comes from cooperative game theory.
The next lesson is:
**Shapley values — allocating predictive credit fairly across interacting features.**