Parameters

A machine-learning model has a structure.

But that structure contains values that must be learned from data.

These learned values are called parameters.

At its simplest:

Parameters are the internal numerical values a model adjusts during learning so that its predictions better match the training data.

For a simple linear regression model:

predicted output = intercept + slope × input

the parameters are:

• the intercept,
• the slope.

The model structure tells us:

Use a straight line.

The parameters tell us:

Which straight line?

---

A simple example

Suppose we want to predict electricity demand from temperature.

We choose a linear model:

Demand = a + b × Temperature

Here:

• a is the intercept,
• b is the slope.

Before learning, we do not know the best values of a and b.

Perhaps initially:

a = 30

b = 0

The model predicts the same demand regardless of temperature.

After learning from data, we might find:

a = 45

b = -0.7

Now the model has learned:

Higher temperatures are associated with lower demand.

The model structure stayed the same.

The parameters changed.

---

Structure versus parameters

This distinction is important.

Suppose we choose:

linear regression.

That choice defines the structure:

ŷ = a + bx

The data then determines the parameters:

a

and:

b.

So:

MODEL DESIGN

↓

STRUCTURE

TRAINING DATA

↓

LEARNED PARAMETERS

↓

TRAINED MODEL

---

Parameters store what the model has learned

Once training is complete, the learned parameters contain much of the useful information extracted from the training data.

The model no longer needs to inspect the entire dataset for every prediction.

Instead, it uses its learned parameters.

Conceptually:

TRAINING DATA

↓

LEARNING

↓

PARAMETERS

↓

NEW INPUT

↓

PREDICTION

The parameters act as a compressed representation of patterns learned from the data.

---

A line is defined by parameters

Consider:

y = a + bx

Different parameter values create different lines.

Model A

a = 10

b = 1

Model B

a = 20

b = -2

Model C

a = 5

b = 0.5

The mathematical structure is identical.

The parameters produce different predictions.

Training is the process of finding parameter values that work well.

---

Parameters can have intuitive meaning

In a simple linear regression, parameters are often relatively interpretable.

Suppose:

Demand = 45 - 0.7 × Temperature

Then:

-0.7

means something like:

For a 1°C increase in temperature, predicted demand falls by approximately 0.7 GW, holding the rest of the model fixed.

That gives the parameter a clear interpretation.

---

But interpretation requires caution

Even when a coefficient is easy to read mathematically, it is not automatically a causal effect.

Suppose a regression finds:

income increases with years of education.

The coefficient may describe an association in the model.

It does not necessarily prove:

One additional year of education causes exactly that income increase.

Parameters describe the model.

Causal interpretation requires stronger assumptions.

---

Multiple parameters

Suppose our demand model uses:

• temperature,
• previous demand,
• hour of day.

A simple linear model might look conceptually like:

Demand = a + b₁ × Temperature + b₂ × Previous demand + b₃ × Hour

Now the model has several parameters:

a

b₁

b₂

b₃.

Each controls how strongly a feature influences the prediction.

---

A parameter vector

When models contain many parameters, we often group them together.

For example:

θ = [a, b₁, b₂, b₃]

The symbol:

θ

is commonly used to represent the collection of model parameters.

Training can then be described as:

Find the values of θ that make the model perform well on the training data.

---

Learning means adjusting parameters

Suppose the model predicts:

38 GW

but actual demand is:

42 GW.

The model has made an error.

The learning algorithm uses that error to adjust its parameters.

Conceptually:

CURRENT PARAMETERS

↓

MAKE PREDICTION

↓

COMPARE WITH ACTUAL VALUE

↓

CALCULATE ERROR

↓

ADJUST PARAMETERS

↓

TRY AGAIN

This loop is at the heart of machine learning.

---

The model starts somewhere

Before training, parameters need initial values.

For a simple model, we may be able to calculate the optimum directly.

For large neural networks, parameters are often initialised with small random values.

So initially, the model may make poor predictions.

Training gradually improves them.

---

Parameters are not manually chosen one by one

For modern models, there may be:

• thousands,
• millions,
• billions

of parameters.

Humans do not set them individually.

Instead, we define:

• model architecture,
• training objective,
• optimisation process.

The learning algorithm adjusts the parameters automatically.

---

Parameters and hyperparameters are different

This distinction is important.

Parameters are learned from the data.

Examples:

• regression coefficients,
• neural-network weights.

Hyperparameters are settings chosen around the learning process.

Examples include:

• learning rate,
• number of layers,
• regularisation strength,
• batch size.

Conceptually:

HYPERPARAMETERS → HOW WE LEARN

PARAMETERS → WHAT THE MODEL LEARNS

---

An analogy

Imagine fitting a radio.

The design of the radio is like the model architecture.

The tuning knobs are like parameters.

Training turns the knobs until the signal becomes clearer.

Hyperparameters are more like deciding:

• how quickly to turn the knobs,
• how many knobs the system has.

---

Parameters in classification

Consider logistic regression.

Inputs might include:

• income,
• debt,
• repayment history.

The model learns weights associated with those features.

Conceptually:

INPUT FEATURES

↓

WEIGHTED COMBINATION

↓

PROBABILITY OF DEFAULT

Each learned weight is a parameter.

---

Positive and negative weights

Suppose a classifier has:

weight for debt = positive

and:

weight for repayment history = negative.

Conceptually:

• higher debt pushes predicted risk upward,
• stronger repayment history pushes it downward.

Again, this describes how the model uses the features.

It does not automatically establish causal relationships.

---

Neural networks contain many parameters

A neural network may contain:

• input units,
• hidden layers,
• output units.

Connections between units have numerical weights.

Many units also have biases.

These weights and biases are parameters.

Training adjusts them so that the network produces better outputs.

---

A neuron as a simple parameterised function

A simplified artificial neuron might do something like:

weighted inputs + bias

↓

activation

Suppose it receives:

x₁

x₂

x₃.

It calculates something conceptually like:

w₁x₁ + w₂x₂ + w₃x₃ + b

where:

• w₁, w₂, w₃ are weights,
• b is a bias.

Those values are learned parameters.

---

Parameters determine sensitivity

Suppose:

w₁ = 10

and:

w₂ = 0.01.

The model is much more sensitive to changes in x₁ than x₂, at least within that part of the model.

Weights determine how information flows through the network.

---

But individual neural-network parameters are hard to interpret

In a simple regression, one coefficient may have a clear interpretation.

In a deep neural network, behaviour is distributed across huge numbers of interacting parameters.

One weight rarely means:

This neuron represents a dog.

Representations emerge through many parameters working together.

---

Distributed representations

Suppose a neural network recognises faces.

The concept:

face

may not live in one single parameter.

Instead, many parameters jointly encode patterns involving:

• edges,
• textures,
• shapes,
• spatial relationships.

Knowledge is distributed throughout the model.

---

Parameters can encode enormous amounts of structure

Large language models may contain billions of parameters.

Training on enormous datasets adjusts those parameters so that the network becomes good at predicting language.

Through that process, the parameters come to encode statistical structure related to:

• grammar,
• facts,
• concepts,
• relationships,
• styles,
• patterns of reasoning.

This does not mean the model stores the training data as one giant database.

Its parameters represent learned statistical structure.

---

Parameters as compressed experience

A useful way to think about this is:

TRAINING EXPERIENCE

↓

OPTIMISATION

↓

PARAMETER VALUES

The parameters become a compressed consequence of the model's training history.

That is why two models with the same architecture but different training data can behave differently.

---

Same architecture, different parameters

Suppose two neural networks have identical architecture.

Model A is trained on:

medical images.

Model B is trained on:

satellite imagery.

The structure is the same.

The learned parameters become very different.

The training data shapes the internal model.

---

Same data, different parameters

Even with the same data and architecture, two training runs can sometimes end with slightly different parameter values.

Why?

Because training may involve:

• random initialisation,
• shuffled batches,
• stochastic optimisation.

Several different parameter configurations may perform similarly.

---

There may be many good parameter solutions

For a complex model, there may not be one unique perfect set of parameters.

Instead, many regions of parameter space may produce:

similarly good performance.

Training searches for one useful solution.

This is important when thinking about uncertainty in model parameters.

---

Parameter space

If a model has two parameters:

θ₁

and:

θ₂,

we can imagine a two-dimensional parameter space.

Every possible pair:

(θ₁, θ₂)

defines a different model.

Training searches through this space.

For modern models with billions of parameters, the space has billions of dimensions.

We cannot visualise it directly.

But the concept remains useful.

---

The loss surface

For every possible parameter configuration, we can calculate:

how badly the model performs.

That creates a loss surface over parameter space.

Conceptually:

PARAMETERS

↓

MODEL PREDICTIONS

↓

LOSS

Training seeks parameter values with:

lower loss.

---

Optimisation searches parameter space

Suppose our current parameters produce:

high loss.

The optimiser tries to adjust them in a direction that reduces the loss.

Then:

evaluate again

and:

adjust again.

This repeated search is optimisation.

In neural networks, one of the most important methods is:

gradient descent.

We will study that shortly.

---

Parameters depend on the loss function

Suppose two models use the same architecture and training data.

Model A minimises:

mean squared error.

Model B minimises:

mean absolute error.

They may learn different parameter values.

Why?

Because the definition of:

what counts as wrong

changed.

So:

LOSS FUNCTION

↓

OPTIMISATION

↓

LEARNED PARAMETERS

The objective shapes what the model learns.

---

Parameters are therefore objective-dependent

This is philosophically important.

We sometimes talk about a trained model as though the data simply revealed the correct parameters.

But the learned parameters depend on:

• data,
• model structure,
• loss function,
• optimisation process.

Different design choices can lead to different models.

The learning outcome is not determined by data alone.

---

Parameters also depend on representation

Suppose we predict house price.

Model A receives:

• floor area,
• location.

Model B receives:

• floor area,
• location,
• transport access.

Even using the same learning algorithm, the learned parameters will differ.

The available features change the relationship the model learns.

---

Parameters reflect correlations in the data

Suppose postcode strongly correlates with house price.

The model may learn parameters that rely heavily on postcode.

If that correlation changes in the future, those learned parameters may become less useful.

Parameters encode historical relationships.

They are not universal laws.

---

Parameters can become stale

Suppose an electricity-demand model was trained several years ago.

Since then:

• EV adoption increased,
• heat pumps increased,
• behaviour changed.

The historical parameters may no longer describe current demand well.

The model may need retraining.

---

Retraining changes parameters

New data arrives.

We train again.

Conceptually:

OLD PARAMETERS

NEW DATA

↓

UPDATED TRAINING

↓

NEW PARAMETERS

The architecture might stay the same.

The learned relationship changes.

---

Online learning

Some systems update parameters continuously or frequently as new data arrives.

This is online learning.

Conceptually:

OBSERVATION

↓

PREDICTION

↓

OUTCOME

↓

PARAMETER UPDATE

↓

NEXT OBSERVATION

The model adapts through time.

---

Stability versus adaptation

But updating parameters too quickly can create problems.

Suppose one unusual observation arrives.

A model that reacts too strongly may overfit to noise.

A model that updates too slowly may fail to adapt to genuine change.

This is the same tension we saw earlier:

stability

versus:

plasticity.

---

Learning rate affects parameter movement

One important hyperparameter is the learning rate.

Conceptually:

small learning rate

→ small parameter changes

large learning rate

→ large parameter changes.

Too small:

learning may be extremely slow.

Too large:

the optimiser may overshoot useful solutions or become unstable.

We will return to this during gradient descent.

---

Parameters and memory

Parameters provide one form of machine memory.

A model may no longer have direct access to every training example.

But its parameters have been shaped by those examples.

This is sometimes called:

parametric memory.

---

Parametric memory versus explicit memory

Consider a language model.

Parametric memory

Knowledge encoded through learned parameters during training.

Explicit or external memory

Information stored separately and retrieved when needed.

Examples include:

• databases,
• documents,
• conversation memory.

These are different ways of giving an AI system access to past information.

---

Training changes parameters; prompting usually does not

Suppose you give a language model a prompt.

The prompt changes:

the context.

It usually does not permanently change the model's trained parameters.

So:

TRAINING → changes parameters

PROMPTING → changes current context

This distinction is fundamental to understanding modern AI.

---

Fine-tuning changes parameters

Suppose we take a pre-trained language model and fine-tune it on:

medical text.

The optimisation process updates some or all of its parameters.

After fine-tuning, the model itself has changed.

That differs from simply supplying a medical document in the prompt.

---

Retrieval does not necessarily change parameters

Suppose an AI retrieves a current document and uses it to answer a question.

The model's parameters may remain unchanged.

The document becomes part of the current input.

So:

PARAMETERS = longer-term learned structure

CONTEXT = information available right now.

This is a useful distinction.

---

Parameters and state are also different

Recall our idea of a state vector.

A system may have:

model parameters θ

and:

current state x(t).

These mean different things.

Parameters describe:

How the model behaves.

State describes:

Where the system currently is.

---

A physical example

Suppose we model a vehicle.

Parameters might include:

• mass,
• drag coefficient.

State might include:

• current position,
• current velocity.

The parameters are relatively persistent.

The state changes continuously.

This distinction becomes very important in dynamical systems.

---

Parameters can themselves change through time

Not every system has fixed parameters.

Suppose battery capacity degrades.

A parameter describing:

maximum capacity

may slowly change.

The system may need to estimate both:

state

and:

parameters.

This becomes an adaptive estimation problem.

---

Model parameter versus physical parameter

The word parameter is used in different contexts.

A physical model might have:

mass = 1,500 kg.

That is a physical parameter.

A neural network might have:

weight = 0.173.

That is a learned model parameter.

Both affect model behaviour.

But their interpretation differs.

---

Parameter estimation

Suppose a physical process follows a known structure:

y = ax + b.

We do not know:

a

or:

b.

Using data to estimate them is called parameter estimation.

Much of statistics and machine learning can be understood as sophisticated parameter estimation.

---

Maximum likelihood

One common principle for estimating parameters is maximum likelihood.

The idea is:

Choose parameter values under which the observed training data would be relatively likely.

Conceptually:

PARAMETERS

↓

HOW LIKELY WOULD THESE DATA BE?

↓

CHOOSE PARAMETERS THAT MAKE OBSERVED DATA MOST PLAUSIBLE

This connects parameter learning directly to probability.

---

A coin example

Suppose a coin has unknown probability:

p

of Heads.

We flip it:

100 times

and observe:

70 Heads.

A natural estimate is:

p ≈ 0.7.

Why?

Because values near 0.7 make the observed sequence more plausible than something like:

p = 0.01.

We have estimated a parameter from data.

---

Bayesian parameter estimation

Bayesian reasoning gives another perspective.

Instead of estimating one value:

p = 0.7,

we can represent:

a distribution over plausible values of p.

For example:

Values around 0.7 appear most plausible, but uncertainty remains.

This gives us:

parameter uncertainty.

---

Parameters can be uncertain

Suppose a model reports:

slope = -0.7.

That does not mean we know the true relationship exactly.

Perhaps the evidence supports values between:

-0.5 and -0.9.

Parameter estimates have uncertainty.

More data can reduce that uncertainty.

---

Parameter uncertainty creates prediction uncertainty

Suppose we are uncertain about the slope of a demand model.

Then future predictions inherit that uncertainty.

So:

UNCERTAIN PARAMETERS

↓

UNCERTAIN PREDICTIONS

This is one component of epistemic uncertainty.

---

More data can constrain parameters

Suppose we fit a line using:

3 observations.

Many different slopes may fit reasonably well.

Now collect:

100,000 observations.

The plausible parameter range may become much narrower.

Data constrains parameter uncertainty.

---

But more data does not fix a bad model

Suppose the true relationship is strongly nonlinear.

We insist on:

a straight line.

Collecting one billion observations will estimate the best straight line extremely precisely.

It may still be the wrong representation.

This is crucial:

Low parameter uncertainty does not imply low model uncertainty.

---

You can be precisely wrong

Suppose the data gives us enormous confidence that:

slope = -0.70001 ± 0.00001.

That looks impressive.

But if the true system cannot reasonably be represented by a linear model, the precision is misleading.

A precise parameter estimate inside the wrong model does not guarantee good predictions.

---

Parameters can overfit

A highly flexible model may have enough parameters to fit almost every training observation.

Suppose:

number of useful examples is small

but:

number of adjustable parameters is enormous.

The model may memorise training-specific patterns.

More parameters increase expressive power.

They can also increase overfitting risk.

---

Parameters and model capacity

Very roughly:

more adjustable parameters

can allow a model to represent:

more complex functions.

This is sometimes described as greater model capacity.

But parameter count alone does not determine effective capacity.

Architecture and regularisation matter too.

---

A polynomial example

Suppose we have:

10 data points.

A straight line has:

2 parameters.

A very high-degree polynomial may have:

10 or more parameters.

The polynomial may pass through every training point.

But between the points it may behave wildly.

Training fit becomes perfect.

Generalisation may become poor.

---

More parameters are not automatically worse

Modern deep-learning models can contain billions of parameters and still generalise remarkably well.

So the simplistic rule:

more parameters = overfitting

is not always correct.

Generalisation depends on:

• data volume,
• architecture,
• optimisation,
• regularisation,
• structure of the task.

Modern AI has complicated traditional intuitions about parameter count.

---

Scaling laws

Large language models have shown that increasing:

• parameters,
• data,
• compute

can often improve performance predictably over wide ranges.

These empirical relationships are sometimes called scaling laws.

This is one reason AI models have become dramatically larger.

---

More parameters require more resources

Parameter count has physical consequences.

Large models require more:

• memory,
• computation,
• electricity.

Training billions of parameters requires substantial hardware.

Inference also requires those parameters to be stored and processed.

So:

PARAMETER SCALE

connects directly to:

PHYSICAL AI INFRASTRUCTURE.

We will explore this later in the course.

---

Memory requirements

Suppose a model has:

100 billion parameters.

Each parameter requires some number of bits or bytes to store.

Even before performing any computation, the model may require enormous memory capacity.

This is one reason:

• GPUs,
• high-bandwidth memory

are so important to modern AI.

---

Quantisation

One way to reduce resource requirements is to represent parameters using fewer bits.

For example, instead of storing each parameter using high numerical precision, we might use lower precision.

This is quantisation.

Conceptually:

LESS PRECISION PER PARAMETER

↓

LESS MEMORY

↓

FASTER / CHEAPER INFERENCE

potentially with some loss of model quality.

---

Parameters must be moved, not just computed

Modern AI performance is not only about arithmetic.

Large models require enormous numbers of parameters to be moved:

• from memory,
• into compute units,
• across accelerators.

Memory bandwidth can become a major bottleneck.

This is another reason hardware architecture matters.

---

Sparse models

Some models do not use every parameter for every input.

For example, mixture-of-experts architectures may activate only part of the model for a particular token.

Conceptually:

HUGE TOTAL PARAMETER SET

but:

SMALLER ACTIVE PARAMETER SET PER INPUT.

This can increase model capacity without increasing computation proportionally.

We will revisit this in the modern-AI modules.

---

Parameter count is not intelligence

A model with:

more parameters

is not automatically:

more intelligent.

Parameter count tells us something about model scale.

Performance also depends on:

• training data,
• architecture,
• training method,
• objective.

Comparing AI systems only by parameter count can be misleading.

---

Parameters interact

In a large neural network, parameters do not contribute independently.

The effect of one weight depends on:

• other weights,
• activations,
• input.

So it is generally not meaningful to ask:

What fraction of the model's intelligence belongs to parameter 8,251,409?

Knowledge emerges from interactions across the network.

---

This is similar to feature interaction

Recall our feature-attribution problem.

The value of one feature can depend on other features.

Likewise, the function of one parameter depends on the larger network.

Complex intelligent systems are often:

relational

rather than:

additive.

---

Features and parameters are different

This distinction is worth making explicit.

Features

Information supplied to the model.

Examples:

• temperature,
• income,
• pixels.

Parameters

Internal values learned by the model.

Examples:

• regression coefficients,
• neural-network weights.

So:

FEATURES = INPUT INFORMATION

PARAMETERS = LEARNED INTERNAL STRUCTURE.

---

Outputs depend on both

A prediction is produced by:

INPUT FEATURES

and:

LEARNED PARAMETERS.

Conceptually:

ŷ = f(x; θ)

where:

• x = input features,
• θ = parameters.

The semicolon simply emphasises that the prediction depends on both.

---

Training changes θ, not x

During training:

• the training examples provide x,
• the known outputs provide y,
• the algorithm changes θ.

So:

x + y

↓

LEARNING

↓

θ changes.

At inference time:

new x + fixed θ

↓

prediction.

---

Inference

Once training is complete, using the learned model to make predictions is called inference.

Conceptually:

Training

DATA → CHANGE PARAMETERS

Inference

INPUT → USE PARAMETERS → OUTPUT

This distinction becomes important when discussing AI compute.

---

Training and inference have different resource profiles

Training may require:

• enormous datasets,
• repeated parameter updates,
• substantial computation.

Inference may involve:

• one forward pass through the learned parameters.

But at huge usage scale, inference can also consume enormous resources.

This connects machine-learning theory to data centres later in the course.

---

Parameters encode the past; inputs describe the present

There is a beautiful connection to the course's timeline.

Very roughly:

PARAMETERS

encode patterns learned from:

the past.

INPUTS

describe information available:

now.

The model combines them to predict:

what comes next.

So:

PAST EXPERIENCE

↓

PARAMETERS

CURRENT INPUT

↓

MODEL

↓

PREDICTION OF UNKNOWN / FUTURE

This is a useful way to think about machine learning through time.

---

A language model example

A language model has parameters learned during training.

Suppose the prompt is:

The capital of France is

The prompt becomes current input.

The parameters contain learned language structure.

Together they produce:

probability distribution over next tokens.

Conceptually:

TRAINED PARAMETERS

CURRENT CONTEXT

↓

NEXT-TOKEN DISTRIBUTION.

---

The parameters do not contain the current conversation by default

The model's trained parameters were learned before the current prompt.

The present conversation enters through:

context.

That is why we distinguish:

long-term learned parameters

from:

current state / context.

---

Training is slow memory formation

One analogy is:

parameter learning = slow memory

The model changes through extensive training.

Current context acts more like:

working memory.

The analogy is imperfect, but useful.

---

Parameter updates can be expensive

Changing billions of parameters requires significant computation.

That is why retraining a foundation model for every new fact would be impractical.

Instead, systems can use:

• context,
• retrieval,
• external memory.

Different information belongs at different timescales.

---

What should live in parameters?

This becomes a system-design question.

Stable general knowledge may be useful in:

parameters.

Fast-changing information may be better stored in:

external databases or retrieval systems.

User-specific temporary context may belong in:

memory or state.

Modern AI increasingly combines all three.

---

Parameters and explainability

For simple models, parameters can help explain predictions.

For complex models, individual weights usually cannot.

This creates the need for other explanation methods:

• feature importance,
• Shapley values,
• probing internal representations.

Explainability becomes more challenging as parameter interactions become more complex.

---

Parameter importance is not feature importance

A feature might strongly affect predictions through thousands of parameters.

Likewise, one parameter might contribute to many different features and concepts.

So:

which parameter matters?

and:

which input feature mattered?

are different questions.

Feature attribution is usually more useful for explaining individual predictions to humans.

---

Regularisation constrains parameters

One way to reduce overfitting is to discourage parameter values from becoming unnecessarily extreme or complex.

This is called regularisation.

Conceptually:

FIT TRAINING DATA WELL

but also:

KEEP MODEL REASONABLY SIMPLE / CONSTRAINED.

Regularisation modifies the learning objective.

---

Parameter penalties

For example, a loss function might include:

prediction error

plus:

penalty for very large parameter values.

The model must trade off:

• fitting the data,
• keeping parameters controlled.

This can improve generalisation.

---

L1 and L2 regularisation

Two common forms are:

L1 regularisation

and:

L2 regularisation.

We do not need the full mathematics yet.

The conceptual distinction is enough:

• L1 can encourage some parameters to become exactly zero,
• L2 tends to shrink parameters more smoothly.

These choices influence the learned model.

---

Parameters can be frozen

Suppose we start with a pre-trained model.

During fine-tuning, we may:

• update all parameters,
• update only some layers,
• freeze most parameters.

Freezing means:

Keep these learned values unchanged.

This can reduce:

• compute,
• overfitting,
• loss of existing capabilities.

---

Parameter-efficient fine-tuning

Modern AI also uses methods that add or adjust only a small number of parameters rather than retraining the entire model.

This allows large models to be specialised more efficiently.

The core idea is:

large shared base model

small task-specific parameter changes.

---

Parameters and continual learning

If a system keeps updating its parameters, it may learn new information.

But parameter updates can also alter old capabilities.

This contributes to:

catastrophic forgetting.

Learning something new can interfere with what was previously encoded.

---

Stable knowledge and changing knowledge

This gives us a broader architecture:

PARAMETERS

for slowly learned patterns.

STATE / CONTEXT

for rapidly changing conditions.

EXTERNAL MEMORY

for information that should be retrievable without retraining.

Intelligent systems may need all three.

---

A useful parameter checklist

When examining a model, ask:

• What are the parameters?
• How many are there?
• What structure do they belong to?
• How are they initialised?
• What data adjusts them?
• What loss function guides the updates?
• How are they optimised?
• Are the parameter estimates uncertain?
• Could they be overfitting the training data?
• Do they still represent the current world?
• How often should they be updated?
• How much compute and memory do they require?
• Can individual parameters be interpreted?
• What information belongs in parameters versus context or external memory?

These questions help reveal what "learning" actually means inside the model.

---

The central idea

A model begins as:

a structure with adjustable values.

Those adjustable values are its:

parameters.

Training uses data to transform:

INITIAL PARAMETERS

↓

PREDICTION

↓

ERROR

↓

UPDATE

↓

BETTER PARAMETERS

until the model captures useful patterns.

So:

DATA

↓

LEARNING ALGORITHM

↓

PARAMETERS

↓

MODEL KNOWLEDGE

↓

NEW INPUT

↓

PREDICTION

Parameters are therefore the mechanism through which much of machine learning stores what it has learned from the past.

They connect:

historical examples

to:

future predictions.

But we have not yet answered the most important question:

How does the machine actually decide how the parameters should change?

To understand that, we first need to look more closely at the process of learning from examples.