Neural decoding: revision 1 to 3

Revision 1

Revision 3

horizon: "present"

categories: ["cybernetics", "minds"]

tags: ["bci", "neural decoding", "machine learning", "fmri", "neural manifolds", "mental privacy"]

summary: "The inference of a stimulus, intention, or mental state from measured neural activity, and the statistical machinery that makes brain–computer interfaces work."

summary: "The inference of a stimulus, intention, or mental state from measured neural activity, and the statistical machinery that makes brain-computer interfaces work."

updated: "2026-07-27"

updated: "2026-10-29"

issues: ["Image reconstruction from fMRI with diffusion models is discussed without a citation."]

---

}

```

**Neural decoding** is the inference of something outside the brain — a stimulus, a movement, a

**Neural decoding** is the inference of something outside the brain, such as a stimulus, a movement,

word, a state — from measurements of activity inside it. It is the statistical half of every

a word or a state, from measurements of activity inside it. It is the statistical half of every

[[brain-computer-interface]], and the half where most recent progress has come from. Decoding is

formally the inverse of encoding, which asks how a neuron or population responds to a given input;

formally the inverse of encoding. Encoding asks how a neuron or population responds to a given

decoding asks what input or intention best explains an observed response.

input; decoding asks what input or intention best explains an observed response.

The word "decoding" invites a misleading picture of a fixed cipher waiting to be broken. In

The word "decoding" suggests a fixed cipher waiting to be broken, and the picture misleads. In

practice a decoder is a model fitted to one person's brain, on one day, for one task, and its

performance degrades as any of those conditions changes. What decoding recovers is what varied

performance degrades as any of those conditions changes. Decoding recovers only what varied

systematically in the data it was trained on, and nothing else.

systematically in the data it was trained on.

## Origins

Motor decoding began with the observation that neurons in primary motor cortex are broadly tuned to

the direction of arm movement, each firing most for a preferred direction and less for others. No

single cell specifies direction, but a weighted vector sum across a population does — the population

single cell specifies direction, but a weighted vector sum across a population does. This population

vector, described in the mid-1980s.[^georgopoulos1986] That result established both the possibility

vector was described in the mid-1980s.[^georgopoulos1986] The result showed that a continuous

of extracting a continuous variable from spike counts and the population-level framing the field

variable could be extracted from spike counts, and it set up the population-level framing the field

still uses.

Sensory decoding developed in parallel in human neuroimaging. Pattern-classification methods applied

to functional MRI showed that the category of an object a person was viewing could be read from

distributed activity in ventral temporal cortex,[^haxby2001] and later that the orientation of an

distributed activity in ventral temporal cortex.[^haxby2001] Later work classified the orientation

attended grating and even the content of binocular rivalry could be classified from voxel

of an attended grating, and even the content of binocular rivalry, from voxel

patterns.[^kamitani2005] These demonstrations are the ancestors of every subsequent "mind-reading"

headline.

## From spikes to intent

Early [[utah-array]] decoders used linear filters mapping binned firing rates to cursor velocity.

The Kalman filter improved on this by treating the intended movement as a hidden state evolving

The Kalman filter improved on this by treating the intended movement as a hidden state evolving over

over time, with the neural data as noisy observations. A further gain came from recognising that

time, with the neural data as noisy observations. A further gain came from recognising that the

the training data are themselves flawed: during calibration the user is trying to correct the

training data are themselves flawed. During calibration the user is trying to correct the cursor, so

cursor, so the true intention points toward the target rather than along the observed trajectory.

the true intention points toward the target rather than along the observed trajectory. Retraining

Retraining with that assumption produced a marked improvement in closed-loop

with that assumption produced a marked improvement in closed-loop control.[^gilja2012]

control.[^gilja2012]

Modern decoders are recurrent or transformer-based networks trained on many hours of data, often

pooled across sessions. For [[speech-neuroprosthesis]] systems they emit phoneme sequences that a

language model then resolves into text. For surface recordings in [[ecog-interfaces]] the input

[language model](https://aiwiki.ai/wiki/large_language_model) then resolves into text. For surface

features differ but the architecture does not. At the opposite extreme, a sixteen-electrode

recordings in [[ecog-interfaces]] the input features differ but the architecture does not. At the

endovascular device such as the [[stentrode]] supports only a binary classifier, and the decoding

other extreme, a sixteen-electrode endovascular device such as the [[stentrode]] supports only a

problem shrinks accordingly — the design of a decoder is inseparable from the sensor feeding it.

binary classifier, and the decoding problem shrinks accordingly. A decoder's design cannot be

separated from the sensor feeding it.

Decoding is not confined to movement and speech. Closed-loop [[deep-brain-stimulation]] systems

decode a physiological marker of symptom state and adjust stimulation in response; seizure-warning

decode a physiological marker of symptom state and adjust stimulation in response. Seizure-warning

systems decode the approach of an ictal event; sleep and workload classifiers decode broad brain

systems decode the approach of an ictal event, and sleep and workload classifiers decode broad brain

states from a handful of channels. Hippocampal decoding underlies the [[memory-prosthesis]]

literature, and the sender's side of a [[brain-to-brain-interface]] is nothing more than a decoder

literature, and the sender's side of a [[brain-to-brain-interface]] is simply a decoder whose output

whose output happens to be routed to another person's stimulator.

is routed to another person's stimulator.

### Population dynamics and manifolds

A shift in framing during the 2010s changed what decoders assume. Rather than treating each neuron

During the 2010s a change in framing altered what decoders assume. Rather than treating each neuron

as encoding a variable, the dynamical-systems view treats motor cortex as a machine whose

as encoding a variable, the dynamical-systems view treats motor cortex as a machine whose population

population state evolves according to internal rules, with rotational structure that appears during

state evolves according to internal rules, with rotational structure that appears during reaching

reaching regardless of the specific movement.[^churchland2012] Population activity turns out to

regardless of the specific movement.[^churchland2012] Population activity turns out to occupy a

occupy a low-dimensional subspace — a neural manifold — of perhaps ten to a hundred dimensions,

low-dimensional subspace, called a neural manifold, of perhaps ten to a hundred dimensions, whatever

whatever the number of recorded neurons.[^gallego2017]

the number of recorded neurons.[^gallego2017]

This has two practical consequences. First, it explains the diminishing returns from higher channel

counts: additional electrodes sample the same manifold more densely rather than revealing new

counts. Additional electrodes sample the same manifold more densely rather than revealing new

independent signals, which is why the arms race in electrode numbers pursued by

independent signals, which is why the race to more electrodes pursued by [[neuralink]] and others

[[neuralink]] and others has not translated proportionally into control performance. Second, it

has not translated proportionally into control performance. Second, it offers a fix for drift.

offers a fix for drift.

## Calibration drift

A decoder trained on Monday works worse on Tuesday. Electrodes shift by micrometres, the set of

recorded neurons turns over, impedances change, and the user's own strategy changes as they learn.

recorded neurons turns over, impedances change, and users change their own strategy as they learn.

The traditional response is to recalibrate at the start of every session, which costs minutes and

The traditional response is to recalibrate at the start of every session, which takes minutes and

requires supervision — a serious obstacle to home use.

needs supervision, a serious obstacle to home use.

The manifold view suggests an alternative: the latent dynamics underlying a behaviour are far more

The manifold view suggests an alternative. The latent dynamics underlying a behaviour are far more

stable than the individual neurons expressing them, so a new day's recordings can be aligned to a

stored latent space rather than relabelled from scratch. Stabilizers built on this principle have

stored latent space instead of being relabelled from scratch. Stabilizers built on this principle

maintained performance across long gaps without new supervised calibration.[^degenhart2020] Related

have maintained performance across long gaps without new supervised calibration.[^degenhart2020]

work has shown that latent dynamics for a learned behaviour remain consistent over

Related work has shown that latent dynamics for a learned behaviour remain consistent over

years.[^gallego2020]

> [!key] Why drift is the practical bottleneck

> Peak decoding accuracy in a laboratory session is not what limits deployment. A device that needs

> Deployment is limited less by peak decoding accuracy in a laboratory session than by stability. A

> a technician every morning cannot be a medical product. Most of the engineering distance between

> device that needs a technician every morning cannot be a medical product. Most of the engineering

> current research systems and an approved implant lies in making decoders that survive weeks

> distance between current research systems and an approved implant lies in making decoders that

> unattended, not in raising a benchmark number.

> survive weeks unattended, rather than in raising a benchmark number.

## Non-invasive and semantic decoding

Decoding from outside the skull is far more limited, and the limits are informative. A widely

discussed 2023 study reconstructed the gist of continuous language from functional MRI while

subjects listened to stories, watched silent films, or imagined telling a story.[^tang2023] The

output was a paraphrase, not a transcript: it captured meaning while frequently getting exact words

output was a paraphrase rather than a transcript, capturing meaning while frequently getting exact

wrong.

words wrong.

The conditions attached to that result are the important part. Each decoder required many hours of

The conditions attached to that result matter most. Each decoder needed many hours of training data

training data from the specific individual, transferred poorly to other people, and could be

from the specific individual, transferred poorly to other people, and could be defeated by an

defeated by an uncooperative subject performing a distracting mental task. fMRI also requires

uncooperative subject performing a distracting mental task. fMRI also requires lying still inside a

lying still inside a large magnet. The study's authors framed these findings as evidence that

large magnet. The study's authors framed these findings as evidence that mental privacy is currently

mental privacy is currently protected by practical barriers rather than by principle — a framing

protected by practical barriers rather than by principle, a framing central to the policy discussion

central to the policy discussion under [[mental-privacy]] and [[neurorights]].

under [[mental-privacy]] and [[neurorights]].

Image reconstruction from fMRI using diffusion models produces striking pictures, but similarly

Image reconstruction from fMRI using diffusion models produces striking pictures. It too depends on

depends on tens of thousands of image-response pairs per subject, and the generative model supplies

tens of thousands of image-response pairs per subject, and the generative model supplies a large

a large share of the detail in the output. Distinguishing what the brain data contributed from what

share of the detail in the output. Separating what the brain data contributed from what the prior

the prior supplied is an active methodological problem.

supplied is an active methodological problem.

## Open problems

Decoders recover trained categories. A speech decoder cannot output a language it never saw; a

Decoders recover trained categories. A speech decoder cannot output a language it never saw, and a

movement decoder cannot produce a gesture absent from calibration. Whether richer or more abstract

content is recoverable at all from any practical measurement is unknown, and it is a different

content can be recovered at all from any practical measurement is unknown, and that is a separate

question from whether that content is present in the signal.

question from whether the content is present in the signal.

Generalization across people is weak. Nearly every high-performing decoder is fitted to one

individual, and efforts to build cross-subject foundation models for neural data are early. Related

individual, and efforts to build cross-subject foundation models for neural data are at an early

to this, the field lacks agreed benchmarks: results are typically reported on bespoke datasets from

stage. The field also lacks agreed benchmarks: results are typically reported on bespoke datasets

single participants, which makes comparison across laboratories difficult.

from single participants, which makes comparison across laboratories difficult.

Finally, decoding is silent about mechanism. That a variable can be read out of a population does

Finally, decoding says nothing about mechanism. That a variable can be read out of a population does

not establish that the brain uses it, a point that recurs in debates over the

not establish that the brain uses it. The point recurs in debates over the

[[neural-correlates-of-consciousness]] and in arguments about what a wiring diagram from

[[connectomics]] would and would not explain. Decoding accuracy is a lower bound on the information

[[connectomics]] would and would not explain. Decoding accuracy sets a lower bound on the

present, not a description of how it is used.

information present; it does not describe how that information is used.

## Outlook

The near-term trajectory is toward decoders that self-calibrate, transfer across sessions, and run

In the near term, decoders are moving toward self-calibration, transfer across sessions, and running

on the implant rather than on a laboratory computer. Those changes matter more for clinical

on the implant rather than on a laboratory computer. For clinical deployment those changes matter

deployment than any further accuracy gain. Beyond that, the question that recurs in discussions of

more than any further gain in accuracy. Beyond that, a question that recurs in discussions of

[[whole-brain-emulation]] applies here in miniature: how much of a system's function can be

recovered from the measurements a practical device can make? For motor intention the answer has

turned out to be a surprising amount from surprisingly few neurons. Whether that generalizes beyond