Mechanisms
Memory Systems · Non-Declarative
Instrumental Learning (Action–Outcome and Habit)
Example
Two people take the same supplement. One would stop the moment they learned it did nothing; the other reaches for the bottle at 9pm without the question arising. Both are doing the same act for the same history, and what differs is how much of it is running on what the act gets them.
Instrumental Learning in Humans
Definition
Doing something and seeing what follows changes whether you do it again, and two systems produce that change. One tracks what the action actually gets you and stays sensitive to whether the outcome is still worth having. The other simply repeats what worked before. Both contribute to any given choice rather than taking turns, and what varies is the balance between them, which differs reliably from person to person (Daw et al., 2011; Gillan et al., 2016).
The diagnostic is outcome devaluation: make the reward worthless, then ask whether behaviour changes. If it does, the goal-directed system is in control. If it does not, the habit system has taken over.
Scope. In rats, control shifts reliably from the goal-directed to the habit system, and the driver appears not to be repetition itself but a fall in experienced contingency, the correlation between changes in response rate and changes in reward rate. Overtraining works by flattening that correlation, which is why interval schedules produce habits and ratio schedules do not even when reinforcement is equated (Yin & Knowlton, 2006). The shift is not unconditional even in rats: with two actions for two outcomes, extended training leaves behaviour goal-directed, because choosing either action keeps the experienced contingency high (Yin & Knowlton, 2006), and offering a choice between responses during training likewise prevents behavioural autonomy (de Wit et al., 2018). The founding demonstration reached a compatible conclusion, finding the extended-training effect may reflect exposure to the reinforcer rather than response practice (Adams, 1982).
The behavioural version of this shift rests on one human study (Tricomi, Balleine, & O'Doherty, 2009). De Wit and colleagues (2018) report five failures across variants of four established devaluation paradigms, including two replications of that study, and a preregistered four-laboratory replication with 306 participants found no effect of training amount at any site (Pool et al., 2022). Human neural evidence for the network shift is covered under Neural Substrate. This row therefore treats the two systems as concurrent and differentially weighted, which human behavioural evidence supports, and treats the transition as established in non-human animals with partial neural support in humans.
Neural Substrate
The two systems map onto distinct cortico-basal ganglia networks rather than onto isolated regions. Goal-directed action depends on an associative network comprising prefrontal and parietal association cortices, dorsomedial striatum, and associative pallidum; habitual responding depends on a sensorimotor network comprising sensorimotor cortices, dorsolateral striatum, and motor pallidum (Yin & Knowlton, 2006). Lesions of dorsolateral striatum leave rats sensitive to outcome devaluation under training that normally produces habits, while lesions or inactivation of posterior dorsomedial striatum abolish sensitivity to both devaluation and contingency degradation. Damage to either network results in the other assuming control. Prelimbic medial prefrontal cortex, which projects heavily to dorsomedial striatum, is likewise required for devaluation sensitivity (Balleine & Dickinson, 1998). The dissociation is mirrored at the cellular level, with dorsomedial striatum expressing NMDA and D1-dependent long-term potentiation and dorsolateral striatum expressing D2-dependent long-term depression (Yin & Knowlton, 2006).
The rodent regions have primate homologues, with dorsomedial striatum corresponding to caudate and dorsolateral striatum to putamen (Yin & Knowlton, 2006). Human imaging is consistent with the network account: learning new motor responses activates caudate and dorsolateral prefrontal cortex, and with well-learned sequences activation shifts to putamen and motor cortices, with the same shift reported for automaticity (Yin & Knowlton, 2006). In the one free-operant devaluation study, cue-related activation in right posterior putamen extending into globus pallidus increased across three days of training while ventromedial prefrontal activity anticipating reward did not diminish, which the authors read as both circuits engaged concurrently and competing for control (Tricomi et al., 2009).
Two cautions. Yin and Knowlton argue explicitly against localizing these functions to structures rather than networks, and reject the multiple memory systems framework that assigns functions to anatomical entities. And the segregation is less clean than the framework implies: ventral striatal prediction error signals carry model-based influence rather than being purely model-free (Daw et al., 2011), and distinct recruitment of the two striatal regions erodes with extended training (Vandaele et al., 2019).
AI at a Glance
In humans, actions that pay off become more likely through two concurrent systems, one that tracks what an action gets and drops it when that outcome stops being worth having and one that repeats what worked, and choice reflects a person-specific mixture of both (Daw et al., 2011; Gillan et al., 2016). Language models reproduce that mixture, weighted toward the goal-directed side, when the two-step task is written into their context, and acquire their standing dispositions through reinforcement learning from human feedback, which is instrumental conditioning applied at training and finished before use (Binz & Schulz, 2023; Coda-Forno et al., 2024; Christiano et al., 2017).
Reinforcement learning from human feedback (RLHF): Training a model by having people compare pairs of its outputs, fitting a reward model to those comparisons, and optimising the model by reinforcement learning against that reward model (Christiano et al., 2017).
Instrumental Learning in AI
How Far It Has Been Measured
Measurement in AI: Indirectly Measured
AI Memory Performance: Comparable to Humans approximate provisional
Where It Lives in the System
Weights, with in-session adjustment at context
What Has Been Measured
- In training, language models are shaped by reinforcement learning from human feedback, meaning their internal settings are adjusted according to how people rated the model's own outputs; this is learning from consequences, but it is judged by output quality, not by instrumental-learning experiments (Ouyang et al., 2022).
- CogBench ran 35 language models in conversation on the probabilistic instrumental learning task from Lefebvre et al. (2017), four interleaved slot-machine pairs over 96 trials, scoring money earned on a scale where 0 is a random chooser and 1 is the average human participant; all seven established models in its main figure reached human level on this task, and GPT-4 and Claude-1 reached it on five of six tasks, while every one of the seven reached it on at least three (Coda-Forno et al., 2024).
- The human results behind that scale are documented: 50 healthy adults in an fMRI experiment and 35 more in a behavioural one chose repeatedly between pairs of symbols that paid out with 25% or 75% probability, learning from wins and losses (Lefebvre et al., 2017).
- The comparison is approximate and uneven: the task and money scale were shared, but every past choice and reward stayed written in the model's prompt while people had to learn and remember them, and CogBench reports only the human average, with no spread and no statistical test against people; GPT-4 therefore matched the human average on a test that favoured it, a tie that counts as Comparable and is provisional (Coda-Forno et al., 2024).
- Systems differ widely beyond CogBench's leaders: of GPT-3.5, GPT-4 and Llama 2 tested on bandit problems, only GPT-4 with step-by-step reasoning and an externally summarised interaction history explored satisfactorily (Krishnamurthy et al., 2024).
- GPT-3 on the horizon bandit task was indistinguishable from people with short horizons (p = .97) but earned significantly lower regret overall (M = 2.72 vs 3.24, human SD = 10.26), with past outcomes shown in the prompt; this result is not used because the human data come from a conference abstract (Zaller et al., 2021) whose sample size and procedure the paper does not report (Binz & Schulz, 2023).
- In saved memory, agents such as Reflexion store written lessons from task feedback to guide later attempts, but no study has tested this, or any AI system, on outcome devaluation or habit formation, so the rating rests on reward learning within a single conversation (Shinn et al., 2023).
Where a Mechanism Can Live
04 Product
The whole configuration, and the only thing a benchmark observes directly.
01
Weights
Across deployments
02
Context Window
Within a session
03
Memory Layer
Across sessions
Which of the three a result belongs to is inferred from it, not seen in it.
Read more
Association cortex
Cortex in the cerebral cortex
The cortex that is neither primary sensory nor primary motor, defined by what it is not.
Association cortex is everything outside the primary sensory and motor areas: disproportionately much of the human cortical sheet, concentrated in prefrontal, lateral temporal and posterior parietal territory. It is divided into unimodal areas, which stay within one sense, and heteromodal areas, which combine several. The category is negative and historical, inherited from nineteenth-century accounts in which these late-myelinating regions were credited with higher intellectual functions, and it groups together tissue with little in common. Work on cortical gradients describes a continuum running from sensory to transmodal cortex rather than a set of named boxes.
Basal ganglia
Subcortical
A set of interconnected subcortical nuclei (striatum, pallidum, subthalamic nucleus, substantia nigra) forming loops with cortex and thalamus.
Not a system defined by adjacency but by circuitry: cortex projects to striatum, striatum to pallidum and substantia nigra, and these back to cortex through thalamus, in several parallel loops that stay segregated from motor through associative to limbic territory. The "direct and indirect pathway" model organized the field and explains a good deal of Parkinson's and Huntington's phenomenology, but its predictions about firing rates have not held cleanly, and much current work treats pattern and timing as well as rate as what the loops carry.
Caudate nucleus
Nucleus in the dorsal striatum
A C-shaped nucleus arching alongside the lateral ventricle, the association-connected half of the dorsal striatum.
Head, body and tail follow the curve of the lateral ventricle, with the head bulging into its frontal horn. It receives from prefrontal and parietal association cortex, keeping it largely separate from the sensorimotor input that dominates the putamen. Caudate volume loss is among the earliest and most characteristic imaging findings in Huntington's disease, visible as widened ventricular horns before motor symptoms. Its activation in human studies is usually reported with feedback-based learning and with goal-directed action selection rather than movement execution.
Dorsal striatum
Subcortical in the striatum
The caudate and putamen together, the striatal territory receiving from sensorimotor and association cortex.
In rodents the caudate and putamen are not separated by an internal capsule and the whole mass is called caudate-putamen, which is why the dorsal/ventral division travels better across species than the primate names do. A further split is functional and well supported: the dorsomedial portion, with association cortex input, tracks action-outcome contingencies, while the dorsolateral portion, with sensorimotor input, supports responses that continue after the outcome is devalued. The shift of control from one to the other across training is a widely used animal model of habit formation.
Dorsolateral prefrontal cortex
Cortex in the prefrontal cortex
The upper outer surface of the prefrontal cortex, on and around the middle frontal gyrus, roughly Brodmann areas 9 and 46.
The dorsolateral prefrontal cortex sits on the lateral convexity of the frontal lobe, centered on the middle frontal gyrus and covering approximately Brodmann areas 9 and 46; its borders are drawn differently by different labs and no sulcus marks them. It is densely connected with posterior parietal cortex, the dorsal striatum, and the mediodorsal thalamus. Activity here tracks working-memory delays, task switching, and rule maintenance, and it is the usual target for prefrontal transcranial stimulation. Whether it stores the contents of working memory, holds the rules for acting on them, or simply reflects task difficulty is genuinely contested, and "executive function" names the dispute rather than settling it.
Globus pallidus
Nucleus in the basal ganglia
The pale, myelin-rich nucleus medial to the putamen, whose two segments are the basal ganglia's internal relay and its output.
Pale because striatal axons entering it are heavily myelinated. Its neurons fire tonically at high rates and are GABAergic, so both segments inhibit their targets continuously and signal by pausing. The internal segment projects to thalamus as a basal ganglia output; the external segment projects to the subthalamic nucleus and back into the circuit, a role the classical model understated. Pallidotomy, and later deep brain stimulation of the internal segment, are established treatments for Parkinsonian motor symptoms and dystonia, which is unusually direct evidence of causal involvement.
Medial prefrontal cortex
Cortex in the prefrontal cortex
The inner wall of the prefrontal cortex, facing the midline between the two hemispheres and wrapping around the corpus callosum's front end.
The medial prefrontal cortex is the flat inner face of the frontal lobe, running from near the frontal pole back to the genu of the corpus callosum, where it meets the cingulate. It is usually divided into dorsal and ventral halves with different connections: the ventral part to amygdala, hypothalamus and brainstem, the dorsal part to lateral frontal and temporal cortex. It is a hub of the rodent memory-consolidation literature and of the human self-reference and default-mode literatures. The term is used loosely and often includes tissue another author would call cingulate.
Motor cortex
Cortex in the frontal lobe
The frontal cortex involved in planning and executing movement, comprising primary motor, premotor and supplementary motor areas.
Motor cortex is a collective term for the frontal regions immediately in front of the central sulcus: primary motor cortex on the precentral gyrus, and the premotor and supplementary motor areas ahead of it. All are agranular, lacking the granular layer IV of sensory cortex, and all send fibers into the corticospinal tract, not just the primary area. The tidy hierarchy of planning-to-execution implied by the names is a simplification; premotor and primary regions are both engaged as a movement begins, in a partial rather than strict hierarchy, and the borders between the fields are drawn cytoarchitectonically, area 4 against area 6 at the precentral sulcus, not by stimulation thresholds.
Prefrontal cortex
Cortex in the frontal lobe
The frontal cortex in front of the motor areas, defined classically by its connections with the mediodorsal thalamus rather than by any one function.
The prefrontal cortex occupies the front of the frontal lobe, anterior to premotor and motor cortex, and accounts for roughly a sixth to a quarter of human cortex depending on where the posterior border is drawn. Three competing definitions are in use (projection from the mediodorsal thalamus, the presence of a granular layer IV, and the absence of movement on electrical stimulation), and they do not enclose the same tissue, which is why rodent-to-primate homology is still argued over. It is reliably engaged by tasks with delays, rules, and conflicting options; what it contributes to those tasks is the open question, not whether it participates.
Putamen
Nucleus in the dorsal striatum
The lateral striatal nucleus, separated from caudate by the internal capsule and dominated by sensorimotor cortical input.
Lies lateral to the globus pallidus and medial to the external capsule and claustrum, forming the outer part of the lentiform nucleus. Its cortical input comes mostly from motor, premotor and somatosensory areas, arranged somatotopically, and it is the striatal territory where dopamine loss in Parkinson's disease is most severe: posterior putamen first, which fits the early motor signs, though the cognitive decline of the disease tracks cortical Lewy pathology rather than this striatal gradient. Its involvement in stimulus-response learning has been tested directly in people, where focal putaminal lesions impair rule-based category learning while sparing information-integration learning.
Striatum
Subcortical in the basal ganglia
The input structure of the basal ganglia: caudate, putamen and nucleus accumbens, receiving from nearly all of cortex.
Ninety-five percent of its neurons are GABAergic medium spiny cells, held at a hyperpolarized resting potential and firing only when converging excitatory input depolarizes them, a pattern described most clearly under anesthesia and in slow-wave sleep. They come in two classes, distinguished by dopamine receptor type and by whether they project straight to the output nuclei or by way of the external pallidum. Dopamine from the midbrain reaches all of it, and acetylcholine from large local interneurons modulates it. The striatum is the usual localization for habit formation and reinforcement-based learning, though how those relate to its role in ordinary movement initiation is unresolved.
Ventral striatum
Subcortical in the striatum
The lower striatal territory (nucleus accumbens and neighboring olfactory tubercle) receiving limbic and prefrontal input.
Heimer's term for striatal tissue defined by its inputs rather than its position above or below the anterior commissure: it takes projections from amygdala, hippocampus, and orbital and medial prefrontal cortex, and sends output to ventral pallidum rather than to the internal pallidal segment. It is the limbic end of the cortico-striatal loops, and the usual localization in human imaging for signals scaling with anticipated outcomes. Human studies frequently report "ventral striatum" where the resolution cannot distinguish accumbens from adjacent caudate or putamen.
Adams (1982)
Variations in the Sensitivity of Instrumental Responding to Reinforcer Devaluation
In
The Quarterly Journal of Experimental Psychology Section B, 34(2b), 77-98
Abstract
In five experiments hungry rats were trained to make a lever press response for a sucrose reinforcer. That sucrose was subsequently devalued by conditioning a food-aversion to it, and the ability of the rats to integrate knowledge about the instrumental contingency with that gained from aversion training was assessed in an extinction test. Experiment I showed successful integration following limited but not extended instrumental training. Experiment II suggested that the crucial factor was the spacing of training; successful integration was seen after massed but not distributed training. The third experiment implicated distributed experience with the reinforcer, rather than distributed...
Find it
Balleine & Dickinson (1998)
Goal-directed instrumental action: contingency and incentive learning and their cortical substrates
In
Neuropharmacology, 37(4-5), 407-419
Abstract
Instrumental behaviour is controlled by two systems: a stimulus–response habit mechanism and a goal-directed process that involves two forms of learning. The first is learning about the instrumental contingency between the response and reward, whereas the second consists of the acquisition of incentive value by the reward. Evidence for contingency learning comes from studies of reward devaluation and from demonstrations that instrumental performance is sensitive not only the probability of contiguous reward but also to the probability of unpaired rewards. The process of incentive learning is evident in the acquisition of control over performance by primary motivational states. Preliminary...
Find it
Binz & Schulz (2023)
Using cognitive psychology to understand GPT-3
In
Proceedings of the National Academy of Sciences, 120(6)
Abstract
We study GPT-3, a recent large language model, using tools from cognitive psychology. More specifically, we assess GPT-3's decision-making, information search, deliberation, and causal reasoning abilities on a battery of canonical experiments from the literature. We find that much of GPT-3's behavior is impressive: It solves vignette-based tasks similarly or better than human subjects, is able to make decent decisions from descriptions, outperforms humans in a multiarmed bandit task, and shows signatures of model-based reinforcement learning. Yet, we also find that small perturbations to vignette-based tasks can lead GPT-3 vastly astray, that it shows no signatures of directed...
Find it
Christiano et al. (2017)
Deep reinforcement learning from human preferences
In
arXiv
Abstract
For sophisticated reinforcement learning (RL) systems to interact usefully with real-world environments, we need to communicate complex goals to these systems. In this work, we explore goals defined in terms of (non-expert) human preferences between pairs of trajectory segments. We show that this approach can effectively solve complex RL tasks without access to the reward function, including Atari games and simulated robot locomotion, while providing feedback on less than 1% of our agent's interactions with the environment. This reduces the cost of human oversight far enough that it can be practically applied to state-of-the-art RL systems. To demonstrate the flexibility of our approach,...
Find it
Coda-Forno et al. (2024)
CogBench: a large language model walks into a psychology lab
In
arXiv
Abstract
Large language models (LLMs) have significantly advanced the field of artificial intelligence. Yet, evaluating them comprehensively remains challenging. We argue that this is partly due to the predominant focus on performance metrics in most benchmarks. This paper introduces CogBench, a benchmark that includes ten behavioral metrics derived from seven cognitive psychology experiments. This novel approach offers a toolkit for phenotyping LLMs' behavior. We apply CogBench to 35 LLMs, yielding a rich and diverse dataset. We analyze this data using statistical multilevel modeling techniques, accounting for the nested dependencies among fine-tuned versions of specific LLMs. Our study...
Find it
Daw et al. (2011)
Model-Based Influences on Humans' Choices and Striatal Prediction Errors
In
Neuron, 69(6), 1204-1215
Abstract
The mesostriatal dopamine system is prominently implicated in model-free reinforcement learning, with fMRI BOLD signals in ventral striatum notably covarying with model-free prediction errors. However, latent learning and devaluation studies show that behavior also shows hallmarks of model-based planning, and the interaction between model-based and model-free values, prediction errors, and preferences is underexplored. We designed a multistep decision task in which model-based and model-free influences on human choice behavior could be distinguished. By showing that choices reflected both influences we could then test the purity of the ventral striatal BOLD signal as a model-free report....
Find it
Gillan et al. (2016)
Characterizing a psychiatric symptom dimension related to deficits in goal-directed control
In
eLife, 5
Abstract
Prominent theories suggest that compulsive behaviors, characteristic of obsessive-compulsive disorder and addiction, are driven by shared deficits in goal-directed control, which confers vulnerability for developing rigid habits. However, recent studies have shown that deficient goal-directed control accompanies several disorders, including those without an obvious compulsive element. Reasoning that this lack of clinical specificity might reflect broader issues with psychiatric diagnostic categories, we investigated whether a dimensional approach would better delineate the clinical manifestations of goal-directed deficits. Using large-scale online assessment of psychiatric symptoms and...
Find it
Krishnamurthy et al. (2024)
Can large language models explore in-context?
In
arXiv
Abstract
We investigate the extent to which contemporary Large Language Models (LLMs) can engage in exploration, a core capability in reinforcement learning and decision making. We focus on native performance of existing LLMs, without training interventions. We deploy LLMs as agents in simple multi-armed bandit environments, specifying the environment description and interaction history entirely in-context, i.e., within the LLM prompt. We experiment with GPT-3.5, GPT-4, and Llama2, using a variety of prompt designs, and find that the models do not robustly engage in exploration without substantial interventions: i) Across all of our experiments, only one configuration resulted in satisfactory...
Find it
Lefebvre et al. (2017)
Behavioural and neural characterization of optimistic reinforcement learning
In
Nature Human Behaviour, 1(4)
Find it
Ouyang et al. (2022)
Training language models to follow instructions with human feedback
In
arXiv
Abstract
Making language models bigger does not inherently make them better at following a user's intent. For example, large language models can generate outputs that are untruthful, toxic, or simply not helpful to the user. In other words, these models are not aligned with their users. In this paper, we show an avenue for aligning language models with user intent on a wide range of tasks by fine-tuning with human feedback. Starting with a set of labeler-written prompts and prompts submitted through the OpenAI API, we collect a dataset of labeler demonstrations of the desired model behavior, which we use to fine-tune GPT-3 using supervised learning. We then collect a dataset of rankings of model...
Find it
Pool et al. (2022)
Determining the effects of training duration on the behavioral expression of habitual control in humans: a multilaboratory investigation
In
Learning & Memory, 29(1), 16-28
Abstract
It has been suggested that there are two distinct and parallel mechanisms for controlling instrumental behavior in mammals: goal-directed actions and habits. To gain an understanding of how these two systems interact to control behavior, it is essential to characterize the mechanisms by which the balance between these systems is influenced by experience. Studies in rodents have shown that the amount of training governs the relative expression of these two systems: Behavior is goal-directed following moderate training, but the more extensively an instrumental action is trained, the more it becomes habitual. It is less clear whether humans exhibit similar training effects on the expression...
Find it
Shinn et al. (2023)
Reflexion: Language Agents with Verbal Reinforcement Learning
In
arXiv
Abstract
Large language models (LLMs) have been increasingly used to interact with external environments (e.g., games, compilers, APIs) as goal-driven agents. However, it remains challenging for these language agents to quickly and efficiently learn from trial-and-error as traditional reinforcement learning methods require extensive training samples and expensive model fine-tuning. We propose Reflexion, a novel framework to reinforce language agents not by updating weights, but instead through linguistic feedback. Concretely, Reflexion agents verbally reflect on task feedback signals, then maintain their own reflective text in an episodic memory buffer to induce better decision-making in...
Find it
Tricomi et al. (2009)
A specific role for posterior dorsolateral striatum in human habit learning
In
European Journal of Neuroscience, 29(11), 2225-2232
Abstract
Habits are characterized by an insensitivity to their consequences and, as such, can be distinguished from goal-directed actions. The neural basis of the development of demonstrably outcome-insensitive habitual actions in humans has not been previously characterized. In this experiment, we show that extensive training on a free-operant task reduces the sensitivity of participants’ behavior to a reduction in outcome value. Analysis of functional magnetic resonance imaging data acquired during training revealed a significant increase in task-related cue sensitivity in a right posterior putamen–globus pallidus region as training progressed. These results provide evidence for a shift from...
Find it
Vandaele et al. (2019)
Distinct recruitment of dorsomedial and dorsolateral striatum erodes with extended training
In
eLife, 8
Abstract
Hypotheses of striatal orchestration of behavior ascribe distinct functions to striatal subregions, with the dorsolateral striatum (DLS) especially implicated in habitual and skilled performance. Thus neural activity patterns recorded from the DLS, but not the dorsomedial striatum (DMS), should be correlated with habitual and automatized performance. Here, we recorded DMS and DLS neural activity in rats during training in a task promoting habitual lever pressing. Despite improving performance across sessions, clear changes in corresponding neural activity patterns were not evident in DMS or DLS during early training. Although DMS and DLS activity patterns were distinct during early...
Find it
Yin & Knowlton (2006)
The role of the basal ganglia in habit formation
In
Nature Reviews Neuroscience, 7(6), 464-476
Abstract
Many organisms, especially humans, are characterized by their capacity for intentional, goal-directed actions. However, similar behaviours often proceed automatically, as habitual responses to antecedent stimuli. How are goal-directed actions transformed into habitual responses? Recent work combining modern behavioural assays and neurobiological analysis of the basal ganglia has begun to yield insights into the neural basis of habit formation.
Find it