Kosslyn The Dual Coding Theory Experiments – Allan Paivio The Picture

The nature of the human mind’s internal representational architecture stands as one of the most enduring and fiercely contested battlegrounds in cognitive science, psychology, and the philosophy of mind. For decades following the decline of radical behaviorism, cognitive scientists grappled with a fundamental ontological question: in what format does the mind encode, manipulate, and retrieve knowledge about the physical world? Does the cognitive apparatus operate exclusively as an amodal, symbolic Turing machine manipulating arbitrary linguistic propositions, or does it harbor functionally distinct, modality-specific representational formats that preserve the continuous geometric and perceptual contours of sensory experience? The quest to resolve these questions gave rise to two monumental frameworks: Allan Paivio’s Dual Coding Theory and Stephen Kosslyn’s depictive representation hypothesis. Together with their rigorous chronometric paradigms, these frameworks revolutionized our understanding of the mental image.

Allan Paivio challenged the amodal orthodoxy by proposing that human cognition is subserved by two structurally and functionally independent, yet richly interconnected, symbolic subsystems: a nonverbal structural system specialized for the direct representation of perceptually grounded events and entities (anchored by perceptual units termed imagens), and a verbal system specialized for linguistic processing (anchored by discrete linguistic units termed logogens). Central to Paivio’s empirical legacy was his relentless documentation of the “Picture Superiority Effect”—a robust cognitive phenomenon demonstrating that pictorial stimuli elicit vastly superior recall and recognition memory compared to their orthographic counterparts, owing to the automatic, additive recruitment of dual referential codes. Paivio elevated the nonverbal pictorial code from a marginalized subjective curiosity into a primary, quantifiable engine of mnemonic efficacy.

Simultaneously, Stephen Kosslyn advanced this spatial revolution by formulating the computational and spatial mechanics of visual mental imagery. Where Paivio established the cognitive independence and associative dynamics of the nonverbal code, Kosslyn addressed its internal spatial topography. Through a series of landmark mental chronometry experiments—exemplified by the iconic fictional island scanning paradigms—Kosslyn demonstrated that mental images are not merely disembodied semantic networks decorated with subjective experiential fluff, but active, depictive representations instantiated within an inherently spatial computational workspace: the visual buffer. Kosslyn showed that the subjective experience of inspecting, scanning, rotating, and zooming into a mental picture is accompanied by strict metric preservation, where internal operations systematically mirror the physical geometry of the distal stimulus. This exhaustive investigation explores the empirical architectures, chronometric methodologies, neurobiological substrates, and enduring theoretical disputes that define Paivio’s Dual Coding Theory and Kosslyn’s depictive paradigm.

1. Historical Foundations of Mental Imagery and Cognitive Architecture

1.1 The Epistemological Shift from Behaviorism to Mental Representation

The scientific legitimation of mental imagery required an epistemological insurrection against the dominant paradigm of early-to-mid-twentieth-century psychology: classical Watsonian and Skinnerian behaviorism. John B. Watson, in his foundational 1913 manifesto, deliberately expunged internal cognitive states, consciousness, and introspective reports from the lexicon of rigorous psychological science. Within the strictures of behaviorist doctrine, the “mental image” was dismissed as an unobservable, non-functional epiphenomenon—a ghostly vestige of Cartesian dualism that eluded empirical operationalization. The internal life of the organism was treated as a black box; behavior was conceptualized purely as the lawful chaining of observable stimuli to discrete, measurable motor responses. To posit that an individual could “inspect” an internal, pictorial mental state was deemed unscientific, unparsimonious, and fundamentally unverifiable.

The mid-twentieth-century cognitive revolution—ignited by advancements in cybernetics, information theory, computer science, and generative linguistics—fundamentally cracked this anti-representational stance. Scholars such as George Miller, Donald Broadbent, and Jerome Bruner argued that complex human performance, from problem solving to language acquisition, could never be adequately explained by passive stimulus-response conditioning. The cognitive revolution introduced the computational metaphor of mind, formalizing human cognition as the algorithmic transformation of internal informational states. Yet, despite this paradigm shift, early cognitive psychology remained profoundly logocentric. Influenced by formal logic and the syntactic parsing engines of early artificial intelligence, pioneers of the cognitive revolution initially exhibited intense resistance toward non-verbal, spatial, or perceptual constructs. The mind was enthusiastically conceptualized as an amodal symbol manipulator operating over propositional, language-like representations. Non-verbal constructs like imagery were still viewed with deep suspicion, considered dangerously close to the unquantifiable subjectivity that behaviorism had spent decades attempting to eradicate.

The epistemological challenge of the 1960s and 1970s was to rescue mental imagery from the realm of idiosyncratic subjective introspection and reposition it as an objective, functionally necessary component of human cognitive architecture. To overcome the entrenched skepticism of both classical behaviorists and amodal cognitive purists, researchers needed experimental paradigms that could quantify internal spatial manipulation without relying on subjective introspective verbalizations. This required the invention of mental chronometry—the precise measurement of reaction times in milliseconds as participants performed systematic, internally directed cognitive operations over visual and verbal information.

1.2 Philosophical Precursors to Imagery Research

Long before cognitive scientists deployed tachistoscopes and response timers, the philosophical ontology of mental imagery had preoccupied thinkers for millennia. The roots of representational imagery can be traced directly to Aristotle’s treatise De Anima, wherein he famously asserted that “the soul never thinks without a phantasm” (mental image). For Aristotle, thought was inextricably linked to sensory memory; sensory impressions were preserved in the mind as quasi-perceptual copies or traces (phantasmata) that formed the foundational material upon which intellective, abstract deliberation operated. Aristotle understood that thought required an internalized sensory medium, positing that contemplation of abstract geometric or categorical truths depended upon the instantiation of concrete perceptual forms.

This perceptual tradition found renewed vigor in the British empiricist school of the seventeenth and eighteenth centuries, notably through the work of John Locke, David Hume, and George Berkeley. In his Treatise of Human Nature (1739), David Hume constructed an entire epistemological framework predicated on the distinction between “impressions”—the lively, direct sensory inputs derived from physical encounters with the environment—and “ideas”—the secondary, faint copies of these impressions preserved in memory and imagination. For Hume, ideas were not fundamentally different in kind from sensations; rather, they were qualitatively identical representations that differed solely in their degree of “force and vivacity.” Berkeley extended this logic to argue against the very possibility of abstract general ideas, contending that one cannot form an image of a triangle without it being specifically isosceles, scalene, or equilateral, thereby emphasizing the inherently concrete, depictive nature of internal conceptual representations.

However, the empiricist consensus was perpetually haunted by the specter of epiphenomenalism. Critics questioned whether these internal sensory impressions possessed authentic functional efficacy or whether they were merely computational “exhaust”—incidental byproducts of underlying physiological or intellectual mechanisms that played no genuine causal role in cognition. If an internal image was merely a passive illustration accompanying cognitive processing, it held no explanatory value for psychology. The core philosophical dispute, which would later erupt into the great imagery debate of the late twentieth century, centered on this functional question: does the cognitive system physically operate upon, measure, and transform analog pictorial structures, or are these phenomenological “pictures in the head” simply an experiential illusion generated by underlying non-spatial, propositional logic networks?

1.3 The Emergence of Paivio and Kosslyn in the Post-Behaviorist Landscape

It was against this historical backdrop of lingering behaviorist skepticism and amodal computational dominance that Allan Paivio and Stephen Kosslyn emerged, each attacking the problem of non-verbal mental representation from distinct yet profoundly complementary empirical angles. Working at the University of Western Ontario in the late 1960s, Allan Paivio approached the problem through the lens of verbal learning, psycholinguistics, and mnemonic efficacy. Paivio observed a striking paradox: while early cognitive psychology modeled human memory primarily through linguistic strings, nonsense syllables, and verbal associative networks, human performance demonstrated catastrophic variance when linguistic stimuli varied in their concrete, image-evoking capacity.

Through systematic, large-scale psycholinguistic experiments, Paivio began quantifying the semantic attributes of thousands of words, establishing concrete scales for imagery value, concreteness, and meaningfulness. He demonstrated that concrete words (such as “apple” or “locomotive”) were memorized, clustered, and recalled with vastly superior accuracy compared to abstract words matched for lexical frequency (such as “justice” or “epistemology”). Paivio posited that this empirical disparity could only be explained by assuming that concrete words possessed privileged access to an autonomous, perceptual-based representational subsystem that operated concurrently with, yet independently from, the language processing engine. This marked the empirical birth of the Dual Coding Theory (DCT), offering a rigorous, behaviorally measurable challenge to linguistic-only models of memory.

Shortly thereafter, during the 1970s at Harvard University, Stephen Kosslyn launched an ambitious empirical program designed to decode the internal metric and functional mechanics of these non-verbal representations. While Paivio focused broadly on structural memory systems, associative processing, and categorical recall, Kosslyn sought to characterize the real-time operational dynamics of visual mental images as spatial arrays. Drawing on cognitive chronometry pioneered by Donders and Sternberg, and directly inspired by Roger Shepard’s groundbreaking mental rotation paradigms, Kosslyn set out to demonstrate that mental images preserve explicit metric distances, spatial arrangements, and visual resolutions.

The convergence of Paivio’s linguistic-pictorial memory paradigms with Kosslyn’s high-precision spatial chronometry permanently altered the representational landscape. Together, they established that non-verbal, pictorial cognitive architecture could be subjected to the same degree of experimental rigor, mathematical formalization, and predictive verification as any classical amodal linguistic paradigm. They demonstrated that the mental image was not an unobservable, esoteric phantasm, but a functionally verifiable computational substrate governed by lawful psychological principles.

2. Allan Paivio’s Dual Coding Theory: Architectural Framework and Mechanics

2.1 Structural Components: Imagens and Logogens

Allan Paivio’s Dual Coding Theory is grounded in the foundational postulate that human cognitive architecture is bifurcated into two functionally independent, structurally distinct, yet dynamically interactive symbolic subsystems: the nonverbal (structural/pictorial) system and the verbal (linguistic) system. These subsystems are specialized to handle distinct classes of informational input from the environment, utilizing structurally divergent internal representational units known as imagens and logogens.

Imagens are the foundational representational building blocks of the nonverbal subsystem. Paivio defined imagens as modality-specific, perceptually grounded internal structures that preserve the continuous, dynamic properties of nonverbal sensory experiences. Critically, imagens are not exclusively visual; they encompass haptic, acoustic, motoric, and visceral sensory dimensions. A visual imagen might represent the continuous structural contour, color distribution, and spatial volume of a violin; an acoustic imagen preserves the characteristic timbre, pitch trajectory, and envelope of its sound; and a motoric imagen encodes the kinesthetic programs required to draw a bow across its strings. Imagens operate through continuous, analog properties, preserving natural geometric and spatial relationships. They are organized hierarchically as nested perceptual structures—for instance, the imagen of a human face contains nested within it sub-imagens corresponding to the eyes, nose, and mouth—and they are processed in a parallel, synchronous fashion, permitting the instantaneous, global inspection of an entire structural scene.

Conversely, logogens—a term adapted and modified from John Morton’s early psycholinguistic models—constitute the structural substrate of the verbal subsystem. Logogens are discrete, symbolic, and amodal or modality-linked linguistic representations that correspond to the lexical units of human language, including spoken words, printed text, morphemes, and syntactic structures. Unlike imagens, logogens do not share any physical, continuous, or structural isomorphism with the referents they designate. The English logogen “horse,” the French logogen “cheval,” and the German logogen “Pferd” share entirely arbitrary, conventionalized relationships with the actual biological quadrupeds they denote. Logogens are organized in sequential, linear, and categorical hierarchies governed by phonological, orthographic, and grammatical rules. While imagens are scanned synchronously and spatially, logogens are fundamentally constrained by temporal, serial processing mechanisms: words must be uttered, read, and syntactically parsed in a sequential progression across time.

The distinction between these two structural units is summarized in the structural properties governing their operation:

• Imagens: Continuous, analog representations; modality-specific (visual, auditory, motoric); organized hierarchically as part-whole spatial relations; processed synchronously and in parallel.
• Logogens: Discrete, arbitrary symbolic representations; linguistically bound (phonemic, graphemic); organized linearly through syntactic and associative hierarchies; processed sequentially across time.

2.2 Operational Levels of Processing: Representational, Referential, and Associative

Within Paivio’s Dual Coding Theory, cognitive activity does not consist merely of static representations resting dormant within memory; it involves the dynamic, lawful propagation of activation across three distinct operational levels of processing: representational processing, referential processing, and associative processing. These three processing levels characterize the flow of informational activation both within and between the verbal and nonverbal cognitive subsystems.

Representational processing constitutes the initial, direct sensory activation of internal codes by external environmental stimuli. This is the entry-level interface where incoming environmental input triggers its direct modality-specific representational unit. When an individual views an actual physical line drawing of a chair, the photic stimulation falling on the retina directly activates the corresponding nonverbal imagen representing that chair via representational processing. Similarly, when the printed word “CHAIR” is visually encountered or the spoken word /tʃɛər/ is acoustically registered, the perceptual input directly activates the internal orthographic or phonological logogen within the verbal subsystem. Representational processing is highly automatic, stimulus-bound, and requires no cross-modal translation; it represents the immediate internal realization of sensory input.

Referential processing defines the cross-system translational mechanisms that bridge the divide between the verbal and nonverbal domains. Referential processing occurs whenever activation crosses subsystem boundaries—that is, when a linguistic logogen triggers a corresponding nonverbal imagen, or conversely, when an imagen activates its appropriate linguistic logogen. For example, when a participant is exposed to the printed word “elephant” and actively generates a visual mental image of the animal’s grey hide, trunk, and large ears, the verbal system is driving activation across the referential link to instantiate an imagen within the nonverbal system. Conversely, when a participant is shown a picture of an elephant and prompted to speak its name aloud, the visual imagen activates the corresponding linguistic logogen (“elephant”) via referential processing. Referential pathways provide the structural bridge that allows humans to verbalize sensory experiences and visualize spoken or written language.

Associative processing refers entirely to internal, intramodal activation that cascades horizontally within a single subsystem, without crossing the border between the verbal and nonverbal domains. Within the verbal subsystem, associative processing is exemplified by classical semantic and lexical networks, where the activation of one logogen automatically spreads to structurally or semantically linked logogens. For instance, the activation of the logogen “doctor” might immediately spread activation to the logogens “nurse,” “hospital,” or “scalpel” through lexical associative pathways. Within the nonverbal system, associative processing operates through the continuous chaining of perceptual scenes and imagens: picturing a hammer might automatically elicit an imagen of an iron nail, a wooden plank, or the motoric sensation of striking a surface. The total cognitive architecture functions as a multi-layered matrix where external stimuli initiate representational processing, which subsequently cascades through associative and referential networks simultaneously.

2.3 Assumptions of Independence and Additivity

The mathematical and empirical core of Paivio’s Dual Coding Theory rests upon two critical architectural postulates: the assumption of functional independence and the assumption of additivity. These two principles provide the theoretical foundation from which Paivio derived falsifiable predictions regarding human memory performance, cognitive load, and multi-modal information processing.

The assumption of functional independence states that the verbal and nonverbal systems can operate completely independently of one another. An individual can process, store, manipulate, and retrieve nonverbal imagens without engaging the verbal system or translating those images into linguistic tokens. Conversely, the linguistic system can parse, analyze, and manipulate logogens purely within an autonomous verbal network without necessitating the instantiation of sensory-based imagens. A subject can memorize a list of abstract phonological tokens or perform complex deductive syllogisms using syntactic transformation rules alone, leaving the nonverbal system largely quiescent. Similarly, one can engage in complex spatial navigation, mental rotation, or aesthetic contemplation of abstract visual forms with zero verbal participation. While the two systems are intimately linked via rich referential connections, neither system requires the continuous co-activation of the other to execute its foundational tasks.

The assumption of additivity posits that when information is successfully encoded into both the verbal and nonverbal subsystems concurrently, the resulting mnemonic trace is the cumulative sum of both individual traces. Because the memory traces stored within the logogenic and imagenic repositories are functionally distinct and independent, they do not compete destructively for the same storage capacity; instead, they establish dual, redundant retrieval pathways. If an item is dual-coded—meaning that both an imagen and a logogen have been laid down in long-term memory—the probability of successfully retrieving that item in a subsequent recall or recognition task is mathematically higher than if it were encoded within only a single system.

Formally, if $P(V)$ represents the independent probability of successfully retrieving an item from the verbal memory trace and $P(NV)$ represents the independent probability of retrieving it from the nonverbal memory trace, the cumulative probability of retrieval failure for a dual-coded item is the product of their independent failure rates, assuming statistical independence:

$$P(\text{Failure}) = [1 – P(V)] \times [1 – P(NV)]$$

Consequently, the overall probability of successful recall, $P(\text{Recall})$, is given by the complementary additivity equation:

$$P(\text{Recall}) = P(V) + P(NV) – [P(V) \times P(NV)]$$

This mathematical formulation yields a clear, definitive prediction: items that spontaneously elicit dual codes, or experimental conditions that explicitly induce participants to form both verbal labels and perceptual images, will consistently yield mnemonic retention rates superior to items or conditions relying upon a single representational format. This simple yet profound architectural assumption served as the computational engine driving Allan Paivio’s experimental demonstrations of human memory, most notably the Picture Superiority Effect.

3. The Picture Superiority Effect: Empirical Evidence and Mechanics

3.1 Experimental Paradigms Demonstrating the Picture Superiority Effect

The Picture Superiority Effect (PSE) represents one of the most robust, highly replicated empirical phenomena in the annals of cognitive psychology. In its most elementary manifestation, the effect demonstrates that human subjects exhibit substantially higher memory performance—manifested across free recall, cued recall, and recognition memory protocols—for visual pictorial stimuli (such as line drawings, paintings, or photographs) compared to the corresponding printed verbal labels (orthographic words) that designate those identical concepts.

In standard free recall paradigms designed by Paivio and his contemporaries (such as Paivio, Rogers, & Smythe, 1968; Paivio & Csapo, 1969), participants were presented with rapid sequences of stimuli consisting of either simple, unambiguous black-and-white line drawings of everyday objects (e.g., an umbrella, a bicycle, a tree) or the printed English words naming those exact objects (“umbrella,” “bicycle,” “tree”). Stimulus presentation was strictly controlled via tachistoscopes or automated projection systems, typically at rates ranging from 1 to 5 seconds per item. Following an intervening distracter task designed to eliminate immediate short-term working memory contributions (such as counting backward by threes), participants were instructed to recall as many items as possible in any order. The results were uniform and dramatic: subjects consistently recalled significantly more items presented as pictures than items presented as printed words, with memory advantages frequently exceeding 30% to 50%.

Recognition memory experiments pushed this disparity into astronomical territories. In monumental studies conducted by Lionel Standing (1973), subjects were exposed to thousands of distinct photographic images over several days. When subsequently tested with pairs of pictures—one previously seen and one novel distracter—participants demonstrated recognition accuracy rates approaching 95% across sets of up to 10,000 images. When the stimuli were replaced with printed verbal phrases or words, performance degraded at exponentially faster decay rates. Visual pictorial memory displayed an almost boundless capacity relative to orthographic memory.

Complementing the direct comparison of pictures and words, Paivio executed extensive investigations into the intrinsic psycholinguistic properties of vocabulary, categorizing words along precise continua of concreteness (C), imagery value (I), and meaningfulness (m). In these experiments, participants were asked to memorize lists of words that held identical word frequencies and syllable lengths, but drastically differed in their rated imagery value. Paivio’s data established that concrete, high-imagery nouns (e.g., “alligator,” “diamond,” “volcano”) produced memory retention curves that closely mirrored those of actual pictures, whereas abstract, low-imagery nouns (e.g., “validity,” “context,” “prudence”) suffered profound retention deficits. Concrete words functioned as linguistic gateways that immediately unlocked nonverbal representations, mirroring the mnemonic robustness of actual physical drawings.

3.2 Differential Encoding and Referential Redundancy

Why do pictures exert such profound superiority over printed text in cognitive memory stores? Allan Paivio explained this phenomenon through the precise mechanics of differential encoding and asymmetrical referential redundancy. The fundamental premise of Dual Coding Theory is that visual pictures and printed words do not possess equivalent access to the verbal and nonverbal systems during initial perception.

When an individual is exposed to a clear, recognizable picture of an object—say, a horse—the visual stimulus immediately and automatically triggers its corresponding nonverbal imagen via representational processing. However, because human adults live within a profoundly linguistic environment, the presentation of a discrete, nameable visual image almost instantly prompts the viewer to covertly emit an internal verbal label (“horse”). The picture automatically traverses the referential pathway from the nonverbal system to the verbal system. As a direct result, pictorial stimuli are spontaneously, effortlessly, and near-universally dual-coded at the moment of perception. The cognitive system constructs both an imagen (visual perceptual trace) and a logogen (verbal lexical trace) in memory without requiring explicit instructional prompts or deliberate strategic rehearsal.

In stark contrast, the presentation of an orthographic verbal stimulus—the printed word “HORSE”—operates under an asymmetrical constraint. The visual word directly activates its corresponding logogen via representational processing within the verbal system. However, the cross-system referential link from a word to an internal mental image is not automatic; it is effortful, probabilistic, and heavily modulated by task demands, cognitive load, and the individual’s idiosyncratic processing goals. Under standard rapid laboratory presentation rates, participants typically read the word, encode the logogen, and perhaps initiate intra-modal verbal associative processing (linking “horse” to “saddle” or “animal”), but they frequently fail to deliberately instantiate a vivid nonverbal mental imagen. Consequently, the printed word remains predominantly single-coded, residing only within the verbal subsystem.

This differential encoding asymmetry is further compounded by interactions with levels-of-processing dynamics. A visual picture inherently presents an integrated, rich perceptual gestalt featuring spatial contours, implied depth, surfaces, and functional boundaries. These features mandate a deeper level of perceptual extraction than the arbitrary orthographic features of printed letters. The picture provides redundant semantic and sensory cues that enrich the nonverbal memory trace, while simultaneously forcing automatic referential labeling. Thus, when recall is demanded, the pictorial stimulus benefits from two completely distinct, highly resilient pathways to retrieval. If the verbal trace decays or suffers lexical interference, the distinct visual-spatial trace remains intact to drive retrieval; if the visual trace becomes clouded, the verbal label secures recall. The orthographic word, lacking this redundant dual footprint, falls victim to retrieval failures with far higher frequency.

3.3 Boundary Conditions and Modulation of the Effect

While the Picture Superiority Effect is exceedingly durable, it is not absolute. Rigorous experimental cognitive psychology demands the identification of boundary conditions—specific empirical manipulations where the effect attenuates, collapses, or paradoxically reverses. Investigating these boundary conditions provided Paivio and subsequent researchers with vital tests verifying the underlying mechanics of Dual Coding Theory.

One primary boundary condition involves conceptual and perceptual similarity interference. If a participant is presented with a list of pictorial stimuli derived from a single, tightly constrained, visually homogeneous category—such as twenty distinct line drawings of slightly varying deciduous leaves or twenty different models of keys—the picture superiority effect degrades rapidly. Under these conditions, the nonverbal imagens suffer profound visual proactive and retroactive interference within the spatial visual buffer. The distinctiveness of the nonverbal code collapses, and because the referential verbal labels are frequently identical or non-distinct (every item is categorized simply as “a leaf”), the advantage of the dual code evaporates. Words describing these objects with precise descriptive adjectives (“serrated maple leaf,” “smooth oval beech leaf”) can paradoxically surpass uniform pictures in recall, because the verbal logogens possess higher categorical discriminability than the visually overlapping imagens.

A second major boundary condition is governed by presentation rate constraints and articulatory suppression. Paivio and Csapo (1969) demonstrated that when the presentation rate of stimuli is driven to extreme, tachistoscopic speeds—such as 100 to 200 milliseconds per item in a Rapid Serial Visual Presentation (RSVP) stream—the picture superiority effect vanishes. At these speeds, the human visual system possesses sufficient time to execute representational processing of the picture (instantiating the imagen), but lacks the requisite chronometric window (typically requiring 300 to 500 milliseconds) to generate the covert referential label (the logogen). Stripped of the temporal window required to achieve dual coding, pictures must compete solely as isolated, single-coded nonverbal traces against rapidly decaying visual iconic stores.

Similarly, executing an articulatory suppression task—forcing the participant to continuously chant a meaningless linguistic string, such as “the-the-the-the,” while viewing pictures—selectively occupies the phonological loop of the verbal system. This concurrent verbal load blocks the referential pathway, systematically preventing the participant from verbally labeling the incoming pictorial stimuli. Under severe articulatory suppression, the magnitude of the picture superiority effect diminishes substantially. Conversely, when tasks switch from standard conceptual or free recall orientations to tasks requiring fine-grained orthographic or phonemic discrimination (e.g., asking participants to identify whether the name of an object rhymes with a target or counting the vowels in an item’s label), the perceptual richness of the picture becomes a distracter. The participant is forced to execute an extra referential translation step to retrieve the word from the picture, slowing down performance and systematically eliminating the pictorial advantage.

4. Stephen Kosslyn and the Depictive Representation Hypothesis

4.1 The Analog versus Propositional Distinction

While Allan Paivio laid the structural groundwork by establishing the autonomy of the nonverbal mnemonic code, Stephen Kosslyn plunged into the computational heart of internal representation. At stake was an architectural dilemma: what is the fundamental computational medium of a mental image? In his depictive representation hypothesis, Kosslyn formulated the analog model of visual cognition, positioning it in diametric opposition to the radical propositional formalism championed by theorists such as Zenon Pylyshyn.

To understand Kosslyn’s contribution, one must delineate the exact philosophical and computational gulf separating depictive (analog) representations from propositional (symbolic) representations. A depictive representation is inherently spatial and analog. In a depictive representation, every part of the representational medium corresponds directly to a part of the represented object, and the metric distances between parts within the representation preserve the physical distances between the corresponding parts of the physical object itself. A depictive representation is isomorphic to the distal stimulus: it is a functional, spatial mapping where space in the representational medium represents space in the real world.

Consider a simple visual scene: a green coffee cup resting to the left of a hardbound book on a wooden desk. In a depictive representation—such as a photograph, a physical drawing, or a Kosslyn-style quasi-pictorial mental image—the physical coordinates of the representation mirror the physical geometry of the scene. The pixels or neurocomputational points representing the cup are situated a quantifiable, metric distance from the points representing the book. Crucially, in an analog depictive representation, one cannot depict the cup and the book without implicitly depicting their spatial relationship, their relative scale, and their orientation within the visual plane. The information is integrated continuously, synchronously, and spatially.

Conversely, a propositional representation is entirely discrete, abstract, and non-spatial. Propositions are formal syntactic expressions structured like language or predicate calculus, consisting of discrete symbols, predicates, and arguments governed by explicit combinatorial truth-value rules. The identical scene of the cup and book would be encoded propositionally via abstract semantic predicates:

$$\text{ON}(\text{CUP}, \text{DESK})$$

$$\text{ON}(\text{BOOK}, \text{DESK})$$

$$\text{LEFT-OF}(\text{CUP}, \text{BOOK})$$

In this propositional format, the physical distance between the linguistic symbols “CUP” and “BOOK” on a page or within computer memory registers holds zero metric correspondence to the physical distance between the actual objects. The symbols are arbitrary tokens; they do not preserve surface geometry, scale, or continuous contours. Spatial relationships must be explicitly stated as distinct predicates rather than falling out naturally from the spatial topology of the medium itself. Kosslyn argued that the human brain does not discard perceptual metrics upon initial sensory processing; rather, it maintains the capacity to instantiate authentic, analog, depictive representations that directly preserve spatial metric properties within an internal coordinate space.

4.2 Kosslyn’s Quasi-Pictorial Visual Buffer Model

To operationalize how the human brain generates and manipulates these depictive representations, Kosslyn articulated the Visual Buffer Model. Central to this computational architecture is the concept of the visual buffer—a specialized short-term spatial processing medium functionally analogous to an internal coordinate array or screen. Kosslyn referred to mental images as “quasi-pictorial” to clarify that he was not claiming a literal, physical framed picture rests inside the cranium; rather, he posited an array of neural processing units organized such that the functional relationships among those units directly emulate a spatial coordinate space.

Kosslyn’s visual buffer model operates through the coordination of four discrete computational operations: generation, maintenance, inspection, and transformation:

• Generation: The visual buffer is transient. When an individual imagines an object from memory, the visual representation must be actively constructed. Long-term memory houses two classes of structural information: “deep” propositional descriptions of the object’s categorical parts and spatial relations, alongside skeletal visual frames or perceptual memories. The generation mechanism reads these long-term structural files and plots them into the visual buffer’s coordinate space, assembling the image part by part (e.g., first generating the fuselage of an airplane, then affixing the wings, tail, and engines).
• Maintenance: Once generated within the visual buffer, the depictive image undergoes rapid, spontaneous decay. Maintenance operations expend attentional and cognitive resources to refresh the pattern of activation across the coordinate matrix, counteracting entropy and preserving the image for active inspection.
• Inspection: Once an image is instantiated within the visual buffer, an internal “mind’s eye” (an attentional parsing mechanism) inspects the array to extract spatial and structural properties that were never explicitly encoded as propositionally labeled facts. For example, if asked what shape an open German Shepherd’s ears are, an individual may generate the image in the buffer, direct internal attention to the coordinates corresponding to the ears, and inspect the depictive trace to classify them as pointed triangles.
• Transformation: The visual buffer permits dynamic geometric operations. The system can zoom into the coordinate space (altering its scale), rotate the array across two- or three-dimensional vectors, or translate the pattern across the buffer’s topography.

Crucially, the visual buffer is computationally embedded within a dual architecture: it interacts dynamically with long-term associative memory stores. The system is neither purely propositional nor exclusively depictive; it is a hybrid, multi-stage computational engine where deep propositional descriptions act as recipes for generating depictive surface displays within an analog coordinate array.

4.3 The Functional Equivalence Doctrine

Kosslyn, working alongside contemporaries such as Ronald Finke and Roger Shepard, codified the theoretical framework that anchored mental imagery to standard sensory processing: the Doctrine of Functional Equivalence. The functional equivalence doctrine posits that visual mental imagery and visual physical perception, while initiated via distinct mechanisms (top-down internal generation versus bottom-up sensory transduction), share largely identical underlying cognitive and neural representational pathways. Mental imagery is not fundamentally divorced from physical vision; it is the reactivation of the visual perceptual apparatus in the absence of bottom-up retinal stimulation.

Finke and Kosslyn systematically outlined three core dimensions of functional equivalence:

1. Perceptual Equivalence: Visual mental imagery activates the identical neurocognitive systems utilized during visual object recognition and spatial localization. If a participant imagines a bright red square, the internal representational machinery recruits many of the exact feature-processing, retinotopically mapped visual areas that are engaged when looking at an actual red square in the environment. Perceptual illusions (such as the Ponzo illusion or the tilt aftereffect) can frequently be induced purely through mentally visualized stimuli, demonstrating that internally generated imagery traces interact directly with low-level visual processing mechanisms.
2. Spatial Equivalence: The geometric arrangement of items within a mental image perfectly preserves the metric relationships, spatial layout, and continuous coordinate vectors of physical space. In the visual buffer, distances are continuous; moving an attentional cursor from point A to point B within an imagined array necessitates traversing all the intermediate spatial points, exactly as saccadic or smooth-pursuit eye movements traverse physical visual space.
3. Transformational Equivalence: Dynamic internal transformations of mental images obey the same physical, kinetic, and temporal laws that govern real-world objects. A mental object cannot instantly teleport from an orientation of 0 degrees to an orientation of 180 degrees; it must rotate continuously through the intermediate trajectory, consuming time in direct, linear proportion to the physical distance or angular magnitude of the transformation.

The functional equivalence doctrine provided Kosslyn with clear, quantifiable empirical predictions: if mental imagery is functionally equivalent to physical perception, then subjecting an individual to chronometric spatial tasks within an imagined domain must produce precise reaction time curves that mathematically duplicate tasks performed in the external physical world.

5. Kosslyn’s Spatial Scanning Experiments: Methodologies and Findings

5.1 The Fictional Island Map Paradigm

To unequivocally prove that visual mental images preserve physical metric space, Stephen Kosslyn, along with Thomas Ball and Brian Reiser (1978), engineered one of the most famous experiments in cognitive science: the Fictional Island Map Experiment. The overarching objective of this experiment was to test whether the time required to mentally scan between two points on an imagined map was a direct, linear function of the actual physical metric distance separating those points on the original stimulus.

The experimental protocol was executed with clinical chronometric precision:

• Stimulus Familiarization: Participants were given an arbitrary, fictional map of an island containing precisely seven distinct landmarks scattered across its topography: a hut, a well, a swamp, an inlet, a beach, a tree, and an isolated rock. Crucially, the physical distances between all possible pairs of these seven objects were systematically varied, producing 21 unique pairwise metric distances ranging from short intervals to long traverses across the entire expanse of the island.
• Internal Representation Training: Participants spent extensive sessions studying the physical map, memorizing the exact spatial relationships, shapes, and locations of the landmarks. To ensure absolute representational accuracy, participants were instructed to close their eyes and visualize the map. The experimenter would then present them with a blank sheet of paper and demand that they draw the map with absolute fidelity, placing the seven objects within a metric tolerance of a fraction of a centimeter. This train-to-criterion protocol was repeated until every single participant could reproduce the map perfectly from memory, guaranteeing that a faithful internal representation was securely lodged within long-term cognitive stores.
• The Scanning Protocol: The physical map was removed completely. Participants closed their eyes and were instructed to generate a vivid, clear mental image of the entire island within their visual buffer. The chronometric trial then commenced via auditory instructions:
  1. The experimenter spoke the name of an initial starting landmark (e.g., “Focus on the hut”).
  2. The participant focused their internal “mind’s eye” directly on the imagined location of that starting landmark.
  3. After a brief interval, the experimenter named a target landmark (e.g., “Tree”) or an item that did not exist on the island (e.g., “Tower”).
  4. The participant’s explicit task was to mentally scan across their visual image of the island, starting from the original location and moving in a continuous line toward the target object.
  5. The moment the mental scan landed on the target object, the participant depressed a reaction-time switch with their dominant hand, halting an electronic timer measuring latencies to the nearest millisecond. If they scanned the image and determined the named target was not present, they pressed an alternative “No” switch.

The empirical results were definitive. Kosslyn and his team found a near-perfect, highly significant linear correlation between the physical metric distance separating two landmarks on the original map and the reaction time required to scan between them in the mental image. The correlation coefficients consistently hovered around $r = 0.97$. Moving a mental focus across a long distance on the imagined map required systematically, linearly more time than traversing a short distance. The internal cursor traversed mental space at a remarkably constant, continuous velocity.

Kosslyn argued that this robust linear chronometric relationship could only occur if the mental image was a genuine depictive representation. If the island was stored purely as a non-spatial, propositional semantic network (e.g., a web of nodes stating “HUT—adjacent to—WELL”), scanning time would be determined by the number of intervening semantic links or hierarchical linguistic nodes traversed, rather than the metric physical distance across open space. The fact that moving across open, empty space on the imagined island consumed time in direct proportion to physical distance proved that the representational medium itself preserved metric coordinates.

5.2 Object Scanning Experiments and Structural Feature Inspection

Prior to the fictional island map paradigm, Kosslyn (1973) executed an equally crucial set of chronometric experiments focused on the scanning and inspection of structural features on individual, isolated objects. These experiments demonstrated that metric scanning principles operate not merely across large environmental topographies, but also across the localized structural details of discrete objects.

In a prototypical implementation, Kosslyn presented participants with drawings of recognizable objects possessing prominent, horizontally separated features—such as a motorized speedboat, an airplane, or an animal. The drawing of the speedboat, for instance, featured a motor at the extreme rear (stern), a captain’s steering wheel in the middle, and an anchor resting at the extreme front (bow). Participants thoroughly memorized these drawings until they could generate exact internal representations.

During the chronometric phase, participants were instructed to generate a mental image of the speedboat and direct their mental focus to a specific spatial anchor point—for example, the rear motor. The experimenter would then present a probe question naming a specific structural component: “Does the boat have a steering wheel?” or “Does the boat have an anchor?” The participant was required to verify the presence of the feature by inspecting their mental image. Kosslyn varied the distance between the initial focus point and the target feature. If the participant was anchored at the rear motor, verifying the steering wheel (located at a medium distance) was achieved significantly faster than verifying the anchor (located at the maximum distance at the opposite end of the vessel).

Critically, Kosslyn incorporated strict experimental controls to eliminate confounding factors like the simple linguistic accessibility or semantic salience of the parts. In control conditions, participants were presented with the identical verification questions but were instructed not to use visual imagery, relying instead on any verbal or conceptual knowledge they possessed. In the non-imagery control condition, the linear relationship between physical feature distance and reaction time completely disappeared; verification latencies were instead governed purely by semantic association strength. This verified that the spatial scanning latencies observed during the experimental conditions were uniquely driven by the continuous physical trajectory of an attentional inspection mechanism moving across a depictive array in the visual buffer.

5.3 Addressing Confounds: Demand Characteristics and Experimenter Expectancy

Despite the striking elegance of Kosslyn’s scanning results, the depictive interpretation faced severe theoretical attacks. The most devastating early methodological critique came from Zenon Pylyshyn (1981), who argued that Kosslyn’s scanning latencies were the artifactual product of demand characteristics and tacit knowledge, rather than an unalterable consequence of an analog visual architecture.

Pylyshyn contended that when participants are explicitly instructed to “form an image,” “focus on a point,” and “scan across the map to a target,” they easily deduce the experimenter’s hypothesis. Participants intuitively know how physical vision operates in the real world: they know that looking from one object to another across a vast physical expanse requires more time than looking between adjacent objects. Pylyshyn asserted that human subjects simply utilize their tacit knowledge of real-world physics and optics to unconsciously simulate the appropriate temporal delays, behaving the way they think a person looking at an actual map ought to behave. The linear reaction time curve, Pylyshyn claimed, was not forced by the computational architecture of a depictive visual buffer, but was deliberately, if unconsciously, generated by participants executing a behavioral performance to satisfy perceived experimenter demands.

To definitively falsify the demand characteristic hypothesis, Kosslyn, Ball, and Reiser formulated crucial methodological modifications designed to strip away instructional bias and covert expectations:

1. The Incidental Scanning Paradigm: Kosslyn modified the task so that scanning was transformed into an incidental, uninstructed byproduct of a completely non-spatial cognitive task. Participants were never instructed to “scan” their mental image. Instead, they were told that the experiment tested how well people could remember the colors or surface materials of objects on the map. Participants were directed to look at a specific landmark; then, a target object appeared, and they were asked to determine its presence or verify a subtle attribute as quickly as possible. Even when explicit instructions to “scan” were utterly omitted from the experimental script, the identical linear relationship between metric distance and reaction time persisted.
2. Experimenter Expectancy Controls: In an ingenious manipulation, Kosslyn hired research assistants to collect the chronometric data, intentionally deceiving them regarding the theoretical hypothesis. One group of experimenters was told that mental scanning follows a U-shaped function (where intermediate distances take the longest, but extreme distances cause rapid “jumping”); another group was told that reaction times should be completely flat across all distances due to instant parallel computational search. If the scanning data were driven by subtle, unconscious cues leaked by the experimenters to the participants, the data should have mirrored the false expectations of the testers. The results definitively refuted this possibility: despite the active counter-expectations of the experimenters, the participants’ data continued to generate the exact same robust, linear distance-to-time slopes.

These rigorous controls demonstrated that the scanning effect was an involuntary, automatic property of the underlying cognitive architecture, successfully neutralizing the critique that Kosslyn’s findings were merely experimental artifacts.

6. The Great Imagery Debate: Kosslyn versus Pylyshyn

6.1 Pylyshyn’s Propositional Counter-Arguments

The philosophical and methodological clash between Stephen Kosslyn and Zenon Pylyshyn—spanning the late 1970s through the early 2000s—is famously immortalized in cognitive science as “The Great Imagery Debate.” Pylyshyn led the amodal, computational opposition, waging an uncompromising crusade against the notion that the mind houses anything resembling an internal pictorial code.

Pylyshyn’s counter-offensive rested upon three foundational pillars:

• The Epiphenomenon Claim: Pylyshyn never denied the subjective reality of the *phenomenology* of mental imagery. He conceded that when people close their eyes, they genuinely experience what feels like an internal visual display. However, Pylyshyn argued vigorously that this experiential picture is entirely epiphenomenal. He compared the mental image to the flashing lights on the exterior of a supercomputer or the steam rising from a locomotive: the lights flash and the steam billows, but they are causal non-entities that do zero computational work in driving the calculation or turning the wheels of the train. Pylyshyn maintained that the actual functional cognitive processing occurs entirely at the deep, amodal computational level—via formal symbolic propositions—and that the subjective “picture” is merely a post-hoc decorative experience playing no causal role in human reasoning.
• The Tacit Knowledge Hypothesis: As established in the scanning debates, Pylyshyn argued that cognitive processes are fundamentally cognitively penetrable. A process is cognitively penetrable if it can be systematically altered by changing a participant’s beliefs, knowledge, goals, or expectations. Pylyshyn demonstrated that if you alter a participant’s beliefs about how a task works, imagery latencies shift radically. Because subjects know that real-world physics takes longer to traverse greater distances, they actively stage a simulation using their tacit knowledge. Pylyshyn claimed that Kosslyn was confusing the properties of what was being *represented* (a physical island with spatial geometry) with the properties of the *representational architecture* itself (which he claimed was wholly non-spatial and propositional).
• Computational Parsimony and Formal Elegance: Pylyshyn championed the computational elegance of a unified representational language. Drawing on the foundational work of Alan Newell, Herbert Simon, and Jerry Fodor, he argued that a cognitive architecture utilizing a single, universal format—symbolic propositions organized in semantic networks or predicate calculus—was vastly more parsimonious than positing two completely different computational systems (a propositional system for language and a mysterious, quasi-pictorial analog system for pictures). For Pylyshyn, visual perception and mental imagery alike were fully translatable into formal semantic networks containing labeled nodes and relational pointers.

6.2 Kosslyn’s Rebuttals: Metric Preservation and Spontaneous Effects

Kosslyn responded to Pylyshyn’s propositional critique with an extensive series of chronometric experiments designed to demonstrate that the cognitive architecture of mental imagery exhibits involuntary, structural constraints that are completely cognitively impenetrable—phenomena that occur automatically and run counter to any intuitive tacit knowledge a participant might hold.

One of Kosslyn’s most compelling demonstrations was the Grain Size Effect (or the visual resolution effect). In physical optics, an observer’s ability to resolve fine spatial details is dictated by the visual angle subtended by the object and the physical grain of the sensor (e.g., the density of retinal photoreceptors). Kosslyn hypothesized that if the mental visual buffer functions as an analog coordinate matrix, it must possess a finite spatial resolution—an internal grain size. In these experiments, Kosslyn asked participants to imagine an animal (e.g., a rabbit) at an extremely small relative scale (visualized as a tiny speck far away in the visual field) versus an extremely large relative scale (visualized up close, filling the entire mental field). Participants were then probed to verify fine physical details: “Does the rabbit have whiskers?”

The chronometric results confirmed Kosslyn’s depictive prediction: participants were significantly slower to verify fine structural details when the animal was imagined at a tiny scale than when it was imagined at a large scale. At the small scale, the details blurred against the spatial resolution limit of the visual buffer, forcing participants to actively execute a “mental zoom” transformation to enlarge the image before the inspection mechanism could successfully read out the whiskers. Critically, Kosslyn showed that this latency difference persisted even when participants were never alerted to the scale-manipulation hypothesis and had no intuitive tacit expectations that visual resolution limits would be computationally simulated within their own minds.

A second foundational phenomenon was the Mental Overflow Effect. In physical vision, as an observer walks closer to a large object (e.g., an elephant), the object eventually exceeds the boundaries of the physical field of view, causing its outer edges to spill off the visual field. Kosslyn instructed participants to imagine walking toward various mental objects of vastly different physical dimensions (e.g., a mouse, an automobile, an elephant) and to halt the mental approach the precise moment the imagined object began to “overflow” the borders of their internal visual field. Participants were then instructed to estimate the imagined physical distance separating them from the object. Kosslyn discovered that the calculated mental angle of visual overflow was invariant across objects: regardless of whether the object was a tiny insect or a massive skyscraper, it systematically overflowed the internal mental visual field at an angular metric equivalent to approximately 20 to 30 degrees of visual angle—strikingly congruent with the high-resolution operational zone of the physical human visual field.

6.3 Integration with Paivio’s Structural Dualism

The fierce resolution of the Great Imagery Debate found vital theoretical reinforcement through Allan Paivio’s Dual Coding Theory. While Kosslyn and Pylyshyn waged war over the micro-computational format of imagery (depictive array versus propositional network), Paivio’s macro-structural model provided the overarching architectural blueprint that accommodated and justified Kosslyn’s findings.

Paivio’s Dual Coding Theory offered a powerful antidote to Pylyshyn’s radical propositional reductionism by establishing that human cognition had evolved as a dual-channel architecture. Paivio argued that Pylyshyn’s insistence on a monolithic, amodal, propositional machine was an artificial, logocentric bias born out of early computer science, failing completely to reflect the biological evolution of the human brain. Long before the evolutionary emergence of human syntax and discrete linguistic logogens, biological organisms successfully navigated, remembered, and manipulated complex spatial environments purely through perceptual and motoric memory structures—what Paivio categorized as the nonverbal system of imagens.

Paivio and Kosslyn differed slightly in their experimental lens: Paivio focused predominantly on structural memory capacity, associative networks, free recall, and the differential processing of pictures versus words across long-term retention. Kosslyn, meanwhile, focused on the real-time, working-memory mechanics of the visual buffer—how those nonverbal images are generated, transformed, and inspected in dynamic metric coordinates. Yet their work was fundamentally unified: Kosslyn’s depictive array served as the high-precision computational instantiation of Paivio’s visual imagen.

The ultimate theoretical synthesis reconciled depictive displays with propositional stores. Kosslyn’s mature model conceded that long-term memory houses propositional structural descriptions (which characterize an object’s categorical parts and spatial relations). However, Kosslyn demonstrated that when these propositional descriptions are retrieved to resolve complex spatial problems, the brain does not calculate the solution using propositional logic networks; instead, it compiles those descriptions into a transient, analog depictive array within the visual buffer. Once this depictive display is active, the system reads out new spatial relationships directly through analog spatial scanning. Paivio’s dualism provided the necessary empirical validation that this nonverbal channel was an autonomous, highly structured pillar of the human mind.

7. Comparative Analysis: Paivio’s Imagens versus Kosslyn’s Depictive Representations

7.1 Representational Granularity and Metric Properties

A rigorous comparative analysis of Allan Paivio’s Dual Coding Theory and Stephen Kosslyn’s Depictive Hypothesis reveals profound points of convergence alongside crucial divergences in representational granularity, operational focus, and structural mechanics.

Regarding representational granularity, Paivio’s construct of the imagen is conceptualized primarily as a holistic, multimodal perceptual gestalt. An imagen is a structural memory unit that encapsulates the characteristic sensory signature of an entity or event. Paivio was less concerned with specifying the precise, pixel-by-pixel computational coordinates of an imagen; his primary empirical objective was to demonstrate its functional independence from verbal logogens and its potency in driving associative recall. For Paivio, an imagen represents a chair by capturing its holistic perceptual affordances—its general visual contours, its tactile feel, and its motoric potential for supporting human posture.

Kosslyn, by contrast, demanded a vastly higher degree of micro-computational specificity. In Kosslyn’s framework, a depictive representation within the visual buffer is explicitly metric and topographic. It is not merely a holistic associative node; it is a point-by-point, spatial array that functions like a 2D or 3D coordinate space. In Kosslyn’s visual buffer, every point in the internal representation corresponds to a distinct spatial coordinate, preserving exact geometric distances, spatial intervals, surface orientations, and local visual gradients. Where Paivio treated the nonverbal code as a structural memory unit embedded in an associative web, Kosslyn treated it as an active computational canvas governed by strict Cartesian coordinates.

This divergence carries over directly into their operational mechanics. In Paivio’s architecture, mental operations over imagens are largely governed by associative retrieval and cross-system referential mapping. An imagen activates another imagen via contiguous perceptual associations or triggers a logogen via referential labeling. In Kosslyn’s framework, mental operations are dynamic, continuous transformations executed within an analog coordinate array. While Paivio’s models measure the probability and latency of retrieving an item based on its imagery value, Kosslyn’s models measure the millisecond-by-millisecond chronometric trajectory of an attentional cursor traversing the metric coordinates of the visual buffer.

7.2 Modality Specificity and Structural Organization

A second fundamental point of comparison centers on the sensory scope and modality specificity of the nonverbal representation. Paivio formulated Dual Coding Theory as a comprehensively multimodal architecture. For Paivio, the nonverbal system is by no means restricted to the visual domain. Imagens exist across the entire spectrum of human sensory experience:

• Acoustic Imagens: Preserving the non-linguistic temporal waveforms, timbres, and melodic profiles of environmental sounds (e.g., the roar of thunder or a musical cadence).
• Haptic and Tactile Imagens: Encoding textures, temperatures, and pressures (e.g., the coarse surface of sandpaper or the cold smoothness of polished marble).
• Motor and Kinesthetic Imagens: Capturing the internal proprioceptive programs governing physical movement and body schema (e.g., the motor sequence of swinging a tennis racket).

Paivio organized these multimodal imagens into complex associative networks linked through rich associative bonds, where a visual imagen could instantly trigger a corresponding motor or acoustic imagen without any intervening verbal translation.

Kosslyn, conversely, focused his empirical and theoretical machinery overwhelmingly on the visuospatial domain. The visual buffer model was explicitly formulated to model human visual mental imagery and its intimate neural overlap with the primary visual cortex (Brodmann Area 17/V1) and the dorsal/ventral visual streams. While Kosslyn acknowledged that other modalities (such as auditory imagery) exist, his depictive hypothesis was tailored to address the spatial topography of visual representation: visual field limits, visual resolution, spatial scanning, and two- and three-dimensional geometric rotations. Kosslyn’s structural organization is compositional and spatial—decomposing visual scenes into spatial parts, categorical locations, and coordinate coordinate arrays—whereas Paivio’s organization is primarily associative, relational, and cross-modally integrated.

The primary comparative distinctions between Paivio’s and Kosslyn’s structural models are systematically organized below:

Theoretical Dimension	Allan Paivio (Dual Coding Theory)	Stephen Kosslyn (Depictive Paradigm)
Core Structural Construct	Imagen (Holistic, multimodal nonverbal unit)	Depictive Array (Topographic visual buffer representation)
Representational Medium	Structural memory store; associative networks	Coordinate-based spatial array; quasi-pictorial canvas
Metric Precision	Qualitative, relational, and functional spatial preservation	Strict, quantitative, Cartesian metric coordinate preservation
Modality Scope	Fully multimodal (visual, auditory, haptic, kinesthetic)	Predominantly visuospatial; linked to visual perceptual pathways
Primary Operations	Representational, referential, and associative activation	Generation, maintenance, inspection, and transformation
Primary Empirical Paradigms	Free recall, paired-associate learning, Picture Superiority Effect	Spatial scanning, mental rotation, size zooming, visual field overflow

7.3 Cognitive Economy and Redundancy

From an evolutionary and computational standpoint, both Paivio and Kosslyn had to defend their nonverbal frameworks against charges of cognitive inefficiency. Standard amodal computational models (such as Pylyshyn’s) asserted that storing and manipulating images is computationally wasteful: why would an organism maintain redundant, heavy pictorial displays when everything can be compactly encoded in sparse propositional logic?

Both Paivio and Kosslyn offered compelling rebuttals grounded in principles of cognitive economy and representational redundancy. Paivio demonstrated that redundancy is not a design flaw; it is an evolutionary survival mechanism. By establishing dual, independent memory traces (logogens and imagens), the human cognitive architecture drastically insulates itself against information loss and retrieval failures. If brain damage, aging, or cognitive interference degrades the verbal pathway, the nonverbal perceptual trace remains accessible to guide behavior, and vice versa. Furthermore, Paivio showed that processing efficiency is optimized when tasks split their cognitive load across two independent channels: presenting complex information simultaneously through visual diagrams and spoken narrative expands total working memory throughput, a principle that later became foundational to modern multimedia educational theories.

Kosslyn addressed cognitive economy through computational mechanics. He argued that attempting to solve complex spatial navigation or mechanical reasoning problems using propositional logic alone triggers a catastrophic combinatorial explosion. Consider a simple mechanical task: determining whether two interlocking gears rotating in a machine will collide. To solve this propositionally, a system must write and execute thousands of complex geometric equations calculating points of contact across continuous time. Depictively, however, the brain simply instantiates the gears in the visual buffer and executes an analog rotation, allowing the spatial constraints of the representational medium to reveal the point of collision automatically and instantaneously. The depictive array offloads heavy algorithmic calculations onto the intrinsic spatial geometry of the visual buffer. Thus, both Paivio’s dual-trace redundancy and Kosslyn’s depictive coordinate arrays represent evolutionary adaptations optimizing cognitive processing speed and mnemonic durability.

8. Mental Transformation Paradigms: Rotation, Scaling, and Translation

8.1 Shepard and Metzler’s Mental Rotation as a Precursor

No discussion of Stephen Kosslyn’s spatial chronometry can be complete without acknowledging the empirical catalyst that launched the metric imagery revolution: the groundbreaking mental rotation experiments executed by Roger Shepard and Jacqueline Metzler (1971). Shepard and Metzler provided the initial empirical proof that internal cognitive processes could operate via continuous, analog, dynamic transformations.

Shepard and Metzler presented participants with pairs of line drawings depicting complex, three-dimensional geometric structures constructed from attached blocks. The two objects in each pair were presented at varying degrees of angular disparity, rotated either within the two-dimensional picture plane (e.g., rotated flat on the page) or in three-dimensional depth (rotated into the Z-plane). The participant’s task was to determine, as quickly and accurately as possible, whether the two drawings depicted identical objects in different orientations or whether they were mirror-image enantiomorphs (structurally distinct objects that could never be rotated to match).

The resulting chronometric data altered cognitive psychology forever. Shepard and Metzler discovered a remarkably pure, deterministic linear relationship between the reaction time required to verify identity and the angular disparity separating the two objects. Whether the rotation occurred in 2D picture space or 3D depth space, the reaction time function was identical: as angular disparity increased from 0 degrees up to 180 degrees, reaction times escalated with steady, linear precision. Participants required approximately one millisecond of processing time for every fraction of a degree of physical rotation.

Shepard and Metzler concluded that participants were mentally rotating the internal representations of these objects through a continuous, analog trajectory within a subjective spatial medium. The internal representation did not magically jump from its initial angle to the target angle; it traveled through every intermediate angle of orientation, precisely obeying the laws of physical motion. Shepard and Metzler’s mental chronometry established the foundational methodology that Stephen Kosslyn would subsequently refine and extend to spatial scanning, zooming, and translation.

8.2 Kosslyn’s Size Zooming and Resolution Paradigms

Building directly upon the transformational logic established by Shepard and Metzler, Stephen Kosslyn designed an array of experiments to prove that scaling (zooming) is an analog transformational operation governed by the intrinsic resolution limits of the visual buffer.

In his seminal size zooming studies, Kosslyn (1975) systematically manipulated the subjective visual scale at which participants were instructed to imagine target animals. In one classic condition, participants were instructed to imagine a rabbit standing immediately adjacent to an elephant. In a second condition, participants were instructed to imagine the exact same rabbit standing immediately adjacent to a tiny fly. Because an elephant is physically massive, participants naturally generated an internal mental scene where the elephant consumed the vast majority of the visual buffer’s coordinate capacity; consequently, the rabbit was forced to be visualized as an extremely small, compressed figure off to the side. Conversely, when paired with a tiny fly, the rabbit expanded to dominate the entire mental display, filling the high-resolution center of the visual buffer.

While holding this dual image in mind, participants were probed with high-speed feature verification questions: “Does the rabbit have whiskers?” “Does the rabbit have front paws?” Kosslyn recorded verification reaction times to the millisecond. The results were unambiguous: participants were consistently and significantly faster to verify the structural features of the rabbit when it was imagined next to the fly (at a large subjective scale) than when it was imagined next to the elephant (at a tiny subjective scale). When imagined at a tiny scale, the fine structural features of the rabbit fell below the resolution threshold of the visual buffer. To detect the whiskers, participants had to mentally “zoom in” on the rabbit, an analog transformation that expanded its coordinate footprint within the buffer and consumed measurable processing time.

To eliminate the confound that participants might simply be thinking about elephants or flies conceptually, Kosslyn introduced a critical control: he instructed participants to imagine the rabbit sitting next to an elephant-sized fly, or a fly-sized elephant. The reaction times were entirely dictated by the imagined physical scale of the rabbit within the visual coordinate space, rather than the conceptual identity of the adjacent animal. Kosslyn demonstrated that zooming within mental imagery is an active, continuous transformation: enlarging an imagined object requires dilating its spatial coordinates across the visual buffer until fine-grained details achieve the visual resolution required for perceptual inspection.

8.3 Paivio’s Interpretation of Chronometric Transformation Studies

How did Allan Paivio incorporate the chronometric findings of Shepard, Metzler, and Kosslyn into the broader architecture of Dual Coding Theory? While Paivio fully embraced these findings as irrefutable evidence against amodal propositionalism, he offered a distinct theoretical interpretation grounded in the dynamic, associative properties of the nonverbal system.

Paivio argued that mental rotation and zooming chronometry reflected the continuous, holistic operational dynamics of multimodal imagens. In Paivio’s view, dynamic transformations in the nonverbal system are not merely abstract geometric operations executed over arbitrary coordinate pixels; they are deeply anchored in embodied perceptual and motoric simulations. When an individual mentally rotates a 3D block figure or zooms in on an imagined animal, the nonverbal system is executing dynamic analog transformations that integrate visual imagens with motoric imagens—recapitulating the continuous sensory consequences of physically turning an object with one’s hands or walking physically closer to inspect a living animal.

Furthermore, Paivio highlighted the continuous interplay between dynamic nonverbal transformations and verbal semantic categorization. In paired chronometric tasks where participants were asked to compare the relative physical sizes of two animals (e.g., “Which is larger: a beagle or an otter?”), Paivio demonstrated that reaction times were a function of the symbolic distance effect. When the physical size difference between two animals was massive (e.g., an elephant versus a toaster), participants answered almost instantaneously. When the physical size difference was marginal (e.g., a wolf versus a German shepherd), reaction times slowed dramatically. Paivio demonstrated that this comparative chronometry engaged both systems: the verbal system parsed the semantic prompt, referentially activated the nonverbal imagens of both animals, and the nonverbal system aligned their continuous perceptual traces to read out the size differential. Paivio viewed mental transformation paradigms not merely as tests of visual buffer mechanics, but as sweeping confirmations of the dynamic, continuous, non-propositional nature of the entire nonverbal cognitive substrate.

9. Neurobiological Foundations: Neuroimaging and Neuropsychological Evidence

9.1 Occipital Cortex and Primary Visual Area (V1) Activation

The ultimate empirical vindication of Stephen Kosslyn’s depictive representation hypothesis emerged with the advent of modern functional neuroimaging techniques, including Positron Emission Tomography (PET) and functional Magnetic Resonance Imaging (fMRI), as well as neuromodulation via repetitive Transcranial Magnetic Stimulation (rTMS). These technologies allowed researchers to look directly inside the living human brain to determine whether the visual buffer possessed a verifiable physical neural substrate.

Kosslyn hypothesized that if the visual buffer is an authentic, metric spatial medium, it must map directly onto the retinotopically organized visual cortices of the human brain—specifically Brodmann Areas 17 and 18 (the primary and secondary visual cortices, V1 and V2), located within the occipital lobe. Primary visual area V1 possesses a precise retinotopic organization: physical space on the human retina is mapped in a direct, topographic, metric layout across the physical surface of the visual cortex. Adjacent columns of neurons in Area 17 process adjacent spatial coordinates in the physical visual field. If mental imagery is truly depictive, internally generating a mental image must activate this exact retinotopic neural canvas, even when the participant’s eyes are completely closed in total darkness.

In a series of landmark neuroimaging studies (Kosslyn et al., 1993, 1995, 1999), Kosslyn and his collaborators demonstrated that when participants closed their eyes and generated vivid visual mental images of objects, PET and fMRI scans revealed statistically significant, robust increases in regional cerebral blood flow localized directly within the primary visual cortex (Area 17/V1). Even more profoundly, Kosslyn documented retinotopic mapping during imagery: when participants were instructed to imagine small objects, the neural activation was tightly clustered in the posterior, foveal representation zone of Area 17. When instructed to imagine large objects that spanned their mental visual field, the activation spread outward into the anterior regions of Area 17 that correspond to the peripheral visual fields. The brain physically mapped the metric scale of mental images onto the topographic coordinates of the primary visual cortex.

To silence the lingering critique that Area 17 activation was merely an epiphenomenal byproduct, Kosslyn et al. (1999) executed a definitive study using repetitive Transcranial Magnetic Stimulation (rTMS). Participants were tested on visual perception and visual mental imagery tasks before and immediately after receiving rTMS directed precisely at Area 17. The magnetic pulses delivered by rTMS temporarily disrupted the normal electrical firing of neurons in the primary visual cortex, creating a temporary, safe “virtual lesion.” The results were unequivocal: disrupting Area 17 significantly impaired and slowed down participants’ performance on both the visual perceptual task and the mental imagery task to an identical degree. This proved that the primary visual cortex is functionally and causally necessary for visual mental imagery, delivering the death blow to the radical proposition that imagery operations bypass low-level sensory cortices.

9.2 Hemispheric Lateralization in Dual Coding

While neuroimaging of Area 17 validated Kosslyn’s depictive buffer, neurobiological research across the cerebral hemispheres provided sweeping empirical confirmation of Allan Paivio’s Dual Coding Theory. Paivio’s assumption of functional independence and structural divergence predicted that the verbal and nonverbal systems would exhibit distinct patterns of hemispheric lateralization within the human brain.

Decades of neuropsychological and neuroimaging research have firmly established that the verbal subsystem—responsible for the processing of logogens, syntactic analysis, phonological encoding, and rapid serial parsing—is heavily lateralized to the left cerebral hemisphere (specifically involving the perisylvian language zones, including Broca’s area in the left inferior frontal gyrus and Wernicke’s area in the left superior temporal gyrus). In contrast, the nonverbal subsystem—governing imagens, holistic scene analysis, metric spatial relationships, and continuous visual-perceptual memory—relies extensively on the right cerebral hemisphere (specifically engaging right parietal, occipitotemporal, and ventral stream networks).

Compelling evidence for this dual-system lateralization emerged from studies of split-brain patients who had undergone surgical complete corpus callosotomy to treat intractable epilepsy, as pioneered by Roger Sperry and Michael Gazzaniga. When pictorial stimuli were tachistoscopically flashed exclusively to the left visual field (projecting directly and solely to the isolated right hemisphere), split-brain patients could easily recognize, match, and draw the pictures using their left hand, or select matching nonverbal objects from a hidden array. However, because the right hemisphere was severed from the left-hemisphere language centers, they were completely unable to verbally name the object they were viewing. The nonverbal imagen was fully intact, operational, and capable of driving complex motoric and spatial behavior within the right hemisphere, completely isolated from the verbal logogen system located across the severed commissure.

Conversely, when printed words were flashed to the right visual field (projecting to the left hemisphere), patients could instantly read the words aloud and execute grammatical parsing, but exhibited profound deficits in mentally visualizing or drawing the continuous, metric contours of the objects described. Functional neuroimaging in healthy individuals further confirmed that encoding concrete, high-imagery words spontaneously elicits bilateral neural activation—activating left-hemisphere language networks alongside right-hemisphere temporal and parietal perceptual networks—whereas abstract, low-imagery words recruit an exclusively left-hemisphere, logogenic neural network. This asymmetric neuroarchitecture perfectly aligns with Paivio’s foundational claim that concrete stimuli achieve dual coding via concurrent, bilateral neural recruitment.

9.3 Neuropsychological Dissociations and Pathologies

The neurobiological reality of both depictive visual buffers and dual coding architecture is further validated by striking double dissociations observed in clinical neuropsychology, where localized brain lesions selectively destroy one cognitive system while leaving the other completely unimpaired.

A classic neuropsychological dissociation occurs between visual object agnosia and visual mental imagery deficits (historically documented as Charcot-Wilbrand syndrome). Patients suffering from visual agnosia due to ventral occipitotemporal damage (such as patient D.F.) are profoundly impaired in visually recognizing, naming, or copying physical objects presented in front of them; they cannot look at a physical apple and recognize it as fruit. However, some of these same agnosic patients, when asked to close their eyes and describe or draw an apple from memory, perform flawlessly. Their internal imagery generation and visual buffer mechanics remain completely functional, despite the collapse of bottom-up perceptual transduction. Conversely, rare patients with focal bilateral posterior cerebral lesions lose the capacity to generate mental visual images entirely: they can recognize physical objects instantly with their eyes open, but if asked to imagine the shape of a dog’s ears or visualize the face of their spouse, they report an internal void—a total collapse of the visual buffer.

Perhaps the most extraordinary neuropsychological confirmation of the metric spatial nature of imagery was documented by Edoardo Bisiach and Claudio Luzzatti (1978) in their legendary study of patients suffering from unilateral hemispatial neglect following damage to the right parietal cortex. Bisiach and Luzzatti asked two Milanese neglect patients to close their eyes and imagine that they were standing in the middle of the famous Piazza del Duomo in Milan, facing the front facade of the great cathedral. The patients were instructed to mentally inspect their visual image of the square and describe all the shops, landmarks, and buildings they saw.

Remarkably, the patients accurately described the landmarks situated on the right side of the imagined square, but completely omitted the landmarks on the left side of the square—precisely mirroring the spatial neglect they exhibited when looking at the physical world with their eyes open. Then, Bisiach and Luzzatti introduced an astonishing manipulation: they instructed the patients to mentally reverse their imagined viewpoint, imagining that they were standing on the steps of the cathedral facing outward toward the plaza. When the mental viewpoint was reversed 180 degrees, the patients’ mental visual buffer inverted its spatial coordinates. Now, they successfully described all the landmarks they had previously omitted (which were now on the imagined right), and completely ignored the landmarks they had just described moments before (which were now positioned on the imagined left). This profound finding proved beyond doubt that mental imagery is instantiated within an internal, metric spatial medium that mirrors the neural coordinates of physical perceptual space.

10. Computational Models of Imagery and Dual Representation

10.1 Kosslyn’s Cathode Ray Tube Metaphor and Matrix Models

To transition the depictive representation hypothesis from a conceptual framework into a rigorous, implementable algorithmic theory, Stephen Kosslyn formulated detailed computational simulations. Central to his early computational work was the famous—and frequently misunderstood—Cathode Ray Tube (CRT) metaphor, later formalized as discrete matrix array models.

Kosslyn utilized the CRT metaphor to demystify how an analog, continuous depictive display could be generated and manipulated by a computational system that also relies on discrete long-term memory files. In an old CRT computer terminal, an internal electron gun fires at a phosphor-coated screen, illuminating specific pixels to produce a continuous, 2D graphic image. The graphic image on the screen is depictive, spatial, and analog: it preserves distances, spatial intervals, and contours. Yet, the underlying programmatic instructions driving the electron gun are stored in the computer’s memory as discrete, binary, alphanumeric code. Kosslyn argued that human cognitive architecture operates in a computationally parallel fashion: long-term memory houses “deep representations” (stored structural descriptions and propositional files), while working memory houses the “surface representation” (the active, quasi-pictorial display instantiated across the visual buffer).

In his computer simulation algorithms (such as the IMAGE model, developed with Steven Shwartz), Kosslyn formalized the visual buffer as a two-dimensional coordinate matrix array:

• The visual buffer is modeled as an array of discrete computational cells, analogous to a grid of pixels defined by Cartesian coordinates $(x, y)$.
• Points in the real world are mapped to specific cells in the matrix. Metric distance is computationally instantiated as the number of intervening matrix cells separating two active data points.
• When an image is generated, a long-term memory file—containing polar coordinate vectors, size scaling factors, and categorical relationship markers—acts as an algorithmic recipe, progressively activating contiguous cells across the matrix to plot the shape of the object.
• Scanning operations are executed by moving an internal attentional pointer incrementally from cell to cell across the matrix along continuous trajectories, directly generating the linear distance-to-time slopes observed in human chronometric experiments.

This computational formalization proved that positing a depictive visual buffer did not require magic or homunculi; it was an entirely viable, mathematically rigorous computational architecture capable of executing visual reasoning with high algorithmic efficiency.

10.2 Connectionist and Neural Network Approaches to Dual Coding

Parallel to Kosslyn’s symbolic-matrix simulations, the ascendance of Parallel Distributed Processing (PDP) and connectionist neural network architectures in the 1980s and 1990s provided a natural computational platform for implementing Allan Paivio’s Dual Coding Theory. Connectionist models operate via large networks of interconnected processing units that pass activation through weighted connections, mirroring the architecture of biological neural networks.

Computational cognitive scientists demonstrated that Dual Coding Theory can be elegantly modeled as a dual-channel connectionist architecture consisting of two distinct, interacting modular subnetworks: a linguistic/logogen network and a perceptual/imagen network. In these PDP models:

• The verbal subnetwork is trained on sequential, orthographic, and phonemic patterns. Its internal representations consist of distributed vectors across lexical and syntactic hidden layers, optimized for serial processing, grammatical category clustering, and discrete semantic classifications.
• The nonverbal subnetwork is trained on topographic, auto-associative, or convolutional feature maps. It processes continuous 2D and 3D spatial feature vectors, capturing edges, surfaces, spatial frequencies, and structural gestalts through parallel, distributed activation patterns.
• The critical referential connections are implemented as bi-directional, cross-network associative weight matrices bridging the hidden layers of the verbal network and the nonverbal network. When an input vector representing a picture is fed into the nonverbal network, activation flows through its internal layers and simultaneously spreads across the cross-network referential weights to ignite the corresponding lexical node in the verbal network—computationally simulating automatic referential labeling.

Crucially, these connectionist dual-coding simulations replicated core behavioral phenomena. When the models were subjected to simulated neural degradation (by randomly pruning synaptic weights or adding Gaussian noise to the network layers), they precisely mimicked human neuropsychological pathologies. Lesioning the cross-network referential weights produced symptoms identical to optic aphasia or referential naming deficits, where the network could identify the object visually within its nonverbal layer but failed to pass activation to retrieve the verbal label. Lesioning the nonverbal layers collapsed the model’s capacity to perform spatial pattern completion while leaving linguistic sentence parsing intact. Connectionist modeling demonstrated that Paivio’s structural dualism is an emergent, mathematically robust property of modular neural network architectures.

10.3 Bayesian and Predictive Coding Perspectives

In contemporary twenty-first-century computational neuroscience, the classic depictive and dual coding frameworks have been re-conceptualized and unified through the lens of Bayesian predictive coding and active inference, championed by theorists such as Karl Friston and Andy Clark.

Under the predictive coding framework, the human brain is conceptualized as a hierarchical, multi-layered Bayesian prediction machine. The central objective of the nervous system is to minimize prediction error—the difference between the brain’s internal top-down generative predictions and the bottom-up sensory data arriving from the peripheral receptors. In physical visual perception, top-down generative priors descend through the cortical hierarchy (from prefrontal and temporal cortices down through V4, V2, and V1) to meet ascending retinal sensory inputs. The primary visual cortex (V1) serves as the high-resolution, retinotopic error-comparator where descending predictions collide with ascending sensory prediction errors.

Within this modern computational architecture, Stephen Kosslyn’s mental imagery is formalized as top-down generative perceptual prediction unconstrained by bottom-up sensory input. When an individual engages in visual mental imagery, the high-level cognitive controllers send massive descending generative signals down the cortical hierarchy to instantiate a highly detailed prior within the retinotopically organized layers of V1. Because the eyes are closed, there are no ascending bottom-up sensory prediction errors to correct or override the descending generative signal; the top-down prior actively occupies the retinotopic array, becoming the dominant neural state of the visual buffer. The scanning, rotation, and inspection of a mental image are computational operations that adjust the spatial parameters of this descending generative model.

Simultaneously, Paivio’s Dual Coding Theory fits naturally into this predictive hierarchy. Language (logogens) functions as an ultra-compact, high-level generative prior that can rapidly parameterize and trigger complex, low-level sensory-motor simulations (imagens). The Picture Superiority Effect emerges organically within predictive processing: a visual picture immediately constrains low-level sensory prediction errors and directly updates the brain’s internal generative model across all levels of the hierarchy, forcing both structural and lexical hypotheses to settle rapidly into high-probability Bayesian posterior states. Predictive coding thus provides a unified computational language that reconciles Paivio’s dual semantic channels with Kosslyn’s retinotopic visual buffer.

11. Contemporary Applications: Education, Technology, and Clinical Practice

11.1 Instructional Design and Multimedia Learning Theories

The theoretical discoveries of Allan Paivio and Stephen Kosslyn have exerted a profound, transformative impact on the science of learning, giving rise to modern Instructional Design and Richard Mayer’s widely celebrated Cognitive Theory of Multimedia Learning (CTML). Mayer’s framework is explicitly grounded in the foundational architecture of Paivio’s Dual Coding Theory and working memory channel limitations.

The central premise of Mayer’s multimedia learning theory is that human working memory possesses two separate, channel-limited processing subsystems: an auditory/verbal channel (processing spoken linguistic logogens) and a visual/pictorial channel (processing visual diagrams, animations, and spatial imagens). Traditional education historically suffered from an overwhelming verbal bias, bombarding students with dense, monolithic text. Grounded in Dual Coding Theory, educational researchers formulated core instructional principles:

• The Multimedia Principle: Students achieve substantially deeper conceptual understanding, transfer, and long-term retention when learning from words and pictures combined than from words alone. Dual-channel presentation ensures redundant, additive encoding into both logogenic and imagenic memory stores.
• The Modality Principle: When presenting visual pictorial information (such as a complex mechanical diagram or scientific animation), the accompanying explanatory verbal text should be delivered as spoken narration rather than printed on-screen text. Printed text forces the eyes to split their visual capacity between reading the words and inspecting the diagram, causing severe cognitive overload within the visual buffer. Spoken narration offloads the verbal information onto the acoustic/phonological channel, allowing the visual buffer to dedicate its complete processing capacity to encoding the spatial diagram.
• The Split-Attention Effect: Grounded in Kosslyn’s spatial scanning metrics, educational displays that physically separate explanatory text from the corresponding parts of a diagram force learners to waste precious working memory resources mentally scanning back and forth across empty space to integrate the two inputs. Integrating text directly inside the diagram adjacent to the relevant visual parts eliminates extraneous spatial scanning, optimizing cognitive throughput.

11.2 User Interface (UI) Design and Data Visualization

The digital revolution and the rise of Human-Computer Interaction (HCI) have drawn heavily upon the empirical principles formulated by Paivio and Kosslyn. Contemporary User Interface (UI) design, User Experience (UX) architecture, and data visualization strategies operate directly upon the constraints of the human visual buffer and the Picture Superiority Effect.

In modern operating systems, mobile interfaces, and digital dashboards, the near-universal adoption of iconographic design is an unadulterated application of the Picture Superiority Effect. Crucial functional actions are represented by pictograms (e.g., a magnifying glass for search, a trash can for deletion, an envelope for messaging) rather than purely orthographic text menus. Visual icons trigger instant representational processing within the visual buffer, eliciting automatic referential labeling far faster than the serial reading and lexical parsing required for printed text. This minimizes visual search latencies and dramatically reduces user cognitive fatigue.

Furthermore, Stephen Kosslyn himself transitioned extensively into applying his depictive imagery paradigms to graphic communication and data visualization, authoring authoritative treatises on effective graph design (e.g., Elements of Graph Design). Kosslyn demonstrated that effective data dashboards must respect the physiological and spatial resolution limits of the human visual buffer:

• Spatial Metric Compatibility: Data variables that represent quantitative, continuous magnitudes (such as time, speed, or revenue) must be mapped onto continuous spatial metrics (such as bar lengths, spatial positions, or continuous scatter plots) rather than discrete categorical colors or arbitrary labels. This allows the human visual buffer to exploit its native analog coordinate mechanics to read out quantitative trends instantaneously without requiring propositional translation.
• Visual Clutter and Resolution Limits: Placing too many data points or complex visual embellishments (“chartjunk”) within a compressed visual area exceeds the grain-size resolution of the visual buffer, inducing spatial interference and degrading the user’s capacity to extract relational patterns. Interfaces designed around Kosslyn’s visual principles optimize visual hierarchy, ensure distinct spatial boundaries, and align layout grids to facilitate effortless internal mental scanning.

11.3 Clinical Interventions and Psychopathology

Beyond education and technology, the nonverbal representational framework has fundamentally revolutionized clinical psychology and psychiatry, particularly in the understanding and treatment of trauma, anxiety disorders, and depression.

In Post-Traumatic Stress Disorder (PTSD), the central pathology is not a deficit in propositional memory; trauma victims can typically state the historical facts of their trauma linguistically without difficulty. The devastating core of PTSD resides in the uncontrollable, involuntary intrusion of terrifyingly vivid, emotionally overwhelming nonverbal imagens—intrusive visual flashbacks, sensory fragments, and somatic terrors. Paivio’s Dual Coding Theory explains why traditional purely verbal talk therapies frequently struggle to resolve trauma: traumatic memories are often encoded under conditions of extreme autonomic arousal where the linguistic logogen system partially decouples from the nonverbal system. The trauma is locked within raw, unintegrated sensory-motor imagens that possess privileged, direct access to the amygdala and autonomic nervous system, bypassing the rational, propositional modulation of the prefrontal cortex.

This realization gave birth to powerful imagery-focused clinical interventions, most notably Imagery Rescripting (ImR) and Eye Movement Desensitization and Reprocessing (EMDR):

• Imagery Rescripting: Rather than merely talking about a traumatic memory in the abstract, the patient is guided to actively generate the terrifying depictive image within their visual buffer. Under the safety of the therapeutic alliance, the patient consciously intervenes within the mental simulation, dynamically transforming and “rescripting” the spatial and narrative trajectory of the image (e.g., imagining their adult self stepping into the childhood scene to halt an abuser). By actively manipulating the nonverbal imagen within the visual buffer, the patient alters its emotional valence and lays down a new, non-threatening memory trace that permanently neutralizes the traumatic image.
• Cognitive Behavioral Interventions: In depression and generalized anxiety, patients frequently suffer from automatic, catastrophic mental simulations of future failure. Therapists utilize dual-modality cognitive training to break these cycles, training patients to actively construct vivid, competing nonverbal images of mastery, resilience, and positive outcomes, directly exploiting the additive mnemonic advantages of Paivio’s dual coding architecture.
• Neurorehabilitation: In stroke rehabilitation and traumatic brain injury recovery, motor imagery therapy—where patients mentally simulate physical movements using kinesthetic and visual imagens—is routinely deployed to stimulate motor cortex plasticity and accelerate physical limb recovery, demonstrating the profound functional power of mental pictorial representations across human health.

12. Theoretical Synthesis: Toward an Integrated Model of Visuospatial Cognition

12.1 Convergence of Paivio’s Architecture and Kosslyn’s Functional Paradigms

More than half a century after Allan Paivio formulated Dual Coding Theory and Stephen Kosslyn initiated his chronometric spatial paradigms, the cognitive sciences have arrived at a mature, unified consensus that synthesizes the greatest insights of both pioneers. The false dichotomies of the early cognitive revolution—which pitted words against pictures, and amodal logic machines against sensory simulation—have been replaced by an integrated, multi-level architecture of human visuospatial cognition.

The synthesis unites Paivio’s macro-structural memory systems with Kosslyn’s micro-functional visual buffer mechanics into a comprehensive taxonomic framework:

• Long-Term Memory Substrate (Paivio’s Domain): Human long-term memory houses two massive, interconnected repositories: a discrete, categorical verbal system composed of logogens, and a multimodal, sensory-motor perceptual system composed of imagens. These repositories are linked through rich referential bridges that enable seamless cross-modal translation and produce additive, resilient dual memory traces (underlying the Picture Superiority Effect).
• Working Memory Simulation Workspace (Kosslyn’s Domain): When the cognitive system requires high-precision spatial problem solving, navigation, or visual inspection, it does not rely on static semantic associations. It activates the visual buffer—a retinotopically organized neural workspace physically instantiated across the primary and extrastriate visual cortices. The system retrieves deep structural descriptions from long-term memory and compiles them into an analog, depictive surface display.
• Dynamic Cross-Validation: Kosslyn’s chronometric scanning, rotation, and zooming paradigms provide the functional operational proof verifying the continuous, analog nature of Paivio’s nonverbal code. Conversely, Paivio’s dual coding principles explain how linguistic labels can rapidly index, retrieve, organize, and modulate the high-resolution depictive displays instantiated within Kosslyn’s visual workspace.

12.2 Current Frontiers in Visuospatial Representation Research

Today, the research paradigms initiated by Paivio and Kosslyn are accelerating into breathtaking new frontiers, driven by advanced computational neuroimaging, cognitive genetics, and individual difference psychology.

A profound modern frontier is the scientific discovery and mapping of aphantasia and hyperphantasia, conceptualized and formalized by neurologist Adam Zeman (2015). Aphantasia represents a fascinating cognitive variation wherein otherwise healthy individuals possess a complete inability to voluntarily generate visual mental images: when asked to visualize a sunset or an apple, their internal visual buffer remains entirely dark, despite possessing intact visual perception and normal semantic knowledge. Conversely, individuals with hyperphantasia experience internal visual imagery of photographic, cinematic vividness. Investigating these conditions has provided unprecedented tests of dual coding and depictive theory: aphantasics exhibit marked reductions in the Picture Superiority Effect when tasks rely on internal visual visualization, yet they compensate brilliantly by deploying sophisticated, non-visual propositional and spatial-coordinate strategies, verifying that the human mind can navigate the world through diverse representational trajectories.

Simultaneously, the rise of Embodied and Grounded Cognition—championed by cognitive scientists like Lawrence Barsalou—has expanded Paivio’s foundational insights far beyond classical laboratory paradigms. Grounded cognition asserts that all human concepts, even the most abstract philosophical notions, are ultimately grounded in sensory-motor simulations, modal perceptual traces, and bodily interactions with the physical environment. Paivio’s nonverbal system is no longer viewed as an isolated module, but as the primary evolutionary foundation of human conceptualization.

Finally, cutting-edge functional neuroimaging utilizing high-resolution fMRI decoding and neural reconstruction algorithms (such as generative deep neural networks and diffusion models) has achieved what was once considered science fiction: actual visual “mind-reading.” Contemporary computational neuroscientists (e.g., Takagi & Nishimoto, 2023) can record blood-oxygen-level-dependent (BOLD) signals from a participant’s visual cortex while they either look at a photograph or close their eyes and mentally imagine a scene, feeding those neural activation patterns into generative AI decoders to visually reconstruct the exact image being held within the participant’s mind. The reconstructed images confirm beyond all doubt that Kosslyn’s depictive visual buffer is physically, spatially instantiated across the human visual cortex.

12.3 Final Academic Assessment of the Dual Representation Paradigm

The historical journey of mental imagery from Watsonian behaviorist exile to the forefront of modern computational neuroscience represents one of the greatest triumphs in the history of psychology. Allan Paivio’s Dual Coding Theory and Stephen Kosslyn’s Depictive Hypothesis dismantled the dogmatic amodal computationalism of the early cognitive revolution, permanently establishing that the human mind does not live by linguistic syntax alone.

Paivio revealed the profound power of the nonverbal code, proving that our memory, learning, and cognition are immeasurably enriched by the redundant, additive architecture of visual and verbal representations. His documentation of the Picture Superiority Effect delivered an enduring empirical standard that continues to shape instructional design, media communication, and cognitive rehabilitation worldwide. Kosslyn, through his audacious mental chronometry and rigorous neurobiological investigations, mapped the topography of the internal visual canvas, demonstrating that visual mental images are depictive, analog representations governed by the spatial metric geometry of the brain’s retinotopic perceptual cortices.

Together, Paivio and Kosslyn fundamentally re-humanized cognitive science. They established that human beings are neither passive stimulus-response automata nor dry, disembodied amodal symbol manipulators. We are deeply sensory, imaginative, and embodied creatures, capable of traversing inner landscapes, inspecting internal pictures, and constructing mental worlds with all the metric richness, continuous beauty, and vivid fidelity of the physical universe.

References

• Aristotle. (1907). De Anima (R. D. Hicks, Trans.). Cambridge University Press.
• Barsalou, L. W. (1999). Perceptual symbol systems. Behavioral and Brain Sciences, 22(4), 577-660. https://doi.org/10.1017/s0140525x99002149
• Bisiach, E., & Luzzatti, C. (1978). Unilateral neglect of representational space. Cortex, 14(1), 129-133. https://doi.org/10.1016/S0010-9452(78)80016-1
• Clark, A. (2013). Whatever next? Predictive brains, situated agents, and the future of cognitive science. Behavioral and Brain Sciences, 36(3), 181-204. https://doi.org/10.1017/s0140525x12000477
• Finke, R. A. (1989). Principles of Mental Imagery. MIT Press.
• Fodor, J. A. (1975). The Language of Thought. Harvard University Press.
• Friston, K. (2010). The free-energy principle: A unified brain theory? Nature Reviews Neuroscience, 11(2), 127-138. https://doi.org/10.1038/nrn2787
• Hume, D. (1739). A Treatise of Human Nature. John Noon.
• Kosslyn, S. M. (1973). Scanning visual images: Some structural implications. Perception & Psychophysics, 14(1), 90-94. https://doi.org/10.3758/BF03198621
• Kosslyn, S. M. (1975). Information representation in visual images. Cognitive Psychology, 7(3), 341-370. https://doi.org/10.1016/0010-0285(75)90015-8
• Kosslyn, S. M. (1980). Image and Mind. Harvard University Press.
• Kosslyn, S. M. (1994). Image and Brain: The Resolution of the Imagery Debate. MIT Press.
• Kosslyn, S. M., Alpert, N. M., Thompson, W. L., Maljkovic, V., Weise, S. B., Chabris, C. F., Hamilton, S. E., Rauch, S. L., & Buonanno, F. S. (1993). Visual mental imagery activates topographically organized visual cortex: PET investigations. Journal of Cognitive Neuroscience, 5(3), 263-287. https://doi.org/10.1162/jocn.1993.5.3.263
• Kosslyn, S. M., Ball, T. M., & Reiser, B. J. (1978). Visual images preserve metric spatial information: Evidence from studies of image scanning. Journal of Experimental Psychology: Human Perception and Performance, 4(1), 47-60. https://doi.org/10.1037/0096-1523.4.1.47
• Kosslyn, S. M., Pascual-Leone, A., Felician, O., Camposano, S., Keenan, J. P., Thompson, W. L., Ganis, G., Sukel, K. E., & Alpert, N. M. (1999). The role of Area 17 in visual imagery: Convergent evidence from PET and rTMS. Science, 284(5411), 167-170. https://doi.org/10.1126/science.284.5411.167
• Kosslyn, S. M., Thompson, W. L., Kim, I. J., & Alpert, N. M. (1995). Topographical representations of mental images in primary visual cortex. Nature, 378(6556), 496-498. https://doi.org/10.1038/378496a0
• Mayer, R. E. (2009). Multimedia Learning (2nd ed.). Cambridge University Press. https://doi.org/10.1017/CBO9780511816819
• Miller, G. A. (1956). The magical number seven, plus or minus two: Some limits on our capacity for processing information. Psychological Review, 63(2), 81-97. https://doi.org/10.1037/h0043158
• Paivio, A. (1971). Imagery and Verbal Processes. Holt, Rinehart and Winston.
• Paivio, A. (1986). Mental Representations: A Dual Coding Approach. Oxford University Press.
• Paivio, A. (1991). Dual coding theory: Retrospect and current status. Canadian Journal of Psychology, 45(3), 255-287. https://doi.org/10.1037/h0084295
• Paivio, A., & Csapo, K. (1969). Concrete image and verbal memory codes. Journal of Experimental Psychology, 80(2), 279-285. https://doi.org/10.1037/h0027273
• Paivio, A., Rogers, T. B., & Smythe, P. C. (1968). Why are pictures easier to recall than words? Psychonomic Science, 11(4), 137-138. https://doi.org/10.3758/BF03331011
• Pylyshyn, Z. W. (1973). What the mind’s eye tells the mind’s brain: A critique of mental imagery. Psychological Bulletin, 80(1), 1-24. https://doi.org/10.1037/h0034650
• Pylyshyn, Z. W. (1981). The imagery debate: Analogue media versus tacit knowledge. Psychological Review, 88(1), 16-45. https://doi.org/10.1037/0033-295X.88.1.16
• Pylyshyn, Z. W. (2002). Mental imagery: In search of a theory. Behavioral and Brain Sciences, 25(2), 157-182. https://doi.org/10.1017/s0140525x02000043
• Shepard, R. N., & Metzler, J. (1971). Mental rotation of three-dimensional objects. Science, 171(3972), 701-703. https://doi.org/10.1126/science.171.3972.701
• Standing, L. (1973). Learning 10,000 pictures. Quarterly Journal of Experimental Psychology, 25(2), 207-222. https://doi.org/10.1080/14640747308400340
• Takagi, Y., & Nishimoto, S. (2023). High-resolution image reconstruction with latent diffusion models from human brain activity. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 14453-14463. https://doi.org/10.1109/CVPR52729.2023.01389
• Watson, J. B. (1913). Psychology as the behaviorist views it. Psychological Review, 20(2), 158-177. https://doi.org/10.1037/h0074420
• Zeman, A., Dewar, M., & Della Sala, S. (2015). Lives without imagery – Congenital aphantasia. Cortex, 73, 378-380. https://doi.org/10.1016/j.cortex.2015.05.019