Neural Networks 17 (2004) 471–510 www.elsevier.com/locate/neunet How laminar frontal cortex and basal ganglia circuits interact to control planned and reactive saccades Joshua W. Browna, Daniel Bullockb,c,*, Stephen Grossberg b a Department of Psychology, Washington University, Campus Box 1125, St. Louis, MO 63130-4899, USA Department of Cognitive and Neural Systems, Boston University, 677 Beacon Street, Boston, MA 02215, USA c Center for Adaptive Systems, Boston University, 677 Beacon Street, Boston, MA 02215, USA Received 30 May 2003; accepted 14 August 2003 Abstract How does the brain learn to balance between reactive and planned behaviors? The basal ganglia (BG) and frontal cortex together allow animals to learn planned behaviors that acquire rewards when prepotent reactive behaviors are insufficient. This paper proposes a new model, called TELOS, to explain how laminar circuitry of the frontal cortex, exemplified by the frontal eye fields, interacts with the BG, thalamus, superior colliculus, and inferotemporal and parietal cortices to learn and perform reactive and planned eye movements. The model is formulated as fourteen computational hypotheses. These specify how strategy priming and action planning (in cortical layers III, Va and VI) are dissociated from movement execution (in layer Vb), how the BG help to choose among and gate competing plans, and how a visual stimulus may serve either as a movement target or as a discriminative cue to move elsewhere. The direct, indirect and hyperdirect pathways through the BG are shown to enable complex gating functions, including deferred execution of selected plans, and switching among alternative sensory-motor mappings. Notably, the model can learn and gate the use of a What-to-Where transformation that enables spatially invariant object representations to selectively excite spatially coded movement plans. Model simulations show how dopaminergic reward and non-reward signals guide monkeys to learn and perform saccadic eye movements in the fixation, single saccade, overlap, gap, and delay (memory-guided) saccade tasks. Model cell activation dynamics quantitatively simulate seventeen established types of dynamics exhibited by corresponding real cells during performance of these tasks. q 2003 Elsevier Ltd. All rights reserved. Keywords: Basal ganglia; Frontal cortex; Cortical layer; Saccade; Gating; Dopamine; Reinforcement learning; Action selection; Planning; Parkinson’s disease 1. Introduction This article proposes detailed mechanistic solutions to several key problems in sensory-motor control: How does the brain learn to balance between reactive and planned movements? How do recognition and action representations in the brain work together to launch movements toward valued goal objects? How does the brain learn and recall the myriad movement plans it needs to switch among different tasks, when each plan may be sensitive to different combinations of scenic cues and timing constraints? The article treats these problems by modeling the saccadic, or ballistic, eye movement system. Solving these * Corresponding authors. Tel.: þ1-617-353-7858/7857; fax: þ 1-617353-7755. E-mail addresses: [email protected] (J.W. Brown); [email protected] cns.bu.edu (D. Bullock); [email protected] (S. Grossberg). 0893-6080/$ - see front matter q 2003 Elsevier Ltd. All rights reserved. doi:10.1016/j.neunet.2003.08.006 problems requires interactions among multiple brain regions, including inferotemporal, parietal, and prefrontal cortex (PFC); basal ganglia (BG); amygdala; cerebellum; and superior colliculus (SC). In amphibians and all land vertebrates, the BG interact with a laminar structure, the optic tectum (OT), or its homolog, the SC, to control orienting actions and, in some species, prey-catching actions (Butler & Hodos, 1996; Marin, Smeets, & Gonzalez, 1998). The mammalian BG also interact with distinct areas of frontal cortex—also laminar structures—to control orienting, cognitive, and manipulative behaviors (Hikosaka & Wurtz, 1989; Passingham, 1993; Strick, Dum, & Picard, 1995). Lesions of the BG uniquely cause devastating disorders of the voluntary movement system, e.g. Parkinsonian akinesia, Huntington’s chorea, and ballism (Albin, Young, & Penney, 1989). Such observations suggest a tight link between volitional movement and BG interactions with laminar 472 J.W. Brown et al. / Neural Networks 17 (2004) 471–510 action control structures, which provide a natural basis for differentiating between plan activation and plan execution. Whereas laminar organization has been neglected in most models of BG function, the present model explains how BG interactions with laminar target structures satisfy staging and learning requirements of voluntary behavior. The model simulates learning and performance of the saccadic tasks that are summarized in Fig. 1 (cf. Hikosaka, Sakamoto, & Usui, 1989, p. 781). It is also used to simulate performance in two related tasks. Recording during such tasks has produced a wealth of electrophysiological data that, in concert with anatomical studies, serve as hard constraints on model development. As a set, these tasks challenge an animal’s ability to plan, withhold, and generate goal-directed movements in a way that satisfies instrumental reward contingencies. Simulations (summarized in Figs. 9 –11) show that the model shown in Fig. 2 can learn and perform all the tasks, and can regenerate seventeen qualitatively distinct types of task-related activation dynamics exhibited by cells in the SC, BG, thalamus, and oculomotor areas of frontal and parietal cortex. A key adaptive challenge is to balance reactive and planned movement (Grossberg & Kuperstein, 1986; Hallett, Fig. 1. Oculomotor tasks of Hikosaka et al. (1989). Black bars indicate intervals of visual stimulus presentations and the trace labeled E gives the horizontal component of eye position (line of gaze). In the fixation task, the subject must maintain gaze on the fixation point, F, despite a brief display of a distracter target, T, at a different locus. In the saccade task, the subject must make a pro-saccade from the fixation point to the target, which appears at a different locus, just as the fixation point shuts off. In the overlap task (similar to a GO/NOGO task), the target and the fixation point are displayed in overlapping intervals. A pro-saccade to the target is rewarded only if generated after the fixation point shuts off. The gap task imposes a delay between the offset of the fixation point and the onset of the target. The gap task target appears at a consistent location across trials, and the subject learns to make an anticipatory pro-saccade to the target location during the gap between fixation light offset and target onset. The delay task requires the subject to remember the location of a briefly-flashed target and later foveate it. [Adapted with permission from Hikosaka et al., 1989, p. 781.] The model in Fig. 2 learned and performed all these tasks. 1978). Rapid reactive movements are needed to ensure survival in response to unexpected dangers. Planned movements often take longer to elaborate. How does the brain prevent reactive movements from being triggered in situations where a more slowly occurring planned movement would be more adaptive? Movement gates can prevent the reactive movement from being launched until the planned movement can effectively compete with it. Then a winning movement command can open its gate and launch its movement. The proposed model shows how a cooperative set of physiological and circuit properties: prevent a reactive movement command from opening the gate before a planned movement command is ready to open it; allow the reactive and planned commands to compete for dominance; yet also allow a reactive movement command to open the gate when no planned movement command is being formed. Conditional movements towards valued goal objects cannot be made until the goal objects are recognized and movement directions specified. Formidable memory storage problems would ensue if the brain had to learn separate object recognition codes for every retinotopic position and size of an object. To achieve efficient object recognition, the What cortical processing stream builds object representations that are ‘positionally invariant’, i.e. independent of the retinotopic position or size of the object (Bar et al., 2001; Sigala & Logothetis, 2002; Tanaka, Saito, Fukada, & Moriya, 1991). Given that recognition codes are independent of position, how does the brain compute how to move to the position of an object after it is recognized? After eliminating the link between an object’s identity and position for purposes of object recognition, the brain needs to re-establish this link for purposes of movement. The Where cortical processing stream elaborates the object positions and directions needed to compute motor commands. The model proposes how interactions across the What and Where processing streams overcome their complementary informational deficiencies to generate movements towards recognized objects. It is not enough to recognize and move towards an object. An animal needs to know when to move towards or away from an object and when not to do so, depending on reward contingencies. Decision criteria include such stimulus properties as color, size, shape, motion and the state of the body, taken individually or in combination. In addition, when confronted with the same scene, an animal may act with respect to different objects depending on its changing needs, such as food if hungry or water if thirsty. The model explains how the brain learns and remembers many plans that involve different sets of discriminative and scheduling constraints, and how it switches among them as needed. According to the proposed model (Fig. 2), these functional problems find a mechanistic solution in BG interactions with the SC and frontal cortex. Reward-related dopaminergic signals modulate learning in the BGs striatum and the frontal cortex (Gaspar, Bloch, & Le Moine, 1995; Schultz, 1998). The trained BG system allows or prevents movements, according to their appropriateness (Bullock & Grossberg, J.W. Brown et al. / Neural Networks 17 (2004) 471–510 473 Fig. 2. A laminar model of BG interactions with the FEF and the SC. Separate gray-shaded blocks highlight the major anatomical regions whose roles in planned and reactive saccade generation are treated in the model. Major anatomical abbreviations, e.g. PPC, are defined in Table 1. Excitatory links are shown as arrowheads, inhibitory as ballheads, in this and all subsequent figures. Filled semi-circles terminate cortico-striatal and cortico-cortical pathways modeled as subject to learning, which is modulated by reinforcement-related dopaminergic signals (dashed arrows). In the FEF block, Roman numerals I– VI label cortical layers; Va and Vb, respectively, are superficial and deep layer V. Further symbols are variable names in the mathematical model. Subscripts xy index retinotopic coordinates, whereas subscript i denotes an FEF zone gated by an associated BG channel (details in Fig. 5). All variables for FEF activities use the ðpÞ ðdÞ symbol F. Processed visual inputs Ixy and Ixyj emerging from visual areas including V4 and posterior IT feed into the model FEF input cells and affect ðIÞ activations Fxyi : Fibers carrying such inputs are predicted to synapse on cells in layer III (and possibly layers II and IV). Visual input also excites the PPC, Pxy ; and anterior IT, Tj : A PFC motivational signal I ðMÞ arouses PFC working memory activity Ci ; which in turn provides a top-down arousal signal to model FEF ðIÞ ðPÞ excite FEF planning cells Fxyi ; which are predicted to reside in layers III/Va (and possibly layer VI cells, with activities FiðGÞ : The FEF input cell activities Fxyi layer II). Distinct plan layer activities represent alternative potential motor responses to input signals, e.g. a saccade to an eccentric target or to a central fixation point. FEF layer VI activities FiðGÞ excite the groups/categories of plans associated with GCZs i and associated thalamic zones k. The BG decide which plan to ðOÞ execute and send a disinhibitory gating signal that allows thalamic activation Vk ; which excites FEF layer Vb output cell activities Fxyi to execute the plan. The model distinguishes (Kemel et al., 1988) a thalamus-controlling BG pathway, whose variables are symbolized by B, and a colliculus-controlling pathway, ðGPiÞ whose variables are symbolized by G. Thus, the striatal direct (SD) pathway activities BðSDÞ and GðSDÞ and SNr xy ; respectively, inhibit GPi activities Bk k ðSNrÞ activities Gxy ; which, respectively, inhibit thalamic activities Vk and collicular activities Sxy : As detailed in Fig. 3, if the FEF saccade plan matches the most salient sensory input to the PPC, then the BG disinhibit the SC to open the gate and generate the saccade. However, if there is conflict between the bottom-up input to PPC and the top-down planned saccade from FEF, then the BG-SC gate is held shut by feedforward striatal inhibition (note BG blocks labeled GABA) ðOÞ until the cortical competition resolves. When a plan is chosen, the resulting saccade-related FEF output signal Fxyi activates PPC, the STN and the SC ðSxy Þ: ðXÞ The SC excites FEF postsaccadic cell activities Fxy ; which delete the executed FEF plan activity. The STN activation helps prevent premature interruption of plan execution by a subsequent plan or by stimuli engendered by the early part of movement. 1991; Crosson, 1985; Hikosaka & Wurtz, 1983; Mink, 1996; Mink & Thach, 1993; Redgrave, Prescott, & Gurney, 1999). BG outputs provide GABA-ergic inhibitory gating of their target structures. In the primate saccadic circuit, cells in the substantia nigra pars reticulata (SNr) tonically inhibit the SC but pause briefly to allow the SC to generate a saccade (Hikosaka & Wurtz, 1983, 1989). Lesions in this system can release a ‘visual grasp reflex’ (Guitton, Buchtel, & Douglas, 1985), i.e. impulsive orienting to any visually salient object. Ancient vertebrate genera, such as frogs, already had a welldeveloped BG system (Marin et al., 1998). Though lacking a precise equivalent of the primate saccadic circuit, frogs can selectively orient while ignoring distracters, but lesions of the BG projection to the OT (SC homolog) impair a frog’s ability to orient selectively (Ewert, Schurg-Pfeiffer, & Schwippert, 1996). 474 J.W. Brown et al. / Neural Networks 17 (2004) 471–510 Thus BG gates help create a difference between physical and motivational salience. Such gating enables an actor to acquire reward for foveating a physically weak stimulus (e.g. a dim and motionless predator) while ignoring a physically strong stimulus. In most visual scenes, many targets compete for foveation. If the saccadic gate is opened before competition among stimuli resolves, the system may foveate the most contrastive target, or may attempt to foveate multiple targets simultaneously by averaging ambiguous SC activity (Lee, Rohrer & Sparks, 1988; Ottes, Van Gisbergen, & Eggermont, 1984). In the proposed model, feedforward striatal inhibitory interneurons (Gernert, Hamann, Bennay, Loscher, & Richter, 2000; Koos & Tepper, 1999; Wilson et al., 1989) keep the BG gate shut until competitive dynamics in posterior parietal cortex (PPC) and the frontal eye field (FEF) have a chance to select a unique saccade goal. The FEF and SC are individually sufficient to generate saccades (Deng et al., 1986; Schiller, True, & Conway, 1980). Yet in the normal animal, the SC is an important common pathway for saccade generation, and focal SC lesions result in transient impairment of all saccade types (Schiller et al., 1980). Imaging studies (Sweeney et al., 1996) have shown that the frontal cortex is more strongly activated in more difficult oculomotor tasks, e.g. those requiring memory-guided saccades or anti-saccades. Such tasks engage elements of the frontal oculomotor system, including the PFC, FEF, and supplementary eye fields (SEF, an oculomotor area in dorsomedial frontal cortex, DMFC). Lesions of these frontal areas suggest that they incorporate distinct, modular contributions to oculomotor planning and control. The frontal oculomotor areas add the ability to use: head-centered or other non-motor-error coordinates (Schiller, 1998; Schlag & Schlag-Rey, 1987; Schlag-Rey, Amador, Sanchez, & Schlag, 1997); working memory (Goldman-Rakic, 1987, 1995; Pierrot-Deseilligny, Israel, Berthoz, Rivaud, & Gaymard, 1993); conjunctions of features (Bichot & Schall, 1999); and internal sequencing (Sommer & Tehovnik, 1999). Taken together, these data suggest a hierarchy. Visual inputs to the SC dominate reactive movements by default, but plans within the frontal cortex can assume control of the SC when simple reactive eye movements are insufficient (Fig. 3). These principles, realized here as mechanisms in a saccadic control model, should also apply to adaptive control of manipulative and cognitive behaviors. The model and results were briefly reported in Brown, Bullock, & Grossberg (2000). 2. Methods The model realizes fourteen major computational hypotheses. These are presented verbally and diagrammatically to frame the subsequent mathematical specification. Table 1 lists abbreviations to be used in reference to neuroanatomical structures. Many of the assumptions are shared with prior verbal formulations and computational models (e.g. Albin et al., 1989; Berns & Sejnowski, 1998; Brown, Bullock, & Grossberg, 1999; Bullock & Grossberg, 1991; Contreras-Vidal & Stelmach, 1995; Crosson, 1985; Dominey, Arbib, & Joseph, 1995; Graybiel, 1998; Hikosaka & Wurtz, 1983, 1989; Houk & Beiser, 1995; Mink, 1996; Mink & Thach, 1993; Redgrave et al., 1999; Schultz, 1998; Suri, Albani, & Glattfelder, 1997; Wichmann & DeLong, 1996; Wickens, 1997), although in virtually all cases the present mathematical implementation of a postulate has predictive implications that distinguish it from postulates in prior models. It is noted below where these predictive implications depart most significantly from prior models. The new model is called TELOS, which is from the ancient Greek telos for goal, end, or completion of a plan, but is also an acronym for TElencephalic Laminar Objective Selector. The BG and cerebral cortex together make up the telencephalon, and the BG take inputs from, and help select the outputs of, laminar oculomotor structures, notably the SC and the FEF in the frontal cortex. Through concerted action of these structures, the current behavioral objective— e.g. a desired eye movement vector or maintained eye fixation—is selected. 2.1. Forebrain circuits enable competitive limited-capacity planning The brain uses analog neuronal states to represent patterned information that is distributed across multiple cells, which have finite ranges of membrane potential, firing rates, and synaptic weights. To use these finite ranges effectively, the brain employs normalizing mechanisms, notably mutual inhibition within on-center off-surround networks whose neurons obey membrane equations. By normalizing their activities, such networks enable relative activity levels to represent patterned information and thereby minimize noise and saturation effects (Grossberg, 1973, 1982). The same normalizing mechanisms can also mediate competition among plans (Bullock & Rhodes, 2003; Grossberg, 1978a). Early modeling proposed how such a competitive network could create a working memory in which the relative priority (intended performance order) among the plans constituting a forthcoming sequence is represented by the relative activation levels (Boardman & Bullock, 1991; Grossberg, 1978a,b) of the plan representations. There is now compelling evidence for both the normalization principle (e.g. Basso & Wurtz, 1998; Cisek & Kalaska, 2002; Pellizzer & Hedges, 2003) and the prediction that relative activation level codes relative priority in working memory (Averbeck, Chafee, Crowe, & Georgopoulos, 2002). The limited number of distinct activation levels that can be simultaneously represented implies a limit on the number of simultaneously active (prioritized) plans in working memory, consistent with many experimental reports (cf. Cowan, 2001). J.W. Brown et al. / Neural Networks 17 (2004) 471–510 475 Fig. 3. Cortical and striatal processes in location-specific gating of the SC by the BG. (A) When multiple stimuli exist as potential saccade goals, the corresponding PPC representations specifically excite striatal SPNs (shown in the rectangle within the BG rectangle) and non-specifically (convergently) excite feedforward inhibitory interneurons (labeled with a capital sigma), via corticostriatal projections. If more than one saccade plan is active, then striatal feedforward inhibition from all active plans prevents any one plan from activating its corresponding striatal SPNs to open the BG gate. This is because the pooled inhibitory input to each SPN can overwhelm the specific excitatory input. Therefore the SC is not released from inhibition from the SNr, and movement is prevented while the conflicting cortical plan activity remains unresolved. (B) Targets compete in PPC via inhibitory interactions. When competition resolves so that the movement plan is unambiguous, the PPC’s excitatory input to striatal SPNs eventually exceeds striatal feedforward inhibition, which wanes as competing plans lose activation and stop convergent excitation of striatal inhibitory interneurons. The winning SPN’s discharge inhibits SNr (opens the normally-closed BG gate), which disinhibits part of the SC map. (C) If the FEF plans a saccade goal that differs from the location of a strong visual stimulus, the competing frontal and parietal activities collectively drive striatal feedforward inhibition to keep the BG gate shut until the conflict resolves. (D) As the frontal cortex imposes its saccade goal on the parietal cortex, the competition between saccade goals resolves, and the BG gate opens to generate the unambiguous saccade. Note: The absence of an icon for FEF activity in (B) indicates not that FEF would be inactive in case (B), but only that FEF contains no plan contrary to PPC in case (B). 2.2. The basal ganglia contextually gate expression of reactive behaviors or plans Parallel channels of the BG embody normally-closed gates that cooperate with the thalamus, the SC and the FEF to withhold saccadic eye movements until a single reactive movement or saccadic plan is selected by competitive dynamics in the SC, FEF, and PPC. This gating hypothesis explains four data sets. (1) GABAergic inhibitory projection neurons of the two BG output nuclei, the GPi and SNr (internal or medial segment of the globus pallidus, and substantia nigra pars reticulata), exhibit tonic firing rates that are sufficiently high to suppress strong activations of recipient neurons in thalamus or SC. (2) Phasic reductions from these high resting rates, or complete pauses, release, and scale the rate of, behaviors controlled by thalamic, collicular, and other targets of GPi or SNr outputs (Bullock & Grossberg, 1991; Hikosaka & Wurtz, 1983, 1989; Horak & Anderson, 1984; Skinner & Garcia-Rill, 1990; Takikawa, Kawagoe, Itoh, Nakahara, & Hikosaka, 2002; Turner, Grafton, Votaw, Delong, & Hoffman, 1998). (3) Lesions in the BG pathway lead to a spectrum of disorders of voluntary movement (Wichmann & DeLong, 2001; Young & Penney, 2001), ranging from a hypokinetic extreme—bradykinesia 476 J.W. Brown et al. / Neural Networks 17 (2004) 471–510 hyperdirect BG pathways (e.g. Levy et al., 1997; Nambu, Tokuno, & Takada, 2002). Table 1 Anatomical abbreviations BG FEF GABA-SI GPe GPi ITa ITp LP/PUL OT PFC PNR-THAL PPC SNc SNr SPN STN VTA Basal ganglia, including striatum, GPi, GPe, STN, SNc, SNr The frontal eye fields GABA-ergic striatal interneurons The external (lateral) segment of the globus pallidus The internal (medial) segment of the globus pallidus Anterior infero-temporal cortex Posterior infero-temporal cortex Lateral posterior/pulvinar nuclear complex of the dorsal thalamus Optic tectum (called the SC in primates) Prefrontal cortex A pallidal- (GPi) or a nigral- (SNr) receiving zone of the thalamus, e.g. mediodorsal, ventral anterior, and ventral lateral pars oralis nuclei Posterior parietal cortex The substantia nigra pars compacta The substantia nigra pars reticulata Spiny projection neuron of the dorsal striatum (caudate or putamen) The subthalamic nucleus Ventral tegmental area (abnormally slow movement) and loss of ability to initiate planned movements—to a hyperkinetic extreme, including saccadic hyperdistractibility, hemi-ballism, and chorea. Thus different lesions produce too much or too little BG inhibition of plan execution. (4) Pathway tracing and physiological studies indicate an impressive degree of independence among a large number of parallel circuits that traverse the BG and return to a specific region of origin in frontal cortex (Alexander & Crutcher, 1990; Middleton & Strick, 2000). The BG thus contain a large set of parallel, programmable, gates. 2.3. Activation of the BG direct pathway can release plan execution Within each parallel BG circuit, the direct pathway-the monosynaptic pathway from the striatum to GPi or SNrenables transient opening of BG gates. A typical voluntary behavior is released via the direct pathway when a small set of striatal medium spiny projection neurons (SPNs) become active and inhibit a set of GABAergic projection neurons in the GPi or SNr (Figs. 2 and 3B). Thus, gate opening works by inhibiting a tonically-on movement inhibitor. Although this interpretation is supported by many saccadic eye movement studies (e.g. Handel & Glimcher, 1999, 2000; Hikosaka & Wurtz, 1983, 1989), its generality remains to be established, in part because of the complexity of interactions between the direct, indirect, and 2.4. Feedforward inhibition in the striatum mediates competition for plan expression The withholding function of the normally-closed BG gates opposes tendencies to react immediately to whatever stimulus is physically most salient, and creates time for plans supported by motivational and cognitive information to activate, compete, and be selected. The striatum of the BG provides a competitive arena in which representations of alternative actions vie for execution. Competition can be mediated by surround inhibition, which could be recurrent (feedback), feedforward, or a combination of the two. Despite receiving massive numbers of excitatory fibers from cortex, the striatum is known as a ‘silent structure’, in which only a small percentage of the dominant neuron type, SPNs, is strongly active at any one time. Prior proposals, that a striatal choice-making competition is mediated primarily by recurrent inhibitory collaterals of SPNs, cannot be logically reconciled with a ‘silent’ striatum, because recurrent competition would require significant supra-threshold activation of many competing SPNs. Moreover, data suggest that striatal recurrent inhibition is weak (Jaeger, Kita, & Wilson, 1994). We therefore propose that cortical plan representations bid in parallel for execution via excitatory inputs to corresponding striatal SPNs, and oppose execution of other plans by exciting GABAergic striatal feedforward interneurons (GABA-SIs) that are known to strongly inhibit striatal SPNs (Gernert et al., 2000; Koos & Tepper, 1999; Wilson et al., 1989). Because the ratio of GABA-SIs to SPNs is , 1:20, such competition by feedforward surround inhibition is entirely consistent with a ‘silent’ striatum. 2.5. Striatal activation requires convergent inputs from distributed plan representations Gate opening, for any cognitive/mnemonic or motoric degree of freedom controlled by the BG system, is usually withheld until a single plan for that degree of freedom dominates alternative active plans. Examples of cases to manage are: two parietal cortex bids (Fig. 3A), or one parietal and one FEF bid (Fig. 3C), to move the eyes in conflicting directions. Expression of two conflicting plans at the same time would result in incoherent behavior. Evidence that this problem does occur when gate opening is too rapid comes from data on saccadic averaging: the probability of mistakenly saccading to the average position, halfway between two visual target loci, is a decreasing function of reaction time (RT) and target separation (Ottes et al., 1984). Evidence that the probability of trying to execute two plans at once can be strongly affected by BG lesions comes from studies of postural responses in PD (Parkinson’s Disease) patients. Such patients exhibit non-adaptive ‘hybrid’ J.W. Brown et al. / Neural Networks 17 (2004) 471–510 responses that appear to reflect execution of two plans that are strict alternatives in healthy control subjects (Horak, Shupert, & Mirka, 1989). In another BG disease syndrome, HD (Huntington’s Disease), involuntary choreic movements, once thought to be random, often result from repeatable phasic coactivations of normally antagonistic muscles (J. Mink, personal communication, September 2002). Although tonic coactivation could arise by involuntary activation of a single normal behavior (e.g. Humphrey & Reed, 1983), phasic coactivations would more likely arise from involuntary co-execution of two (or more) plans that are normally mutually exclusive. Incoherent behavior can be avoided if broad feedforward inhibition in the striatum is complemented by a high threshold for activation of SPNs, so that they can be activated only by strong convergent excitation from cortex. Then SPNs will not activate until the several cortical representations associated with a degree of freedom, which are hypothesized to converge in the striatum, simultaneously project significant excitation to the same localized striatal region. As competition between plans resolves in favor of one winner, the cortical state becomes more coherent, in the sense that the remaining highly active sites are mutually compatible representations that help to specify the winning plan. Simultaneously, the signal from cortex to striatum becomes more focused, as diffuse signals from losing plans decline and signals from the winning plan’s distributed representations grow and converge to excite the associated small region of striatum (Fig. 3D). This hypothesis computationally clarifies reports that convergent cortical afferents to distinct localized regions of the striatum originate from cortical areas that are themselves strongly interconnected (Cavada & Goldman-Rakic, 1991; Flaherty & Graybiel, 1991; Gerfen & Wilson, 1996; Yeterian & Hoesen, 1978). It is also consistent with the gap junctions that may mediate broad activation of GABA-SIs (Kita, Kosaka, & Heizmann, 1990), and SPNs’ specialization to switch from OFF to ON states only if they receive strong convergent cortical excitation (Wilson, 1995a). 2.6. Indirect pathway activation enables deferral of a chosen plan In ‘simple’ (as opposed to ‘choice’) RT tasks, an oftrewarded plan can be cognitively primed for execution, yet await a permissive event that signals when it may be released. During the priming interval, excitation of SPNs in the direct pathway could inhibit the GPi or SNr and cause premature execution unless such gate opening were stopped until occurrence of the permissive event. The trainable ‘GO signal’ function provided by the direct pathway is therefore complemented by a trainable ‘STOP signal’ provided by the indirect BG pathway. Thus the model distinguishes two bases for non-execution of a plan. In one case, the plan may fail to satisfy the conditions for winning the direct pathway’s striatal competition. In another case, an action 477 that satisfies those conditions can be blocked by activity in the indirect pathway. Evidence that activation in the indirect pathway can be potent enough to serve this function comes from studies of PD, in which hyperactivity of the indirect pathway has been associated with inability to initiate a planned action (Wichmann & DeLong, 1996). The striatal source cells for the indirect pathway affect the GPi (or SNr) via two routes: striatum-GPe-GPi (or SNr) (Hazrati, Parent, Mitchell, & Haber, 1990; Parent & Hazrati, 1995; Smith & Bolam, 1990), and striatum-GPe-STN-GPi (or SNr) (e.g. Wichmann & DeLong, 1996). Consistent with recent experimental results (Hassani, Moroux, & Feger, 1996; Levy et al., 1997), this hypothesis differs from various prior models by treating the former (shorter) route as the more potent, and thus the more important for the STOP function. Below, hypothesis 14 specifies a distinct function for the STN-GPi link, which is part of the so-called ‘hyper-direct’ pathway. 2.7. Thalamo-striatal feedback guides learning of indirect pathway STOP responses Learning necessary to generate a STOP signal occurs during trials on which premature release of a movement leads to non-reward. In the model, such experiences can lead to learned activation of the indirect channel associated with the released movement, even though striatal activation in the indirect channel is not a normal part of releasing the movement. How does the brain identify which indirect pathway it should recruit in order to STOP a given direct pathway? We propose that the feedback pathways (Fig. 2) from the thalamus to the striatum (De Las Heras, Mengual, Velayos, & Gimenez-Amaya, 1998; McFarland & Haber, 2000) help solve this credit-assignment problem by routing a specific teaching signal to the indirect channel associated with whatever direct channel has just activated. Because the thalamo-striatal pathways have been neglected in prior models of the BG, and there is little pertinent electrophysiological work, this hypothesis is currently based largely on anatomical and computational considerations. 2.8. Laminar maps enable gated interactions between planning and executive cells Alternative plans are coded, in the SC and FEF, at distinct spatial positions in 2-D cellular arrays or maps. To complement the BGs parallel gating system, ‘planning’ cells encode preparatory activities and send bids to the BG, and associated ‘executive’ cells generate phasic outputs if and when their BG gate opens. This hypothesis predicts a multilayered—i.e. laminar—organization and a distinctive pattern of connectivity. Such is true of the SC, which projects to the BG (via dorsal thalamus, e.g. LP/PUL) from superficial layer maps (e.g. Butler & Hodos, 1996; Hall & Lee, 1993; Hutsler & Chalupa, 1991) and receives SNr outputs in deeper layer maps that are representationally and 478 J.W. Brown et al. / Neural Networks 17 (2004) 471–510 topographically ‘in-register’ with superficial layers. As schematized in Fig. 2, there is also abundant anatomical evidence for a similar stratification of cells and fiber systems in the frontal cortex (Berendse, de Galis, & Groenewegen, 1992; Canteras, Shammah-Lagnado, Silva, & Ricardo, 1990; Gerfen & Wilson, 1996; Giuffrida, Volsi, Maugeri, & Perciavalle, 1985; Jones, Coulter, Burton, & Porter, 1977; Levesque, Charara, Gagnon, Parent, & Deschenes, 1996; Royce & Bromley, 1984; Wilson, 1995a,b; Yeterian & Pandya, 1994). A large portion of cortical fibers projecting to the striatum emerge from small pyramidal cells of layers II – III or III– Va (depending on species and cortical area). Although layer Vb large pyramids send some collaterals to BG (including STN and striatum; see hypothesis 14 below), such “Brainstem projecting neurons make up a very small proportion of the total population of cortico-striatal cells…. The vast majority of (cortico-striatal) neurons are more superficially situated, smaller pyramidal neurons (Wilson, 1995a, p. 38)”, which also have cortico-cortical connections. There is accumulating physiological evidence (Bruce, Goldberg, Bushnell, & Stanton, 1985; Iwabuchi & Kubota, 1998; Sawaguchi, 2001; Segraves & Goldberg, 1987; Turner & DeLong, 2000) that the cortico-striatal cells in layers III– Va have different task-related properties than the large pyramidal cells that are clustered in layer Vb and send outputs to the brainstem and effectors. For the part of FEF (area 8) that they examined, Iwabuchi and Kubota (1998) found that neurons with early onsets coupled to a GO cue (signaling whether to release a response) were numerous in layer III, whereas neurons whose onsets were late and movement-coupled were more numerous in layer V. Thus there was a normal sequence involving activation of layer III neurons before layer V output neurons. These data support the hypothesis of at least distinct, if not completely segregated (Sommer & Wurtz, 2000; Wurtz, Sommer, Pare, & Ferraina, 2001) laminar distributions of ‘planning’ and ‘executive’ cells within areas of frontal cortex. Some models assume or imply that such a functional differentiation is achieved across cortical areas, rather than by layers within areas. However, moving across areas typically implies a change in cortical representation. Columnar laminar neocortex can efficiently implement the planning/ execution distinction across layers. Although the concept of planning has some connotations that go beyond the competence of the current model, the current usage of ‘plan cell’ is consistent with prior usage (e.g. Bullock & Grossberg, 1991). A neural representation qualifies as a plan if it helps specify a possible forthcoming response, and if the circuit controlling the effects of the representation’s activation provides a basis for withholding the active representation’s access to the effector apparatus until a decision has been reached to execute the plan. To this the current analysis adds another condition: that the putative plan representations must send bids to the telencephalic decision centers that gate their access to effectors. Thus the response representations that exist in ungated reflex systems do not qualify as plans; nor do gatable output stages that send no projections to telencephalic decision centers. Finally, although the plan cells in the current model can remain active across significant delays after stimulus offset (hypothesis 12 and Fig. 9F and G, below) this may not be true of all cells properly so-called. 2.9. Small BG gating channels control large topographic zones in laminar target structures How broad a region of frontal cortex (or SC) is affected by opening of a particular BG gate? On gross anatomical grounds, the cortico-striatal-GPi/SNr projection has been described as a ‘funnel’, and indeed there is an order of magnitude reduction in numbers of cells from the pool of cortico-striatal cells to the pool of striatal SPNs (Zheng & Wilson, 2002), and another order of magnitude reduction from the pool of SPNs to the pool of GPi/SNr projection neurons (Oorschot, 1996; Wickens, 1997). The ‘funnel’ usage has waned because it was interpreted to imply mixing rather than the (now well-established) segregation of channels, but the radical reduction in cell counts implied by ‘funnel’ remains a key constraint on models. In frontal cortex, many more cells are reached by the projection from PNR-THAL (pallidum or nigra receiving thalamus) than there are cells in PNR-THAL, or in the GPi/SNr. We thus hypothesize a one-to-many relationship between cells in GPi/SNr and gatable cells in frontal cortex. It is probable that, similar to SNr – SC interactions (Handel & Glimcher, 1999, 2000; Hikosaka & Wurtz, 1983, 1989), pausing by each small cluster of cells in the GPi/SNr directly affects (via PNR-THAL) a much larger set of cells that are distributed across a moderate-sized zone of a given frontal cortical area. Further cells will then be affected by secondary, e.g. cortico-cortical, interactions. Thus action-selection theories of the BG cannot plausibly assume a one-to-one relationship between plan representations and BG channels. Models must explain how the effect of gating can be highly selective and accurate even though the number of cellular degrees of freedom in the GPi/SNr is relatively small. In many cases, two or more cortical plans will have similarly high activation levels, and the gating signal needs to select the cortical plan that has won the striatal competition rather than any highly active alternatives. The current model hypothesizes that loops through the BG circuit respect fronto-cortical topography: each frontal zone receives thalamo-cortical fibers from a zone of PNR-THAL that receives fibers from a zone of GPi/SNr that receives fibers from the striatal zone targeted by the frontal zone of origin. This principle has been established for the large-scale topographic organization of frontal cortex. For example, there are separate motor (area 4), premotor (area 6), oculomotor (FEF) and prefrontal circuits (Alexander & Crutcher, 1990; Middleton & Strick, 2000). To explain how the system achieves sufficient selectivity, the model proposes that such circuits also J.W. Brown et al. / Neural Networks 17 (2004) 471–510 respect finer-grained topographic organization. In particular, selectivity can be achieved even if the cortical activation levels of two plans are indistinguishable, so long as the plans exist in different gateable cortical zones (GCZs). For simplicity, the model (see Fig. 5) defines three BG channels and three associated GCZs in FEF, but these discrete elements could be replaced by an overlapping continuum in a straightforward way in a population model. The main computational requirement is that the cortical activity foci representing two competing plans be sufficiently separated that one receives a reliably stronger GO signal from the PNR-THAL. Hypotheses 4, 5 and 9 imply a kind of complementarity in the macrocircuit’s management of competitive plan execution. Any two cells that represent different plans in the same GCZ are likely to compete via strong mutual intracortical inhibition, so that when the competition resolves and the gate opens for the entire GCZ, only the winning plan will execute. If two plan cells pertinent to a given degree of freedom do not reside in the same GCZ, then they are less likely to compete via such strong mutual intra-cortical inhibition, so a large activation of one may not imply a small activation of the other. However, in this case, striatal competition can ensure that only one of the GCZs receives a strong GO signal. In either case, a non-viable ‘hybrid’ response is generally avoided. 2.10. Selective gating of premotor zones enables fast sensory-motor remapping A gating system can control which of several alternative sensory-motor or cognitive-motor pathways are used to generate behavior. This ability is fundamental because a visual stimulus may serve either as a movement target or as a discriminative cue to move elsewhere. Sensory/cognitive and motor information is often represented in different coordinate systems. The model shows how learned associations can form between such different coordinate representations, notably from spatially invariant sensory/cognitive representations to spatially coded motor representations. For example, in the SC map, plan representations cluster on the basis of eye movement direction and amplitude (a vector code), not on the basis of the stimulus modality (vision, audition, or touch) that may excite them. In contrast, premotor frontal cortex embodies superimposed gradients, across the cortical map, based on modality and source of cortico-cortical input. This creates a multiplexed patchiness in receptive field structure and the sensitivity of local cortical zones to perceptual, categorical, and mnemonic inputs. Within each patch, many movement vectors are also represented, so such movement features are not the basis of map structure. In particular, in premotor cortex (area 6), there is a well-established medio-lateral gradient of sensitivity to proprioceptive vs. exteroceptive inputs (Mushiake, Inase, & Tanji, 1991; Passingham, 1993). In FEF, which spans major parts of areas 8 and 45, there is 479 a medio-lateral gradient for saccadic amplitude, from largeto-small saccades, in the rostral bank of the arcuate sulcus (Bruce et al., 1985). Moreover, this medio-lateral gradient for amplitude reflects differences in the nature of the afferents arriving in subareas of the FEF (Barbas, 1988; Barbas & Mesulam, 1981; Bullier, Schall, & Morel, 1996; Schall, Hanes, Thompson, & King, 1995b; Schall, Morel, King, & Bullier, 1995a). The medial, ‘large-saccade’, part of FEF receives afferents from dorsal stream visual and auditory areas that process information about stimuli in, or even beyond, the visual periphery. This medial FEF area appears to lack significant inputs from ventral stream areas TEO/V4 (ITp) and TE (ITa), both of which preferentially represent stimuli in the foveal and parafoveal regions of the visual field. In the lateral FEF, a strong projection from ITp terminates in FEF zone 45A and a strong projection from ITa terminates in FEF zone 45B (Bullier et al., 1996). Thus FEF zones receive functionally distinct types of inputs. The model’s GCZs in FEF reflect these input differences. Specifically, the model incorporates the fact that cells in ITp are sensitive to simple features falling within particular retinotopic loci (Kobatake & Tanaka, 1994; Tanaka et al., 1991; Komatsu & Ideura, 1993), whereas ‘position invariant’ cells in ITa are sensitive to feature complexes (‘objects’) regardless of specific retinotopic locus (Gross, Desimone, Albright, & Schwartz, 1985; Tanaka et al., 1991). These systematic differences in the types of cues to which the FEF sub-areas are sensitive are consistent with the more general pattern observed for motor and premotor areas of frontal cortex (Passingham, 1993). The model proposes that sensitivity to distinct classes of inputs covaries with the GCZs. Thus, unlike BG gating of SC, BG gating of cortex operates on plan clusters defined by cue type and modality. The emphasis on cue sensitivity differences in FEF may seem surprising, because it was established long ago that FEF neurons may show no selectivity for stimulus properties such as shape or color (e.g. Mohler, Goldberg, & Wurtz, 1973). However, evidence on the anatomical projections from feature sensitive areas such as ITp, when combined with physiological evidence on the emergence of feature selectivity in FEF neurons when features are consistently rewarded (Bichot, Schall, & Thompson, 1996), support the present hypothesis, as well as hypothesis 12 (below) regarding reward-guided learning mediated by weight changes on the IT to FEF pathways. Fig. 5A illustrates this hypothesis. Maps of retinotopic inputs and motor error outputs at cortical and collicular levels are modeled by 3 £ 3 cell grids. Each square within a grid represents a model cell with a receptive or movement field corresponding to its grid position, so each grid represents fixation (the null direction) plus eight saccade directions (each with unit amplitude). Fig. 5B shows that each of the three FEF zones has input, plan, and output layers. Overall, the model has twelve gateable zones, three cortical zones of 9-cells each, and nine collicular zones of 1-cell each. The model approximates the SNr –SC system 480 J.W. Brown et al. / Neural Networks 17 (2004) 471–510 with a map-like array of nine SNr cells, each with output restricted to a corresponding region of the 9-cell SC map. Each SNr output to the SC is specific for a zone coding similar saccadic directions and amplitudes (Bayer, Handel, & Glimcher, 1999; Hikosaka & Wurtz, 1989). Thus, there is not a single non-specific release of the entire SC when a BG gate opens. Although gating is specific to a subset of the SC map, it is coarse relative to the direction-specifying signals that are received by the SC from other structures, such as the PPC, FEF, and the retina (which is not modeled). Thus, saccadic accuracy depends mostly on non-nigral inputs to the SC, but saccadic occurrence, and to a lesser extent, velocity, depend on the degree of release of nigral inhibition (Takikawa et al., 2002). Greater realism could be achieved by scaling up the resolution of the SC map relative to the SNr map, but this was unnecessary for present purposes. For the model FEF, there are three BG gates and corresponding PNR-THAL cells, each used to excite the entire grid within one of the three model GCZs. These GCZs correspond to patches of FEF that differ in whether they receive inputs from ITa or ITp, and, for the two GCZs with inputs from ITp, whether they respond more strongly to one discriminative visual feature or another. Thus each FEF GCZ controls a full range of saccadic directions, but its cells have a similar sensitivity to one or another visual feature, e.g. ‘red’ or ‘T-shaped’. In the following exposition, one of these GCZs is called the GCZf (GCZ, fixation feature) because cells in its ITp receptive field respond selectively to salient features of the fixation stimulus that distinguish it from the target stimuli to which the monkey will saccade after fixation light offset (Fig. 1). The second GCZ is called the GCZt (GCZ, target feature) because its ITp receptive field is selective for salient features of target stimuli that distinguish them from the fixation stimulus. The model’s third FEF gateable zone (Fig. 5B) is called the GCZo because it corresponds to the part of area 45 that receives inputs from ITa (area TE) cells that show selectivity for feature complexes (‘objects’) regardless of where these complexes fall on the retina. Although the GCZo does not receive a retinotopic projection from ITp, it does have other inputs, from the SC and LIP (not shown in Fig. 5), that endow it with retinotopically organized planning and output layers. These two properties allows the GCZo to learn to associate a spatially invariant object category with a saccade direction, thus mapping a What stream sensory cue to a Where stream saccade plan. These GCZo cells thus mediate a learned What-to-Where transformation between different coordinate systems. Although cells in the GCZf and GCZt are in the What stream, in a sense they mediate a Where-toWhere transformation because their feature-specific filtering also reflects retinotopic organization within ITp. With these three GCZs, the model is able to open one gate in the presence of two or more competing visual stimulus representations, and release movement only toward the stimulus that matches the feature or object class associated with the open gate. Thus, the correct movement will occur provided that the plan cell sitting in the correct GCZ becomes able (via reinforcement learning, see below) to win the striatal competition to open its gate. In a larger scale implementation, these fully discrete channels would be replaced by overlapping populations. 2.11. Sustained plan cell activity is a local cortical circuit property modulated by layer VI Cortical cells whose activity correlates with plans often show task-dependent sustained firing that outlasts the stimulus presentations that cue the initial activation of these cells. In the model, sustained activation results from local, intra-cortical, recurrent excitation whenever the plan cell resides in a GCZ that has been aroused or primed. Such activation occurs when input from PFC reaches plan cells via a cellular link in cortical layer VI; see Fig. 2. These same layer VI cells are a source of excitation to the PNR-THAL cells whose disinhibition allows plans to execute. Differential priming via layer VI allows the frontal cortex to bias the pool of candidate plans to favor those that fall under the particular input –output relation enabled by the primed GCZ (Asaad, Rainer, & Miller, 2000; Hoshi, Shima, & Tanji, 1998). In contrast, some models have proposed that the basis of sustained frontal activations is a cortico-thalamo-cortical loop that starts with a cortical plan cell, traverses a thalamic stage subject to inhibition by the GPi or SNr, and returns to the plan cell of origin. In such a model, a BG gate must open to enable sustained activity of each plan cell. The latter seems unnecessarily restrictive for brief maintenance of candidate plans in FEF before one plan is selected and released, and may not even be necessary for prolonged storage in PFC (cf. Durstewitz, Seamans, & Sejnowski, 2000). Moreover, anatomical studies (cited in Table 2) show that the thalamo-cortical projection (to layers III and V) from PNR-THAL may not even reach, and is certainly not restricted to, the layer VI cells of origin of the cortico-thalamic projection. Thus, there seems to be no anatomical basis for restricting modeled thalamo-cortical input so that it excites only plan cells, and therefore no basis for preventing this input from activating output cells in layer Vb. If so, it cannot serve the priming function proposed in alternative models. 2.12. Reinforcement signals guide learning of cortico-cortical and cortico-striatal links Reinforcement learning can modify links from representations of arbitrary contexts and cues to plan representations. It can also affect interactions among candidate plans. Phasic, non-specific, dopaminergic signals emerging from the VTA/SNc in response to unpredicted rewarding events (Schultz, 1998) reach both frontal cortex and the striatum, where they modulate synaptic plasticity (Bao, Chan, & Merzenich, 2001; Berger, Trottier, Verney, Gaspar, & Alvarez, 1988; Calabresi et al., 2000; Charpier & Deniau, J.W. Brown et al. / Neural Networks 17 (2004) 471–510 481 Table 2 Selected anatomical studies From To Species References Layer VI Distant cortex Layer III Layer III, V Layer III, V Layer II, III Layer V Striatum Cat Cat Cat Rhesus monkey, rat Layer VI VL (Thalamus) Layer V Gpe Gpe STN VA, MD nuclei of thalamus Layer V, III STN GPi SNr GPe, GPi Rhesus monkey, rat Rat Rat Squirrel monkey, rhesus monkey Rat Squirrel monkey SC, intermediate layers Ita Itp FEF (via thalamus) FEF zone 45B FEF zone 45A Cebus monkey Monkey Monkey Kang and Kayano (1994) Iriki et al. (1991) Kimura, Caria, Melis, and Asanuma (1994) Jones et al. (1977), Levesque et al. (1996), Yeterian and Pandya (1994) Yeterian and Pandya (1994), Levesque et al. (1996) Zin-Ka-Ieu et al. (1998) Canteras et al. (1990), Gerfen (1992) Hazrati et al. (1990), Parent and Hazrati (1993, 1995) Smith and Bolam (1990) Parent and Hazrati (1993), Shink, Bevan, Bolam, & Smith (1996) Lynch et al. (1994), Sommer and Wurtz (1998) Bullier et al. (1996) Bullier et al. (1996) 1997; Cohen, Braver, & Brown, 2002; Eyny & Horvitz, 2003; Gurden, Takita, & Jay, 2000; Law-Tho, Desce, & Crepel, 1995; Reynolds, Hyland, & Wickens, 2001; Wickens, Begg, & Arbuthnott, 1996). In the model, arbitrary cueing of plan activation is learned through reinforcement-guided modification of cortico-cortical links terminating in frontal cortex; see Fig. 2. Then these cueactivated plan representations compete while exciting GABA-SIs and SPNs in the striatum. Learned weights on the cortico-striatal projections to SPNs can affect the competition for plan execution and thereby facilitate appropriate queuing of plans that are rewarded if performed in the proper context/sequence, but not otherwise. Consistent with recent data (Zheng & Wilson, 2002), this proposal departs from views of the striatum as a general adaptive pattern recognizer capable of representing arbitrary combinations of cortical states (Amos, 2000; Graybiel, 1998). In our model’s BG direct and indirect channels (see Fig. 4), phasic dopaminergic bursts and dips (Brown et al., 1999; Schultz, 1998) have opposite learning effects (Eyny & Horvitz, 2003), mediated by the differential expression of D1 and D2 dopamine receptors in striatal SPNs of the two channels. In particular, DA bursts cause increments of weights on recently active pathways to D1, substance P (SP) and dynorphin (DYN) expressing SPNs of the direct channel, whereas DA dips cause increments of weights on recently active pathways to D2 and enkephalin (ENK) expressing SPNs of the indirect channel. This is consistent with observations (review in Steiner & Gerfen, 1998) that D1 and D2 receptor activations normally exert opposite effects on IEG (immediate-early gene) expression in SPNs. D1 activation induces IEG expression in D1 SPNs, but D2 activation does not induce IEG expression in D2 SPNs. Instead, given cortical input or NMDA receptor activation, D2 blockade induces IEG expression in D2 SPNs. Note that the D2 blockade may be equivalent to what occurs naturally when DA input dips due to pausing of SNc cell activity (Schultz, 1998). Overall, we emphasize cortico-cortical learning for control of the direct channel and cortico-striatal learning for control of the indirect channel. The latter is supported by recent data of Berretta, Parthasarathy, and Graybiel (1997). Unlike some (e.g. Young & Penney, 2001), we do not interpret the Berretta et al. (1997) data to imply a scarcity of adaptive mono-synaptic cortico-striatal links to the direct pathway. Many other data, e.g. the nature of cortical and SNc axon synapses with spines on virtually all SPNs, and the IEG and plasticity studies cited above, argue otherwise. The model predicts that dopaminergic (DA-ergic) reinforcement signals interact with traces of stimulus and plan activations, not with plan activations, or responses, as such. When a delayed reinforcement signal signifies Fig. 4. Direct and indirect pathways in the BG circuit model. The BG direct pathway projects from the striatum to the GPi or SNr. Because it disinhibits thalamo-cortical excitation, activation of the direct pathway normally generates a ‘GO signal’ that enables action plans to be executed. Activation of the indirect pathway, which projects from the striatum to the GPe to the GPi or SNr, inhibits PNR-THAL and can thus STOP planned actions, even if there is activation in the direct pathway that would otherwise suffice to generate a GO signal. The feedback from thalamus to striatum helps solve a credit assignment problem inherent in learning stimulus control of STOP signals (see text for details). Weights on cortico-striatal afferents of SPNs are subject to learning that depends partly on dopaminergic (DA) signals. In the indirect pathway, a phasic DA dip disinhibits D2 receptors and permits LTP. In the direct pathway, a DA burst rather than a dip permits LTP by activating D1 receptors. Model cortico-striatal synapses on SPNs also exhibit LTD, consistent with Calabresi et al. (2000). 482 J.W. Brown et al. / Neural Networks 17 (2004) 471–510 a positive outcome, it can strengthen an input’s ability to activate the plan representation causally responsible for the rewarded act at a time when the plan representation has already been de-activated by response feedback. In the model, the time between termination of plan cell activation and the reinforcement signal is assumed to be bridged by an intracellular second messenger, which provides a short-term ‘trace’ of the earlier plan activation. Such traces are predicted to be of brief duration, consistent with data showing that reinforcement learning falls off steeply when reward occurs more than a few seconds after the response. The time scale of these traces is shorter than that of working memory traces in dorsolateral PFC, which is not needed for reinforcement-guided learning or performance of externally cued behavior (Goldman-Rakic, 1987). 2.13. Feedback from superior colliculus to FEF serves learning and performance functions STN (Parent & Hazrati, 1995). Further components of lockout protection may be supplied by specialized membrane properties (Wilson, 1995a) and recurrent GABAergic collaterals of SPNs, but these have not yet been incorporated in the model. Although hypothesis 14 differs from the recent proposal regarding the ‘hyperdirect’ pathway made by Nambu et al. (2002), their proposal was based on electrical stimulation studies that did not take account of either the laminar differences between (most) cortico-striatal and cortico-STN cells or the physiological differences between such cells, as suggested by the results of Iwabuchi and Kubota (1998) and explicitly probed by Turner and DeLong (2000). The macrocircuit implied by these fourteen hypotheses is schematized in Fig. 2, which also serves as a key to most of the important variable names in the mathematical specification of the model. Table 2 tabulates anatomical studies supportive of the connectivity shown in Fig. 2. Whenever any collicular zone becomes active enough to exceed a threshold, it excites FEF plan layer cells tuned to the same direction and grid position, across the three GCZs. This excitation of FEF plan cells by SC (via thalamus; Sommer & Wurtz, 1998) serves as a teaching signal that enables GCZs not initially responsible for SC activation to learn to take over responsibility for activating a specific SC map region on future occasions. This teaching signal is specific, and therefore more like the specific thalamostriatal teaching signal proposed in hypothesis 7 than the non-specific DA teaching signal proposed in hypothesis 12. When the collicular activation becomes large enough to initiate a saccade, the SC to FEF excitation also activates FEF inhibitory interneurons that can selectively ‘reset’ FEF plan cells tuned to the same direction as the activated SC cells. The model interprets the post-saccadic cells (see Fig. 9E) found in FEF (Bruce et al., 1985) to be inhibitory reset neurons (Fig. 2) that terminate specific plan representations once plan execution has initiated. 2.14.1. Mathematical model Mathematical specification of the TELOS model’s hypotheses required a system of differential equations, whose exposition follows. The Results section may be read before studying these equations. Figs. 6a and b depict every model cell type, together with labeled inputs. These labels refer to variables in the equations. Each cell type obeys a nonlinear membrane (or ‘shunting’) equation (Grossberg, 1973; Hodgkin, 1964), in which excitatory and inhibitory inputs affect separate conductances and thus do not act as additive injected currents that directly affect activation/ potential. Often, a second and third equation further define the membrane equation by computing the cell’s net excitatory and inhibitory inputs, symbolized by superscripts (E) and (I). In addition to explaining the form of the equations, this section cites additional data that support model details. 2.14. Cortico-subthalamic signals enhance lockout of competing plans during plan execution 3. Visual inputs to the cortical cell types Once a degree of freedom has been allocated to execute a plan, it should be protected from further plan bids and from interfering reactive movements (cf. Mink, 1996) while that plan is executing. The basis for such protection in the model is distributed, and includes the persistence of plan cell activity into the movement interval. One anatomical basis for locking out disruptive bids during execution is excitation of the BG’s ‘hyperdirect’ pathway (cortex-STN-GPi) by collaterals of the executive pathways. This hypothesis explains observations that collaterals of frontal cortical layer Vb projection neurons excite the STN (Canteras et al., 1990; Gerfen & Wilson, 1996), which in turn excites the GPe and GPi or SNr. It also explains why the GPi/SNr receives focused inhibition from striatum but broad excitation from p External visual stimuli Ixyj were convolved with a Gaussian kernel to approximate visual cortical receptive field properties, in order to generate the pre-processed internal signal Ixyj (Fig. 2). Specifically, Ixyj ¼ X ðp;qÞ[C p Ipqj ! 2ðp 2 xÞ2 2 ðq 2 yÞ2 ; exp ð0:7Þ2 ð1Þ where C is the set of eight nearest neighbors in the Cartesian input space. In the position-sensitive GCZs of the FEF, the signal Ixyj ðpÞ then generated two further signals, namely Ixy and ðpÞ ðdÞ Ixyj (Fig. 2). The positional FEF input Ixy is a transient, J.W. Brown et al. / Neural Networks 17 (2004) 471–510 rapid-onset, input to FEF: 8X ðonÞ < I if 50 ms # t 2 txyj # 80 ms j xyj ðpÞ Ixy ¼ :0 otherwise ð2Þ This models the property that FEF input cells respond quickly (with about 50 ms latency) to stimuli in their retinotopic receptive fields, regardless of stimulus features. ðonÞ The symbol txyj is the time at which the convolved stimulus signal Ixyj rises above 0.3, an arbitrary threshold chosen to register activity but ignore spurious noise, and t is the ðpÞ current simulation time. Eq. (2) specifies that input Ixy to FEF spans the interval of 50– 80 ms after Ixyj rises above threshold. The interval was set to accord with data on FEF cell onset and offset; see first burst in Fig. 9A. By default, ðpÞ Ixy ¼ 0 outside that interval, or if no inputs are active. The ðonÞ quantity txyj is not reset unless Ixyj first falls below the threshold 0.3. If the features possessed by the external stimulus are preferred by an FEF cell, then it fires a second burst at around 100 ms. This is modeled with the stimulusðdÞ discriminating FEF input signal, Ixyj ; assumed to arrive from visual area V4 and ITp: 8 < Ixyj if 100 ms # t 2 tðonÞ # 130 ms xyj ðdÞ ð3Þ Ixyj ¼ :0 otherwise The interval 100 –130 ms. was based on timing of the onset of the second burst in the data; see second burst in Fig. 9A. ðpÞ Input I(pc) xy to the PPC is similar to the positional Ixy input to FEF, except that there is no upper time limit, because PPC activity is sustained as long as the visual input remains on (Fig. 10C). Thus, 8X ðonÞ < I if t 2 txyj . 50 ms j xyj ðpcÞ Ixy ¼ :0 otherwise 483 ð4Þ The 50 ms onset delay reflects the time required for light impinging on the retina to activate PPC, as seen in Fig. 11C. In addition to these positionally-sensitive visual signals, the model also invokes signals to the gated cortical zones that respond to objects independent of their positions. Thus inputs Tj from ITa cells (Figs. 2 and 6a) respond to visual features of type j regardless of their spatial position (hence the absence of a subscript xy). The dynamics of spaceinvariant anterior IT cells Tj were modeled by: d T ¼ 150ð1 2 Tj ÞIjðITÞ 2 30Tj ; dt j ð5Þ Thus Tj can be excited by model input IjðITÞ to a maximum value of 1, and can also spontaneously decay. 4. Reactive and attentive processing: PPC, SC, and SNr gating signal The model supposes that developmental processes have enabled the FEF representations of oculomotor plans to be in register with the SC and PPC representations; see Gancarz and Grossberg (1999) for a consistent model that proposes how this happens. PPC cell activities Pxy (Figs. 2 and 6a) represent responses in the lateral bank of the intraparietal cortex (LIP), which code visual stimuli in motor error coordinates (Gnadt & Andersen, 1988). These activities were Fig. 5. Grid of gatable zones and laminar organization. (A) Maps of retinotopic inputs and motor error outputs at cortical and collicular levels are represented by two-dimensional grids. For convenience, the cells are indexed by a Cartesian coordinate system, which could be replaced by a space-variant representation. (B) The model FEF contains three distinct gatable cortical zones (GCZs), each possessing input, plan, and output layers, as well as some distinctive cortical afferents. These three GCZs are indexed by i, whereas k (see Fig. 2) indexes the corresponding three BG-thalamic channels that control them. 484 J.W. Brown et al. / Neural Networks 17 (2004) 471–510 cells was defined by X ðOÞ ðOÞ ðPÞ þ 4 ðPCÞ PðEÞ f ðFxyi ÞÞ þ 2ð½Fxyi Þ ; xy ¼ 5ðIxy þ modeled by: d ðIÞ P ¼ 25½ð1 2 Pxy ÞPðEÞ xy 2 Pxy Pxy : dt xy ð6Þ Eq. (6) says that Pxy can be excited by net input PðEÞ xy to a maximum value of 1, and inhibited by net input PðIÞ xy to a minimum value of zero. The excitatory input PðEÞ to PPC xy ð7Þ i ðPCÞ which includes excitation from visual signals (Ixy ; defined ðOÞ below in (8)), FEF output layer cells (Fxyi ; defined in (21)), ðPÞ and FEF plan layer cells (Fxyi ; defined in (14)). Here the summation index i ranges from 1 to 3 to represent different FEF zones, as in Fig. 5. The notation ½wþ ¼ maxðw; 0Þ Fig. 6. Modeled afferent signals, and mathematical symbols for physiological variables, associated with model cell types. As implied by Fig. 2, the model includes a large number of cell types distinguishable by their afferent and efferent connections. See text for details. (a) Cell types of the cortical model. (b) Cell types of the subcortical model. J.W. Brown et al. / Neural Networks 17 (2004) 471–510 485 Fig. 6 (continued ) means that the argument w is thresholded at zero. The visual signals to PPC can drive reactive saccades. However, the potent input from FEF can override the visual signals and reprogram the PPC to reflect the spatial goal of a previously rewarded saccadic plan. For reactive saccades, the strengths of competing visual signals must be unequal for a decision to be made. Both cortical magnification and motivational signals play a role in the competition that determines PPC attentional focusing (Platt & Glimcher, 1999). Here cortical magnification ðPCÞ sufficed, and the visual inputs Ixy to the model PPC cells were defined by: ðPCÞ ðpcÞ Ixy ¼ Ixy þ 0:01ð1:1lx 2 1l þ ly 2 1lÞ: ð8Þ This ensures that given multiple inputs of equal strength, peripheral inputs are more likely to evoke a saccade than foveal inputs, and objects left or right of the fovea are more likely to attract the eye than objects above or below. Nonetheless, this peripheral bias is not sufficient to maladaptively break fixation of a visible stimulus. The inhibitory input PðIÞ xy to PPC cells in (6) was defined by X ðRÞ PðIÞ ð½Ppq þ Þ4 : ð9Þ xy ¼ 1 þ 10Pxy þ 200 ðp;qÞ–ðx;yÞ Term PðRÞ xy specifies recurrent self-inhibition, which ensures phasic PPC activity, as seen in Fig. 11C. Dynamics 486 J.W. Brown et al. / Neural Networks 17 (2004) 471–510 for the interneuron mediating self-inhibition were defined by d ðRÞ ðRÞ P ¼ 2½ð1 2 PðRÞ xy ÞPxy 2 Pxy : dt xy ð10Þ The summation term in (9) describes quickly-acting lateral inhibition. Activities Sxy of SC cells (Figs. 2 and 6b) were excited by ðOÞ signals from PPC (Pxy) and the FEF output layer ðFxyi Þ; and inhibited by the topographic and potent SNr output, GðSNrÞ xy ; and a tonic signal set to a value of 10: X d ðOÞ Sxy ¼ ð1 2 Sxy Þð60Pxy þ 5 f ðOÞ ðFxyi ÞÞ dt i 2 0:3þ þ 10Þ 2 Sxy ð800½GðSNrÞ xy X d ðGÞ Fi ¼ ð1 2 FiðGÞ Þð80Tj þ 2000 WjiðCGÞ ½Cj 2 0:35þ Þ dt j 2 160FiðGÞ : ð11Þ GðSNrÞ xy from SNr (defined in (32)) The inhibitory input tends towards 1 when the BG gate is shut, and towards zero when open. In the former case, Eq. (11) implies that SC activity will be low even in the presence of strong excitatory inputs. When the gate opens as GðSNrÞ is driven near zero, Sxy xy can become highly active in response to inputs from PPC or FEF. In the simulations, the SNr output signal GðSNrÞ had the xy same indices and spatial resolution as the model SC map. Although the in vivo SC map has much higher resolution than the SNr output, this higher resolution was not needed for the data simulated. Also, the simple SC model used here has SC burst cells but omits SC buildup cells (Munoz & Wurtz, 1995). A consistent but more detailed SC model, with buildup cells as well as burst cells, was described in Grossberg, Roberts, Aguilar, and Bullock (1997), but the added detail was not needed for the data modeled in this article. Because of its simplified SC treatment, the present model does not include connections from FEF plan cells to the SC, a projection that is consistent with the computational hypotheses of the model and that appears to be justified by physiological observations (e.g. Sommer & Wurtz, 2000). 5. Planning in the frontal eye fields ðIÞ The activity Fxyi of an FEF input cell (Figs. 2 and 6a) was defined by: d ðIÞ ðIÞ ðdÞ ðIÞ ðpÞ F ¼ 60ð1 2 Fxyi Þ½Ixyj þ Ixy 2 Fxyi ½100 dt xyi X ðdÞ þ 30 ðIrsðpÞ þ Irsj Þ: (12)) among FEF visual cells (Schall et al., 1995b) helps to normalize the overall activity and allows a stronger response to a unique, isolated stimulus than to multiple stimuli, consistent with the ‘oddball discrimination’ effect observed in the FEF (Thompson, Bichot, & Schall, 1997). The model FEF layer VI cells arouse a zone of cells in the columns above (see Figs. 2 and 6a). Model layer VI activity FiðGÞ was defined by ð12Þ ðr;sÞ–ðx;yÞ ðIÞ Here the index i in the subscript of Fxyi ranges from 1 to 3 and denotes FEF GCZ, as in Fig. 5. These model FEF ‘visual cells’ (Schall et al., 1995a) respond to excitation from areas including V4 and ITp, and are predicted to reside in granular or supragranular layers. The excitatory inputs ðpÞ ðdÞ Ixyj (where j denotes stimulus feature) and Ixy were defined P in (2) and (3). Feedforward lateral inhibition (the term in ð13Þ Input Tj from ITa provides a training signal that enables positionally-invariant inputs Cj ; from prefrontal working memory cells to learn to select a zone of FEF cells, constituting a premotor zone i; by activating module i’s layer VI cells. The adaptive weight WjiðCGÞ ; on the path from PFC site j to FEF module i, denotes a synaptic strength that can be modified by reinforcement learning; see Eq. (55). Model FEF plan layer cells correspond anatomically to layer III/Va cells from which the cortico-striatal projection arises (Figs. 2 and 6a), and physiologically to FEF ‘visuomovement cells’. As shown in Fig. 5, they may be fixationor saccade-related, and they may perform a ‘What – Where’ or a ‘Where – Where’ transformation via distinct cortical zones. The combination of these attributes leads to four kinds of plan layer cells with distinct connectivity patterns. Within a cortical zone, cells generally compete, but there is no general between-zone competition. However, learned selectivity of zone activation was found to develop more quickly if the same saccade vector (including the special case of fixation) does not have simultaneous strongly active representations in different zones. This rate effect occurs because weight change is a function of activation level. Although not necessary for learning as such, the learning speed advantage warrants the hypothesis that competition exists between like vectors in different zones. Moreover, a basis exists in the model for self-organization of such vector-specific competition based on a ‘fire-together, wiretogether’ principle: As shown in Eqs. (15) and (18) below, all plan cells coding for vector ðx; yÞ receive a common excitatory input ksfp ½Sxy 2 0:25þ from SC zone ðx; yÞ and a ðXÞ common inhibitory input Fxy from FEF post-saccadic cells excited by SC zone ðx; yÞ: For these reasons, the model presented here includes between-zone competition between cells coding for the same vector. However, different saccade vector plans may simultaneously be highly active in different zones, to provide alternative plans from which to choose. ðPÞ Activity Fxyi of FEF plan layer cell xyi was defined by d ðPÞ ðPÞ ðPEÞ ðPÞ ðPIÞ F ¼ 500½ð1 2 Fxyi ÞFxyi 2 ðFxyi þ 0:4ÞFxyi dt xyi ð14Þ J.W. Brown et al. / Neural Networks 17 (2004) 471–510 ðPEÞ where Fxyi is the total excitatory input to a plan layer cell, ðPIÞ and Fxyi is the total inhibitory input. The excitatory input ðPEÞ Fxyi of (14) was defined by ðPEÞ ðIÞ ðPÞ Fxyi ¼ Fxyi þ 0:025qðFiðGÞ ; 0:15Þ þ kfpr f ðPÞ ðFxyi Þ X ðTPÞ þ 0:1 j Wjxyi Tj þ ksfp ½Sxy 2 0:25þ : qðx; aÞ ¼ 0 if x , a x if x $ a : ð½xþ Þ8 : ð0:5Þ8 þ ð½xþ Þ8 m–i k ð15Þ þ5 X ðPÞ f ðPÞ ðF11m ÞÞ: X ðOÞ ½Fpqk 2 0:6þ ðp;qÞ–ð1;1Þ ð18Þ m–i ð16Þ It allows background noise to be filtered out while even small suprathreshold signals can generate robust activity. Parameter kfpr of (15) scales the FEF plan layer ðPÞ recurrent excitation, f ðPÞ ðFxyi Þ; which is restricted to the cell of origin. The sigmoid signal function f ðPÞ in (15) was defined by f ðPÞ ðxÞ ¼ ðPIÞ of (14) were defined by: The inhibitory inputs Fxyi X ðPIÞ ðPÞ ðXÞ ðPÞ ¼ 0:06 þ 5Fxy þ kfpri ð f ðPÞ ðFxym ÞÞ þ kfpi ðfjðWÞ ðFxyi Þ Fxyi X ðPÞ ÞÞ þ kfop ð 2 f ðPÞ ðFxyi ðIÞ from the FEF input layer was defined in (12). Input Fxyi ðGÞ Term Fi is input from layer VI, defined by (13). The threshold signal function qðx; aÞ that transforms FiðGÞ was defined by ( 487 ð17Þ ðPÞ With Fxyi as its argument, the steep sigmoid specified by (17) ensures that recurrent excitation within the plan layer can result in self-sustaining activity when a plan layer cell’s activity exceeds 0.5 due to inputs from layer VI, ITa, and/or SC. Term Tj in (15) is input from ITa that is defined by ðTPÞ (5). It is multiplied by the adaptive synaptic weight Wjxyi that is defined by (56). This weight is non-zero only for projections to FEF What – Where (not Where –Where) GCZs. This link allows a category representation in ITa to learn to activate a saccadic plan. The term Sxy (defined by (11)) is from the SC (Lynch, Hoover, & Strick, 1994). The rectified signal ½Sxy 2 0:25þ grows linearly with Sxy above the signal threshold 0.25 but is zero for all Sxy below 0.25. Parameter ksfp is the SC-to-FEF plan layer synaptic strength. Fixation-related plan cells are those cells with Cartesian coordinates ðx; yÞ ¼ ð1; 1Þ: There were three such cells, one in each FEF zone. For saccade-related cells, kfpr ¼ 0:18; ksfp ¼ 8; for fixation-related cells, kfpr ¼ 0:0; ksfp ¼ 0:0: Thus saccade-related, but not fixation-related, cells can receive movement-related corollary discharges from the SC and exhibit selfsustaining recurrent excitation. ðXÞ Here Fxy is inhibition from FEF postsaccadic cells, defined below by (26). The parameter kfpri scales FEF plan layer recurrent inhibition from same-direction saccaderelated cells in other zones. Recurrent inhibition prevents multiple representations of the same saccade vector from being active, to prevent interference between different GCZs while strategies are learned. For plan cells in the fixation and target feature zones, which can mediate Where – Where transformations, kfpri ¼ 0:1: However, kfpri ¼ 0:0 for cells in the What – Where zone, GCZo: Since this zone lacks the positional inputs of the Where– Where zones, interference during learning will not occur. Of the final two terms in Eq. (18), only one is non-zero for any single cell, because the model’s saccade- and fixationrelated plan cells have different patterns of inhibitory afferents. The parameter kfpi scales recurrent inhibition from saccade-related cells to saccade-related cells tuned to different directions, indexed by the coordinates ðx; yÞ: The parameter kfop scales the FEF output layer and plan layer inhibition to fixation-related plan cells. Thus, for saccaderelated cells, kfpi ¼ 0:1; but kfop ¼ 0:0: For fixation-related cells, kfpi ¼ 0:0; but kfop ¼ 1:0: The summation index k denotes the zone ðk ¼ 1; 2; 3Þ of cells representing motor error vectors ðp; qÞ: In the fourth term of (18), the lateral inhibitory signal, ðPÞ fjðWÞ ðFxyi Þ; takes a different form for Where – Where (indexed by j ¼ 1) and What –Where (indexed by j ¼ 2) cells. Although Where– Where plans encoding the same saccade vector inhibit each other across zones, Where– Where plan cells do not inhibit What – Where plan cells. Lateral inhibition from saccade-related plan cells to saccade-related plan cells in either Where –Where zone, was defined by X X ðPÞ ðPÞ f1ðWÞ ðFxyi Þ¼ f ðPÞ ðFpqk Þ; ð19Þ k[V ðp;qÞ–ð1;1Þ where V is the set of all plan layer cells in the Where– Where zones, for which index k ¼ 0 and 1. (More generally, the indices i and k refer to zones in the cortex and BG, respectively. Use of separate indices allows computation of the interaction among zones, e.g. in Eq. (52)). ðPÞ ðPÞ In (18), the subtraction of f ðPÞ ðFxyi Þ from f1ðWÞ ðFxyi Þ fpr ðPÞ ðPÞ excludes self-inhibition. Excitatory term k f ðFxyi Þ of ðPÞ ðPÞ (15) and inhibitory term kfpi ðf1ðWÞ ðFxyi Þ 2 f ðPÞ ðFxyi ÞÞ of (18) together define a recurrent on-center, off-surround network, 488 J.W. Brown et al. / Neural Networks 17 (2004) 471–510 which acts to normalize and contrast-enhance network activity (Grossberg, 1973). For cells in the What –Where zone, kfpri ¼ 0:0 in (18), ðPÞ and the lateral inhibitory signal f2ðWÞ ðFxyi Þ was X ðPÞ ðPÞ Þ¼ f ðPÞ ðFpqi Þ: ð20Þ f2ðWÞ ðFxyi ðp;qÞ–ð1;1Þ Lateral inhibition came only from saccade-related cells of the same zone. Model FEF output cells correspond to FEF movement or presaccadic cells, and are predicted to reside in layer Vb; see Figs. 2 and 6a. They are divided into fixationðOÞ and saccade-related categories. Their activity Fxyi was defined by: d ðOÞ ðOÞ ðOEÞ ðGÞ ðOÞ ðOIÞ F ¼ 125½ð1 2 Fxyi ÞFxyi Fi 2 ðFxyi þ 0:6ÞFxyi ; dt xyi ð21Þ ðOEÞ ðOIÞ excites the output cells and Fxyi inhibits where Fxyi ðGÞ them. The arousal signal Fi from layer VI, defined by ðOEÞ (13), multiplies Fxyi ; which was defined by: ðOEÞ Fxyi ¼ 1:5½Vk 2 0:5 þ ðPÞ þ 0:4qðFxyi ;0:2Þ þ ksfo ½Sxy 2 0:4þ : ð22Þ In (22), 1:5½Vk 2 0:5þ is the thalamocortical input; see (38). It conveys a GO signal from the BG if thalamic activity Vk exceeds 0.5. Subscript k of Vk has the same ðOEÞ range (1 – 3) as i of Fxyi because there is a one-to-one relationship between FEF zones and associated thalamic sites controlled by distinct BG channels. Signal function ðPÞ q(x,a) was defined in (16). Input Fxyi is from the plan ðOEÞ layer and Sxy is from the SC. The product of Fxyi and ðGÞ Fi in (21) makes output contingent on a conjunction of arousal with plan layer, thalamocortical, and SC inputs. In particular, corollary discharges from the SC cannot activate output layer cells whose corresponding zone is not active. Parameter ksfo in (22) is the strength of the SC’s projection to the FEF output layer. For saccaderelated cells, ksfo ¼ 10; for fixation-related cells, ksfo ¼ 0: Therefore, a corollary discharge from the SC can excite saccade-related but not fixation-related cells. ðOIÞ The inhibitory input Fxyi in (21) differs for fixationrelated and saccade-related output cells. The latter are inhibited by both postsaccadic cells and output layer cells that are tuned to other saccadic vectors. The inhibitory ðOIÞ signals Fxyi to saccade-related FEF output cells, for which ðx; yÞ – ð1; 1Þ; were defined by: X X ðOIÞ ðOÞ ðOÞ ðXÞ ¼ 0:3 þ 6Fxy þ f ðOÞ ðFpqk Þ 2 f ðOÞ ðFxyi Þ; Fxyi k ðp;qÞ–ð1;1Þ ð23Þ ðXÞ where Fxy is input from postsaccadic cells defined by (26), the summation index k denotes zone, and the output layer signal function f ðOÞ was a sigmoid: f ðOÞ ðxÞ ¼ ð½xþ Þ10 : ð½xþ Þ10 þ 0:410 ð24Þ Because of the steep shape of the sigmoid in (24), output layer cell activity must exceed 0.4 to inhibit other output layer cells. The high threshold for competition among FEF output cells creates an interval—before any cells exceed this threshold on the basis of dual inputs from the plan layer and the thalamus—during which FEF output cells can be transiently activated by visually driven signals arriving via the plan layer. Such a transient visual response is seen in the presaccadic movement cell shown in Fig. 9D. The inhibitory signal to fixation-related FEF output cells was defined by X X ðOIÞ ðOÞ ðOÞ ðXÞ F11i ¼0:3 þ 6Fxy þ f ðOÞ ðFpqk Þ 2 f ðOÞ ðF11i Þ k þ 10 X X k ðp;qÞ–ð1;1Þ ðp;qÞ¼ð1;1Þ ðOÞ ½Fpqk 2 0:6þ : ð25Þ Again, summation index k denotes zone. The final term in (25) allows saccade-related FEF output cells to inhibit fixation-related output cells via inhibitory interneurons. Fixation-related plan and output cell activities, specified in Eqs. (14) – (18) and (21,22, 25), respectively, begin to shut off immediately prior to a saccade due to the inhibition from saccade-related FEF plan and output cells. ðXÞ FEF postsaccadic cell activities Fxy were defined by: d ðXÞ ðXÞ ðXÞ F ¼ 500ð1 2 Fxy Þ½Sxy 2 0:6þ 2 10Fxy : dt xy ð26Þ Postsaccadic cells delete executed plans; see Eq. (18). They are excited by the corresponding SC cell activities Sxy; see Eq. (11). The SC signal was modeled by thresholding the SC activity. The 0.6 threshold for exciting them is also the threshold of saccade initiation in the SC. 6. Model BG and PNR-THAL cells Cortical afferents to the striatum of the BG include projections from inferotemporal cortex (Hoesen, Yeterian, & Lavizzo-Mourey, 1981; Steele & Weller, 1993), parietal cortex (Cavada & Goldman-Rakic, 1991), and frontal cortex (Parthasarathy, Schall, & Graybiel, 1992; Strick et al., 1995). All three classes of afferents project to the BG model. BG components that ultimately project to the SC are denoted by the symbol G, and those that ultimately return to frontal cortex via the PNRTHAL are denoted by the symbol B. The model coordinate systems are also distinct because the thalamus-projecting population codes a zone’s choice, J.W. Brown et al. / Neural Networks 17 (2004) 471–510 while the colliculus-projecting population codes motor error. The SC-projecting striatal SPN direct pathway activity GðSDÞ (Figs. 2 and 6b) was defined by: xy d ðSDÞ ðSDEÞ ðSDIÞ G ¼ 30½ð1 2 GðSDÞ 2 ðGðSDÞ xy ÞGxy xy þ 0:58ÞGxy ; dt xy ð27Þ where the excitatory input is GðSDEÞ ; and the inhibitory input xy is GðSDIÞ xy : The excitatory signal was defined by: X ðOÞ GðSDEÞ ¼ N þ 75½Pxy 2 0:25þ þ 100 f ðOÞ ðFxyi Þ; ð28Þ xy i where N refers to a dopamine burst (Ljungberg, Apicella, & Schultz, 1992), which can activate the striatum and modulate corticostriatal reinforcement learning (Wickens et al., 1996). Signals from parietal cortex ½Pxy 2 0:25þ ; and ðOÞ FEF output cells, f ðOÞ ðFxyi Þ; promote opening of the BG-toSC gate to allow execution of the plans that they represent. The model assumes that the excitatory inputs are strong but specific: they are pooled only across parietal and FEF sites that represent the same motor vector. In contrast, the inhibitory inputs are convergent: they are pooled across parietal and FEF sites regardless of the motor error vector represented. The inhibitory input GðSDIÞ was defined by: xy X X ðOÞ ðOÞ ðSDIÞ þ Gxy ¼ 1 þ 20ð ½Ppq 2 0:25 þ f ðFpqi ÞÞ; ð29Þ pq pqi which models fast-acting feedforward striatal inhibitory interneurons (Koos & Tepper, 1999; Wilson et al., 1989) that are excited by convergent parietal cortex and FEF outputs from all positions ðp; qÞ and zones i: Because of the relative strengths of the inputs in (28) and (29), a planrepresenting saccade vector ðx; yÞ can always activate the striatum if the main source of inhibition in the summations of (29) comes from cells at grid coordinate ðx; yÞ: This condition will be met when the cortical competition resolves in favor of one winning plan. Otherwise, inhibition can overwhelm excitation; see Fig. 3. Also noteworthy is that Eq. (29) does not include feedback inhibition from SPNs. This is consistent with electrophysiological data of Jaeger et al. (1994) indicating that such inhibition is weak. The theory does not, however, preclude a role for feedback inhibition; see Eq. (41). Colliculus-controlling striatal SPN activity GðSIÞ xy (Figs. 2 and 6b) was defined by: d ðSIÞ ðSIÞ G ¼ 30½ð1 2 GðSIÞ ð30Þ xy Þ10NSxy 2 ðGxy þ 0:58Þ; dt xy where N corresponds to a transient depression in the dopamine signal (a dopamine dip), as described by Ljungberg et al. (1992). The excitatory action of N is attributable to reduced binding of inhibitory D2 receptors on indirect path SPNs. Excitatory input, Sxy ; from the SC informs the indirect channel that the SC region that is gated by the corresponding direct channel is being activated. 489 As noted for the analogous case of thalamo-striatal feedback in hypothesis 7, such an input to the indirect channel can serve as a training signal for other excitatory afferents to SPNs; see Eq. (34). However, no such afferents were needed to SPNs represented by Eq. (30) in the present simulations. The constant term (0.58) of Eq. (30) specifies tonic inhibition. (Figs. 2 Colliculus-controlling GPe activity GðGPeÞ xy and 6b) was defined by: d ðGPeÞ G ¼ 30½0:5ð1 2 GðGPeÞ Þ xy dt xy þ 2 ðGðGPeÞ þ 1Þð0:2 þ 0:8½GðSIÞ xy xy Þ: ð31Þ The first term of (31) represents a combination of intrinsic excitation and tonic excitation from the STN. Input GðSIÞ from the striatal indirect pathway SPN provides xy inhibition. SNr activity GðSNrÞ (Figs. 2 and 6b) was defined by: xy d ðSNrÞ ðSNrÞ þ G ¼ 100ð1 2 GðSNrÞ þ 1Þð54½GðSDÞ xy Þ 2 ðGxy xy dt xy þ 80½GðGPeÞ þ Þ: xy ð32Þ The first term of (32) represents a combination of intrinsic excitation and tonic excitation from the STN. Both ðGPeÞ Þ the striatal direct pathway SPN ðGðSDÞ xy Þ and the GPe ðGxy provide inhibition. The thalamus-projecting BG system differs from the SCprojecting system in having adaptive cortico-striatal pathways. The thalamus-projecting striatal SPN direct pathway activity BðSDÞ (Figs. 2 and 6b) was defined by: k X ðPÞ d ðSDÞ ðPSDÞ Bk ¼50½ð1 2 BðSDÞ Þð70 ½Fxyi 2 0:33þ Wxyik k dt xyi X þ 8 Tj WjkðTSDÞ þ N þ 2½Vk þ Þ j þ 0:58Þð1 þ 1:17ð 2 ðBðSDÞ k X xyi ðPÞ Fxyi þ2 X Vi ÞÞ: i ð33Þ Subscript k ranges from 1 to 3 to denote the three thalamus-controlling BG channels associated with the three ðPÞ FEF zones. The FEF plan layer activities Fxyi ; defined in (14), and the anterior IT activities Tj ; defined in (5), provide learned excitation to striatal direct pathway SPNs. The ðPSDÞ adaptive weight Wxyik is defined by (53), and the adaptive ðTSDÞ weight Wjk is defined by (54). A dopamine burst N provides a dopamine-mediated enhancement of striatal activity via D1 receptors. Thalamic inputs Vk provide excitatory feedback that can help train FEF and ITa inputs to activate the striatal direct pathway. Activity in the FEF plan layer and thalamus also use feedforward striatal inhibitory 490 J.W. Brown et al. / Neural Networks 17 (2004) 471–510 interneurons (the last two terms in (33)) to inhibit striatal direct pathway activity, as in (27) and (29). Striatal indirect pathway SPN activity BðSIÞ (Figs. 2 k and 6b) was defined by: X ðPSIÞ ðPÞ d ðSIÞ k þ Þ Bk ¼30½ð12BðSIÞ Wxyik ½Fxyi 20:33þ þ10N½V k Þ70 dt xyi 2ðBðSIÞ k þ0:58Þð1þ0:17 X ðPÞ þ ½Fxyi Þ: ð34Þ xyi ðPÞ Learned excitatory inputs from the FEF plan layer Fxyi ðPSIÞ are adaptively weighted by the Wxyik and add with excitatory input Vk from the thalamus that is enhanced when a dip ðN.0Þ occurs in the dopaminergic input, which acts on D2 receptors. Indirect pathway SPNs are also suppressed by feedforward inhibitory interneurons, which ðPÞ are driven by FEF plan layer inputs Fxyi : The adaptive ðPSIÞ weights Wxyik in (34) are defined by (52). The SNr/GPi receives input from the STN, the striatal direct pathway projections, and the GPe. The SNr/GPi activity BðGPiÞ (Figs. 2 and 6b) was defined by: k d ðGPiÞ B ¼100½ð1 2 BðGPiÞ Þð0:77 þ 2½BðSTNÞ þ Þ k dt k 2 ðBðGPiÞ þ 1Þð0:54½BðSDÞ þ þ 0:8½BðGPeÞ þ Þ: k k k ð35Þ With ðp;qÞ–ð1;1Þ; the summation index spans inputs from saccade-related, but not fixation-related, cells. STN hyperactivity in Parkinson’s disease has, in the past (Wichmann & DeLong, 1996), been attributed to GPe hypoactivity and the subsequent loss of GPe-STN inhibition. This argument has supported the prevailing model of BG function in which the GPe-STN projection is an essential link in the indirect pathway. However, Levy et al. (1997) reviewed evidence that GPe activity is not significantly reduced in Parkinsonism, suggesting that loss of GPe-STN inhibition may not be sufficient to account for STN hyperactivity in Parkinson’s disease. Moreover, dopaminergic lesions directly in the STN, which contains D1 and D2 receptors (Brown et al., 1979; Martres, Bouthenet, Sales, Sokoloff, & Schwartz, 1985), result in five times the STN hyperactivity induced by GPe lesions alone (Hassani et al., 1996), although the precise mechanism for this is not clear. Therefore we have modeled the GPe-STN projection as weak relative to the GPe-GPi/SNr projection (Parent & Hazrati, 1995; Smith & Bolam, 1990). Activities Vk in the PNR-THAL, e.g. VA and VLo, are excited by afferents from cortical layer VI (the category layer, FkðGÞ ) and inhibited by the GPi ðBðGPiÞ Þ: Thalamic k activity Vk (Figs. 2 and 6b) was defined by: d V ¼400½ð1 2 Vk Þð0:5½FkðGÞ 2 0:47þ þ f ðVÞ ðVk Þ þ vðtonicÞ Þ dt k 2 ðVk þ 0:1Þ½0:2 þ 3:4½BðGPiÞ 2 0:2þ k By (35), the GPi receives tonic excitation and glutamatergic excitation BðSTNÞ from the STN, and is inhibited by striatal direct pathway SPN outputs BðSDÞ and GPe outputs k BðGPeÞ : In the case of the oculomotor system, the SNr rather k than GPi projects to the PNR-THAL, which projects to FEF. These SNr projections are distinct from SNr-to-SC projections modeled by GðSNrÞ in Eq. (32) above (Kemel, Desban, xy Gauchy, Glowinski, & Besson, 1988). The GPe activity BðGPeÞ (Figs. 2 and 6b) is excited by the k STN output BðSTNÞ and inhibited by the striatal indirect path SPN outputs BðSIÞ : k d ðGPeÞ B ¼ 30½ð1 2 BðGPeÞ Þð0:46 þ 0:25BðSTNÞ Þ k dt k þ 2 ðBðGPeÞ þ 1Þð0:2 þ 0:8½BðSIÞ k k Þ: ð36Þ The STN activity B (STN ) (Figs. 2 and 6b) is excited by afferents from layer Vb (the output layer, F ðOÞ ) of the FEF and inhibited by the GPe outputs BðGPeÞ : k X X d ðSTNÞ ðOÞ B ¼25½ð12BðSTNÞ Þð0:016þ10 ½Fpqi 20:5þ Þ dt i ðp;qÞ–ð1;1Þ 20:1BðSTNÞ X k ½BðGPeÞ þ : k ð37Þ þ X f ðVÞ ðVi ÞViðXÞ Þ; ð38Þ i–k where the feedback signal function f ðVÞ was defined by f ðVÞ ¼ ð½xþ Þ2 : ð39Þ The term vðtonicÞ is 0.0 if the thalamic cell projects to a Where– Where FEF zone (i.e. k , 2) and 0.1 if the thalamic cell projects to a What –Where FEF zone (i.e. k ¼ 2). This small tonic level biases the system to use What – Where FEF plans initially during learning. For example, in learning the delay task, the model first attempts a saccade to the target that was learned in the gap task. This target is recalled via a learned What –Where transformation. Subsequent punishment trains the model to suppress the response learned in the gap task. Thalamic recurrent inhibition, which is well established in the visual thalamus (Lo & Sherman, 1994), mediates a local winner-take-all competition that allows only the thalamocortical projection of one FEF zone at a time to generate output. Since lateral inhibition within a zone in the plan layer allows only one plan to be active per zone, this helps ensure that multiple conflicting plans are not simultaneously executed. In the thalamus, newly formed J.W. Brown et al. / Neural Networks 17 (2004) 471–510 plans can shut off a prior plan and instate themselves as current winner if the ability of the prior winner to maintain itself wanes through time. This will occur if selfexcitation and/or recurrent surround inhibition use habituative signals. The current implementation uses only the latter. Thus, a transmitter level, symbolized VkðXÞ in (38) and defined in (40), is assumed to mediate GABA-ergic lateral inhibition and to habituate (inactivate, or depress) with continued firing. As a result, a new plan can be instated and can effectively inhibit a previously active plan whose inhibitory signals have habituated. Thalamic habituative transmitter VkðXÞ was defined (cf., Grossberg, 1982) by: d ðXÞ V ¼ 0:25ð1 2 VkðXÞ Þ 2 12:5VkðXÞ ½Vk 2 0:4þ ; dt k ð40Þ where VkðXÞ habituates via term VkðXÞ ½Vk 2 0:4þ if activity Vk . 0:4; but accumulates to a baseline of 1.0 via term ð1 2 VkðXÞ Þ when the thalamic signal Vk is inactive. In vivo, this habituative dynamic may be attributable to short-term depression of excitatory synapses on inhibitory interneurons of the thalamus. 7. Second messenger traces and working memory equations Reward signals subserving reinforcement learning arise after an action has been generated, with a delay of hundreds of milliseconds or even seconds. However, movement-related cell discharges shut off rapidly after a movement is initiated (Figs. 9C and F). Thus, reinforcement learning signals must modify synapses for which the pre- and post-synaptic cells were previously, but are no longer, active. In order for credit or blame to be properly assigned to synapses on non-discharging cells that previously helped activate a movement, a record or trace of recent cell activity persists after spiking ceases (Dominey et al., 1995; Fiala, Grossberg, & Bullock, 1996; Grossberg & Merrill, 1992; Grossberg & Schmajuk, 1987; Houk, Adams, & Barto, 1995; Klopf, 1982; Sutton & Barto, 1981). Such traces can take the form of intracellular second messengers, which are critical for some aspects of learning in many plastic brain areas (Hemart, Daniel, Jaillard, & Crepel, 1995; Otani, Auclair, Desce, Roisin, & Crepel, 1999; Sung, Choi, & Lovinger, 2001; Vezina & Kim, 1999). Other options, such as working memory representations of previous movements, occur in situations where the cell activation can itself be sustained over the interstimulus interval. Trace variables that embody these proposed second messenger effects were introduced in the BG as follows. To support reinforcement learning for awhile after direct pathway activity ceases, a trace, or transient record, of recent striatal direct pathway activity was carried by second 491 messenger activity B ðSDÞ (Fig. 6b): k d ðSDÞ B ¼750ð1 2 B ðSDÞ Þ½BðSDÞ 2 0:4þ k k dt k X 2 B ðSDÞ ð0:75 þ 75 f ðBÞ ðBðSDÞ ÞÞ: i k ð41Þ k–i The threshold 0.4 ensures that the trace forms only if is sufficiently large. The trace B ðSDÞ influences learning BðSDÞ k k ðPSDÞ of the weights Wxyik in (33); see (53). The final term in (41) formalizes the assumption that GABA-ergic recurrent collaterals of striatal direct pathway projection neurons suppress second messenger-based records of activity within neighboring striatal projection neurons. This hypothesis may explain why collaterals of striatal SPNs exist despite their apparent failure to produce potent lateral inhibition (Jaeger et al., 1994). The inhibitory feedback signal function f ðBÞ in (41) was defined by: f ðBÞ ¼ ð½xþ Þ2 : ð42Þ ðPSIÞ Wxyik Learning by the adaptive weights in (34) that impinge on indirect pathway cells suppress a negatively reinforced response. Second messenger activity B ðSILÞ serves k as a trace of thalamic input Vk to striatal indirect channel ðPSIÞ SPNs, and determines whether the weights Wxyik from the FEF plan layer to these SPNs are eligible to learn, should negative reinforcement occur; see (52): d ðSILÞ ¼750ð1 2 B ðSILÞ Þ½Vk 2 0:5þ B k dt k X 2 B ðSILÞ ð0:75 þ 75 f ðVÞ ðVi ÞViðXÞ Þ: k ð43Þ i–k Here Vk is thalamic activity defined by (38), and ViðXÞ is transmitter that habituates with continued firing, as defined in (40). The signal function f ðVÞ was defined by f ðVÞ ¼ ð½xþ Þ2 : ð44Þ The second term in (43) models the hypothesis that the trace is suppressed in neighboring cells via habituative feedforward inhibition from striatal interneurons (Kawaguchi, 1997; Kawaguchi, Wilson, Augood, & Emson, 1995) that are excited by thalamic signals Vk : Simulations showed that this striatal habituation to Vk could be identical to the thalamic habituation to Vk ; which was already modeled in (38) and (40). For convenience, the same variable, ViðXÞ ; was therefore used in both (38) and (43). A trace of FEF plan layer activity F ðPÞ xyi (Fig. 6a) was specified by: d ðPÞ ðPÞ F ¼ 15ð1 2 F ðPÞ xyi Þ½Fxyi 2 G dt xyi 2 0:75F ðPÞ xyi : pt þ ð45Þ Eq. (45) compactly describes the dynamics of two physically distinct traces of FEF plan cell activity. One of these traces modulates learning in anterior IT projections to the plan layer (see Eq. (56)). It is naturally interpreted 492 J.W. Brown et al. / Neural Networks 17 (2004) 471–510 as a process localized in the ITa-recipient dendrites of FEF cortico-striatal cells. The second trace (whose dynamics are also described by (45)) modulates learning (Berretta et al., 1997) in the plan layer projection to the striatal direct pathway (see Eq. (53)). It is naturally interpreted as a process in the axon terminals of projections from FEF plan layer cells to striatal direct pathway SPNs. For Where –Where cells, G pt ¼ 0:4; for What – Where cells, G pt ¼ 0:15 to compensate for weaker What – Where plan cell activity by reducing the threshold for trace activation. A plan layer cell is prevented from learning to activate those striatal indirect-pathway SPNs that can prevent the same plan from being executed. Otherwise, when a dopamine dip occurred, a plan could learn to prevent itself from ever being executed (a kind of learned helplessness). However, plans can block each other’s execution via learned FEF-to-indirect pathway links. The model accomplishes these goals using traces within plan layer cells that gate plan-layer to indirect-pathway learning. These traces create eligibility only for recently active but unexecuted plans because plan execution generates feedback, via model GABA-ergic FEF postsaccadic cells, that resets activities and traces only within plan and output cells responsible for execution of the current response. This trace of plan layer activity, F ðPAÞ xyi ; (Fig. 6a) was defined by: d ðPAÞ ðPÞ pt þ ðXÞ ðPAÞ F ¼15ð12 F ðPAÞ xyi Þ½Fxyi 2 G 2 Fxyi ½0:75þ75Fxy : dt xyi ð46Þ Like the trace in (45), this trace is activated only if FEF ðPÞ plan layer activity Fxyi ; which is defined in (14), satisfies ðPÞ Fxyi . G pt : However, unlike other trace variables in the model, which decay slowly once their source cell’s activity falls below G pt ; this trace collapses quickly when its source cell is shut off, notably when an FEF post-saccadic cell ðXÞ generates an inhibitory signal Fxy ; defined in (26). For convenience, the dependence on this inhibitory event is shown directly in the final term of Eq. (46), where a ðPÞ function that became large when Fxyi nears zero would also work. This captures the hypothesis that a small activity in the source cell is necessary to prevent rapid collapse of this type of trace. This trace is interpreted as a process in cortico-striatal axon terminals that abut dendrites of indirect pathway SPNs. Consistent with this proposal, Berretta et al. (1997) showed that applying a GABA agonist in motor cortex prevents learning-related gene expression in projections to the striatal indirect, but not direct, pathways. The model captures this functional difference by introducing the dependence on cortical inhibition in (46) but not in (45). A trace of layer VI activity F ðGÞ (Fig. 6a) was used to i support reinforcement learning, after movement completion, of adaptive weights between PFC context representations and FEF zones that can be primed by PFC inputs to layer VI. This trace variable was defined by: d ðGÞ ðGÞ 2 0:5þ F ¼750ð1 2 F ðGÞ i Þ½Fi dt i X ðGÞ ðGÞ 2 F ðGÞ f ðFk ÞÞ: i ð0:75 þ 75 ð47Þ k–i The inhibitory summation term in (47) represents lateral inhibition among layer VI cells. The inhibitory signal function f ðGÞ in (47) was defined by f ðGÞ ¼ ð½xþ Þ4 : ð48Þ Eqs. (47) and (48) ensure that a synapse between a PFC fiber and a layer VI cell will remain eligible to learn until the trace either passively decays or is actively shut off by competitive activity in a neighboring layer VI cell. A trace of ITa activity T j (Fig. 6a) was defined by d T ¼ 750ð1 2 T j Þ½Tj 2 0:4þ 2 0:75T j dt j ð49Þ and may be realized by second messenger activity within terminals of anterior IT cell synapses with FEF plan cells. Although the What pathway trace in (49) facilitates learning, it does not control performance. Working memory areas in the PFC provide a basis for persistent activation of contextual representations that is useful for controlling performance, specifically which premotor zones are set in an active or primed state. As noted in (47), model PFC What representations learn to activate layer VI cells via longrange cortico-cortical projections. PFC What working memory activities Cj (Figs. 2 and 6a) were defined by: d C ¼ 30½ð1 2 Cj Þð1:5I ðMÞ þ Tj þ 4f ðCÞ ðCj ÞÞ 2 ðCj þ 0:3ÞÞ1 dt j X þ 0:35 f ðCÞ ðCk ÞÞÞ: ð50Þ k–j Motivational input I ðMÞ is either 0 or 1. Model PFC representations require such an input I ðMÞ ; assumed to be from limbic areas, in order to become aroused. Then a specific ITa signal Tj can activate a working memory representation as follows. Positive feedback 4f ðCÞ ðCj Þ via the sigmoidal signal function f ðCÞ ; f ðCÞ ¼ x8 x8 þ 0:68 ; ð51Þ P is balanced by recurrent inhibition 0:35 k–j f ðCÞ ðCk Þ from other cells. By (51), Cj activity must exceed approximately 0.6 for recurrent excitation to engage and maintain Cj in a self-sustaining regime. Input Tj from ITa cannot activate working memory alone, because it cannot drive activity Cj above 0.6. However, when motivational input I ðMÞ is active, the sum of the nonspecific I ðMÞ and the specific input Tj can drive one or multiple prefrontal cells Cj above 0.6 and into the selfsustaining regime; cf. Grossberg (1971, 1982b). Once self-sustaining, Cj persists only as long as at least one of J.W. Brown et al. / Neural Networks 17 (2004) 471–510 I ðMÞ and Tj is active. In a more complete model, sustained activity in PFC might also depend on reciprocal excitation between frontal and parietal cortices (Chafee & Goldman-Rakic, 2000). The anterior IT-to-striatal direct pathway weights WjkðTSDÞ (Fig. 6b) in (33) were defined by: d ðTSDÞ jkðTSDÞ Þ W ¼ðN½T j 2 WjkðTSDÞ þ 2 0:1ðN þ NÞW dt jk qðB ðSDÞ ; 0:5Þ: k 8. Adaptive weight equations Five similar learning equations describe the functional dependence of adaptive weight changes on the intracellular traces of recent activity defined above. Learning of the planðPSIÞ to-indirect pathway weights Wxyik (Fig. 6b) in (34) was defined by d ðPSIÞ W ¼ N½500qð B ðSILÞ ; 0:35Þ½qðF ðPAÞ xyi ; 0:5Þ k dt xyik ðPSIÞ þ ðPSIÞ 2 Wxyik 2 Wxyik : ð52Þ generated by the A negative reinforcement signal N; omission of primary reward at the expected time during training (Brown et al., 1999), allows recently active but unexecuted plans, which are marked by plan layer traces F ðPAÞ xyi of (46), to learn to suppress the previously selected ðPSIÞ zone with synaptic efficacy Wxyik : The zone is suppressed via activity in striatal indirect pathway SPNs, whose signals B ðSILÞ ; defined in (43), are still on k when signal N arrives. The signal function q of Eq. (16) in learning rules mimics the voltage-dependent effect of NMDA channel blocking with magnesium (Mayer, Westbrook, & Guthrie, 1984). Noise below the threshold is filtered out, while even a small activity above the threshold causes a robust learning response. ðPSDÞ Learning of the plan-to-direct pathway weights Wxyik (Fig. 6b) in (33) was defined by: d ðPSDÞ ðPSDÞ þ W ¼ðN½F ðPÞ xyi 2 Wxyik dt xyik ðPSDÞ xyik 2 0:1ðN þ NÞW ÞqðB ðSDÞ ; 0:5Þ: k Expansion yields two products on the right hand side. The first product specifies a process of LTP: weights increase toward F ðPÞ when a dopamine burst N coincides with recent presynaptic activity F ðPÞ from the plan layer and recent postsynaptic activity B ðSDÞ : The second k product specifies a process of LTD: weights decrease slowly when a dopamine burst or dip (N or N; respectively) coincides with recent postsynaptic activity B ðSDÞ : that occurs in the absence of recent k presynaptic activity. Thus localized FEF planning activities become potent exciters of the striatal direct pathway if they predict striatal activations leading to forthcoming rewards, but they can lose this potency if they predict striatal activations leading to punishment, or if reward is forthcoming in their absence. ð54Þ Here positive reinforcement N allows the traces T j of ITa representations in (49), in conjunction with striatal direct pathway SPN trace activity B ðSDÞ ; to drive learning in the k adaptive weights WjkðTSDÞ : Both positive ðNÞ and negative ðNÞ reinforcement allow weight decay in response to irrelevant stimuli, whose ITa representations are inactive when the direct pathway and reinforcement are active. By selectively augmenting only recently active and positively reinforced inputs, the learning rules in Eqs. (53) and (54) prevent irrelevant stimuli and plans from learning to open the BG gate. The PFC-to-FEF layer VI weights WkiðCGÞ (Fig. 6a) in (13) were governed by d ðCGÞ jiðCGÞ Þ W ¼ð500N½qðCj ;0:5Þ 2 WjiðCGÞ þ 2 0:1ðN þ NÞW dt ji £ qðF ðGÞ i ;0:5Þ: ð55Þ By (13), this weight can enable the PFC working memory to learn to activate the appropriate zone in layer VI, and by (55) the weight grows when the working memory activity Cj and layer VI trace F ðGÞ i are both active while a reinforcement learning signal N is active. The weight declines when F ðGÞ i is active without Cj ; or if Cj predicts punishment. The anterior IT-to-FEF plan layer (What – Where) ðTPÞ weights Wjxyi (Fig. 6a) in (15) were defined by: d ðTPÞ ðTPÞ þ W ¼ð500N½F ðPÞ xyi 2 Wjxyi dt jxyi ðTPÞ jxyi 2 0:1ðN þ NÞW ÞqðT j ; 0:5Þ: ð53Þ 493 ð56Þ Here active ITa traces T j ; and active plan layer traces F ðPÞ xyi ðTPÞ together drive learning of adaptive weights Wjxyi when reinforcement N occurs. The ITa projection reaches only What-to-Where zone cells in the FEF plan layer, in keeping with the topographical distribution of ventral and dorsal stream afferents to FEF (Schall et al., 1995a). In other ðTPÞ words, Wjxyi ¼ 0 if i corresponds to a Where-to-Where plan layer cell. Eqs. (55) and (56) explicate hypothesis 12, that DA-ergic signals influence cortical reinforcement learning and plasticity in a manner similar to the way they influence reinforcement learning in the BG. Simulations. Details regarding simulated inputs, of visual stimuli and reward signals, are provided in the Results section. The simulations were implemented with a Cþ þ program that used the fourth order Runge-Kutta method for integrating the system of differential equations. Parameters. In many cases, parameters were found for a given equation by specifying several equilibrium values of the governed variable and specific corresponding inputs to 494 J.W. Brown et al. / Neural Networks 17 (2004) 471–510 the governing equation. For example, the GPi is required to have low activity (0.2) when the striatal direct pathway afferents are active and the GPe is tonically active. This provides one constraint. Second, it is required to have higher activity (0.5) when the striatal direct pathway projection is inactive and the GPe is tonically active. This provides a second constraint. Finally it must have high activity (0.8) when the striatal direct pathway is inactive and the indirect pathway is active, resulting in a loss of GPe – GPi inhibition. This provides a third constraint. Together, these three constraints can be used to simultaneously fit as many parameters. The constrained inputs here refer to all variables that are not parameters, such as afferent signals from other parts of the model. For a given governing equation, each desired equilibrium value and input condition pair constitute a constraint on the parameters. This equilibrium-fitting approach required much less computational power than fitting spike histograms to the model output curves. The governing differential equation can be set to zero (equilibrium), and each constraint then yields an algebraic equation in the unknown parameters. In principle, if the number of constraints equals the number of unknown parameters in a given equation, the resulting system of equations can be solved exactly. In practice, two gradient descent methods were used to solve the equations. In the first method, the objective function is the sum of the squared time derivatives (which we refer to as STID) of the governing equation for each constraint. The objective function is minimized if the time derivatives are zero, indicating that equilibrium is achieved for all constraints. In the second method, the objective function is the sum of the squared equilibrium errors (which we refer to as SEE) between the desired equilibrium value and the actual equilibrium value for the constraint, given the parameter choice. The two methods are not equivalent, because the time derivative and equilibrium error are not linearly related, due to the nonlinearity of the equations themselves. For example, doubling the equilibrium error does not necessarily double the time derivative. Although the SEE method was more accurate in some cases, it was the slower of the two because it required solving each constraint for its equilibrium value at each iteration, given the parameters. The STID method simply calculated the time derivative directly from the governing equation. For this reason, STID was run to get an initial parameter estimate, and then SEE was used to refine the estimate. In cases where the constraints were not especially complex, parameters were adjusted manually. The equilibrium-fitting technique could not specify the overall time constant for a given governing equation; these were also adjusted manually. In cases where each term in the governing equation contained a parameter to be fitted, the solver tended to find the trivial solution, corresponding to a zero constant multiplier for each term in the equation. This was prevented by constraining the sum of parameters to be normalized to a constant value n equal to the number of parameters to be fit. For all model variables, the parameters found were not particularly sensitive, and they could be rounded up to only a few significant digits. For example, the plan-tooutput layer weight scale factor in Eq. (22) was rounded from 0.377 to 0.4, and the anterior IT to striatal scale factor in Eq. (33) was rounded from 7.858 to 8. Even when all of the model parameters were rounded to one or two significant digits simultaneously, there was no significant difference in model learning or performance after vs. before parameter rounding, indicating a relative insensitivity to parameter choice. 9. Results Model simulations of oculomotor tasks. Learning and performance of seven oculomotor tasks were simulated: the five tasks shown in Fig. 1, and two additional ones described below. With the exception of the fixation task, all the tasks required generation of a saccade, and Fig. 7 summarizes the activation dynamics implied by hypotheses 1 – 14 (and the system of equations) for a typical plan-execute episode. However, the cortical paths via which the BG gate was opened, the need to use the indirect BG pathway to STOP unrewarding gate opening, and the time required to open the BG gate following fixation point offset, varied across tasks. The TELOS model was trained on the five tasks that required learning in a single order: saccade, fixation, overlap, gap, and delay. (The tasks could be learned in other orders, but no others were simulated.) The model was required to correctly perform one task trial before proceeding to learn a subsequent task. Trials were given at 5 s intervals, which provided sufficient time for the model activity to subside and equilibrate between trials. Time t was reinitialized to be zero at the beginning of each trial. At time t ¼ 0; a motivational signal, I ðMÞ was always set to 1.0 and input to the model dorsolateral PFC (the ‘What’ working memory activity Cj in Fig. 2; also see Fig. 6a), which in turn excited model FEF layer VI cells. These cells excited overlying model FEF plan cells (layers II/III/Va), thereby enabling subsequent inputs from IT and PPC to induce reverberating activations in these plan cells. Input I ðMÞ remained on until reward or punishment signals (N or N; respectively) terminated at the end of the trial. Input to the p model of a visible external stimulus, Ixyj ; always set to 1.0 ðITÞ the internal input, Ij ; which excited the ITa (TE) stage activity Tj, shown on the upper left of Fig. 2. (The superscript asterisk marks the associated variable as external to the brain.) Subscript ðx; yÞ indexes the input’s visual field (grid) location, j ¼ 1 or 2 indexes its distinctive feature value (to distinguish Fixation points from other, Target, inputs), and its ITa-coded (feature-based and position independent) classification. When a saccade was performed, visual inputs shifted position 30 ms after J.W. Brown et al. / Neural Networks 17 (2004) 471–510 495 Fig. 7. Model of a canonical plan-execute episode. (A) Top-down arousal from higher or more anterior cortical areas excites layer VI, which in turn primes action plan representations in layer III/Va. (Layer VI-to-III projections are pre-set in the present model but could be learned.) The combination of priming and sensory input supraliminally activates layer III/Va plan cells. (B) Layer III/Va activity persists after the offset of sensory input and excites specific movement-related cells in layer Vb, though not sufficiently to supraliminally activate layer Vb cells. (C) Layer III/Va plan cells excite the direct pathway of the BG, which removes the tonic inhibition of the thalamus. This disinhibition allows the existing layer VI signals to thalamus to supraliminally activate disinhibited thalamic cells. (D) These thalamic cells excite layer Vb cells. The combination of layer III/Va and thalamic inputs drives localized layer Vb activity above the threshold needed to generate output to the SC and STN. (E) Buildup of activity in direction-tuned cells of the SC sends feedback to FEF that initially excites plan cells tuned to the same direction. Once the buildup becomes large enough to initiate movement, the stronger feedback to FEF transiently activates postsaccadic cells, which are modeled as inhibitory interneurons that shut off the movement-related activity in layers III/Va and Vb. This prevents perseveration of already executed plans. saccade initiation to bring the saccade’s goal into the grid’s ‘fovea’ at locus ðx; yÞ ¼ ð1; 1Þ: Featural information from the shifted scene began updating the FEF representations after a 50 ms. consistent with the empirical delay (see Fig. 9). The model output was determined by the state of the SC, which was modeled (Fig. 5) as a map of ‘motor errors’ (Grossberg & Kuperstein, 1986; Lee et al., 1988), i.e. of eye movement vectors which, if performed, would zero the difference, or error between current and desired eye position ðx; yÞ: The SC map was excited by the model’s PPC stage, FEF output layer, or both (Figs. 2 and 6b). Whenever an SC cell’s activity rose above 0.25, it began to excite FEF plan cells tuned to the same direction. If the SC cell’s activity reached a threshold of 0.60, it was deemed to have initiated a saccade (Hanes & Schall, 1996) corresponding to the cell’s motor error vector representation. Such initiation resulted in excitation of FEF inhibitory interneurons capable of suppressing FEF plan cell firing (Burman & Bruce, 1997). After a 100 ms delay following saccade initiation, a reward signal ðN ¼ 1:0Þ was given if the saccade was correct, and a punishment signal ðN ¼ 1:0Þ was given if the response was incorrect. The reinforcement learning signals N and N could not be simultaneously active. They correspond to dopamine bursts and dips, respectively, and they lasted 100 ms (Ljungberg et al., 1992; Brown et al., 1999). Application of the model to the saccade task (Fig. 1) is summarized by the canonical plan-execute sequence of 496 J.W. Brown et al. / Neural Networks 17 (2004) 471–510 Fig. 7, in which there is no STOP signal generation in the indirect pathway. At time t ¼ 0; the fixation light was p presented in the fovea: Ixyj was set to 1.0 for ðx; y; jÞ ¼ ð1; 1; 1Þ; i.e. location (1,1) where j ¼ 1 indicates that the input possessed the distinctive feature of a Fixation point. The input IjðITÞ to ITa was set to 1.0 for j ¼ 1; to activate a spatially-invariant, feature-based representation of the fixation light. At t ¼ 200 ms; the fixation light shut off, p and the target Ixyj appeared to the right, at ðx; yÞ ¼ ð2; 1Þ; with j ¼ 2 to signify that the input possessed the distinctive feature of a Target. This target input activates ITp and then the corresponding target feature GCZ in FEF. Via the Target cortical zone GCZt, the model initiated a saccade at t ¼ 279 ms; so the RT between fixation light offset and saccade onset was just 79 ms (Fig. 9). At t ¼ 309 ms; which is 30 ms after saccade onset, the target moved to the fovea as a result p of the saccade, meaning that Ixyj shifted from ðx; yÞ ¼ ð2; 1Þ to ðx; yÞ ¼ ð1; 1Þ; where j ¼ 2. At t ¼ 379 ms, which was 100 ms after saccade initiation, reward was given by setting N to 1.0 for 100 ms. At t ¼ 479 ms, the trial ended and all p inputs were shut off, including I ðMÞ ; Ixyj ; IjðITÞ ; and N. Thus, on the first trial, the GCZt mediated correct performance of the saccade task, which involves visually-guided saccades with no STOP requirements. In addition, SC activation provided a teaching signal that caused like-tuned planning cells in the object cortical zone GCZo to be briefly excited. Because of the predictive relationship between fixation light onset, plan cell excitation in the GCZo, and generation of a positive reinforcement signal 100 ms after performance of a rightward saccade, an ITa-FEF link began to be learned between fixation light identity and the plan for a rightward saccade in the What – Where module—even though that module had not generated the reinforced saccade. (This link would have enabled anticipatory excitation of an FEF plan for a rightward saccade in the saccade task if training of the saccade task had continued; see below). The fixation task (Fig. 1) was presented next. At t ¼ 0; the fixation light appeared in the fovea. It activated the link learned (as a consequence of the last rewarded trial in the saccade task) from the ITa representation of the fixation light to GCZo plan cell for a rightward saccade. The GCZo then generated an anticipatory saccade, even before the target light could excite a saccade through GCZt. This which erroneous response led to a punishment signal, N; triggered a weight increment on the pathway from the FEF fixation plan cell to the BG indirect pathway associated with GCZo; see Fig. 2. The punishment signal also caused a decrement in the ITa to FEF link that had caused the anticipatory saccade. Consequently, on the next trial, the anticipatory activation of the GCZo plan cell by fixation light onset was no longer able to open the BG gate. At t ¼ 200 ms, the distracting target appeared to the right at ðx; yÞ ¼ ð2; 1Þ while the fixation light remained visible. Distracter onset excited the rightward saccade plan cell in GCZt, which was able to open its BG gate. The model thus erroneously initiated a saccade to the distracter target, at 361 ms, which led to activation of the punishment signal. This caused a large increment in the weight on the link from the FEF fixation plan cell to the BG indirect pathway associated with the GCZt. On the next trial, neither GCZt nor GCZo were able to open their BG gates in response to the fixation light. Consequently, the model was able to avoid looking at the distracter, and to maintain fixation, on this trial. The model was rewarded beginning at t ¼ 500 ms. Because the fixation light was still on and activating the learned pathway to GCZo, this reward further strengthened the link (from 0.3 to 0.9) between the fixation stimulus representation in ITa and the rightward saccade plan in GCZo. However, the rightward saccade plan is not executed while the fixation cue is present, because the fixation plan has also learned to activate the indirect pathway, which STOPs saccades driven by GCZo as long as the fixation cue is present. The overlap task (Fig. 1) was presented next and the model performed correctly on the first trial. It did not saccade immediately upon target onset, because the model had previously learned to withhold a planned saccade despite distracters while the fixation light remained on. It did saccade at fixation light offset because then the fixation plan vanished as a source of a STOP signal. The fixation light appeared in the fovea at t ¼ 0: At t ¼ 200 ms, the target appeared to the right at ðx; yÞ ¼ ð2; 1Þ: Fixation light offset at t ¼ 500 ms removed the source of excitation of the indirect pathway, and thereby released, at t ¼ 559 ms, a correct saccade to the target. Because of the weights acquired during the two prior tasks, both the GCZt and GCZo helped generate this saccade. Both weights were helpful because the response required in the overlap task was the same as that required in both prior tasks. Following the correct saccade, a reward was given, but only modest further change occurred in model adaptive weights. The gap task (Fig. 1) was presented with a 500 ms delay between fixation light offset and target onset. At fixation light offset, an anticipatory saccade was performed correctly on the first trial. This was because the fixation light feature activated a saccade plan of the appropriate direction, mediated by the ITa-FEF pathway, which had been learned in the saccade task and not unlearned in the intervening tasks. Indeed, it had been strengthened in the fixation task. The anticipatory saccade was generated only once the fixation light shut off, because then the fixation light was no longer able to drive the indirect pathway to suppress the anticipatory saccade. The fixation light appeared in the fovea at t ¼ 0 and shut off at t ¼ 200 ms. A saccade to the anticipated target location at ðx; yÞ ¼ ð2; 1Þ was generated via GCZo at t ¼ 340 ms. Reward was given and the trial ended prior to t ¼ 700 ms (at which point the target would have appeared had the anticipatory saccade not been generated). Had the anticipatory saccade not occurred, the appearance of the target would have driven a visually-guided saccade to foveate it. This version of the gap task differs from that often used, e.g. in studies of J.W. Brown et al. / Neural Networks 17 (2004) 471–510 express saccades (Fischer & Weber, 1993). In that version, reward is withheld if the saccade is anticipatory. Although this is intended to force the animal to wait until the target stimulus appears, animals and humans nevertheless generate some anticipatory saccades, and our simulations indicate that the model is also capable of learning to generate anticipatory saccades. The delay (or memory) task (Fig. 1) was next presented. The fixation light was visible in the fovea from t ¼ 0 until t ¼ 800 ms. The target appeared in the upper right at ðx; yÞ ¼ ð2; 2Þ and was visible from t ¼ 200 until t ¼ 500 ms. When the fixation light shut off, two saccade plans had formed in the plan layer: one was maintained by recurrent excitation in GCZt to represent the remembered target location; the other, in GCZo, represented the fixation light activated, ITa-FEF mediated, anticipated location of the target that had been used in prior tasks. At t ¼ 987 ms, an ‘anticipatory’ saccade to the previously rewarded locus was erroneously initiated by GCZo, and punishment was given. This caused a large weight increment in the STOP pathway between the GCZt plan cell and the indirect pathway controlling GCZo. On the next trial, this new link stopped the previously learned tendency to initiate an anticipatory saccade via GCZo, and thus a saccade was correctly initiated after the delay by GCZt. The model’s saccade to the remembered target location at t ¼ 969 ms was rewarded. The brief 300 ms delay between target offset and fixation light offset was chosen for convenience, and could have been much longer for a wide set of parameters. Because the model is noiseless, there is no limit on how long it can maintain a plan cell activation representing the target locus, provided that recurrent excitation is strong enough and that no other salient visual inputs are presented during the memory interval. Since the model had now learned to perform each of the tasks correctly, the training ended, and the next trials marked a final testing phase of the simulation. The state of the simulation was saved at the beginning of the next trial and served as the initial condition for testing each of the tasks. The tasks were presented as during training and were all performed correctly. RTs (measured from fixation light offset to saccade onset) from this final test are listed in Table 3. They are similar to those measured on the first correct trial during initial training of each task. Subsequent to training, a visual discrimination task was performed to assess choice between simultaneously presented peripheral stimuli, one possessing the fixation light feature, and one the target light feature. The usual fixation light was first presented in the fovea at t ¼ 0: At t ¼ 200 ms, Table 3 Simulated saccadic RTs for tasks Task Saccade Overlap Gap Delay Choice Reaction Time (ms) 79 55 163 233 210 497 a stimulus that also possessed the fixation light feature appeared in the upper left at ðx; yÞ ¼ ð0; 2Þ; and a stimulus possessing the target feature simultaneously appeared in the upper right at ðx; yÞ ¼ ð2; 2Þ: GCZf activity quickly began to reflect both the foveal and the peripheral fixation feature input. Competition between these two GCZf activities resolved in favor of the peripheral activity. Since GCZf received a permissive signal from the PNR-THAL, at t ¼ 402 ms, with the foveal fixation light still on, the model initiated a saccade to the upper left; i.e. to the peripheral stimulus that possessed the fixation light feature. This response was ‘correct’ given past training, because the model had never been punished for saccading to such a stimulus, but had been punished for saccading to stimuli possessing the target feature whenever the fixation light remained on. At t ¼ 432 ms, the stimulus at the upper left moved to the fovea, and all other stimuli were extinguished. The RT appears under ‘choice’ in Table 3. Also, subsequent to training, the model was tested on a countermanding saccade task (Hanes, Patterson, & Schall, 1998; Logan & Cowan, 1984). This was essentially a saccade task, except that on a fraction of trials, the fixation light reappeared after a brief time off. The reappearance constituted an imperative STOP signal. It instructed the system to cancel the planned saccade and instead maintain fixation. Provided that the STOP signal was not presented too late, the model was able to cancel the saccade by using a fixation light activated STOP signal mediated by the BG indirect pathway. The last two tests illustrate that the model can perform some tasks on which it was not trained. Adaptive weights were adjusted as described above, in accord with learning equations detailed in the Mathematical Model section. An algorithm (see Mathematical Model) had previously adjusted other model parameters to minimize the difference between model cell dynamics and the observed cell dynamics of a large number of electrophysiological cell types identified in published experimental studies of the modeled areas. Table 4 lists electrophysiological reports that were used to guide parameter adjustment. Activation traces from many of these experimental reports are reprinted below (in Figs. 9 – 11) alongside simulated activation traces of corresponding model cells. Clearly, the model learned faster than real monkeys. This was largely due to simulation parameters, in particular, a lack of any uncontrolled distracting stimuli, no noise, unflagging motivational input signals, maximal between-task compatibility, and a conveniently high learning (weight adjustment) rate. Less favorable parameters would amplify the number of trials to criterion. Illustration of model cell dynamics in one task. Panels A – U of Fig. 8 depict the dynamics of 22 model cells during the overlap task; see Fig. 1. Panel A shows activations of two model FEF input-layer cells whose dynamics differ because only one cell’s receptive field is such that it (dotted line) receives both an early input based on the target’s 498 J.W. Brown et al. / Neural Networks 17 (2004) 471–510 Table 4 Electrophysiological cell types Area Cell type names Simulation-data comparison figure References FEF 1. Visual 2. Visual with second burst 3. Visuomovement 4. Discriminating visuomovement 5. Transient Sensory-Movement 6. Presaccadic 7. Postsaccadic 8. Sustained Sensory-Movement 9. Preparatory Set/Fixation 10. Burster 11. Pause-burster 12. Burster 13. Burster 14. Pauser 15. Movement-related 16. Saccade-related 17. Fixation-related Fig. 2A Fig. 2A Fig. 2B Fig. 2B Fig. 2C Fig. 2D Fig. 2E Fig. 2F Fig. 2G Fig. 3A Fig. 3A Fig. 3B Fig. 3C Fig. 4A Fig. 4C Fig. 4B Fig. 4D Schall et al. (1995a) Schall (1991) Schall and Bichot (1998) FEF FEF FEF FEF FEF FEF Gpe Gpi Motor Thalamus STN SNr PPC SC SC retinotopic locus and a delayed input, from ITp, based on a conjunction of the target’s position and feature. The other cell (solid line) shares the initial input but lacks the second because it is tuned to a different position-feature conjunction in ITp. All other panels show only one cell. Panels M and N respectively show a winning and a losing FEF planlayer cell. The Fig. 8 caption gives the number of the equation (see Mathematical Model) that mediated each dynamic. Reinforcement learning of plan activations. The trial-bytrial progress of successful learning and performance of the task series was described above. The dotted lines in induced Fig. 8L – N depict the second messenger traces ðFÞ by recent plan cell activations. These intracellular trace variables allowed the model to bridge the time delays between plan cell offset and the later arrival at planning cells of signals indicating rewarding or non-rewarding response outcomes. Consequently, reinforcement learning in model cortico-cortical synapses was able to bring plan cell activation under control of inputs to FEF. The simulations Schall (1991), Segraves and Goldberg (1987) Bruce et al. (1985), Schall (1991), Segraves and Goldberg (1987) Bruce et al. (1985), Schall (1991) Hanes et al. (1998), Schall (1991) Schall et al. (1995b), Schall (1991) Turner and Anderson (1997) Turner and Anderson (1997) Schlag-Rey and Schlag (1984), Turner and Anderson (1997) Wichmann et al. (1994) Hikosaka and Wurtz (1989) Kalaska and Crammond (1995) Munoz and Wurtz (1995) Munoz and Wurtz (1993) exemplify computational hypothesis 12, that reinforcement learning establishes stimulus control over plans, not responses, even though plan cell activation is suppressed by feedback inhibition from post-saccadic cells well before reward or non-reward signals have been generated. More generally, the model illustrates how learning and performance processes evolve in parallel and in real-time, with no need for ‘off-line’ learning. Comparisons of model with real cell dynamics. Figs. 9 –11 show side-by-side comparisons of 17 model cell activations with neurophysiological data (citations in Table 4). These comparisons illustrate the ability of the model to match data across the range of tasks simulated. In particular, Fig. 9A data and simulation are from the delay task, 9B from the discrimination task, and 9C –9G from the overlap task. Fig. 10 data and simulation are from the delay task, and Fig. 11 from the overlap task. Fig. 9 shows plots of model cell behavior alongside reprinted plots of electrophysiological data for various real FEF cells with distinctive types of task-related dynamics. Fig. 8. Activation dynamics of 22 simulated cell types during performance of the overlap task. From left to right in each column of plots, the three vertical lines respectively mark target onset time, fixation light offset time, and saccade onset time. (A) Two FEF input layer cells (governed by Eq. (12) whose retinotopic receptive fields cover the target position. Only one shows a second burst due to delayed input (from ITp) conditional upon target’s position and features. (B) FEF layer VI cell Eq. (13). (C) GPi/SNr cell Eq. (35) associated with the Fixation-feature GCZ. (D) GPi/SNr cell Eq. (35) associated with the Target-feature GCZ.(E) Striatal SPN in the direct pathway Eq. (27) associated with SC saccade-related cells. (F) SNr saccade-related cell Eq. (32) that receives inhibition from the cell in (E). (G) SNr cell Eq. (32) associated with SC fixation cells. (H) Fixation cell Eq. (11) of the SC. (I) Saccade-related cell Eq. (11) of the SC. (J) FEF post-saccadic cell Eq. (26). (K) Thalamic cell Eq. (38) associated with the Target GCZ. The late (post movement) discharge of this cell is associated with postsaccadic fixation of the Target object. There is little pre-movement thalamic activity, because the movement to the still visible target is generated by the reactive, PPC-SC, pathway (not via FEF) once Fixation stimulus offset removes the indirect channel’s STOP signal. (L) FEF fixation plan cell Eq. (14) of layer variable Eq. (45). (M) Winning FEF saccade plan cell Eq. (14) of layer III/Va, showing discharge of the sustained III/Va. Dotted line shows associated trace ðFÞ sensory-movement type. (N) Losing FEF saccade plan cell Eq. (14) showing discharge of the transient sensory-movement type. (O) Striatal direct path SPN Eq. (33) associated with FEF fixation cells. (P) Striatal direct path SPN Eq. (33) associated with FEF saccade-related cells. (Q) PPC fixation-related cell Eq. (6). (R) PPC saccade-related cell Eq. (6). (S) FEF layer Vb fixation cell Eq. (21). (T) Striatal SPN of the indirect pathway Eq. (34) that inhibits the FEF Target-feature module. This cell receives strong input from the FEF fixation cell in (L). (U) GPe cell Eq. (36) in the pathway that inhibits the FEF Target-feature module. This cell receives strong inhibition from the cell in (T). J.W. Brown et al. / Neural Networks 17 (2004) 471–510 499 500 J.W. Brown et al. / Neural Networks 17 (2004) 471–510 Fig. 9. Comparison of dynamics of real and simulated FEF cells. Data appear above model fits in all panels except (G). Fig. 9A simulation and data are from the delay task, 9B from the discrimination task, and 9C, D, E, F, and G from the overlap task. (A) Discharge of FEF visual cells and activation level of model FEF input layer cells. [Data adapted with permission from Schall et al., 1995a.] (B) FEF visuomovement cells and model FEF plan layer cells. In the simulation plot, the solid trace is from the winning plan and the dashed trace from a losing plan. [Data adapted with permission from Schall and Bichot, 1998.] (C) FEF transient sensory-movement cells and model FEF losing plan layer cells [Data adapted with permission from Schall, 1991.] (D) FEF presaccadic movement cells and model FEF layer Vb (output) layer cells. (E) FEF postsaccadic cells and model FEF inhibitory feedback cells [(D) and (E) data plots adapted with permission from Schall, 1991.] (F) FEF sustained sensory-movement cells and model FEF plan layer cells that represent planned but not yet executed saccades. [Data adapted with permission from Hanes et al., 1998.] (G) FEF preparatory set cells (left) and model FEF fixation cells (right). Vertical line on right denotes saccade onset. [Data adapted with permission from Schall et al., 1995b.] J.W. Brown et al. / Neural Networks 17 (2004) 471–510 501 Fig. 10. Comparison of dynamics of simulated and real BG and PNR–THAL cells for the delay task. (A) On left, real GPe cells increase their firing rate prior to movement onset whereas GPi cells pause. On the right are corresponding model GPe and GPi cells. In the model, GPi pausing permits movement execution. Model STN cell activity beginning at movement initiation transiently increases both the GPi and GPe activities. [Data adapted with permission from Turner and Anderson, 1997.] (B) Above, real thalamic cells show a transient burst just prior to movement initiation. Below, the model thalamic burst reflects excitation from layer 6 and transient disinhibition (gating) by the GPi. [Data adapted with permission from Schlag-Rey and Schlag, 1984.] (C) Above, real STN cells show a transient increase in activity immediately following movement onset. Below, the model STN burst reflects a transient corollary discharge from FEF output layer cells. [Data adapted with permission from Wichmann et al., 1994.] Fig. 11. Comparison of dynamics of simulated and real PPC, SC, and SC-related BG cells for the overlap task. Data appear above simulation in each panel. (A) Real SNr cell pauses during saccades near its preferred direction. Model SNr cell pauses to allow model SC to generate a saccade near the cell’s preferred direction. [Data adapted with permission from Hikosaka & Wurtz, 1989, Fig. 10b.] (B) Real SC burst cell activity during the overlap task compared with model SC burst cell. [Data adapted with permission from Munoz & Wurtz, 1995.] (C) Real PPC activity during a task similar to the overlap task compared with model PPC activity during the overlap task. [Data adapted with permission from the reaching study of Kalaska & Crammond, 1995.] (D) Real fixation-cell activity in the rostral SC, where fixation cells pause briefly during a saccade, compared to a model SC fixation cell. [Data adapted with permission from Munoz & Wurtz, 1993.] 502 J.W. Brown et al. / Neural Networks 17 (2004) 471–510 Fig. 9A shows FEF visual cells (Schall et al., 1995a). These cells fire weakly at 50 ms after stimulus onset (time t ¼ 0) in the receptive field. If the stimulus is a preferred feature for the cell, a second burst occurs at 100 ms due to input from ITp. In the model (see Figs. 2 and 6a), these are FEF input ðIÞ layer cells (Fxyi in Eq. (12)). Fig. 9B shows traces from two FEF discriminating visuomovement cells (Schall & Bichot, 1998). Such cells initially respond to either a preferred or non-preferred stimulus, but begin to discriminate with a bifurcation around 100– 120 ms after stimulus onset. In the ðPÞ model, these are plan layer cells (Fxyi in Eq. (14)), and their bifurcation is due to both recurrent, competitive dynamics and the selectiveness of their inputs (shown in Fig. 9A). Either factor may explain the slightly faster decline of activity in the ‘losing’ model cell than in the corresponding real cell. Fig. 9C exemplifies transient sensory-movement cells (Schall, 1991), which fire both in response to the stimulus and around the time of the saccade. In the model, ðPÞ these are plan layer cells (Fxyi ; Eq. (14)) that lose the competition for sustained activity, due to recurrent inhibition in (18). A corollary discharge from the SC (term Sxy in (15)) drives the burst just before movement onset. That the 9C real cell has some sustained activity whereas the 9C model cell has none is not significant because both data and model indicate a continuum from none to the robust sustained activity shown in 9F. Fig. 9D shows presaccadic cells (Schall, 1991), which peak at the time of saccade onset, when activity is sufficient to initiate a saccade but inhibition from ðXÞ postsaccadic cells (Fxy ; Eq. (26)) has not yet taken effect. In the model, these are layer Vb output cells ðOÞ (Fxyi ; Eq. (21)) that mediate execution of the saccade. Some movement-related layer Vb cells that projected to SC also showed visual rather than movement activity when the stimulus in the receptive field was not to be foveated, and others showed weak but sustained, working memory-like activity. These were epiphenomenal in the model and emerged due to the complex constraints on parameters governing movement gating in layer V. However, these unexpected findings agree with and functionally interpret otherwise puzzling observations of non-motor responses in SC-projecting cells in layer V of FEF (Sommer & Wurtz, 2000). Fig. 9E shows postsaccadic cells (Schall, 1991), which begin firing at the point of saccade initiation. In the model, these cells ðXÞ (Fxy ; Eq. (26)), which are driven by a corollary discharge of movement initiation from SC burst cells, serve to shut off the successfully executed saccadic plan. Fig. 9F shows sustained sensory-movement cells (Hanes et al., 1998), which are active from stimulus onset until just after saccade initiation. These contrast with preparatory set or fixation cells, which cease activity before, rather than after, saccade initiation. In the model, the sustained sensory-movement cells (Eq. (14)) exist in the plan layer (II/III/Va) and represent planned but not yet executed saccades. Their sustained activity is initiated by afferents from the input layer (F ðIÞ in (12)), maintained by recurrent self-excitation, and shut off by inhibition from postsaccadic cells (F ðXÞ in (26)). Fig. 9G shows FEF preparatory set cells (Schall et al., 1995b). The vertical line on the right denotes saccade onset. Set cells remain active during fixation, and shut off prior to saccade initiation. In the model FEF, such cells exist in ðPÞ ðOÞ both the plan (Fxyi ; Eq. (14)) and the output layers (Fxyi ; Eq. (21)) to maintain fixation. They are driven by foveal activity and are shut off by inhibition from saccadeðOÞ related output cells Fxyi ; subject to parameter kfop in (18) and the last term in (25), when saccade initiation is imminent. Fig. 10 shows neurophysiological data collected from BG and PNR –THAL cell types (see Fig. 2) and the dynamics of corresponding simulated cells. Fig. 10A (left side) shows data from GPe and GPi from the reaching study of Turner and Anderson (1997). GPe cells (BðGPeÞ ; k Eq. (36)) increase their firing rate prior to movement onset, whereas GPi cells (BðGPiÞ ; Eq. (35)) show a pause k prior to movement onset. Similar pausing data have been reported by Hikosaka and Wurtz (1983) for SNr in a saccade task, as shown in Fig. 11. In the simulation (Fig. 10A, right), the GPi pause is due to increased inhibition from both the GPe (BðGPeÞ ; Eq. (36)) and the k striatum (BðSDÞ ; Eq. (33)), and the pause disinhibits the k PNR-THAL, thereby allowing the movement to be executed. STN activity ðBðSTNÞ ; Eq. (37)) that begins at movement initiation transiently increases both the GPi and GPe activities. Fig. 10B shows PNR-THAL cells (Schlag-Rey & Schlag, 1984; Eq. (38)) that show a transient burst just prior to movement initiation. In the model, this reflects excitation from FEF layer VI ðFiðGÞ Þ and transient disinhibition (gating) in the GPi-thalamic projection. Fig. 10C shows STN cells (Wichmann, Bergman, & DeLong, 1994) that exhibit a transient increase in activity immediately following movement onset. In the model, this reflects a transient corollary ðOÞ discharge from FEF output layer cells (Fxyi ; Eq. (21)) to ðSTNÞ the STN ðB ; Eq. (37)). Fig. 11 shows the dynamics of PPC, SC, and SCrelated BG cells. Fig. 11A shows an SNr cell pausing prior to saccade initiation. In the simulation (GðSNrÞ xy ; Eq. (32)), the similar pause is caused by inhibitory action of the model direct path from the striatum (GðSDÞ xy ; Eq. (27)). Indeed, the mirror image of the SNr off-response, also shown in Fig. 8F, can be seen in the striatal on-response shown in Fig. 8E, consistent with data reviewed in Hikosaka and Wurtz (1989). Fig. 11B shows SC burst cell activity (Munoz & Wurtz, 1995) during the overlap task. In the simulation (Sxy ; Eq. (11)), the excitatory inputs that energize the burst come from both PPC cells ðOÞ (Pxy) and FEF output cells ðFxyi Þ: Although the former excite the SC throughout the overlap interval, the SC shows a burst response only after the removal of inhibition from the model SNr ðGðSNrÞ xy Þ: Fig. 11C shows J.W. Brown et al. / Neural Networks 17 (2004) 471–510 PPC activity (Kalaska & Crammond, 1995) during a GO –NOGO task on a GO trial similar to an overlap task trial. The real and simulated PPC activities show sustained activity during the delay interval followed by a pre-movement burst response. However, note that the model lacks the tonic baseline activity seen in the real cell. The model (Pxy ; Eq. (6)) attributes the sustained ðPÞ delay-interval activity to inputs from visual cortex ðIxy Þ ðPÞ and the FEF planning layer III/Va ðFxyi Þ; and the burst ðOÞ response to input from the FEF output layer Vb ðFxyi Þ: Fig. 11D shows fixation-related activity in the SC (Munoz & Wurtz, 1993). Such SC fixation cells pause briefly during a saccade. In the simulation (Sxy ; Eq. (11)), the pause results from both the collapse of excitatory ðOÞ input from FEF fixation-related output cells ðF11i Þ and the onset of inhibition from the fixation-related component of the SNr ðGðSNrÞ 11 Þ: 10. Discussion General issues. The theory developed in this paper proposes how the BG interact with laminar circuits in the frontal cortex and SC to help satisfy the staging requirements of conditional voluntary behavior. Fourteen functional hypotheses were elaborated to clarify the roles played by components of the known circuitry treated by the theory. The mathematical model based on this theory is able to account for a wide range of anatomical, neurophysiological, and psychophysical data about planned and reactive saccadic eye movements. During performance, the model regenerated seventeen distinct types of activation dynamics exhibited by electrophysiologically-recorded cell types, while also predicting the behavior of several additional cell types. By virtue of learning guided by reinforcement signals, and selective cue sensitivities of premotor mapping zones, the model system was able to learn five, and to perform seven, oculomotor tasks, without forgetting previously learned tasks, and with a single set of parameters. As such, the TELOS model goes some distance toward explaining both procedural learning of arbitrary conditionaction rules (Wise & Murray, 2000) and the ability to switch at will among the large number of rules that may eventually be learned and simultaneously preserved in memory. Comparison with other models. The model introduces new concepts while also incorporating and extending concepts from previous models. The concept of zones consisting of cortex and BG channels has been discussed previously (Beiser & Houk, 1998; Houk & Wise, 1995; Redgrave et al., 1999) in terms of cortico-BG modules. Consistent with data on distinct sensitivities of frontal premotor areas (review in Passingham, 1993), we have extended the module concept to denote groups of cells organized into gatable cortical zones (GCZs) that each mediate a distinctive sensorimotor strategy or mapping. The resultant GCZ concept shares features with the ‘mixture of 503 experts’ model (Jacobs, Jordan, Nowland, & Hinton, 1991), which has been discussed with references to BG function (Graybiel, Aosaki, Flaherty, & Kimura, 1994; Graybiel, 1998). However, whereas previously the striatum has been suggested as the location of the ‘experts’ (Graybiel, 1998), in the present model of primate action control, the ‘experts’ are cortical, indeed frontal, and the BG function as a GCZ selector. BG control of cortico-thalamo-cortical projections. Previous models have assumed reciprocal excitation between the cortex and thalamus (Arbiband Dominey, 1995; Beiser & Houk, 1998; Frank, Loughry, & O’Reilly, 2001; Houk & Wise, 1995; Taylor & Taylor, 2000), which was proposed as a basis for sustained, working memory-like activity enabled by BG disinhibition of the thalamus. The working memory in turn biases, rather than determines, movement selection (Houk & Wise, 1995). Studies of the laminar anatomy of frontal cortex show thalamocortical projections terminating on cells in layers III and V, but not layer VI (Iriki, Pavlides, Keller, & Asanuma, 1991; Zin-Ka-Ieu, Roger, & Arnault, 1998). Since layer VI is the main source of corticothalamic projections, its apparent omission as a recipient of thalamocortical afferents challenges the cortico-thalamo-cortical loop theory of working memory storage. A reciprocal thalamocortical excitation remains possible if layer III cells excite layer VI, or if thalamocortical fibers terminate on dendrites of layer VI neurons that extend into overlying laminae, but the known thalamic excitation of layer V will cause outputs that would—on the loop theory—make it impossible for a single cortical area to function both to reverberate a plan during a delay interval and then to also execute that plan at the end of the delay. In our theory, both functions are realized by a single cortical area. Given the anatomical constraints and the ease with which basic working memory functionality can be achieved by local circuits, or even by intracellular mechanisms, our theory treats plan maintenance during delays as a local circuit property and treats BG-mediated disinhibition of thalamus as a mechanism for gating excitation of layer V and, thereby, execution of action plans. Finally, it is worth noting that nothing in our interpretation precludes a role for the BG in selective activation of prefrontal areas supporting working memory. Lateral vs. feedforward inhibition within the striatum. Recurrent competition between striatal SPNs, mediated by potent lateral inhibition among SPNs, has been assumed by previous models (Arbib& Dominey, 1995; Beiser & Houk, 1998; Contreras-Vidal & Schultz, 1999). However, other data (Jaeger et al., 1994) challenge the potency assumption. Also, it is difficult to reconcile the idea of many actively competing SPNs with observations of a relatively silent striatum. The present model resolves the dilemma by emphasizing the competitive role of the GABA-SIs (GABA-ergic striatal interneurons), which, though outnumbered by SPNs 20 to 1, mediate potent feedforward inhibition (Koos & Tepper, 1999; Wilson et al., 1989). In 504 J.W. Brown et al. / Neural Networks 17 (2004) 471–510 our model, such cells prevent disparate plans from gaining simultaneous control of the same executive degrees of freedom. Although the model GABA –SIs mediate striatumlevel competition between vying plans, choice-making cannot be wholly localized to the striatum. The present model broadly distributes plan-selective dynamics across competitive inhibitory mechanisms in the thalamus, SC, and both supragranular and infragranular cortical layers. Finally, the existence of striatal recurrent inhibition is compatible with the present model. Although not needed to make the selection, recurrent inhibition would be useful in vivo to help sustain the striatal choice in the face of noise and modest input fluctuations (which were not present in our deterministic simulations). BG direct and indirect pathways. The few BG computational models that treat the indirect pathway have ascribed to it roles other than a STOP function. In some models, the indirect pathway generally opposes the direct pathway effect, without explicit justification for why this is needed (Contreras-Vidal & Stelmach, 1995; Suri et al., 1997; Suri, Bargas, & Arbib, 2001). Gurney, Prescott, & Redgrave (2001a,b) suggested that the components of the indirect pathway may be thought of as forming a ‘control’ pathway, which modulates the tonic arousal of the GPi in proportion to the overall level of dopamine and excitation of the striatum. Berns and Sejnowski (1998) assumed that reciprocal activity between the STN and GPe mediated a kind of working memory which subserved sequence production. Contreras-Vidal and Stelmach (1995) modeled the GPe’s possible role in enhancing the effects of dopamine loss in Parkinson’s disease. Our model reconceptualizes the indirect pathway and suggests a specific rationale for why it is needed in addition to the direct pathway, namely to actively STOP inappropriate action plans from execution until such time as they become appropriate. Dominey and Arbib model. Arbib and Dominey (1995) and Dominey et al. (1995) reported a model of the FEF, BG, and related oculomotor areas, with which our model shares several features. Their model also used retinotopic and motor error map representations throughout. Their FEF model contained visual, memory, fixation, and saccade ‘layers’, which although not identified with cortical laminae, resemble the cortical layers of the present model. Their visual layer was similar to our visual or input layer, except that our visual cells do not drive FEF output (output) cells directly. Their FEF memory layer corresponds to saccade-related cells in our plan layer, and their ‘fixation ON’ layer corresponds to fixation-related cells in our plan layer (both here predicted to reside in cortical layers II/III/Va). Their FEF output layer corresponds to our output layer (here predicted to reside in layer Vb), in that both drive the SC, although their output layer does not receive corticothalamic projections, which enable gating of movement execution in our model. Their model also had distinct FEF- and SC-controlling BG channels and assumed a motor error map representation in nigro-SC projections, as does ours. The two models address a large and overlapping body of data. Additional important features distinguish the two models. Our explicit treatment of the laminar anatomy of frontal cortex led us to model the BG as a system for gating plan execution rather than for reverberation of working memories. Our model relies on neither striatal lateral inhibition nor reciprocal thalamocortical excitation. Whereas the Dominey et al. (1995) model simulates sequence learning, our model simulates multiple task learning and, during performance of the multiple tasks, simulates the dynamics of seventeen electrophysiologicallyidentified cell types spread across all the brain regions treated. Their model predicts that corticostriatal projections mediate conditional motor learning, whereas our model predicts that corticocortical projections from anterior or posterior IT to the FEF plan layers (II/III/Va) mediate conditional motor learning, including ‘What-to-Where’ transformations. Our model also implicates the indirect channel in the conditioned STOP function neglected in prior models. Their model predicts that working memory activity is shut off by a nonspecific, inhibitory corollary discharge from the SC to the thalamus. Ours proposes that the SC drives postsaccadic cells in cortex to shut off specific, already-executed, movement plans. Future extensions. The TELOS model may be extended in several ways. Its treatment of the ability of the frontal cortex to replace a parietal target representation excited by a transient visual input with a remembered target plan, and to generate an internally guided saccade following resolution of the parietal competition, applies with minor changes to how primates perform the anti-saccade task (Amador, Schlag-Rey, & Schlag, 1998; Everling, Spantekow, Krappmann, & Flohr, 1998; Funahashi, Chafee, & Goldman-Rakic, 1993; Hallett, 1978). One of these changes would be the addition of the SEFs to expand the representational scope (Chen & Wise, 1995a, 1995b) of the model’s premotor GCZs. More generally, although we have focused primarily on the FEF, BG, and SC, the anatomical principles used to structure the model govern many other parts of the frontal cortex (e.g. Middleton & Strick, 2000). Therefore the model should generalize to other frontal cortical areas. For another example, the prefrontal cortex (PFC, e.g. areas 9 and 46) is represented in the model as a single layer working memory (Fig. 2) which could itself be expanded into a multi-layered system similar to the present FEF model. The output layer (or even plan layer) of such an expanded prefrontal model could provide the control signals to the FEF layer VI. This would constitute a kind of hierarchical decision-making scheme, in which the PFC decides, based on current working memory activity, what strategies will be primed in lower areas such as FEF. These lower areas then determine the specifics of a planned movement. Another fruitful direction for extensions is the explanation of serial behavior. In the existing model, if a ‘losing’ plan remains, after the winning plan is executed and has its J.W. Brown et al. / Neural Networks 17 (2004) 471–510 activity deleted by postsaccadic cell activity, then the losing plan may be executed subsequently. This means that the model can serve as part of what has been called a competitive queuing system, in which several to-be-performed plans are simultaneously active before movement begins. Then through iterative choice making (and deletion of plans as they are executed), the corresponding actions are executed in a sequence determined by the initial gradient of activation levels, namely from most to least active. Evidence for such gradient representations of winning versus losing plans, and of planned sequences, have recently been reported in prefrontal and dorsal premotor cortices (Averbeck et al., 2002; Cisek & Kalaska, 2002). This mechanism can be used to help learn and perform movement sequences (Boardman & Bullock, 1991; Bullock & Rhodes, 2003; Grossberg, 1982; Grossberg & Kuperstein, 1986; Hartley & Houghton, 1996), for which the BG system has been implicated (e.g. Aldridge & Berridge, 1998; Kermadi, Jurquet, Arzi, & Joseph, 1993; Martin, Phillips, & Iansek, 1994; Weiss, Stelmach, & Hefter, 1997). Acknowledgements J.B. was supported in part by Defense Advanced Research Projects Agency and the Office of Naval Research (ONR N00014-95-1-0409, ONR N00014-92-J-1309, and ONR N00014-95-1-0657). D.B. was supported in part by Defense Advanced Research Projects Agency and the Office of Naval Research (ONR N00014-95-1-0409, ONR N00014-92-J-1309) and the National Institute of Mental Health (R01 DC02852). S.G. was supported in part by Air Force Office of Scientific Research (AFOSR F49620-01-10397), Defense Advanced Research Projects Agency and the Office of Naval Research (ONR N00014-95-1-0409, ONR N00014-92-J-1309, ONR N00014-95-1-0657, ONR N00014-01-1-0624), and the National Science Foundation (NSF IRI-97-20333). References Albin, R. L., Young, A. B., & Penney, J. B. (1989). The functional anatomy of basal ganglia disorders. Trends in Neurosciences, 12, 366– 374. Aldridge, J. W., & Berridge, K. C. (1998). Coding of serial order by neostriatal neurons: A natural action approach to movement sequence. Journal of Neuroscience, 18, 2777–2787. Alexander, G., & Crutcher, M. (1990). Functional architecture of basal ganglia circuits: neural substrates of parallel processing. Trends in Neurosciences, 13, 266–271. Amador, N., Schlag-Rey, M., & Schlag, J. (1998). Primate antisaccades. I. Behavioral characteristics. Journal of Neurophysiology, 80, 1775–1786. Amos, A. (2000). A computational model of information processing in the frontal cortex and basal ganglia. Journal of Cognitive Neuroscience, 12, 505–519. Arbib, M. A., & Dominey, P. F. (1995). Modeling the roles of basal ganglia in timing and sequencing saccadic eye movements. In J. Houk, J. Davis, 505 & D. Beiser (Eds.), Models of information processing in the basal ganglia (pp. 149–162). Cambridge, MA: MIT Press. Asaad, W. F., Rainer, G., & Miller, E. K. (2000). Task-specific neural activity in the primate prefrontal cortex. Journal of Neurophysiology, 84, 451–459. Averbeck, B. B., Chafee, M. V., Crowe, D. A., & Georgopoulos, A. P. (2002). Parallel processing of serial movements in prefrontal cortex. Proceedings of the National Academy of Science, 99, 13172–13177. Bao, S., Chan, V. T., & Merzenich, M. M. (2001). Cortical remodelling induced by activity of ventral tegmental dopamine neurons. Nature., 412, 79–83. Bar, M., Tootell, R. B. H., Schacter, D. L., Greve, D. N., Fischl, B., Mendola, J. D., Rosen, B. R., & Dale, A. M. (2001). Cortical mechanisms specific to explicit object recognition. Neuron, 29, 529 –535. Barbas, H. (1988). Anatomic organization of basoventral and mediodorsal visual recipient prefrontal regions in the rhesus monkey. Journal of Comparative Neurology, 276, 313–342. Barbas, H., & Mesulam, M. M. (1981). Organization of afferent input to subdivisions of area 8 in the rhesus monkey. Journal of Comparative Neurology, 200, 407–431. Basso, M. A., & Wurtz, R. H. (1998). Modulation of neuronal activity in superior colliculus by changes in target probability. Journal of Neuroscience, 18, 7519–7534. Bayer, H. M., Handel, A., & Glimcher, P. W. (1999). Do the saccaderelated neurons of the substantia nigra pars reticulata (SNr) carry information about saccadic amplitude or endpoint? Society for Neuroscience Abstracts, 25, 1921. Beiser, D. G., & Houk, J. C. (1998). Model of cortical-basal ganglionic processing: Encoding the serial order of sensory events. Journal of Neurophysiology, 79, 3168–3188. Berendse, H. W., de Galis, G. Y., & Groenewegen, H. J. (1992). Topographical organization and relationship with ventral striatal compartments of prefrontal corticostriatal projections in the rat. Journal of Comparative Neurology, 316, 314–347. Berger, B., Trottier, S., Verney, C., Gaspar, P., & Alvarez, C. (1988). Regional and laminar distribution of the dopamine and serotonin innervation in the macaque cerebral cortex: a radioautographic study. Journal of Comparative Neurology, 273, 99– 119. Berns, G., & Sejnowski, T. (1998). A computational model of how the basal ganglia produce sequences. Journal of Cognitive Neuroscience, 10, 108– 121. Berretta, S., Parthasarathy, H. B., & Graybiel, A. M. (1997). Local release of GABAergic inhibition in the motor cortex induces immediate-early gene expression in indirect pathway neurons of the striatum. Journal of Neuroscience, 17, 4752–4763. Bichot, N. P., & Schall, J. D. (1999). Effects of similarity and history on neural mechanisms of visual selection. Nature Neuroscience, 2, 549– 554. Bichot, N. P., Schall, J. D., & Thompson, K. G. (1996). Visual feature selectivity in frontal eye fields induced by experience in mature macaques. Nature, 381, 697–699. Boardman, I., & Bullock, D. (1991). A neural network model of serial order recall from short-term memory. Proceedings of the International Joint Conference on Neural Networks, II, 879–884. Brown, J., Bullock, D., & Grossberg, S. (1999). How the basal ganglia use parallel excitatory and inhibitory learning pathways to selectively respond to unexpected rewarding cues. Journal of Neuroscience, 19, 10502–10511. Brown, J., Bullock, D., & Grossberg, S. (2000). How laminar frontal cortex and basal ganglia circuits interact to control planned and reactive saccades. Society for Neuroscience Abstracts, 26. Brown, L. L., Makman, M. H., Wolfson, L. I., Dvorkin, B., Warner, C., & Katzman, R. (1979). A direct role of dopamine in the rat subthalamic nucleus and an adjacent intrapeduncular area. Science, 206, 1416–1418. Bruce, C. J., Goldberg, M. E., Bushnell, M. C., & Stanton, G. B. (1985). Primate frontal eye fields. II. Physiological and anatomical correlates of 506 J.W. Brown et al. / Neural Networks 17 (2004) 471–510 electrically-evoked eye movements. Journal of Neurophysiology, 54, 714– 734. Bullier, J., Schall, J. D., & Morel, A. (1996). Functional streams in occipitofrontal connections in the monkey. Behavioral Brain Research, 76, 89–97. Bullock, D., & Grossberg, S. (1991). Adaptive neural networks for control of movement trajectories invariant under speed and force rescaling. Human Movement Science, 10, 3–53. Bullock, D., & Rhodes, B. (2003). Competitive queuing for serial planning and performance. In M. Arbib (Ed.), (2nd ed.) (pp. 241 – 244). Handbook of brain theory and neural networks, Cambridge, MA: MIT Press. Burman, D. D., & Bruce, C. J. (1997). Suppression of task-related saccades by electrical stimulation in the primate’s frontal eye field. Journal of Neurophysiology, 77, 2252– 2267. Butler, A. B., & Hodos, W. (1996). Comparative vertebrate neuroanatomy. New York: Wiley-Liss. Calabresi, P., Gubellini, P., Centonze, D., Picconi, B., Bernardi, G., Chergui, K., Svenningsson, P., Fienberg, A. A., & Greengard, P. (2000). Dopamine and cAMP-regulated phosphoprotein 32 kDa controls both striatal long-term depression and long-term potentiation, opposing forms of synaptic plasticity. Journal of Neuroscience, 20, 8443–8451. Canteras, N. S., Shammah-Lagnado, S. J., Silva, B. A., & Ricardo, J. A. (1990). Afferent connections of the subthalamic nucleus: a combined retrograde and anterograde horseradish peroxidase study in the rat. Brain Research, 513, 43–59. Cavada, C., & Goldman-Rakic, P. S. (1991). Topographic segregation of corticostriatal projections from posterior parietal subdivisions in the macaque monkey. Neuroscience, 42, 683–696. Chafee, M. V., & Goldman-Rakic, P. S. (2000). Inactivation of parietal and prefrontal cortex reveals interdependence of neural activity during memory-guided saccades. Journal of Neurophysiology, 83, 1550–1566. Charpier, S., & Deniau, J. M. (1997). In vivo activity-dependent plasticity at cortico-striatal connections: evidence for physiological long-term potentiation. Proceedings of the National Academy of Science, 94, 7036–7040. Chen, L. L., & Wise, S. P. (1995a). Neuronal activity in the supplementary eye field during acquisition of conditional oculomotor associations. Journal of Neurophysiology, 73, 1101–1121. Chen, L. L., & Wise, S. P. (1995b). Supplementary eye field contrasted with the frontal eye field during acquisition of conditional oculomotor associations. Journal of Neurophysiology, 73, 1122–1134. Cisek, P., & Kalaska, J. F. (2002). Simultaneous encoding of multiple potential reach directions in dorsal premotor cortex. Journal of Neurophysiology, 87, 1149– 1154. Cohen, J. D., Braver, T. S., & Brown, J. W. (2002). Computational perspectives on dopamine function in prefrontal cortex. Current Opinion in Neurobiology, 12, 223–229. Contreras-Vidal, J., & Schultz, W. (1999). A predictive reinforcement model of dopamine neurons for learning approach behavior. Journal of Computational Neuroscience, 6, 191 –214. Contreras-Vidal, J., & Stelmach, G. E. (1995). A neural model of basal ganglia-thalamocortical relations in normal and parkinsonian movement. Biological Cybernetics, 73, 467–476. Cowan, N. (2001). The magical number 4 in short-term memory: A reconsideration of mental storage capacity. Behavioral and Brain Sciences, 24, 87–185. Crosson, B. (1985). Subcortical functions in language: a working model. Brain and Language, 25, 257–292. De Las Heras, S., Mengual, E., Velayos, J. L., & Gimenez-Amaya, J. M. (1998). Re-examination of topographic distribution of thalamic neurons projecting to the caudate nucleus. A retrograde labeling study in the cat. Neuroscience Research, 31, 283–293. Deng, S. Y., Goldberg, M. E., Segraves, M. A., Ungerleider, L. G., & Mishkin, M. (1986). The effect of unilateral ablation of the frontal eye fields on saccadic performance in the monkey. In E. Keller, & D. Zee (Eds.), Advances in the Biosciences (Vol. 57). Adaptive processes in visual and oculomotor systems, New York: Pergamon. Dominey, P., Arbib, M., & Joseph, J. (1995). A model of corticostriatal plasticity for learning oculomotor associations and sequences. Journal of Cognitive Neuroscience, 7, 311 –336. Durstewitz, D., Seamans, J. K., & Sejnowski, T. J. (2000). Dopaminemediated stabilization of delay-period activity in a network model of prefrontal cortex. Journal of Neurophysiology, 83, 1733– 1750. Everling, S., Spantekow, A., Krappmann, P., & Flohr, H. (1998). Eventrelated potentials associated with correct and incorrect responses in a cued antisaccade task. Experimental Brain Research, 118, 27 –34. Ewert, J. P., Schurg-Pfeiffer, E., & Schwippert, W. W. (1996). Influence of pretectal lesions on tectal responses to visual stimulation in anurans: field potential, single neuron and behavior analyses. Acta Biologica Hungarica, 47, 89 –111. Eyny, Y. S., & Horvitz, J. C. (2003). Opposing roles of D1 and D2 receptors in appetitive conditioning. Journal of Neuroscience, 23, 1584–1587. Fiala, J., Grossberg, S., & Bullock, D. (1996). Metabotropic glutamate receptor activation in cerebellar Purkinje cells as substrate for adaptive timing of the classically conditioned eye-blink response. Journal of Neuroscience, 16, 3760–3774. Fischer, B., & Weber, H. (1993). Express saccades and visual attention. Behavioral and Brain Sciences, 16, 553 –610. Flaherty, A. W., & Graybiel, A. M. (1991). Corticostriatal transformations in the primate somatosensory system. Projections from physiologically mapped body-part representations. Journal of Neurophysiology, 66, 1249–1263. Frank, M. J., Loughry, B., & O’Reilly, R. C. (2001). Interactions between the frontal cortex and basal ganglia in working memory: A computational model. Cognitive, Affective, and Behavioral Neuroscience, 1, 137–160. Funahashi, S., Chafee, M. V., & Goldman-Rakic, P. S. (1993). Prefrontal neuronal activity in rhesus monkeys performing a delayed anti-saccade task. Nature, 365, 753–756. Gancarz, G., & Grossberg, S. (1999). A neural model of saccadic eye movement control explains task-specific adaptation. Vision Research, 39, 3123–3143. Gaspar, P., Bloch, B., & Le Moine, C. (1995). D1 and D2 receptor gene expression in the rat frontal cortex: cellular localization in different classes of efferent neurons. European Journal of Neuroscience, 7, 1050–1063. Gerfen, C., & Wilson, C. (1996). The basal ganglia. In L. W. Swanson, A. Bjorklund, & T. Hokfelt (Eds.), Handbook of chemical neuroanatomy (Vol. 12) (pp. 371–468). Integrated systems of the CNS, Part III, Holland: Elsevier Science B.V. Gernert, M., Hamann, M., Bennay, M., Loscher, W., & Richter, A. (2000). Deficit of striatal parvalbumin-reactive GABAergic interneurons and decreased basal ganglia output in a genetic rodent model of idiopathic paroxymal dystonia. Journal of Neuroscience, 20, 7052–7058. Giuffrida, R., Volsi, G. L., Maugeri, G., & Perciavalle, V. (1985). Influences of the pyramidal tract on the subthalamic nucleus in the cat. Neuroscience Letters, 54, 231 –235. Goldman-Rakic, P. S. (1987). Circuitry of primate prefrontal cortex and regulation of behavior by representational memory. In V. B. Mountcastle, F. Plum, & S. R. Geiger (Eds.), (Vol. I) (pp. 373 –417). Handbook of physiology, New York: Oxford University Press. Goldman-Rakic, P. S. (1995). Toward a circuit model of working memory and the guidance of voluntary motor action. In J. Houk, J. Davis, & D. Beiser (Eds.), Models of information processing in the basal ganglia (pp. 131 –148). Cambridge, MA: MIT Press. Gnadt, J. W., & Andersen, R. A. (1988). Memory related motor planning activity in posterior parietal cortex of macaque. Experimental Brain Research, 70, 216–220. Graybiel, A. M. (1998). The basal ganglia and chunking of action repertoires. Neurobiology of Learning and Memory, 70, 119–136. J.W. Brown et al. / Neural Networks 17 (2004) 471–510 Graybiel, A. M., Aosaki, T., Flaherty, A., & Kimura, M. (1994). The basal ganglia and adaptive motor control. Science, 265, 1826–1831. Gross, C. G., Desimone, R., Albright, T. D., & Schwartz, E. L. (1985). Inferior temporal cortex and pattern recognition. Experimental Brain Research, Supplement, 11, 179– 201. Grossberg, S. (1971). On the dynamics of operant conditioning. Journal of Theoretical Biology, 33, 225– 255. Grossberg, S. (1973). Contour enhancement, short term memory, and constancies in reverberating neural networks. Studies in Applied Mathematics, 52, 213 –257. Grossberg, S. (1978a). A theory of human memory: self-organization and performance of sensory-motor codes, maps, and plans. On Progress in Theoretical Biology, Vol. 5 (R. Rosen and F. Snell, eds), pp. 233–374. New York: Academic Press. Grossberg, S. (1978b). Behavioral contrast in short term memory: serial binary memory models or parallel continuous memory models. Journal of Mathematical Psychology, 3, 199–219. Grossberg, S. (1982a). Processing of expected and unexpected events during conditioning and attention: a psychophysiological theory. Psychological Review, 89, 529 –572. Grossberg, S. (1982b). Studies of mind and brain. Boston: Kluwer/Reidel Press. Grossberg, S., & Kuperstein, M. (1986). Neural dynamics of adaptive sensory-motor control: Ballistic eye movements. Holland: Elsevier. Grossberg, S., & Merrill, J. (1992). A neural network model of adaptively timed reinforcement learning and hippocampal dynamics. Cognitive Brain Research, 1, 3–38. Grossberg, S., Roberts, K., Aguilar, M., & Bullock, D. (1997). A neural model of multimodal adaptive saccadic eye movement control by superior colliculus. Journal of Neuroscience, 17, 9706– 9725. Grossberg, S., & Schmajuk, N. A. (1987). Neural dynamics of attentionally modulated Pavlovian conditioning: Conditioned reinforcement, inhibition and opponent processing. Psychobiology, 15, 195–249. Guitton, D., Buchtel, H. A., & Douglas, R. M. (1985). Frontal lobe lesions in man cause difficulties in suppressing reflexive glances and in generating goal-directed saccades. Experimental Brain Research, 58, 455–472. Gurden, H., Takita, M., & Jay, T. M. (2000). Essential role of D1 but not D2 receptors in the NMDA receptor-dependent long-term potentiation at hippocampal-prefrontal cortex synapses in vivo. Journal of Neuroscience, 20, RC106. Gurney, K., Prescott, T. J., & Redgrave, P. (2001a). A computational model of action selection in the basal ganglia: A new functional anatomy. Biological Cybernetics, 84, 401 –410. Gurney, K., Prescott, T. J., & Redgrave, P. (2001b). A computational model of action selection in the basal ganglia: Analysis and simulation of behavior. Biological Cybernetics, 84, 411–423. Hall, W. C., & Lee, P. (1993). Interlaminar connections of the superior colliculus in the tree shrew. I. The superficial gray layer. Journal of Comparative Neurology, 332, 213–223. Hallett, P. E. (1978). Primary and secondary saccades to goals defined by instructions. Vision Research, 18, 1279–1296. Hanes, D. P., Patterson, W. F. I. I., & Schall, J. D. (1998). Role of frontal eye fields in countermanding saccades: Visual, movement, and fixation activity. Journal of Neurophysiology, 79, 817–834. Hanes, D. P., & Schall, J. D. (1996). Neural control of voluntary movement initiation. Science, 274, 427 –430. Handel, A., & Glimcher, P. W. (2000). Contextual modulation of substantia nigra pars reticulata neurons. Journal of Neurophysiology, 83, 3042–3048. Handel, A., & Glimcher, P. W. (1999). Quantitative analysis of substantia nigra pars reticulata activity during a visually guided saccade task. Journal of Neurophysiology, 82, 3458–3475. Hartley, T. A., & Houghton, G. (1996). A linguistically constrained model of short-term memory for nonwords. Journal of Memory and Language, 35, 1– 31. 507 Hassani, O. K., Moroux, M., & Feger, J. (1996). Increased subthalamic neuronal activity after nigral dopaminergic lesion independent of disinhibition via the globus pallidus. Neuroscience, 72, 105–115. Hazrati, L. N., Parent, A., Mitchell, S., & Haber, S. N. (1990). Evidence for interconnections between the two segments of the globus pallidus in primates: a PHA-L anterograde tracing study. Brain Research, 533, 171– 175. Hemart, N., Daniel, H., Jaillard, D., & Crepel, F. (1995). Receptors and second messengers involved in long-term depression in rat cerebellar slices in vitro: a reappraisal. European Journal of Neuroscience, 7, 45– 53. Hikosaka, O., & Wurtz, R. H. (1983). Visual and oculomotor functions of monkey substantia nigra pars reticulata. IV. Relation of substantia nigra to superior colliculus. Journal of Neurophysiology, 49, 1285–1301. Hikosaka, O., Sakamoto, M., & Usui, S. (1989). Functional properties of monkey caudate neurons I. Activities related to saccadic eye movements. Journal of Neurophysiology, 61, 780–798. Hikosaka, O., & Wurtz, R. H. (1989). The basal ganglia. In R. Wurtz, & M. Goldberg (Eds.), The neurobiology of sacccadic eye movements (pp. 257– 281). Amsterdam: Elsevier. Hodgkin, A. L. (1964). The conduction of the nervous impulse. Springfield, IL: Thomas. Hoesen, G. W. V., Yeterian, E. H., & Lavizzo-Mourey, R. (1981). Widespread corticostriate projections from temporal cortex of the rhesus monkey. Journal of Comparative Neurology, 199, 205 –219. Horak, F. B., & Anderson, M. E. (1984). Influence of globus pallidus on arm movements in monkeys, II. Effects of stimulation. Journal of Neurophysiology, 52, 305– 322. Horak, F. B., Shupert, C. L., & Mirka, A. (1989). Components of postural dyscontrol in the elderly: A review. Neurobiology of Aging, 10, 727– 738. Hoshi, E., Shima, K., & Tanji, J. (1998). Task-dependent selectivity of movement-related neuronal activity in the primate prefrontal cortex. Journal of Neurophysiology, 80, 3392–3397. Houk, J., Adams, J., & Barto, A. (1995). A model of how the basal ganglia generate and use neural signals that predict reinforcement. In J. Houk, J. Davis, & D. Beiser (Eds.), Models of information processing in the basal ganglia (pp. 249–270). Cambridge, MA: MIT Press. Houk, J. C., & Wise, S. P. (1995). Distributed modular architectures linking basal ganglia, cerebellum, and cerebral cortex: Their role in planning and controlling action. Cerebral Cortex, 2, 95–110. Humphrey, D. R., & Reed, D. J. (1983). Separate cortical systems for control of joint movement and joint stiffness: Reciprocal activations and coactivation of antagonist muscles. In J. E. Desmedt (Ed.), Motor control mechanisms in health and disease. (Advances in Neurology, 39, 347– 372). New York: Raven Press. Hutsler, J. J., & Chalupa, L. M. (1991). Substance P immunoreactivity identifies a projection from the cat’s superior colliculus to the principal tectorecipient zone of the lateral posterior nucleus. Journal of Comparative Neurology, 312, 379–390. Iriki, A., Pavlides, C., Keller, A., & Asanuma, H. (1991). Long-term potentiation of thalamic input to the motor cortex induced by coactivation of thalamocortical and corticocortical afferents. Journal of Neurophysiology, 65, 1435–1441. Iwabuchi, A., & Kubota, K. (1998). Laminar organization of neuronal activities in area 8 of rhesus monkeys during a symmetrically reinforced visual go/no-go task. International Journal of Neuroscience, 94, 1–25. Jacobs, R. A., Jordan, M. I., Nowland, S. J., & Hinton, G. E. (1991). Adaptive mixtures of local experts. Neural Computation, 3, 79–87. Jaeger, D., Kita, H., & Wilson, C. J. (1994). Surround inhibition among projection neurons is weak or nonexistent in the rat neostriatum. Journal of Neurophysiology, 72, 2555–2558. Jones, E. G., Coulter, J. D., Burton, H., & Porter, R. (1977). Cells of origin and terminal distribution of corticostriatal fibers arising in the sensory-motor cortex of monkeys. Journal of Comparative Neurology, 173, 53–80. 508 J.W. Brown et al. / Neural Networks 17 (2004) 471–510 Kalaska, J. F., & Crammond, D. J. (1995). Deciding not to GO: neuronal correlates of response selection in a GO/NOGO task in primate premotor and parietal cortex. Cerebral Cortex, 5, 410–428. Kang, Y., & Kayano, F. (1994). Electrophysiological and morphological characteristics of layer VI pyramidal cells in the cat motor cortex. Journal of Neurophysiology, 72, 578–591. Kawaguchi, Y. (1997). Neostriatal cell subtypes and their functional roles. Neuroscience Research, 27, 1–8. Kawaguchi, Y., Wilson, C. J., Augood, S. J., & Emson, P. C. (1995). Striatal interneurones: chemical, physiological and morphological characterization. Trends in Neurosciences, 18, 527 –535. Kemel, M. L., Desban, M., Gauchy, C., Glowinski, J., & Besson, M. J. (1988). Topographical organization of efferent projections from the cat substantia nigra pars reticulata. Brain Research, 455, 307–323. Kermadi, I., Jurquet, Y., Arzi, M., & Joseph, J. P. (1993). Neural activity in the caudate nucleus of monkeys during spatial sequencing. Experimental Brain Research, 94, 352–356. Kimura, A., Caria, M. A., Melis, F., & Asanuma, H. (1994). Long-term potentiation within the cat motor cortex. Neuroreport, 5, 2372– 2376. Kita, H., Kosaka, T., & Heizmann, C. W. (1990). Parvalbuminimmunoreactive neurons in the rat neostriatum: A light and electron microscope study. Brain Research, 536, 1–15. Klopf, A. H. (1982). The hedonistic neuron: a theory of memory, learning, and intelligence. Washington, DC: Hemisphere. Kobatake, E., & Tanaka, K. (1994). Neuronal selectivities to complex object features in the ventral visual pathway of the macaque cerebral cortex. Journal of Neurophysiology, 71, 856–867. Komatsu, H., & Ideura, Y. (1993). Relationships between color, shape, and pattern selectivities of neurons in the inferior temporal cortex of the monkey. Journal of Neurophysiology, 70, 677 –694. Koos, T., & Tepper, J. M. (1999). Inhibitory control of neostriatal projection neurons by GABAergic interneurons. Nature Neuroscience, 2, 467 –472. Law-Tho, D., Desce, J. M., & Crepel, F. (1995). Dopamine favours the emergence of long-term depression versus long-term potentiation in slices of rat prefrontal cortex. Neuroscience Letters, 188, 125–128. Lee, C., Rohrer, W. H., & Sparks, D. L. (1988). Population coding of saccadic eye movements by neurons in the superior colliculus. Nature, 332, 357–360. Levesque, M., Charara, A., Gagnon, S., Parent, A., & Deschenes, M. (1996). Corticostriatal projections from layer V cells in rat are collaterals of longrange corticofugal axons. Brain Research, 709, 311–315. Levy, R., Hazrati, L.-N., Herrero, M.-T., Vila, M., Hassani, O.-K., Mouroux, M., Ruberg, M., Asensi, H., Agid, Y., Feger, J., Obeso, J., Parent, A., & Hirsch, E. C. (1997). Re-evaluation of the functional anatomy of the basal ganglia in normal and parkinsonian states. Neuroscience, 76, 335–343. Ljungberg, T., Apicella, P., & Schultz, W. (1992). Responses of monkey dopamine neurons during learning of behavioral reactions. Journal of Neurophysiology, 67, 145– 163. Lo, F.-S., & Sherman, S. M. (1994). Feedback inhibition in the cat’s lateral geniculate nucleus. Experimental Brain Research, 100, 365–368. Logan, G. D., & Cowan, W. B. (1984). On the ability to inhibit thought and action: A theory of an act of control. Psychologial Review, 91, 295–327. Lynch, J. C., Hoover, J. E., & Strick, P. L. (1994). Input to the primate frontal eye field from the substantia nigra, superior colliculus, and dentate nucleus demonstrated by transneuronal transport. Experimental Brain Research, 100, 181–186. Marin, O., Sweets, W. J., & Gonzalez, A. (1998). Evolution of the basal ganglia in tetrapods: A new perspective based on recent studies in amphibians. Trends in Neurosciences, 21, 487–494. Martin, K. E., Phillips, J. G., & Iansek, R. (1994). Inaccuracy and instability of sequential movements in Parkinson’s disease. Experimental Brain Research, 102, 131–140. Martres, M. P., Bouthenet, M. L., Sales, N., Sokoloff, P., & Schwartz, J. C. (1985). Widespread distribution of brain dopamine receptors evidenced with [125I] iodosulpride, a highly selective ligand. Science, 228, 752–755. Mayer, M. L., Westbrook, G. L., & Guthrie, P. B. (1984). Voltagedependent block by Mg2 þ of NMDA responses in spinal cord neurones. Nature, 309, 261–263. McFarland, N. R., & Haber, S. N. (2000). Convergent inputs from thalamic motor nuclei and frontal cortical areas to the dorsal striatum in the primate. Journal of Neuroscience, 20, 3798–3813. Middleton, F. A., & Strick, P. L. (2000). Basal ganglia and cerebellar loops: motor and cognitive circuits. Brain Research Reviews, 31, 236 –250. Mink, J. W. (1996). The basal ganglia: focused selection and inhibition of competing motor programs. Progress in Neurobiology, 50, 381– 425. Mink, J. W., & Thach, W. T. (1993). Basal ganglia intrinsic circuits and their role in behavior. Current Opinion in Neurobiology, 3, 950– 957. Mohler, C. W., Goldberg, M. E., & Wurtz, R. H. (1973). Visual receptive fields of frontal eye field neurons. Brain Research, 61, 385–389. Munoz, D. P., & Wurtz, R. H. (1993). Fixation cells in monkey superior colliculus I. Characteristics of cell discharge. Journal of Neurophysiology, 70, 559–575. Munoz, D. P., & Wurtz, R. H. (1995). Saccade-related activity in monkey superior colliculus I. Characteristics of burst and buildup cells. Journal of Neurophysiology, 73, 2313– 2333. Mushiake, H., Inase, M., & Tanji, J. (1991). Neuronal activity in the primate premotor, supplementary, and precentral motor cortex during visually guided and internally determined sequential movements. Journal of Neurophysiology, 66, 705–718. Nambu, A., Tokuno, H., & Takada, M. (2002). Functional significance of the the cortico-subthalamo-pallidal hyperdirect pathway. Neuroscience Research, 43, 111–117. Oorschot, D. E. (1996). Total number of neurones in the neostriatal, pallidal, subthalamic, and substantia nigral nuclei of the rat basal ganglia: a stereological study using the cavalieri and optical dissector methods. Journal of Comparative Neurology, 366, 580–599. Otani, S., Auclair, N., Desce, J. M., Roisin, M. P., & Crepel, F. (1999). Dopamine receptors and groups I and II mGluRs cooperate for longterm depression induction in rat prefrontal cortex through converging postsynaptic activation of MAP kinases. Journal of Neuroscience, 19, 9788–9802. Ottes, F. P., Van Gisbergen, J. A., & Eggermont, J. J. (1984). Metrics of saccade responses to visual double stimuli: Two different modes. Vision Research, 24, 1169–1179. Parent, A., & Hazrati, L. N. (1995). Functional anatomy of the basal ganglia. II. The place of the subthalamic nucleus and external pallidum in basal ganglia circuitry. Brain Research Reviews, 20, 128–154. Parthasarathy, H. B., Schall, J. D., & Graybiel, A. M. (1992). Distributed but convergent ordering of corticostriatal projections: analysis of the frontal eye field and the supplementary eye field in the macaque monkey. Journal of Neuroscience, 12, 4468–4488. Passingham, R. (1993). The frontal lobes and voluntary action. Oxford: Oxford University Press. Pellizzer, G., & Hedges, J. H. (2003). Motor planning: effect of directional uncertainty with discrete spatial cues. Experimental Brain Research, 150, 276–289. Pierrot-Deseilligny, C., Israel, I., Berthoz, A., Rivaud, S., & Gaymard, B. (1993). Role of the different frontal lobe areas in the control of the horizontal component of memory-guided saccades in man. Experimental Brain Research, 95, 166–171. Platt, M. L., & Glimcher, P. W. (1999). Neural correlates of decision variables in parietal cortex. Nature, 400, 233–238. Redgrave, P., Prescott, T. J., & Gurney, K. (1999). The basal ganglia: A vertebrate solution to the selection problem? Neuroscience, 89, 1009–1023. Reynolds, J. N., Hyland, B. I., & Wickens, J. R. (2001). A cellular mechanism of reward-related learning. Nature, 413, 67–70. Royce, G. J., & Bromley, S. (1984). Florescent double labeling studies of thalamostriatal and corticostriatal neurons. In J. S. McKenzie, R. E. Kemm, & L. N. Wilcock (Eds.), The basal ganglia (pp. 131 –146). New York: Plenum Press. J.W. Brown et al. / Neural Networks 17 (2004) 471–510 Sawaguchi, T. (2001). Laminar and topographic distributions of prefrontal cortical neurons related to an oculomotor delayed response task in monkeys. Society for Neuroscience Abstracts, 27, 85220. Schall, J. D. (1991). Neuronal activity related to visually guided saccades in the frontal eye fields of rhesus monkeys: Comparison with supplementary eye fields. Journal of Neurophysiology, 66, 559 –579. Schall, J. D., & Bichot, N. P. (1998). Neural correlates of visual and motor decision processes. Current Opinion in Neurobiology, 8, 211– 217. Schall, J. D., Hanes, D. P., Thompson, K. G., & King, D. J. (1995b). Saccade target selection in frontal eye field of macaque. I. Visual and premovement activation. Journal of Neuroscience, 15, 6905–6918. Schall, J. D., Morel, A., King, D. J., & Bullier, J. (1995a). Topography of visual cortex connections with frontal eye field in macaque: convergence and segregation of processing streams. Journal of Neuroscience, 15, 4464–4487. Schiller, P. H. (1998). The neural control of visually-guided eye movements. In J. E. Richards (Ed.), Cognitive neuroscience of attention. Mahwah, NJ: Lawrence Erlbaum Assoc. Schiller, P. H., True, S. D., & Conway, J. L. (1980). Deficits in eye movements following frontal eye field and superior colliculus ablations. Journal of Neurophysiology, 44, 1175–1189. Schlag, J., & Schlag-Rey, M. (1987). Evidence for a supplementary eye field. Journal of Neurophysiology, 57, 179 –200. Schlag-Rey, M., Amador, N., Sanchez, H., & Schlag, J. (1997). Antisaccade performance predicted by neuronal activity in the supplementary eye field. Nature, 390, 398 –401. Schlag-Rey, M., & Schlag, J. (1984). Visuomotor functions of central thalamus in monkey. II. Unit activity related to visual events, targeting, and fixation. Journal of Neurophysiology, 51, 1175– 1195. Schultz, W. (1998). Predictive reward signals of dopamine neurons. Journal of Neurophysiology, 80, 1– 27. Segraves, M. A., & Goldberg, M. E. (1987). Functional properties of corticotectal neurons in the monkey’s frontal eye field. Journal of Neurophysiology, 58, 1387–1419. Shink, E., Bevan, M. D., Bolam, J. P., & Smith, Y. (1996). The subthalamic nucleus and the external pallidum: two tightly interconnected structures that control the output of the basal ganglia in the monkey. Neuroscience, 73, 335–357. Sigala, N., & Logothetis, N. K. (2002). Visual categorization shapes feature selectivity in the primate temporal cortex. Nature, 415, 318 –320. Skinner, R. D., & Garcia-Rill, E. (1990). Brainstem modulation of rhythmic functions and behaviors. In W. R. Klemm, & R. P. Vertes (Eds.), Brainstem mechanisms of behavior (pp. 465 –496). New York: Wiley. Smith, Y., & Bolam, J. P. (1990). The output neurones and the dopaminergic neurones of the substantia nigra receive a GABAcontaining input from the globus pallidus in the rat. Journal of Comparative Neurology, 296, 47–64. Sommer, M. A., & Tehovnik, E. J. (1999). Reversible inactivation of macaque dorsomedial frontal cortex: effects on saccades and fixations. Experimental Brain Research, 124, 429 –446. Sommer, M. A., & Wurtz, R. H. (1998). Frontal eye field neurons orthodromically activated from the superior colliculus. Journal of Neurophysiology, 80, 3331–3333. Sommer, M. A., & Wurtz, R. H. (2000). Composition and topographic organization of signals sent from the frontal eye field to the superior colliculus. Journal of Neurophysiology, 83, 1979–2001. Steele, G. E., & Weller, R. E. (1993). Subcortical connections of subdivisions of inferior temporal cortex in squirrel monkeys. Visual Neuroscience, 10, 563–583. Steiner, H., & Gerfen, C. R. (1998). Role of dynorphin and enkephalin in the regulation of striatal output pathways and behavior. Experimental Brain Research, 123, 60 –76. Strick, P. L., Dum, R. P., & Picard, N. (1995). Macro-organization of the circuits connecting the basal ganglia with the cortical motor areas. In J. Houk, J. Davis, & D. Beiser (Eds.), Models of information processing in the basal ganglia (pp. 117– 130). Cambridge: MIT Press. Sung, K. W., Choi, S., & Lovinger, D. M. (2001). Activation of group I 509 mGluRs is necessary for induction of long-term depression at striatal synapses. Journal of Neurophysiology, 86, 2405–2412. Suri, R. E., Albani, C., & Glattfelder, A. H. (1997). A dynamic model of motor basal ganglia functions. Biological Cybernetics, 76, 451–458. Suri, R. E., Bargas, J., & Arbib, M. A. (2001). Modeling functions of striatal dopamine modulation in learning and planning. Neuroscience, 103, 65– 85. Sutton, R. S., & Barto, A. G. (1981). Toward a modern theory of adaptive networks: expectation and prediction. Psychological Review, 88, 135– 170. Sweeney, J. A., Mintun, M. A., Kwee, S., Wiseman, M. B., Brown, D. L., Rosenberg, D. R., & Carl, J. R. (1996). Positron emission tomography study of voluntary saccadic eye movements and spatial working memory. Journal of Neurophysiology, 75, 454–468. Takikawa, Y., Kawagoe, R., Itoh, H., Nakahara, H., & Hikosaka, O. (2002). Modulation of saccadic eye movements by predicted reward outcome. Experimental Brain Research, 142, 284–291. Tanaka, K., Saito, H., Fukada, Y., & Moriya, M. (1991). Coding visual images of objects in the inferotemporal cortex of the macaque monkey. Journal of Neurophysiology, 66, 170–189. Taylor, J. G., & Taylor, N. R. (2000). Analysis of recurrent cortico-basal ganglia-thalamic loops for working memory. Biological Cybernetics, 82, 415–432. Thompson, K. G., Bichot, N. P., & Schall, J. D. (1997). Dissociation of visual discrimination from saccade programming in macaque frontal eye field. Journal of Neurophysiology, 77, 1046–1050. Turner, R. S., & Anderson, M. E. (1997). Pallidal discharge related to the kinematics of reaching movements in two dimensions. Journal of Neurophysiology, 77, 1051–1074. Turner, R. S., & DeLong, M. R. (2000). Corticostriatal activity in primary motor cortex of the macaque. Journal of Neuroscience, 20, 7096–7108. Turner, R. S., Grafton, S. T., Votaw, J. R., Delong, M. R., & Hoffman, J. M. (1998). Motor subcircuits mediating the control of movement velocity: a PET study. Journal of Neurophysiology, 80, 2162–2176. Vezina, P., & Kim, J. H. (1999). Metabotropic glutamate receptors and the generation of locomotor activity: interactions with midbrain dopamine. Neuroscience and Biobehavioral Reviews, 23, 577–589. Weiss, P., Stelmach, G. E., & Hefter, H. (1997). Programming of a movement sequence in Parkinson’s disease. Brain, 120, 91– 102. Wichmann, T., Bergman, H., & DeLong, M. R. (1994). The primate subthalamic nucleus. Journal of Neurophysiology, 72, 494–506. Wichmann, T., & DeLong, M. (1996). Functional and pathophysiological models of the basal ganglia. Current Opinion in Neurobiology, 6, 751– 758. Wichmann T., & DeLong, M., (2001). Basal ganglia circuits in movement and movement disorders. In K. Kultas-Llinsky and I. A. Llinsky (Eds.). Basal ganglia and thalamus in health and movement disorders (pp. 1125). Kluwer Academic. Wickens, J. (1997). Basal ganglia: Structure and computations. Network: Computation in Neural Systems, 8, R77–R109. Wickens, J., Begg, A., & Arbuthnott, G. (1996). Dopamine reverses the depression of rat corticostriatal synapses which normally follows high-frequency stimulation of cortex in vitro. Neuroscience, 70, 1–5. Wilson, C. (1995a). The contribution of cortical neurons to the firing pattern of striatal spiny neurons. In J. Houk, J. Davis, & D. Beiser (Eds.), Models of information processing in the basal ganglia (pp. 29– 50). Cambridge, MA: MIT Press. Wilson, C. (1995b). Corticostriatal neurons of the medial agranular cortex of rats. In M. Kimura, & A. Graybiel (Eds.), Functions of the corticobasal ganglia loop (pp. 50–72). Tokyo: Springer-Verlag. Wilson, C. J., Kita, H., & Kawaguchi, Y. (1989). GABAergic interneurons, rather than spiny cell axon collaterals, are responsible for the IPSP responses to afferent stimulation in neostriatal spiny neurons. Society for Neuroscience Abstracts, 15, 907. 510 J.W. Brown et al. / Neural Networks 17 (2004) 471–510 Wise, S. P., & Murray, E. A. (2000). Arbitrary associations between antecedents and actions. Trends in Neurosciences, 23, 271 –276. Wurtz, R. H., Sommer, M. A., Pare, M., & Ferraina, S. (2001). Signal transformations from cerebral cortex to superior colliculus for the generation of saccades. Vision Research, 41, 3399–3412. Yeterian, E. H., & Hoesen, G. W. V. (1978). Cortico-striate projections in the rhesus monkey: The organization of certain cortico-caudate connections. Brain Research, 139, 43 –63. Yeterian, E. H., & Pandya, D. N. (1994). Laminar origin of striatal and thalamic projections of the prefrontal cortex in rhesus monkeys. Experimental Brain Research, 99, 383 –398. Young, A. B., & Penney, J. B. (2001). What we know and what we have left to learn. In K. Kultas-Ilinsky, & I. A. Ilinsky (Eds.), Basal ganglia and thalamus in health and movement disorders (pp. 3–10). Dordrecht: Academic Press. Zheng, T., & Wilson, C. J. (2002). Corticostriatal combinatorics: The implications of cortico-striatal axonal arborizations. Journal of Neurophysiology, 87, 1007–1017. Zin-Ka-Ieu, S., Roger, M., & Arnault, P. (1998). Direct contacts between fibers from the ventrolateral thalamic nucleus and frontal cortical neurons projecting to the striatum: a light-microscopy study in the rat. Anatomy and Embryology (Berlin), 197, 77–87.
* Your assessment is very important for improving the work of artificial intelligence, which forms the content of this project