Author: sunnylee

How memory meets perception during naturalistic scene understanding

As we navigate our visual world, what we see is seamlessly integrated with our memory of the broader environment. One long-standing puzzle in neuroscience is how perceptual and mnemonic systems – which are topographically distinct in the brain – interface to give rise to memory-guided visual experience. In this talk, I will present a series of studies using head-mounted virtual reality and fMRI to explore this question. First, we investigated the neural basis of scene perception and memory using fine-grained individual-subject fMRI. To our surprise, this analysis revealed a new network of brain areas that collectively bridge the scene perception and spatial memory systems of the human brain, the “place memory network” (Steel et al., Nat Coms, In Press). Specifically, we reveal three areas, each lying immediately anterior to a region of the scene perception network in posterior cerebral cortex, that selectively activate when recalling familiar real-world locations. Despite their close proximity to the scene-perception areas, network analyses show that these regions constitute a distinct functional network that interfaces with memory systems during naturalistic scene understanding. We hypothesize that these regions may provide remembered, contextual representations to support ongoing perception. In a second study, we use head-mounted virtual reality (VR) and behavior to test whether memory for a broad spatial environment, in fact, impacts ongoing scene perception in behavior. Using a priming paradigm, we show that a briefly presented scene view reinstates associated views of the panoramic environment, facilitating subsequent perception. Together, these studies illuminate how perceptual and mnemonic systems interact during naturalistic visual experience, and point to a new mechanistic step for understanding how the brain implements memory-guided visual behaviors in posterior cerebral cortex.

Do Hering’s sensations determine lexical color categories?

In principle, languages could create lexical color categories that partition color space in culture-specific ways. Nonetheless, the color categories in the lexicons of world languages are strikingly similar. Why is this so? One traditional explanation is based on Hering’s elemental sensations (redness, greenness, blueness and yellowness), which demarcate privileged regions of color space. Several recent studies have challenged the special status of Hering’s sensations. Are they the cause or the by-product of “red”, “green”, “blue” and “yellow” color categories of the Indo-European languages? Here, we describe the results of three studies that address this issue. The first two studies examined the understanding of Hering’s elemental sensations in subjects speaking languages that are missing terms for some of these sensations. Color naming by the Hadza people is sparse and distributed: most speakers do not use terms for all the Hering sensations (or most other basic color categories of IndoEuropean languages), yet each person uses a different subset of the terms. Thus, the language as a whole demonstrates a complete Hering lexicon, though no idiolect does so. Somali-speaking observers use a term “grue” that does not respect the color boundaries of the Hering color terms, and particularly their term for “yellow” often names colors of every hue. A third study examined the elemental sensations in English-speaking deuteranomalous trichromats. Their initial physiological encoding of color differed from that of color-normal observers. Nevertheless, their choices of broadband lights corresponding to the unique and binary hues of Hering’s theory were similar, but not identical, to those of color normal observers. These results highlight the importance of language in mediating color understanding but are inconsistent with the view that culture alone guides color category formation.

Neural signatures of attentional engagement during narratives and its consequences for event memory

“I’m in an encoding state of mind”: Neural mechanisms of memory formation

How do we successfully form new memories? A classic approach to assessing memory formation is the comparison of activity patterns during the study of items that will later be remembered compared to the study of items that will later be forgotten — a contrast dubbed the subsequent memory effect (SME). Although this approach has been fruitful, there are many reasons why we may forget, meaning that the SME contrast is a coarse assay of memory formation. Here we use alternative approaches in which we refine the SME contrast and investigate the role of mnemonic states in memory formation. We record scalp EEG or fMRI while participants perform memory tasks and use a combination of univariate and multivariate approaches to assess successful memory formation. We find that neural SME signals reflect memory organization processes and that mnemonic states, estimated through multivariate analysis methods, may be better predictors of subsequent memory than traditional univariate signals. A central goal of our future work is to identify the factors that induce mnemonic states and to link the engagement of mnemonic states to performance across cognitive tasks.

Lingua Franca Use as a Hidden Barrier to Conflict Resolution

Leigh Burnett

When international negotiation faces a communication barrier a solution is to use a lingua franca that is native to neither side, presumably putting the parties on equal footing through the use of a common language. Here we show that despite common belief, using a lingua franca erects hidden barriers to conflict resolution. In three studies in which native Hebrew speaking Israelis evaluated an outline for a peace building proposal, they consistently perceived it less favorably when it was in English than when it was presented in their native Hebrew. The use of the lingua franca elicits a less positive emotional response as compared to the use of a native language, thereby reducing how favorably the proposal was evaluated as being for their side. This has implications for cross-national negotiation in diplomacy as well as commerce.

Environmental influences on thought content and connectedness

Kathryn Schertz

Exposure to natural environments, compared to urban environments, has generally been shown to be beneficial for positive affect, pro-social thinking, and feelings of
connectedness. As most studies use pre- and post-exposure testing, the time course of these changes in thoughts and feelings is unclear. Additionally, individual differences, such as trait impulsivity, which may mediate the relationship between environmental exposure and changes in affect and cognition, are less well understood. I will present a study where we investigated these questions by surveying participants at three timepoints during a one hour walk in both a natural and urban environment. Our results showed that differences between positive affect and environmental connectedness were detected by the first survey time point (20 minutes), and most differences remained significant throughout the entire walk. We did not replicate the association of trait impulsivity with changes in positive affect. However, exploratory analysis uncovered several other traits that should be further investigated as individual difference measures that may influence whether an individual reaps the benefits of spending time in natural environments.

Description’s role in language and concept learning

Learning from description seems straightforward: if you hear about a “blue dax,” you might think you should learn that daxes are likely to be blue. But, counterintuitively, you should do the opposite. This is because people describe the rare or interesting features of objects, and not their typical features. If you hear about a “blue dax,” it is likely that most daxes are not blue. In this talk, I will draw on corpus analyses of conversation, language modeling, and experiments with adults to suggest that learning from description poses a problem for associative models of language and concept learning. Corpus analyses of parent–child conversations show that when talking to children as young as 14 months old, parents use description to highlight the surprising, atypical features of things (e.g., “brown apple”) rather than their typical, generalizable features (e.g., “red apple”). That is, people remark on features that are surprising given their world knowledge, even when talking to a child whose world knowledge is still nascent. Because language is structured in this way, models of word meaning that associate co-occurring words (here, Word2vec) fail to capture typicality relationships between nouns and adjectives well. Finally, adults take into account that description remarks upon the remarkable when interpreting utterances about novel categories (e.g., “Pass me the blue dax”): they do not simply associate the descriptor with the novel category, but infer that the descriptor points out an atypical feature (e.g., that it is rare for daxes to be blue). Overall, we find that people produce and interpret description based on principles of informative communication—not veridical description of the world—and this raises problems for associative models of language and concept learning as well as natural language processing.

Cognitive Effort and Decision-making: Integrating Computational, Behavioral, And Psychophysiological Approaches

Our ability to perform tasks is constrained by our limited mental resources, which mandates that people should minimize use of cognitively “effortful” processing when possible. Recent theories posit that decisions to expend effort are governed by a cost-benefit tradeoff, whereby the potential benefits of effort can offset its perceived costs. I will present a series of recent, computationally-informed experiments combining behavioural experimentation and pupillometry to gain critical insights into understanding when and why we allocate—or withhold—cognitive effort, both from an individual differences perspective, and at the level of the task by examining the effect of changes in costs and benefits. We find that individual differences in cognitive capacity—and relatedly, intrinsic motivation—govern trial-to-trial adjustments to cognitive effort expenditure in accordance with shifts in costs (i.e., opportunity costs) and benefits (i.e., rewards). Further, we find that task-evoked pupillary responses can elucidate internal computations of these effort allocation decisions.