“Understanding” and prediction in artificial intelligence models
Allyson Ettinger, Assistant Professor, Department of Linguistics
October 29, 11:30am-1pm
Joseph Regenstein Library, 122
In recent years, the field of natural language processing (NLP) in artificial intelligence has made what appears to be incredible progress, with models even surpassing human performance on certain evaluations. How should we interpret these advances? Have these models achieved so-called language “understanding”? Operating on the premise that “understanding” will necessarily involve the capacity to extract the meaning of language inputs, in this talk I will discuss a series of projects leveraging targeted tests to examine NLP models’ ability to capture linguistic meaning in a systematic fashion. I will first discuss work probing model representations for compositional meaning, with a particular focus on disentangling compositional information from encoding of lexical properties. I’ll then explore models’ ability to extract and deploy meaning information during word prediction, applying tests inspired by psycholinguistics to examine the types of information that models are able to encode and access for anticipating words in context. In all cases, these investigations draw on insights and methods from linguistics and cognitive science in order to maintain human-driven standards for what constitutes language “understanding”, and to ensure that tests are adequately controlled. The results of these studies suggest that although NLP models show a good deal of sensitivity to word-level information, and to a number of semantic and syntactic distinctions, they show little sign of capturing higher-level compositional meaning, of handling logical impacts of meaning components like negation, or of retaining access to detailed representations of meaning information conveyed in prior context. I will discuss implications of the findings both for currently dominant model paradigms in NLP, and for the study of language processing in humans.
Dr. Allyson Ettinger’s research is focused on language processing in humans and in artificial intelligence systems, motivated by a combination of scientific and engineering goals. For studying humans, her research uses computational methods to model and test hypotheses about mechanisms underlying the brain’s processing of language in real time. In the engineering domain, her research uses insights and methods from cognitive science, linguistics, and neuroscience in order to analyze, evaluate, and improve natural language understanding capacities in artificial intelligence systems. In both of these threads of research, the primary focus is on the processing and representation of linguistic meaning.
If you need any additional accommodations to participate in the Forum, please contact Carmen Caswell (caswellc@uchicago.edu).
Lunch will be served, but registration is strongly encouraged to ensure an accurate order count.
Due to University of Chicago policies to ensure the health of the community during the COVID-19 pandemic, in-person attendance at this event is restricted to members of the University of Chicago community only. However, if you would like to hear the talk remotely, please complete the form below.