Expectation-based syntactic processing in humans and machines

Abstract:
Psycholinguistics and computational linguistics are the two fields most dedicated to accounting for the computational operations required to understand natural language. Today, both fields find themselves responsible for understanding the behaviors and inductive biases of "black-box" systems: the human mind and artificial neural network (ANN) models, respectively. In this talk I highlight how the two fields can productively contribute to one another, with a focus on the study of syntactic processing. I first describe the surprisal theory of interpretation, prediction, and differential processing difficulty in human language comprehension. I then show how surprisal theory and controlled experimental paradigms from psycholinguistics can help us probe ANN language model behavior for evidence of human-like grammatical generalizations. We find that ANNs exhibit a range of subtle behaviors, including embedding-depth tracking and garden-pathing over long stretches of text, that suggest representations homologous to incremental syntactic state in human language processing. These ANNs also learn abstract word-order preferences and many generalizations about the long-distance filler-gap dependencies that are a hallmark of natural language syntax, perhaps most surprisingly including many filler-gap "island" constraints. However, even when trained on a human lifetime's worth of linguistic input these ANNs fail to learn a number of key basic facts about other core grammatical dependencies. Finally, I comment on the respects in which the departures of recurrent neural network language models from the predictions of the "competence" grammars developed in generative linguistics might provide a "performance" account of human language processing -- and on the respects in which they might not.

Bio:
Roger Levy is a professor in the Department of Brain and Cognitive Sciences at the Massachusetts Institute of Technology. He directs MIT's Computational Psycholinguistics Laboratory. Before joining MIT, he was faculty in the Department of Linguistics at UC San Diego and received his PhD in Linguistics from Stanford University. His research focuses on theoretical and applied questions in the processing and acquisition of natural language.