Syllabus for Perspectives on Large Language Models: Computational, Cognitive, Social at The University of Chicago (Cognitive Science 20100, Linguistics 20110, Fall 2023)
Instructor: Eugene Yu Ji (yuji1@uchicago.edu)
Time and Location: Mon and Wed 3:00 – 4:20 pm Harper Mem Library 130
Office hours: Weds 8 – 9 pm Online or by appointment
Description: In this interdisciplinary course, students will delve into the multifaceted world of large language models (LLMs), investigating their computational, cognitive, and social dimensions. The course covers an array of topics, such as the history and evolution of language models, computational underpinnings and training methodologies, cognitive and social aspects of language understanding, knowledge production, communication, and creativity, as well as crucial ethical and social considerations, encompassing fairness, alignment, toxicity. We will examine the scientific and practical applications and limitations of LLMs across diverse domains and contemplate the future prospects and challenges LLMs pose for science, technology, and society. Through lectures, discussions, debates, hands-on exercises, and case studies, our goal is to foster a comprehensive understanding of LLMs, empowering students to critically assess these models, create their own LLMs-based projects, and contribute to ongoing dialogues regarding LLMs and AI’s broader implications. Prior experience in cognitive science, linguistics, or computer science is beneficial but not mandatory.
Note: This course primarily focuses on cultivating reflective and critical thinking about LLMs more than direct programming, and most in-class implementations will involve evaluating and experimenting with ChatGPT in multiple scientific and applied dimensions. Students with programming and other quantitative skills are, however, encouraged to utilize them to facilitate their learning. Please familiarize yourself with every week’s materials before each lecture/discussion. Active participation in class is expected.
General requirements: There are two meetings per week, usually with one lecture introducing the main topic of the week and one discussion session led by the instructor. Students are required to do readings before lectures/discussions and actively participate in lectures, discussions, and debates. Grades are composed of weekly posts (15%), midterm debate reflective report (20%), group project presentation and project report (30%), Take-home final (25%), and participation (10%).
Weekly posts (15%): ~300 words each reflecting on any aspects of the reading(s) of the week. You can choose to skip one weekly response without penalty. Readings can be found either through the link on the syllabus or on Canvas. Due by 12 pm every Monday starting from week 2 to week 9.
Debates and midterm debate reflective report (20%) (choose one debate only): There are two in class debates (see the sections of weeks 5 and 7 on the syllabus about the topic), one on Wednesday of week 5 (Oct. 25) and the other on Wednesday of week 7 (Nov. 8). The student can choose either one of the two debates to write a 3 -5 page reflective report. The report needs to summarize the debate and formulate one’s own critical arguments on the topic and the debate itself. The reflective report is due in one week after each debate (Wednesday Nov. 1 and Wednesday Nov. 15, respectively).
Group project presentation and final project report (30%): Students form project groups (2 – 3 people per group) in week 3, and each group proposes a project topic by the end of week 6. Groups will showcase the projects on the last day of the class on November 29. Students are required to write a 7 – 9 page project report and submit them by Friday December 8. The report should be independently written by each individual of the group.
Take-home final (25%): Prompts for the take-home final will be released on the first day of the exam week (December 4), and the deadline for submitting your exams is Saturday December 9. The exam will include two parts:
Part one is a short essay question based on one of the given essay prompts. The short essay needs to be around 4 – 6 page long and draw materials from at least three pieces of reading or video assignments through the quarter.
Part two is an experimentation evaluation, in which the student is asked to critically evaluate one prompt example of human-ChatGPT interaction.
Familiarity with readings and participation (10%): Students are expected to read materials prior to the class, and actively participate in discussions, debates, and lectures. If you have any difficulties in speaking out in the classroom, please let the instructor know as early as possible.
Page formatting: Double-spaced, Times New Roman 12.
Notice: This course is about creatively experimenting with LLMs, so please be innovative and bold in that respect (e.g., you are strongly encouraged to explore how much ChatGPT can help you understand the assigned readings). However, as we navigate the explorative waters, it is essential to adhere to the university’s standards for academic honesty and integrity (https://college.uchicago.edu/student-services/academic-integrity-student-conduct). If ever in doubt, please consult the university guidelines or engage in a discussion with the instructor. Please also note that any experimentations or implementations from this course must not be applied to assignments or exams in other university classes without explicit permission from the respective instructor.
Week 1 (September 27): Introduction
– Assignments:
Q&A with Richard Feynman (September 1985): “Can Machines Think?”
Week 2 (October 2 and 4): What is a Language Model?
– Assignments:
Goldsmith, John & Bernard Laks (2019). “The Era of Machines” (231 – 237) from Chapter 4 “Psychology: 1900 – 1940”, and “The Chrome Machine of Logic” (461 – 477) and “The Logicians’ Grammar” (477 – 485) from Chapter 8 “Logic: 1900 – 1940”. Battle in the Mind Fields. The University of Chicago Press.
Simon, Herbert & Allen Newell (1964). “Information Processing in Computer and Man.” American Scientist, 52, 281–300.
Weaver, Warren (1953). “Recent Contributions to The Mathematical Theory of Communication.” A Review of General Semantics, 10, 4: 261–81.
Part of Chris Manning’s Stanford lecture on Word Vector Representation (2017): https://www.youtube.com/watch?v=ERibwqs9p38&t=2020s (3:51 – 22:56 only).
– Experiments:
Play with the word embedding demo at http://vectors.nlpl.eu/explore/embeddings/en/#. Each of you should try five to ten target words as your inputs. Please evaluate the results first by yourself and we will discuss them in class on Wednesday October 4.
Week 3 (October 9 and 11): How to Make Language Models Large and Generative and Why?
– Assignments:
Jaspreet (2016). “A Concise History of Neural Networks.” Towards Data Science https://towardsdatascience.com/a-concise-history-of-neural-networks-2070655d3fec.
John Ewald (Google Cloud Tech)’s short introduction to large language models (15 mins):
Join Ellie Pavlick’s crash course in generative AI in the 2023 MIT GenAI summit (20 mins): https://www.youtube.com/watch?v=f5Cm68GzEDE
Ouyang et al. (2022). “Training Language Models to Follow Instructions with Human Feedback.” arxiv.org: https://arxiv.org/pdf/2203.02155.pdf (This is the paper directly behind the creation of ChatGPT!)
Optional: Trouvron et al. (2023). “LLaMA: Open and Efficient Foundation Language Models.” arxiv.org: https://arxiv.org/pdf/2302.13971.pdf (LLaMA is MetaAI’s equivalent model of OpenAI’s ChatGPT).
– Experiments:
Please use ChatGPT or other applications available to you to read the Ouyang et al. (2022). It’s a long paper and you certainly don’t need to understand every page of it. Instead, try to explore possible ways that are most fruitful to you to digest any aspect(s) of the paper you are interested in. Don’t skip the seemingly long and tedious appendix sections – a lot of fun and important training/feedback prompt examples actually appear there!
Please evaluate two to three prompt examples from Ouyang et al. (2022) that you find most interesting or puzzling, and be prepared to share and discuss them in Wednesday’s class.
Week 4 (October 16 and 18): Evaluating LLMs: Capacities
– Assignments:
Chomsky, Noam, Ian Roberts and Jeffrey Watumull (March 8, 2023): “The False Promise of ChatGPT.” The New York Times: https://www.nytimes.com/2023/03/08/opinion/noam-chomsky-chatgpt-ai.html
Brockmen, Greg (OpenAI’s co-founder)’s TED talk (April 20, 2023). “The Inside Story of ChatGPT’s Astonishing Potential” (30 mins): https://www.youtube.com/watch?v=C_78DM8fG6E
Li et al. (2023). “Counterfactual Reasoning: Testing Language Models’ Understanding of Hypothetical Scenarios.” arxiv.org: https://arxiv.org/pdf/2305.16572.pdf
Goldstein, Ariel et al. (2022). Shared computational principles for language processing in humans and deep language models. Nature neuroscience, 25(3), 369-380: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC8904253/pdf/41593_2022_Article_1026.pdf
– Experiments:
Please use ChatGPT or other similar applications to evaluate a specific capacity and a specific limitation that you think are interesting and/or significant. How to justify your evaluations?
Week 5 (October 23 and 25): Evaluating LLMs: Knowledge and Education
Abbott, Andrew (2016): “The Future of Expert Knowledge.” Lecture at Hamburger Institut für Sozialforschung (9:15 – 1:05:47): https://www.youtube.com/watch?v=yVrP2DOHeq0&t=616s
Roose, Kevin (August 24, 2023): “How Schools Can Survive (and Maybe Even Thrive) With A.I.” The New York Times: https://www.nytimes.com/2023/08/24/technology/how-schools-can-survive-and-maybe-even-thri ve-with-ai-this-fall.html
– First class debate (October 25):
Will ChatGPT and other similarly powerful AI tools have positive or negative impacts on higher education? In your opinion, what sociologist Andrew Abbott’s criticisms in 2016 would hold or not hold for accessing today’s large language models like ChatGPT?
Week 6 (October 30 and November 1) Evaluating LLMs: Science and Society
– Assignments:
Griffiths, Thomas (2015). “Manifesto for a New (computational) Cognitive Revolution.” Cognition, 135. 21-23. https://cocosci.princeton.edu/tom/papers/ComputationalCognitiveRevolution.pdf
(For the following two papers: please glance at both first and do close reading on at least one of them)
Birhane, Abeba et al. (2023). “Science in the Age of Large Language Models.” Nature Review, 5, 277–280. https://doi.org/10.1038/s42254-023-00581-4;
Krenn, Mario et al. (2022). “On Scientific Understanding with Artificial Intelligence.” Nature Review, 4, 761–769. https://www.nature.com/articles/s42254-022-00518-3
Andrew Ng’s Stanford Lecture (August 29, 2023). “Opportunities in AI” (37 mins):
Geoffery Hinton’s interview: “ ‘Godfather of Artificial Intelligence’ Talks Impact and Potential of AI” (March 25, 2023). CBS Saturday Morning (42 mins): https://www.youtube.com/watch?v=qpoRO378qRY. Also see: Metz, Cade (May 1, 2023). “ ‘The Godfather of A.I.’ Leaves Google and Warns of Danger Ahead.” The New York Times: https://www.nytimes.com/2023/05/01/technology/ai-google-chatbot-engineer-quits-hinton.html.
– Experiments:
Use ChatGPT to explore an idea of research (e.g., How digital information affects cognition of children in poverty) or an idea of a startup in industry (e.g., developing a new app teaching people playing musical instruments). Ask as many questions as you wish to ChatGPT to render the idea as specific, concrete, and practical as possible. Does it work out as you wish to?
November 1: Debate reflective report due (if you opt for writing on the first debate).
By the end of Week 6: Topics of group projects due.
Week 7 (November 6 and 8): Evaluating LLMs: Creativity and Multimodality
– Assignments:
Daston, Lorraine (2022). “Algorithmic Intelligence in the Age of Calculating Machines” from Rules, 122 -150. Princeton University Press.
Malihe, Alikhani et al. (2023). “Text Coherence and its Implications for Multimodal AI: Frontier in Artificial Intelligence, Section of Language and Computation, Volume 6: http://www.frontiersin.org/articles/10.3389/frai.2023.1048874/full
Yang et al. (2023). “The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision).” arxiv.org: https://arxiv.org/pdf/2309.17421.pdf
Hsu, Tiffany and Steven Lee Myers (April 8, 2023). “Can We No Longer Believe Anything We See?” The New York Times: https://www.nytimes.com/2023/04/08/business/media/ai-generated-images.html
OpenAI’s Dall E.3: https://openai.com/dall-e-3 (Dall E.3 is an image-and-text LLM model that can work together with ChatGPT Plus, which was released by Open AI on September 21 2023. We will show and discuss some examples generated by Dall E.3 in class).
Optional (vision and touch): Guzey et al. (2023). “See to Touch: Learning Tactile Dexterity through Visual Incentives.” arxiv.org: https://arxiv.org/pdf/2309.12300.pdf
Optional (text and speech): Meta AI’s Introduction to its new multimodal AI model for speech translation (released in August 22, 2023): https://ai.meta.com/blog/seamless-m4t/
– Second class debate (November 8):
Will AI’s creativity threaten or replace humans’ creativity? Use cases from course materials or your own examples to defend your position.
Week 8 (November 13 and 15): (Re-)Visiting the Cognitive, Social, and Ethical Aspects of LLMs
(Please first glance at all the materials of this week and then choose at least three topics to do close reading depending on your background and interests.)
– Assignments:
Goldsmith, John & Bernard Laks (2019). “Chapter 1: Battle in the Mind Fields” (1 – 51) from Battle in the Mind Fields. The University of Chicago Press.
Kasirzadeh, Atoosa and Iason Gabriel (2023). “In Conversation with Artificial Intelligence: Aligning Language Models with Human Values.” Philosophy and Technology, 36, 27. https://doi.org/10.1007/s13347-023-00606-x
Törnberg, Petter (2023). “ChatGPT-4 Outperforms Experts and Crowd Workers in Annotating Political Twitter Messages with Zero-shot Learning.” arxiv.org: https://arxiv.org/pdf/2304.06588.pdf
Sourati, Jamshid & Evans, James. A (2023): “Accelerating (social) science with human-aware artificial intelligence.” Nature Human Behavior.
(The online talk about this paper is here: https://www.youtube.com/watch?v=nAmfMs5scto. Feel free to read/watch either or both).
Deshpande, Ameet et al. (2023). “Toxicity in ChatGPT: Analyzing Persona-assigned Language Models.” arxiv.org: http://arxiv.org/pdf/2304.05335.pdf
Garg, Nikhil et al. (2018). “Word Embeddings Quantify 100 Years of Gender and Ethnic Stereotypes.” PNAS, 115 (16) E3635-E3644. https://www.pnas.org/doi/epdf/10.1073/pnas.1720347115
– Experiments:
(a) Revisit the word embedding demo from week 2 (http://vectors.nlpl.eu/explore/embeddings/en/#). Does the word embedding model show any sign of social or cultural bias or fairness? How to evaluate that?
(b) Experiment how much ChatGPT can handle problems of bias and fairness. How to evaluate that?
In both experiments, we need to think about how to “operationalize” the concepts of bias and fairness.
November 15: Debate reflective report due (if you opt for writing on the second debate).
Thanksgiving week: No class
Week 9 (November 27): Upcoming Directions of LLMs and the Future of AI/Human Knowing
(Please first glance at all the materials of this week and then choose at least three topics to do close reading depending on your background and interests. Reviewing relevant readings/materials from previous weeks is strongly encouraged.)
– Assignments:
Overview: “AI for the Next Era: OpenAI’s Sam Altman on the New Frontiers of AI”, podcast by Greylock (September 13, 2022): https://greylock.wpengine.com/greymatter/sam-altman-ai-for-the-next-era/
Pragmatic knowledge: Gubelmann, Reto et al. (2023). “When Truth Matters – Addressing Pragmatic Categories in Natural Language Inference (NLI) by Large Language Models (LLMs).” In Proceedings of the 12th Joint Conference on Lexical and Computational Semantics (SEM 2023), 24–39, Toronto, Canada. Association for Computational Linguistics. https://aclanthology.org/2023.starsem-1.4.pdf
Embodied AI: Czerski, Elisa (2023). “Embodied AI: How will it define our future?” Rise of AI Conference 2023 (27 mins): https://www.youtube.com/watch?v=DHNQ6jD-t3w
Multimodal AI: Please revisit materials from Week 7.
AI-aided science: Please revisit materials from Week 6.
Human/AI Knowing: Abbott, Andrew (2009): “The Future of Knowing.” Lecture at the University of Chicago Alumni Association: https://home.uchicago.edu/aabbott/Papers/futurek.pdf
– Experiments:
(a). Please review Warren Weaver’s article “Recent Contributions to The Mathematical
Theory of Communication.” from Week 2. Illustrate how ChatGPT may have or have not realized the vision Weaver proposed in 1953.
(b). Are the developments or insights from this week and previous weeks’ materials aligned with Weaver’s vision? Can you use ChatGPT and/or other resources to assist in addressing this question?
Week 9 (November 29): Group project presentations
December 4: Take-home final prompts released
December 8: Group project report due
December 9: Take-home final due