Hello everyone,
Time flies and we’re entering week 8! This week, we’ll focus primarily on exploring the social and ethical questions of LLMs. The diversity and complexity of these issues are well-reflected in our second debate and our readings for the week. As outlined in the syllabus, I recommend starting with a quick review of the abstracts of all papers, then selecting two or three for a more in-depth analysis.
A must-read for everyone is the first chapter of Goldsmith and Laks. It offers a comprehensive overview and a “scientific-social diagnosis” of how linguistics, computation, and cognition first collided with one another in history, which is very useful and highly relevant for us to reflect on how the social and scientific issues are intertwined for the AI advancements today.
- Monday: I’ll conclude our discussion on multimodal creativity and do a comprehensive survey on the topic of social and ethical challenges associated with LLMs.
- Wednesday: The session will be dedicated to this week’s experiment: exploring the approaches of word embedding and ChatGPT in addressing bias and fairness. Be prepared for a group discussion and to collectively present your examples, similar to our previous in-class activities on word embedding and evaluative strategies of LLMs.
Additionally, if you haven’t yet talked about your individual experiments on scientific or startup ideas, plan to do so on either Monday or Wednesday. I’ll try to allocate 10-15 minutes each day for that.
Lastly, the deadline for your debate two report is midnight on Wednesday, November 15. Please send it to my email: yuji1@uchicago.edu. Remember you will get a final grade boost if you post it on our course blog.
Looking forward to an engaging Week 8!
Eugene
Week 8 (November 13 and 15): (Re-)Visiting the Cognitive, Social, and Ethical Aspects of LLMs
(Please first glance at all the materials of this week and then choose at least three topics to do close reading depending on your background and interests.)
– Assignments:
Goldsmith, John & Bernard Laks (2019). “Chapter 1: Battle in the Mind Fields” (1 – 51) from Battle in the Mind Fields. The University of Chicago Press.
Kasirzadeh, Atoosa and Iason Gabriel (2023). “In Conversation with Artificial Intelligence: Aligning Language Models with Human Values.” Philosophy and Technology, 36, 27. https://doi.org/10.1007/s13347-023-00606-x
Törnberg, Petter (2023). “ChatGPT-4 Outperforms Experts and Crowd Workers in Annotating Political Twitter Messages with Zero-shot Learning.” arxiv.org: https://arxiv.org/pdf/2304.06588.pdf
Sourati, Jamshid & Evans, James. A (2023): “Accelerating (social) science with human-aware artificial intelligence.” Nature Human Behavior.
(The online talk about this paper is here: https://www.youtube.com/watch?v=nAmfMs5scto. Feel free to read/watch either or both).
Deshpande, Ameet et al. (2023). “Toxicity in ChatGPT: Analyzing Persona-assigned Language Models.” arxiv.org: http://arxiv.org/pdf/2304.05335.pdf
Garg, Nikhil et al. (2018). “Word Embeddings Quantify 100 Years of Gender and Ethnic Stereotypes.” PNAS, 115 (16) E3635-E3644. https://www.pnas.org/doi/epdf/10.1073/pnas.1720347115
– Experiments:
(a) Revisit the word embedding demo from week 2
(http://vectors.nlpl.eu/explore/embeddings/en/#). Does the word embedding model show any sign of social or cultural bias or fairness? How to evaluate that?
(b) Experiment how much ChatGPT can handle problems of bias and fairness. How to evaluate that?
In both experiments, we need to think about how to “operationalize” the concepts of bias and fairness.
November 15: Debate reflective report due (if you opt for writing on the second debate).