I am fascinated with Yang et al.’s paper on the potential of LMMs and the variety of tasks they show promising potential in. Here are some of the questions I have:

• Regarding replicability and consistency: I wonder if the examples provided can be replicated with similar/identical prompts with reasonable variance. The example that GPT-4V can correctly count the number of apples after being told “you are an expert in counting” in the prompt seems both fascinating and a bit incredible. I am trying to replicate some of these examples with my individual experiment, but at the same time, I am wondering in what way can I reliably establish consistency and eliminate confounding variables in producing the output.
• As I read Dretske’s “What Good is Consciousness” in Mind, Brain and Meaning class, an interesting philosophical problem is, what good is multimodality? An easy question is that humans have multimodal processing capabilities, and it would be great if AI can too. But a subtler point is, what are the ways that humans uniquely benefit or take advantage of our multimodal capabilities (per Dretske’s discussion, if there is a benefit to multimodality per se) that we are trying to have AI imitate or mimic? Specifically, Dretske makes the distinction between object-awareness and fact-awareness, and this could be interesting in evaluating ChatGPT’s output on its propositional knowledge about an input (say, a picture) versus its ability to recognize an object in the picture.
• On the section of explaining jokes and memes, I am especially impressed by GPT-4V’s capability for two reasons beyond giving a satisfactory answer to why certain jokes and memes are funny. (1) GPT-4V is able to point out multiple plausible reasons why someone could find a meme funny, and all of them sound reasonable enough. (2) People don’t explain explicitly why jokes are supposed to be funny or how they find them funny, so this might not be abundant in the training dataset, but GPT-4V is able to do a pretty impressive job. This might be taking a curious stab at the study of jokes and funniness, which is something that I am interested in philosophically too.