Quicksand The Replacements Quotes

For many kinds of science, another time-consuming and otherwise complicated step involves surveying or interviewing participants, also called human subjects. This is difficult: it’s often hard to recruit an appropriate sample of the population of interest, or find ways to ask questions that get at the topic of interest without causing harm to the people being asked (for instance, asking about past trauma without retraumatizing participants). Sometimes, what’s needed is expert opinions, but the relevant experts are too busy or are unwilling to do the relatively low-paid labor of providing the information required. But what if chatbots could be designed to answer questions as if they were people with different kinds of lived experience or different expertise? How convenient! You might hope at this point that we’re making this up, but researchers have actually proposed using “in silico” samples for political science surveys35 and psychological experiments.36 We’ve already discussed37 a form of this type of methodology in Chapter 3—in a paper written by OpenAI researchers, they determined what kinds of tasks in what kinds of jobs could be handled by an LLM by asking the LLM itself. But this idea is ludicrous on its face: “silicon samples” are, obviously, not real people. Even if researchers are “just asking questions” about whether this is a reasonable methodology, it’s misguided. When other researchers take their results and use it as a justification to use “silicon samples” in social science research, it replaces empirical foundations with quicksand.