Can you tell whether an online forum thread reflects genuine community dialogue, or just a single artificial intelligence chatting with itself? According to a new study published in Scientific Reports, advanced AI systems can now orchestrate realistic multi-user discussions that convincingly mimic genuine social media threads, deceiving human readers in a substantial portion of tests.
Passing the Group Turing Test
Researchers evaluated whether cutting-edge generative tools could reproduce the varied dynamics of human interactions found in comment sections on Reddit. Across the study’s evaluations, artificial conversations were mistaken for human-authored discussions 39% of the time. The illusion was particularly persuasive when examining threads created by Meta’s open model, Llama 3 70B: human participants correctly identified them as artificial only 56% of the time—a score scarcely better than random chance.
How the Experiment Was Conducted
To measure the realism of artificial social dynamics, the research team gathered authentic human discussion threads from Reddit across shared subjects. They then tasked two prominent large language models—OpenAI’s GPT-4o and Meta’s Llama 3 70B—with generating synthetic discussion threads on the exact same topics.
Human evaluators were shown human and AI-generated threads side-by-side to see whether they could spot the machine-made content. By requiring the AI to orchestrate multiple interacting personas, each displaying distinct tones and viewpoints, the setup expanded the classic Turing test from a traditional one-on-one dialogue into a rigorous “collective” test of social believability.
Opportunities for Digital Research
While the results reveal how easily humans can be fooled, the study highlights substantial benefits for digital science. High-fidelity social simulation provides scientists and engineers with a safe, synthetic sandbox to evaluate content recommendation algorithms prior to public deployment.
Rather than experimenting directly on living communities, social media platforms could leverage synthetic forums to model the outcomes of novel content moderation policies. Simulated environments could also help researchers measure the efficacy of digital mental health interventions and prosocial prompts without exposing real individuals to psychological risks or algorithmic disruptions.
The Danger of Synthetic Public Opinion
Alongside scientific opportunities, the researchers warn of severe threats to online information ecosystems. The ability to conjure diverse online communities out of thin air creates dangerous avenues for automated astroturfing, where bad actors can manufacture artificial consensus around controversial social or political debates.
Traditional bots are often simple to recognize due to repetitive phrasing, but multi-agent language models can simulate nuanced debates, supportive rejoinders, and varying viewpoints within a single thread. The authors caution that as synthetic group dynamics become indistinguishable from reality, social networks will face profound hurdles in defending authentic human discourse against deceptive disinformation campaigns.
Source
- The collective turing test: large language models can generate realistic multi-user discussions
- Researchers: Azza Bouleimen, Giordano De Marzo, Taehee Kim, Nicolò Pagan, H. Metzler, Silvia Giordano, Anikó Hannák, David García
- Journal: Scientific Reports
- Read the original study