Leading AI models can replicate general human survey results but consistently fail to capture the counterintuitive weirdness of real human thought.
April 26, 2026
Original Paper
Stochastic Parrots or Singing in Harmony? Testing Five Leading LLMs for their Ability to Replicate a Human Survey with Synthetic Data
SSRN · 6644227
The Takeaway
LLMs tend to parrot conventional wisdom and average out their answers into a consensus that lacks novelty. When tested against real human surveys, they miss the outliers and unique insights that drive human progress. This proves that while AI is great at summarization, it cannot yet simulate the spark of original or unconventional thinking. The models are effectively trapped in a harmony of existing data. This means that for tasks requiring genuine creative leaps or contrarian insights, humans are still irreplaceable.
From the abstract
How well can AI-derived synthetic research data replicate the responses of human participants? This article presents a comparison between a human-respondent survey of 420 Silicon Valley coders and developers and "synthetic" survey data designed to simulate real organizational survey takers generated by five leading Generative AI Large Language Models (LLMs): ChatGPT Thinking 5 Pro, Claude Sonnet 4.5 Pro plus Claude Code / CoWork 1.123, Gemini Advanced 2.5 Pro, Incredible 1.0, and DeepSeek 3.2. A