In a recent article, Qualtrics' Derrick McLean made the case that the model behind your synthetic research tool matters more than the interface polished on top of it. That's something I agree with, even if I part ways with much of the argument that follows.
Full disclosure before I go further: I co-founded FishDog, which competes in this category. So please read what follows as an argument, not a neutral survey. Derrick knows this space well, and his piece is a serious one, which is exactly why it's worth taking apart rather than nodding along to.
I think Derrick is asking an important question but then proceeds to answer it with the wrong axis. I'd like to take a close look at his argument because I believe that it results in the wrong conclusion.
The ladder, and the true rung
Derrick's argument goes like this. There are three ways to build synthetic respondents.
Prompt-engineered personas are a toy, fine for qualitative poking, useless for numbers.
Retrieval-grounded twins are more realistic but drag their source biases along with them.
And fine-tuned models, retrained on real survey data to reproduce how humans answer surveys, are the only approach with the statistical integrity to survive a conjoint or a factor analysis.
Qualtrics sells the third option, and claims it is 12 times more accurate than a general model at predicting human survey responses.
Let me start where I am in agreement. General-purpose language models do flatten variance, massively so. Ask a stock model to role-play a demographic and it gives you the median of everything it has ever read about that demographic. The edges disappear, the disagreement disappears.
You end up with a plausible, articulate, suspiciously agreeable answer that clusters around what the persona typically believes. If you're making multi-million dollar decisions, that is a real failure mode.
So the diagnosis is correct. It's the prescription that I think goes sideways.
Fine-tuning fixes the symptom by training a better patient
If the problem is that a general model produces a flattened, too-agreeable answer, the Qualtrics fix is to fine-tune the model on real survey responses until its outputs match how real people fill out surveys.
Now, let's think a minute about what that optimises for. You are training a model to be an excellent survey respondent. Not an excellent representation of a person, an excellent representation of a person answering a survey.
Those are not the same thing, and the gap between those two is the oldest problem in research: the say-do gap. The difference between what someone tells a clipboard and what they do on a Tuesday is one of the most stable findings in behavioural science. Traditional surveys capture the said. Fine-tuning a model on survey data teaches it to reproduce the said, with higher fidelity, at scale.
This doesn't escape the failure mode of survey research, it now automates it. The 12x-more-accurate number, whatever its provenance, is a measure of how well the model predicts survey answers. If survey answers were the ground truth, we would not have a market research crisis. We would have market research.
The ladder is pointed the wrong way
Let's step back for a minute and look at the three rungs together. Prompt personas, retrieval twins, fine-tuned models. Every rung is a different technique for making one synthetic individual answer more like a real individual would. The axis is response fidelity: how convincingly does a manufactured respondent behave.
In my opinion, that is the wrong axis, and the giveaway here is the middle rung. The “digital twin” framing, cloning individuals and hoping the clone inherits the right instincts, has always been a construction problem that mimics a fidelity problem. When you build a synthetic person, you decide what goes into them. Every one of those decisions is a place for your own assumptions and biases to move in because creation is where bias lives.
The question that matters isn't how faithfully does each fake person answer. It's who is in the room, and does that room look like the market. That is a population question, not a model-tuning question, and no amount of fine-tuning on the top rung fixes a room you assembled out of your own priors on the way in.
Recruit, don't create
Now we're arriving at the fork in the category, and it's the bet that we're building FishDog on.
You can create a synthetic audience or you can recruit one.
Creation starts from a specification. You decide the audience should be 60 percent female, skew urban, index high on convenience, and you generate people to match. It's fast, it's controllable, and it almost guarantees that the data can only tell you what you already assumed, because you assumed it into existence.
Recruitment starts the other way around. You draw from a grounded population built on real profile data, you screen for the people who actually fit the brief, and you let the composition of the market fall out of the recruiting rather than the spec. Representativeness lives in recruitment, not in creation, for the same reason a good panel provider's value is in who they can reach, not in how convincingly they can fake a respondent.
Fine-tuning is a creation technique. A better creation technique than prompting, I agree. But it optimises the wrong end of the pipe. It makes each manufactured answer smoother while leaving untouched the question of whether the sample was ever the market to begin with.
The average is the lie
Here's the part that should worry a decision-maker most, and it's where the fine-tuned-model pitch works against the buyer.
We recently ran a study on how senior leaders actually consume research, 100 respondents, VP and above. Not one of the 100 trusted a report that opened with consensus. Ninety-two of them acted on divergence instead, on the split, the range, the who-disagrees-with-whom. The recurring line, in different words, was that the average is the lie. The moment you compress a market to a single confident number, you have deleted the only thing that tells a decision-maker where the risk and the opportunity sit.
Now recall what a model fine-tuned to predict survey responses is engineered to produce. A tighter, more accurate, more defensible central estimate. It is optimised, by construction, to be a better average. That's precisely the artefact the people paying for research told us they distrust and don't act on. A synthetic method whose headline virtue is a smoother mean is solving for the thing that already fails.
The output that changes a decision is the distribution: how many said zero, how many said twenty, whether the fringe is growing or shrinking, and which real, nameable respondents hold which view. Preserve the disagreement and you have something a person can defend in a room. Flatten it into a confident number and you have expensive theatre.
Where the other side has a point
For narrow, well-bounded, high-frequency survey tasks, tracking a stable attitude in a large mainstream population, a model tuned on years of matching survey data will do that specific job well, and might genuinely beat a naive prompted persona on calibration.
If your decision truly rests on a single mainstream point estimate that a traditional survey would also have given you, the fine-tuned approach is a reasonable, faster substitute.
And I could be wrong about the size of that carve-out. If it turns out most research budgets are spent chasing exactly those stable mainstream estimates, then the response-fidelity axis matters more than I'm giving it credit for. I don't believe that's where the valuable decisions live, but it's the place my argument could break.
But the place it doesn't break is validation, and validation is where the “12x” number belongs in context. In independent testing, EY found synthetic survey results correlate 95 percent with human responses, and the most interesting work is understanding where the other 5 percent goes, not selling the correlation as a trophy.
A number without its failure mode attached is marketing. The bar we hold ourselves to is that a claim be true and not misleading, which means shipping the limits alongside the wins.
The model matters. So does the question you point it at.
McLean is right that the model matters and that interface polish is a distraction from it. I'd go further than he does. The model is one layer. The population feeding it is another, and it's the one that decides whether your synthetic study reflects a market or reflects you.
Fine-tune all you like. If you created the room instead of recruiting it, and if you tuned the model to hand you a cleaner average, you've built a faster way to get a confident answer to the wrong question. The synthetic research worth paying for doesn't ask which model imitates a survey-taker best. It asks who's in the population, whether they were recruited or invented, and whether the disagreement survived the trip to the slide.
If it doesn't change what you build, buy, or believe, it's just expensive theatre. That test doesn't care which model you fine-tuned.