This article is the one we would rather not write. But any tool that claims to have no limitations is not being honest, and honesty is rather the point of research. So here it is: a complete and candid account of what synthetic research cannot do, where it falls short, and the circumstances in which you should spend your budget on traditional methods instead. We sell [FishDog](https://fish.dog). We have every commercial incentive to present it as a solution to all research problems. It is not. What follows is our best attempt to tell you exactly where the boundaries are, so that you can make intelligent decisions about when to use synthetic research, when to use traditional research, and when to use both.
The Honest Case Against Ourselves
There is a particular genre of technology marketing that might be called the "limitless potential" school. The product can do anything. The platform scales infinitely. The AI understands everything. The customer testimonials are uniformly rapturous. The competitive comparison shows green ticks in every column and the competitor's column is a graveyard of red crosses. Nobody believes it, but everybody publishes it, because the alternative -- admitting that your product has genuine limitations -- feels like commercial suicide.
It is not. It is the opposite. When a company tells you honestly what their product cannot do, two things happen. First, you trust them more, because they have demonstrated that their interest in accuracy exceeds their interest in a sale. Second, you use the product more effectively, because you understand its operating parameters rather than discovering them through expensive failures.
This article is an exercise in the first principle. We build synthetic research tools. We believe they are genuinely useful for a wide range of product marketing, brand strategy, and go-to-market decisions. We have published nineteen articles in this series demonstrating specific applications, each with working methodology and real examples. But synthetic research has real limitations, and pretending otherwise would undermine the credibility of everything we have written so far.
The limitations fall into seven categories. Some are fundamental -- they derive from what synthetic personas are, and no amount of engineering will eliminate them. Others are practical -- they reflect the current state of the technology and may be resolved as the field matures. We will distinguish between the two, because the distinction matters for how you plan your research programme.
Limitation 1: Genuine Novelty
Synthetic personas are, at their core, sophisticated statistical models of human behaviour. They are trained on vast quantities of human-generated text, which means they carry the accumulated knowledge, opinions, preferences, and biases of the populations they represent. This is their strength: they can tell you how a 45-year-old marketing director in Manchester would likely respond to a new pricing model, because they have internalised the patterns of how such people think, decide, and communicate.
It is also their most fundamental limitation. If a product category does not yet exist, synthetic personas have no reference frame from which to evaluate it. They cannot tell you how consumers would react to the first smartphone in 2005, because the concept of a smartphone did not exist in the collective understanding that the personas draw upon. They would give you an answer -- synthetic personas always give you an answer, which is part of the danger -- but the answer would be an extrapolation from adjacent categories rather than a genuine reaction to something unprecedented.
This matters most for genuinely novel products. Not "novel" in the marketing sense, where every iterative improvement is described as "revolutionary," but novel in the structural sense: products that create new categories, that solve problems people do not yet know they have, or that require a conceptual leap to understand.
Consider the difference. A new flavour of sparkling water is not genuinely novel. Synthetic personas can evaluate it effectively, because they understand the sparkling water category, they have preferences about flavour profiles, and they can articulate what would make them choose one brand over another. A brain-computer interface for consumer use, by contrast, is genuinely novel. Synthetic personas will respond to questions about it, but their responses will be anchored to existing mental models -- they will compare it to things they already understand, like smartphones or voice assistants -- rather than engaging with the product on its own terms.
The practical test is straightforward: if you can explain your product by reference to existing categories ("it is like X but for Y," "it replaces Z with a better approach"), synthetic research will serve you well. If explaining your product requires the listener to first accept a premise they have never encountered, synthetic research will give you articulate but unreliable data. In such cases, you need real humans encountering the real product, because you are researching not just preferences but comprehension.
Limitation 2: Sensory and Physical Experience
This one is absolute. Synthetic personas cannot taste your coffee. They cannot feel the texture of your packaging. They cannot smell your candle, wear your shoes, sit in your chair, or experience the tactile satisfaction of pressing a well-engineered button on a physical product.
This sounds obvious, and it is, but the implications are more extensive than most people initially appreciate. A significant proportion of consumer product differentiation is sensory. The reason people pay a premium for certain chocolate brands is not the ingredient list -- it is the mouthfeel, the snap of the bar, the way it melts. The reason certain cosmetics command loyalty is the texture, the scent, the way the product feels on the skin. These are not incidental attributes. They are the product.
Synthetic research can tell you what consumers value in abstract terms. You can ask personas what matters to them when choosing chocolate, and they will tell you about taste, quality of ingredients, brand values, and price. This is useful for understanding the decision framework. But you cannot ask them whether your specific chocolate tastes better than a competitor's, because they have not tasted either. The response you would get would be an inference based on brand perception, ingredient quality, and price point -- which is interesting intelligence in its own right, but it is not sensory evaluation and should not be treated as such.
The boundary here is clear. If the research question involves the physical experience of a product, synthetic research is the wrong tool. Use it for everything around the physical experience -- brand perception, purchase drivers, price sensitivity, competitive positioning, packaging design preferences expressed in visual terms -- but do not use it as a substitute for sensory panels, product testing, or any methodology that requires the respondent to interact with a physical object.
Limitation 3: Extreme Emotional Contexts
Synthetic personas can model a wide range of human emotional responses. They can simulate frustration with a product experience, excitement about a new feature, anxiety about a purchasing decision, and the quiet satisfaction of a problem well solved. These are emotions that exist within the normal range of consumer and professional experience, and the personas model them credibly.
What they cannot model with confidence is extreme emotional context. Grief. Trauma. Crisis. The emotional states that fundamentally alter how a person processes information, makes decisions, and relates to the world around them.
This limitation matters for specific categories. Healthcare products marketed to patients with serious diagnoses. Financial services marketed to people experiencing bankruptcy or foreclosure. Insurance products marketed in the aftermath of catastrophic loss. Mental health services. Bereavement support. Crisis communication. In these contexts, the emotional state of the audience is not merely relevant to their decision-making -- it is the dominant factor, and it operates in ways that are difficult to simulate because they are difficult to generalise.
A synthetic persona modelling a recently bereaved person will produce responses that are plausible. They will reference appropriate emotions. They will mention the difficulty of decision-making during grief. But they will be drawing on aggregate patterns rather than inhabiting a specific emotional reality, and the difference matters when the stakes are high and the margin for error is low.
If your research involves populations in extreme emotional states, use traditional qualitative methods. Depth interviews conducted by trained researchers with appropriate ethical oversight. Focus groups with proper screening and support protocols. The data will be harder to collect, slower to analyse, and more expensive to obtain. It will also be real.
Limitation 4: Highly Specific Niche Populations
Synthetic personas are drawn from broad population models. They can be filtered by country, age, gender, income bracket, profession, industry, and a range of other demographic and psychographic attributes. This filtering is effective for populations that are statistically well-represented in the training data: American marketing managers, British software engineers, German automotive executives, Canadian healthcare administrators.
The filtering becomes less reliable as the target population becomes more specific. If your target audience is left-handed violin makers in rural Japan, the persona pool will not cover it. Not because the system is poorly designed, but because there is insufficient data about that specific intersection of attributes to generate personas that credibly represent it.
The practical threshold is roughly this: if your target population numbers in the tens of thousands or more, synthetic research will serve you well. If it numbers in the hundreds, the personas will be approximations drawn from adjacent populations rather than genuine representations of the specific group. If it numbers in the dozens, you should not use synthetic research at all -- you should find those actual people and talk to them, because they exist in small enough numbers that direct research is both feasible and necessary.
This limitation is worth considering carefully, because the temptation to over-specify is strong. A study of "enterprise CFOs at Fortune 500 companies evaluating treasury management software" is specific but still represents a population of several hundred people with well-documented characteristics. Synthetic research handles this well. A study of "chief risk officers at Nordic reinsurance firms with more than 15 years of experience in catastrophe modelling" is probably too narrow for reliable synthesis, because the global population matching that description might fit comfortably in a single conference room.
The honest guidance is: use synthetic research for populations that are defined by common attributes, and use traditional research for populations that are defined by rare combinations of attributes. The former plays to the strengths of statistical modelling. The latter requires the irreducible specificity of real human beings.
Limitation 5: Regulatory and Compliance Requirements
Certain industries require human-subject research by regulation, by established practice, or by professional standards that predate synthetic alternatives. Pharmaceutical clinical trials. Medical device usability testing. Food safety evaluation. Financial suitability assessments. Educational programme accreditation studies. These are contexts in which the regulatory framework specifies that research must involve actual human participants, and no synthetic alternative will satisfy the requirement regardless of its accuracy.
This is not a limitation of the technology in the intellectual sense -- synthetic research may well produce insights that are as accurate or more accurate than the traditional methods mandated by regulators. It is a limitation in the practical sense: the output will not be accepted by the people who need to accept it.
The nuance here is that synthetic research can still play a valuable supporting role in regulated industries, even if it cannot serve as the primary evidence base. A pharmaceutical company conducting a clinical trial cannot replace its human subjects with synthetic personas. But it can use synthetic research to design better trial protocols, test patient communication materials, evaluate marketing messages for a drug awaiting approval, or explore pricing and access strategy for a product that has not yet launched. The key is understanding which questions require regulatory-grade evidence and which require commercial intelligence. The former demands real human data. The latter is often better served by synthetic methods, precisely because it can be conducted faster, cheaper, and without the ethical complications of involving real patients in commercial research.
Limitation 6: Social Desirability Bias
This limitation is more subtle than the others, because it involves a problem that synthetic research was partly designed to solve -- and yet does not entirely eliminate.
Social desirability bias is the tendency for research participants to give answers they believe are socially acceptable rather than answers that reflect their actual views. In traditional research, this manifests as people overstating their environmentally conscious purchasing habits, understating their price sensitivity, claiming to read more than they do, and generally presenting an idealised version of themselves that bears a family resemblance to reality without being entirely faithful to it.
Synthetic personas exhibit a version of this bias, because the language models that generate them were trained on human-generated text, which itself reflects social desirability. When people write about their preferences online -- in reviews, surveys, social media posts, forum discussions -- they are already filtering through a social desirability lens. The personas inherit this filter.
The result is that synthetic personas can, in certain contexts, over-index on socially approved preferences. They may slightly overstate their willingness to pay a premium for sustainability. They may slightly understate their responsiveness to discount pricing. They may express more enthusiasm for innovative products than their behaviour would justify, because enthusiasm for innovation is socially rewarded in ways that cautious scepticism is not.
The word "slightly" is doing significant work in those sentences. Synthetic personas exhibit markedly less social desirability bias than real humans in a traditional research setting, because they are not sitting across from a moderator whose approval they unconsciously seek. They are not in a focus group where peer dynamics shape their responses. They are not completing a survey that was sent by the company whose product they are evaluating, with all the implicit pressure that entails. But the bias is not zero. It is reduced, not eliminated, and researchers should account for this in their interpretation.
The practical mitigation is straightforward: treat directional findings with confidence and precise magnitudes with appropriate scepticism. If seven out of ten synthetic personas say they would choose your product over a competitor, you can be confident in the preference direction. Whether the real-world ratio is seven out of ten or six out of ten is less certain, and decisions that hinge on that level of precision should be validated with traditional methods.
Limitation 7: Stated Preference vs Revealed Behaviour
This is perhaps the most important limitation, and it applies to all research -- not just synthetic research -- but it applies to synthetic research with particular force.
There is a well-documented gap between what people say they will do and what they actually do. This gap is not dishonesty. It is a fundamental feature of human cognition. People genuinely believe they will switch to the new product, exercise more, cook at home, cancel the subscription they never use, and read that book they bought six months ago. They mean it when they say it. And yet the subscription renews, the gym membership goes unused, and the book remains on the nightstand, gathering dust and moral authority in equal measure.
Synthetic personas inhabit the "stated preference" side of this divide. They can tell you what they would do, what they value, what they prefer, and how they would decide. They cannot tell you what they will actually do when confronted with the full complexity of real life -- the inertia, the time pressure, the competing priorities, the sheer gravitational pull of the status quo.
This limitation is not unique to synthetic research. Every survey, every focus group, every depth interview in the history of market research has produced stated preferences rather than revealed behaviour. The only methodologies that capture revealed behaviour are observational studies, A/B tests with real traffic, point-of-sale data, and actual market experiments. But the limitation is worth stating explicitly because the fluency of synthetic persona responses can create an illusion of behavioural certainty that exceeds what the data actually supports.
A synthetic persona that says "I would definitely switch to this product" is telling you that the stated preference, based on the information provided, favours switching. It is not telling you that the person will actually switch when the time comes. The gap between those two things is where a great deal of product launch optimism goes to die.
Where Synthetic Research Excels
Having catalogued seven limitations with what we hope is uncomfortable honesty, it is worth restating what synthetic research does exceptionally well. Not as a counterbalance to soften the criticism, but because an accurate understanding of any tool requires knowing both its boundaries and its strengths.
Speed. A synthetic research study can be designed, fielded, and analysed in minutes. Not days. Not weeks. Minutes. This is not a marginal improvement over traditional research timelines. It is a categorical change. It means that research can be embedded into decision-making processes that previously operated without evidence because there was no time to gather it. The product meeting that makes a positioning decision in an hour. The pricing discussion that happens on a Thursday for a Monday launch. The competitive response that needs to be formulated before end of day. These decisions have always been made. They have rarely been made with research support. Synthetic research changes that equation.
Scale. You can run a study with ten personas or a hundred. You can test twelve messaging variants in a single afternoon. You can run the same study across five countries simultaneously. Traditional research at this scale would require months of fieldwork, hundreds of thousands in budget, and a project management infrastructure that most companies simply do not have. Synthetic research makes scale a variable you can adjust rather than a constraint you must accept.
Cost. A synthetic study costs a fraction of its traditional equivalent. This matters not because companies want to spend less on research -- though they do -- but because it means research can be applied to decisions that were previously too minor to justify the investment. You would not commission a traditional research study to decide between two subject lines for an email campaign. You might commission a synthetic one, because the cost is negligible and the insight is genuine.
Hard-to-reach populations. Recruiting senior executives for traditional research is expensive and slow. Recruiting them in specific industries, at specific company sizes, in specific geographies, is exponentially harder. Synthetic panels make these populations accessible for research questions that do not require the real person in the room. You will not replace the executive advisory board with synthetic personas, but you can supplement it with synthetic data that extends your understanding beyond the twelve executives who agreed to participate.
Consistency. In traditional longitudinal research, panel attrition is a persistent problem. People drop out. They change jobs. They lose interest. They give more thoughtful answers in wave one than wave five. Synthetic panels do not have this problem. The same personas, with the same characteristics, can be queried at regular intervals with perfect consistency. This makes longitudinal tracking -- of brand perception, message resonance, competitive positioning -- genuinely practical rather than aspirationally planned and operationally abandoned.
The 80/20 Rule for Research Methodology
The most useful framework for thinking about synthetic versus traditional research is the 80/20 rule, applied honestly.
Approximately eighty per cent of the research questions a product marketing team encounters can be addressed effectively with synthetic methods. These are questions about messaging preference, competitive perception, pricing sensitivity at the directional level, brand positioning, feature prioritisation, purchase driver analysis, and market segmentation. They are questions where the goal is to understand patterns, preferences, and perceptions across a population -- precisely the territory where synthetic personas are most reliable.
The remaining twenty per cent requires traditional methods. These are questions where sensory experience matters, where regulatory compliance demands human subjects, where the target population is too narrow for reliable synthesis, where the emotional context is extreme, or where the gap between stated preference and revealed behaviour makes the stakes too high for anything less than observed human data.
The twenty per cent is not an afterthought. It is often the twenty per cent that matters most. The clinical trial. The sensory panel. The depth interview with the bereaved family member who is evaluating end-of-life care options. The usability test where you watch a real person struggle with your interface and learn things that no stated preference data would reveal. These are irreplaceable research activities, and any framework that suggests synthetic methods can substitute for them is doing a disservice to both the researcher and the researched.
The eighty per cent, however, is where most teams currently do no research at all. They make the decision based on instinct, the opinion of the most senior person in the room, or the results of a study that was conducted eighteen months ago and has been treated as current ever since. Synthetic research does not need to be perfect to be enormously better than nothing. And for the eighty per cent of questions where it is appropriate, it is not merely adequate -- it is faster, cheaper, and often more honest than the traditional alternative.
When to Use Real Research Instead
The guidance here should be direct. Use traditional research methods -- qualitative interviews, quantitative surveys with real respondents, observational studies, ethnographic research, sensory panels, clinical trials -- in the following circumstances.
When the product is genuinely novel and requires the respondent to encounter something they have never conceptualised. When the research question involves physical sensory experience. When the target population is too narrow or too specific for reliable synthetic representation. When the emotional context of the research is extreme and the ethical responsibility to respondents demands trained human facilitation. When regulatory or compliance frameworks require human-subject data. When the decision is high-stakes and the margin between stated preference and revealed behaviour could change the outcome. When you are testing usability and need to observe real interaction rather than hypothetical response.
The common thread across these circumstances is that they involve either the irreducible specificity of real human experience or the institutional requirements of external stakeholders. Synthetic research is powerful within its operating parameters. Outside those parameters, it produces plausible-sounding data that should not be trusted, and the appearance of plausibility makes it more dangerous than having no data at all, because bad data that looks good will be acted upon with confidence.
The Hybrid Model: Synthetic and Traditional Together
The most sophisticated research programmes do not choose between synthetic and traditional methods. They use both, deliberately, in a sequence designed to maximise the strengths of each.
The pattern that works best in practice is: synthetic first, traditional second.
Start with a synthetic study to establish the landscape. Identify the key themes, the major preference patterns, the competitive dynamics, and the areas of uncertainty. This takes hours, not weeks, and it costs a fraction of what the traditional phase will require. Critically, it also produces a far better brief for the traditional research that follows.
One of the persistent problems with traditional qualitative research is poorly designed discussion guides. The moderator asks questions that are too broad, too leading, or too disconnected from the actual decision dynamics of the market. This happens because the people designing the discussion guide do not have enough prior knowledge to ask the right questions. Synthetic research eliminates this problem. You walk into the qualitative phase already knowing the territory, already knowing which themes emerged, already knowing where the interesting tensions lie. The discussion guide writes itself, because you are no longer exploring in the dark.
The traditional phase then does what it does best: it adds depth, nuance, and the human specificity that synthetic methods cannot provide. The depth interview with the real buyer who explains not just what they prefer but why, with digressions, contradictions, and the kind of messy human detail that synthetic data smooths away. The usability session where you watch a real person interact with your product and see the moment of confusion that no survey would have captured. The sensory panel where trained evaluators provide the kind of granular product feedback that can only come from direct experience.
The hybrid model is more expensive than synthetic alone. It is substantially cheaper and faster than traditional alone, because the synthetic phase reduces the scope and improves the focus of the traditional phase. And it produces better outcomes than either method in isolation, because each method compensates for the other's weaknesses.
If you have the budget for only one approach, use synthetic for the eighty per cent of questions it handles well, and accept the limitations for the rest. If you have the budget for both, use synthetic to map the landscape and traditional to explore the terrain that matters most. If you have the budget for only traditional research, you probably have the budget for synthetic research too -- you are simply allocating it differently, and you should reconsider that allocation.
A Note on Intellectual Honesty
We have now written twenty articles in this series. Nineteen of them demonstrate what synthetic research can do. This one documents what it cannot. We could have folded these caveats into a paragraph at the bottom of each previous article, the way pharmaceutical advertisements list side effects in small print while the voiceover describes a life transformed. We chose to dedicate an entire article to limitations instead, because we believe the credibility of the entire series depends on the reader trusting that we are telling the truth -- all of it, including the parts that are commercially inconvenient.
The validation data supports a 92% directional accuracy rate for synthetic research conducted through FishDog. That number is real, it is independently audited, and we stand behind it. But 92% is not 100%, and the 8% where synthetic research diverges from real-world outcomes is not randomly distributed. It clusters around the limitations described in this article: novel categories, sensory evaluation, extreme emotional contexts, narrow populations, and the gap between stated preference and revealed behaviour.
Knowing where the 8% lives is as valuable as knowing what the 92% covers. Perhaps more so, because it tells you where to invest your traditional research budget for maximum impact rather than spreading it evenly across questions that synthetic methods could have answered for a fraction of the cost and time.
The definitive guide to FishDog is available for teams that want to explore synthetic research within its proper boundaries. The boundaries are real. So are the capabilities. Both deserve your honest assessment.
Phillip Gales is co-founder at [FishDog](https://fish.dog). He has financial interests that the reader should weigh accordingly.
The Claude Code and FishDog for Product Marketing Series
This is part of a series exploring how AI agents handle the core disciplines of product marketing. Each article covers one function of the PMM stack, explains the methodology, and links to a companion Claude Code guide you can run yourself.
Part 1: How to Develop Product Positioning (guide)
Part 2: How to Build Competitive Battlecards (guide)
Part 3: How to Research Pricing (guide)
Part 4: How to Test Product Messaging (guide)
Part 5: How to Run Voice of Customer Research (guide)
Part 6: How to Segment Customers (guide)
Part 7: How to Validate GTM Strategy (guide)
Part 8: How to Build a Content Marketing Engine (guide)
Part 9: How to Build Sales Enablement Materials (guide)
Part 10: How to Research a Product Launch (guide)
Part 20: Limitations of Synthetic Research with Claude Code and FishDog -- this article