Using synthetic research to test product concepts with real-world personas before committing engineering resources to something nobody wants
The Feature That Nearly Killed the Company
In 2013, a well-funded enterprise software company spent fourteen months building a collaborative document editing feature. Their product team had done the research. Users complained about version control issues. Multiple people editing the same document created conflicts. The team designed an elegant real-time collaboration engine, hired three additional engineers, and delivered a technically impressive solution that synchronized edits across users with sub-second latency.
Nobody used it.
Not because the technology failed. It worked beautifully. The problem was that users had been describing a version control problem, not a collaboration problem. They did not want to edit documents simultaneously. They wanted to know which version was current and who had changed what. The team had built a sophisticated answer to a question nobody was asking.
Fourteen months. Three additional hires. An entire quarter's roadmap displaced. All because the concept was never tested against the people who would actually use it.
This story is not unusual. It is, in fact, the norm. According to research by the Standish Group, 64% of features in typical software products are rarely or never used. Not because they are poorly built, but because they solve the wrong problem, address the wrong audience, or exist because someone in a meeting thought they sounded like a good idea.
The gap between "this seems like a good idea" and "this is something people will actually adopt" is where concept testing lives. It is Stage 6 of the product research lifecycle, the bridge between generating ideas and committing resources. Everything before this stage has been about understanding the landscape: framing the problem (Stage 1), exploring it in depth (Stage 2), segmenting the audience (Stage 3), synthesising and prioritising findings (Stage 4), and generating candidate solutions (Stage 5). Now, armed with a shortlist of promising concepts, you face the question that determines whether the next six months of development will produce value or waste: is this the right thing to build?
That is a different question from the one that comes later. Usability testing, beta programmes, and post-launch measurement all ask "did we build it right?" Concept testing asks the prior question, the one that matters more, and the one that teams routinely skip.
What Concept Testing Actually Is
Concept testing is the process of presenting a product idea to target users in sufficient detail for them to form a meaningful reaction, but not in so much detail that you have already built it.
The balance is critical. Too vague and you are testing a vibes check. "Would you like a product that makes your life easier?" produces nothing actionable. Too detailed and you have crossed the line into prototyping, which is slower, more expensive, and commits you to implementation decisions that should still be open.
The sweet spot is what researchers call a minimum viable concept: a description rich enough to provoke genuine reactions but lean enough to modify based on what you learn.
A minimum viable concept typically includes:
What the product does (the core value proposition, in plain language)
Who it is for (so the respondent can assess personal relevance)
How you would use it (enough interaction detail to be concrete)
What outcome it delivers (the benefit, not the feature)
How it differs from alternatives (the reason to choose this over the status quo)
Notice what is absent: technology stack, implementation details, pricing tiers, integration architecture, roadmap timelines. These matter enormously to the team building the product. They matter almost nothing to the person deciding whether to use it.
The Taxonomy of Testable Concepts
Concept testing is not limited to new products. It applies to any significant investment where the question "will people want this?" precedes the question "can we build this?"
Feature concepts. You are adding a major capability to an existing product. Before your engineers spend a quarter building it, does the target audience understand it, want it, and believe it would improve their experience? The collaborative editing feature from the opening of this article would have benefited enormously from thirty minutes of concept testing.
Product redesigns. You are fundamentally rethinking the user experience. A redesign is expensive and disruptive. It affects every existing user. Concept testing reveals whether the new direction resonates or whether you are about to alienate your base in pursuit of an audience that does not exist.
New product lines. You are extending into an adjacent market. The assumptions that hold for your current product may not hold for a new category. Concept testing with the new target audience surfaces misalignments before you discover them in market.
Service extensions. You are adding consulting, support, managed services, or other non-software offerings. These involve hiring, training, and operational complexity. Testing the concept before scaling the team is prudent in the same way that measuring before cutting is prudent.
Pricing models. You are considering a shift from subscription to usage-based, from freemium to paid-only, from per-seat to per-team. Pricing model changes affect every customer relationship. Concept testing reveals which models users understand, which they trust, and which provoke immediate resistance.
Business model pivots. You are rethinking who pays, how value is delivered, or what the core product actually is. This is the highest-stakes version of concept testing, and consequently the context where skipping it is most catastrophic.
What Concept Testing Is Not
Concept testing is not a focus group. Focus groups produce social dynamics that distort individual reactions. Concept testing seeks individual, uncontaminated responses.
Concept testing is not a survey. Surveys measure what people say at scale. Concept testing explores why people react the way they do. It is qualitative, not quantitative. The goal is depth of understanding, not statistical significance.
Concept testing is not usability testing. Usability testing requires a working prototype and asks whether people can use it effectively. Concept testing requires only a description and asks whether people want what is being described.
And concept testing is not product messaging validation. This distinction deserves particular emphasis, because the two are frequently confused. Product messaging testing (covered in the How To: Product Messaging article) evaluates whether your communication about a product resonates. It assumes the product concept is sound and asks whether you are describing it effectively. Concept testing makes no such assumption. It evaluates the product idea itself, not the words you use to sell it. You can have a brilliant concept with terrible messaging, or terrible concept with brilliant messaging. These are different problems requiring different research.
The Seven Questions That Reveal Whether a Concept Will Work
The purpose of concept testing is not to ask people whether they "like" your idea. Liking is cheap. People like all sorts of things they would never pay for, use regularly, or recommend to others. The purpose is to understand whether the concept is comprehensible, relevant, differentiated, believable, and actionable.
Each of these dimensions maps to a specific failure mode, and each can be probed with a well-crafted question.
1. Comprehension: "Do They Understand It?"
Present the concept description and ask: "Based on what I've described, what do you understand this product to do? Explain it back in your own words."
This is the most important question in the study, and it should always come first. If people cannot articulate what your product does after reading a clear description, you have a comprehension problem. And a comprehension problem at the concept stage becomes a marketing problem, an onboarding problem, and an adoption problem downstream.
The tell is not whether they get it exactly right. It is whether they get the core value proposition right. If your concept is a sleep-tracking app and personas describe it as "a fitness tracker," they have understood the mechanism but missed the point. That gap tells you something important about how the concept will land in market.
2. Relevance: "Does It Matter to Them?"
"How relevant is this to your life right now? Is this solving a problem you actually have, or is it solving a problem that exists in theory but does not bother you in practice?"
A concept can be perfectly understood and entirely irrelevant. "I get what it does. I just don't need it." This is a segmentation signal. If some personas find it highly relevant and others shrug, you have learned something about who your real audience is, which may differ from who you assumed it was.
Relevance also has a temporal dimension. "I would have needed this two years ago" or "I might need this someday" are not the same as "I need this now." Concepts that solve future or past problems face adoption barriers that concepts solving present problems do not.
3. Differentiation: "Is It Different From What They Already Have?"
"How does this compare to what you currently use to handle this problem? Is it meaningfully better, or roughly the same as what's already available?"
This is where concepts go to die. The team has spent weeks crafting something they believe is unique, only to discover that users perceive it as roughly equivalent to a tool they already have, or worse, as a slightly different version of something they tried and abandoned.
Differentiation is not about feature count. It is about perceived distance from the status quo. A concept that does one thing dramatically better than existing options is more differentiated than a concept that does twenty things slightly better. Users evaluate concepts relative to their current solution, not relative to your competitor matrix.
4. Believability: "Do They Trust the Claim?"
"Do you believe this product could actually deliver what I've described? What makes you sceptical, if anything?"
Some concepts trigger immediate scepticism. "An app that predicts what your customers want before they know themselves" sounds impressive in a pitch deck and implausible to anyone who has used a prediction engine. Believability problems at the concept stage become trust problems at the sales stage.
The most dangerous believability gap is the one you do not know about. Your team has spent months immersed in the technology and genuinely believes the claim is modest. The target user, encountering the claim cold, thinks it sounds like marketing fiction. Concept testing surfaces this gap before you build an entire go-to-market strategy on a claim your audience does not believe.
5. Intent: "Would They Actually Try It?"
"If this product existed today, what would you do? Would you try it, investigate further, or ignore it entirely? What would need to be true for you to give it a go?"
Intent is not prediction. People are notoriously poor at predicting their own future behaviour. But the conditions they attach to their intent are highly informative. "I would try it if it integrated with Slack" tells you something about adoption requirements. "I would try it if it were free" tells you something about perceived value. "I would try it if someone I trusted recommended it" tells you something about the sales motion.
The conditional is more valuable than the answer. "Would you use this?" yields a polite "probably." "What would need to be true for you to use this?" yields actionable design requirements.
6. Concerns: "What Worries Them?"
"What concerns, hesitations, or objections come to mind? What might stop you from using this even if you thought it was a good idea?"
Every concept has friction points. Some are fundamental and some are solvable, but you cannot address what you have not surfaced. Concerns fall into predictable categories: privacy and data security, learning curve and time investment, cost and value, reliability and trust, switching costs from current solution, and social proof or lack thereof.
The pattern across responses matters more than any individual concern. If seven of ten personas mention data privacy, you have a systemic trust issue that must be addressed in the concept itself, not papered over with a reassuring sentence in the FAQ.
7. Improvement Suggestions: "What Would Make It Better?"
"If you could change anything about this concept to make it more useful to you, what would you change? What is missing?"
This is the generative question. It turns respondents into co-designers. Their suggestions will not all be feasible or coherent, but the patterns reveal what matters. If multiple personas independently suggest the same missing element, that element probably belongs in the concept.
The gap between what you described and what they wish you had described is your design brief for the next iteration. It is also, frequently, the insight that separates a concept that tests adequately from one that tests exceptionally.
The Minimum Viable Concept: Getting the Stimulus Right
The quality of your concept test depends entirely on the quality of your stimulus. A poorly written concept description produces noise. A well-written one produces signal. The difference is not literary flair. It is clarity, specificity, and restraint.
The Anatomy of a Good Concept Description
A concept description for testing purposes should be 100 to 200 words. Shorter than that and respondents lack enough information to form a genuine reaction. Longer than that and you are testing their reading comprehension rather than their response to the concept.
Structure it as follows:
Opening sentence: State the problem in the user's language. Not your language, not your investor's language, the user's language. "You know how it takes forever to figure out which version of a document is current?" beats "Enterprise document management faces persistent version control challenges."
Core proposition (2-3 sentences): What the product does and how. Be specific enough to be concrete but not so specific that you are describing a wireframe. "This app connects to your existing tools and automatically tracks every change, who made it, and when" is the right level of detail.
Benefit statement (1-2 sentences): What the user gets. Frame it as an outcome, not a feature. "You always know which version is current, and you can see exactly what changed and why" is an outcome. "Real-time version tracking with audit trails" is a feature list.
Differentiator (1 sentence): What makes this different from what they are already doing. "Unlike tracking changes manually or asking colleagues which file is newest, this works automatically across all your tools."
What to Leave Out
Do not include pricing. Pricing shapes reactions so powerfully that it overwhelms concept evaluation. You are not testing price sensitivity at this stage. You are testing concept appeal. If a concept does not resonate at any price, knowing the optimal price point is irrelevant.
Do not include brand names. If you are testing a concept for an established brand, the brand's existing reputation will colour every response. Test the concept on its own merits first. Brand effects can be layered in later.
Do not include implementation timelines. "Coming in Q3" creates urgency that distorts intent measurement. "Available now" creates expectation you cannot meet. Present the concept as hypothetical.
Do not include competitor comparisons. "Unlike Product X, our solution..." anchors the respondent to the competitor and transforms your concept test into a competitive evaluation. These are different research activities.
Common Mistakes in Concept Descriptions
The feature list disguised as a concept. "Our product has real-time sync, version history, branching, merging, commenting, permissions management, and SSO." This is a specification, not a concept. Users respond to stories and outcomes, not feature catalogues.
The jargon trap. "Leveraging proprietary NLP models, our platform provides actionable intelligence through semantic analysis of unstructured data." Approximately zero target users would describe their problem using these words. Write in the language your users actually speak.
The solution without a problem. "A beautiful iOS app that..." Describe the problem before describing the solution. Without the problem, the solution has no context and the respondent has no basis for evaluating relevance.
The overclaimed benefit. "Saves you hours every day." Maybe it does. But if the claim exceeds what the respondent considers plausible, you have undermined believability and polluted your intent data. Understate rather than overstate. A concept that exceeds modest expectations outperforms one that fails to meet extravagant ones.
How FishDog Transforms Concept Testing
Traditional concept testing is slow, expensive, and fragile. Recruiting eight to twelve participants who match your target demographic takes one to three weeks. Scheduling and conducting interviews takes another week. Synthesis takes a third. By the time you have results, the team has either waited impatiently or, more commonly, started building anyway. The research arrives as a retrospective verdict on a decision already made.
FishDog inverts this dynamic. By testing concepts with synthetic personas that match your target user's demographics, behaviours, and context, you get qualitative reactions in minutes rather than weeks. This changes not just the speed of concept testing but its fundamental role in the product process.
When testing is fast, you test more concepts. When you test more concepts, you make better decisions. When you make better decisions, you build fewer features that nobody uses.
Creating a Research Group for Concept Testing
The research group should match the target segment you identified in Stage 3. Concept testing is not the moment to broaden your audience. It is the moment to be precise about who this concept is for.
```json { "name": "Mid-Career PMs - Concept Test", "group_size": 10, "filters": { "country": "USA", "age_min": 28, "age_max": 45 } } ```
Ten personas is the standard. It provides enough diversity to surface patterns without producing so much data that synthesis becomes unwieldy. If your concept targets multiple distinct segments, run separate studies for each. Mixing segments in a single study obscures which reactions come from which audience.
Structuring the Study
A concept testing study in FishDog follows the seven-question framework described above. The concept description itself is embedded in the study objective, so every persona encounters it before answering questions.
Study objective example:
We are testing a product concept for mid-career product managers. The concept is: [insert 100-200 word concept description]. We want to understand whether this concept is comprehensible, relevant, and appealing to people in this role.
The seven questions, adapted for a specific example:
"Based on the product I described, what do you understand it to do? Explain it in your own words."
"How relevant is this to the challenges you face in your current role? Is this solving something that genuinely frustrates you?"
"How is this different from the tools and processes you currently use for this task? Is the difference meaningful?"
"Do you believe the product could actually deliver what I described? What makes you sceptical?"
"If this existed right now, would you try it? What conditions would need to be met?"
"What concerns or hesitations come to mind? What might prevent you from using it?"
"What would you change or add to make this concept more useful to you?"
Running Comparative Tests
The real power of FishDog for concept testing emerges when you test multiple concepts against the same audience. In Stage 5, you generated a shortlist of two to three promising concepts. Now you test each one.
Create three studies using the same research group profile. Present a different concept in each. Ask the same seven questions. Compare the results.
The comparison reveals patterns that single-concept testing cannot:
Concept A scores high on comprehension and relevance but low on differentiation. Users understand it and want it, but they think they already have something similar.
Concept B scores high on differentiation and intent but low on believability. Users think it sounds different and exciting, but they doubt it can deliver.
Concept C scores consistently moderate across all dimensions. No red flags, but no enthusiasm either.
These are three different strategic situations requiring three different responses. Concept A needs sharper positioning. Concept B needs proof of concept or credibility signals. Concept C might need to be killed, because moderate enthusiasm at the concept stage typically translates to indifference at the adoption stage.
Running three concept tests with traditional methods would take six to nine weeks and cost thousands in recruitment and incentives. With FishDog, it takes an afternoon.
The Claude Code Workflow
Claude Code integrates directly with FishDog's API, which means concept testing can be executed from the command line as a natural part of the product development process. Here is the practical workflow.
Step 1: Prepare Your Concepts
Write each concept as a 100-200 word description following the structure described above. Store them in a document you can reference. This is the stimulus material, and it deserves careful crafting. Spend thirty minutes getting the language right. It will save you from testing a poorly described concept and concluding, incorrectly, that the concept itself is flawed.
Step 2: Create the Research Group
Using Claude Code, create a research group matching your target segment:
``` Create a FishDog research group called "Enterprise PMs - Concept Test" with 10 personas, USA, ages 28-45 ```
Claude Code calls the FishDog API to recruit synthetic personas matching these parameters. The group is available within minutes.
Step 3: Create and Run the Study
For each concept, create a study with the seven-question framework:
``` Create a FishDog study called "Concept Test: Automated Sprint Planning" with objective: "We are testing the following product concept with mid-career product managers: [concept description]. We want to understand comprehension, relevance, differentiation, believability, intent, concerns, and improvement ideas."
Ask these 7 questions:
What do you understand this product to do? Explain it in your own words.
How relevant is this to challenges you actually face in your current role?
How does this compare to what you currently use for this task?
Do you believe this product could deliver what I described? What makes you sceptical?
If this existed today, would you try it? What would need to be true?
What concerns or hesitations come to mind?
What would you change or add to make this more useful? ```
Claude Code creates the study, asks the questions, and polls for responses. When all personas have responded, it triggers study completion to generate AI-synthesised insights.
Step 4: Generate a Share Link
Once the study is complete, generate a shareable link so the entire team can review the raw responses and AI-generated insights:
``` Generate a share link for study "Concept Test: Automated Sprint Planning" ```
This produces a URL that anyone can access, no FishDog account required. Share it in your Slack channel, attach it to your product brief, paste it into the meeting agenda. The research is the artefact, not a summary someone wrote about the research.
Step 5: Synthesise Across Concepts
If you tested multiple concepts, compare the results. Look for:
Which concept had the highest comprehension (personas could explain it back accurately)?
Which concept had the strongest relevance signal?
Which concept generated the most detailed improvement suggestions (a proxy for engagement)?
Which concept raised the fewest fundamental concerns?
Where did responses converge across concepts (indicating segment-level preferences)?
The synthesis is the decision-making input. It does not make the decision for you. It arms you with evidence that makes the decision defensible.
Interpreting Results: What Good Looks Like (and What Should Worry You)
Concept testing produces qualitative data. There is no single metric that tells you "this concept passed." Instead, you are looking for patterns across responses that signal strength, weakness, or the need for iteration.
Strong Signals
High comprehension with low prompting. When seven or more personas accurately describe what the product does in their own words, without restating your description verbatim, the concept is clear. Paraphrasing indicates genuine understanding. Recitation indicates memorisation.
Unprompted enthusiasm. Responses that include phrases like "I wish this existed" or "I have been looking for something like this" are stronger signals than responses that say "yes, I would probably use it." Unprompted enthusiasm is rare in research. When it appears, pay attention.
Specific improvement suggestions. When personas suggest concrete additions or modifications, they have engaged deeply enough with the concept to think beyond it. Vague suggestions ("make it faster") indicate surface engagement. Specific suggestions ("I would want it to integrate with Jira and pull in sprint data automatically") indicate someone who is mentally designing alongside you.
Warning Signals
Comprehension failure. If four or more personas misunderstand the concept, the description needs rewriting and retesting. Do not proceed to development with a concept that people cannot explain. You will spend the next two years trying to market something that does not make sense to its audience.
Relevance mismatch. If personas say "I understand it, but I do not need it," you have either targeted the wrong segment or overestimated the problem's severity. Return to Stage 2 (Discovery Research) to verify that the problem is as acute as you believed.
The "nice to have" response. "Sure, that sounds fine" is the most dangerous response in concept testing. It is not rejection. It is not enthusiasm. It is polite indifference, and polite indifference does not drive adoption. Products that test as "nice to have" at the concept stage almost never achieve product-market fit. Users do not change their behaviour for something that sounds fine.
Differentiation failure. "That sounds like [existing product]" means your concept has a positioning problem, a genuine novelty problem, or both. If the concept genuinely differs from the competitor but personas do not perceive the difference, you have a communication issue. If the concept does not genuinely differ, you have a strategy issue. These require different responses.
Conditional intent with high barriers. "I would use it if it were free, required no setup, and integrated with every tool I already use" is not intent. It is a wish list. The conditions reveal what users consider table stakes, and if your concept cannot meet those conditions, intent is theoretical.
When to Kill a Concept
Not every concept deserves iteration. Some should be killed.
This is uncomfortable. The team has invested intellectual energy and emotional attachment. The concept has a name, a Slack channel, and possibly a logo someone made during a slow afternoon. Killing it feels like failure.
It is not failure. It is the entire point of concept testing. Killing a weak concept at Stage 6 costs you a few hours of research and a bruised ego. Killing it after twelve months of development costs you a year, a team's morale, and whatever else you could have built instead.
Kill a concept when:
Comprehension is persistently low across multiple description iterations. Some ideas are inherently difficult to explain. That is not a writing problem. It is a complexity problem that will plague every customer interaction.
Relevance is low across all segments. If the concept does not matter to anyone in your target audience, iterating the concept is less useful than revisiting the problem.
Differentiation is zero. If users genuinely cannot distinguish your concept from existing solutions, and the existing solutions are adequate, you are entering a market with no wedge.
Concerns are fundamental and unsolvable. "I would never give a third party access to my financial data" is not an objection you can overcome with better onboarding copy. It is a structural constraint on your addressable market.
Conversely, do not kill a concept based on a single negative response. Concept testing surfaces individual variation. One persona who dislikes the idea in a group of ten is noise. Five personas who share the same specific concern is signal.
From Concept Testing to What Comes Next
Concept testing is a gate, not a destination. Its output is a decision: proceed, iterate, or abandon. That decision feeds directly into the subsequent stages of the product lifecycle.
If the concept passes, you move to feature prioritisation (Stage 7) with confidence that the overall direction is sound. The concerns and suggestions from concept testing become design requirements. The comprehension test results inform your messaging. The relevance data refines your target segment.
If the concept needs iteration, you revise and retest. This is where FishDog's speed matters most. Traditional concept testing makes iteration impractical because each cycle takes weeks. With synthetic research, you can revise a concept description in the morning, retest it over lunch, and have results by the afternoon. Three iterations in a single day is entirely feasible.
If the concept fails, you return to Stage 5 (Solution Ideation) with the knowledge of why it failed. This is not wasted work. It is the most efficient possible way to learn that a particular direction is unproductive. The alternative, building it and discovering the same thing nine months later, is the expensive version of the same lesson.
At the end of Stage 6, you should have:
A validated concept (or a decision to abandon). Not a concept you hope will work. A concept you have tested and have evidence for.
A comprehension baseline. You know how users describe the product in their own words. This becomes the foundation of your messaging.
A concerns register. You know what worries people. These become design constraints and FAQ content.
An improvement roadmap. You know what users wish the concept included. These become candidate features for prioritisation.
Segment-level data. You know which audiences respond most strongly. This refines your go-to-market targeting.
This output transforms the next stage from guesswork into evidence-based planning. You are no longer deciding what to build based on internal opinion. You are deciding based on what your target users told you, in their own words, about whether the thing you are proposing matters to them.
That is what concept testing is for. Not to validate your excitement. To test it.
Quick Reference: Concept Testing Checklist
Before Testing:
[ ] Problem validated (Stage 1) and discovery complete (Stage 2)
[ ] User segments defined (Stage 3) and prioritised (Stage 4)
[ ] Solution concepts generated and shortlisted (Stage 5)
[ ] Concept descriptions written (100-200 words each, plain language)
[ ] Research group designed matching target segment
During Testing:
[ ] 7-question framework applied (comprehension, relevance, differentiation, believability, intent, concerns, improvements)
[ ] Multiple concepts tested comparatively where possible
[ ] Same research group profile used across concept variations
[ ] Share links generated for team review
After Testing:
[ ] Comprehension assessed (can personas explain it back?)
[ ] Relevance signals mapped to segments
[ ] Differentiation gaps identified
[ ] Concern patterns documented as design constraints
[ ] Go/iterate/kill decision made for each concept with supporting evidence
[ ] Results shared with engineering and design stakeholders
Further Reading
Testing Business Ideas by David Bland and Alexander Osterwalder
The Mom Test by Rob Fitzpatrick (particularly Chapter 3 on asking questions that do not lead)
Inspired: How to Create Tech Products Customers Love by Marty Cagan (Chapters 33-36 on concept validation)
Sprint: How to Solve Big Problems and Test New Ideas in Just Five Days by Jake Knapp (Day 5 methodology)
This is article 6 of 9 in the Product Stage series. Previous: [Stage 5: Solution Ideation](stage_05_solution_ideation.md). Next: Stage 7: Feature Prioritisation.
Want to test product concepts before committing engineering resources? FishDog lets you validate ideas with synthetic personas in minutes, not months. Stop building features nobody uses.