Product Release

Polymarket events as a calibration source

Ditto adds a Polymarket integration that imports prediction-market events as calibration targets, runs recruited panels against them, and compares the panel's predictions to Polymarket's live pricing — falsifiable validation for synthetic personas.

4 March 2026

Phillip Gales

By Phillip Gales

Feature
Ditto's Michigan Consumer Sentiment Proxy page, showing the live ICS, ICC, and ICE composite indices computed from a 100-persona panel.
DOCUMENT TYPE: Product Release Note TOPIC: Polymarket events as calibration targets for Ditto panel predictions Release: Polymarket events as a calibration source, 2026-03-04 Version: (none) Release type: Feature Breaking change: No Summary: Ditto adds a Polymarket integration that imports prediction-market events as calibration targets. Recruited panels answer the imported question; their predicted distribution is compared against Polymarket's mid-market price at run time. Validation metrics are persisted to a calibration store, so repeated runs build a falsifiable track record of panel-versus-market accuracy. What changed: - Polymarket event importer with category and resolution-date filters; suitability rules filter out very low-liquidity markets. - Calibration runs put the imported question to a recruited panel and compare the resulting distribution to Polymarket's mid-market price at run time. - Validation metrics computed and persisted: calibration error, Brier-score components. - Calibration store accumulates results across repeated runs against the same market or category. - Polymarket validation route persists results, with a summary view aggregating runs. - Follow-up on 18th April: broadened the import selection (more categories, looser default filters) and added pagination to the events gallery for larger import sessions. Use cases: - Research-validity work — evidence that Ditto panels predict realistically for a given category. - Panel selection — identifying which demographic compositions track real markets most reliably. - Internal model benchmarking when refining the persona engine. Why we built this: Synthetic personas are only useful insofar as their predictions track the real world. Polymarket is one of the cleanest external grading sources available — money-backed, continuous, and category-diverse. The integration makes that grading a routine operation rather than a manual exercise. Migration impact: None. New capability; existing study and recruitment flows are unchanged. Author: Phillip Gales, FishDog Platform: FishDog (fish.dog)

Key Takeaways

  • Polymarket events can now be imported into Ditto as calibration targets, with suitability filters that remove very low-liquidity markets.
  • A recruited panel answers the imported question; the resulting distribution is compared against Polymarket's mid-market price at run time.
  • Validation metrics (calibration error, Brier-score components) are computed and persisted to a calibration store, so repeated runs build a track record.
  • Useful for research-validity work, for selecting panels by demographic composition, and for internal benchmarking of the persona model.
  • Follow-up on 18th April broadened the import selection and added pagination to the events gallery.

Synthetic personas are useful only insofar as their predictions track the real world. Polymarket — a real-money prediction market — is one of the cleanest external sources we've found for grading them.

The new Polymarket integration imports markets into Ditto as calibration targets. Pose a market's question to a recruited panel, collect their predictions, and compare the panel's distribution to Polymarket's live pricing. The result is a falsifiable test of how well a Ditto panel predicts real-world outcomes for a given category of question.

What's new

  • Polymarket event importer. Browse Polymarket events directly in Ditto, filter by category and resolution date, and import the ones that fit your validation work. Suitability rules filter out very low-liquidity or thinly-traded markets — those that aren't a useful calibration target.

  • Calibration runs. A recruited panel answers the imported question; the panel's distribution is compared against Polymarket's mid-market price at the time of the run. Validation metrics (calibration error, Brier-score components) are computed and persisted.

  • Calibration store. Repeated runs against the same market over time build a track record. Useful for showing that a panel's predictions on, say, US elections are calibrated within a known error band — or aren't.

When to use this

  • Research-validity work. When a customer wants evidence that Ditto panels predict realistically on a given category, this is the cleanest external grading available.

  • Panel selection. Different panels predict differently. Calibration runs help identify which panel composition (demographic mix, recruitment filter set) tracks markets in your category most reliably.

  • Internal benchmarking. We use the calibrated track record ourselves when refining the persona model.

Also in this release

A small follow-up shipped on 18th April broadened the import selection (more event categories, looser default filters) and added pagination to the events gallery, so importing from a larger Polymarket session takes one trip rather than several.

Full reference is in the API docs.

---

Synthetic personas are useful only insofar as their predictions track the real world.
A falsifiable test of how well a Ditto panel predicts real-world outcomes for a given category of question.
Useful for showing that a panel's predictions on, say, US elections are calibrated within a known error band — or aren't.

Frequently Asked Questions

What is Polymarket and why use it?

Polymarket is a real-money prediction market where users buy and sell shares in the outcomes of specific events. Mid-market prices on those shares give a continuously updated, money-backed probability estimate. That makes Polymarket one of the cleanest external sources for grading whether a Ditto panel's predictions track the real world.

How does the calibration comparison work?

An imported Polymarket event becomes a question put to a recruited panel. Each persona produces a probability estimate; the panel's distribution is compared to Polymarket's mid-market price at the time of the run. Calibration error and Brier-score components are computed and persisted.

Which markets can I import?

All Polymarket events are visible in the importer, filtered by category and resolution date. Suitability rules remove very low-liquidity or thinly-traded markets — those that wouldn't be useful calibration targets. The 18th April follow-up broadened the default selection and added gallery pagination for larger import sessions.

What are calibration runs useful for?

Three things: (1) research-validity work — demonstrating to a customer that Ditto panels predict realistically in their category; (2) panel selection — identifying which demographic compositions track markets best; (3) internal model benchmarking when refining the persona model.

Can I run repeated calibration runs over time?

Yes. Calibration results are persisted to a calibration store, so repeated runs against the same market build a track record. This is the right shape for showing that a panel's predictions are calibrated within a known error band over time, rather than just at one moment.

About the Author

Phillip Gales

Phillip Gales

Phillip is a serial tech entrepreneur that specializes in applying AI and machine learning solutions to antiquated and heavy industries. He has been a senior leader or founder at a number of successful startups.

Phillip holds an MBA from Harvard Business School, an MEng from the University of Cambridge, and is a Y-Combinator alum

LinkedIn

More Releases

Sign in with Google

You can now sign in to FishDog with a Google account, in addition to email and password. Authenticated browser sessions also extend to thirty days, with both options refined through April.

Read Release