Fit for Which Purpose? How to Judge Whether Synthetic Health Data Is Good Enough

Synthetic health data is not "good" or "bad". It is fit or unfit for a specific purpose, and the only honest evaluation states the purpose first. A synthetic cohort that reproduces population-level associations well enough for trial feasibility can be useless for detecting a rare adverse event. Evaluation therefore has three questions, in a fixed order: what decision will the data support; does the synthetic data preserve the statistics that decision depends on; and what does it leak about the real people behind it. Any evaluation that skips the first question cannot answer the other two.

Why "fit for purpose" is the only defensible standard

The health-informatics literature settled this question for real data a decade ago. Weiskopf and Weng's 2013 review of EHR data-quality assessment found that quality was being measured along five dimensions (completeness, correctness, concordance, plausibility, currency) with no agreed methods and, critically, no reference to what the data were for. Kahn and colleagues' 2016 harmonized framework made the point explicit: data quality for secondary use is assessed relative to a task, in categories of conformance, completeness and plausibility, each of which can be verified only against a stated expectation.

The cleanest recent demonstration of why this matters comes from emergency cardiology. Brandberg and colleagues asked, in a 2024 JAMIA study of 1,000 chest-pain patients at a Stockholm emergency department, whether the medical history in the electronic health record was fit for a very specific purpose: computing three established risk scores. It was not. The HEART score could be computed from the EHR for 31% of patients, EDACS for 7%, and T-MACS for 25%. The same patients' self-reported computerized histories supported the scores in 75%, 74% and 83% of cases. Same hospital, same patients, same week. The answer to "is the data good enough?" was no, no, no for one collection method and yes, yes, yes for the other, because the question was asked against a decision rather than in the abstract.

Synthetic data inherits this logic exactly. A synthetic dataset is a manufactured artifact with a stated relationship to a source; whether it is "good enough" is a property of the pair (dataset, purpose), never of the dataset alone. Every credible evaluation framework for synthetic data converges on the same structure, whether the Alan Turing Institute's 2022 report, Chen and colleagues' 2021 review in Nature Biomedical Engineering, or Giuffrè and Shung's 2023 review in npj Digital Medicine: utility is task-specific, privacy is attack-specific, and the two must be reported together.

What are the purposes synthetic health data is actually used for?

Purposes are not equal, and they should not be listed as if they were. Ordered from the most to the least tolerant of synthesis error:

The bottom three rows are where most disappointment with synthetic data originates, and they are also where the strongest privacy mechanisms bite hardest: differential privacy protects individuals precisely by limiting how much any one rare record can shape the output. That is a feature for the first four rows and a disqualification for the last three. An evaluation that does not begin by placing the intended use in this table has not begun.

Which utility measures match which purpose?

Once the purpose is fixed, the utility measures follow from it. Reporting all of them for every dataset is neither necessary nor honest; reporting the wrong one is a common way to make a dataset look better than it is.

The replication approach deserves emphasis because it is the one non-experts can interpret. El Emam and colleagues' 2024 study in Scientific Reports tested whether analyses published from real health data could be replicated on synthetic versions of the same data, and reported the agreement in effect size and direction. That is what "utility" should mean to a clinician or a data-access committee: not a divergence statistic, but "the finding you would have reached on the real data, you also reach on this one, and here is how often."

How is privacy evaluated, and why are similarity scores not enough?

Privacy evaluation has two questions, corresponding to two attacks. Membership inference: can an adversary tell whether a given individual was in the source data? Linkage: can a synthetic record be connected to a real person? Rocher, Hendrickx and de Montjoye showed in 2019 that 99.98% of Americans could be correctly re-identified in any dataset from 15 demographic attributes, a result that applies to any release that carries real individuals' attribute combinations, synthetic or not.

The industry's habitual answer is a similarity-based privacy metric: the distance from each synthetic record to its nearest real neighbor, compared to some baseline. Two results should retire that habit. Stadler, Oprisanu and Troncoso demonstrated at USENIX Security 2022 that synthetic data without formal guarantees provides no consistent protection against inference and can leak more about outliers than the original data would. Ganev's 2024 analysis went further, showing by construction that datasets can pass every common similarity-based test while containing exact copies of real records, and concluded that such metrics cannot establish the "identification is effectively precluded" standard that European regulators apply. The EDPB's draft anonymisation guidelines (02/2026, in consultation until 30 October 2026) point the same way: anonymity is assessed against reasonably likely means, not asserted from a score.

There is only one kind of privacy statement that can be made before an attack is run rather than after: a formal bound. Generation under differential privacy yields a stated (ε, δ), a ceiling on how much any single individual can shift the output. That ceiling is a property of the process and holds regardless of which attack is tried next. Empirical attack tests remain useful, but as confirmation of a bound, not as a substitute for one.

What should a fitness-for-purpose label contain?

If evaluation is relative to purpose, then the output of evaluation should be a label that travels with the dataset and states the purpose. Skyttberg and colleagues, studying data-quality governance across nine Swedish emergency departments, found that quality depends on a technical layer and a human layer, and that improvement fails when only one is addressed. A synthetic-data label has the same two layers. The technical layer (a privacy budget, divergence statistics, attack results) is what an auditor checks and almost nobody reads. The human layer, which states what this dataset is for and what it must not be used for, is what people act on.

A label that respects both layers carries five things:

1. Purpose grades: which rows of the table above the dataset was evaluated for, and passed.
2. Utility per grade: the measures appropriate to each passed grade, with the numbers.
3. A disclosed privacy budget: the (ε, δ) under which it was generated, and the accounting method, so that budgets compose across releases from the same source.
4. Provenance: a tamper-evident record of the source, the release version of every code system, and the generation run, so that the evaluation can be repeated years later.
5. Contraindications, printed rather than implied: the purposes for which the dataset was not evaluated or failed, starting with rare events and safety signals.

The fifth item is the one that separates a research artifact from a marketing one. A dataset that lists what it is not good for has been evaluated; a dataset that only lists what it is good for has been described.

Who should do the evaluation?

Not the generator's maker, or not only. The Brandberg study is credible because a hospital cardiology group and a university informatics group ran it against a pre-registered protocol; a vendor reporting its own utility scores is credible in proportion to how much it discloses about the cases where it failed. The practical standard for synthetic health data should be the same as for any measurement instrument: the producer reports its own numbers with full method and limits; an independent party (a university degree project, a data-access body's technical reviewer, a published replication) confirms or contradicts them on data the producer did not choose. Evaluation designed so that a negative result is publishable is the only evaluation worth citing.

Frequently asked questions