Data quality in real-world studies is a more complex challenge than in interventional trials. In a trial, the protocol defines every data point, the case report form specifies every field, and monitoring verifies critical values systematically. In a real-world study, data often originates from multiple sources, with varying levels of structure, completeness, and consistency. The challenge is not to impose trial-level quality processes on data that cannot support them — it is to build a quality framework that is proportionate, systematic, and actually works.
Defining what quality means for your study
Quality is not a single property. A real-world dataset might be internally consistent but incomplete. It might be complete but contain systematic measurement error. It might have excellent completeness for primary outcomes and poor completeness for covariates.
The first step in a data quality framework is being specific: what does quality mean for this study, and what are the consequences of quality failures? For a primary endpoint that drives the primary analysis, a high standard of completeness and accuracy is essential. For secondary covariates where some missingness is expected and adjustment methods exist, a different standard may be appropriate.
Defining quality requirements at the study design stage — before data collection begins — allows data collection procedures to be designed to meet those requirements. Defining them after the data are collected means working with whatever was gathered, which is a substantially weaker position.
A 2021 literature review in the Journal of the American Medical Informatics Association, examining 120 published data quality frameworks across health, public health, and service settings, made the same point at scale: data quality has to be consistently defined in context, because the attributes that matter shift with what the data will be used for. Most of the frameworks it examined converged on a similar small set of dimensions, which is a useful checklist when scoping a study:
| Dimension | What it captures | Where it typically matters most |
|---|---|---|
| Completeness | How much of the expected data is actually present | Primary endpoints, key covariates |
| Accuracy | Whether recorded values reflect the true value | Lab measurements, clinical assessments |
| Consistency | Whether the same concept is recorded the same way across sources | Multi-site or multi-source studies |
| Timeliness | Whether data is captured close enough to the event it describes | Patient-reported outcomes, symptom diaries |
| Plausibility | Whether values fall within a credible range given other known information | Any field prone to entry error |
No study needs to score highly on every dimension for every variable. The point of naming them explicitly is to make a deliberate choice about which dimensions matter most for which variables, rather than defaulting to "as accurate as possible everywhere," which is neither achievable nor necessary.
Source diversity and harmonisation
Many real-world studies collect data from multiple sources: participant-reported outcomes, study assessments, laboratory measurements, and externally sourced data from registries or linked datasets. Each source has its own data structure, collection cadence, and quality characteristics.
Harmonising data from multiple sources — ensuring that variables collected differently can be combined into a coherent analysis dataset — requires explicit decisions about mapping and coding. These decisions should be made prospectively and documented. When harmonisation decisions are made retrospectively, they often reflect what made the analysis easier rather than what was scientifically correct, introducing analytical flexibility that undermines reproducibility.
Completeness monitoring during data collection
One of the most effective data quality interventions is the simplest: monitoring completeness in real time and acting on it. Real-world studies, particularly those with extended follow-up, generate completeness data continuously. Reviewing it systematically — who is providing complete data, where the gaps are, which instruments are performing poorly — allows problems to be corrected while the study is still running.
A participant who has stopped completing their dietary diary can be contacted before weeks of missing data accumulate. A data collection instrument generating systematic missing values can be investigated and corrected. A site not recording a particular element can be retrained.
None of this is possible if completeness is only reviewed at the end of the study. Building completeness monitoring into the study management plan, with defined review frequency and clear responsibilities, is a structural commitment — not an optional extra.
Query management: calibrated, not exhaustive
In interventional trials, data queries are generated systematically for defined out-of-range or missing values. In real-world studies, a similar approach is appropriate, but it needs to be carefully calibrated.
Generating queries for every imperfect data point in a large real-world dataset is not feasible — and many imperfections are expected and acceptable. The goal is to identify and resolve errors that would materially affect the analysis, not to produce a perfect dataset where none is possible.
Query management in real-world studies should focus on: primary endpoint data, critical covariates, values that are implausible given other available information, and systematic rather than random errors. Random incompleteness in low-priority fields is better handled through the missing data analysis than through laborious querying.
A 2024 implementation study in JMIR Medical Informatics applied this kind of framework to a real-world oncology dataset tracking time to treatment discontinuation — a variable that, like many real-world endpoints, is assembled from multiple underlying data points rather than recorded directly. The study's approach is a useful template regardless of therapeutic area: rather than querying every record, it defined a small set of automated plausibility checks against the specific variables the endpoint depended on, then reserved manual review for the subset of records those checks flagged. The general lesson transfers cleanly to nutritional and observational research — identify which handful of derived or composite variables the primary analysis actually depends on, and concentrate query effort there rather than spreading it evenly across the dataset.
Analysis and the pre-specified plan
The final quality gate is the analysis itself. Real-world datasets are rich with variables, and the temptation to explore associations that were not pre-specified is strong. Analytical flexibility that is not disclosed — running analyses multiple ways and reporting the most favourable result — is a form of data quality failure even when no individual data point is incorrect.
Pre-specifying the analysis plan, including primary outcomes, exposure definitions, covariate sets, and missing data handling, and registering it before the analysis begins, protects against this. It does not prevent exploratory analysis — it requires that exploratory analyses are clearly labelled as such.
For studies intended to generate evidence for regulatory or HTA purposes, the pre-specified analysis plan is not optional. Reviewers with experience of real-world evidence will ask to see it — and the absence of one will raise questions that are difficult to answer after the fact.