Treatment effects may remain the same even when trial participants differed from the target population

Abstract

Objectives: Randomized controlled trials (RCTs) have been criticized for lacking external validity. We assessed whether a trial in people with type I diabetes mellitus mirrored the wider population and applied sample-weighting methods to assess the impact of differences on our trial's findings.

Study design and setting: The Relative Effectiveness of Pumps over MDI and Structured Education trial was nested within a large UK cohort capturing demographic, clinical, and quality of life data for people with type I diabetes mellitus undergoing structured diabetes-specific education. We first assessed whether our RCT participants were comparable with this cohort using propensity score modeling. After this, we reweighted the trial population to better match the wider cohort and re-estimated the treatment effect.

Results: Trial participants differed from the cohort in regard to sex, weight, HbA1c, and also quality of life and satisfaction with current treatment. Nevertheless, the treatment effects derived from alternative model weightings were similar to that of the original RCT.

Conclusion: Our RCT participants differed in composition to the wider population, but the original findings were unaffected by sampling adjustments. We encourage investigators take steps to address criticisms of generalizability but doing so is problematic; external data, even if available, may contain limited information and analyses can be susceptible to model misspecification.

Trial registration: ClinicalTrials.gov NCT01616784.

Keywords: External validity; Generalizability; Randomized controlled trials; Sample reweighting.