- relate to a candidate primary outcome or the comparison of interest, and;
- inform what is an important or realistic difference for that outcome, comparison, and population.
DELTA 2 recommended reporting items for the sample size calculation of a randomised controlled trial with a superiority question
- View popup
- View inline
Summary points
- Central to the design of a randomised controlled trial is an a priori sample size calculation, which ensures a high probability of the study achieving its prespecified main objective
- An incorrect sample size can result in a study that is unable to inform clinical practice (hence directly or indirectly harming patients), or could expose excess patients to the uncertainty inherent in a clinical trial
- The target difference between treatments that is considered realistic or important by one or more key stakeholder groups plays a critical part in the sample size calculation of a randomised controlled trial
- Guidance on how to choose the target difference and undertake a sample size calculation for funders and researchers is presented in this article
- 10 recommendations are made regarding choosing the target difference and undertaking a sample size calculation, along with recommended reporting items for trial proposal, protocols, and results papers
- This article on choosing the target difference for a randomised controlled trial and undertaking and reporting the sample size calculation has been dual published in The BMJ and BMC Trials journals
Development of the DELTA 2 guidance
The DELTA 2 guidance is the culmination of a five stage process to meet the stated project objectives (fig 1), which included two literature reviews of existing funder guidance and recent methodological literature, a Delphi process to engage with a wider group of stakeholders, a two day workshop, and finalisation of the core guidance.
DELTA 2 project components of work
- Download figure
- Open in new tab
- Download powerpoint
The literature review was conducted between April and December 2016 (searching up to April 2016). The Delphi study had two rounds: one held in 2016 before a two day workshop in Oxford (September 2016), and another between August and November 2017. The general structure of the guidance was devised at the workshop. It was substantially revised on the basis of feedback from stakeholders received through the Delphi study. In addition, stakeholder engagement events were held at various meetings throughout the development of the guidance: the Society for Clinical Trials meeting and Statisticians in the Pharmaceutical Industry conferences both held in May 2017, a Joint Statistical Meeting in August 2017, and a Royal Statistical Society Reading local group meeting in September 2017. These interactive sessions provided feedback on the scope (in 2016) and then draft guidance (in 2017). The core guidance was provisionally finalised in October 2017 and reviewed by the funders’ representatives for comment (Methodology Research Programme advisory group). The guidance was further revised and finalised in February 2018. The full guidance document incorporating case studies and relevant appendices is available here.5 Further details on the findings of the Delphi study and the wider engagement with stakeholders are reported elsewhere.6 The guidance and key messages are summarised in the remainder of this paper.
The target difference and sample size calculations in randomised controlled trials
The role of the sample size calculation is to determine how many patients are required for the planned analysis of the primary outcome to be informative. It is typically achieved by specifying a target difference for the key (primary) outcome that can be reliably detected and the required sample size calculated. In this summary paper, we restrict considerations to the most common trial design looking at a superiority question (one which assumes no difference between treatments and looks for a difference), although the full guidance considers equivalence and non-inferiority designs that invert the hypothesis and how the use of the target difference differs for such designs.5
The precise research question that the trial is primarily set up to answer will determine what needs to be estimated in the planned primary analysis, which is known formally as the “estimand.” A key part of characterising the research question is choosing the primary outcome, which needs careful consideration. The target difference should be a difference that is appropriate for that estimand.78910 Typically (for superiority trials), an intention to treat or treatment policy estimand—that is, according to the randomised groups irrespective of subsequent compliance with the treatment allocation—is used. Other analyses that deal with different estimands8911 of interest (eg, those based on the effect on receipt of treatment and the absence of non-compliance) could also inform the choice of sample size. Different stakeholders can have somewhat differing perspectives on the appropriate target difference.12 However, a key principle is that the target difference should be viewed as important by at least one (and preferably more) key stakeholder groups—that is, patients, health professionals, regulatory agencies, and healthcare funders. In practice, the target difference is not always formally considered and in many cases appears, at least from trial reports, to be determined on convenience, the research budget, or some other informal basis.13 The target difference can be expressed as an absolute difference (eg, mean difference or difference in proportions) or a relative difference (eg, hazard or risk ratio), and is also often referred to, rather imprecisely, as the trial “effect size.”
Statistical calculation of the sample size is far from an exact science.14 Firstly, investigators typically make assumptions that are a simplification of the anticipated analysis. For example, the impact of adjusting for baseline factors is difficult to quantify upfront, and even though the analysis is intended to be an adjusted one (such as when randomisation has been stratified or minimised),15 the sample size calculation is often conducted on the basis of an unadjusted analysis. Secondly, the calculated sample size can be sensitive to the assumptions made in the calculations such that a small change in one of the assumptions can lead to substantial change in the calculated sample size. Often a simple formula can be used to calculate the required sample size. The formula varies according to the type of outcome, how the target difference is expressed (eg, a risk ratio versus a difference in proportions), and somewhat implicitly, the design of the trial and the planned analysis. Typically, a sample size formula can be used to calculate the required number of observations in the analysis set, which varies depending on the outcome and the intended analysis. In some situations, ensuring the sample size is sufficient for more than one planned analysis may be appropriate.
When deciding on the sample size for a randomised controlled trial, it is necessary for researchers to balance the risk of incorrectly concluding that there is a difference when no actual difference between the treatments exists, with the risk of failing to identify a meaningful treatment difference when the treatments do differ. Under the conventional approach, referred to as the statistical hypothesis testing framework,16 the probabilities of these two errors are controlled by setting the significance level (type I error) and statistical power (1 minus type II error) at appropriate levels (typical values are two sided 5% significance and 80% or 90% power, respectively). Once these two inputs have been set, the sample size can be determined given the magnitude of the between group difference in the outcome it is desired to detect (the target difference). The calculation (reflecting the intended analysis) is conventionally done on the basis of testing for a difference of any magnitude. As a consequence, it is essential when interpreting the analysis of a trial to consider the uncertainty in the estimate, which is reflected in the confidence interval. A key question of interest is what magnitude of difference can be ruled out. The expected (predicted) width of the confidence interval can be determined for a given target difference and sample size calculation, which is another helpful aid in making an informed choice about this part of a trial’s design.17 Other statistical and economic approaches to calculating the sample size have been proposed, such as precision and bayesian based approaches16181920 and the value of information analysis,21 although they are not at present commonly applied.22
The required sample size is very sensitive to the target difference. Under the conventional approach, halving the target difference quadruples the sample size for a two arm, 1:1, parallel group superiority trial with a continuous outcome.23 Appropriate sample size formulas vary depending on the proposed trial design and statistical analysis, although the overall approach is consistent. In more complex scenarios, simulations can be used but the same general principles hold. It is prudent to undertake sensitivity calculations to assess the potential effect of misspecification of key assumptions (such as the control response rate for a binary outcome or the anticipated variance of a continuous outcome).
The sample size calculation and the target difference, if well specified, help provide reassurance that the trial is likely to detect a difference at least as large as the target difference in terms of comparing the primary outcome between treatments. Failure to clarify sufficiently what is important and realistic at the design stage can lead to subsequent sample size revisions, or an unnecessarily inconclusive trial due to lack of statistical precision or ambiguous interpretation of the findings.2425 When specifying the target difference with a definitive trial in mind, the following guidance should be considered.
Specifying the target difference for a randomised controlled trial
Different statistical approaches can be taken to specify the target difference and calculate the sample size but the general principles are the same. To aid those researchers new to the topic and to encourage better practice and reporting regarding the specification of the target difference for a randomised controlled trial, a series of recommendations is provided in box 1 and table 1. Seven broad types of methods can be used to justify the choice of a particular value as the target difference, which are summarised in box 2.
Methods that can help inform the choice of the target difference
Methods that inform what is an important difference
- Anchor: The outcome of interest can be anchored by using either a patient’s or health professional’s judgment to define what an important difference is. This approach can be achieved by comparing a patient’s health before and after treatment and then linking this change to participants who showed improvement or deterioration using a more familiar outcome (for which either patients or health professionals more readily agree on what amount of change constitutes an important difference). Contrasts between patients (eg, individuals with varying severity of a disease) can also be used to determine a meaningful difference.
- Distribution: Approaches that determine a value based on distributional variation. A common approach is to use a value that is larger than the inherent imprecision in the measurement and therefore likely to represent a minimal level needed for a noticeable difference.
- Health economic: Approaches that use the principles of economic evaluation. These approaches compare cost with health outcomes, and define a threshold value for the cost of a unit of health effect that a decision maker is willing to pay, to estimate the overall incremental net benefit of one treatment versus the comparator. A study can be powered to exclude a zero incremental net benefit at a desired statistical significance and power. A radically different approach is a (bayesian) decision-theoretic value of information analysis that compares the added value with the added cost of the marginal observation, thus avoiding the need to specify a target difference.
- Standardised effect size: The magnitude of the effect on a standardised scale defines the value of the difference. For a continuous outcome, the standardised difference can be used (most commonly expressed as Cohen’s d effect size, the mean difference divided by the standard deviation). Cohen’s cutoff sizes of 0.2, 0.5, and 0.8 are often used for small, medium, and large effects, respectively. Thus, a medium effect corresponds simply to a difference in the outcome of 0.5 standard deviations. When measuring a binary or survival (time-to-event) outcome, alternative metrics (eg, an odds, risk, or hazard ratio) can be used in a similar manner, although no widely recognised cutoff points exist. Cohen’s cutoff points approximate odds ratios of 1.44, 2.48, and 4.27, respectively.26 Corresponding risk ratio values vary according to the control group event proportion.
Methods that inform what is a realistic difference
- Pilot study: A pilot (or preliminary) study may be carried out if there is little evidence, or even experience, to guide expectations and determine an appropriate target difference for the trial. Similarly, a phase 2 study could be used to inform a phase 3 study, although this approach would need to take account of methodological differences (eg, inclusion criteria and outcomes) that should be reflected in specification of the target difference.
Methods that inform what is an important or a realistic difference
- Opinion seeking: The target difference can be based on opinions elicited from health professionals, patients, or others. Possible approaches include forming a panel of experts, surveying the membership of a professional or patient body, or interviewing individuals. This elicitation process can be explicitly framed within a trial context.
- Review of evidence base: The target difference can be derived from current evidence on the research question. Ideally, this evidence would be from a systematic review or meta-analysis of randomised controlled trials. In the absence of randomised evidence, evidence from observational studies could be used in a similar manner.
Broadly speaking, two different approaches can be taken to specify the target difference for a randomised controlled trial. A difference that is considered to be:
- Important to one or more stakeholder groups
- Realistic (plausible), based on either existing evidence, or expert opinion.
A large literature exists on defining and justifying a (clinically) important difference, particularly for quality of life outcomes.272829 In a similar manner, discussions of the relevance of estimates from existing studies are also common; there are several potential pitfalls to their use, which needs careful consideration of how they should inform the choice of the target difference.2 It has been argued that a target difference should always be both important and realistic,30 which would seem particularly apt when designing a definitive (phase 3) superiority randomised controlled trial. In a sample size calculation for a randomised controlled trial, the target difference between the treatment groups strictly relates to a group level difference for the anticipated study population. However, the difference in an outcome that is important to an individual might differ from the corresponding value at the population level. More extensive consideration of the variations in approach is provided elsewhere.23
Reporting the sample size calculation
The approach taken to determine the sample size and the assumptions made should be clearly specified. This information should include all the inputs and formula or simulation results, so that it is clear what the sample size was based on. This information is critical for reporting transparency, allows the sample size calculation to be replicated, and clarifies the primary (statistical) aim of the study. Under the conventional approach with a standard trial design (1:1 allocation, two arm, parallel group, superiority design) and unadjusted statistical analysis, the core items that need to be stated are the primary outcome, the target difference appropriately specified according to the outcome type, the associated nuisance parameter (that is, a parameter that, together with the target difference, uniquely specifies the difference on the original outcome scale—eg, the event rate in the control group for a binary primary outcome), and the statistical significance and power. More complicated designs can have additional inputs that should be considered, such as the intracluster correlation for a cluster randomised design.
A set of core items should be reported in all key trial documents (grant applications, protocols, and main results papers) to ensure reproducibility and plausibility of the sample size calculation. The full list of recommended core items are given in table 1, which is an update of the previously proposed list.31 When the sample size calculation deviates from the conventional approach, whether by research question or statistical framework, the core reporting set can be modified to provide sufficient detail to ensure that the sample size calculation is reproducible and the rationale for choosing the target difference is transparent. However, the key principles remain the same. If the sample size is determined on the basis of a series of simulations, this method should be described in sufficient detail to provide an equivalent level of transparency and assessment. Additional items to give more explanation of the rationale should be provided if space allows (eg, in grant applications and trial protocols). Trial result publications can then reference these documents if sufficient space is not available to provide a full description.
Discussion
Researchers are faced with a number of difficult decisions when designing a randomised controlled trial, the most important of which are the choice of trial design, primary outcome, and sample size. The sample size is largely driven by the choice of the target difference, although other aspects of sample size determination also contribute.
The DELTA 2 guidance provides help on specifying a target difference and undertaking and reporting the sample size calculation for a randomised controlled trial. The guidance was developed in response to a growing recognition from funders, researchers, and other key stakeholders (such as patients and the respective clinical communities) of a real need for practical and accessible advice to inform a difficult decision. The new guidance document therefore aims to bridge the gap between the existing (limited) guidance and this growing need.
The key message for researchers is the need to be more explicit about the rationale and justification of the target difference when undertaking and reporting a sample size calculation. Increasing focus is being placed on the target difference in the clinical interpretation of the trial result, whether statistically significant or not. Therefore, the specification and reporting of the target difference, and other aspects of the sample size calculation, needs to be improved.
Acknowledgments
This paper summarised key findings of the new guidance produced by the DELTA 2 study, commissioned by the UK Medical Research Council (MRC)/National Institute for Health Research (NIHR) Methodology Research Programme, in response to an open commissioned call to organise a workshop on this topic in order to produce guidance. The authors are all researchers who have been involved in randomised trials of varying types, with most involved for more than 10 years; they have varying backgrounds and have worked in various clinical areas and on both academic and industry funded studies. The members of the original DELTA (Difference ELicitation in TriAls) 2 group were Jonathan Cook, Doug Altman, Jesse Berlin, Martin Bland, Richard Emsley, Dean Fergusson, Lisa Hampson, Catherine Hewitt, Craig Ramsay, Joanne Rothwell, Robert Smith, William Sones, Luke Vale, Stephen Walters, and Steven Julious. As part of the guidance development process, the two day workshop held in Oxford in September 2016 included the following participants: Doug Altman, David Armstrong, Deborah Ashby, Martin Bland, Andrew Cook, Jonathan Cook, David Crosby, Richard Emsley, Dean Fergusson, Andrew Grieve, Lisa Hampson, Catherine Hewitt, Steven Julious, Graeme MacLennan, Tim Maughan, Jon Nicholl, José Pinheiro, Craig Ramsay, Joanne Rothwell, William Sones, Nigel Stallard, Luke Vale, Stephen Walters, and Ed Wilson.
We thank the participants in the Delphi study and the one-off engagement sessions with various groups, including the attendees to the Society for Clinical Trials, Statisticians in the Pharmaceutical Industry, and Joint Statistical Meeting conference sessions, along with the other workshop participants who kindly provided helpful input and comments on the scope and content of this document; Robert Smith for his role as a member of the public who provided helpful public perspective and input in the workshop and also development and revision of the guidance document; Stefano Vezzoli for in-depth comments that helped to refine this document; and the MRC Methodology Research Programme advisory panel that commissioned the study and provided helpful feedback on a draft version.
Footnotes
- Contributors: JAC and SAJ conceived of the idea and drafted the initial version of the manuscript. WS, LVH, CH, JAB, DAs, RE, DAF, SJW, ECFW, GM, NS, JCR, MB, LB, CRR, AC, DAr, DAl, and LDV contributed to the development of the guidance and commented on the draft manuscript. All authors have read and approved the final version. The corresponding author attests that all listed authors meet authorship criteria and that no others meeting the criteria have been omitted.
- Funding: Funding for this work was received from the MRC-NIHR UK Methodology Research Programme in response to an open commissioned call for an effect size methodology state-of-the-art workshop. The Health Services Research Unit, Institute of Applied Health Sciences (University of Aberdeen), is core funded by the Chief Scientist Office of the Scottish Government Health and Social Care Directorates. The funders had no involvement in study design, collection, analysis, and interpretation of data; reporting; or the decision to publish.
- Competing interests: All authors have completed the ICMJE uniform disclosure form at www.icmje.org/coi_disclosure.pdf and declare: support from the MRC-NIHR UK Methodology Research Programme for the submitted work; JAB is employee of Johnson & Johnson and holds shares in this company; LVH is an employee of Novartis; all the other authors have been involved in design and conducting randomised trials through their roles; no other financial relationships with any organisations that might have an interest in the submitted work in the previous three years; and no other relationships or activities that could appear to have influenced the submitted work.
- Ethical approval: Ethics approval for the Delphi study, which is part of the DELTA 2 project, was sought and received from the University of Oxford’s medical sciences interdivisional research ethics committee (IDREC - R46815/RE001). Informed consent was obtained for all participants in the Delphi study.
This is an Open Access article distributed in accordance with the terms of the Creative Commons Attribution (CC BY 4.0) license, which permits others to distribute, remix, adapt and build upon this work, for commercial use, provided the original work is properly cited. See: http://creativecommons.org/licenses/by/4.0/.
References
- Altman DG ,
- Schulz KF ,
- Moher D ,
- et al. ,
- CONSORT GROUP (Consolidated Standards of Reporting Trials)
