Resources for pilot and feasibility studies

Pilot and feasibility studies differ from studies to evaluate the effectiveness or efficacy of an intervention in that they are concerned with addressing areas of uncertainty and assessing feasibility. This different focus affects objectives and means that their design, analysis, and reporting are different and need special consideration.

For example, a pilot trial assessing feasibility of conducting a future definitive trial might collect data on items such as recruitment and retention to assess feasibility. Analysis is likely to be based on descriptive statistics since the study is not powered for formal hypothesis testing for effectiveness/efficacy, and reporting would need to include detail on items such as identification and consent of participants, criteria for deciding whether to progress to the definitive study, unintended consequences during the study, and implications for progression.

This page provides resources to help trialists and researchers to conduct a pilot or feasibility study.   

See Introduction to pilot and feasibility studies for an overview of pilot and feasibility studies.

See also our mini webinar series highlighting key points to consider when conducting pilot and feasibility studies prior to a definitive randomised controlled trial. 

Resources below are separated into sections on Design, Analysis and Reporting.

Design
Analysis
Reporting
Design: Introduction

As is described in Introduction to pilot and feasibility studies, pilot and feasibility studies are conducted to inform a later definitive study. Their aim is to fill gaps in knowledge that need to be clarified before progressing to a definitive study. The intention is that by resolving points of uncertainty at an early stage the ultimate study is more likely to succeed.

The CONSORT extension for pilot and feasibility trials can be used for designing and planning a pilot trial, not just reporting.

Additional guidance has also been published for reporting of pilot trial protocols, and non-randomised studies.

Design: Example objectives

The Introduction to pilot and feasibility studies web page gives an overview of why to do pilot and feasibility studies. The objectives for feasibility and pilot studies should focus on feasibility and areas of uncertainty about that feasibility in a putative trial of effectiveness or efficacy. Here we present some example objectives, though we stress that investigators should base their own objectives on the uncertainties about feasibility to be addressed in their own putative definitive trial:

  • Confirming whether the proposed intervention is needed
  • Confirming whether the proposed intervention is acceptable to all stakeholders
  • Investigating whether the relevant population will agree to take part in a potential randomised study
  • Investigating the feasibility of recruiting under-represented groups
  • Checking whether participants can be recruited
  • Checking whether participants can be retained
  • Testing the integrity of the study protocol
  • Investigating whether blinding can be achieved
  • Testing the randomisation procedure
  • Testing data collection procedures
  • Obtaining estimates to help with sample size calculation
  • Exploring and/or selecting potential outcome measures, including selecting the primary outcome

The following thesis on the role of progression criteria in the design, conduct, analysis and reporting of external randomised pilot trials provides a useful review of example feasibility objectives contributing to progression criteria.

Effect sizes

The objective of a pilot or feasibility study should not be to assess effectiveness or efficacy, because, by definition, the study has not been powered to assess this. Evaluating effectiveness would normally be the primary objective of the definitive study. See Introduction to pilot and feasibility studies and Resources – analysis for more on this.

Authors sometimes use the effect size determined in a pilot study to power a main study assessing effectiveness or efficacy, but this focuses on the effect that they think might be possible to achieve rather than the effect that it is important to detect. Pilot studies are normally too small to estimate treatment effects precisely. The inaccurate pilot study effect size may overestimate the true effect size, leading to a seriously underpowered main study.

Further reading 

The following papers give further discussion about objectives in pilot and feasibility studies:

Design: Choosing the design

The design and data collection for a pilot or feasibility study should be appropriate to answer the pre-specified objectives. It should be stated explicitly which objectives will be answered using quantitative methods and which using qualitative methods, and it might be beneficial to separate the objectives into primary objective(s) and secondary objectives.

For pilot trials, primary objective(s) are often related to decisions about whether to progress to a future definitive randomised controlled trial or not. Secondary objectives, on the other hand, may focus on the way in which the pilot trial progresses. This distinction may look different for feasibility studies where the next step in the trial trajectory may be another pilot or feasibility study.

As stated, some objectives might be better answered by a qualitative approach using interviews or focus groups (is the proposed intervention acceptable?), some by a non-randomised intervention study (can the necessary data be collected, can participants be retained?), and others by a pilot trial (can participants be recruited and randomised?). 

The following examples use varying designs to answer their objectives:

Mixed methods study

Objective: To assess the feasibility of establishing a comprehensive primary angioplasty service. We aimed to compare satisfaction at intervention hospitals offering angioplasty-based care and control hospitals offering thrombolysis-based care.

Outcomes: Postal survey of patients and their carers, supported by semi-structured interviews.

Survey

Objective: To determine feasibility of an RCT comparing operative with non-operative treatment for femoroacetabular impingement surgery.

Outcomes: Surgeon and patient opinion via a questionnaire.

Single arm trial

(intervention without a control group)

Objective: To pilot an intervention to avoid the use of syringes and contamination of materials amongst injecting drug users.

Outcomes: Adoption of each of four components; whether pre-post changes in blood residues indicated that intervention merited further testing.

Randomised controlled trial

Objective: To assess feasibility of a randomised controlled trial management of reduced fetal movement.

Outcomes: Recruitment, retention, acceptability, adherence to protocol, prevalence of poor perinatal outcomes.

Design: Sample size

Justification should be given for the number of participants, or clusters and cluster size, included in the pilot or feasibility study. The sample size rationale should be based on the primary objective(s) of the pilot or feasibility study. The sample size should be large enough to address all the feasibility questions we want to answer.

For example, if the primary objective of the pilot or feasibility study is to estimate some quantitative measure (e.g. variance of an outcome variable to inform the sample size calculation for the future definitive trial, or rates of acceptance, recruitment, retention), then the sample size should be set to ensure a desired degree of precision around the estimate (although this may give too large a number and may need to be balanced with logistics). Most of the literature on sample size for a pilot trial assumes the objective is to estimate parameters for the sample size calculation for the future effectiveness or efficacy trial. In this case, one may decide the sample size for the pilot trial in relation to the sample size required in the future definitive trial.

However, the primary objective for most pilot and feasibility studies is related to assessing the feasibility of trial processes and intervention implementation.

The CONSORT extension states: “Many pilot trials have key objectives related to estimating rates of acceptance, recruitment, retention, or uptake … for these sorts of objectives, numbers required in the study should ideally be set to ensure a desired degree of precision around the estimated rate.” The CONSORT extension also recommends that researchers formulate decision processes, which “might involve formal progression criteria to decide whether to proceed, to proceed with amendments, or not to proceed with progression to a definitive trial”.

Some researchers have developed a method to calculate sample size which is based on chosen progression criteria. There is also a freely-accessible website that introduces the methods and how to use the tool for sample size calculation and evaluation of pilot studies that are designed to formally assess multiple progression criteria.

The review below reported the ‘usual’ target sample size of a pilot trial to be 30 per arm (median) and 20 to 50 per arm (interquartile range); though the figures were slightly different for continuous (median 30, IQR 20 to 43) versus binary (median 50, IQR 25 to 81) outcomes.

A review of school-based cluster randomised pilot trials found the median (IQR; range) achieved sample size was 7.5 (4.5 to 9; 2 to 37) schools, 8 (5.5 to 9.5; 2 to 37) clusters and 274 (179 to 557; 29 to 1567) pupils. The planned sample size was 7.5 (5 to 8; 2 to 20) schools, 7.5 (5 to 8; 2 to 20) clusters and 320 (150 to 1,200; 50 to 1852) pupils. 

Design: Informed consent

Khan et al. assessed the transparency of informed consent in pilot and feasibility studies and found that transparency of informed consent is often inadequate, and Sim identifies some of the associated ethical issues.

Design: Progression criteria

Progression criteria are pre-specified criteria on which to base the decision about whether or not to proceed to the next stage in the research process. 

Avery et al. propose a traffic light system for specifying progression criteria, where green (or go) indicates that the criteria have been met and the trial should proceed, amber (or amend) indicates that some changes should be made to the larger trial, and red (or stop) indicates that the investigators should not move forward with the larger trial. 

Mellor et al. have provided recommendations for progression criteria during external randomised pilot trial design, conduct, analysis and reporting:

The following papers provide detail on usual items that are included for assessing progression but also indicate that many pilot and feasibility studies fail to include clear progression criteria and reporting is often inadequate.

As detailed in Design: Sample Size, the following paper looks at determining sample size for progression criteria.

The following paper is an example of a methodological study to determine empirical progression criteria thresholds for feasibility outcomes to inform future HIV pilot randomised trials:

Design: Pragmatic and implementation trials

There are unique considerations and areas of uncertainty that should be taken into account in pilot studies for pragmatic and implementation trials.

The following paper looks at the elements of pragmatic trials that might need to be examined as part of a pilot study, and how we might design a pilot or feasibility study to address these areas of uncertainty:

This paper provides a resource for those conducting pilot or feasibility studies in advance of large-scale implementation trials:

Reporting: Introduction

Good reporting of pilot and feasibility studies is needed to inform other researchers who might want to use the results when preparing for similar future definitive trials, and to show how the pilot or feasibility study could inform the future definitive trial. This page provides resources to help with the reporting of pilot and feasibility studies. Researchers are encouraged to follow the reporting recommendations for pilot and feasibility studies:

Reporting: Current reporting quality

Current reporting of pilot and feasibility studies needs improving, as shown by the following reviews of these studies:

Friedman argues why we should report results from pilot studies in the following paper:

Reporting: Guidelines for reporting

CONSORT stands for Consolidated Standards of Reporting Trials and encompasses various initiatives developed by the CONSORT Group to alleviate the problems arising from inadequate reporting of randomised controlled trials. The main product of CONSORT is the CONSORT Statement, which is an evidence-based, minimum set of recommendations for reporting randomised trials. It offers a standard way for authors to prepare reports of trial findings, facilitating their complete and transparent reporting, and aiding their critical appraisal and interpretation.

The CONSORT statement has recently been updated:

Initially developed for individually randomised trials, an extension for randomised pilot and feasibility trials was developed. The extension applies to CONSORT 2010 but can be used in conjunction with the new CONSORT 2025. The CONSORT extension for randomised pilot and feasibility trials aims to provide reporting guidance for any randomised study in which a future definitive randomised trial, or part of it, is conducted on a smaller scale, regardless of its design (e.g., cluster, factorial, crossover) or the terms used by authors to describe the study (e.g., pilot, feasibility, trial, study). This extension does not directly apply to internal pilot studies built into the design of a main trial, non-randomised pilot and feasibility studies, or phase II studies, but these studies all have some similarities to randomised pilot and feasibility studies and so many of the principles might also apply.
There are some key differences between pilot and feasibility trials and randomised trials designed to evaluate effectiveness or efficacy, particularly in the type of information that needs to be reported and in the interpretation of standard CONSORT reporting items. Some of the original CONSORT Statement items are retained, but most have been adapted, some removed, and new items added.
The CONSORT extension for reporting of randomised pilot and feasibility trials comprises a 26-item checklist and a flow diagram. The checklist items focus on reporting how the trial was designed, analysed, and interpreted; the flow diagram displays the progress of all participants through the trial (see Available downloads below). The CONSORT extension document explains and illustrates the principles underlying the CONSORT extension for randomised pilot and feasibility trials:

The following paper outlines the methods and processes used to develop the CONSORT extension:

The following editorials have been published to introduce the CONSORT extension for pilot trials:

Reporting: Available downloads
Reporting: Pilot and feasibility studies journal

In 2015, the journal Pilot and Feasibility Studies was launched for papers about pilot and feasibility studies (for example, protocols, manuscripts about design and analysis, discussion and review articles, and reports of completed pilot and feasibility studies):

The following editorials relevant to reporting have been published in this journal or elsewhere:

Analysis: Introduction

Lancaster et al. and Grimes and Schultz suggest that analysis of pilot studies should mainly be descriptive, as hypothesis testing requires a powered sample size which is usually not available in pilot studies:

Thabane et al. and Arain et al. state that any testing of an intervention needs to be reported cautiously:

Remember that some trials with small sample sizes may be incorrectly labelled as a ‘pilot’. If a trial is assessing clinical effectiveness as a primary outcome the original CONSORT statement for RCTs should be applied.

The recommendation not to carry out hypothesis testing in pilot trials is directed toward averting inappropriate analysis of clinical outcomes (since the study will be underpowered and the evaluation of clinical effectiveness is the aim of the definitive trial). However, it may be reasonable to carry out formal statistical evaluation of feasibility outcomes if the sample size of the pilot study is designed for this purpose (see Design: Sample size – sample size for progression criteria)

Analysis: Value of information approach

The idea behind the value of information (VOI) approach is that errors are costly and information is valuable since it reduces the risk of making wrong judgments. VOI analysis quantifies the uncertainty surrounding trial results, estimates the expected benefits of reducing this uncertainty with additional research such as a trial, and subsequently informs optimal future trial design. Based on this approach, if the expected benefit of an intended clinical trial outweighs its expected cost, then the study is potentially worthwhile. VOI analysis can also optimize additional aspects of research design such as possible comparator arms and alternative follow-up periods, by considering trial designs that optimize the expected benefits of research.

 

Analysis: Items from the CONSORT extension for pilot trials

The CONSORT extension for pilot trials (See Reporting: Guidelines for reporting) states that the objectives of a pilot trial should determine what is reported. Methods should be specified for how each of the pilot or feasibility study objectives will be addressed, and this can be qualitative or quantitative.

The CONSORT extension for pilot trials has three items related to the analysis of pilot trials. Although the CONSORT extension is for reporting, it is also useful for suggesting appropriate analysis.The examples and text that follows are quotations from this paper:

Item 17a

For each objective, results including expressions of uncertainty (such as 95% confidence interval) for any estimates. If relevant, these results should be by randomised group.

Example 1 given in CONSORT statement (feasibility outcome):

“The ABSORB [A bioabsorbable everolimus-eluting coronary stent system for patients with single de-novo coronary artery lesions] study aimed to assess the feasibility and safety of the BVS [bioasorbable everolimus- eluting stent] stent in patients with single de-novo coronary artery lesions. …Procedural success was 100% (30/30 patients), and device success 94% (29/31 attempts at implantation of the stent).”

“It is important that the reported results of a pilot trial reflect the objectives. Results might include, for example, recruitment, retention or response rates, or other sorts of rates, as in example 1. Because the sample size in a pilot trial is likely to be small, estimates of these rates will be imprecise and this imprecision should be recognised, for example, by calculating a confidence interval around the estimate. Commonly, authors do not give such a confidence interval, but if the numerator and denominator are given the confidence interval can be calculated. In example 1 the Wilson 95% confidence interval for 100% (30/30) is 88.65% to 100% and for 94% (29/31) is 79.78% to 98.21%.”

See this practical resource for calculating a confidence interval.

“If authors do report differences between trial arms (and this is not necessary if it is not consistent with the objectives of the trial) then confidence intervals again provide readers with an assessment of precision (example 2), which usually indicates considerable uncertainty.”

Example 2 given in CONSORT statement (proposed outcome in future definitive trial):

“Rates of initiation of lifestyle change also favoured the individualized assessment arm but less clearly. At 3 months, 75% of the individualized assessment arm and 68% of the usual assessment arm had initiated changes in their lifestyle (unadjusted odds ratio, 1.38 [95%CI, 0.55 to 3.52]). At 6 months, the percentages were 85% and 75%, suggesting increased initiation of change over time in both arms, with the gap widening slightly (unadjusted odds ratio, 1.86 [95% CI, 0.64 to 5.77]) . Wide CIs again point to the degree of uncertainty around this conclusion”

“If samples in the pilot trial and future definitive RCT are drawn from slightly different populations, confidence intervals calculated from the pilot will not directly indicate the likely upper and lower bounds of the relevant measure in the future definitive RCT, but can nevertheless highlight the lack of precision effectively.”

Item 17b

The original CONSORT statement states that one should give both the relative risk and absolute difference if the outcome is a proportion: However, in pilot trials this is not applicable.

“Because of the imprecision of estimates from these trials and the fact that samples in these trials can be unrepresentative (see item 17a), we caution against any reliance on estimates of effect size from pilot trials for clinical implications. Information from outcome data, however, can be legitimately used for other purposes, such as estimating inputs for sample size for the future definitive RCT. Thus item 17b, which is underpinned by rationale around clinical implications, is not applicable.”

Item 18

Results of any other analyses performed that could be used to inform the future definitive trial.

Example given in CONSORT statement “Sensitivity analysis. At both six and 12 weeks, findings were insensitive to the exclusion of those catheterised throughout their hospital stay (and also to the exclusion of those who were never incontinent following the removal of a catheter). However, at both time points, odds ratios reduced when those with pre-stroke incontinence were excluded.”

“It is possible that the results of analyses that were not initially planned might have important implications for the future definitive RCT. Such findings should be reported and discussed in relation to how they might inform the future definitive RCT. In the example, although numbers were small, the authors inferred from the unplanned sensitivity analyses that those with pre-stroke incontinence were at least as likely, or more likely, to benefit from the intervention than those continent pre-stroke, and concluded that this group of patients should be included in the full trial.”

Analysis: Missing data

Dealing with missing data within an analysis of outcome data is less important in pilot and feasibility studies; what is more important is to investigate missing data and understand why they are missing and what can be done to prevent missing data in the future definitive study. Simple descriptive statistics can identify the extent of missing data for various questions/questionnaires in the dataset, and models can be fitted to see whether the probability of missingness is associated with baseline characteristics such as age, gender, ethnicity etc (subject to the sample size being sufficient to sustain such an analysis). Furthermore, qualitative work can be performed to discover the burden of data completion to patients and why some particular items may be incomplete.

Analysis: Deciding whether to proceed with a future trial

Since pilot trials aim to assess the feasibility of a definitive trial, the analysis should indicate whether researchers intend to proceed with the intended future trial. Researchers should also consider how the pilot trial findings have informed the future definitive trial design, for example, if changes to the proposed design have been made based on their pilot trial findings. If applicable, this should also be based on any progression criteria that were pre-specified to inform this decision.

The following paper also provides an example of how this decision can be made by considering problems faced during the pilot trial and potential solutions to those problems:

Decision-making on progression for the formal evaluation of feasibility outcomes using the method of Lewis et al (2021) [highlighted in the Design – Sample size section] should consider multi-criteria assessment.