Resources for pilot and feasibility studies

Pilot and feasibility studies differ from studies to evaluate the effectiveness or efficacy of an intervention in that they are concerned with addressing areas of uncertainty and assessing feasibility. This different focus affects objectives and means that their design, analysis, and reporting are different and need special consideration.

For example, a pilot trial assessing feasibility of conducting a future definitive trial might collect data on items such as recruitment and retention to assess feasibility, analysis is likely to be based on descriptive statistics since the study is not powered for formal hypothesis testing for effectiveness/efficacy, and reporting would need to include detail on items such as identification and consent of participants, criteria for deciding whether to progress to the main trial, unintended consequences during the trial, and implications for progression.

This page provides resources to help trialists and researchers to conduct a pilot or feasibility study.   

See Introduction to pilot and feasibility studies for an overview of pilot and feasibility studies.

See also our mini webinar series highlighting key points to consider when conducting pilot and feasibility studies prior to a definitive randomised controlled trial. 

Resources below are separated into sections on Design, Analysis and Reporting.

Design
Analysis
Reporting
Design: Introduction

As is described in Introduction to pilot and feasibility studies, pilot and feasibility studies are conducted to inform a later definitive study. Their aim is to fill gaps in knowledge that need to be clarified before progressing to a full study. The intention is that by resolving points of uncertainty at an early stage the ultimate study is more likely to succeed.

The CONSORT extension for pilot trials can be used for designing and planning a pilot trial, not just reporting.

Additional guidance has also been published for reporting of pilot trial protocols.

Design: Example objectives

The Introduction to pilot and feasibility studies web page gives an overview of why to do pilot and feasibility studies. The objectives for feasibility and pilot studies should focus on feasibility and areas of uncertainty about that feasibility in a putative trial of effectiveness or efficacy. Here we present some example objectives, though we stress that investigators should base their own objectives on the uncertainties about feasibility in their own studies:

  • Confirming whether the proposed intervention is needed
  • Confirming whether the proposed intervention is acceptable to all stakeholders
  • Investigating whether the relevant population will agree to take part in a potential randomised study
  • Checking participants can be recruited
  • Checking participants can be retained
  • Testing integrity of study protocol
  • Testing randomisation procedure
  • Testing data collection
  • Obtaining estimates to help with sample size calculation
  • Exploring potential outcome measures
Effect sizes

The objective of a pilot or feasibility study should not be to assess effectiveness or efficacy, because by definition the study has not been powered to assess this. Effectiveness would normally be the primary outcome of the ultimate definitive study. See Introduction to pilot and feasibility studies and Resourcess – analysis for more on this.

Authors sometimes use the effect size determined in a pilot study to power a main study assessing effectiveness or efficacy, but this can lead to a seriously underpowered main study. It is discouraged because pilot studies are not powered to estimate effect sizes:

Further reading 

The following papers give further discussion about objectives in pilot and feasibility studies:

Design: Choosing the design

The design and data collection for a pilot or feasibility study should be appropriate to answer the pre-specified objectives. It should be stated explicitly which objectives will be answered using quantitative methods and which using qualitative methods, and it might be beneficial to separate the objectives into primary objectives (often those on which decisions about progressing to a future definitive randomised controlled trial (RCT) may be made) and secondary objectives.

For the example, some objectives might be better answered by a qualitative approach using interviews or focus groups (is the proposed intervention acceptable?), some by a non-randomised intervention study (can the necessary data be collected, can participants be retained?), and others by a pilot trial (can participants be recruited and randomised?).

The following examples use varying designs to answer their objectives:

Mixed methods study

Objective: To assess the feasibility of establishing a comprehensive primary angioplasty service. We aimed to compare satisfaction at intervention hospitals offering angioplasty-based care and control hospitals offering thrombolysis-based care.

Outcomes: Postal survey of patients and their carers, supported by semi-structured interviews

Survey

Objective: To determine feasibility of RCT comparing operative with non-operative treatment for femoroacetabular impingement surgery

Outcomes: Surgeon and patient opinion via a questionnaire

Single arm trial

(intervention without a control group)

Objective: To pilot an intervention to avoid the use of syringes and contamination of materials amongst injecting drug users

Outcomes: Adoption of each of four components; whether pre-post changes in blood residues indicated that intervention merited further testing

Randomised controlled trial

Objective: To assess feasibility of randomised controlled trial (RCT) management of reduced fetal movement

Outcomes: Recruitment, retention, acceptability, adherence to protocol, prevalence of poor perinatal outcomes

Design: Sample size

Justification should be given for the numbers included in the pilot or feasibility study. The sample size rationale should be based on the primary objective of the pilot or feasibility study.

For example, if the primary objective of the pilot or feasibility study is to estimate some quantitative measure (e.g. variance of an effect size to inform sample size calculation for future definitive trial, or rates of acceptance, recruitment, retention), then the sample size should be set to ensure a desired degree of precision around the estimate (although this may give too large a number and need to be balanced with logistics). Most of the literature on sample size for a pilot trial assumes the objective is to estimate parameters for the sample size calculation for the future effectiveness or efficacy trial. In this case, one may decide the sample size for the pilot trial in relation to the sample size required in the future definitive trial.

However, the primary objective for most pilot and feasibility studies is related to assessing the feasibility of trial processes and intervention implementation. The sample size rationale can be qualitative or quantitative, so in this scenario may be based on logistics, resources, and time.

Design: Informed consent

Khan et al. assessed the transparency of informed consent in pilot and feasibility studies and found that transparency of informed consent is often inadequate

Design: Progression criteria

Progression criteria are pre-specified criteria on which to base the decision about whether or not to proceed to the next stage in the research process. 

Avery et al. propose a traffic light system for specifying progression criteria, where green (or go) indicates that the criteria have been met and the trial should proceed, amber (or amend) indicates that some changes should be made to the larger trial, and red (or stop) indicates that the investigators should not move forward with the larger trial. 

A recent review of progression criteria indicates that reporting is inadequate.

The following paper looks at determining sample size for progression criteria.

Design: Pragmatic trials

There are unique considerations and areas of uncertainty that should be taken into account in pilot studies for pragmatic trials. The following paper looks at the elements of pragmatic trials that might need to be examined as part of a pilot study, and how we might design a pilot or feasibility study to address these areas of uncertainty.

Reporting: Introduction

Good reporting of pilot and feasibility studies is needed to inform other researchers who might want to use the results when preparing for similar future definitive trials, and to show how the pilot or feasibility study could inform the future definitive trial. This page provides resources to help with the reporting of pilot and feasibility studies.

Reporting: Current reporting quality

Current reporting of pilot and feasibility studies needs improving, as shown by the following reviews of these studies:

Friedman argues why we should report results from pilot studies in the following paper:

Reporting: Guidelines for reporting

CONSORT stands for Consolidated Standards of Reporting Trials and encompasses various initiatives developed by the CONSORT Group to alleviate the problems arising from inadequate reporting of randomised controlled trials. The main product of CONSORT is the CONSORT Statement, which is an evidence-based, minimum set of recommendations for reporting randomised trials. It offers a standard way for authors to prepare reports of trial findings, facilitating their complete and transparent reporting, and aiding their critical appraisal and interpretation.

Initially developed for individually randomised trials, an extension for randomised pilot and feasibility trials is now available. The CONSORT extension for randomised pilot and feasibility trials aims to provide reporting guidance for any randomised study in which a future definitive randomised trial, or part of it, is conducted on a smaller scale, regardless of its design (e.g., cluster, factorial, crossover) or the terms used by authors to describe the study (e.g., pilot, feasibility, trial, study). This extension does not directly apply to internal pilot studies built into the design of a main trial, non-randomised pilot and feasibility studies, or phase II studies, but these studies all have some similarities to randomised pilot and feasibility studies and so many of the principles might also apply. 

There are some key differences in pilot and feasibility trials from randomised trials designed to evaluate effectiveness or efficacy, particularly in the type of information that needs to be reported and in the interpretation of standard CONSORT reporting items. Some of the original CONSORT Statement items are retained, but most have been adapted, some removed, and new items added.

The CONSORT extension for reporting of randomised pilot and feasibility trials comprises a 26-item checklist and a flow diagram. The checklist items focus on reporting how the trial was designed, analysed, and interpreted; the flow diagram displays the progress of all participants through the trial (see Available downloads below). The CONSORT extension document explains and illustrates the principles underlying the CONSORT extension for randomised pilot and feasibility trials:

The following paper outlines the methods and processes used to develop the CONSORT extension:

The following editorials have been published to introduce the CONSORT extension for pilot trials:

Reporting: Available downloads
Reporting: Pilot and feasibility studies journal

In 2015, the journal Pilot and Feasibility Studies was launched for papers about pilot and feasibility studies (for example, protocols, manuscripts about design and analysis, and discussion and review articles):

The following editorials relevant to reporting have been published in this journal or elsewhere:

Analysis: Introduction

Lancaster et al. and Grimes and Schultz suggest that analysis of pilot studies should mainly be descriptive, as hypothesis testing requires a powered sample size which is usually not available in pilot studies:

Thabane et al. and Arain et al. state that any testing of an intervention needs to be reported cautiously:

Remember that some trials with small sample sizes may be incorrectly labelled as a ‘pilot’. If a trial is assessing effectiveness as a primary outcome the original CONSORT statement for RCTs should be applied.

Analysis: Value of information approach

One analysis of a pilot that could be undertaken is to decide the sample size for a subsequent trial using the value of information (VOI) approach:

The idea is that errors are costly and information is valuable since it reduces the risk of making wrong judgments. VOI analysis quantifies the uncertainty surrounding trial results, estimates the expected benefits of reducing this uncertainty with additional research such as a trial, and subsequently informs optimal future trial design. Based on this approach, if the expected benefit of an intended clinical trial outweighs its expected cost, then the study is potentially worthwhile. VOI analysis can also optimize additional aspects of research design such as possible comparator arms and alternative follow-up periods, by considering trial designs that optimize the expected benefits of research.

Analysis: Items from the CONSORT extension for pilot trials

The recently published CONSORT extension for pilot trials states that the objective of a pilot trial should determine what is reported. Methods should be specified for how each of the pilot or feasibility study objectives will be addressed, and this can be qualitative or quantitative.

The CONSORT extension for pilot trials has three items related to the analysis of pilot trials. Although the CONSORT extension is for reporting, it is also useful for suggesting appropriate analysis.The examples and text that follows are quotes from this paper:

Item 17a

For each objective, results including expressions of uncertainty (such as 95% confidence interval) for any estimates. If relevant, these results should be by randomised group.

Example 1 given in CONSORT statement (feasibility outcome):

“The ABSORB [A bioabsorbable everolimus-eluting coronary stent system for patients with single de-novo coronary artery lesions] study aimed to assess the feasibility and safety of the BVS [bioasorbable everolimus- eluting stent] stent in patients with single de-novo coronary artery lesions. …Procedural success was 100% (30/30 patients), and device success 94% (29/31 attempts at implantation of the stent).”

“It is important that the reported results of a pilot trial reflect the objectives. Results might include, for example, recruitment, retention or response rates, or other sorts of rates, as in example 1. Because the sample size in a pilot trial is likely to be small, estimates of these rates will be imprecise and this imprecision should be recognised, for example, by calculating a confidence interval around the estimate. Commonly, authors do not give such a confidence interval, but if the numerator and denominator are given the confidence interval can be calculated. In example 1 the Wilson 95% confidence interval for 100% (30/30) is 88.65% to 100% and for 94% (29/31) is 79.78% to 98.21%.”

See this practical resource for calculating a confidence interval.

“If authors do report differences between trial arms (and this is not necessary if it is not consistent with the objectives of the trial) then confidence intervals again provide readers with an assessment of precision (example 2), which usually indicates considerable uncertainty.”

Example 2 given in CONSORT statement (proposed outcome in future definitive trial):

“Rates of initiation of lifestyle change also favoured the individualized assessment arm but less clearly. At 3 months, 75% of the individualized assessment arm and 68% of the usual assessment arm had initiated changes in their lifestyle (unadjusted odds ratio, 1.38 [95%CI, 0.55 to 3.52]). At 6 months, the percentages were 85% and 75%, suggesting increased initiation of change over time in both arms, with the gap widening slightly (unadjusted odds ratio, 1.86 [95% CI, 0.64 to 5.77]) . Wide CIs again point to the degree of uncertainty around this conclusion”

“If samples in the pilot trial and future definitive RCT are drawn from slightly different populations, confidence intervals calculated from the pilot will not directly indicate the likely upper and lower bounds of the relevant measure in the future definitive RCT, but can nevertheless highlight the lack of precision effectively.”

Item 17b

In the original CONSORT statement states that one should give both the relative risk and absolute difference if the outcome is a proportion: However, in pilot trials this is not applicable.

“Because of the imprecision of estimates from these trials and the fact that samples in these trials can be unrepresentative (see item 17a), we caution against any reliance on estimates of effect size from pilot trials for clinical implications. Information from outcome data, however, can be legitimately used for other purposes, such as estimating inputs for sample size for the future definitive RCT. Thus item 17b, which is underpinned by rationale around clinical implications, is not applicable.”

Item 18

Results of any other analyses performed that could be used to inform the future definitive trial.

Example given in CONSORT statement “Sensitivity analysis. At both six and 12 weeks, findings were insensitive to the exclusion of those catheterised throughout their hospital stay (and also to the exclusion of those who were never incontinent following the removal of a catheter). However, at both time points, odds ratios reduced when those with pre-stroke incontinence were excluded.”

“It is possible that the results of analyses that were not initially planned might have important implications for the future definitive RCT. Such findings should be reported and discussed in relation to how they might inform the future definitive RCT. In the example, although numbers were small, the authors inferred from the unplanned sensitivity analyses that those with pre-stroke incontinence were at least as likely, or more likely, to benefit from the intervention than those continent pre-stroke, and concluded that this group of patients should be included in the full trial.”

Analysis: Missing data

Dealing with missing data is less important in pilot and feasibility studies; what is more important is to investigate missing data and understand why they are missing and what can be done to prevent missing data in the future definitive study. Simple descriptive statistics can identify the extent of missing data for various questions/questionnaires in the dataset, and models can be fitted to see whether the probability of missingness is associated with baseline characteristics such as age, gender, ethnicity etc. Furthermore, qualitative work can be performed to discover the burden of data completion to patients and why some particular items may be incomplete.

Analysis: Deciding whether to proceed with a future trial

Since pilot trials aim to assess the feasibility of a definitive trial, the analysis should indicate whether researchers intend to proceed with the intended future trial. Researchers should also consider how the pilot trial findings have informed the future definitive trial design, for example if changes to the proposed design have been made based on their pilot trial findings. If applicable, this should also be based on any progression criteria that was pre-specified to inform this decision.

The following paper also provides an example of how this decision can be made by considering problems faced during the pilot trial and potential solutions to those problems: