Guidance

This is best practice guidance

Although not legally required, it's an essential activity.

This Guide covers:

- England

From:

- National Institute for Health and Care Excellence (NICE)

Page last reviewed: 22 Jan 2023

Piloting digital technologies in a health or care service

Before deciding whether to adopt a digital healthcare technology, you may need to pilot it in your service.

Understanding local piloting

Before adopting a digital healthcare technology, you should have evidence showing its clinical and cost effectiveness in a setting like yours. If evidence relevant to your context is limited, the developer should have plans to generate it. Testing the technology in your service (local piloting) may be part of this plan.

You might do a pilot study for other reasons, including to:

  • learn about potential risks or technical and implementation issues associated with the technology in a controlled way before ‘going live’ (deploying) it
  • understand how the technology affects current care or operational workflows, and whether this creates risks or issues that need to be managed
  • learn how to use the technology most effectively and adjust processes to get the best value from it
  • plan processes for monitoring the technology after it is deployed

Piloting gives stakeholders confidence that the technology creates benefits in your setting, and that any decision to adopt it is well-evidenced.

Planning a pilot study

Your pilot study should have:

  • a defined scope
  • a data-collection plan
  • clear criteria for success or failure
  • a defined time period (start and finish dates)
  • defined roles and responsibilities for you and the developer so that both your interests are met appropriately

The pilot study should generate results that help you decide whether to continue using the technology. So, your data-collection plan should define what and how much data to collect and how it will be analysed. This will allow you to assess the benefits of the technology. Types of data you might collect include:

  • healthcare professionals’ experiences of using the technology
  • experiences of patients or service users
  • clinical or care outcomes
  • cost effectiveness
  • efficiency gains or losses
  • operational or financial impacts on your service

Resources to help you design a pilot study

Resources that may help you design a pilot study include:

  • NICE’s real-world evidence framework. This provides general guidance on planning, doing and reporting real-world evidence studies. This will help because you’ll be doing the pilot study in a real-world setting, using data collected routinely in your service
  • The DECIDE-AI reporting guidelines, which aim to improve reporting of studies
  • NICE’s evidence standards framework for digital health technologies. This provides standards on evidence requirements for different types of digital healthcare technologies (which it refers to as DHTs). See the table below for evidence standards that are particularly relevant for thinking through different stages of pilot development

Relevant NICE evidence standards for developing a pilot

Planning a pilot study

Make sure the value proposition for how the technology will fit into local clinical pathways and its expected benefits are clear:

Planning for deployment

Make sure the technology deployment plan for the pilot provides relevant information about deployment requirements:

Clinical or care outcomes

Identify the data collection requirements to demonstrate the relevant outcomes:

Experiences of patients or service users

Identify the data collection requirements to demonstrate the relevant outcomes:

Experiences of health care professionals

Identify the data collection requirements to demonstrate the relevant outcomes:

Efficiency gains or losses and their operational or financial impact on your service

Identify the data collection requirements to demonstrate the value of the technology:

Is there anything wrong with this page? Let us know