Introduction

Classical medical technology assessment (MTA) is focused on the analysis of the costs and benefits of a technology from various perspectives, such as economic, clinical or policy perspective [1]. The definition of MTA is the analysis of the implications of a medical technology in terms of its safety, efficiency, effectiveness, accessibility and equity, with the aim of supporting appropriate use of medical technologies by improving input to decision-making in policy and practice [2]. These analyses are usually conducted at the end of the development process of a medical technology, typically after large clinical trials, when clinical and cost-effectiveness data are available [1]. The rationale is that a full and proper assessment can be made only when enough data are available. The main goal of classical MTA is to support health policymaking about market approval or reimbursement of a technology [3]. However, the methods employed in MTA can be used in other ways. Some researchers have shown that similar methods can be conducted earlier in the development of a technology. The relevance of early MTA is that it could help to allocate public support effectively. Perhaps more importantly, from the industry perspective it can also inform research and development decisions to increase the chance of later market approval and reimbursement [4]. Relevant information acquired in an early stage can lead to changes that will improve the device during the development process in order to produce the most beneficial medical technology for society [1]. The main difference between classical and early MTA is that classical MTA is conducted to support decision-making by regulators, payers and patients about the overall value of a technology, while early MTA helps manufacturers and investors to decide about the management of the development, as well as their regulatory and reimbursement strategy [1].

Different tools are available to perform early MTA studies, including early health economic modelling, clinical trial simulation and multi-criteria decision analysis [4]. However, the number of published articles on this topic is very limited. The aim of this study is to describe the most commonly used methods in early MTAs of emerging technologies and examine which methods have been used in early MTAs of medical devices and tests. 

Materials and methods

Since the research question of the difference between early MTA studies of medical tests and other technologies was too specific and focused to employ one specific searching keyword, an explorative literature research was conducted. The following databases were used: PubMed, The Cochrane Library, Embase, and Google Scholar by various keywords and MESH terms (health technology assessment, early technology assessment, early health technology assessment, medical technology assessment, economic evaluation, early stage, emerging technology, drugs, medical devices, and medical tests). In addition, the reference lists of relevant publications were examined. Since the literature base on this topic is very limited we did not make any restrictions about the year of the publication, but search only for articles published in English.

Results

Differences between classical and early medical technology assessments

Classical MTA is usually conducted at the end of the development, when data is available about efficacy and safety, which are usually derived from clinical trials. At that stage the technology is ready to be introduced to the market and the main investments have already been made. If the technology does not obtain market approval or reimbursement, the manufacturer or the pharmaceutical firm can face serious financial consequences [1]. In the last decade, many parties have recognized that economic analyses can be conducted earlier in the development process to obtain optimal future results. This would help the industry to produce technologies which are going to get market approval and reimbursement from the national health insurers. However the basic steps of classical and early technology assessment are the same, such as decisions about the design of the study, measuring and valuing costs, measuring and valuing benefits, discounting, sensitivity analysis, which plays a more important role in early MTA, and finally, applying a decision rule, e.g. calculating an incremental cost-effectiveness ratio (ICER) [5]. It is difficult to define the cut-off point between classical and early MTA, which is before the technology is introduced to the market (Figure 1).

Figure 1. Medical technology development

Figure 1 shows how technology can develop over time, starting first as an idea or concept, which is converted into a prototype that it later studied using steadily more rigorous methods. According to Vallejo-Torres et al. the development process of a medical device can be divided into three stages: early, mid and late stage [8]. Since clinical and economic data are not yet available in the early stage, MTA has to rely on assumptions about these parameters [8]. In the mid stage, uncertainty about the effects and costs still plays a role, but some evidence from pre-clinical studies is available. The goal is to identify the parameters which have the biggest impact on cost-effectiveness estimates and reduce uncertainty with clinical evidence. In the late stage, all data from clinical studies is available and much less uncertainty plays role. These studies are already considered as classical MTA and designed to inform market approval and reimbursement decisions.

Economic evaluations, or cost-effectiveness analyses, represent a frequently used component in classical HTAs but can, of course, be used in early HTAs. According to Hartz and John there are six different applications of an early economic evaluation. These are shown in the first row of Figure 1 and are listed below [4].

• In the case of strategic R&D decision making, economic evaluation helps the manufacturer to avoid investing in potentially unsuccessful products.
• In pre-clinical preliminary market assessments, a prototype of the product is already available and the manufacturer or the investor would like to know what the potential target population, epidemiological factors, costs and effects are. For this purpose, they need data about the cost-effectiveness of the current therapies, because the less effective the available technologies, the more likely the new technology will be cost-effective.
• Go/no go decisions need to be made at various points in time. Obviously the data available needs to be used optimally and the amount of data will change over time. For example, data from market assessments must be used properly (and perhaps together with an economic model) to decide whether to continue developing the technology.
• Early economic evaluations can also help to design future trials. Usually, this means the design of a phase III trial, which is performed to determine the clinical effectiveness of the medical technology. The identification of the input parameters that have the most impact on cost-effectiveness is a crucial issue. It could contribute to a better resource allocation and to decide what kind of methods and studies are needed during the trial.
• For assessment of future reimbursement and pricing scenarios, economic evaluation under different scenarios is carried out. These data could be useful for policy makers about the emerging technology for future planning.
• For price determination many types of information besides the results of an economic evaluation are needed, such as consumer willingness-to-pay (WTP) and market characteristics. However early economic evaluation or MTA is crucial for deciding if the new technology will be profitable in a given country or market. It could also help to identify the level of efficacy or effectiveness that needs to be obtained by the new technology for a given price [4].

The last row of Figure 1 shows the different questions raised by the manufacturer and investor in the different stages of the development process. These questions can be answered using early MTA studies [9].

In sum, early MTA (including economic evaluations) can be applied in different ways to plan the future development of a technology. It therefore has the potential to help the manufacture to produce a product that is profitable for them, beneficial for the patient and affordable and cost-effective for payers.

Methods used in early medical technology assessment

This section contains a non-exhaustive list of methods that can be used in early MTA studies of medical devices and diagnostic tests. This list is based on what was found in the literature regarding early MTA in general.

Early health economic modelling

Modelling is a frequently used technique in health economic evaluations, since they are simplified representations of real-life and therefore easy to use [12]. They can be used in many ways, such as converting efficacy to effectiveness or short-term results to long-term results [13]. Just as modelling is commonly used to perform economic evaluations, so can modelling be used to facilitate an early economic evaluation. Moreover, early modelling requires the same inputs as late models [4] and both rely on the same methods [13]. According to Annemans et al., it can function as an input into go/no go and priority setting decisions of the manufacturer, since it is able to predict the future economic value of the emerging technology. Early modelling can help to focus on potentially more cost-effective technologies and it can also serve with information for design further development. One special problem of early models is that a lot of uncertainty plays a role, due to a very limited data about the new technology and the inputs of the model [13]. Therefore, many scenarios have to be modelled during an early MTA.

Headroom method

The headroom method is a relatively simple threshold approach developed at the University of Birmingham that estimates the maximum amount that a technology could cost and yet still be considered cost-effective [6]. According to the developers, “the headroom method is an approach to help avoid misguidedly investing in those technologies that will never be cost-effective.” [6]. The main question to be answered is “Would it be cost-effective if it works as well as one would hope?” and the user can determine the range of prices at which the new technology would be cost-effective versus the comparator (e.g., current care).

The headroom method has three stages:

1. Strategic considerations, or structuring and defining the business problem situation.
2. Defining the clinical problem, or defining all conditions of the current treatment, strengths and weaknesses, as specifically as possible. This information will enable calculation of the effectiveness gap (maximum health gain in quality-adjusted life-years, QALYs) (maxΔQALY) assuming different scenarios (optimistic, realistic, pessimistic, etc.).
3. Headroom analysis, where headroom is defined by calculating the maximum incremental cost of the new technology versus the comparator by multiplying the maximum health gain by the willingness-to-pay to gain one QALY ( max ΔCost = WTP threshold*maxΔQALY).

The headroom method can help to make investment decisions without building a complex model with a lot of uncertainty. It is a useful tool for investors and manufacturers, because it provides information about the possible price in the future and the possible profit [6]. This tool could be used throughout the entire development process, since updating the inputs and recalculating the headroom will lead to better predictions about the potential cost-effectiveness.

There are also limitations of the headroom method. One important one is that it only works when a payer uses an explicit WTP threshold, such as the GBP 20 000 and GBP 30 000 thresholds in the UK. However, most countries do not have such an explicit threshold. Secondly, it only focuses on cost-effectiveness, when in fact reimbursement decisions may be based on other factors.

The Bayesian analytical framework

Bayesian statistics have been increasingly used in health economic evaluations over the past years. It is certainly a useful tool for early MTAs since it allows evaluations to be performed repeatedly as the knowledge base evolves [4]. Spiegelhalter et al. define the approach as “the explicit quantitative use of external evidence in the design, monitoring, analysis, interpretation and reporting of a health technology assessment.”[16]. It is a mathematical-statistical mechanism where a prior assumption about a parameter, usually a probability distribution, is modified by the new information. The two main questions that can be answered by the Bayesian approach are “how might new evidence change what we currently believe?” and “if we continue the study, what is the chance we will get a significant result?” [16]. Spiegelhalter and his colleagues have listed several advantages and disadvantages of this approach in a thorough review about the Bayesian methods. The main advantages are that all evidence regarding a specific problem can be taken into account, that potential biases can be explicitly modelled, and the outputs can be used as inputs in a later health economic model. However, the most important disadvantage may be that specification of expected utilities is difficult and may require many assumptions about the use of the new technology [17].

Since both diagnostic tests and medical devices are fast changing technologies, this approach could be a very useful tool for assessing their likely cost-effectiveness. According to Vallejo-Torres et al. the Bayesian Analytical Framework could help the development of new technologies in three ways:

• Enhancing the estimation of likely cost-effectiveness in the investment decision process, and avoiding investments in a technology that could never be cost-effective,
• Helping companies to prioritize and make the choice between competing possibly cost-effective ideas or prototypes,
• Identifying in the early stages of development those parameters that have the largest impact on the likely cost-effectiveness of the product.

The suggestion by Vallejo-Torres et al. is to start the development process with a simple health economic analysis and develop it further every time, when more data becomes available. They state that the Bayesian approach would be more feasible in the mid-stage of medical device development and it combines the new, but limited, data with the prevailing beliefs at that moment [8].

Clinical trial simulation

Clinical trial simulation (CTS) is a technique which synthesises available knowledge about the technology under development using mathematical relationships and models [18]. It can estimate different efficiency and tolerability profiles before clinical data are available [18]. It makes it possible to explore key assumptions before actual studies using human subjects and perform virtual studies to identify any weaknesses or limitations of the proposed study design [19,20]. Its use can therefore help manufacturers to minimize the duration and costs of technology development [21]. The aims of CTS are to maximise the use of information from previous phases of the development and thereby improve trial protocols, maximize the probability of meeting the targets of the trial and maximise the results that a trial can yield. It can help to improve efficiency and also supply information that would otherwise not be available by other means [4].

Clinical trial simulation is typically done by computer simulation, where the real-world situation is mapped and then the simulation is used to predict and describe the situation and investigate the assumptions. The simulation should capture all crucial aspects of the real world to help manufacturers draw some conclusions about further development design [22].

Most of the literature on CTS is about drug development, since clinical trials are much more important in the regulation and reimbursement policies for drugs than they are for medical device [18]. In drug development, CTS can help with dosage optimization, adaptation of a trial design and decisions about the optimal sample size and planning of the Phase III trial [4]. One interesting type of CTS is longitudinal stochastic modelling, which is a simulation technique that can describe individual behaviours. This could be important in assessments of medical devices and tests, due to learning effects and uncertainties about the usage of the device [20].

Multi-criteria decision analysis (MCDA)

Multi-criteria decision analysis (MCDA) is a method to support decisions between two or more discrete alternatives. It helps decision-makers in data organization and transparent decision making [9]. It has many validated methods, including analytic hierarchy process (AHP), conjoint analysis and contingent valuation. However, AHP is the only one that has been applied in early MTAs of medical devices. Further research about the usability of other MCDA methods in early MTAs would be valuable.

The analytic hierarchy process is a descriptive measurement theory which derives dominance priorities from a series of pairwise comparisons of homogeneous or similar elements on the basis of a common criterion or attribute, and then scales them using a hierarchy structure [23]. This process make it possible to include patient preferences beyond clinical effectiveness as well as other criteria not included in other approaches like economic evaluations. Therefore, its relevance for medical devices and diagnostic tests is noteworthy, since these other factors may play an important role in the uptake and cost-effectiveness of the technology [24]. Since its results can be used as inputs for health economic modelling and since it includes patient preferences and additional effects of the medical technology, this method could be used by both manufacturers, to make go/no go decisions about further development, and payers about market approval or reimbursement.

Hummel et al. has used AHP to elicit expected relative diagnostic effectiveness, patient comfort and safety data, and then converted these relative priorities to absolute estimations to compare a new diagnostic method for breast cancer (photo-acoustic mammography, PAM) with the current practice (magnetic resonance imaging, MRI) [24]. They then used these data as input in a health economic model (Markov model). They concluded that AHP can support the assessment of an emerging technology when clinical evidence is not available. However, they also added that the method has various methodological challenges, such as the best way to convert the relative AHP-derived priorities to absolute estimations and add weights to the additional criteria.

Value of information (VOI) analysis

The underlying principle of value of information (VOI) analysis is to compare the costs and benefits of obtaining additional information, or in other words, to assess the value of investing in further research [4]. It can answer different questions, such as “Should additional information be collected to better inform that decision?”[25]. The aim of the analysis is to calculate the expected value of perfect information (EVPI), which reflects the maximum possible payoff from additional research, since making wrong decisions has an opportunity cost and extra information is valuable if it reduces the chance of a wrong decision [18,26]. If the EVPI is higher than the cost of additional research, reducing uncertainty surrounding cost-effectiveness by performing research is beneficial [18,27]. EVPI reaches its maximum when the uncertainty is the highest about whether to continue or terminate the research and development of the new technology [28].

Originally, the expected cost of making decisions under uncertainty is equal to the EVPI, which is the maximum a decision-maker would be willing to pay to eliminate uncertainty. This can be derived from the probability that the decision will be wrong and the possible consequences of this wrong decision [26]. Additionally, partial EVPI can be calculated to focus the further research only on those parameters which have the most influence on the results [28]. By estimating the partial EVPI we can see which parameters contribute to the uncertainty the most [26].

Miller has described VOI analysis for drug development, but we can also apply his findings to other medical technologies [18]. He concluded that VOI analysis is relevant in early MTAs for drugs, since the major cost of drug development is spent on obtaining additional information about the drug.

Early HTA studies of medical devices and tests

Six publications describing early medical technology assessments were found in the literature. The six articles describe the assessment of eight technologies, four of which were diagnostic tests and four of which were other types of medical devices. Table 1 summarizes the aims and results of these studies. The studies focused on technologies in different medical specialties, the most frequent of which was oncology (n=3). In most cases, the primary aim of the study was to estimate the potential cost-effectiveness of the new technology. Interestingly, only one of the eight studies (tissue engineered urethral tissue) concluded that the technology was not likely to be cost-effective.

Table 1. Summary of early MTA studies

Table 2. Methodology of early MTA studies

Table 2 provides more details about the studies and also shows the methodologies that were used. All studies were conducted to yield information for use by manufacturers, although some mentioned other users as well such as policymakers and investors. Most studies used modelling techniques, sometimes along with other methods such as MCDA or CTS. This combination meant that a model served as the core of the study and that the other methods provided input data for that model. Regarding the application (or general purpose) of the study, most studies were performed as part of a pre-clinical preliminary market assessment or were performed to support a go/no go decision. For price determination only the headroom method was used, but we can see, that most of them were intended to support different decisions [14,15,24,29-31].

Differences between medical technologies

The aim of this study was to examine the methodology of early MTAs of medical devices and tests that could be used and have been used in the past. We can distinguish between three kinds of medical technologies: drugs, medical devices and diagnostic tests.

A possible definition of diagnostic tests is technologies which do not interfere in the treatment, but only provide information to the clinician about the patient and disease progression [32]. Their value can be measured by their sensitivity and specificity, but as Fineberg perfectly summarized: “The ultimate value of the diagnostic test is that difference in health outcome resulting from the test: In what ways, to what extent, with what frequency, in which patients is health outcome improved because of this test?”[34]. Most of their impact is indirect and the link between the performance of the test and health benefits of the patient is complex, although one should not forget that the testing of patients can also have its risks or side-effects [12,33]. For example, in the case of a diagnostic test used to establish a diagnosis, several parameters have to be considered, including disease prevalence (prior probability), diagnostic accuracy (sensitivity, specificity), any direct effects of testing, and the benefits and risks of subsequent treatment on the diseased and non-diseased groups (both correctly and incorrectly diagnosed patients). The direct effects of a medical test are the testing-induced emotional, cognitive and behavioural changes and the complications of a dangerous test [12].

In that sense, it could be argued that both early and classical HTAs of tests are harder to perform than other HTAs. Moreover, tests can be used in various ways, for a variety of disease and purposes. Many so-called diagnostic tests are not actually used for diagnosis per se, but for disease susceptibility testing, prognosis, selecting therapies, treatment response monitoring, monitoring for disease recurrence, etc. This diversity can make it hard to define the target condition of the test and the comparator in the economic evaluation [33].

The methodology of assessing the value of drugs is quite well defined. In stark contrast, it is not always clear how much evidence of effectiveness is needed in the case of medical devices and tests [36]. Double-blind randomized controlled trials are part of the development process of pharmaceuticals and the data obtained from those studies serve as an input for MTAs. In the case of diagnostic tests, and also some medical devices, it can be more difficult to design such a study, and MTAs of these technologies are not always supported by RCT data [32]. Some RCTs of tests may require larger sample sizes and well-defined protocols that link testing, results and treatment decisions, since we need to evaluate all the effects and future consequences [33].

Taken together, there are essentially no overall differences in the methodology of early MTA of different technologies. However, upon closer inspection, one could imagine that there are nevertheless some factors that could lead to differences in the ways to perform early MTA. For example, since there are differences in the requirements for approval and reimbursement, one could expect differences in the choice of methodology and the way in which a methodology is applied. In that way, rational goal-directed approaches can well lead to different choices.

Discussion

The aim of this study was to introduce the most commonly used methods in early MTAs of medical devices and tests. Various methods have been described in the literature for use in early MTA of drug and devices. We described six methods: early health economic modelling, headroom method, Bayesian analytical framework, clinical trial simulation, multi-criteria decision analysis and value of information analysis. The methods examined here can all help to make better decisions about whether and how to further develop medical technologies. They are not only relevant to drugs but also to medical devices and tests. Of these methods, one could argue that the methods are complementary since their purposes are not identical. For example, early health economic modelling can be viewed as an engine which can use the results from other methods (e.g., the analytic hierarchy process) to perform various calculations beyond just cost-effectiveness analyses. In fact, a model would be able to support clinical trial simulations or value of information analyses. Viewed in that way, it is not necessary to see the different methods as isolated options but rather as a set of tools that can be used together to perform early HTA. Each of the methods has its strengths and weaknesses. For example, the headroom method is a quick and easy model, which helps to make investment decisions without building a complex model. However it only works when explicit WTP thresholds are used by the payer.

A literature search only identified six publications describing early MTA studies of medical devices and tests. They described the assessments of eight technologies (four diagnostic tests and four other types of medical devices). Published studies have so far not utilised all of the available methodologies. While early MTA follows the same steps as classical MTA, repeated assessments and sensitivity analysis play a more significant role.

The limited number of studies can be explained by the fact that early MTAs are rarely published because they primarily support internal decisions by a company [13]. This means that a literature review will always have its limitations and that additional research will have to involve interviews with the different stakeholders to explore what methods they use in the early stages of technology development. Only then will it be possible to see what is done now and to explore what improvements can be made.

In the case of medical devices and diagnostic tests there are special features which may determine the methodology of early MTA, such as the learning curve phenomenon or their sometimes indirect impact on patient recovery. While more research on the differences between medical devices and tests would also be valuable, one could argue that the diversity amongst both devices and tests is so great that a comparison between the early MTA of devices versus that of tests is only a partial solution. Instead, it may be possible that the most appropriate early MTA approach might vary from technology to technology, amongst both devices and tests.

In conclusion, the concept of early MTA represents a new way to evaluate technologies that should receive more attention in the future. Early MTA can help to reduce the time and investments required in developing new technology but also help to develop more effective and cost-effectiveness medical technologies.