Introduction

Clinical trial evidence accompanied with real-world evidence provides a comprehensive dataset for healthcare system stakeholders to make sound decisions about whether a particular drug or treatment should be approved, reimbursed, or reassessed, if already on the market. As a result Real World Evidence became one of the burning topics of today’s pharmaceuticals industry. Key drivers for this demand are the health technology assessors and payers, and their need for evidence-based health economic data, especially based on local conditions.

There are many individual examples of regulators and health technology assessment (HTA) agencies using real-world evidence for decision-making. However, we have identified just one analysis from an international perspective comprising several Western countries that focused on comprehensive appraisal on the usage of RWE in HTA processes [1]. Furthermore, we have not found out any recent studies on use of RWE in decision-making processes in Poland. Under these circumstances, there is a need to carry out a detailed analysis of the adoption of RWE in decision-making processes on public funding of drugs in Poland.

In Poland for almost a decade now a reimbursement decision making system operates, based on a multi-step process, one of the elements of which is a pharmacoeconomic analysis. Health Technology Assessment (HTA) guidelines in Poland require from the Applicant to submit data on clinical efficacy, effectiveness and safety in clinical and economics analysis [2]. In addition, according to the Reimbursement Act [3] risk sharing solutions based on health outcomes are acceptable, which results in possibility for RWE development in Poland.

There is a general perception that the role of RWE in public payer decision-making is becoming increasingly important. HTA guidelines [2] emphasize the value of RWE, however there are numerous organizational and legal barriers which impede locally generated data use in Poland. Moreover, in practice, RWE is not appreciated as it might seem from the HTA guidelines.

As pointed by Wierzbicka et al. 2015 [4] and Skrzekowska-Baran et al. 2015 [5] the main limitations of RWE development in Poland are: low awareness of RWE concept, unstable legislation, high costs, low computerization level, lack of cooperation standards and data integration. In opposition to limitations there are some opportunities such as: increasing need, reimbursement regulations that require separate reporting of efficacy and effectiveness data, international trends reflected in public debate and e-health [4,5].

Authors [4,5] conducted a questionnaire survey to assess use of RWE in clinical practice among medical practitioners. Questions included: the demand for RWE concerning treatment outcomes, interest in RWE, or the most reliable sources of RWE information in Poland. The survey results indicate that physicians are generally not satisfied with the level of access to real life outcomes. About 90% of physicians surveyed stated they lacked access to RWE, and only 10% of the respondents expressed the opposite view. In the opinion of physicians, pharmaceutical companies should broaden their knowledge in the field of new scientific reports regarding the use of RWE [4,5].

In addition, in accordance with the principles of evidence-based medicine (EBM) clinical decisions and ultimately reimbursement decisions should be based on the evidence of the highest reliability [6]. Studies considered as RWE are at the low level of the classical hierarchy of quality of clinical evidence. Those studies provide data with high external validity understood as the extent to which the results of the study can be applied to everyday clinical practice, or the extent to which one may expect similar treatment effects in a real life practice to those observed in studies [6,7]. RWE should be treated as a similarly important source of clinical data as the research of the highest internal validity, i.e. RCTs.

To better understand how recommendations and decisions on public funding of drugs are made in Poland the following general description of the decision making system is provided. Health technology assessment is carried out by the Agency for Health Technology Assessment and Tariff System (AOTMiT), based on the manufacturer submission that obligatorily includes HTA report. Analysts of AOTMiT prepare a Verification Analysis, on the basis of which the Transparency Council issues a Statement. The next step in the appraisal is a Recommendation of the President of AOTMiT issued on the basis of both VA and STC. Therefore AOTMiT generates three types of documents relating to a single decision-making process: Verification Analysis of AOTMiT created by analysts, Statement of the Transparency Council of AOTMiT and Recommendation of the President of AOTMiT. The latter is the most important document emerging on the basis of the previous two, as it is a legally named formal basis on which the Minister of Health takes a final reimbursement decision. 

Materials and methods

We conducted an analysis of available data from the AOTMiT [8] on the use and impact of RWE in decision-making processes. The analysis included decision-making processes on including into reimbursement and setting official ex-factory price in Poland. The Agency gets involved into such process by order of the Minister of Health according to Art. 35 paragraph 1 of the Reimbursement Act [3].The analyzed period was between January 2012 and September 2015.We were looking for data on effectiveness and safety in Verification Analyses (analytic input to further appraisal), Statements of the Transparency Council and Recommendations of the President of AOTMiT. 

All activities in the analysis were divided into four phases: screening, selection, categorization and descriptive statistics.

In the screening phase the primary eligibility criterion of a decision making process for further consideration was availability of at least one document of AOTMiT. Should there be no documents either issued or published by the Agency, there would be no sources to verify existence of RWE in a given decision making process.

We searched for keywords related to RWE using Acrobat Reader DC^® with advanced settings "case-sensitive" and "match any of the words". The search terms included:

PASS PAES RWE RWD post Post faz Faz ejestr ejestrze ejestrach ejestry ejestru ejestrów PSUR real Real zeczywist ost-auth ostauth ostmarket ost-market orejestrac obser prakty doświadcze Obser Prakty Doświadcze

During the selection phase decision making processes have been selected that contained RWE related to the effectiveness or safety.

The following were considered to be typical examples of clinical RWE: pragmatic clinical trials, observational studies, registries, named patient programs, early access programs, case/case series, Post-Authorization Effectiveness Study (PAES), Post-Authorization Safety Study (PASS), Periodic Safety Update Report (PSUR), and databases of public payers/insurers.

In the categorization phase we performed classification in the four steps:

(A) presence of RWE across 3-stage HTA appraisal,

(B) type of RWE,

(C) evaluation of RWE and

(D) impact of RWE.

Presence of RWE was categorized into 5 categories: Yes – from the Applicant; Yes – from the AOTMiT; Not known; No; No, but mentioned need for RWE. The identified types of RWE were related to effectiveness, safety, effectiveness and safety and not known. In regard to how RWE was evaluated four categories were applied: Sufficient, Insufficient, Lack of assessment and Not known. We based the assessment of impact of RWE on where the RWE was cited: in a justification chapter or outside of it. We also identified if the need for RWE was expressed in a justification chapter. This category was only applied to Statements of Transparency Council and Recommendations of the President of AOTMiT.

In the final phase of the study all the necessary statistical calculations have been conducted. The quantitative analysis included descriptive statistical parameters such as percentages of occurrences of values in each of the three data sets (Verification Analyses of AOTMiT, Statements of the Transparency Council of AOTMiT and Recommendations of the President of AOTMiT). In addition, an analysis of consistency of parameters for RWE within the decision-making processes was performed. Primarily it was based on quantitative compilation of the number of processes with a given set of elementary values for RWE parameters.

Results

We analyzed 914 documents published by AOTMiT including: Verification Analyses of AOTMiT (N=308), Statements of the Transparency Council of AOTMiT (N=302) and Recommendations of the President of AOTMiT (N=304), that were associated with 316 distinctive reimbursement decision making process.

For 8 processes the data were incomplete, i.e. one or two documents were not available compared to a complete triad. The least missing items were in regard to Verification Analyses, which may be due to the fact that we analyzed all the processes until the most recent ones which might have had only a VA developed yet without Statements of the Transparency Council and Recommendations of the President issued or published.

The analysis of individual documents have shown that RWE was identified in 53% (n/N – 162/308) of Verification Analyses, 21% (n/N – 63/302) of Statements of the Transparency Council and 35% (n/N – 106/304) of Recommendations of the President of AOTMiT (Fig. 1A).

Predominantly RWE was included in reimbursement applications by the applicants – Marketing Authorisation Holders, however in 4% of Verification Analyses analysts found RWE themselves. Out of all 316 analysed processes 162were defined as "RWE-positive"with majority of evidence related to practical safety (89%) and effectiveness (55%) (Fig. 1B).

Unfortunately, in up to ¾ of "RWE-positive" processes RWE was not evaluated. When RWE was assessed, AOTMiT analysts and the President of AOTMiT slightly more often judged it was insufficient than sufficient, while the Transparency Council of AOTMiT was nearly twice more restrictive (Fig.1C).

It should be noted that under the three-tier assessment, prepared documents have defined scope and format of the sections where the assessment of evidence is carried out. In order to estimate impact of RWE on appraisal, the place of quotation was analysed. Of the cases where RWE was cited, the Transparency Council put it twice more often into justification chapters of their Statements than the President of AOTMiT in his Recommendations. However, in majority of cases the reference to RWE was outside the justification chapter (Fig. 1D). 

Figure 1. RWE in the documents of AOTMiT: A) presence of RWE, B) type of RWE, C) evaluation of RWE, D) impact of RWE on the outcome of the decision-making process

In the next step, we analyzed consistency of decision-making processes.In 31% (N=96) of decision-making processes there were discrepancies in regard to RWE citation among AOTMiT analysts, the Transparency Council and the President of AOTMiT. Quite often (N=42; 14%) the information about RWE presented in Verification Analyses was ignored by both the Transparency Council and the President of AOTMiT. Moreover, when RWE was found by AOTMiT analysts, this evidence was omitted in later stages of appraisal in 6 cases (n/N – 6/96; 1.9%). Similarly, in one case (n/N – 1/96; 0.3%), the Transparency Council relied on RWE submitted by the applicant, which was not mentioned by the AOTMiT analysts or the President of AOTMiT. On the other hand, omitting results on the effectiveness occurred in 1.6% (n/N – 5/96) of the decision-making processes evaluated by the Transparency Council and 2.6% (n/N – 8/96) by the President of AOTMiT. In 5 cases (n/N – 5/96; 1.6%), both the President of AOTMiT and the Transparency Council ignored RWE, rated as adequate by AOTMiT analysts.

The impact assessment showed that 37% (n/N – 114/308) of the analyzed decision-making processes included a citation to RWE in the Statements of the Transparency Council and/or Recommendations of the President of AOTMiT. In the rest of the analyzed processes (n/N – 194/308) there were no RWE citation. Need for RWE was unanimously expressed by both the Transparency Council and the President in two cases. Such need was also expressed by the President of AOTMiT in 3 processes despite the Transparency Council had referred to RWE in their justification chapter.

In order to classify processes to specific disease areas, ATC classification was applied. Analysis of 307 (for 1 process there was not ATC code) reimbursement processes in Poland by ATC codes showed that the largest number of "RWE-positive"processes concerned antineoplastic and immunomodulatory drugs – ATC L (n/N – 55/104).The following biggest groups are drugs used in the treatment of infections – ATC J (n/N – 25/43) and diseases of the central nervous system(CNS) – ATC N (n/N – 24/35).Within the group of drugs in ATC L (N=104) category we observed that RWE was used in 30 of the 56 processes related to cytostatic drugs and in 17 of 31 to drugs used in hormone therapy. In the remaining 17 processes (N=104) RWE was included in 7 and 1 reimbursement process related to use of immunostimulatory and immunosuppressive drugs (Fig. 2).

Figure 2. The use of RWE in decision-making processes in Poland by ATC codes

In our study, we analyzed the use of RWE in relation to the type of public funding applied for, proposition of a risk sharing scheme and orphan drug status (Tab. 1). It is worth to highlight that the highest number of reimbursement processes with RWE was in case of drug programs and pharmacy reimbursement. We noticed similar percentage of reimbursement processes with RWE that included a proposition of RSS and without it (49% vs. 53% respectively).Much greater number of reimbursement processes with RWE applied to non-orphan drugs. However, it should be noted that over 70% of processes on orphan drugs contained RWE. Probably this is due to the fact that often the best reliable evidence for orphan drugs are just studies related to real life practice (Tab. 1).

Table 1. The use of RWE in relation to type of reimbursement mode applied for, proposition of a risk sharing schemes and orphan drug status

*Data not available in the analyzed documents published by AOTMiT

We also analyzed the character of the Statements of the Transparency Council and Recommendations of the President of AOTMiT in relation to prevalence of RWE (Tab. 2). More than half of positive Statements and Recommendations appear with RWE. Moreover the reimbursement Decisions of the Minister of Health also had a similar trend. The analysis of use of the RWE data in the reimbursement decisions over the years shows no clear trend, but on average RWE represented 50% (Fig. 3).The results of this study showed that there were more processes using RWE, both absolutely and relatively among processes with a positive reimbursement decision of the Minister of Health in comparison to processes with a presumably negative decision (86 vs. 66 and 54% vs. 50% respectively). Among the processes that are presumably in progress a much smaller share of processes took RWE into account (23%).

The final parameter for the analysis was the time of marketing authorization and evaluation by AOTMiT. It should be emphasized that AOTMiT judged relatively few drugs approved before 2000, so the number of processes using RWE for drugs from this period is also low. Most processes with RWE concern drugs authorized in the period 2006–2013, with the largest share of processes with RWE identified for drugs authorized in 2006 (15 of 18). In case of drugs registered after this year the number of processes that included RWE is rather constant (up to ca. 20), even though the number of all processes increases almost for each subsequent year of registration. This analysis also indirectly shows that there was an apparent considerable backlog of decisions, as a large number of decision making processes in the years 2012-2015 related to drugs authorized nearly ten years later (e.g. 2006 or 2007).

Figure 3. Use of RWE in relation to marketing authorization year

Table 2. Use of RWE in relation to the character of the Statement of the Transparency Council, Recommendation of the President of AOTMiT and the reimbursement decision of the Minister of Health

* “Presumably negative” and “presumably in progress” categories are related to the fact that only positive outcome of the reimbursement decision making process is publicly available through reimbursement lists. Negative decisions are only communicated to the applicant. Therefore the assumption had to be made that a process is presumably negative if after 180 days from submitting an application no positive decision was published in at least one reimbursement announcement. The process is deemed “presumably in progress” if the time from submission of an application is less than 180 days and no positive decision was published.

Longitudinal analysis of RWE in AOTMiT evaluations shows steady increase in number of processes with RWE in the years 2012–2014 with apparent decline in 2015 to estimated 36 processes per annum. Furthermore the share of processes with RWE had a peak in 2014. The share for 2015 including September data is 36%[1], which is the lowest in the analysed period (Fig. 4). The probable reasons for the low participation of RWE in 2015 is a greater share of new drugs in the evaluation processes by AOTMiT, for which there are no RWE (especially safety data from the period after the registration of the drug).

Figure 4. Use of RWE depending on the year of assessment of the drug by AOTMiT

Discussion

The aim of our study was to evaluate the use and impact of the RWE in decision-making processes on public funding of drugs in Poland.

To the best of our knowledge our study is the first analysis in Poland to focus on the use and impact of RWE in decision-making processes based on the documents from AOTMiT. Results of our study suggest that RWE has a growing importance and adoption in HTA.

It should be mentioned that we have found one study [1] related to the analysis of the decisions of HTA Agencies, particularly the National Institute for Health and Care Excellence (NICE), Scottish Medicines Consortium (SMC),Canadian Agency for Drugs and Technologies (CADTH), Common Drug Review (CDR) and pan-Canadian Oncology Drug Review (pCODR), Pharmaceutical Benefits Advisory Committee (PBAC), French National Authority for Health (fr. Haute Autorité de Santé; HAS), Federal Joint Committee (ger. Gemeinsame Bundesausschuss; G-BA).Authors examined a total of 1,840 HTA decisions of which only 106 included RWE (6%), understood as observational studies, whereas 1,734 HTA decisions did not include observational data. Of all the decision making processes which included RWE a positive decision was in 40% (42), a recommendation with restrictions in 38% (40) and a negative recommendation in 22% of processes (24). The authors concluded that RWE played some part in public funding decision making processes. It is worth noting that analyzed HTA decisions had a very low share of observational studies [1].

Also we found a systematic review conducted in Medline by Szkultecka-Dębeket al. 2015 [9] which aim was to identify existing guidelines for RWE collection and analysis. Authors [9] have identified only two publications that were dedi­cated to RWE and provided some guidelines to researchers. In conclusion they suggested that actually published guidelines are focused on a specific type of RWE studies. Furthermore authors have acknowledged a potential for use of RWE in decision-making process but they underline the need of specific guidelines in relation to RWE methodology research [9].

Our study includes both limitations and strengths. The primary limitation of the study is a problem of limiting analytical work only to extract the data already presented in the documentation prepared by AOTMiT (Verification Analyses, Statements of the Transparency Council, Recommendations of the President of AOTMiT) without detailed verification of studies classification of evidence as experimental or RWE (passive attitude). Other important limitation is whether categorizing of evidence as RWE made by AOTMiT is correct. The analysis was conducted based on the classification of RWE made by AOTMiT analysts and contained in the documentation. For example there were cases where extended phases of RCTs were considered RWE.

The strengths of this study include strict methodology concerning predefined assessment criteria. Data extraction and calculations were conducted independently by two authors. Other important strengths of this study concern a comprehensive extensive set of all available documents analyzed for a period of 2012-2015. The analysis was also carried out extensively for different aspects (features of the process and the use of RWE). The analyses included both comparative approach between documents within a single decision-making and among the overall documents.

Decision-making processes in Poland require that Applicants present evidence with the highest available reliability. According to requirements under the HTA guidelines [2] and the Directive of the Ministry of Health [10] the major important evidence is constituted by data from experimental studies (mostly RCTs). Our analysis shows that RWE is apparently becoming meaningful in Poland. However, it can be assumed that the potential of RWE has not yet been fully developed. There is an unmet need to further even more detailed analyzing of the use and impact of RWE in decision-making processes in Poland and also to communicate it to decision makers at all stages of public funding for drugs.

Conclusions

General results of the study suggest broad adoption of RWE in pharmacoeconomics reports in Poland. RWE related to effectiveness or safety was found in surprising 53% of decision-making processes on public funding of drugs in Poland. Moreover, applications with RWE more often had a positive Statement/Recommendation/Decision compared to those without RWE. The results of this study showed that there were more processes using RWE, both absolutely and relatively among processes with a positive reimbursement decision of the Minister of Health in comparison to processes with a presumably negative decision (86 vs. 66 and 54% vs. 50% respectively). During a 3-step evaluation process there was the need for RWE reported or AOTMiT added RWE to the application evidence "ex officio". In several cases the need for RWE was unanimously expressed by both the Transparency Council and the President of AOTMiT or supplemented by verifying bodies which confirm usefulness and practical implementation of such data in decision-making process. Detailed analysis showed that usage of RWE is important and has positive impact on reimbursement decisions in many cases. However, there was lack of consistency in the identification of RWE on the three distinct levels of assessment and appraisal within the HTA Agency. Yet still the role of practical evidence is smaller than of clinical evidence which reduces the attractiveness of collecting RWE for the applicant. Although not perfect, the evaluation process seems to be opened legally to RWE and obligatory guidelines are more encouraging than discouraging applicants to invest in RWE in Poland. A growing importance of RWE can be observed in Poland, however sustainability of this growth remains uncertain and requires constantly increasing effort from all stakeholders.

Acknowledgments

None

Funding

The funding for this work has been provided by Janssen-Cilag Polska.