Introduction

Randomized clinical trials (RCTs) are the gold standard in research and there are regulations in place about how to perform such studies, what the best methodology is, what the standards are to ensure good quality of delivered evidence and how to report the results. However RCTs cannot answer all scientific questions and real world data are those which can provide additional information in relation to medical treatments, especially taking into account the potential impact on decision making process at the time to provide access to new treatments. But yet there is a need to use the right methodology for data collection in order to ensure good data quality. This seems not to be that well defined when we are looking at the real world data studies.

This is the reason why as a follow up to our first paper related to real world evidence need we decided to search for existing guidelines or recommendations addressed to projects and studies where real world data is collected [1].

Objective: Our aim was to investigate what is recommended to perform good quality and reliable real world data studies. Are there any existing guidelines specific for RWD? What is the recommended methodology to collect and analyze RWD?  

Methods: The performed search was based on the Internet. The Medline-PubMed databases have been reviewed and the search strategy was based on terms: “real world data” [All Fields] AND "guidelines"[All Fields] OR “guidance” [All Fields] OR “recommendation” [All Fields]. The initial search was performed in June 2015, however due to obtained findings not specific to guidelines related to real world we decided to perform an additional search in July 2015. The additional second search was focused on “real world data” [Title]. All the obtained abstracts have been analyzed for the accuracy and the selected ones have been searched for the full publications.

Results: Based on the performed search we have obtained in total 97661 records. We have restricted the search to “humans” 69517 and then for publications within last 5 years  which allowed to obtain 23597 records (diagram 1). When analyzing the obtained records we observed that they were not fulfilling our criteria, as most of the publications were related to clinical guidelines for treatment of different conditions, not methodological guidelines about how real world data studies should be performed. With the 2^nd search, we decided to restrict the search only to titles with the words we were interested in and we obtained 67 records. We have restricted the search to “humans” which allowed to obtain 40 publications. Reviewing all obtained records from the performed search we analyzed in detail the abstracts and after analysis of the full texts we found 2 publications fulfilling our criteria (diagram 2). The final selection was done independently by 2 authors before the final inclusion decision was taken. Since we were not satisfied with the findings we continued searching additionally on the internet site of scientific organizations (ISPOR) and HTA organizations (NICE) where we found two additional publications which we decided to include in our discussion.

Diagram 1. First search strategies in the library database – PubMed

Diagram 2 – Second search strategies in the library database – PubMed

The RWD topic is in the area of interest of the ISPOR organization and their Task Force published the results in 2007 of the group work defining what we understand by real world data and also pointed out to the limitations in relation to RWD. The Task Force identified the need for good practices for collecting and reporting RWD and also the need for good process in using real world data in coverage and reimbursement decisions as well as the need to consider the costs and benefits of the data collection [3]. In the PubMed database search we found that already in 1999 Joel Hay raised the question about how we should evaluate RWD. The author saw the potential significant impact of real world data on decision making process as such information obtained from retrospective or observational trials can answer decision-makers questions. However there are limitations when collecting and analyzing data from real world. Hay provides examples of some retrospective analyses and discuss potential confounding factors and different factors influencing the results [2].

In his paper Willke recommends to learn from ISPOR publications about RWD however for those who start their interest in RWD he presents the “Ten commandments” for conducting and reporting CER based on analysis of RWD. In his paper the author provides an overview of what should be taken into account at the time of planning, analyzing and reporting results from real world research [4].

A recent publication by Roche, Reddel, Martin et al. focuses on the quality standards for real world research and helps to understand methodological issues related to comparative and observational studies when using clinical and administrative datasets. The authors provide the researchers and reviewers with a tool to be used both for conducting and reviewing RWD studies. The proposed checklist  includes the key issues to be taken into account in relation to RWD projects. The authors divide the process whose aim is to ensure good quality of the data into preparation phase, analysis and reporting of results and the discussion of results [5].

Discussion

With the growing interest in real world data as supportive element to the findings from randomized clinical trials we found relatively few published guidelines or recommendations on how to collect and analyze real world data. The reason could be that there are some guidelines focused on a specific type of RWD. There may exist guidelines for epidemiological studies, on how to report the results or guidelines for observational studies. 

In 2007 there was an initiative called The Strengthening the Reporting of Observational Studies in Epidemiology (STROBE). Within this initiative recommendations on what should be included in an accurate and complete report of an observational study were developed. The STROBE recommendations covered three main study designs: cohort, case-control and cross-sectional studies. There are also checklists for the different types of observational studies available [6].

In Brazil in view of the Brazilian Network for Health Technology Assessment recommendations to use observational studies to develop economic evaluations a search was done to analyze to what extent RWD are used for HTA. The authors of the paper: “Real World data for Health and Technology assessment in Brazil: an unmet need“ aimed to identify the requirements and needs for epidemiological data regarding HTA submissions in Brazil. After reviewing different sources such as HTA requirements, reports and dossiers, as well as local guidelines and regulations about principles for real world data requirements for HTA they found that in 11.8% of the submissions there were no real world data used and also lack of epidemiologic data was a common issue. The Authors concluded  that for HTA in Brazil the use of real world data is an important need [7].

In relation to HTA analysis in UK NICE issued recently guidance for observational trials providing guidance which covers both aspects: for submission and review of observational data as part of the appraisal process. NICE being aware of the limitations in relation to observational data like bias in comparison to RCTs, patients selection, follow up and no pre-specified end-points  prepared recommendations how  the quality and transparency of assessments using observational data can be improved. The authors present an algorithm with the aim to support the best methods selection for the analysis [8].

Conclusion

Based on the performed search we can observe the growing interest in the real world data, however the published guidelines are usually focused on specific type of RWD studies and there are no many guidelines available in relation to real world data methodology research  in general.

In view of the potential use of RWD in decision making specific guidelines on how to conduct RWD research are needed.