CER was popularized with the passing in 2009 of the American Reinvestment and Recovery Act (ARRA). Among other objectives, CER aims to compensate for limitations in the inferences that can be made from randomized controlled trials (RCTs) about an intervention's performance when applied in the messier real world.⁴ CER blends the established fields of outcomes research informed by classical epidemiology and health technology assessment that includes economic evaluation. CER is a concept that bundles these various evaluation approaches for population-level decision making, although techniques like health technology assessment and economic analyses have been practiced at the policy level for decades in other countries where fears about so-called rationalized health care are less part of the political landscape.^5,6

An important aspect of CER is exploration of the development of informatics infrastructures, networks for data linkage, and biobanking standards to achieve the evaluation requirements for personalized medicine. Ginsburg and Kuderer briefly sketch seven active CER projects in genomic medicine under the National Cancer Institute ARRA-funded program. They share the common theme of infrastructure development (technical and social) for information capture, linking, and sharing—the third strategy for QI listed. Such infrastructure development may be the first necessary step in foundation building for CER to facilitate the subsequent analytic components. So, it is not yet clear from these brief descriptions where the comparisons of alternative interventions will be made in the analysis phases of CER and whether such comparisons on the basis of large data sets will necessarily provide more valid inferences than those from well-executed RCTs.

But the articles by McShane and Hayes¹ and by Ginsburg and Kuderer² raise the concept of clinical utility as the underlying rationale for the relevance of CER, claiming that analytic and clinical validity, although necessary, are not sufficient for adopting technological innovations into the population. CER claims to address the clinical utility issue. The term clinical utility as usually applied is restricted to the clinical (physical, emotional, and social) consequences of interventions, both good and harmful, to the individuals exposed to them.⁷ Issues such as financial or out-of-pocket costs to individuals and convenience can be considered part of clinical utility; but Ginsburg and Kuderer include costs to society as part of the concept.

It is worthwhile distinguishing between clinical utility and health systems utility, with the former applying to decisions or judgments by health care providers and patients about consequences to individuals and the latter to policy advisors and decision makers considering consequences for the population and the system as a whole. What individuals demand for themselves and what is appropriate for society as a whole can come into conflict; and it is not clear at this time that the public in the United States is ready for setting priorities, interpreted as health care rationing. For the CER agenda to achieve its objectives in informing policies about the affordability of new interventions, whether well or imperfectly evaluated, it will be important to overcome such attitudes. A poignant illustration of differences in societal attitudes related to this issue is the contrast between how the US Food and Drug Administration and the United Kingdom National Institutes of Health and Clinical Excellence approached decisions about approval of bevacizumab for metastatic breast cancer, with economic evaluation missing from the former and an integral part of the latter evaluation.^8,9 As a concept, personalized medicine does create the expectation of health care for individuals, which could undermine policy informing aspirations of CER for setting priorities for the population as a whole.

Proponents of CER also claim that it has the potential to address limitations in the inferences that can be made at a population level from the results of RCTs that are conducted in more restricted settings. But we need to avoid assuming that the failure to reproduce the results of RCTs for interventions when applied to the real world is necessarily a comment on the potential benefits of the intervention itself; discordance between RCT results and those in the real world may have more to do with the circumstances under which the intervention is delivered. Exploration of these circumstances, rather than rejection of an intervention that might benefit subpopulations when appropriately applied, is a potentially important agenda for CER.

Because of the variety of factors that can affect the performance of an intervention in the real world, it is not unreasonable to question CER results obtained from selected populations as necessarily generalizable to other populations or regions where there may be differences in health services delivery mechanisms, access to services, insurance arrangements, age structures, and socioeconomic factors among others. We already know from the pioneering work of Wennberg and Gittelsohn¹⁰ that there can be considerable variation in patterns of care and outcomes across geographic regions.¹¹ We will need to be more explicit about how CER can most appropriately contribute, what inferences can be made from its comparisons, and what further explorations of the data might be needed before accepting them at face value.

A key contribution from these papers is reconciliation of the lingering doubts created by the political debates around ARRA in which a coalition of influential partners opposed the $1.1 billion investment in evaluation science on the grounds that it would stymie progress in personalized medicine.¹² Both articles confirm, through CER, the relevance of traditional evaluation approaches for personalized medicine interventions.

The two sets of authors set up a vision for improved conduct and reporting of biomarker validation studies for personalized medicine using tools (often presented as recommendations or statements) patterned after the evidence-based movement (eg, CONSORT; Assessment of Multiple Systematic Reviews [AMSTAR]; Grading of Recommendations Assessment, Development, and Evaluation [GRADE]; Appraisal of Guidelines for Research and Evaluation [AGREE]).^13–16 But, just as McShane and Hayes¹ and Ginsburg and Kuderer² highlight the distinction between real-world circumstances requiring CER approaches and the controlled environments of RCTs, their own recommendations for use of QI tools (eg, Biospecimen Reporting for Improved Study Quality, Reporting Recommendations for Tumor Marker Prognostic Studies [REMARK], Strengthening the Reporting of Observational Genetic Association Studies, and the Evaluation of Genomic Applications in Practice and Prevention [EGAPP] initiative) will themselves face real-world challenges for adoption.^17–20

Analogous to the gap identified between recommendations from practice guidelines and their use in the real world,²¹ the recommendations from these QI tools for genomic medicine will require active implementation. Lessons for promoting adoption of standards can be learned from the emerging fields of knowledge translation and implementation science^22–24 that show dissemination of recommendations through the literature to be a relatively ineffective, and perhaps the least effective, strategy to promote behavior change. In this context, it is surprising how timid is the language in the publications of tools for improved reporting in genomic medicine, often couching recommendations as voluntary, awareness-raising, and intended for readers as opposed to doers of research. For example, the publication of QI standards in genomic medicine by the EGAPP Working Group is accompanied by disclaimers in almost apologetic terms about any authority (other than moral) for influence or enforcement of its recommendations.²⁰ Although this may be appropriate given the positioning of the EGAPP Working Group, it does speak to the need for bolder approaches to stimulate adoption of these QI tools. On the other hand, it is encouraging that tools such as these have been successfully promoted and adopted in the evidence-based realm by influential journals and other organizations—from which lessons can be learned. It is encouraging that Journal of Clnical Oncology has adopted REMARK as a reporting standard. A recent proposal of a tool to facilitate adoption of the Biospecimen Reporting for Improved Study Quality recommendations by pathologists is a welcome action-oriented strategy for promoting their use.²⁵ In addition, the recent Institute of Medicine Report on Evolution of Translational Omics²⁶ and the availability of ARRA related-funding could stimulate accelerated uptake.

As CER and QI tools seek to address the real-world circumstances that affect quality in the conduct and reporting of genomic research, so too do these strategies face their own real-world challenges, some of which I highlight here.

The laudatory visions of McShane and Hayes¹ and Ginsburg and Kuderer² for improvements in the quality and reporting of predictive/prognostic biomarker research—so crucial to the future of personalized medicine—sets an important agenda, but more effort is required to address real-world challenges for implementation. The QI effort for genomic medicine is itself challenged by real-world barriers to adoption of standards. To accelerate the quality agenda in genomic medicine we need to add to CER the insights and methodologies of health services research and the emerging fields of implementation science and knowledge translation. To this I would add the imperative of including policy analysis research to broaden our perspectives about how societal trends, attitudes, and changing values reflected in the notion of personalized medicine can influence the acceptability to all stakeholders of more rigorous evaluation standards and their consequences, especially if more rigorous evaluation is perceived by the public and by clinicians as counter to the availability of innovation earlier in the development cycle of an intervention.