Real-World Outcomes of an Automated Physician Support System for Genome-Driven Oncology

PURPOSE Matching patients to investigational therapies requires new tools to support physician decision making. We designed and implemented Precision Insight Support Engine (PRECISE), an automated, just-in-time, clinical-grade informatics platform to identify and dynamically track patients on the basis of molecular and clinical criteria. Real-world use of this tool was analyzed to determine whether PRECISE facilitated enrollment to early-phase, genome-driven trials. MATERIALS AND METHODS We analyzed patients who were enrolled in genome-driven, early-phase trials using PRECISE at Memorial Sloan Kettering Cancer Center between April 2014 and January 2018. Primary end point was the proportion of enrolled patients who were successfully identified using PRECISE before enrollment. Secondary end points included time from sequencing and PRECISE identification to enrollment. Reasons for a failure to identify genomically matched patients were also explored. RESULTS Data were analyzed from 41 therapeutic trials led by 19 principal investigators. In total, 755 patients were accrued to these studies during the period that PRECISE was used. PRECISE successfully identified 327 patients (43%) before enrollment. Patients were diagnosed with 29 tumor types and harbored alterations in 43 oncogenes, most commonly ERBB2 (21.3%), PIK3CA (14.1%), and BRAF (8.7%). Median time from sequencing to enrollment was 163 days (interquartile range, 66 to 357 days), and from PRECISE identification to enrollment 87 days (interquartile range, 37 to 180 days). Common reasons for failing to identify patients before enrollment included accrual on the basis of molecular alterations that did not match pre-established PRECISE genomic eligibility (140 [33%] of 428) and external sequencing not available for parsing (127 [30%] of 428). CONCLUSION PRECISE identified 43% of all patients accrued to a diverse cohort of early-phase, genome-matched studies. Purpose-built informatics platforms represent a novel and potentially effective method for matching patients to molecularly selected studies.


INTRODUCTION
Genomic data are increasingly used to guide both routine and investigational treatment decisions for patients with cancer. These therapeutic advances have been driven by a convergence of factors. Primarily, selective and potent inhibitors of a wide range of critical signaling nodes are now available in the clinic. Simultaneously, clinically validated broad next-generation sequencing platforms that permit the detection of multiple potentially actionable alterations are now accessible at the point of care. 1,2 These improvements, however, have not been accompanied by commensurate advancements in the systems that are available to help physicians interpret and act on these data. Indeed, previous experience with patients with advanced solid tumor who undergo genome profiling suggests that only a minority-5% to 24%, depending on the breadth of sequencing and practice setting-are subsequently enrolled in genome-matched trials. [3][4][5][6][7][8] Numerous challenges exist in identifying and enrolling patients in appropriate genome-driven clinical trials. Clinicians must correctly interpret sequencing data not only at the time of the initial results, but also longitudinally, as our understanding of genomic biomarkers and associated clinical evidence continuously evolves. 9 Enrolling patients in trials also requires access to, and up-to-date knowledge of, a changing portfolio of studies, each with its own study-specific eligibility criteria and dynamic slot availability. Finally, all of this information must be readily accessible to the clinician at critical decision points in a patient's care.
To address these challenges, multiple strategies have emerged to facilitate matching patients to genomedriven clinical trials (Appendix Table A1). The most straightforward of these approaches involves directly annotating molecular sequencing reports at initial sign-out to indicate the clinical significance of each alteration and list potentially suitable clinical trials. Multiple initiatives are underway to harmonize the annotation of individual variants and incorporate multitiered levels of evidence for actionability. [10][11][12] Some commercial laboratories and academic institutions have created on-demand molecular tumor boards that review molecular sequencing reports to make treatment recommendations and manually curate existing clinical trials for alterations of interest. 13 All of these approaches have potential limitations. Annotated molecular sequencing reports generate potentially cumbersome, long lists of interpretations, typically with no prioritization or information on study availability. These annotations are also static and can rapidly become out of date as new biologic insights emerge or novel drugs become available. Similarly, molecular tumor boards are time consuming, difficult to scale, and potentially influenced by participants' knowledge base and anecdotal experience.
In response to these limitations and to address the ongoing unmet need of matching patients to clinical trials at our center, we created Precision Insight Support Engine (PRECISE), an automated, just-in-time, clinical-grade informatics platform to identify and dynamically track patients on the basis of molecular and clinical criteria. PRECISE was designed to empower clinical investigators to proactively identify and recruit optimal candidates to their genome-driven trials. Here, we analyze real-world use of this tool to determine how PRECISE facilitated enrollment in genome-driven, early-phase clinical trials at our center.

Genomic Testing and Patient Selection
All genomic data used in patient matching were generated in the Memorial Sloan Kettering (MSK) Clinical Laboratory Improvement Amendments-certified diagnostic molecular pathology laboratory using several laboratory-developed tests reflecting the evolution of profiling technologies used during the study period. Initially, limited profiling data were generated via a mass spectrometry-based hotspot assay covering eight genes (MassARRAY; Sequenom, San Diego, CA) or an amplicon-based next-generation sequencing test covering 48 genes (TruSeq Amplicon Cancer Panel; Illumina, San Diego, CA). 14 A customized hybrid capture-based next-generation sequencing assay (MSK-IMPACT) was later performed per previously published methods. 15 MSK-IMPACT can identify all classes of genomic alterations-single-nucleotide variants, indels, copy number alterations, and select structural rearrangements-in up to 468 genes, depending on the assay version. Data from all next-generation sequencing assays were captured in MPath, an internally developed application that uses the opensource database management system MySQL. These data are used for clinical report sign-out and delivery to the data warehouse that PRECISE uses for patient identification. 16,17 Sequenced patients had cancer types for which profiling was considered routine at the time performed or underwent clinical testing under a prospective institutional review board-approved protocol that was designed to evaluate the utility of profiling in these cancer types (ClinicalTrials.gov identifier: NCT01775072).

PRECISE Platform
The PRECISE platform was available to any principal investigator of an IRB-approved therapeutic study that included molecular eligibility criteria. In situations in which PRECISE might identify patients with whom study investigators had no therapeutic relationship, an IRB-approved waiver of authorization was also required. To generate the PRECISE cohort for each study, principal investigators worked with MSK information systems to craft inclusion and exclusion criteria on the basis of molecular characteristics and other structured data elements available through the MSK data warehouse using previously published methods, listed in the Appendix. 16 The functionality of the PRECISE platform evolved during the study period on the basis of user feedback (Fig 1). In total, PRECISE underwent three major functionality releases. Initially, in 2012, PRECISE could generate a list of all molecularly eligible patients who were alive at the time of the initial cohort creation. Investigators could opt to receive notifications of newly identified patients at a self-chosen frequency, typically between daily and weekly. Beginning in 2014, PRECISE was enhanced to offer notifications, customized per study, of potentially clinically important events that could prompt a treatment change, such as upcoming appointments or restaging scans. For most of the study duration, PRECISE notifications were provided directly to the research team, which empowered them to further screen potential patients for appropriateness and notify treating physicians accordingly. Beginning in June 2016, PRECISE was again enhanced to permit individualized, patient-specific, customized notifications to be sent directly to the primary treating oncologist. Notifications could be automatically triggered by multiple prespecified events, including sign-out of new sequencing data that identify a qualifying alteration or consecutively rising tumor markers. Use of this direct-to-primary oncologist notification function was at the discretion of the research team. All notifications to the research team and the primary oncologist were generated as emails and did not directly integrate with the electronic medical record.

Cohort Eligibility, Data Collection, and Statistical Analysis
All PRECISE cohorts for early-phase studies-pilot, phase I, and phase I and II-that included at least one genomic eligibility criterion and that were created after April 16, 2014, were included for analysis. The data lock was performed on January 31, 2018. Therapeutic protocol documents were manually reviewed to ensure that the studies met these criteria.
Data on PRECISE cohort criteria, use, and notification were parsed from usage logs. Patient enrollment data were obtained from the MSK clinical research database, which centrally maintains patient registrations to all therapeutic studies at MSK, including the date of enrollment. An internally developed MPath application was used to obtain the date that sequencing data was signed out and entered into the medical record for each patient, as well as the specific qualifying genomic alteration present.
The MSK institutional review board evaluated and approved a retrospective research protocol to evaluate PRECISE platform outcomes. Primary outcome measure was to determine what proportion of patients who were enrolled in the evaluated genome-driven studies was facilitated by PRECISE. For the purpose of this analysis, enrollment was considered to be facilitated if PRECISE identified the patient as eligible and generated a notification to either the research team or the primary oncologist before the enrollment date. Studies with no accrual during the evaluation period were excluded. Secondary outcome measures included determining the time from sequencing and PRECISE identification to patient enrollment in the relevant therapeutic study. Timing was expressed using descriptive statistics. To identify areas for future PRECISE functionality enhancement, reasons for the failure of PRECISE identify eligible patients before study enrollment were also explored through manual record review.

Study Characteristics
A total of 41 therapeutic trials used PRECISE for genomic matching during the study period (Table 1) evaluated small-molecule monotherapy, and an additional 9.8% (four of 41) included a small-molecule monotherapy in combination with other investigational agents. The first trial opened to accrual on July 13, 2012, and the last began accruing on October 11, 2017. Of the 41 trials, 14 used PRECISE from initial study opening and the rest of the trials began using the system after protocol activation. Eight of 41 trials used the direct-to-oncologist notification system.

Principal Investigator Characteristics
The 41 therapeutic trials were led by 19 unique principal investigators (PIs) who individually led between one and five studies (

Patient Matching
During the study period, a total of 755 patients who were treated primarily by 150 unique oncologists were enrolled in 41 trials. PRECISE prospectively identified 43% (327 of 755) of cases before patient enrollment, successfully notifying study investigators and/or the primary oncologist of the potential match (Fig 2). These patients had a wide range of tumor types and harbored multiple classes of genomic alterations that targeted a variety of genes (  . 15 2 (10.5) *The total adds up to more than 41 as studies may include more than one agent class.
†Twelve principal investigators had dual affiliations with a diseasespecific group and the phase I group.
Tao et al capture by PRECISE, such as presence of triple-negative breast cancer, which was not always readily documented in the medical record.
To further understand how physicians and patients use tumor genomic data to guide investigational treatment decisions, we evaluated the timing of enrollment relative to two important milestones: the completion of sequencing and the initial PRECISE identification. Upon evaluation of the 327 patients who were identified by PRECISE and successfully enrolled in a genome-driven study, there was significant variability in time intervals between these three events at the individual patient level. Median time from sequencing to therapeutic enrollment was 163 days (interquartile range, 66 to 357 days; range, 5 to 1,281 days) and from PRECISE identification to enrollment 87 days (interquartile range, 37 to 180 days; range, 1 to 850 days; Fig 3). Reasons for delay from sequencing and PRECISE identification to study enrollment included the availability of alternative routine therapy or a lack of need for treatment among patients without evidence of active disease.

Causes of Patient Attrition Before Enrollment
To better understand the reasons why patients who were identified by PRECISE did not subsequently enroll in the relevant therapeutic study, a representative cohort involving a multitumor phase I and II expansion study of a targeted small molecule was selected for manual record review. In total, PRECISE initially identified 98 patients on the basis of the genomic inclusion criteria and being listed as alive (Fig 4). Of these 98 patients, 22 were immediately determined to be permanently ineligible or excluded as a result of a static characteristic that rendered the patient ineligible for the trial indefinitely. These included having a second primary cancer (n = 6), a nonqualifying malignancy (n = 6), being deceased but not listed as such in the medical record Outcomes of a Physician Support System for Genome-Driven Oncology structured data elements that were readily available for use by PRECISE-nonsolid tumor and disallowed concurrent mutation-but that were not included in the initial PRECISE cohort criteria. Another 45% (10 of 22) involved a data element that was sometimes available-second primary cancer, prohibited prior therapy, and HIV/AIDS-for use but not under all circumstances. For example, exclusions on the basis of prior therapy will miss agents administered at other medical centers or some oral anticancer agents, especially if dispensed by third-party pharmacies.
‡Includes multiple alterations in the same gene. §Consists of one case each of loss of immunohistochemistry expression of BAP1 and unknown RET alteration.
Tao et al therapies, rapidly deteriorating, or deemed inappropriate because of other issues, such as prior nonadherence. In total, only 4% (four of 98) of patients who were initially identified by PRECISE successfully enrolled in this study.

DISCUSSION
Analyzing accrual patterns to early-phase, genome-driven studies using PRECISE during a 4-year period, we found that PRECISE helped facilitate nearly one half (43%) of all  May qualify in future (n = 70) 71.4% 6.1%

Reasons for nonenrollment
Oncologist chose other therapy Patient too ill Adherence concerns (n = 5) (n = 3) (n = 1) (n = 1)  Precision Insight Support Engine (PRECISE) cohort for a phase I and II study. Reasons for permanent or temporary exclusion from study qualification are depicted in blue (criteria that is available for use by PRECISE), red (criteria that is sometimes available and could potentially be captured as an extractible structured data element), or teal (criteria that is not yet available for use by PRECISE). (*) Listed as alive in electronic medical record but actually deceased.

Reasons for deferred qualification
Outcomes of a Physician Support System for Genome-Driven Oncology enrollment in these studies (Appendix Fig A1). To our knowledge, this report represents the first effort to evaluate real-world outcomes of an automated, just-in-time, clinicalgrade informatics platform to facilitate patient matching. We also discovered that a significant time interval often elapses between the initial generation of tumor genomic data and subsequent enrollment in a matched study. Specifically, among those who ultimately accrue to a matched study, median duration from data generation to enrollment was 5 months, with some outliers enrolling up to 42 months later. These data emphasize the importance of supporting physician decision making longitudinally through a patient's entire treatment.
Although these pilot data are encouraging, our analysis also identified areas of ongoing challenge for such matching systems as PRECISE. Through manual curation of one representative PRECISE cohort, we found that only 4% of patients who were initially identified as potentially eligible ultimately enrolled. Of importance, nearly three quarters of initially identified patients were not immediately eligible on the basis of clinical factors that were not readily available for use by PRECISE, most commonly related to challenges in algorithmically defining patient disease status. The result is that PRECISE had a high false-positive rate that may ultimately limit the utility of this system for some indications, particularly for recruiting patients with more prevalent genomic alterations. Indeed, a previous analysis of overall match rates at our institution that was conducted during roughly the same time period found a match rate of only 11%, despite 37% of patients harboring a potentially actionable alteration. 5 Taken together, these data demonstrate that even with the use of a sophisticated decision support system, there is an ongoing need for additional improvement in the methodologies to match patients to clinical trials on the basis of tumor genomic and clinical information. A major area necessary for improving matching efficiency by automated informatics platforms is better integration of additional clinical eligibility criteria, such as disease status and response to therapy. This requires developing agreed-upon standards, including discrete, structured criteria, for extracting clinical data from the electronic medical record.
In the future, leveraging natural language processing and information extraction technologies may also enhance the ability to accurately capture unstructured data.
This analysis has some important limitations. Foremost, we considered any patient about whom PRECISE successfully notified the PI or treating physician of potential eligibility before enrollment as an accrual that was potentially facilitated by the system. However, we cannot determine exactly how many of these enrollments might have occurred without the use of this system. Indeed, our study was retrospective in design and we cannot definitively make conclusions on the incremental value of PRECISE. Moreover, this analysis represents real-world use of PRECISE by each PI, who were responsible for setting his or her own cohort criteria and using the results as he or she felt best complimented the practice. Finally, PRECISE was not one system but rather an evolving platform during the study duration. Additional evaluation is needed to determine how the utility of this system is affected by recent features, such as full automation of direct-to-treating physician alerts triggered by critical events, like scans that show progression or rising tumor markers.
PRECISE is not the only system designed to address the emerging need of matching patients to relevant clinical trials on the basis of the patient's genomic profile. Several other centers have developed strategies by which to achieve this goal that range from on-site or virtual molecular tumor boards to automated matching platforms 13,18-23 (Appendix Table A1). Each of these systems offers physician decision support that aims to bridge the gap between genomic alteration detection and identification of the appropriate genome-driven therapy. Potential advantages of an automated informatics approach include scalability and the ability to track patients longitudinally and respond to changing molecular and clinical data.
In summary, this pilot study of real-world use of PRECISE to guide enrollment in genome-driven studies suggests that this type of real-time decision support system can meaningfully facilitate patient enrollment. As the use of genomic profiling in oncology care increases, new tools are necessary to maximize the utility of this information and bring precision oncology care to patients. This study reinforces the potential of automated bioinformatics platforms as an important means to increase clinical trial accrual and enhance the delivery of precision oncology care to patients.

PRECISE Technical Specifications and Workflow
PRECISE (Precision Insight Support Engine) leverages an existing IBM DB2 data warehouse and was designed using open-source software where possible. Notifications are generated from scheduled queries and use an open-source Java e-mail package implemented as a database user-defined function to send emails from the SQL statement. A Web application using open-source JavaScript libraries allows system administrators to capture cohort logic and settings and provides end users the ability to annotate patient status for their study. Additional details on the technical specifications of PRECISE, workflow support, and the notification system are described in previously published methods. 16 Outcomes of a Physician Support System for Genome-Driven Oncology  Outcomes of a Physician Support System for Genome-Driven Oncology