The Hilltop Institute’s “Predicting Avoidable Hospital Events” model—otherwise known as the Hilltop Pre-AH Model™—originally began with a lunch. In mid-2018, Ian Stockwell, PhD—head of the analytics & research team here at Hilltop—and Howard Haft, MD—executive director of the Maryland Primary Care Program (MDPCP)—met over lunch to discuss the newly established MDPCP. This program is a key element of Maryland’s Total Cost of Care Model and is intended to help primary care providers practice advanced primary care. One of the central ideas of advanced primary care is that primary care providers can (and should) be pro-active, transformational players in a patient’s health care ecosystem, helping patients with disease prevention, chronic disease management, and even the avoidance of unnecessary hospitalizations. Dr. Stockwell, having already developed a predictive model for the Maryland Department of Health on nursing facility admissions, told Dr. Haft that it sounded like a predictive algorithm could be useful in triaging the large patient panels assigned to each primary care practice. Hilltop could develop a model to predict which patients are likely to incur an avoidable hospitalization in the near future, and providers could incorporate these risk predictions into their care processes in order to proactively care for high-risk individuals. Dr. Haft concurred.
The model-building process took about eight months. The first step, which took about three months, was a comprehensive literature review. We sought to identify, based on previously published research, which risk factors appeared to be predictive of avoidable hospital events. At Hilltop, we view this as a crucial element of predictive model development. Predictive models can, at one extreme, be “black boxes.” It is entirely possible to skip the literature review and feed raw claims into a complex neural network and allow the algorithm to find relationships between (for example) claim-level diagnosis codes and the outcome of choice (in this case, the occurrence of an avoidable hospital event). However, a black-box method of model development makes it difficult to meaningfully determine why certain patients have high (or low) risk scores. We knew that, for this model, we needed maximum transparency. We needed to make a model that identified high-risk patients in a way that allowed care providers to know why a certain patient was high-risk.
The literature review generated a list of about 200 risk factors that have been shown to be predictive of avoidable hospital events. We knew it was important for these risk factors to be operationalizable in administrative claims. The model was set to use Medicare fee-for-service claims, and even though a certain claim (an individual’s current blood glucose level, for example) might be predictive of avoidable hospitalization, this information is not available in administrative claims, so we couldn’t use this risk factor. We also knew that we would also have patient ZIP codes, so we took the time to create a library of about 40 social determinants of health, with the intention of capturing elements of an individual’s environment that may determine their risk of avoidable hospitalizations. We used various publicly available data sources to create these risk factors at the ZIP code level.
The next step was coding. This was the “needle-in-a-haystack” phase of model development. We were swimming in data—tens of millions of claims for hundreds of thousands of patients in practices participating in the MDPCP—but we needed to turn the data into the meaningful risk factors we had identified in the previous step. We used a combination of existing code, further research, and internal Hilltop expertise to turn the risk factors from words on a page to lines of code in a script. We also developed a dual-version coding system (one version in SAS, one version in Stata) to debug and cross-check. This took about three and a half months.
Next, we were ready to develop the model. We had the data and the risk factors and had to decide which model to use to best identify individuals at a high risk of incurring avoidable hospital events. There are many different models that can accomplish this, each with its particular strengths and weaknesses. We knew that we wanted something transparent, stable, and flexible. We were designing a model that could be in production for years, so we wanted to be able to add risk factors. We settled on a discrete time survival model that is operationalized as a person-month multivariable logistic regression. Once we had decided on this modeling strategy, it took a couple of weeks to generate the first internal estimates, and another month to fine-tune the modeling process.
Finally, in September 2019, we moved the model into production. Using three years of data, we trained the model so it would estimate the relationships between our risk factors and the occurrence of future avoidable hospital events. Then, we applied these relationships to the most recent data in order to generate risk scores for each individual in the MDPCP cohort. The first risk scores were released through Maryland’s health information exchange, Chesapeake Regional Information System for our Patients (CRISP), on October 11, 2019. We have been updating, re-training, re-scoring, and releasing new risk scores monthly since then. We monitor model performance each month, and we’re proud to report that the model works well: the top 10% riskiest patients experience about half of all avoidable hospital events in a given month. This means we’re helping primary care providers identify the patients that truly are at risk of future avoidable hospitalizations.
Of course, we didn’t just build the model. We also developed extensive documentation in order to be as transparent as possible with stakeholders and end-users; met many times with our partners at the MDPCP project management office (PMO), CRISP, hMetrix, and other organizations to incorporate their feedback; conducted extensive internal testing in order to make sure that our estimates are as accurate as possible; delivered many trainings and presentations on the model; developed automated processes to conduct data quality checks; and more. Since the model has been in production, we’ve added a feature to let providers know the particular reasons for an individual’s risk, we’ve added risk factors, and we’ve adapted our internal procedures to ensure model quality during the COVID-19 pandemic. Going forward, we plan to use more detailed patient address data in order to better model the social determinants of health. We’re grateful for the support from our partners at the PMO and within Hilltop, and we look forward to helping providers practice precision primary care with this and other models in the coming years.
Morgan Henderson, PhD
Senior Data Scientist
Analytics & Research Team
As a public university, UMBC is dedicated to integrating teaching, research, and service to benefit the citizens of Maryland. The Hilltop Institute—through our longstanding partnership with the Maryland Medicaid program and our mission to advance the health and wellbeing of people and communities through research and analysis—exemplifies UMBC’s commitment to public service.
Hilltop’s partnership with the Maryland Medicaid program is built on a foundation of mutual trust, a spirit of collaboration, and a shared commitment to increasing access to affordable and high-quality health care for all Marylanders. Since 1994, Hilltop has performed data-driven research and policy analysis for the Medicaid program to inform policymaking and support program development, operations, and evaluation. Today we are actively engaged in furthering health care financing and delivery system reform in Maryland and supporting implementation of the state’s Total Cost of Care Model.
A number of other partnerships between state Medicaid programs and public universities exist across the country. While similar to Hilltop, each is unique in its own way. What these partnerships have in common is a shared vision for a more equitable and robust publicly funded health care system. Hilltop has been privileged to work with state agency and university representatives from partnerships in 24 other states through our work as a founding member of AcademyHealth’s State-University Partnership Learning Network (SUPLN). Our annual meetings, collaboration calls, and frequent communications have engendered collegial relationships that have resulted in timely joint research investigations aimed at addressing salient state health policy issues such as the opioid crisis and the response to the COVID-19 pandemic. SUPLN’s Medicaid Outcomes Distributed Research Network (MODRN) is enabling us to produce powerful cross-state Medicaid data analyses that were not previously possible.
In working with other SUPLN colleagues, we have come to better understand the attributes that distinguish a successful state-university partnership and the benefits that can accrue to both the state and the university. To capture these learnings, we have prepared briefs that articulate the value proposition for both the state agency and the public university. In partnership with the Millbank Memorial Fund and AcademyHealth, we are pleased to share the value propositions. Our hope is that these briefs will encourage additional states to pursue formation of partnerships. SUPLN stands by, ready to assist. The briefs can also provide guidance to existing partnerships on strengthening their roles and relationships and solidifying support within their states. Partnerships such as these have never been more important as we as a nation endeavor to ensure access to affordable health care and ensure all Americans good health and prosperity.
Cynthia H. Woodcock
The Hilltop Institute at UMBC
Chair, Steering Committee
AcademyHealth State-University Partnership Learning Network (SUPLN)