|By Jamie Chen, Johnny Chiu, and Junxiong Liu |
This paper was written to record our experience as one of the top 5 finalists of the the 2018 Humana-Mays Healthcare Analytics Case Competition.
Introduction
Hackathons and case competitions are great ways to test our knowledge by applying it to real-world problems, and this fall, our team (J-crew) participated in the 2018 Humana-Mays Healthcare Analytics Case competition that was open to all students pursuing Master’s degrees in the US. We were ecstatic when we were notified that our report was selected among the top five finalists from a pool of 246 teams from 42 major universities and that we were invited to the final presentation in Texas. We have extracted insights and designed modeling methods that closely aligned with business needs, built healthcare domain knowledge along the way, and learned from other finalists on creative problem-solving.
Case background
Heart disease remains the No.1 cause of deaths in the US. In 2016, heart disease cost America $555 billion, and this number is forecasted to reach $1.1 trillion by year 20351. The goal of the analysis was to help Humana better understand members’ likelihood of Acute Myocardial Infarction (AMI), also known as heart attack, through predictive modeling and establishing key indicators to inform Humana’s business decisions.
To come up with insights that are closely tailored to Humana’s business needs, we first spent time researching on Humana’s business and studied the company’s 2017 Annual Report. As a leading provider of Medicare Advantage plans, Humana received more than 90% of annual revenue from members’ premiums. More than half of Humana’s members enrolled in medical benefit plans and the remaining enrolled in specialty products. Given the high cost of AMI treatment, the ability to identify members with high risks of being diagnosed with AMI will help inform Humana’s pricing decision and allow Humana to provide preventative treatments in advance to target members, thus increasing premium revenue and reducing benefits expenses.
Source 1: American Heart Association Heart Disease and Stroke Statistics 2017 Report
![](https://sites.northwestern.edu/msia/files/2018/12/jcrew_fig1-2bmhv6s.png)
Dataset and Preprocessing
To perform this analysis, we were given a dataset with 100,000 records, with each record corresponding to a member that continued their enrollment in Humana’s Medicare Advantage plan in 2016 and with no indication of AMI. Each member had 448 features, which contained information about member characteristics, product characteristics, quality of medication and utilization of plan. The response variable was AMI_flag, indicating whether the member was diagnosed with AMI in the first quarter of 2017.
![](https://sites.northwestern.edu/msia/files/2018/12/jcrew_fig2-1vfev05.png)
Our general approach to this problem began with data preparation, followed by model fitting, insights, and predictions. Data preparation encompassed data cleansing, missing value imputation, feature inspection as well as additional feature creation. During feature inspection, we removed features that were highly sparse or had extremely low variance (< 0.5%), checked on multicollinearity, and recomputed missing values.
One key finding during exploratory data analysis allowed us to reduce the number of features from 448 to 226 while having features that yielded stronger signals to the response variable. Specifically, we noticed that variables for member disease history (visits related to different conditions and Place of Service features) included a feature for each of the four quarters in 2016 and some of these quarterly visits were 0. Therefore, we decided to sum up the four quarterly visits for the same visit condition or place of services to consolidate information and help with mode fitting. As shown in the figure below, annual visits to the emergency room showed a clear increasing trend for members with AMI after we aggregated the quarterly visits. This transformation proved to be a success during model fitting as it reduced model training time and improved model performance and interpretability.
![](https://sites.northwestern.edu/msia/files/2018/12/jcrew_fig3-215fm4m.png)
Modeling approach
Model fitting consisted of developing classification models which predicted a member’s likelihood of AMI diagnosis. We experimented a variety of models and performed model diagnostics and validation procedures to determine which models best fit our data and met business goals. One challenge in predicting AMI likelihood was the extremely unbalanced dataset. Of the 100k members, less than 3% of them were diagnosed with AMI as of quarter one in 2017 and these were the people that we especially want our model to learn from.
We designed a “Round-Robin” approach to learn from this minority class and help Humana identify existing members with a high likelihood of developing AMI in the future. A simplified version is demonstrated in the figure below: the majority (non-AMI) class can be divided into four folds, and at each time, we will combine AMI class with each of the non-AMI fold to train the model, and predict on the remaining three non-AMI folds. In the end, each member in each non-AMI fold receives three different predictions and the final prediction will be selected from a majority vote of the three predictions.
![](https://sites.northwestern.edu/msia/files/2018/12/jcrew_fig4-woarpz.png)
During actual implementation, we divided non-AMI members to 10 equal folds instead of four as shown in the demo to better address the class imbalance. In terms of the actual machine learning algorithm, we found that Logistic Regression was proved to be the best performing model with the highest AUC (0.603), in comparison to XGBoost and Random Forest. During the prediction phase, we labeled the top ~2500 members with the highest predicted AMI probability as AMI positive, and as a result, we were able to identify 2.6% of the current non-AMI members as high risks.
![](https://sites.northwestern.edu/msia/files/2018/12/jcrew_fig5-zrf1bc.png)
To further understand our predicted members with high AMI risks, we conducted descriptive analytics and compared among the three groups: non-AMI members, non-AMI members with high risks identified by the model, and members already diagnosed with AMI. As we see from the table of the key features identified by the model below, the predicted high AMI risk members consistently showed traits that were indicative of AMI risks such as highest in age and CMS medical risk score, the most number of antidiabetics prescriptions and emergency room visits, and 90% of them has been diagnosed with coronary artery disease before. These results further provided confidence in our Round-Robin method, which was robust and can generate predictions for every non-AMI member. In addition, It is also flexible as it can be customized based on data quality and business needs (i.e. changing from 4 folds to 10 folds, experimenting with different machine learning algorithms).
![](https://sites.northwestern.edu/msia/files/2018/12/jcrew_fig6-1h4za60.png)
Insights and application
For the identified non-AMI members with high AMI risks, we recommended Humana to take preventative actions and utilize their existing health and wellness services. For example, Humana can assign a case manager to a small group of these members and enroll them in specially designed care program to make sure they participate in additional screenings and activities to help lower their AMI risks.
Besides stratifying members with high risks, we provided key features shared among different prediction models along with their thresholds from exploring the probability distributions within the two classes. The figure below showed the 12 most important features shared among the different classification models we applied. For each of these features, we conducted additional analyses to identify a risk threshold. Take CMS Medical risk score for example, after plotting the distribution for AMI + and AMI – classes, we saw that the maximum separation of the two classes occurred after passing the score of 2.2, where a higher percentage of members with AMI can be identified compared to members without AMI. Therefore, during Humana’s initial screening of new customers’ product enrollment eligibility, Humana can collect information specifically related to these 12 factors and provide an initial product recommendation after comparing the collected data to their thresholds.
![](https://sites.northwestern.edu/msia/files/2018/12/jcrew_fig7-1wtyl8g.jpg)
Reflection on experience
This experience has helped us grow not only as data scientists but also taught us how to think as strategists and communicate to top executives in the healthcare insurance industry. During the onsite presentation, we met with other finalists and learned a lot from watching their presentations. One important lesson we learned was that while we dedicated most of our presentation on modeling and results, we lacked tangible recommendations on detailed advice of what Humana could do with the stratified high-risk people because of our limited domain knowledge. We could have told a fuller story by planning out the recommendations in different phases and provide a more detailed outlook. Training machine learning models is an iterative process where the models improve little by little after each iteration, we believe iterating through experiences like these also helps us develop a better understanding of how our skills could be applied and we looked forward to taking on the next challenge together!
![](https://sites.northwestern.edu/msia/files/2018/12/jcrew_fig8-qpap7f-632x666.jpeg)
(From left to right: Jamie Chen, Junxiong Liu, Johnny Chiu )
Special thanks to Humana and Texas A&M University – Mays Business School for hosting the annual competition that stimulated our interest in healthcare analytics! Thank you to the judges for evaluating our presentation and giving feedback.