first predictive model in healthcare looked like a house run.
It answered the business query. The performance metrics were strong. The logic was clean.
It also would have failed spectacularly in production.
That lesson modified how I take into consideration data science and what it takes to achieve success in healthcare within the age of AI.
Looking back, that failure would repeat itself throughout my profession, nevertheless it was crucial to my growth and success as a knowledge scientist: a fancy model in a notebook is value nothing when you don’t understand the environment your model is supposed for.
Data Analyst
After three grueling months on the hunt for my first job in the true world, in a market with a fresh appetite for data but that was also teeming with talent, I used to be finally given my first big break. I landed an entry-level data analyst position on the Business Intelligence team at a big hospital system. There was a lot to learn. An enormous hurdle, and one which many individuals wanting to get into the healthcare data realm may also must jump, was familiarizing myself with the ins and outs of Epic, the biggest EHR (electronic health record) vendor by market share. Stretching my legs in SQL with the extremely complex data in an EHR was no easy feat. For the primary few months, I used to be leaning on my senior coworkers to write down the SQL I would want for evaluation. This frustrated me; how could I even have just finished a master’s degree in statistics and still be struggling to select up the SQL mindset?
Well, with practice (plenty of practice) and patience from my coworkers (plenty of patience) it will definitely all began to make sense in my head. As my comfort grew, I dove into the world of Tableau and dashboarding. I grew fascinated with the means of making aesthetically pleasing dashboards that told data stories that desperately needed telling.
Throughout my first 12 months, my manager was extremely supportive, checking in recurrently and asking what my profession goals were and the way she could help me achieve them. She knew my background in class was more technical than the ad-hoc analyses I used to be doing as an entry level data analyst, and that I wanted to construct predictive models. In a bittersweet end to my first chapter, she offered to transfer me to a different team to get me this experience. That team was the Advanced Analytics team. And I used to be going to be a Data Scientist.
Data Scientist I
From day one, I worked closely with a knowledge science guru who had a deep knowledge of healthcare and the technical capabilities to match, giving him the flexibility to deliver amazing products and pave the way in which for our small team. He was the primary in our system to develop a custom predictive model and get it live within the production environment, producing scores on patients in real-time. These scores were getting used in clinical workflows. When my manager asked me what my skilled goals were for the upcoming 12 months, I had an instantaneous and certain response: I desired to get a custom predictive model into production.
I started with a number of POCs (Proofs of Concept). My first model was a linear logistic regression model that attempted to predict the likelihood of complications from diabetes. While a superb first attempt, my data sampling approach was all flawed, and in peer review, my colleague pointed it out. One in all the important thing lessons I learned from my first attempt at a predictive model in healthcare was
An example of this: You can not simply gather each patient’s current lab values and use those as features in your model. In case you expect the model to make predictions, say quarter-hour after arrival within the ED, you could account for that. Thus, when gathering two years of historical data to coach a model, you could gather each patient’s lab values as they existed quarter-hour after arrival, i.e. on the time of their simulated prediction date and time, not what those lab values are today/currently. Failing to accomplish that creates a model which will perform higher in POC than it does in real-time production environments, because you’re giving the model access to data it might not have available to it on the time of prediction, an idea often known as .
Lesson learned, I used to be able to try again. I spent the following few weeks developing a model to predict appointment no-shows. I used to be very intentional on how I gathered data, I used a more robust and powerful algorithm, XGBoost, and once more got to the peer review stage. The model’s AUC (Area Under the Receiver Operating Characteristic curve) was astounding, sitting within the low 0.9s and blowing everybody’s expectations for a no-show model out of the water. I felt unstoppable. Then, all of it got here crumbling down. During a deep dive into the surprisingly strong performance, I noticed crucial feature was the scheduled appointment time. Take that feature out, and AUC dropped into the mid-0.5s, meaning the model predictions were virtually no higher than random guessing. To analyze this strange behavior, I jumped into SQL. There it was. Throughout the database, every patient who didn’t show as much as their appointment also had a scheduled appointment time of midnight. Some data process retrospectively modified the appointment time of all patients who never accomplished their appointment. This gave the model a near-perfect feature for predicting no-shows. Each time a patient had an appointment at midnight, the model knew that patient was a no-show. If this model made it to production, it might be making predictions weeks before upcoming appointments, and it might not have this magic feature to drag up its performance. Data leakage, my arch nemesis, was back to haunt me. We tried for weeks to salvage the performance using creative feature engineering, a bigger data set for training, more intensive training processes, nothing helped. This model wasn’t going to make it, and I used to be heartbroken.
I ultimately hit my stride. My first big predictive model success also had an amusing name: the DIVA model. DIVA stands for Difficult Intravenous Access. The model was designed to notify nurses when they could have difficulty placing IVs on certain patients and may contact the IV team for placement as an alternative. The goal was to scale back failed IV attempts, hopefully raising patient satisfaction and reducing complications that would arise from such failures. The model performed well, but not suspiciously well. It passed peer review, and I developed the script to deploy it into production, a process much harder than I could’ve imagined. The IV Team loved their latest tool, and the model was getting used inside clinical workflows across the organization. I achieved my goal of getting a model into production and was thrilled.
Data Scientist II
Following the successful implementation of a number of other models, I used to be promoted to Data Scientist II. I continued to develop predictive models, but in addition carved out time to learn concerning the ever-growing world of AI. Soon, demand for AI solutions increased. Our first official AI project was an internal department challenge where we’d employ language models to summarize financial releases of publicly traded corporations in an automatic fashion. This project, like most other AI-related projects, was quite different than the standard ML model development I used to be used to, but the variability was welcomed. I had a lot fun diving into the world of ETL processes, effective prompting, and automation. While we are only getting our feet wet with AI initiatives, I’m excited for the brand new kinds of business problems we will now create solutions for.
My role as a knowledge scientist has evolved as AI systems have improved. Developing DS/ML and AI solutions requires much less technical work effort now, and I almost consider myself as part data scientist, part AI project manager throughout the process. The AI systems now we have access to now can write code, bug test, and make edits very effectively with tactical prompting on our end. That said, there may be a growing concern concerning the impact and feasibility of AI initiatives, with various reports suggesting that almost all AI projects fail before seeing production. I imagine
Our understanding of predictive models fundamentals coupled with domain knowledge from inside our industries (healthcare, in my case), continues to be very much needed to create solutions which are effective and may provide value. Gone are the times once we could rely solely upon our technical acumen to offer value. Coding is now handled by LLMs. Automation is way more accessible with cloud providers. An authority that may translate the needs of the business right into a strategic plan that guides AI to an efficient solution is what is required now. The trendy data scientist is the right candidate to be that translator.
Wrapping Up
Data science, as with every profession path in tech, is all the time changing and evolving. As you’ll be able to see above, my role has modified a lot within the years since college. I even have climbed a number of rungs of the company ladder, going from an entry-level data analyst to a Data Scientist II, and I can say with confidence that the talents required to achieve success have shifted because the years have passed by and technological advances have been made, but it’s important to recollect the teachings learned along the way in which.
My models failed.
Those failures shaped my profession.
In healthcare, especially with AI magic at our fingertips, a successful data scientist isn’t the one who can construct probably the most complex models.
A successful data scientist is one who understands the environment the model is supposed for.
