A glance right into a scientific review paper that asked that query and located answers
Artificial intelligence (AI) is becoming increasingly prevalent in our day by day lives. From advice systems in almost every webshop, to automatic translation of foreign languages on web sites you visit. For some industries this transition appears to be going more easily than for others though. The medical field appears to be especially difficult to enter, but why? There’s a lot academic activity dedicated to AI within the medical space, so what’s keeping these technological breakthroughs from having tangible impact in healthcare? Sendak et al. tried to seek out a solution to that query of their review paper “A path for translation of Machine Learning Products into healthcare delivery” (2020). Their findings really resonated with what I do know from experience at UbiOps in working with MedTech startups, so in this text I’ll walk you thru their paper.
Before we dig into the paper itself, let’s have a fast have a look at the present state of machine learning within the medical field. Enthusiasm and excitement about the chances of machine learning within the medical field have been immense, resulting in an astonishing amount of literature on the subject. Every other week you’ll be able to read a few latest study that has been done on using ML for cancer detection, and using ML for drug discovery also appears to be a hot topic. A large number of conferences, organizations and academic journals have been set as much as disseminate knowledge surrounding the very topic of ML in healthcare.
Though the research is proliferating fast, evidence of tangible clinical impact stays scant. You would possibly think “Oh but it surely at all times takes some time before latest technologies develop into mature enough to be applied in practice”, but at the identical time we are able to see that latest findings on using ML for higher user retention are quickly adopted by the likes of TikTok, Instagram and LinkedIn. Panch et al eloquently describes this ‘inconvenient truth’ of machine learning in healthcare as
“at present the algorithms that feature prominently in research literature are actually not, for essentially the most part, executable on the front lines of clinical practice.”
Luckily there are some healthcare corporations which have successfully managed to integrate AI/ML into their products. Corporations like Ellogon who help doctors in choosing the fitting patient for cancer immunotherapy show that it is feasible to make the transition from proof of concept to a mature product that may easily be fully integrated into existing medical protocols.
Let’s have a have a look at the research from Mark Sendak et al. to seek out out.
Mark Sendak and his colleagues got down to perform a narrative review that might help understand find out how to translate machine learning into healthcare. They combined their very own first hand experiences in constructing machine learning products with 21 case studies of machine learning models that successfully made their way into clinical care. And this is strictly what I find so interesting about their research, the undeniable fact that they tried to learn from people who actually made the step into production.
Based on their evaluation of those 21 case studies,
The authors managed to map all of the 21 success stories back to what they check with as “The translational path” (see figure). They note that, in going from proof of concept to proper product with ML, there are 4 key phases. These phases are:
- The strategy of identifying the fitting problem to resolve, and designing and developing a Machine Learning tool that may create actionable insights.
- Evaluate whether the product can actually improve clinical care and patient outcomes, whether it’s accurate and reliable, and whether there’s a business case to be made for the product.
- This step describes the method where a prood of concept is actually scaled as much as an integrated product. It requires scaling the deployment of the model and diffusing it to early adopters.
- It’s vital to notice that no ML product is ever finished. The models should be repeatedly monitored and updated to avoid faulty behavior. Especially in healthcare the latter can have serious repercussions.
These phases aren’t necessarily sequential, and teams can find themselves moving forwards and backwards between them in an iterative fashion. See the figure below for more details on the translational path.
The review does a fantastic job of describing quite a few challenges and frustration points with regards to creating ML powered products in healthcare, from technical infrastructure challenges to moral risks. I is not going to undergo all of them but I need to spotlight a number of points that I recognised.
Domain knowledge versus productionisation knowledge
When developing medtech tools there’s at all times this tension between domain knowledge and productionisation knowledge. You’ll be able to only have so many individuals in your team, and at the identical time it’s essential make sure that enough medical examiners are involved, but in addition the fitting individuals who can actually construct and deploy your solution. Where to place the main focus is extremely dependable, but Sendak et al do a fantastic job of highlighting the importance of getting each these capabilities in your team ultimately if you desire to achieve success.
After all not every skill set must be represented by an actual person on the team! Certain things will also be outsourced, or tools could be brought in that maintain the usual tasks so experts can deal with what’s unique to your solution. I see so many corporations getting swept up in constructing their very own platforms with a bunch of open source tooling because that’s free. But let’s not forget the prices related to having all of the people in your team which have to speculate time and energy into organising all of that tech! While they’re busy attempting to get a deployment tool working, you’re losing time that you would have spent on actually improving your model and driving value…
When is a model “good”?
It’s often unclear what differentiates a very good model from a foul one, and what performance you need to be striving for in your specific case. If left undiscussed, this will result in a mismatch in expectations and reality. Vital to notice here is that this doesn’t only concern model accuracy, but in addition usability and potential economic performance. A model that has amazing accuracy, but takes 10 hours to run, will probably not be very useful, nor reasonably priced. Every case is different, and it’s key that a conversation is had early on to discover and agree on the relevant model performance metrics.
Demonstrating validity of the product in an isolated context is just not enough
Simply because the product performed great on controlled test environments and test datasets, doesn’t mean that the product will perform well in a real-life setting. It’s vital to get real life data fed through the product, and to make use of it to evaluate its performance. At UbiOps we deal with deployment and serving and we’ve got seen repeatedly that performance can change tremendously after the model is introduced to actual production data! It’s vital to get to that stage early, even when it’s just as a shadow deployment.
Integration into production environments is difficult
The authors note that there is usually a large difference between the actual production environment and the event/sata storage environments. In all the instance cases they investigated they found that always significant effort and investment were needed to integrate products into the present systems. One study estimated the price to validate and integrate the Kidney Failure Risk Equation into clinical workflows at a single site at nearly $220,000. That’s only a single site!
Data is spread across cloud and OnPremise environments
A crucial issue that this review brings to light, is the undeniable fact that data is unfolded across various cloud solutions and on premise data centers. This typically starts causing issues within the scaling out phase. Being aware of this whilst you design your product and architecture can greatly profit the transition from proof of concept to properly rolled out product.
Constantly changing regulatory frameworks
One other major challenge pertains to compliance, data security and the large amount of regulation and required certification for medical devices and software. Not to say the undeniable fact that the foundations are repeatedly changing. It’s difficult to remain on top of all the pieces and ensure that each a part of your product is fully compliant. Data security is particularly an obstacle, as the information is so sensitive.
What now?
I’ve walked you thru the translational path and its major obstacles, so what now? I feel all of it starts with while you got down to create a latest ML powered product within the healthcare sector. Familiarize yourself with the challenges of people who went before you, how are you going to learn from their mistakes?
In my view crucial thing is to not be afraid of truly attending to that diffuse and scale step. , albeit in a shadow mode. Only after making that step are you able to start moving towards a product that really has impact and value.
So how do you ensure that you could actually run things in production? Well, and that means that you can deal with what you’re good at: constructing models for medical use cases. Investing in MLOps tools that may assist you deploy quickly, and monitor what’s occurring, can really help make sure that you could deal with the actual challenges at hand, quite than standard infrastructure challenges. I see so many corporations getting swept up in constructing their very own platforms with a bunch of open source tooling because that’s free. But let’s not forget the prices related to having all of the people in your team investing time and energy into organising all of that tech! While they’re busy attempting to get a deployment tool working, you’re losing time that you would have spent on actually improving your model and driving value…
In terms of data security, it might help to make sure that the tooling that you simply are using already has the fitting certifications (like ISO certifications). Specifically in Europe it might also make sense to work with more area of interest cloud providers quite than the large three. Working with a cloud provider specialized in medical data you’ll have rather a lot less headache in proving that you simply are compliant with all the foundations.
There are various aspects at play with regards to getting AI adopted in routine clinical care. From regulations, to architecture problems, and even just getting the fitting people involved. Sendak et al. managed to capture all of the phases and obstacles succinctly of their “translational path”. Being aware of the 4 distinct phases of this translational path and the several obstacles which may come your way will certainly assist in setting you up for fulfillment.