In theory, the potential of machine learning represents a paradigm shift — a competitive edge in a constantly changing tech industry. Wielding these tools — and the data that drive them — can create a separation between smaller companies and their larger, possibly more antiquated counterparts. For many business leaders, data is the most valuable commodity they can leverage.
But data science is where the theoretical and the practical collide. While many companies invest in machine learning models to improve their products, a sobering truth tends to undermine even the most realistic of ambitions. As VentureBeat reported in 2019, nearly 90 percent of machine learning models never make it into production.
“There’s a common problem that comes up when a data science team has built a successful proof of concept, and then they need to build a production system around it,” said Matt Doherty, an engineer at healthtech company ClosedLoop.
A huge gulf — oftentimes a structural barrier to entry — tends to exist between conceiving a promising idea and bringing it to market, Doherty explained. Without support from an operations team to monitor and audit the methodologies in place, it’s difficult to adapt the model as new challenges arise.
“You know the statistic that 90 percent of data science projects never make it into production?” Doherty said. “This is how the journey starts for the 90 percent.”
Yet while there are many barriers to adopting machine learning operations, or MLOps, they’re far from insurmountable. As Built In Austin learned through conversations with Doherty and Philip Ollapally of information technology company Rex, there are plenty of solutions available to overcome these challenges and deliver a higher rate of successful data science outcomes.
Why did your teams adopt MLOps to transition your work into production?
At ClosedLoop, our production models are guiding real-world healthcare decisions at scale, and to do that, we have to solve some thorny operational issues.
First, healthcare data is diverse, messy and constantly changing. For example, due to an upstream issue, our data might say that no one in Illinois refilled a prescription in the past week. We need to identify these problems immediately, so we depend on MLOps solutions like data-versioning and feature-drift monitoring.
Second, we need to be able to continue iterating and improving models once they are in production. Our MLOps solution allows us to change any part of the pipeline, from initial ETL through hyperparameter optimization, without affecting the production model. This means we can experiment with radical changes while also easily deploying production updates.
Finally, our models must be continuously monitored and audited for algorithmic bias. In healthcare, there are many different modeling pitfalls for machine learning practitioners to fall into — and the consequences can be dire, like inadvertently perpetuating racial or gender disparities. MLOps is used to catch these types of problems before they reach production.
Healthcare data is diverse, messy and constantly changing.”
What benefits do teams that fold in operations stand to gain?
Without support from an operations team, it’s often tough to get anything running at all. At this point, a data scientist might try to do it all in Jupyter or hack something together with Bash scripts.
To overcome this, MLOps can be employed much earlier. Done well, MLOps is about doing your initial experimentation and modeling using tools and processes that are designed to transition into a production workflow. If you don’t do this, you will inevitably have to redo all the work to put the model in production — often with the help of a separate engineering team. The problem with this approach is that the handoffs create opportunities for miscommunication, it’s difficult to monitor the models in production and changes become extremely expensive. Early use of MLOps can overcome all of these problems.
How has the nature of your day-to-day work changed since you started implementing MLOps methodologies?
For one thing, we spend a lot less time dealing with operations emergencies. This is something we expected to see as we built out our MLOps tooling.
Something less expected was how much more time we were able to spend iterating and improving on our models. It turns out that getting a model up and running in a production workflow is a really major milestone, which is obvious in hindsight. But because we’re able to hit that point so quickly, it’s easy then to justify spending additional time and resources on incremental modeling improvements, since we can be confident we’ll be able to realize a return on our investment. This is how the ML virtuous cycle gets started: incremental investment turns into incremental return, which is then reinvested, and so on. And all because of MLOps!
Why did your teams adopt MLOps to transition your work into production?
At Rex, we firmly believe machine learning is a critical enabler to scale our products. We made early investments in machine learning that are already paying off. Our MLOps team functions as an extension of the core operations team and owns the full life cycle of every machine learning initiative, from framing the problem, assessing data readiness and model training to developing integrations and production deployment. They not only understand the nuances of operations, but are also involved in data collection and labeling efforts. The tight collaboration between MLOps, engineering and product teams has made it possible for us to get our machine learning models into production at a record pace.
Machine learning is a critical enabler to scale our products.”
What are the main barriers to entry for teams looking to adopt MLOps practices?
Establishing key performance indicators tied to outcomes is crucial in enabling buy-in from end users and leaders. At Rex, we achieved this by tying initiatives to business outcomes, like driving automation and improving job quality or customer experience. Our MLOps teams are measured both on accuracy and coverage of models.
Don’t underestimate the importance of data quality, as the accuracy of machine learning models is only as good as the data used for training the models. Our MLOps validates ground truth within the model training datasets.
Keep the scope narrow when scoping out new machine learning initiatives. We account for variability and experiment with a few models until we achieve the desired levels of accuracy. In some cases, ensemble models yield more accurate outcomes.
Containerization and endpoint integration offset the rigidity of open source model outputs; we use containerization and APIs to minimize the effort required by our back-end engineering team.
Lines get blurry during production deployment between DevOps, machine learning and data engineering teams. This is in part due to the novelty of machine learning for most organizations, combined with the shortage of ML product managers. At Rex, our MLOps leaders bridge this gap.
How has the nature of your day-to-day work changed since you started implementing MLOps methodologies?
Most organizations are still figuring out efficient ways to implement MLOps methodologies, and there isn’t a set path. While Rex’s MLOps team is still not optimized, we’re getting better at tracking and automating ad-hoc tasks through each production cycle. We’re now in a rhythm of accurately forecasting efforts, tracking tasks and including machine learning initiatives as part of our sprint cycles. This is especially important for us now as we develop more complex AI and machine learning models, including our computer vision and natural language processing initiatives.
