Scaling using AI/ML by constructing Continuous Integration (CI) / Continuous Delivery (CD) / Continuous Training (CT) pipelines for ML based applications

Background

In my previous article:

MLOps in Practice — De-constructing an ML Solution Architecture into 10 components

I talked in regards to the importance of constructing CI/CT/CD solutions to automate the ML pipelines. The aim of MLOps automation is to repeatedly test and integrate code changes, repeatedly train latest models with latest data, upgrade model performance when required, and repeatedly serve and deploy models to a production environment in a protected, agile and automatic way.

In today’s article, we’re going to dive into the subject of MLOps automation. Specifically we’re going to cover the next components:

• Why is MLOps automation essential?
• A high-level introduction to DevOps and CI/CD and its relevance to ML
• What’s special about MLOps in comparison with DevOps?
• A sample CI/CD architecture for ML based systems

Now let’s start by understanding why MLOps automation is essential.

Why is MLOps automation essential?

Constructing ML systems is an especially iterative process — These iterations could be reflected in the next 2 features:

• Experiment-driven: Firstly, training ML models could be very experiment-driven. To get a satisfactory performance of a selected ML model, data scientists generally must conduct a whole lot (and even possibly 1000’s) of experiment runs around mixtures of various feature engineering techniques, model architecture definitions, and hyperparameters. Manually experimenting with all these potential mixtures and choosing the best-performing one, could be very tedious and time-consuming, particularly when one data science team needs to administer tens (and even a whole lot) of ML applications concurrently.
• Data-dependent: Secondly, different from traditional software, ML based applications are extremely data dependent. The ML based applications potentially require frequent updates depending on how quickly the underlying data changes. When frequent updates change into essential, manually re-training an ML model and deploying the newer version into the production environment could be very time-consuming. More importantly, manual updates can significantly decelerate the applying release cycle, which might result in missed market opportunities or un-met regulatory requirements.

Without MLOps automation, it is extremely difficult for any organization to using AI/ML without always recruiting data science or ML resources for brand new ML use cases / applications.

One other point price mentioning, before we get into the detail, is that, implementing ML in a production environment doesn’t only mean deploying your model as an API for prediction. Quite, it means deploying an ML pipeline that may automate the retraining and deployment of latest models. Establishing a CI/CD system allows you to mechanically test and deploy latest pipeline implementations

What’s DevOps and CI/CD ?

In response to Wikipedia,

DevOps is a strategy within the software development and IT industry. Used as a set of practices and tools, DevOps integrates and the work of software development (Dev) and IT operations (Ops) as a method for improving and shortening the systems development life cycle.

The important thing essence of DevOps is “automation”, to hurry up the discharge of the software development cycle. As was explained in the primary a part of this text, ML based applications need fast updates and releases, as a result of the indisputable fact that performance of ML models relies on always evolving data profiles.

Hence, although being quite different from traditional software, ML based applications are still a bit of software, fundamentally. Subsequently, when data scientists and ML engineers deploy ML based applications, there are still huge advantages from adopting the DevOps principals and methodologies, which have been leveraged by traditional software development for years.

As many data scientists don’t come from the software engineering and compute science background, the concept of DevOps and CI/CD could also be unfamiliar to them. Subsequently, before we get into the detail of designing CI/CD pipelines for ML based applications, let’s fist level set, to quickly undergo what a CI/CD pipeline encompasses.

To the guy data scientists, please don’t be scared away by DevOps and CI/CD. It will not be that difficult particularly once you don’t aim to change into an DevOps expert. I’ll try to clarify them in relatively easy layman terms. In actual fact, understanding how it really works on the high-level can bring many advantages. For instance, you’ll be able to have a feel for what your peers / counterparts like ML engineers and DevOps engineers are working on and this will make the communications with them much smoother. For those who wish, you may also expand your skills from developing ML models to constructing CI/CD pipelines, to regularly becoming a full-stack data scientist.

To place things simply, you’ll be able to consider a CI/CD pipeline as an automatic workflow including a series of steps that should be executed before the software is reliably and securely deployed to the production environment. CI/CD pipelines are the backbone of a DevOps methodology. A CI/CD pipeline is usually developed in a yaml file (you’ll be able to check with the instance below). As suggested by the name, a CI/CD pipeline generally be comprised of two parts:

• Continuous Integration (CI) — The tasks in a CI pipeline are focused on testing and validating the source codes and constructing the codes into executable artifacts and usable applications. The testing within the CI pipeline is usually divided into two categories, one is unit testing and the opposite is integration testing. Unit testing makes sure each function (def function()) works as expected and integration testing focuses on verifying that different modules or services, utilized by your software, work thoroughly together. Integration testing for ML based systems is around ensuring every step (ingestion, splitting, transforming, training, evaluation and prediction) of the ML pipeline works well together.
• Continuous Delivery (CD) — The tasks in CD are generally triggered after a successful CI run, with CD specializing in automating the infrastructure provisioning and application release process. Nowadays Terraform is widely used for infrastructure provisioning and it is suggested to check the Terraform scripts as well, before deploying. Apart from infrastructure provisioning, on the CD stages, the pre-built artifacts will likely be deployed right into a staging / testing environment, which is sort of a cloned version of the production environment. On this staging environment, there will likely be rigorous testing scripts executed to discover bugs that slipped through the initial pre-build testing process.

All of the above tasks included in a CI/CD pipeline could be repeatedly executed mechanically. Subsequently the true value of a CI/CD pipeline lies in automation. Finally, a CI/CD pipeline is a design decision. There is no such thing as a right or improper answer here. You’ll be able to resolve what tasks to be included, depending on the necessity of your personal applications.

What’s special about MLOps in comparison with DevOps?

We’ll discuss how MLOps is different from DevOps repectively by CI and CD. Let’s start with CI.

CI for ML based systems

For the CI part, certainly one of the important thing differences between MLOps and DevOps pertains to the scope of testing. As was explained within the previous part, the testing involved with a CI pipeline of traditional software is principally unit testing and integration testing, which mainly focuses on testing the standard of code. On account of the individuality in ML based systems, the testing ought to be expanded to incorporate data testing and model testing. Data testing includes data quality checking, unit testing of the feature engineering logics, validity in train/test data split and verifying data distribution. Model testing includes model performance validation against the test dataset, model output schema check and testing the performance of model prediction services by way of throughput and latency.

CD for ML based systems

The CD a part of traditional software is the delivery of a software or a service, while the CD a part of a ML based system is about deploying a multi-step pipeline to mechanically retrain and deploy models, which adds an additional layer of complexity. Subsequently, there are two forms of CD pipelines for ML based systems. One is continuous delivery for ML training pipelines, also called Continuous Training (CT) pipelines, and the opposite is continuous delivery for the deployment of model prediction services. These two forms of CD pipelines also work closely with one another. Generally a latest run of the ML training pipeline will trigger a latest deployment of the ML model prediction service.

Continuous Training (CT) pipelines are generally triggered by two forms of changes.

• The primary variety of change is code related. For instance, data scientists make some changes to the model training source codes including feature engineering logics changes, model architectures changes, hyperparameters changes, or configuration/variable changes. Code changes will at all times trigger a latest CI/CD pipeline run and a latest execution of the ML training pipeline to generate a more recent version of the model.
• The second variety of change is data related, meaning model retraining is required as a result of underlying data drift. If there is no such thing as a code change and only latest data is provided, model retraining could be executed by triggering the ML training pipeline within the production environment to create a more recent version of the model.

Whatever the variety of CT pipeline, a more recent version of the ML model is generated. If the newer version model has a greater performance, it would be registered within the model registry store, which can trigger the ML deployment service to generate a latest model prediction service.

CD for the deployment of model prediction service is followed by a latest run of ML training pipeline. Typically, there are two forms of ML deployment. One is online (real-time) model deployment where ML models are normally packaged either as REST API endpoints or self-contained Docker images with the REST API endpoints. The opposite is offline (batch) model deployment where ML models are used to directly rating files. With offline model deployment, the trained models are called and fed with a batch of information at a certain interval, (equivalent to once per day or once per week depending on how the models are utilized in certain business contexts), to periodically generate predictions to be used. For those who are keen on learning more about ML model deployment patterns, you’ll be able to check with my previous article:

MLOps in Practice — Machine Learning (ML) model deployment patterns

In today’s article, we’re specializing in the CD pipeline design for the deployment of model prediction service. Whatever the variety of model deployment to your ML based applications, you need to include the next tasks once you design and construct the CD pipelines to your ML model deployment:

• Infrastructure provisioning and testing for scripts of infrastructure provisioning
• Input data validation to make sure that the info sent to the model for prediction meets the required input schema
• Model performance test by segmentations to make sure that the model still performs as expected
• Throughput and latency test of the model prediction service. This is especially relevant for online model service

A sample CI/CD/CT architecture for ML based systems

Below is a sample CI/CD/CT reference architecture for ML based systems

This reference architecture includes the potential tasks respectively for CI, CT and CD.

• Continuous Integration — unit test, integration test, data test and model test
• Continuous Training — data extraction, data validation, feature engineering, model training, model evaluation and model validation.
• Continuous Delivery — infrastructure provisioning, pre-production deployment, model throughput testing and model latency testing.

There are just a few popular CI/CD tools that may all be leveraged to implement the above reference architecture, equivalent to Github actions, Azure DevOps pipelines and Jenkins pipeline. Please leave a comment to let me know should you would love to see an article on practical implementation of CI/CD/CT pipelines. I’m greater than completely satisfied to put in writing one.

The top

Please be happy to let me know if you might have any comments and questions on this topic or other MLOps related topics! I generally publish 1 article related to data and AI every week. Please be happy to follow me on Medium so you could get notified when these articles are published.

If you must see more guides, deep dives, and insights around modern and efficient data+AI stack, please subscribe to my free newsletter — , thanks!

Note: Just in case you haven’t change into a Medium member yet, and you actually should, as you’ll get unlimited access to Medium, you’ll be able to enroll using my referral link!

Thanks a lot to your support!