Data Scientists be like….I have the Model…..everyone else be like…until it hits production
Your very own ML Model is the Lion King to the Jungle…..ahem..ahem..
Infrastructure is almost an afterthought in data science today (MLOps)
Each ML workflow i.e lifecycle involves 3 broad areas which can be grouped together group (Build, Train, Deploy)
Build phase requires a combination of Data Engineering and Data Analytics(hire Big Data Engineer — who engineers the entire data pipeline to gathers the required data from several heterogeneous sources under one roof in the desired form)
Train phase requires Machine Learning skills and finally (hire Data Scientists to research, experiment, prototype and develop a PoC that can validate an ML business use-case)
Deploy phase requires primarily Software Engineering skills and a bit of ML skills (hire Software Engineers to take the PoC into a full-blown and scalable web-service or software product as per the required business standards)
But when it comes to putting those algorithms into production for inference, outside of AWS’s popular SageMaker, there’s not a lot to choose from. MLOps tools remove the burden of manually deploying, monitoring, and managing machine learning pipelines in production
MLOps tool automates many of the operational tasks required to turn a machine learning model from a promising piece of code running in Spark, Flink, TensorFlow, or PyTorch processing engines into a secure, governed, and production-ready machine learning system.
There are like non-standardized 20 steps we do manual deployment of ML Models in production
Did you just say 20 steps…..well deal with it….or not?
Just like DataOps Standards(standards for data e.g. Apache Atlas model of data governance i.e metadata standards and governance standards for data operations and data management) MLOps needs to be defined
MLOps tool (Features)
Serve as a catalog(or a repository) for ML models from data science platforms (keeping all the pipelines up to date)
ML model lineage and provenance, including tracking of who configured and approvals the models
Centralised policy-based management of production and training ML models
Deployment of ML models to distributed compute resources (Reproducible MLOps workflows)
Monitoring of ML health indicators to spot divergence between results of training models and results of production inference models (tracking the health and validation of your ML predictions )
Sandboxing of models for AB testing, including management of “canary” models to monitor for changes in real-world conditions (correlate predictions or outputs from your ML algorithms to business results)
Snapshotting of ML model and data states for later analyses;
A central dashboard for monitoring ML activities (doing ML governance to track all configuration and parameters over time; monitoring diagnostics)
Orchestrating the updating between training pipelines and inference pipelines
Normalise machine learning metadata and monitoring capabilities to look at mathematical factors to predict things like skew, drift, accuracy, or the need to retrain models
Like REALLY…….naah…this is purely being used for dramatization here….
INDUSTRY EXAMPLES of internal MLOps Systems
Airbnb’s Bighead: End-to-End Machine Learning Platform (https://www.slideshare.net/databricks/bighead-airbnbs-endtoend-machine-learning-platform-with-krishna-puttaswamy-and-andrew-hoh)
Uber’s Michaelangelo: to seamlessly build, deploy, and operate machine learning solutions at Uber’s scale (https://eng.uber.com/michelangelo-machine-learning-platform/)
Their team in the Cafeteria be like
Automating the end-to-end lifecycle of Machine Learning applications
Machine Learning applications are becoming popular in our industry, however the process for developing, deploying, and continuously improving them is more complex compared to more traditional software, such as a web service or a mobile application. They are subject to change in three axis: the code itself, the model, and the data. Their behaviour is often complex and hard to predict, and they are harder to test, harder to explain, and harder to improve. Continuous Delivery for Machine Learning (CD4ML) is the discipline of bringing Continuous Delivery principles and practices to Machine Learning applications.
Pertinent MLOps Questions
Say an early team wants to deploy 5 to 10 models. That is manageable. But what happens when an organization wants to ramp it up to 1,000 or 2,000 models?
How do ML/AI leading platforms like LinkedIn, Airbnb, and Uber solve complex MLOps-related problems by building their own internal MLOps systems ?
What are the implementation risks to business if
Predictions go wrong (roll-out of ML models manually is a complex and error prone process)
Models go stale (under-fitting or over fitting because of data divergence from new data vs the old data trained models)
Undeployed models lying at development level.
Imagine no more nightmares from the black fat Panda
There are two really different worlds to machine learning engineering
ML Algos: theoretical data science side, where people talk about neural networks and hidden layers and back propagation and
PyTorch
TensorFlow
MLOps: Actual system side of things, which is Kubernetes and Docker and Nvidia and running on GPUs and dealing with S3 and different AWS services (The starting point is a trained model i.e deployment into customers’ AWS accounts using its Kubernetes engine i.e point us at a model, and basically get it converted it into a Web API. It handles all the productionalization challenges around it.)
TOOL1: CLOUDERA (CML & SDX)
Cloudera Machine Learning (CML) & Cloudera SDX for models
Data scientists, machine learning engineers, and operators can collaborate in a single unified solution, drastically reducing time to value and minimizing business risk for production machine learning models.
Allows visibility into the entire ML lifecycle to eliminate silos and blind spots for full lifecycle transparency, explainability and accountability (Unique model cataloging and lineage capabilities)
Securely deploy machine learning models to production, ensure accuracy, and scale use cases.(Full end-to-end machine learning lifecycle management)
Track and monitor both technical aspects and accuracy of predictions in a repeatable, secure, and scalable way (A first-class model monitoring service )
Detect model performance and drift over time with native storage and access to custom and arbitrary model metrics.
Measure and track individual prediction accuracy, ensuring models are compliant and performing optimally.
TOOL2: SAGEMAKER
https://aws.amazon.com/sagemaker/
AWS Sagemaker fits best for us. It provides Jupyter NoteBooks running R/Python kernels with a compute instance that we can choose as per our data engineering requirements on demand
You can visualize, process, clean and transform the data into our required forms using the traditional methods we use (say Pandas + Matplotlib or R +ggplot2 or other popular combinations)
Post data engineering, you can train the models using a different compute instance based on the model’s compute demand, say memory optimized or GPU enabled
Leverage a smart default high-performance hyperparameter tuning settings for a variety of models.
Leverage performance-optimized algorithms from the rich AWS library or bring our own algorithms through industry standard containers.
Deploy the trained model as an API, again using a different compute instance appropriate to meet business requirements and scale elastically.
The entire process of provisioning hardware instances, running high capacity data jobs, orchestrating the entire flow with simple commands while abstracting the mammoth complexities and finally enabling serverless elastic deployment works with a few lines of code and yet is cost effective.
TOOL3: CORTEX
Simply put
Deploy models from any framework as production APIs
Cortex deploys simple python functions with heavy external deps to Kubernetes (Cortex can be thought as a tool for deploying, scaling, and monitoring Python functions on AWS, but few important nuances that makes Cortex different from lambda.
Inference workloads are read-only, benefit a lot from GPU infrastructure, and often memory hungry.
ML specific optiomizations for constraints (for example: prioritizing high-memory GPU spot instances).
ML specific features like prediction monitoring and support for ONNX and TensorFlow serving exported files.
Model retraining but we're spending a lot of time upfront on making inference easy at scale.
It is a tool for scaling real-time inference
E.g. shifting inference workloads from CPUs to GPUs in the AWS cloud, or vice versa
Automatically spinning up more AWS servers under the hood when calls to the ML inference service are high,
and spinning down the servers when that demand starts to drop
built-in AWS cost-optimization capabilities
built-in software logs and monitors all activities, which is a requirement in today’s security- and regulatory-conscious climate
The software works with a range of frameworks, including TensorFlow, PyTorch, scikit-learn, and XGBoost
It can take advantage of EC2 spot instances, and integrates with AWS services like Elastic Kubernetes Service (EKS), Elastic Container Service (ECS), Lambda, and Fargate. The Kubernetes management alone may be worth the price of admission.
Other Products: algorithmia.com
Hidden Technical Debt in Machine Learning Systems
In the famous Google paper published by Sculley et al. in 2015 "Hidden Technical Debt in Machine Learning Systems", they highlight that in real-world Machine Learning (ML) systems, only a small fraction is comprised of actual ML code. There is a vast array of surrounding infrastructure and processes to support their evolution. They also discuss the many sources of technical debt that can accumulate in such systems, some of which are related to data dependencies, model complexity, reproducibility, testing, monitoring, and dealing with changes in the external world.
Notes: Extra detailed reading from a curated list of articles
Source1: https://www.datanami.com/2020/01/27/an-open-source-alternative-to-aws-sagemaker/
Source2: https://www.datarevenue.com/en-blog/airflow-vs-luigi-vs-argo-vs-mlflow-vs-kubeflow
Source4: https://towardsdatascience.com/why-do-we-need-aws-sagemaker-79bce465f19f
Source6: https://martinfowler.com/articles/cd4ml.html