Blueprints
Run a MLOps pipeline to deploy a model and create issues if the model is below a given threshold
About this blueprint
AI Python
This blueprint showcases how to run a full MLOps pipeline. It's triggered when a commit is made on a given repository and will train and then deploy a machine learning model to a docker registry.
If the model training results are below a given threshold, the flow will send an alert in Slack and create a corresponding issue in GitHub.
yaml
id: mlops
namespace: blueprint
inputs:
- id: threshold
type: FLOAT
defaults:
- 0.92
tasks:
- id: wdir
type: io.kestra.plugin.core.flow.WorkingDirectory
workerGroup:
key: my-gpu-group
tasks:
- id: clone
type: io.kestra.plugin.git.Clone
url: "{{secret('GITHUB_REPO')}}"
branch: "{{trigger.body['branch']}}"
- id: train
type: io.kestra.plugin.scripts.python.Commands
containerImage: "{{vars.python_image}}"
beforeCommands:
- pip install -r requirements
commands:
- python train/main.py
- python test/validate_model.py
outputFiles:
- out/test_results.json
- out/model.pkl
- id: check_results
type: io.kestra.plugin.core.flow.If
condition: "{{inputs.threshold <= .train.outputFiles['out/test_results.json']) | jq '.accuracy' }}"
then:
- id: slack_success
type: io.kestra.plugin.notifications.slack.SlackIncomingWebhook
url: " {{secret('SLACK_WEBHOOK')}}"
payload: |
{
"text": "New model validated with success {{.train.outputFiles['out/test_results.json']}}"
}
- id: build_container
type: io.kestra.plugin.docker.Build
inputFiles:
model.pkl: "{{ outputs.train.outputFiles['out/model.pkl'] }}"
dockerfile: |
FROM ubuntu
...
tags:
- my-private-repo/my-model:latest
- id: deploy
type: io.kestra.plugin.core.http.Request
method: POST
uri: https://internal-deploy-api/deploy
else:
- id: slack_fail
type: io.kestra.plugin.notifications.slack.SlackIncomingWebhook
url: " {{secret('SLACK_WEBHOOK')}}"
payload: |
{
"text": "Model build unsuccessful, model accuracy below threshold {{.train.outputFiles['out/test_results.json']) | jq '.accuracy' }}"
}
- id: create_git_issue
type: io.kestra.plugin.github.issues.Create
title: Training failed {{flow.id}}-{{execution.id}}
body: "Training failed on {{taskrun.startDate}} as accuracy {{.train.outputFiles['out/test_results.json']) | jq '.accuracy' }} below threshold of {{inputs.threshold}}"
triggers:
- id: github_commit
type: io.kestra.plugin.core.trigger.Webhook
key: "{{secret('GITHUB_WEBHOOK')}}"