Thread
Unless you've built a machine learning system before, you'd think it's just like building regular software.
It's not.
Here is a reason they are different and what you can do about it:
1 of 13
It's not.
Here is a reason they are different and what you can do about it:
1 of 13
For the most part, regular software looks like this:
• Build once
• Run forever*
I know there's no such thing as *forever*. Things change all the time.
But this rate of change pales in comparison with machine learning systems.
Here is an example:
2 of 13
• Build once
• Run forever*
I know there's no such thing as *forever*. Things change all the time.
But this rate of change pales in comparison with machine learning systems.
Here is an example:
2 of 13
A real, straightforward example:
You build a deep learning model to process pictures that users capture with their phones.
The goal of the model: recognize different shoe brands in the pictures.
Everything works, but something happens:
3 of 13
You build a deep learning model to process pictures that users capture with their phones.
The goal of the model: recognize different shoe brands in the pictures.
Everything works, but something happens:
3 of 13
Apple releases a new iPhone.
A beautiful, groundbreaking new camera that is now packing pixels differently.
The pictures look stunning, but the model starts having problems.
Guess what happened?
4 of 13
A beautiful, groundbreaking new camera that is now packing pixels differently.
The pictures look stunning, but the model starts having problems.
Guess what happened?
4 of 13
We trained our model on different pictures.
The world changed, but our model didn't.
Although the new pictures aren't enough reason to break our model, there're enough differences to reduce its performance.
Unfortunately, there's more:
5 of 13
The world changed, but our model didn't.
Although the new pictures aren't enough reason to break our model, there're enough differences to reduce its performance.
Unfortunately, there's more:
5 of 13
Nike just released a new pair of shoes. Adidas did the same.
Both go viral. Everyone is buying them!
When we trained our model, these shoes didn't exist. We didn't know about them, so our model doesn't either.
The model's performance takes another hit.
6 of 13
Both go viral. Everyone is buying them!
When we trained our model, these shoes didn't exist. We didn't know about them, so our model doesn't either.
The model's performance takes another hit.
6 of 13
An excellent system turned into a mediocre one in just a few weeks.
The model is the same. The world isn't.
In machine learning, we call this problem "drift."
7 of 13
The model is the same. The world isn't.
In machine learning, we call this problem "drift."
7 of 13
It's natural for those who haven't built complete systems before to assume their work is done as soon as they finish training their model.
In reality, that's when the work truly starts.
Building a model is simple. Building a model that works is another ball game.
8 of 13
In reality, that's when the work truly starts.
Building a model is simple. Building a model that works is another ball game.
8 of 13
Interestingly, most people know how to tackle drift.
But they are missing a critical step: They realize that drift happens when it's too late.
When everything breaks, they scramble and fix things.
You need to do better. You need good *monitoring*.
9 of 13
But they are missing a critical step: They realize that drift happens when it's too late.
When everything breaks, they scramble and fix things.
You need to do better. You need good *monitoring*.
9 of 13
Monitoring helps you see problems coming.
More often than not, it gives you time to react before your system breaks.
I've tried to implement a monitoring system myself.
I'd rather eat glass, so I have a recommendation for you:
10 of 13
More often than not, it gives you time to react before your system breaks.
I've tried to implement a monitoring system myself.
I'd rather eat glass, so I have a recommendation for you:
10 of 13
Look into @cometml's Model Production Monitoring:
comet.com/docs/v2/guides/model-production-monitoring/mpm-overview/?utm_source=svpino&utm_medium=refer...
You have access to 3 fundamental levers:
1. Identify if your model is struggling
2. Track drift across inputs and outputs
3. Get alerts if anything is wrong
11 of 13
comet.com/docs/v2/guides/model-production-monitoring/mpm-overview/?utm_source=svpino&utm_medium=refer...
You have access to 3 fundamental levers:
1. Identify if your model is struggling
2. Track drift across inputs and outputs
3. Get alerts if anything is wrong
11 of 13
@Cometml gives you what you need to answer two questions:
1. Am I going to have a problem with my model?
2. Do I already have a problem with my model?
Not having a robust monitoring system is like driving a car without a gas gauge.
12 of 13
1. Am I going to have a problem with my model?
2. Do I already have a problem with my model?
Not having a robust monitoring system is like driving a car without a gas gauge.
12 of 13
For those who prefer to build everything in-house:
A poorly designed monitoring system is worse than no system at all.
A broken alarm at home that doesn't go off when the thieves come in will make you think everything is fine.
Until it's too late.
13 of 13
A poorly designed monitoring system is worse than no system at all.
A broken alarm at home that doesn't go off when the thieves come in will make you think everything is fine.
Until it's too late.
13 of 13
