Evaluating Machine Learning Models -- A Beginner's Guide
•Download as PPTX, PDF•
21 likes•14,160 views
This is an overview on evaluating machine learning models: when, how, metrics, datasets, methods. Topics include metrics, validation, hyperparameter tuning, A/B testing, and multi-armed bandits. It's a summary of my short report on the topic: http://oreil.ly/1LkP2tn.
Recommended
Recommended
More Related Content
Viewers also liked
Viewers also liked (18)
Recently uploaded
Recently uploaded (20)
Evaluating Machine Learning Models -- A Beginner's Guide
- 1. Evaluating Machine Learning Models – A Beginner’s Guide Alice Zheng, Dato September 15, 2015 1
- 2. 2 My machine learning trajectory Applied machine learning (Data science) Build ML tools Shortage of experts and good tools.
- 3. 3 Why machine learning? Model data. Make predictions. Build intelligent applications.
- 4. 4 Machine learning pipeline I fell in love the instant I laid my eyes on that puppy. His big eyes and playful tail, his soft furry paws, … Raw data Features Models Predictions Deploy in production GraphLab Create Dato Distributed Dato Predictive Services
- 5. The ML Jargon Challenge
- 6. Typical machine learning paper 6 … semi-supervised model for with large-scale learning from sparse data … sub-modular optimization for distributed computation… evaluated on real and synthetic datasets… performance exceeds start-of-the-art methods
- 7. What it looks like to ML researchers 7 Such regularize! Much optimal So sparsity Wow! Amaze Very scale
- 8. What it looks like to normal people 8
- 9. What it’s like in practice 9 Doesn’t scale Brittle ? Hard to tune ? ? Doesn’t solve my problem on my data
- 10. Achieve Machine Learning Zen 10
- 11. 11 Why is evaluation important? • So you know when you’ve succeeded • So you know how much you’ve succeeded • So you can decide when to stop • So you can decide when to update the model 11
- 12. 12 Basic questions for evaluation • When to evaluate? • What metric to use? • On what data? 12
- 13. 13 When to evaluate Online evaluation Historical data Online evaluation results Offline evaluation Live data Offline evaluation on live data Prototype model Training results Validation results Deployed model
- 15. 15 Types of evaluation metric • Training metric • Validation metric • Tracking metric • Business metric “But they may not match!” Uh-oh Penguin
- 16. 16 Example: recommender system • Given data on which users liked which items, recommend other items to users • Training metric - How well is it predicting the preference score? - Residual mean squared error: (actual – predicted)2 • Validation metric - Does it rank known preferences correctly? - Ranking loss
- 17. 17 Example: recommender system • Tracking metric - Does it rank items correctly, especially for top items? - Normalized Discounted Cumulative Gain (NDCG) • Business metric - Does it increase the amount of time the user spends on the site/service?
- 18. 18 Dealing with metrics • Many possible metrics at different stages • Defining the right metric is an art - What’s useful? What’s feasible? • Aligning the metrics will make everyone happier - Not always possible: cannot directly train model to optimize for user engagement “Do the best you can!” Okedokey Donkey
- 19. Model Selection and Tuning
- 20. Model Selection and Tuning Historical data Hyperparameter tuning Training data Validation data Model training Model Training results Validation results
- 21. 21 Key questions for model selection • What’s validation? • What’s a hyperparameter and how do you tune it? 21
- 22. 22 Model validation • Measure generalization error - How well the model works on new data - “New” data = data not used during training • Train on one dataset, validate on another • Where to find “new” data for validation? - Clever re-use of old data
- 23. 23 Methods for simulating new data Hold-out validation Data Training Validation K-fold cross validation Data 1 2 3 K… Validation set Bootstrap resampling Data Resampled dataset
- 24. 24 Hyperparameter tuning vs. model training Best model parameters Best hyperparameters Hyperparameter tuning Model training
- 25. 25 Hyperparameters != model parameters Feature 2 Feature 1 Classification between two classes Model parameter Hyperparameter: How many features to use
- 26. 26 Why is hyperparameter tuning hard? • Involves model training as a sub-process - Can’t optimize directly • Methods: - Grid search - Random search - Smart search • Gaussian processes/Bayesian optimization • Random forests • Derivative-free optimization • Genetic algorithms
- 28. 28 ML in production - 101 Model Historical Data Predictions Live Data Feedback Batch training Real-time predictions
- 29. 29 ML in production - 101 Model Historical Data Real-time predictionsBatch training Predictions Model 2 Live Data
- 30. 30 Why evaluate models online? • Track real performance of model over time • Decide which model to use when
- 31. 31 Choosing between ML models Model 2 Model 1 2000 visits 10% CTR Group A Everybody gets Model 2 2000 visits 30% CTR Group B Strategy 1: A/B testing—select the best model and use it all the time
- 32. 32 Choosing between ML models A statistician walks into a casino… Pay-off $1:$1000 Pay-off $1:$200 Pay-off $1:$500 Play this 85% of the time Play this 10% of the time Play this 5% of the time Multi-armed bandits
- 33. 33 Choosing between ML models A statistician walks into an ML production environment Pay-off $1:$1000 Pay-off $1:$200 Pay-off $1:$500 Use this 85% of the time (Exploitation) Use this 10% of the time (Exploration) Use this 5% of the time (Exploration) Model 1 Model 2 Model 3
- 34. 34 MAB vs. A/B testing Why MAB? • Continuous optimization, “set and forget” • Maximize overall reward Why A/B test? • Simple to understand • Single winner • Tricky to do right
- 35. 35 That’s not all, folks! Read the details • Blog posts: http://blog.dato.com/topic/machine-learning- primer • Report: http://oreil.ly/1L7dS4a • Dato is hiring! jobs@dato.com alicez@dato.com @RainyData
Editor's Notes
- Features sit between raw data and model. They can make or break an application.