Why can a model score brilliantly and still be useless?
A model can score brilliantly and still be useless when it is tested on the same data it learned from, because it may have memorised those examples rather than learned something that carries over to new customers.
Picture a fictional 120-person meal-kit subscription business, Pantrybox. An analyst, Kofi, builds a model to predict which subscribers will cancel next month so the retention team can call them first. His update to the head of customer, Sian, reads: 'Model A: 99% accuracy!! uses 85 columns incl. customer ID, postcode, every past order date. Model B (simple, 2 columns): 69% on training, 68% on test. Model A on last month's new data: 71%. About 30% of customers cancel each month.' Sian wants to know which model to use, and why the 99% one did so badly last month.
| Model | Training accuracy | Test accuracy | What it suggests |
|---|---|---|---|
| Model A, 85 columns | 99% | 71% | Overfitting: memorised training customers |
| Model B, 2 columns | 69% | 68% | Underfitting: barely beats always guessing 'stays' |
How do you diagnose overfitting and underfitting step by step?
Diagnose both problems by comparing performance on the training data with performance on data the model has never seen, and by comparing both against a simple baseline. The pattern of those three numbers tells you which problem you have.
1. Hold data back
Before training, set aside a test set the model never sees. Testing a model on the data it learned from is a basic mistake, because a model that simply repeats what it has seen scores perfectly and predicts nothing.
2. Set a baseline
Work out what a trivial rule would score. At Pantrybox, about 70% of customers stay each month, so guessing 'stays' for everyone scores about 70%. A useful model must beat that clearly.
3. Compare training and test
High training score with a much lower test score signals overfitting. Low scores on both, close to the baseline, signal underfitting. Similar, good scores on both suggest a reasonable fit.
4. Check stability
Cross-validation trains and tests the model several times on different slices of the training data and averages the results. If scores swing widely between slices, the model is fragile.
5. Choose a fix that matches the problem
Overfitting and underfitting need opposite remedies, so name the problem before you change anything.
How do you fix overfitting and underfitting?
Fix overfitting by making the model simpler or its data more representative, and fix underfitting by giving the model more useful information or more flexibility. In plain words:
For overfitting
Remove columns that identify individuals rather than behaviour, such as customer ID; use training data that represents the customers you will actually score; prefer a simpler model; and apply regularisation, which penalises complexity during training so the model favours simpler patterns.
For underfitting
Add features that plausibly relate to cancelling, such as skipped deliveries, recent complaints or a price change; allow a more flexible model; and check the data is not so noisy or mislabelled that no model could learn from it.
For both
Keep the test set untouched until the end. If you tune the model repeatedly against the test set, information leaks into the model and the test score stops being an honest estimate.
What does a weak and an improved churn model decision look like?
The weak decision picks Model A because 99% is the biggest number on the slide. In use it is barely better than guessing, and the retention team wastes calls on customers who were never going to leave.
The improved decision reads: 'Model A memorised individual customers: it includes customer ID and exact order dates, so it recognised people it had seen rather than learning why people cancel. Model B is too simple to beat the 70% baseline. Next step: Model C uses six behaviour features, including skipped deliveries in the last month, support tickets and weeks since the last menu change, with customer ID and postcode removed. Target: a test score clearly above 70% and within a few points of its training score, checked with five-fold cross-validation, before the retention team relies on it.' Sian now knows why the 99% model failed and what good looks like.
How do you check a model is neither overfitted nor underfitted?
Check a model by confirming that its test score is close to its training score, clearly better than the baseline, and stable across cross-validation folds and across a fresh month of data. Also ask whether any feature would only be known after the outcome, such as a cancellation reason; that is leakage, and it makes both scores look better than the model really is.
What are the common mistakes with overfitting and underfitting?
The most common mistake is reporting training accuracy as if it were real-world accuracy. Others: ignoring the baseline, so 70% looks impressive when it is what guessing achieves; adding every available column in the hope of a higher score; peeking at the test set while tuning; and treating overfitting as a reason to add complexity, when it usually needs the opposite.
Where does this lesson sit in Tosheo.ai?
This lesson is in the Machine Learning category and follows the introduction to models in the machine learning course. In Tosheo.ai the hands-on task gives you a fictional model report to diagnose, and the rubric feedback checks that you compared training, test and baseline results before recommending a fix. It is a professional foundation for working alongside data teams; the optional Code track covers training and evaluating models in code.
Sources and scope
Google Machine Learning Crash Course: overfitting (checked 19 October 2026) ↗
scikit-learn user guide: cross-validation (checked 19 October 2026) ↗