Supervised Learning: Classification, Regression & Key

Supervised learning is the most widely used type of machine learning. You provide the algorithm with labelled training data — inputs paired with correct outputs — and it learns the mapping between them. Once trained, the model predicts outputs for new, unseen inputs.

This lesson covers the two main supervised learning tasks (classification and regression), the algorithms you need to know, and how to evaluate whether your model actually works.

Classification vs regression in supervised learning

Supervised learning solves two types of problems:

Classification — predict a category. Examples: spam or not spam, disease diagnosis (positive/negative), image recognition (cat/dog/bird). The output is a discrete label.
Regression — predict a continuous number. Examples: house price, temperature tomorrow, customer lifetime value. The output is a numerical value.

The distinction matters because different algorithms and evaluation metrics apply to each. Classification uses accuracy, precision, recall, and F1. Regression uses MAE (mean absolute error), RMSE (root mean squared error), and R-squared.

Key supervised learning algorithms

These algorithms cover the vast majority of practical supervised learning:

Linear / logistic regression — simple, interpretable, fast. The baseline for regression and binary classification.
Decision trees — split data by asking yes/no questions. Easy to understand and visualise.
Random forests — ensemble of many decision trees. Reduces overfitting, handles messy data well. Often the best first choice.
Gradient boosting (XGBoost, LightGBM) — builds trees sequentially, each correcting the previous one's errors. Wins most Kaggle competitions. Slightly harder to tune.
Support vector machines (SVM) — find the optimal boundary between classes. Effective with smaller datasets.
k-Nearest Neighbours (kNN) — classify based on the closest training examples. Simple, no training phase, but slow at inference for large datasets.

Start with random forests — they work well out of the box and require minimal tuning. Move to gradient boosting when you need maximum performance.

Evaluating supervised learning models

Never trust training accuracy alone. Always evaluate on held-out data:

Split data into train (80%) and test (20%) sets — train_test_split in scikit-learn
Use cross-validation for more robust estimates — cross_val_score
For classification: check the confusion matrix, precision, recall, F1, and AUC-ROC
For regression: check MAE, RMSE, and R² — RMSE penalises large errors more
Plot learning curves to diagnose overfitting (train accuracy high, test low) or underfitting (both low)

The most informative evaluation is a confusion matrix — it shows exactly where the model makes mistakes and helps you decide whether those mistakes are acceptable.

When evaluating supervised learning, start with the fundamentals: what problem it solves, who it serves, and how it compares to the alternatives buyers consider first.

Start with the fundamentals. Define who you are serving, the single outcome you want, and the budget you are comfortable committing before you compare any options.

A practical rule is to favour clarity over cleverness. The choice that is easiest to understand, spell, remember and act on tends to win over the long run.

Context changes the answer. What is optimal for a fast-moving startup is rarely optimal for an established brand, so weigh your own stage, market and ambitions.

Avoid the common mistakes: rushing the decision, copying competitors blindly, and ignoring the total cost of ownership beyond the very first year of use.

Think in terms of trade-offs rather than a single perfect answer. Every option trades memorability, cost, reach and flexibility in a slightly different mix.

Document your reasoning. Writing down why you chose one path makes it far easier to revisit the decision later and to explain it to partners and investors.

Test with real people. A quick check with a handful of your actual customers will surface problems that no amount of internal debate ever reveals.

Plan for growth. The option that fits today should still make sense when your audience, catalogue or team is several times larger than it is right now.

Finally, revisit the decision periodically. Markets, tools and best practices move quickly, and a choice that was ideal last year may deserve a fresh look today.