Understand classification vs regression, training/validation/test splits, overfitting, and common algorithms.
Published May 1, 2025
Supervised learning trains a model on labeled data — input-output pairs — to learn a mapping function that predicts outputs for new inputs.
Classification: predict a discrete label
Regression: predict a continuous value
1. Data Collection → 2. Preprocessing → 3. Feature Engineering
→ 4. Train/Val/Test Split → 5. Model Training → 6. Evaluation
→ 7. Hyperparameter Tuning → 8. Deployment → 9. Monitoring
from sklearn.model_selection import train_test_split
# 60% train, 20% validation, 20% test
X_train, X_temp, y_train, y_temp = train_test_split(X, y, test_size=0.4)
X_val, X_test, y_val, y_test = train_test_split(X_temp, y_temp, test_size=0.5)
# Train: fit model parameters
# Validation: tune hyperparameters (choose best model)
# Test: final unbiased performance estimate (use ONCE)
Linear Regression (for regression)
y = w₁x₁ + w₂x₂ + ... + b
Loss: MSE = Σ(yᵢ - ŷᵢ)²/n
Optimize: gradient descent → update w, b to minimize loss
Logistic Regression (for binary classification)
P(y=1) = sigmoid(w·x + b) = 1 / (1 + e^(-w·x+b))
Loss: Binary cross-entropy
Decision Trees — split data by feature thresholds Random Forest — ensemble of decision trees (reduces overfitting) SVM — find maximum-margin hyperplane Neural Networks — learn hierarchical representations
Underfitting (high bias):
Model too simple → misses patterns in training data
Train loss high, Val loss high
Fix: add complexity (more features, deeper model)
Overfitting (high variance):
Model memorizes training data → fails on new data
Train loss low, Val loss HIGH
Fix: regularization (L1/L2), dropout, more data, cross-validation
Ideal:
Train loss low, Val loss ≈ Train loss
Classification:
Accuracy = correct / total
Precision = TP / (TP + FP) ← how often we're right when we say positive
Recall = TP / (TP + FN) ← how often we catch actual positives
F1 = 2 × (Precision × Recall) / (Precision + Recall)
ROC-AUC: area under Receiver Operating Characteristic curve
Regression:
MSE = mean squared error
RMSE = root MSE (same units as target)
MAE = mean absolute error (more robust to outliers)
R² = proportion of variance explained (0 to 1)
from sklearn.model_selection import cross_val_score
# 5-fold cross-validation
scores = cross_val_score(model, X, y, cv=5, scoring='accuracy')
print(f"Mean: {scores.mean():.3f} ± {scores.std():.3f}")