What is grid search, and how does it work?

What you need to know

You give grid search a grid: a list of candidate values for each hyperparameter. It builds every combination (the Cartesian product) and evaluates each one with cross-validation — splitting the training data into k folds, training on k−1 and validating on the remaining one, k times.

1. Define the grid — for example max_depth in [3, 6, None] and min_samples_leaf in [1, 10]: 3 × 2 = 6 combinations.
2. Cross-validate each combination — with 5 folds, that is 6 × 5 = 30 model fits.
3. Average the scores for each combination and pick the best.
4. Refit the best combination on all the training data (refit=True is the default).
5. Evaluate once on the held-out test set.

In scikit-learn

1import pandas as pd2from sklearn.datasets import make_classification3from sklearn.ensemble import RandomForestClassifier4from sklearn.model_selection import GridSearchCV, train_test_split56X, y = make_classification(n_samples=2000, n_features=20, n_informative=6,7                           weights=[0.9], random_state=0)          # 10% positives8X_tr, X_te, y_tr, y_te = train_test_split(X, y, stratify=y, random_state=0)910param_grid = {"max_depth": [3, 6, None], "min_samples_leaf": [1, 10]}11search = GridSearchCV(RandomForestClassifier(n_estimators=200, random_state=0),12                      param_grid, cv=5, scoring="roc_auc", n_jobs=-1)13search.fit(X_tr, y_tr)1415results = pd.DataFrame(search.cv_results_)[["param_max_depth", "param_min_samples_leaf",16                                             "mean_test_score", "std_test_score"]]17print(results.round(3).to_string(index=False))18print("best:", search.best_params_, "| test ROC-AUC:", round(search.score(X_te, y_te), 3))

param_max_depth  param_min_samples_leaf  mean_test_score  std_test_score              3                       1            0.902           0.011              3                      10            0.903           0.019              6                       1            0.922           0.017              6                      10            0.919           0.025           None                       1            0.930           0.020           None                      10            0.922           0.025best: {'max_depth': None, 'min_samples_leaf': 1} | test ROC-AUC: 0.953

Three things to read here. scoring="roc_auc" sets the metric — always choose it deliberately; the default is accuracy. cv_results_ holds every combination's mean and standard deviation, which is more informative than best_params_ alone. And the standard deviations (around 0.02) are as large as the differences between the top four rows, so those settings are effectively tied; the test score is then measured once, at the end.

If the pipeline includes preprocessing, search over the whole Pipeline (for example {"model__max_depth": [...]}) so the scaler and encoder are re-fitted inside every fold and nothing leaks.

Strengths and weaknesses

• Strengths — exhaustive over the grid, easy to explain, fully reproducible, trivially parallel (n_jobs=-1).
• Weakness 1: cost grows multiplicatively. 4 hyperparameters × 5 values = 625 combinations; with 5 folds, 3,125 fits.
• Weakness 2: wasted budget. If one hyperparameter hardly matters, every one of its values multiplies the cost while adding nothing.
• Weakness 3: only the values you listed. If the best learning rate is 0.03 and your grid has 0.01 and 0.1, you will never find it.

A common practical pattern is coarse-to-fine: a wide grid on a log scale first (0.001, 0.01, 0.1, 1), then a narrower grid around the winner.

A real-life example

A telecom company tunes a churn model's C for logistic regression. Only one hyperparameter matters, so a grid of 8 values on a log scale from 0.001 to 100, with 5-fold CV, is just 40 fits and runs in under a minute. Grid search is the right tool here. Later, when the team switches to gradient boosting with six hyperparameters, a grid with five values each would need 15,625 combinations, and they move to random search.

Follow-up questions to expect

• "How many fits does a grid search run?" — The product of the number of values for each hyperparameter, times the number of CV folds, plus one final refit.
• "What is HalvingGridSearchCV?" — An experimental scikit-learn search that evaluates all candidates on a small budget, keeps the best third or half, and gives them more data — much cheaper for large grids.
• "Why use a log scale?" — For strength and rate settings, the difference between 0.001 and 0.01 matters as much as between 0.1 and 1. A linear grid would waste most points on large values.