Lecture 8: Hyperparameter optimization

Varada Kolhatkar

Focus on the breath!

Take a moment to settle in. Relax your shoulders and take a slow breath in and out.

Check-in

How are you feeling today?

• 1. Things are more or less under control!
• 2. Excited about hyperparameter optimization!
• 3. Lost 😞
• 4. Tired / sleepy 😴
• 5. Thinking of lunch 🥗

Today’s big questions

1. How do we choose settings for a complete pipeline?
2. How do grid and random search spend a search budget?
3. How much should we trust the winning validation score?

We will use slides to explain the choices and a live notebook to run the searches. Keep the central puzzle in mind: does a higher validation score after more trials necessarily mean a better model?

Recap: Parameters vs. hyperparameters

Component	Learned during fit	Chosen before fitting
Logistic regression	Coefficients, intercept	C
Decision tree	Splits, leaf predictions	max_depth
Bag of words	Vocabulary	max_features

How should we choose the hyperparameters?

A search compares models fitted under different settings. The vocabulary is learned from training documents; its maximum size is a setting we choose. We have already compared tree depths with loops and validation scores.

For each setting, we already know how to:

1. Fit a model within each CV training fold.
2. Score it on the corresponding validation fold.
3. Compare mean validation scores.

Now automate this workflow for combinations of settings.

Recall choosing a tree depth with a loop. The goal is not to maximize training accuracy. Automated tools handle fitting, scoring, bookkeeping, selection, and refitting. We still choose the search space and evaluation procedure.

Choosing a candidate pipeline

Our demo predicts whether a listener likes a song.

We will tune a pipeline with:

• Preprocessing for numerical, categorical, binary, and text features.
• An RBF SVM classifier.

The settings interact: changing the representation can change which model settings work well.

The Spotify target describes one listener’s preference, not universal song quality. We omit artist identity. The notebook constructs the column transformer and compares a most-frequent baseline with the default SVC using CV. Keep all preprocessing inside the searched pipeline.

Why does tuning get expensive?

The demo tries every combination of:

• 2 vocabulary sizes (max_features).
• 4 regularization settings (C).
• 4 RBF kernel settings (gamma).

How many model fits does five-fold cross-validation require?

\[2 \times 4 \times 4 = 32\text{ candidate pipelines}\] \[32 \times 5 = 160\text{ CV fits}\]

More combinations and more folds mean more fits.

Each CV fit trains the entire pipeline, including preprocessing. The 160 fits exclude the final refit of the selected pipeline on all training data. With five hyperparameters and ten values each, there are 100,000 candidates and 500,000 CV fits. Automation alone does not reduce this cost if it evaluates the same candidates and folds.

Demo 1: Data, pipeline, and baseline

Lecture 8 Class Demo

Run Section 1:

• Reserve the test set.
• Inspect the Spotify data and preprocessing pipeline.
• Compare the dummy and default SVC baselines.

Which rows may the scaler and vocabulary learn from in each CV fold?

Notebook: /Users/kvarada/CS/2026-27/330/cpsc330-2026W1/content/classes/varada/lecture-08-hyperparameter-optimization-demo.ipynb

Run Section 1 only, then return to slides. Each transformer may learn only from the fold’s training portion. The pipeline is what enforces this. CountVectorizer receives the string column selector song_title; other transformers receive lists of columns.

Hyperparameter optimization with loops

• Define a parameter space.
• Iterate through possible combinations.
• Evaluate model performance.

What are some limitations of this approach?

Two ways to search in sklearn

Method	Candidates evaluated	Useful when…
GridSearchCV	Every specified combination	We have a small set of combinations to compare
RandomizedSearchCV	A fixed number of sampled combinations	We want to explore a larger space within a budget

Both use CV on training data, select a candidate, and by default refit on all training data.

Neither guarantees the best possible settings on future data. Both support parallel execution. We choose the pipeline, parameter space, CV procedure, scoring metric, and computational budget.

Naming pipeline hyperparameters

pipe_svm = make_pipeline(preprocessor, SVC())

Search key	What changes?
svc__C	SVM’s C
svc__gamma	SVM’s gamma
columntransformer__countvectorizer__max_features	Song-title vocabulary limit

Use step__parameter, nesting further when needed.

These names come from make_pipeline and make_column_transformer. Inspect pipe_svm.get_params() to find available keys. The double underscores navigate through the pipeline’s structure.

How to use grid search

param_grid = {
    "columntransformer__countvectorizer__max_features": [100, 400],
    "svc__gamma": [0.001, 0.01, 0.1, 1.0],
    "svc__C": [0.01, 0.1, 1.0, 10.0],
}
grid_search = GridSearchCV(
    pipe_svm, param_grid, cv=5, scoring="accuracy",
    n_jobs=-1, return_train_score=True,
)
grid_search.fit(X_train, y_train)

Exhaustive within this grid, including preprocessing in every fit.

This is the notebook’s grid. n_jobs=-1 uses all available CPU cores; it can reduce elapsed time while increasing memory use. return_train_score=True requests training scores for diagnosis. refit=True is the default. All code on slides is illustrative; execution and outputs live in the notebook.

What does the fitted search contain?

Attribute	Meaning
best_params_	Selected settings
best_score_	Winning mean CV validation score
best_estimator_	Selected pipeline refitted on all training data
cv_results_	Scores and timings for all candidates

mean_test_score in cv_results_ refers to CV validation folds.

It is not a score on our reserved test set.

The search object can predict directly after refitting. Training scores help diagnose fitting behaviour; std_test_score describes fold variation, not a confidence interval for future accuracy. In the demo, inspect the compact table rather than the whole results dictionary.

Choosing useful search ranges

• For positive scale settings such as C and gamma, explore orders of magnitude.
• Inspect promising regions and interactions, rather than only the winner.
• If a winner lies at a boundary, consider extending that range.

Use training-set CV to guide further searches. Keep the test set reserved.

The demo heatmaps use the already-computed results, holding vocabulary size fixed in each panel. No additional fits are needed. A boundary result does not establish an optimum or guarantee that extending the range helps. Every refinement based on CV adds to model selection.

How to use random search

list_search = RandomizedSearchCV(
    pipe_svm, param_distributions=param_grid, n_iter=20,
    cv=5, scoring="accuracy", random_state=123,
    n_jobs=-1, return_train_score=True,
)
list_search.fit(X_train, y_train)

• Sample 20 of the grid’s 32 combinations.
• random_state makes candidate sampling reproducible.
• n_iter counts candidates, not folds or elapsed time.

Using the same lists allows a comparison within the same search space. All-list configurations are sampled without replacement. A smaller budget can miss the grid’s winner. Random search is not inherently faster per fit; fewer candidates reduce total work.

Sample distributions instead of lists

param_dist = {
    "columntransformer__countvectorizer__max_features":
        randint(100, 2001),
    "svc__C": loguniform(1e-4, 1e4),
    "svc__gamma": loguniform(1e-4, 1e4),
}

Pass param_dist as param_distributions to RandomizedSearchCV.

• randint: integer values; the upper endpoint is excluded.
• loguniform: equal-sized intervals on log scale have equal probability.

These are exactly the demo’s distribution ranges. randint includes 100 through 2000. Log-uniform assigns equal probability to tenfold intervals such as 0.01–0.1 and 0.1–1. Uniform sampling over a wide positive interval places most samples at its larger orders of magnitude. The notebook illustrates the difference with histograms. Distributions are our choices about where to search, not learned from data.

Effectiveness of random search

Nine grid evaluations and nine random evaluations; random search tries more distinct values along the important dimension.

If one dimension matters more, a grid repeats its values while varying other dimensions.

Source: Bergstra and Bengio (2012)

The horizontal dimension matters more in the illustration. Random sampling over continuous ranges can try nine distinct values there; a three-by-three grid tries only three. We generally do not know which dimensions matter most. This does not guarantee random search wins.

Your turn: Count the fits

The demo uses 32 grid candidates, five-fold CV, and the default final refit.

1. How many pipeline fits does grid search require?
2. How many does random search require with n_iter=20?
3. What do we give up by evaluating fewer candidates?

Grid: 32 × 5 + 1 = 161. Random: 20 × 5 + 1 = 101. The extra fit refits the selected pipeline on all training data. Random search leaves combinations unevaluated and may miss a useful candidate. Parallelism changes elapsed time, not the fit count.

Demo 2: Run and inspect the searches

Run Sections 2 and 3, stopping before final test evaluation.

• Inspect selected settings, scores, and timings.
• Compare the heatmaps for vocabulary sizes 100 and 400.
• Try random search with lists and distributions.

Where would you spend more computation?

Pause before Section 4. Ask students to predict which regions might be useful before interpreting the heatmaps. Discuss the uniform/log-uniform plots as they appear. The distribution search uses a different space, so its score difference does not isolate the search algorithm. Return to slides before showing test accuracy.

More trials, higher score, better model?

A validation score reflects:

• Predictive ability.
• Variation from the particular evaluation examples.

Selecting the highest score can select a candidate that benefited from favourable noise.

The winning validation score can be optimistic about new-data performance.

This selection effect is optimization bias, also called overfitting the validation set. It differs from overfitting the training examples.

Optional pizza analogy, kept off the slides: practise recipes at home (training), ask friends to taste (validation), repeatedly adjust for their ratings (selection), then serve competition judges (test). Tailoring the recipe to those friends may not generalize. Optimization bias can arise even when friends and judges come from the same population; distribution mismatch is not required.

Winning by guessing

A quiz has 10 questions, each with 4 options.

• One randomly completed answer sheet: expected accuracy 25%.
• Generate 100 random answer sheets.
• Keep the sheet that scores highest on these questions.

Will the winning sheet usually exceed 25%?

Pause for answers. The selected score usually exceeds 25%. On independent new questions, expected accuracy remains 25%. Selection improved the observed score without adding knowledge. Questions represent validation examples and answer sheets represent candidates. Adapted from the random-answer example attributed to Rodolfo Lourenzutti in Chapter 8. Real model candidates are correlated, so this analogy does not quantify bias in our search.

Final evaluation: Protect the test set

1. Make modelling decisions using training-set CV.
2. Select the final configuration and refit on all training data.
3. Evaluate once on the untouched test set.

If a test result guides another modelling decision, that set is now part of development.

An independent evaluation must also be sufficiently large and representative. A disappointing test score should be reported honestly. If it guides further development, obtain fresh held-out evaluation for the revised model. Test accuracy may be above or below CV accuracy because evaluation samples and training sizes differ; a gap alone does not measure optimization bias.

Your turn: Find two evaluation problems

A student:

1. Fits CountVectorizer on all song titles before splitting.
2. Searches 200 SVM settings.
3. Reports the winning mean_test_score as final test accuracy.

What went wrong? How would you fix it?

The vocabulary learned from held-out examples. Split raw data first and put preprocessing inside the searched pipeline. mean_test_score is a CV validation score used for selection. Evaluate the selected refitted pipeline on untouched test data after development is complete.

Demo 3: Evaluate the selected model

Run Section 4 of the notebook.

selected_name = max(searches, key=lambda name: searches[name].best_score_)
selected_search = searches[selected_name]
selected_search.score(X_test, y_test)

Why might this differ from best_score_?

The selection uses CV only. The search’s selected pipeline is already refitted, so score can evaluate it directly. Different samples, different training sizes, and selection effects can all contribute to differences. Stop here for the core demo. Do not use the displayed test result to revise the search. The notebook’s optional extension is a separate training-only experiment.

Summary

• Tune the complete pipeline within CV.
• Grid search evaluates every specified combination.
• Random search controls the candidate budget.
• Search ranges matter; inspect more than the winning score.
• Count candidates \(\times\) folds + final refit.
• Selection can make the winning CV score optimistic.
• Finish with an untouched test set.

Optional: Nested cross-validation

An outer CV loop evaluates the whole selection procedure:

1. Run a complete inner search within each outer training portion.
2. Evaluate its selected pipeline on the outer held-out portion.
3. Summarize the outer evaluation scores.

Evaluation across multiple splits, at greater computational cost.

Outer held-out data must not guide the inner search. Outer scores also require protection from repeated development decisions. This evaluates a selection procedure rather than one fixed fitted model.

Optional: Searches that adapt

• Successive halving: begin with many candidates and small resource allocations; give more resources to promising candidates.
• Bayesian optimization: use previous trial results to guide later choices.

Both require sensible search spaces and protected evaluation data.

The notebook has an optional halving extension. Early rankings with fewer examples can be noisy. Treat it as a separate training-only experiment, not an opportunity to revise choices in response to the core demo’s test score. These methods are beyond the core requirements today.

Further reading

• Chapter 8 lecture notes
• scikit-learn hyperparameter-tuning guide
• GridSearchCV API
• RandomizedSearchCV API
• Bergstra and Bengio (2012)