Check-in

How are you feeling today?

• 1. Things are more or less under control!
• 2. Excited about hyperparameter optimization!
• 3. Lost 😞
• 4. Tired / Sleepy 😴
• 5. Secretly thinking of lunch 🥗

Announcements

• Important information about midterm 1
  • https://piazza.com/class/mekbcze4gyber/post/162
• HW3 was due on Monday, Sept 29th 11:59 pm.
• HW4 has been released

Recap: iClicker Logistic Regression 1

Which of the following are True?

• 1. Logistic regression can be used for binary as well as multi-class classification tasks.
     
• 2. Logistic regression computes a weighted sum of features and applies the sigmoid function.
     
• 3. The sigmoid function ensures outputs between 0 and 1, interpreted as probabilities.
     
• 4. The decision boundary in logistic regression is linear, even though the sigmoid is applied.
• 5. When the weighted sum is 0, \(\hat{p}\) = 0.5.

Recap: iClicker Logistic Regression 1

Which of the following are True?

• 1. Logistic regression coefficients always have to be positive.
     
• 2. Larger coefficients (in absolute value) indicate stronger feature influence on the prediction.
     
• 3. For \(d\) features, the decision boundary is a \(d-1\) dimensional hyperplane.
     
• 4. In sklearn, very small C value shrinks coefficients, often leading to underfitting.
     
• 5. A larger C value allows larger coefficients and a more complex model.

Recap: Logistic regression

• For a dataset with \(d\) features, the decision boundary that separates the classes is a \(d-1\) dimensional hyperplane.
  
• Complexity hyperparameter: C in sklearn.
  • Higher C \(\rightarrow\) more complex model meaning larger coefficients
  • Lower C \(\rightarrow\) less complex model meaning smaller coefficients

Data

sms_df = pd.read_csv(DATA_DIR + "spam.csv", encoding="latin-1")
sms_df = sms_df.drop(columns = ["Unnamed: 2", "Unnamed: 3", "Unnamed: 4"])
sms_df = sms_df.rename(columns={"v1": "target", "v2": "sms"})
train_df, test_df = train_test_split(sms_df, test_size=0.10, random_state=42)
X_train, y_train = train_df["sms"], train_df["target"]
X_test, y_test = test_df["sms"], test_df["target"]
train_df.head(4)

Model building

• Let’s define a pipeline

pipe_svm = make_pipeline(CountVectorizer(), SVC())

• Suppose we want to try out different hyperparameter values.

parameters = {
    "max_features": [100, 200, 400],
    "gamma": [0.01, 0.1, 1.0],
    "C": [0.01, 0.1, 1.0],
}

Hyperparameter optimization with loops

• Define a parameter space.
• Iterate through possible combinations.
• Evaluate model performance.
• What are some limitations of this approach?

sklearn methods

• sklearn provides two main methods for hyperparameter optimization
  • Grid Search
  • Random Search

Grid Search

• Covers all possible combinations from the provided grid.
• Can be parallelized easily.
• Integrates cross-validation.

Grid search example

from sklearn.model_selection import GridSearchCV

pipe_svm = make_pipeline(CountVectorizer(), SVC())

param_grid = {
    "countvectorizer__max_features": [100, 200, 400],
    "svc__gamma": [0.01, 0.1, 1.0],
    "svc__C": [0.01, 0.1, 1.0],
}
grid_search = GridSearchCV(pipe_svm, 
                  param_grid = param_grid, 
                  n_jobs=-1, 
                  return_train_score=True
                 )
grid_search.fit(X_train, y_train)
grid_search.best_score_

np.float64(0.9782606272997375)

Random Search

• More efficient than grid search when dealing with large hyperparameter spaces.
• Samples a given number of parameter settings from distributions.

Random search example

from sklearn.model_selection import RandomizedSearchCV
from scipy.stats import randint

pipe_svc = make_pipeline(CountVectorizer(), SVC())

param_dist = {
    "countvectorizer__max_features": randint(100, 2000), 
    "svc__C": uniform(0.1, 1e4),  # loguniform(1e-3, 1e3),
    "svc__gamma": loguniform(1e-5, 1e3),
}
random_search = RandomizedSearchCV(pipe_svm,                                    
                  param_distributions = param_dist, 
                  n_iter=10, 
                  n_jobs=-1, 
                  return_train_score=True)

# Carry out the search
random_search.fit(X_train, y_train)
random_search.best_score_

np.float64(0.9818506556179762)