Lecture 2: From Data to a First Model

Terminology, Baselines, Decision Trees

Varada Kolhatkar

Focus on the breath!

Announcements

• Homework 1: due September 14, 11:59 pm

• Homework 2: released; due September 21, 11:59 pm

• Watch the pre-lecture videos and check pinned posts on Ed.

• Course deadlines

• iClicker Cloud link: https://join.iclicker.com/GHJB

• My OH: Tuesdays and Thursdays from 12:30 to 1 PM.

Keep logistics brief. The core sequence is designed around four student activities; allow time for the answer discussions.

Today’s big questions

• How do we formulate a prediction problem as a supervised machine learning problem?
• How do we train, predict, and evaluate models using scikit-learn?
• How does a decision tree learn rules and make predictions?

Learning outcomes: identify examples, features, targets, training, predictions, and classification error; distinguish classification from regression; use fit–predict–score; fit and visualize a DecisionTreeClassifier; distinguish learned parameters from hyperparameters; interpret depth and decision boundaries. Regression implementations and the extended glossary remain in Chapter 2.

Formulate a prediction problem

Which job might make you happy?

You’ve graduated and have several job offers. Which one would make you happiest?

You decide to look for patterns in the experiences of friends who value similar things in a job.

Image created by GPT-6 Astra

Invite two or three suggestions before showing the data. This is an invented teaching dataset, not evidence about what causes happiness. We are predicting a reported outcome, not choosing a job on someone’s behalf.

What do we need to know?

Before thinking about machine learning, let’s define the prediction problem.

• Problem: Help someone choose a job they’ll be happy in.
• What things about a job might affect how happy you are with it?
• What exactly should we predict? How could we represent it?

Frame ML as a tool for solving a problem. First clarify the decision we want to support, then define a prediction that could help.

Ask aloud: “Would this prediction actually help someone make their decision?” Predicting happiness can inform a job choice, but does not make the decision for the person.

For the target, students might suggest Happy/Unhappy, 1/0, or a numerical satisfaction score. Use their suggestions to emphasize that what the target represents matters. Encoding Happy as 1 and Unhappy as 0 does not turn the problem into regression.

The numerical satisfaction target is hypothetical; it is not in our dataset.

This sequence describes supervised ML; unsupervised tasks do not have a designated prediction target.

Toy job happiness dataset

Supportive colleagues	Salary ($)	Free coffee	Boss vegan	Happy?
0	70,000	0	1	Unhappy
1	60,000	0	0	Unhappy
1	80,000	1	0	Happy
1	110,000	0	1	Happy
1	120,000	1	0	Happy
1	150,000	1	1	Happy
0	150,000	1	0	Unhappy

Each row describes one person in a job. 1 = yes; 0 = no.

Read one row aloud. The chapter applies the same concepts to quiz grades; this lecture uses job happiness so we can inspect all seven examples. The boss_vegan column is an arbitrary candidate feature, not a claim about diet and happiness.

Recap: the goal of supervised learning

Use examples with known features \(X\) and targets \(y\) to learn a mapping function \(f\).

\[ \underbrace{X_{\text{new}}}_{\text{features}} \xrightarrow{\text{learned model } f} \underbrace{\hat{y}_{\text{new}}}_{\text{predicted targets}} \]

The goal: predict targets accurately for new examples.

The model approximates the relationship between features and targets; it need not reproduce every training label perfectly. Explain that y is a known target and y-hat is a model prediction. Now ask students to identify X and y in the job dataset.

Your turn: identify \(X\) and \(y\)

Supportive colleagues	Salary ($)	Free coffee	Boss vegan	Happy?
0	70,000	0	1	Unhappy
1	60,000	0	0	Unhappy
1	80,000	1	0	Happy

First three examples: which columns are inputs, and which is the target?

Features \(X\): the four job attributes
Target \(y\): Happy?
Example: one row
\(n = 7\) examples and \(d = 4\) features

The first row is outlined and the target column is shaded. Ask students to identify each role before revealing the labels. Features need to be available when making a prediction; the unknown answer cannot also be an input.

Training vs. Prediction

• Training: learn the mapping

  • Known features \(X\) + known targets \(y\) \(\to\) fitted model \(f\)

• Prediction: use the mapping

  • New features \(X_{\text{new}}\) \(\to\) fitted model \(f\) \(\to\) predictions \(\hat{y}_{\text{new}}\)

We need known targets to train the model, not to make predictions.

Training is also called fitting. In the job example, training uses job attributes and known happiness labels. For a new job, we supply its attributes to the fitted model; the happiness label is unknown. A prediction is the model’s proposed answer and may differ from the actual target. Known targets are also needed to evaluate prediction accuracy.

Optional reminder: unsupervised learning has no designated target—for example, grouping similar jobs by their attributes. Ask whether grouping jobs requires happy? labels; the answer is no.

Same job features, different targets

Supportive colleagues	Salary ($)	Free coffee	Boss vegan	Happy?	happiness_score
0	70,000	0	1	Unhappy	35
1	60,000	0	0	Unhappy	48
1	80,000	1	0	Happy	85

• Happy or unhappy \(\to\) predict a discrete label \(\to\) classification
• Happiness score from 0 to 100 \(\to\) predict a numerical quantity \(\to\) regression

Other regression examples: predicting temperature, house price, or travel time.

iClicker 2.1: Classification or regression?

Link: https://join.iclicker.com/GHJB

Select all regression problems.

• A. Predict whether a flight will be delayed.
• B. Predict a flight’s delay in minutes
• C. Predict a student’s percentage grade in CPSC 330.
• D. Predict whether you’ll get a seat on the bus.
• E. Predict whether you’ll hit snooze tomorrow morning.

Train, predict, and evaluate models using scikit-learn

scikit-learn

• In this part of the course we use a framework called scikit-learn.
• A popular framework for tabular data
• Easy to use.
• 67.3K stars on Github

Separate the inputs from the target

X = toy_happiness_df.drop(columns=['happy?'])
y = toy_happiness_df['happy?']

Object	Contains	Shape
X	Four feature values for each person	(7, 4)
y	One known happiness label per person	(7,)

Keep the answer out of the inputs.

Your turn: predict without the features

If you must predict the same label for everyone, which label should you choose? How many predictions will be correct?

Allow 30–45 seconds before advancing. Ask for both a label and a count. This introduces the most-frequent baseline before any library code.

A baseline gets four out of seven right

Actual	Baseline prediction	Correct?
Unhappy	Happy	No
Unhappy	Happy	No
Happy	Happy	Yes
Happy	Happy	Yes
Happy	Happy	Yes
Happy	Happy	Yes
Unhappy	Happy	No

Always predict Happy: accuracy \(= 4/7 \approx 57\%\)
Classification error: \(3/7 \approx 43\% = 1 - \text{accuracy}\)

A baseline gives us a reference point. Getting some predictions right does not establish that a model has learned anything useful from the job attributes.

Fit a baseline with scikit-learn

from sklearn.dummy import DummyClassifier

dummy = DummyClassifier(strategy='most_frequent')
dummy.fit(X, y);

fit(X, y) learns from the training data.

Here, it finds the most frequent target. This baseline ignores the feature values.

Scikit-learn calls model objects estimators. Introduce the library in the context of this task rather than opening its homepage. DummyClassifier receives X for a consistent API even though this strategy does not use its values.

Predict, then check accuracy

dummy.predict(X)

array(['Happy', 'Happy', 'Happy', 'Happy', 'Happy', 'Happy', 'Happy'],
      dtype='<U5')

print(f'Training accuracy: {dummy.score(X, y):.1%}')

Training accuracy: 57.1%

predict(X) returns predicted labels.
score(X, y) compares predictions with known targets.

For this classifier, score returns accuracy. We are scoring the data used for fitting, so this is training accuracy. We have not measured performance on new people.

Method	You provide	It does
fit(X, y)	Features and known targets	Learns the model
predict(X_new)	Features	Returns predicted targets
score(X, y)	Features and known targets	Returns a default evaluation measure

Why doesn’t predict require y?

We don’t know the answer yet!

For classifiers used here, score is accuracy. Regressors usually return R²; score does not always mean accuracy. For supervised evaluation, known targets are needed to judge the predictions. Prediction and evaluation can be separate uses of a fitted model, not mandatory consecutive steps on the same examples.

Decision trees

Can we do better than the baseline?

Supportive colleagues	Salary ($)	Free coffee	Boss vegan	Happy?
0	70,000	0	1	Unhappy
1	60,000	0	0	Unhappy
1	80,000	1	0	Happy
1	110,000	0	1	Happy
1	120,000	1	0	Happy
1	150,000	1	1	Happy
0	150,000	1	0	Unhappy

Propose one yes/no question that separates Happy from Unhappy.

Invite rules involving supportive_colleagues or salary. Unlike the baseline, these rules use feature values. A decision tree learns a hierarchy of questions from the data.

Which question is more effective?

• Visual
• Data

Question	No group	Yes group	Comments
Supportive colleagues?	0 😊, 2 ☹️	4 😊, 1 ☹️	Cleaner split
Boss vegan?	2 😊, 2 ☹️	2 😊, 1 ☹️	Mixed labels

Supportive colleagues	Salary ($)	Free coffee	Boss vegan	Happy?
0	70,000	0	1	Unhappy
1	60,000	0	0	Unhappy
1	80,000	1	0	Happy
1	110,000	0	1	Happy
1	120,000	1	0	Happy
1	150,000	1	1	Happy
0	150,000	1	0	Unhappy

Supportive colleagues gives a cleaner split.

What are we trying to learn?

• We want to learn which questions to ask and in what order.
• How many different combinations of feature-based questions could a decision tree potentially ask?

Supportive colleagues	Salary ($)	Free coffee	Boss vegan	Happy?
0	70,000	0	1	Unhappy
1	60,000	0	0	Unhappy
1	80,000	1	0	Happy
1	110,000	0	1	Happy
1	120,000	1	0	Happy
1	150,000	1	1	Happy
0	150,000	1	0	Unhappy

Decision tree Training (high level)

• Training a decision tree is a search process: we look for the “best” tree among many possible ones.
• There are different algorithms for learning trees. Check this out.
• At each step, we evaluate candidate questions using measures such as:
  • Information gain
  • Gini index
• The goal is to split the data into groups with greater certainty (more homogeneous outcomes).

For classification, measures such as Gini impurity or entropy quantify class mixture. No formulas are needed here. Numeric features do not need to be binarized: a tree can learn salary <= 75000. A leaf predicts its most frequent training class. This is a sequence of local choices; it does not guarantee the best possible whole tree.

Fit a decision tree

from sklearn.tree import DecisionTreeClassifier

model = DecisionTreeClassifier(max_depth=2, random_state=1)
model.fit(X, y);

Same fit interface. A different learning algorithm.

max_depth=2 allows at most two questions along a prediction path.

random_state makes the example reproducible. Leave the parameter/hyperparameter distinction until students have seen what this model learned.

Read the learned tree

Root: first question, Branch: answer to a question, Leaf: prediction

At each question, True goes left and False goes right. For a 0/1 feature, supportive_colleagues <= 0.5 means no supportive colleagues. Internal nodes ask questions; leaves return the majority label. Here the algorithm selected salary first, although supportive colleagues was better than boss_vegan in our earlier comparison.

Prediction

What would be the prediction for the example below using this tree ?

• supportive_colleagues = 0
• salary = 100,000
• coffee_machine = 0
• vegan_boss = 1
  

Prediction with sklearn

• supportive_colleagues = 0
• salary = 100,000
• coffee_machine = 0
• vegan_boss = 1

test_example = [[0, 100000, 0, 1]]
print("Model prediction: ", model.predict(test_example))

Model prediction:  ['Unhappy']

Does the tree beat the baseline?

model.score(X, y)

1.0

Model	Correct / total	Training accuracy
Always Happy	4 / 7	57%
Decision tree (max_depth=2)	7 / 7	100%

Yes, on the training data.
We still don’t know how well it predicts happiness for new people.

Ask students what the word training contributes to the claim. Do not introduce a full train/test workflow yet; that is the central question for Lecture 3.

iClicker Check-in

Link: https://join.iclicker.com/GHJB

How are you feeling about the workflow and decision trees?

• A. I can explain both to a neighbour.
• B. I understand the ideas but need to review the code.
• C. I understand the code but need to review how decision trees work.
• D. I’d benefit from a pause and another example.

Use the response to choose a brief pause or revisit one prediction. This is a confidence check, not a scored content question.

Parameters vs. Hyperparameters

Before fit: hyperparameters	During fit: parameters
We set max_depth=2.	The tree learns to split on salary at 75,000.
This limits the allowed tree depth.	It learns another split on supportive_colleagues at 0.5.

Tree depth: the number of edges on the longest root-to-leaf path.

Leaf predictions are also learned from the data. Hyperparameters can later be selected using validation; for now, emphasize that they are supplied before fitting. A maximum depth is a limit, not a guarantee of exactly that depth.

Decision boundary

• A decision boundary is the line, curve, or surface that separates classes.
• Points on one side \(\rightarrow\) Model predicts Class Happy
  
• Points on the other side \(\rightarrow\) Model predicts Class Unhappy

Decision boundary with max_depth=1

• Points: observed people and their actual labels
• Shaded regions: the model’s predicted class
• Decision boundary: where the predicted class changes

Decision boundary with max_depth=1

A depth-one tree is called a decision stump. A shallow tree can stop too soon. Which person does this tree get wrong?

The person earning $150,000 without supportive colleagues.

Decision boundary with max_depth=2

Contrast it with the depth-two tree, which can ask another question within the higher-salary group.

iClicker 2.3: Baselines and trees

Link: https://join.iclicker.com/GHJB

Select all true statements.

• A. Changing salary values must change a most-frequent baseline’s predictions.
• B. predict needs features; score also needs known targets.
• C. Decision-tree features must be binary.
• D. A tree predicts by following a path from root to leaf.

The same labeled training examples are assumed in A; changing features alone does not change the majority label. Ask students to explain why C is false using the salary split.

Exit ticket

A new job offers $100,000, supportive colleagues = 1, and free coffee.

1. What path and prediction does it get?
2. Name one learned parameter and one hyperparameter.
3. Does 100% training accuracy make this prediction reliable?

Allow one to two minutes individually or in pairs. The absent boss_vegan value does not prevent hand-tracing this tree, because that feature never appears in it; calling predict with the original fitted estimator would still require the full feature schema.

Next: will it work on new examples?

We can now represent a problem with \(X\) and \(y\), compare against a baseline, and fit and read a decision tree.

A model can get every training example right.
What would convince us it has learned something useful?

Lecture 3: evaluating on unseen data, overfitting, and model complexity.

More details on this material: Chapter 2: From Data to a First Model

Additional reading in Chapter 2: terminology across disciplines, prediction versus statistical inference, DummyRegressor and DecisionTreeRegressor, and further decision-boundary examples. Here inference means applying a fitted model when used operationally; statistical inference asks about population quantities or relationships and their uncertainty. Prediction does not by itself establish causation.