CPSC 330 Lecture 15: DBSCAN and hierarchical clustering

Varada Kolhatkar

Focus on the breath!

Announcements

Today’s big questions

1. When does a nearest-centre grouping fail?
2. How can density define a cluster and identify noise?
3. How can we explore nested groups with a dendrogram?
4. What evidence makes a clustering useful?

iClicker 15.1: recalling K-Means

Select all statements that are true.

• 1. With \(n\) examples and \(d\) features, K-Means learns \(k\) centres, each with \(d\) coordinates.
• 2. The parameter \(k\) plays the same role in KNN and K-Means.
• 3. Changing feature units can change a distance-based clustering.
• 4. When choosing the number of clusters, we should prefer values that produce clusters of roughly equal size.

Answers: A and C. KNN uses k neighbours for a prediction; K-Means requests k groups. Units affect distances unless preprocessing compensates. Equal group sizes are not a general clustering goal. Cluster IDs are arbitrary.

K-Means limitations

Shape of clusters

• Good for compact, roughly spherical clusters of comparable sizes.

K-Means uses nearest-centre assignments under Euclidean distance. Comparable sizes describe a favourable setting, not a requirement or an equal-size constraint. Scaling, representation, initialization, and outliers also affect the result.

When nearest-centre assignments fail

Even with the intended \(k\), K-Means can split natural groups.

Ask students where nearest-centre boundaries will cut each dataset. We supply the intended number of groups in every case, so choosing k is not the difficulty. Moons and rings require curved group boundaries; elongated groups can also be split by nearest-centre assignments. More initializations cannot remove this geometry. The original-data row is unlabelled; generating labels do not define the only useful partition.

Can we do better?

DBSCAN

• Density-Based Spatial Clustering of Applications with Noise
• Forms clusters from connected dense regions, leaving isolated points as noise.

X, y = make_moons(n_samples=200, noise=0.08, random_state=42)
dbscan = DBSCAN(eps=0.2, min_samples=5)
dbscan.fit(X)
plot_original_clustered(X, dbscan, dbscan.labels_)

DBSCAN follows connected dense regions instead of assigning points to the nearest centre. The default min_samples is 5. We did not supply the number of clusters. Later parameter comparisons use the slightly different moons dataset from the lecture notes, held fixed across those comparisons.

Two choices define “locally dense”

• eps: radius of a point’s neighbourhood.
• min_samples: minimum number of examples in that neighbourhood, including the point itself, needed to make it a core point.

We choose these parameters; the number of clusters emerges.

Their values depend on the features, scale, and distance metric.

Core, border, and noise

• Core: at least min_samples examples within eps, counting itself.
• Border: not core, but within eps of a core point.
• Noise: neither core nor within eps of any core point.

Only core points extend a cluster. Border points can join it.

Your turn: who can extend the group?

eps=0.5, min_samples=4. Why is point 5 noise even though it is near point 4?

Points 0–3 are core, 4 is border, and 5 is noise. Point 4 cannot extend the cluster to 5. The two panels show the same data with one eps-neighbourhood highlighted at a time. Point 1 has five points in its neighbourhood; point 4 has only three (1, 4, and 5). Count each centre point itself. Point 5 is within eps of border point 4, but no core point. Its distance from 4 is 0.45, safely inside eps rather than exactly on the boundary. A chain of border points cannot connect clusters.

How DBSCAN grows clusters

1. Visit an unprocessed point.
2. If core, start a cluster and add its neighbours.
3. Expand through any core neighbours.
4. Repeat until no further expansion is possible.
5. Continue with unprocessed points.

Points never reached from core points remain noise.

A point initially marked noise can later join a cluster. Border points never propagate a label. Core components are fixed for fixed data and parameters; an ambiguous border point reachable from two components can receive either label depending on processing order. Animation source: https://dashee87.github.io/data%20science/general/Clustering-with-Scikit-with-GIFs/ .

DBSCAN in scikit-learn

model = DBSCAN(eps=0.22, min_samples=6)
labels = model.fit_predict(X_moons)
# After fitting, these labels are also in model.labels_

• One label per input row; -1 means noise.
• Other integers are arbitrary group identifiers.
• Standard DBSCAN has no predict method for new points.

Adding observations can change density connections. Fitting the expanded dataset can change memberships; it is not equivalent to K-Means assigning new observations to fixed centres.

Your turn: human DBSCAN

Keep the same students and positions for every run.

• Each student is a point; distance is physical seat distance.
• Start with eps about 1 m and min_samples=4.
• Count neighbours including yourself; identify all core points first.
• Give connected core points one label, then label their border neighbours.
• Everyone else is noise. Border points do not spread labels.

Allow about 4 minutes. Use one clearly bounded group of seats, and count only participating students. Students can remain seated and signal roles by raised hands. Use row/seat IDs to record memberships. Start a new label only for a core point not already reached. If a border point touches two core components, assign it to the first component processed, without merging the components.

Repeat: change one parameter

Run	eps	min_samples	What changed?
Baseline	~1 m	4	Identify core, border, noise
Wider neighbourhood	~3 m	4	Which groups connect?
Stricter density	~1 m	15	Who remains core?

Reset labels between runs. Keep positions fixed.

Compare each changed run with baseline, not with the previous run. Choose radii appropriate to the room. The larger radius can be judged from seats rather than asking students to move or reach. Exact outcomes depend on seating density; do not promise a particular cluster count.

Predict: increase eps

Hold min_samples=6 fixed. Which setting fragments or merges the moons?

Large dots: core. Small dots: border. Grey crosses: noise.

Small neighbourhoods fragment the groups; the middle setting recovers the moons; the large setting joins them. Colours are local identifiers within each panel, not correspondence across runs. Increasing eps can grow or merge core components; the cluster count need not change monotonically because new components can also appear from noise.

Predict: increase min_samples

Hold eps=0.22 fixed. Does every parameter change alter the partition?

The first two settings retain two groups here; the stricter threshold fragments them. A stricter core requirement cannot create new core points, but cluster counts can change in either direction as components split or disappear.

How should we tune DBSCAN?

1. Choose a meaningful representation, scaling, and metric.
2. Try a small grid of eps and min_samples values.
3. Inspect cluster sizes, noise fraction, and representative examples.
4. Check whether useful interpretations persist at nearby settings.

Can we use our usual grid search or random search?

Usually there is no target y or natural held-out prediction score. GridSearchCV is not a drop-in solution for standard DBSCAN: it has no predict method or default score. Custom evaluation is possible, but internal scores alone do not establish usefulness. Almost all noise suggests trying larger eps or smaller min_samples; one large group suggests the reverse. These are diagnostic directions, not guarantees. Suitable numerical settings are representation-specific.

One density threshold may not suit every group

A small radius can discard a sparse group; a large radius can merge distinct groups.

K-Means receives K=3 and offers an alternative partition, not a guaranteed correct answer. DBSCAN uses one global density definition. Neither method is universally better.

A dense bridge can join groups

DBSCAN follows core-point connections, even when we would prefer two groups.

Contrast this with the six-point example: border points alone do not extend a cluster. Tuning cannot always reconcile our desired grouping with density connectivity. High-dimensional representations can also make distances less informative.

What if we want nested groups?

Agglomerative hierarchical clustering

Start with one cluster per example.

1. Merge the two closest clusters.
2. Recalculate which clusters are closest.
3. Repeat until all examples form one cluster.

After \(n-1\) merges, we have a hierarchy of nested groups.

Merges are not undone. We can choose a level of detail afterward.

Your turn: read the merge sequence

Find the first pair that merges. Where are those points in the scatter plot?

Points 0 and 2 merge first, then 5 and 9, then 4 and 7. Each merge reduces the cluster count by one. The input is standardized, as in the notes.

Reading a dendrogram

• Leaves: individual examples, identified by original row number.
• Connections: merges of two points or groups.
• Height: merge dissimilarity under the chosen linkage rule.

Horizontal spacing does not measure similarity. Branches can rotate.

Large vertical gaps suggest possible cuts, rather than proving a correct number of groups.

For Ward, height relates to the increase in within-cluster squared deviation, not simply the distance between individual points. Leaf order in the display differs from original row order.

What does “closest clusters” mean?

A distance metric compares points. A linkage criterion compares clusters.

Linkage	Merge the pair with the smallest…
Single	Minimum cross-cluster point distance
Complete	Maximum cross-cluster point distance
Average	Average cross-cluster point distance
Ward	Increase in total within-cluster squared deviation

Ward requires Euclidean distance and favours compact groups. It does not enforce equal sizes.

Example: single linkage, three clusters

To go from three clusters to two, which clusters would you merge?

Single linkage compares the closest pair of points across each pair of clusters.

Use the same standardized eleven-point dataset as the earlier Ward dendrogram, but recompute the hierarchy with single linkage. Ask students to identify the nearest cross-cluster pair in the scatter plot, then locate the next merge above the dashed cut in the dendrogram. Distances between cluster centres do not determine this merge.

Example: single linkage, two clusters

Points 6 and 8 are the closest cross-cluster pair: merge their two clusters.

The same hierarchy gives two clusters when we raise the cut past the next merge.

Point out which two groups have joined; colours are assigned from the new flat labels and can change between cuts. Only one close pair is needed to join two groups, even if other points are far apart. This explains why single linkage can form long chains. The following slide compares this behaviour with other linkage criteria.

Your turn: does linkage change the result?

Same data, Euclidean distance, and two clusters. Which rule keeps the long chain together?

• Single: one nearby pair can join groups \(\to\) loose, elongated clusters.
• Complete: the farthest pair determines closeness \(\to\) tighter clusters.

Pause before explaining the two panels. Single linkage joins all 25 chain points because consecutive points are close, even though the endpoints are far apart. The compact group remains separate. Complete linkage splits the chain: joining the entire chain would require a large maximum pairwise distance. Some chain points instead join the nearby compact group. Here the largest within-cluster distance is smaller under complete linkage. This illustrates a tendency, not a guarantee of round or equally sized clusters, and neither partition is inherently correct. Colours identify groups within each panel; their numerical labels are arbitrary across methods. Features use the same units and both panels use identical axis scales.

Distance and linkage are separate choices

Z = linkage(X, method="complete", metric="cosine")

• Cosine: dissimilarity between individual feature vectors.
• Complete: compare clusters using their farthest pair.

Choose the metric for the representation. Ward cannot use cosine distance.

For example, cosine can compare directions of numerical document vectors; avoid zero vectors. Single, complete, and average support other metrics. Keep this API distinction concise; extended examples are in the notes.

From the tree to flat cluster labels

Cut horizontally: each connected group below the cut becomes a cluster.

Two ways to cut the same hierarchy

# At most three clusters
labels = fcluster(Z, t=3, criterion="maxclust")

# A merge-height threshold
labels = fcluster(Z, t=0.7, criterion="distance")

Both reuse the same fitted hierarchy Z.

Changing dendrogram colours alone does not extract labels.

fcluster returns labels in original input-row order, starting at 1. maxclust requests at most t clusters, not a guarantee of exactly t when merge heights tie. The distance threshold depends on the linkage rule. The figure’s branch colours match its flat labels.

Optional: a smaller display

dendrogram(Z, truncate_mode="lastp", p=8)

• Truncation collapses branches for display.
• All observations still contribute to the fitted hierarchy.
• It does not reduce fitting cost; standard methods often need roughly quadratic memory.

Parenthesized leaf labels count observations in collapsed groups. Display limits and computational limits are separate. A full dendrogram is not required to extract flat labels.

iClicker 15.2: interpreting clusters

Select all statements that are true.

• 1. Hierarchical clustering avoids choosing random initial centres.
• 2. Truncating a dendrogram reduces the memory needed to fit the hierarchy.
• 3. DBSCAN’s cluster count emerges from its density settings.
• 4. We can combine clustering results by taking a majority vote of their numeric labels.
• 5. A high silhouette score proves that clusters capture the meaning we care about.

Answers: A and C. B confuses display with fitting. D is false because IDs can be permuted without changing memberships. E is false because distance-based separation does not establish semantic usefulness. Tied hierarchical merge choices may still require deterministic tie-breaking.

Which grouping is useful?

Your turn: compare the methods

With a neighbour: fill in the table and give one reason for each choice.

Consideration	K-Means	DBSCAN	Hierarchical
Meaning of a group	?	?	?
Main choices	?	?	?
Noise handling	?	?	?
Assign new points with predict?	?	?	?

Allow 2 minutes. Answers: nearest centre / connected dense region / nested merge sequence. Main choices: k (and initialization) / eps, min_samples, metric / metric, linkage, cut. K-Means and standard hierarchical flat cuts assign every point; DBSCAN can leave noise. Standard scikit-learn K-Means provides predict; DBSCAN and standard agglomerative clustering do not. K-Means is usually the most scalable of these; hierarchical methods often face quadratic memory. K-Means uses Euclidean geometry; DBSCAN supports other metrics; hierarchical compatibility depends on linkage.

Evaluating clusters: evidence beyond one score

• Inspect representative members and exceptions in each group.
• Check nearby parameter settings and reasonable preprocessing changes.
• Use silhouette as supporting evidence; it can undervalue curved groups.
• For DBSCAN, report the noise fraction alongside any score.

Stable, separated groups can still reflect the wrong similarity.

If computing silhouette on non-noise points, explicitly report that exclusion. At least two clusters must remain, with more points than clusters. Treating all -1 points as a single ordinary cluster is often misleading. For documents, read representative texts and noise examples. Stability alone cannot establish usefulness. External partition comparisons such as adjusted Rand index handle arbitrary label permutations; ordinary label accuracy does not.

Class demo: inspect and explain groups

Compare density settings and hierarchy cuts. For each result:

1. Describe what members share and inspect exceptions.
2. Explain what changed when the settings changed.
3. Decide whether the grouping serves the task.

Use the corresponding lecture notebook, 15_DBSCAN-hierarchical.ipynb, for parameter exploration, cut examples, and interpretation. The previous slide linked a 2025 class demo; use the current course notebook instead. No new student-facing notebook URL is assumed.

Today’s takeaways

• DBSCAN: core points extend density-connected groups; border points join; noise remains unassigned.
• Density settings: tune together and inspect noise, stability, and examples. Varying densities and bridges can cause trouble.
• Hierarchical clustering: repeated merging gives nested groups; linkage defines closeness and a cut gives flat labels.
• Representation defines similarity: useful groups require interpretation, beyond a numerical score.

Exit question

Two DBSCAN settings both produce three clusters.

One marks half the dataset as noise; the other marks almost none.

What would you inspect before choosing, and why is cluster count alone insufficient?

Look at memberships, sizes, representative examples, exceptions, noise examples, and stability across nearby settings. Relate the interpretation to the task; neither low noise nor the desired count is automatically better.

Happy Halloween