Unsupervised Learning

This course explores the foundations of Unsupervised Machine Learning and Representation Learning through a rigorous probabilistic and statistical lens. Students will learn how to extract patterns, model underlying distributions, and discover hidden structures in unlabeled data using advanced statistical frameworks and information theory.

Final Objectives

The goal is to provide a deep understanding of how machine learning algorithms discover structure in data without human supervision. By the end of this course, students will be able to bridge algorithmic concepts like clustering, dimensionality reduction, and self-supervised learning with their theoretical statistical roots, including density estimation, generative modeling, and mutual information maximization.

Course Content

Click on each lesson to access its detailed planning and study materials.

Part 1: Density Structures and Clustering

• Lesson 1: Data Space, Parametric Generative Models, and Anomalies

  • ML Concept: What it means to “model data” without labels. Profiling to identify atypical behaviors.
  • Statistical Concept: Empirical vs. theoretical distribution. Assuming observed data was generated by a known theoretical distribution (e.g., Multivariate Gaussian). Using the likelihood of a point under the fitted model to define statistical exclusion regions (anomaly thresholds based on probability tails).
  • Objectives: Understand how to fit theoretical distributions to unlabeled data and use likelihood for anomaly detection.
  • Expected Competencies: Ability to fit a Multivariate Gaussian to data and define robust, probability-based anomaly thresholds.

• Lesson 2: Vector Spaces, KNN, and Kernel Density Estimation (KDE)

  • ML Concept: High-dimensional nearest neighbor search (ANN) and local density estimation.
  • Statistical Concept: The curse of dimensionality and concentration of measure (analysis of \(L_1\), \(L_2\), and Cosine metrics). The core of the lesson: showing that the distance to the \(k\)-th nearest neighbor, \(d_k(x)\), is inversely proportional to the local density (\(p(x) \propto 1/d_k(x)^d\)). Transitioning from this hard count to KDE, where density is smoothed by a local Gaussian weight function.
  • Objectives: Grasp the geometric behavior of high-dimensional spaces and smooth local density estimation.
  • Expected Competencies: Ability to evaluate distance metrics in high dimensions and implement KDE to estimate underlying data distributions.

• Lesson 3: Density Topography and Graphs: Hierarchical Clustering and HDBSCAN

  • ML Concept: Identifying clusters of arbitrary shapes and isolating stochastic noise.
  • Statistical Concept: Non-parametric estimation of density level sets (\(p(x) \ge \lambda\)). Using the \(k\)-neighbor distance property (Lesson 2) as a density metric to build graph paths (Minimum Spanning Tree). The concept of statistical persistence in HDBSCAN to cut the density tree where clusters are real rather than sample fluctuations.
  • Objectives: Connect density estimation to graph theory to isolate robust, arbitrarily shaped clusters.
  • Expected Competencies: Ability to apply HDBSCAN, interpret the cluster hierarchy, and distinguish between true clusters and stochastic noise based on statistical persistence.

• Lesson 4: Gaussian Mixture Models (GMM) and the EM Algorithm

  • ML Concept: Probabilistic clustering (Soft Clustering) for overlapping sample spaces.
  • Statistical Concept: KMeans viewed as the limiting case of a GMM with identical spherical covariances and variance approaching zero. Introduction to Latent Variables (from which hidden population the point was born). The Expectation-Maximization (EM) algorithm to infer posterior probabilities (E-Step) and update parameters via weighted maximum likelihood (M-Step).
  • Objectives: Understand latent variable models and iterative probabilistic optimization via EM.
  • Expected Competencies: Ability to implement a GMM, mathematically derive the E and M steps, and explain KMeans as a constrained GMM.

• Lesson 5: Model Selection and Introduction to Variational Inference

  • ML Concept: Choosing the optimal number of components (\(K\)) and evaluating latent models.
  • Statistical Concept: The collapse of pure likelihood in complex models. Using BIC as an asymptotic approximation of Model Evidence. Introducing Variational Inference and KL Divergence as tools to bypass the intractability of true Evidence, establishing the Evidence Lower Bound (ELBO) as the definitive criterion for selecting and fitting models with hidden variables.
  • Objectives: Evaluate complex models rigorously and introduce the mathematical foundations of variational approximations.
  • Expected Competencies: Ability to use BIC for GMM selection and mathematically formulate the ELBO.

Part 2: Dimensionality Reduction and Self-Supervision

• Lesson 6: The Linear World: Traditional PCA, PPCA, and Linear Autoencoders

  • ML Concept: Linear dimensionality reduction and feature projection.
  • Statistical Concept: Maximizing projected variance via spectral decomposition of the sample covariance matrix. Probabilistic PCA (PPCA) formulated as a generative model of linear Gaussian latent variables with isotropic noise. Mathematical proof that the linear Autoencoder shares the same projection subspace as PCA.
  • Objectives: Bridge algebraic projections (PCA) with probabilistic generative models (PPCA) and neural architectures.
  • Expected Competencies: Ability to perform PCA via eigendecomposition, formulate PPCA, and prove its geometric equivalence to a linear autoencoder.

• Lesson 7: The Non-Linear World: Traditional and Variational Autoencoders (VAE)

  • ML Concept: Non-linear dimensionality reduction and deep data generation.
  • Statistical Concept: Introducing non-linear activation functions creating complex network bottlenecks. Applying Variational Inference and the ELBO (learned in Lesson 5) in the non-linear scenario (VAE): using the reparameterization trick to force the deep latent space to conform to a clean Gaussian prior, without needing to focus on algebraic derivation massacres.
  • Objectives: Extend latent variable modeling to non-linear deep networks using variational approximations.
  • Expected Competencies: Ability to explain the architecture of a VAE, apply the reparameterization trick, and understand how to optimize the ELBO using gradient descent.

• Lesson 8: Reduction for Visualization: MDS and t-SNE

  • ML Concept: Collapsing high dimensions into interpretable 2D or 3D visual maps.
  • Statistical Concept: Proximity graph optimization. The contrast between preserving raw geometric distances (Stress in MDS) and preserving probabilistic neighborhood structures (t-SNE). Using asymmetric KL Divergence combined with the heavy-tailed Student’s t-distribution to solve the crowding problem.
  • Objectives: Understand the mathematical objectives behind topological dimensionality reduction algorithms.
  • Expected Competencies: Ability to distinguish when to use MDS vs. t-SNE and explain how heavy-tailed distributions prevent crowding in low dimensions.

Part 3: Information Discovery and Large-Scale Structures

• Lesson 9: Bayesian Networks and Probabilistic Graphical Models (PGMs) for Frequent Item Mining

  • ML Concept: Association rules (Support, Confidence, Lift) in massive and sparse binary data.
  • Statistical Concept: Translating business rules into joint probabilities, conditionals, and independence tests. Using PGMs to model directed dependencies, isolating and eliminating spurious associations caused by latent confounding variables.
  • Objectives: Model complex categorical dependencies and distinguish causal structures from spurious correlations.
  • Expected Competencies: Ability to extract association rules, construct a basic Bayesian Network graph, and identify conditional independencies.

• Lesson 10: Recommendation Systems: Matrix Factorization (SVD and NMF)

  • ML Concept: Collaborative filtering and preference prediction in matrices with missing data.
  • Statistical Concept: Geometric and additive approach for hidden factors. Presenting Truncated SVD for matrix approximation and NMF (Non-Negative Matrix Factorization) focused on the statistical intuition that positivity constraints completely change the interpretation of components (additive parts vs. abstract combinations that cancel out).
  • Objectives: Formulate recommendation as an optimization problem over latent matrix factors.
  • Expected Competencies: Ability to implement collaborative filtering via SVD and explain how NMF leads to interpretable, parts-based representations.

• Lesson 11: Distribution Alignment and Domain Shift (Data Drift) Detection

  • ML Concept: Monitoring models in production and ensuring stochastic data consistency.
  • Statistical Concept: High-dimensional non-parametric hypothesis testing to evaluate whether two population samples belong to the same underlying PDF (\(P(X_{\text{train}}) \stackrel{?}{=} P(X_{\text{test}})\)). Introduction to Maximum Mean Discrepancy (MMD) using kernel tricks to compare statistical moments directly in Hilbert spaces.
  • Objectives: Rigorously detect data drift using advanced kernel-based statistical tests.
  • Expected Competencies: Ability to apply MMD to detect multivariate domain shift between training and production data streams.

• Lesson 12: Foundations of Contrastive Representation Learning

  • ML Concept: Extracting meaning and creating robust feature vectors (embeddings) from unstructured data without human supervision.
  • Statistical Concept: Optimization grounded in Information Theory. Using the InfoNCE loss to maximize the lower bound of Mutual Information between different stochastic views and perturbations of the same data object.
  • Objectives: Understand modern self-supervised learning through the lens of mutual information maximization.
  • Expected Competencies: Ability to formulate contrastive learning objectives and mathematically explain how InfoNCE approximates mutual information.