Machine Learning

Machine Learning

Course: Stanford / Coursera — Andrew Ng Week: 1–11 | Started: July 1, 2019

Table of Contents

1. What is Machine Learning?
2. Types of Learning
   • Supervised Learning
   • Unsupervised Learning
3. Model Representation
4. Cost Function
5. Gradient Descent
6. Multivariate Linear Regression
7. Feature Scaling
   • Mean Normalization
   • Learning Rate Tips
   • Polynomial Regression
8. Normal Equation
9. Logistic Regression
   • Hypothesis Representation
   • Decision Boundary
   • Cost Function
   • Advanced Optimization
10. Multiclass Classification
11. Regularization
    • The Problem of Overfitting
    • Regularized Linear Regression
    • Regularized Logistic Regression
12. Neural Networks
    • Motivation
    • Model Representation
    • Forward Propagation
    • Cost Function
    • Backpropagation
    • Gradient Checking
    • Random Initialization
    • Putting It All Together
13. ML Diagnostics
    • Evaluating a Hypothesis
    • Model Selection & Cross Validation
    • Bias vs Variance
    • Learning Curves
    • Debugging & Neural Network Tips
14. Designing ML Systems
15. Support Vector Machines
    • SVM Cost Function
    • Large Margin Intuition
    • Kernels
    • When to Use SVM vs Logistic Regression
16. K-Means Clustering
17. Dimensionality Reduction & PCA
18. Anomaly Detection
    • Gaussian Distribution
    • Anomaly Detection Algorithm
    • Developing & Evaluating
19. Recommender Systems
20. Large-Scale Machine Learning
21. Application Example: Photo OCR

What is Machine Learning?

“Machine learning is a science of getting computers to learn without being explicitly programmed.” — Arthur Samuel (popularized by Andrew Ng)

Formal definition (Tom Mitchell): A computer program is said to learn from experience E with respect to some task T and some performance measure P, if its performance on T, as measured by P, improves with experience E.

Example — Playing Checkers

Types of Learning

Supervised Learning

The algorithm is given a labeled dataset — every training example has a known “right answer.” The goal is to learn a mapping from inputs to outputs.

Key characteristics:

• Labeled training data with direct feedback
• Goal: predict output for unseen inputs

Two main problem types:

Note: Classification can have more than 2 categories — e.g., 0 = benign, 1 = malignant type A, 2 = malignant type B.

Unsupervised Learning

The algorithm is given data with no labels. It must find hidden structure on its own — we don’t tell it what the “right answer” looks like.

Key characteristics:

• No labeled examples, no feedback
• Approach problems with little or no idea what results should look like

Two main techniques:

Reinforcement Learning (mentioned briefly): the algorithm learns by interacting with an environment and receiving rewards or penalties — covered separately.

Model Representation

Linear regression draws the best-fit straight line through data in a scatter plot. This line is called the regression line and is useful for making predictions.

Training Set — Housing Prices Example

Notation

How the Supervised Learning Pipeline Works

Training Set
     │
     ▼
Learning Algorithm
     │
     ▼
  h (hypothesis)
     │
  x ──► h(x) ──► ŷ (predicted output)

The hypothesis h maps input x to a predicted output ŷ.

Hypothesis Function — Univariate Linear Regression

For a single input feature, the hypothesis is:

• θ₀ — the y-intercept (bias term)
• θ₁ — the slope (weight for feature x)

This is called univariate linear regression — one variable, one feature.

Cost Function

The cost function measures how wrong the hypothesis is across all training examples. We want to choose θ₀ and θ₁ to minimize it.

Goal: Choose θ₀, θ₁ so that h(x) is as close to y as possible for all training examples (x, y).

Mean Squared Error (MSE) Cost Function

• The ½ is a convenience factor that cancels when taking the derivative
• This is also called the squared error loss function

Optimization objective:

Intuition — Simplified Example (θ₀ = 0)

Set θ₀ = 0 to simplify: h(x) = θ₁x (line through the origin). Now J is purely a function of θ₁ — a parabola.

Worked example with 3 training points: (1,1), (2,2), (3,3)

At θ₁ = 0.5:

When we plot J(θ₁) vs θ₁, we get a convex parabola with a single global minimum — this is the ideal landscape for optimization.

Visualizing with Both Parameters

When θ₀ ≠ 0, plotting J(θ₀, θ₁) gives a 3D bowl-shaped surface (convex). We can also view it as 2D contour plots — the center of the smallest contour ellipse is the minimum.

Gradient Descent

Gradient descent is the algorithm used to find the values of θ₀ and θ₁ that minimize J.

Intuition: Imagine standing on a hilly landscape and taking small steps downhill in the direction of steepest descent — eventually reaching the valley (minimum).

Algorithm

Start with some initial θ₀, θ₁ (often both = 0), then repeat until convergence:

Critical: Update θ₀ and θ₁ simultaneously using temp variables, not sequentially.

temp0 := θ₀ − α · ∂J/∂θ₀
temp1 := θ₁ − α · ∂J/∂θ₁
θ₀ := temp0
θ₁ := temp1

α (alpha) is the learning rate — controls the step size:

• Too small → very slow convergence
• Too large → may overshoot, fail to converge, or diverge

Gradient Descent for Linear Regression

After computing the partial derivatives of the MSE cost function:

Repeat until convergence:

(Update both simultaneously)

Key Properties

• “Batch” Gradient Descent — each update uses the entire training set (all m examples)
• The cost function J of linear regression is always convex (bowl-shaped) → only one global minimum, no local minima
• Starting from any initial guess and repeatedly applying these updates will make the hypothesis more and more accurate

Multivariate Linear Regression

Week 2 — extends linear regression to multiple input features (variables) to better predict the output.

Training Set — Extended Housing Example

Notation

Hypothesis Function — Multivariate

Vectorized form (using the convention x₀ = 1):

Setting x₀ = 1 (a “dummy” feature) allows the entire hypothesis to be written as a clean dot product θᵀx.

Gradient Descent for Multiple Variables

Repeat until convergence (update all θⱼ simultaneously for j = 0, 1, …, n):

Note that for j = 0, since x₀⁽ⁱ⁾ = 1, this reduces to the familiar θ₀ update.

Summary

Feature Scaling

To make gradient descent converge faster, get every feature into approximately the same range:

Without scaling, features on very different scales cause elongated contours in J — gradient descent zigzags slowly to the minimum.

Mean Normalization

Replace each feature $x_i$ with:

where $\mu_i$ is the mean and $s_i$ is the range (max − min) or standard deviation.

Example — Housing:

Learning Rate Tips

Debugging gradient descent — plot $J(\theta)$ vs. number of iterations:

• $J$ should decrease on every iteration
• If $J$ is increasing → learning rate $\alpha$ is too large (overshooting)
• If $J$ decreases very slowly → $\alpha$ too small

Choosing $\alpha$: try values like 0.001, 0.003, 0.01, 0.03, 0.1, 0.3, 1 and pick the largest one that consistently decreases $J$.

Polynomial Regression

When a straight line doesn’t fit well, combine or transform features:

• Combine features: e.g., $x_3 = x_1 \cdot x_2$ (frontage × depth = area) — still linear regression in $x_3$
• Polynomial features: $h_\theta(x) = \theta_0 + \theta_1 x + \theta_2 x^2 + \theta_3 x^3$
  • Feature scaling becomes even more important here (ranges explode)
  • Can also use $\sqrt{x}$ as a feature

Normal Equation

An analytical alternative to gradient descent that solves for $\theta$ directly (no iteration needed):

Gradient Descent vs. Normal Equation

Normal Equation & Non-invertibility

Sometimes $X^T X$ is singular (non-invertible). Causes:

1. Redundant/dependent features — e.g., $x_1$ = size in ft², $x_2$ = size in m² (linearly dependent)
2. Too many features — $m \leq n$ (more features than training examples)

Fixes: delete redundant features, use regularization, or reduce $n$.

Use pinv (pseudo-inverse) in Octave/MATLAB — it handles singular matrices gracefully.

Logistic Regression

Week 3 — The most widely used classification algorithm. Outputs a probability between 0 and 1.

Binary classification: $y \in {0, 1}$

• $y = 1$: positive class (presence of something, e.g., malignant tumor)
• $y = 0$: negative class (absence)

Why not linear regression? Predictions $h_\theta(x)$ can exceed 1 or go below 0, and a single outlier can skew the decision boundary badly.

Hypothesis Representation

• Output: $0 \leq h_\theta(x) \leq 1$
• Interpretation: $h_\theta(x) = P(y=1 \mid x;\theta)$ — probability that $y=1$ given input $x$
• Since probabilities must sum to 1: $P(y=0 \mid x;\theta) + P(y=1 \mid x;\theta) = 1$

The sigmoid function plateaus near 0 for $z \ll 0$ and near 1 for $z \gg 0$, crossing 0.5 at $z=0$.

Decision Boundary

The decision boundary is a property of the hypothesis (i.e., $\theta$), not the training set.

Prediction rule:

Since $g(z) \geq 0.5$ when $z \geq 0$:

The decision boundary is the line (or surface) $\theta^T x = 0$ that separates $y=0$ and $y=1$ regions. With higher-order polynomial features, the boundary can be non-linear (circles, ellipses, etc.).

Logistic Regression Cost Function

We can’t use MSE for logistic regression — $J(\theta)$ would be non-convex with many local minima.

Per-example cost:

Intuition: if $y=1$ and $h_\theta(x) \to 0$, cost $\to \infty$ (penalizes confident wrong predictions heavily).

Simplified (combined) form:

Full cost function:

This is convex — gradient descent converges to the global minimum.

Gradient descent update (identical form to linear regression):

Advanced Optimization

Alternatives to gradient descent that auto-select step size and are often faster:

These are more complex but generally outperform vanilla gradient descent. Use a library implementation — don’t write from scratch.

Multiclass Classification

One-vs-All (One-vs-Rest): for $y \in {1, 2, \ldots, K}$, train $K$ separate binary logistic regression classifiers:

Each classifier $h_\theta^{(i)}$ is trained to distinguish class $i$ from all other classes combined.

Prediction: given a new input $x$, pick the class with the highest probability:

Regularization

The Problem of Overfitting

Overfitting occurs when we have too many features relative to training examples — the model memorizes the training data instead of learning general patterns.

Addressing overfitting:

1. Reduce features — manually select, or use a model selection algorithm
2. Regularization — keep all features but shrink parameter magnitudes $\theta_j$

Regularization works especially well when there are many features, each contributing a little to predicting $y$.

Regularization: Cost Function

Add a penalty term that shrinks all $\theta_j$ (except $\theta_0$, the bias):

$\lambda$ (regularization parameter) controls the trade-off:

• Too small $\lambda$ → overfitting (no regularization effect)
• Too large $\lambda$ → underfitting (all $\theta_j \approx 0$, hypothesis becomes $h_\theta(x) \approx \theta_0$)

Small $\theta$ values → simpler hypothesis → less prone to overfitting.

Regularized Linear Regression

Gradient descent (note the shrinkage factor on $\theta_j$):

The factor $(1 - \alpha\lambda/m) < 1$ slightly shrinks $\theta_j$ on every step.

Normal equation with regularization:

Adding $\lambda L$ also fixes the non-invertibility issue when $m \leq n$.

Regularized Logistic Regression

Gradient descent updates are the same form as regularized linear regression (just with the logistic hypothesis).

Neural Networks

Motivation

Why not just use logistic regression with many features?

For non-linear classification with $n$ features, adding all quadratic terms gives $O(n^2/2)$ features — impractical. For computer vision (e.g., 100×100 pixel images), $n \approx 10{,}000$ pixels → millions of quadratic features. Neural networks learn complex non-linear hypotheses efficiently.

Origins: Algorithms that try to mimic how the brain works.

NN Model Representation

Biological analogy:

Artificial neuron: takes inputs, applies weights $\theta$, passes result through sigmoid activation $g(z) = \frac{1}{1+e^{-z}}$.

Network anatomy:

Notation:

• $a_i^{(l)}$ — activation of unit $i$ in layer $l$
• $\Theta^{(l)}$ — weight matrix mapping layer $l$ to layer $l+1$
• If layer $l$ has $s_l$ units and layer $l+1$ has $s_{l+1}$ units, then $\Theta^{(l)} \in \mathbb{R}^{s_{l+1} \times (s_l + 1)}$

Example — 3 input features, 1 hidden layer (3 units), 1 output:

Forward Propagation

Vectorized form: define $z^{(l)} = \Theta^{(l-1)} a^{(l-1)}$, then:

Add bias $a_0^{(l)} = 1$ before computing the next layer.

Forward propagation = computing activations layer by layer, from input → hidden → output. The network learns its own features in the hidden layers.

Multiclass output: for $K$ classes, the output layer has $K$ units and $h_\Theta(x) \in \mathbb{R}^K$:

NN Cost Function

Setup:

• $L$ = total number of layers; $s_l$ = units in layer $l$ (excluding bias)
• Binary classification: $s_L = 1$, $y \in {0,1}$
• Multi-class ($K$ classes): $s_L = K$, $y \in \mathbb{R}^K$, $K \geq 3$

Generalized cost function (regularized):

The double sum over $k$ accounts for all output nodes; the triple sum regularizes all weights (excluding bias terms).

Backpropagation

Goal: compute $\frac{\partial J}{\partial \Theta_{ij}^{(l)}}$ for all layers — needed for gradient descent (or advanced optimizers).

Key concept: $\delta_j^{(l)}$ = “error” of node $j$ in layer $l$ — how much that node’s activation contributed to the overall error.

Algorithm (for one training example):

1. Set $a^{(1)} = x$ (forward propagation through all layers to get all $a^{(l)}$)
2. Compute output error: $\delta^{(L)} = a^{(L)} - y$
3. Propagate backwards: $\delta^{(l)} = (\Theta^{(l)})^T \delta^{(l+1)} \odot g’(z^{(l)})$ for $l = L-1, \ldots, 2$
4. No $\delta^{(1)}$ — we don’t assign error to inputs
5. Accumulate: $\Delta^{(l)} := \Delta^{(l)} + \delta^{(l+1)}(a^{(l)})^T$

For the full training set, run FP then BP for each example, then average:

Gradient Checking

BP implementation is easy to get subtly wrong — it may appear to work but converge to a wrong solution. Gradient checking numerically verifies that BP is correct.

Numerical gradient approximation (two-sided difference):

For each parameter $\theta_j$:

Check that DVec ≈ gradApprox. Use $\varepsilon \approx 10^{-4}$.

Important: Disable gradient checking before training — it is very slow. Use it only to verify BP, then turn it off.

Random Initialization

Symmetry problem: if all weights start at 0, every hidden unit in a layer computes the same function → all updates are identical → network never learns distinct features.

Solution — Symmetry breaking: initialize each $\Theta_{ij}^{(l)}$ to a random value in $[-\varepsilon, \varepsilon]$:

Theta = rand(out, in+1) * 2 * eps - eps

Putting It All Together

Architecture choices:

• Default: same number of units in each hidden layer
• More units per layer → more expressive, but higher computational cost
• More hidden layers → generally better, train all with same # units per layer

Training procedure:

$J(\Theta)$ for neural networks is non-convex — gradient descent may find a local minimum. In practice, this usually works well enough.

ML Diagnostics

Machine Learning Diagnostic: a test you run to gain insight into what is/isn’t working with a learning algorithm, guiding what to try next.

What to try next when your model isn’t working:

Evaluating a Hypothesis

Split data into training set and test set (typically 70/30).

Linear regression — test set error:

Logistic regression — test set error:

Misclassification error (0/1 error):

Model Selection & Cross Validation

Problem: choosing the best model (degree $d$, feature set, $\lambda$) using test set error is optimistically biased — the model is implicitly fit to the test set.

Solution — Three-way split:

Model selection procedure:

1. Train each candidate model on the training set
2. Compute $J_{\text{CV}}(\theta)$ for each — pick the model with lowest CV error
3. Estimate generalization error using $J_{\text{test}}(\theta)$ on the held-out test set

This way the test set remains a truly unbiased estimate — never used for selection decisions.

Bias vs Variance

As polynomial degree $d$ increases, $J_\text{train}$ always decreases, but $J_\text{CV}$ first decreases then increases (U-shape):

Regularization and bias/variance:

Learning Curves

Plot $J_\text{train}$ and $J_\text{CV}$ vs. training set size $m$ to sanity-check and diagnose your algorithm.

• High bias — getting more data won’t help: both curves converge to a similarly high error; gap is small
• High variance — getting more data will likely help: $J_\text{train}$ is low, $J_\text{CV}$ is much higher; large gap that narrows as $m$ grows

Debugging & Neural Network Tips

Neural Networks — small vs. large:

Number of hidden layers: Start with 1 hidden layer; try more using a CV set — choose the best.

Designing ML Systems

Recommended Approach

1. Start with a simple algorithm that can be implemented quickly; test it on CV data.
2. Plot learning curves to decide whether more data, more features, etc. will help.
3. Error analysis: manually examine misclassified CV examples to identify patterns and inspire new features.

Large Data Sets

Large datasets are not always useful — they help only when both conditions hold:

1. Low-bias algorithm: use a model with many parameters (logistic/linear regression with many features, neural network with many hidden units) so $J_\text{train}(\theta)$ is small.
2. Large training set: makes overfitting unlikely (low variance) → $J_\text{train}(\theta) \approx J_\text{test}(\theta)$, so $J_\text{test}(\theta)$ is also small.

Key test before collecting more data:

• Can a human expert look at all the features and predict $y$ accurately?
• Can we get a large enough training set to train the algorithm with many parameters?

Support Vector Machines

SVM Cost Function

Starting from logistic regression: $h_\theta(x) = g(\theta^T x)$, $g(z) = \frac{1}{1+e^{-z}}$

For $y=1$: want $\theta^T x \gg 0$; for $y=0$: want $\theta^T x \ll 0$.

The per-example logistic cost is replaced by hinge-like costs:

SVM optimization objective:

Hypothesis:

$C$ vs. $\lambda$: $C \approx 1/\lambda$ — large $C$ → low bias / high variance; small $C$ → high bias / low variance.

Large Margin Intuition

SVMs are also called large margin classifiers — they find a decision boundary that separates positive and negative examples with the maximum possible margin (distance to the nearest training points on each side). With a very large $C$, the SVM can be sensitive to outliers; a moderate $C$ is more robust.

Kernels

Motivation: Learn complex non-linear decision boundaries by defining new features using landmarks and similarity functions.

Gaussian (RBF) Kernel:

• $f_i \approx 1$ when $x$ is close to landmark $l^{(i)}$; $f_i \approx 0$ when far
• $\sigma^2$ controls width — smaller $\sigma^2$ → sharper boundary (higher variance)
• Feature scaling is important before applying the Gaussian kernel
• Works well when $n$ is small and $m$ is large

Using an SVM (practical):

• Use off-the-shelf software (libsvm, scikit-learn, etc.)
• Specify: parameter $C$, and choice of kernel
• No kernel = linear kernel (good when $n$ is large or $m$ is small)

When to Use SVM vs Logistic Regression

Multi-class SVM: use one-vs-all method (train $K$ SVMs).

K-Means Clustering

In the clustering problem, we are given an unlabeled dataset and want an algorithm to automatically group data into coherent clusters.

Algorithm:

1. Randomly initialize $K$ cluster centroids $\mu_1, \mu_2, \ldots, \mu_K \in \mathbb{R}^n$
2. Repeat until convergence:
   • Cluster assignment step: assign each example to the nearest centroid: \(c^{(i)} := \underset{k}{\arg\min} \|x^{(i)} - \mu_k\|^2\)
   • Move centroid step: update each centroid to the mean of its assigned points: \(\mu_k := \text{average of } \{x^{(i)} : c^{(i)} = k\}\)

Random initialization:

• Choose $K < m$; randomly pick $K$ training examples and set centroids equal to them
• Run multiple times; keep the result with the lowest distortion cost

Choosing $K$:

• Manually choosing is still often the best approach
• Elbow method: plot cost $J$ vs. $K$ — look for an “elbow” where returns diminish

Dimensionality Reduction & PCA

Applications: data compression (reduce memory/storage, speed up algorithms), data visualization (project to 2D/3D).

Principal Component Analysis (PCA)

Goal: Project data onto a $k$-dimensional subspace to minimize average squared projection error.

PCA ≠ Linear Regression — PCA minimizes perpendicular projection distances, not vertical prediction errors.

Algorithm:

1. Preprocessing: mean normalize (and optionally scale) each feature $x_j$
2. Compute covariance matrix: \(\Sigma = \frac{1}{m} \sum_{i=1}^{m} x^{(i)} (x^{(i)})^T \quad (\Sigma \in \mathbb{R}^{n \times n})\)
3. SVD: $[U, S, V] = \text{svd}(\Sigma)$
4. Select top $k$ columns of $U$: $U_\text{reduce} \in \mathbb{R}^{n \times k}$
5. Project: $z^{(i)} = U_\text{reduce}^T x^{(i)} \in \mathbb{R}^k$

Reconstruction: $x_\text{approx} = U_\text{reduce} \cdot z$

Choosing $k$: pick the smallest $k$ such that ≥ 99% of variance is retained:

Using the $S$ matrix from SVD, equivalently pick smallest $k$ such that:

PCA can also speed up supervised learning — reduce $n$ features to $k$ before training.

Anomaly Detection

Gaussian Distribution

For $x \in \mathbb{R}$:

Parameter estimation from ${x^{(1)}, \ldots, x^{(m)}}$:

Anomaly Detection Algorithm

1. Choose features $x_i$ that may be indicative of anomalous behavior
2. Fit $\mu_1, \ldots, \mu_n$ and $\sigma_1^2, \ldots, \sigma_n^2$ from training data
3. For a new example $x$, compute (assuming feature independence): \(p(x) = \prod_{j=1}^{n} p(x_j;\mu_j,\sigma_j^2) = \prod_{j=1}^{n} \frac{1}{\sqrt{2\pi}\,\sigma_j} \exp\!\left(-\frac{(x_j-\mu_j)^2}{2\sigma_j^2}\right)\)
4. Flag as anomaly if $p(x) < \varepsilon$

Developing & Evaluating

Dataset split (closer to supervised learning):

Since $y=0$ dominates, use skew-aware metrics: precision/recall, F₁ score.

Anomaly Detection vs Supervised Learning:

Choosing features:

1. Transform skewed features to look Gaussian: $\log(x_i)$, $x_i^{1/c}$, etc.
2. Error analysis: inspect missed anomalies; engineer new features (e.g., CPU load / network traffic ratio)
3. Prefer features that take on unusually large or small values during anomalies

Recommender Systems

A class of algorithms for learning which features matter for user-item interactions.

Applying mean normalization per user helps when a user has rated very few items.

Large-Scale Machine Learning

The Problem

Traditional Batch Gradient Descent uses all $m$ examples per update — computationally expensive at scale (100,000+ examples).

Stochastic Gradient Descent (SGD)

Use 1 example per update. Repeat multiple passes over data:

Converges much faster; oscillates near (rather than exactly at) the minimum.

Mini-Batch Gradient Descent

Use $b$ examples per update ($b$ typically 2–100):

Online Learning

Learn from a continuous stream of incoming data — process one example, then discard:

Naturally adapts to changing user behavior. Example: predict CTR (Click-Through Rate).

Map-Reduce

For datasets too large for one machine:

1. Split training set into $P$ parts; send each to a separate machine
2. Each machine computes a partial gradient sum
3. Combine on a master server; apply update

Key insight: any algorithm expressible as a sum over training examples can be parallelized. Works across a multi-core single machine too. Hadoop is a popular Map-Reduce implementation.

Application Example: Photo OCR

ML Pipeline

Complex ML systems are built as pipelines — a sequence of modules, each solving a sub-problem:

Image → Text Detection → Character Segmentation → Character Recognition → Text Output

Each stage is an independent ML problem. Ceiling analysis identifies where to invest effort.

Sliding-Window Detection

Scan an image with a fixed-size patch; at each position apply a classifier ($y=0$ or $y=1$). Slide by a stride value, repeat. Then repeat with larger patches.

Artificial Data Synthesis

When labeled data is scarce:

1. Collect and label more data (can be costly but often worth it)
2. Amplify existing data — apply distortions to labeled examples (different fonts, audio noise, etc.)
3. Crowdsource (e.g., Amazon Mechanical Turk) — relatively inexpensive

Caution: Duplicating training examples without new variation won’t help — ensure low bias first (check learning curves) before investing heavily in more data.

Ceiling Analysis

Measure accuracy gain from “perfecting” each pipeline stage by manually supplying ground-truth outputs:

The stage with the largest accuracy jump has the highest payoff — focus engineering effort there.

Summary

Notes from Andrew Ng’s Machine Learning course (Stanford/Coursera)

Deep Dive into LLMs like ChatGPT

Notes from Andrej Karpathy’s talk. Mental models for what ChatGPT is — what it’s good at, not good at, and the sharp edges to be aware of.

Source: Deep Dive into LLMs like ChatGPT — Andrej Karpathy

How to Build ChatGPT

Training happens in sequential stages.

Stage 1: Pre-Training

Step 1 — Download and process the internet

Raw web data goes through a pipeline (e.g. FineWeb): URL filters (CommonCrawl) → text extraction → quality filters. This produces the pre-training dataset.

Step 2 — Tokenization

NNs expect a 1D sequence of symbols from a finite vocabulary. Raw text is compressed using the byte pair encoding (BPE) algorithm: repeatedly merge the most frequent adjacent symbol pair into a new token. The result is a vocabulary. Play with GPT-4 tokenization at tiktokenizer.vercel.app.

Step 3 — Training the Neural Network

The NN (a transformer) is a giant mathematical expression. Training adjusts its parameters so the probability of the next token matches the training distribution.

Visualize internals at bbycroft.net/llm.

Step 4 — Inference

Generating new tokens from the trained model. ChatGPT only does the inference part (plus the assistant layer on top). Pre-training and tokenization happen once; inference runs on every query.

The Base Model

A base model is an internet document simulator — given prefix tokens, it predicts the next token based on everything it saw during training. It is stochastic by nature.

• Information seen more often in training → remembered more reliably
• Regurgitation: reciting training data verbatim (undesirable)
• Hallucination: best-guess token prediction when the model hasn’t seen the answer

Base models are not assistants. They are glorified autocomplete. Few-shot prompting is a trick to coax assistant-like behavior from a base model.

Released base models need two things: the Python code (e.g. openai/gpt-2) and the parameter weights (just numbers). Play with open base models at app.hyperbolic.xyz.

Stage 2: Post-Training — Making an Assistant

All the heavy compute/data/cost is in pre-training. Post-training is relatively cheap and fast.

1. Humans write a dataset of conversations (question + ideal answer pairs)
2. Conversations are tokenized (similar protocol to pre-training)
3. Base model is fine-tuned on this data → Supervised Fine-Tuning (SFT)

This is called the InstructGPT paper approach. OpenAI didn’t release the data; HuggingFace OpenAssistant is an OSS equivalent.

The model now has a “human labeler persona” — it answers questions in the style of the experts who wrote the training conversations. If the prompt matches the post-training distribution, the SFT model responds. Otherwise it falls back to pre-training knowledge (the whole internet).

Today, LLMs themselves are used to generate post-training data (e.g. UltraChat), eliminating the need for humans.

LLM Physiology

Hallucinations

The model statistically emits tokens from its training distribution. For questions it doesn’t know the answer to, it confidently guesses. Mitigations:

• Teach uncertainty: train the model on examples where the answer is “I don’t know” so the relevant neurons activate on uncertainty (ref)

• External tools: model detects uncertainty, emits a special <SEARCH_WEB> token, fetches new data, adds it to the context window, then infers from the richer context. Knowledge in model parameters = vague recollection; knowledge in context window tokens = working memory.

Knowledge of Self

Without intervention, a model asked “Who are you?” gives a statistically plausible answer (often “ChatGPT by OpenAI”) because that text is prevalent in pre-training data — even if the model is something else entirely. Fix: hardcode a system prompt at conversation start, or fine-tune with identity-specific conversations.

Models Need Time to Think

Each token spends a finite amount of compute in the NN. To solve hard problems:

• Spread computation across tokens — let the model “think out loud” before answering
• Use a code interpreter — models can’t do mental arithmetic but can write and run Python
• Models are bad at spelling because they see tokens (chunks), not individual characters; the tokenizer isn’t character-aware

Summary: Three Stages

RLFT (Reinforcement Learning on Fine-Tuned models): generate many candidate answers, pick the best, train on it, repeat. All major providers do this internally; DeepSeek was the first to open-source the approach and demonstrate its “reasoning” capability. Models learn to allocate more tokens to thinking — essentially chain-of-thought emerging from RL.

• GPT-o* series = thinking/reasoning models (RLFT)
• GPT-4* series = mostly SFT post-trained models
• For factual questions → SFT models. For math/logic → reasoning models.

RLHF (Reinforcement Learning from Human Feedback): extends RL to unverifiable domains by training a “reward model” NN on human preferences, then using it as a simulator. Downside: the reward function can be gamed since it’s a NN approximation, not real human feedback.

Future Capabilities

• Multimodal: audio, video, images are just more tokens — same model handles all modalities
• Agents: models operating autonomously over longer horizons
• Test-time training: currently impossible to add knowledge after training except via the (finite) context window; multimodal use cases will push context windows larger

How to Keep Up

• lmarena.ai — human-preference model rankings (somewhat gamed, but good first pass)
• ainews newsletter — daily summaries

How to Access Models