From Linear Models to Neuron Networks

Understanding Deep Learning — and why it changed everything

CAI1000 · Module 3/4 Bridge · Supplemental

Course Context

You Already Know the Core Idea

Data examples & labels

→

ML Algorithm finds patterns

→

Model makes predictions

What you've seen so far

Supervised & unsupervised learning
Decision trees, regression, k-means
Train / test splits and evaluation
Features → predictions

What's next in Modules 5 & 6

AI that understands language
AI that recognizes images
ChatGPT, image generation, self-driving cars
Powered by deep learning

The Problem

Traditional ML Had a Hidden Bottleneck

Traditional ML: feature engineering

📋 Detect spam email?
Expert manually picks: word counts, sender domain, punctuation rate, caps ratio…
🖼️ Recognize a cat?
Expert manually defines: ear shape, whisker angles, fur texture, eye position…

🧑‍💼

The Bottleneck

Human expertise required at every step.
Doesn't scale. Doesn't generalize.

The question researchers asked:

What if instead of telling the machine what to look for…
we let it figure that out itself?

Core Concept

The Neural Network: Layers That Learn

Inspired by the brain, but a very simplified version of it.

Input layer

Raw data goes in — pixel values, word tokens, numbers. No human hand-crafting required.

Hidden layers

The network automatically discovers which patterns matter.

Output layer

A prediction: cat vs dog, spam vs not spam,translate this sentence.

How Layers Work

Each Layer Learns Something More Abstract

Imagine the network learning to recognize a human face from a photo.

Raw Pixels 255, 247, 201…

→

Edges & Lines layer 1–2

→

Shapes & Curves layer 3–5

→

Face Parts eyes, nose, mouth

→

"Person" final prediction

The same principle applies whether the input is images, text, audio, or sensor data — layers build progressively richer representations.

The Breakthrough

Neural networks existed for decades, what changed was the scale.

2012

AlexNet wins ImageNet by a landslide

1,000×

More training data available in 2012 than in 1990, thanks to the internet

GPUs

Graphics cards originally built for video games turned out to be perfect for training neural nets in parallel

Scaling Law

As you increase a neural network size (N), data set size (D), and total compute (C), it keeps getting better following a predictable power-law relationship.

From AlexNet to GPT-4

AlexNet (2012) parameters 60 million

GPT-3 (2020) parameters 175 billion

GPT-4 (est.) parameters ~1.8 trillion

Each parameter is one weight in the network, part of (M).

Comparison

ML vs. Deep Learning — Picking the Right Tool

Traditional ML: feature engineering

📊 You have a small or mid-size dataset
🔍 You need to explain the decision to stakeholders
⚡ You need fast training on a laptop
📋 Your features are already well-understood
Decision Trees Regression K-Means

Deep Learning

🗂️ You have massive amounts of data
🖼️ Your input is raw images, audio, or text
💡 Nobody knows which features matter — let it learn
🎯 You need state-of-the-art accuracy
CNNs Transformers LLMs

Coming Up

This Is the Engine Under the Hood

The next two modules will make a lot more sense now that you know what's powering them.

Module 5 — Natural Language Processing

ChatGPT, Claude, Gemini — all powered by Transformer neural networks with billions of layers and hundreds of billions of weights.

→ Pre-training: Learn from the entire internet (deep learning at scale)
→ Fine-tuning: Specialize for conversations and instructions
→ Each token you generate = one forward pass through the network

Module 6 — Computer Vision

Image recognition, object detection, medical imaging, self-driving cars — all powered by Convolutional Neural Networks (CNNs).

→ Convolutional layers scan for local patterns (the abstraction ladder in action)
→ Teachable Machine will let you train your own CNN

Deep learning is the basis for modern AI.