Deep Learning
Transformers
The architecture behind ChatGPT, Claude and Gemini. Build attention and a small Transformer step by step.
- Advanced
- About 14 hours
- 4 modules
- 13 lessons
- Free
This course is not released yet. We publish courses one at a time, each one complete. The full syllabus is below.
What you will learn
- AttentionThe Problem With Recurrence, Attention Intuition, Self-Attention Step by Step and more
- The ArchitecturePositional Encoding, The Encoder Block, The Decoder Block and more
- VariantsEncoder-Only: BERT, Decoder-Only: GPT, Encoder-Decoder: T5
- Build ItProject: A Tiny Transformer in PyTorch
Course syllabus
Not released yet · full syllabus below
- 1
Attention
5 lessons- 1The Problem With RecurrenceSoon
- 2Attention IntuitionSoon
- 3Self-Attention Step by StepSoon
- 4Queries, Keys and ValuesSoon
- 5Multi-Head AttentionSoon
- 2
The Architecture
4 lessons- 6Positional EncodingSoon
- 7The Encoder BlockSoon
- 8The Decoder BlockSoon
- 9Layer Norm and ResidualsSoon
- 3
Variants
3 lessons- 10Encoder-Only: BERTSoon
- 11Decoder-Only: GPTSoon
- 12Encoder-Decoder: T5Soon
- 4
Build It
1 lessons- 13Project: A Tiny Transformer in PyTorchSoon