All Courses
10 lessons

Build GPT

A working GPT, built lesson by lesson

Lessons

  1. 01

    Tokenizer (Byte Pair Encoding)

    BPE from scratch — merges, vocabs, edge cases.

    Hard
  2. 02

    Build Vocabulary

    Train a BPE vocab on real text.

    Medium
  3. 03

    Tokenization Edge Cases

    Whitespace, unicode, emoji, and the quirks of tiktoken.

    Medium
  4. 04

    GPT Data Loader

    Streaming tokens into the model efficiently.

    Medium
  5. 05

    GPT Dataset

    Context windows, next-token targets, packing.

    Medium
  6. 06

    Code GPT

    Assemble the full GPT architecture.

    Hard
  7. 07

    Train Your GPT

    AdamW, warmup, cosine decay — the real recipe.

    Hard
  8. 08

    Make GPT Talk Back

    Sampling: temperature, top-k, nucleus.

    Medium
  9. 09

    KV-Cache

    The single trick behind fast inference.

    Hard
  10. 10

    Grouped Query Attention

    Llama-style attention: memory savings without accuracy loss.

    Hard