Velamuvi AIAcademy
Generative AI

Tokenization

How text becomes numbers. Byte-pair encoding, why token counts matter and how to count them, with a live tokenizer.

  • Intermediate
  • About 4 hours
  • 1 modules
  • 5 lessons
  • Free

This course is not released yet. We publish courses one at a time, each one complete. The full syllabus is below.

What you will learn

  • TokensWhat Is a Token?, Byte-Pair Encoding, Special Tokens and more

Course syllabus

Not released yet · full syllabus below

  1. 1

    Tokens

    5 lessons
    1. 1What Is a Token?Soon
    2. 2Byte-Pair EncodingSoon
    3. 3Special TokensSoon
    4. 4Counting TokensSoon
    5. 5Tokenisation Across LanguagesSoon

Recommended next