In artificial intelligence (AI), tokens are the basic units of data used for processing text. Tokenization is the process of breaking down input text into smaller chunks. Here’s how tokens are created:

1. Text Splitting: The input text is divided into meaningful units. These can be words, subwords, or even characters. For example, in GPT models, “unbelievable” may be split into [“un”, “believ”, “able”].
2. Encoding with a Vocabulary: Each token is assigned a unique ID from a predefined vocabulary. Larger models use vocabularies with tens of thousands of tokens.
3. Byte Pair Encoding (BPE): Many models use BPE or WordPiece algorithms, which combine frequent character sequences into tokens, making the process efficient and compact.
4. Embedding Representation: Tokens are mapped to high-dimensional vectors, allowing the AI model to understand their semantic meaning.
5. Contextual Processing: During inference, tokens are processed in context, enabling the model to generate coherent and meaningful responses.

Byte Pair Encoding (BPE) – A Tokenization Technique

Byte Pair Encoding (BPE) is a popular tokenization algorithm used in large language models like GPT. It balances efficiency and flexibility by breaking words into smaller, reusable units.

How BPE Works:

1. Initialize with Characters: The text is first split into individual characters (or bytes).
2. Merge Frequent Pairs: The most common adjacent pairs of characters are merged into new tokens.
3. Repeat Iteratively: This process continues until the vocabulary reaches a predefined size.
4. Efficient Representation: Rare words are split into smaller, reusable subwords, while frequent words remain whole.

Example:
The word “unhappiness” might be tokenized as:

• Initial: ["u", "n", "h", "a", "p", "p", "i", "n", "e", "s", "s"]
• After merging frequent pairs: ["un", "hap", "pi", "ness"]

Byte Pair Encoding (BPE) vs. WordPiece

Both BPE and WordPiece are tokenization algorithms used in AI language models, but they differ in their merging strategies.

Byte Pair Encoding (BPE)

• Merges the most frequent adjacent pairs of characters or tokens.
• Creates efficient, reusable subwords.
• Used in GPT models.
• Simplicity makes it faster in training and inference.

WordPiece

• Selects the pair that maximizes likelihood (based on a probabilistic model).
• Used in BERT and T5 models.
• Improves rare word representation by balancing frequency and semantic meaning.