Data Tokenization

Tokenization is not a one-size-fits-all process; it employs distinct algorithms tailored to each data type's inherent structure.

• Text: Uses subword algorithms like Byte-Pair Encoding (BPE) or WordPiece to balance vocabulary size and sequence length.
• Audio: Typically involves converting raw waveforms into spectrograms, which are then split into fixed-time frames (e.g., 25ms windows).
• Image/Video: Employs patch-based tokenization, where an image is divided into a grid of non-overlapping patches (e.g., 16x16 pixels), each treated as a token.
• Sensor/Telemetry: Involves segmenting continuous time-series data into fixed-interval windows or event-based segments.

A core output of tokenization is a vocabulary, a finite set of all possible tokens, and a deterministic mapping from raw data to token IDs.

• The vocabulary is created from a training corpus. For text, this might be 30k to 100k unique subword units.
• Each unique token is assigned a unique integer ID. The sequence ["un", "##aff", "##able"] might map to [1256, 5432, 7891].
• This mapping allows models to process discrete symbols as numerical tensors. The size of the vocabulary is a key hyperparameter influencing model performance and memory usage.

Tokenization directly governs how data fits into a model's fixed context window, a hardware-imposed limit on the number of tokens processed at once (e.g., 128K).

• Long documents, audio files, or videos must be chunked into sequences that fit within this window.
• Chunking strategies are critical: naive splitting can sever semantic meaning. Advanced methods use semantic or sentence-aware boundaries.
• For inference, the total token count of a prompt (input + system instructions) must not exceed the context window, or truncation is required.

Effective tokenization aims to create tokens that correspond to meaningful semantic or acoustic units, not arbitrary byte splits.

• Subword tokenization ensures common words are single tokens ("cat"), while rare words are split into known subwords ("tokenization" -> "token", "##ization"). This handles out-of-vocabulary words.
• In vision, a patch containing a dog's eye is a more meaningful unit than random pixels.
• Poor tokenization that fractures semantic units (e.g., splitting "context" into "con" and "text") can degrade model understanding and increase sequence length unnecessarily.

Tokenization is the first step before the embedding layer, which converts token IDs into dense vector representations.

• Each token ID in the vocabulary corresponds to a specific row in the model's embedding matrix.
• This matrix is a trainable parameter of size [vocab_size, embedding_dimension] (e.g., [50,000, 4096]).
• The tokenized sequence [1256, 5432, 7891] is looked up in this matrix to produce a sequence of vectors that the transformer's attention mechanisms can process. The quality of the initial tokenization affects what the embedding layer can learn.

For a given model and tokenizer, tokenization is a deterministic function. It should also be losslessly reversible (detokenization) for human interpretation.

• The same input string will always produce the same sequence of token IDs.
• Detokenization reconstructs the original data format from tokens, though some information (like exact whitespace) may be lost.
• This reversibility is crucial for debugging, for tasks like text generation where the model's token outputs must be converted back to human-readable text, and for aligning model outputs with original data segments.

Byte-Pair Encoding (BPE) is the dominant algorithm for modern LLMs like GPT-4 and LLaMA. It works by iteratively merging the most frequent pairs of characters or bytes in a training corpus to build a vocabulary of subword units.

• Process: Starts with a base vocabulary of individual characters. The algorithm scans the corpus, finds the most frequent adjacent pair (e.g., 't' and 'h' becoming 'th'), merges them into a new token, and adds it to the vocabulary. This repeats until a target vocabulary size (e.g., 50,000 to 100,000 tokens) is reached.
• Benefit: Strikes a balance between word-level and character-level tokenization. It can represent common words as single tokens ('the') while decomposing rare words into meaningful subwords ('unfathomable' → 'un', 'fathom', 'able'), handling out-of-vocabulary words effectively.
• Example: The word 'playing' might be tokenized as ['play', 'ing']. The sentence 'Tokenization is foundational.' could become ['Token', 'ization', ' is', ' foundation', 'al', '.'].

Audio tokenization converts continuous sound waves into discrete symbolic representations, enabling models like Whisper and AudioLM to process speech.

• Process: Raw audio (a waveform) is first converted into a compressed, continuous representation using a neural codec (e.g., EnCodec, SoundStream). This codec uses an encoder to produce a latent sequence, which is then quantized. The quantizer maps each latent vector to the nearest entry in a learned codebook, producing a sequence of discrete token IDs.
• Two Common Approaches:
  • Speech-to-Text then Text Tokens: Models like Whisper transcribe audio to text, which is then tokenized using a standard text tokenizer (BPE).
  • Direct Audio Tokens: Models like AudioLM and Voicebox use a neural audio codec to produce discrete acoustic tokens that capture timbre, prosody, and content, which can be modeled autoregressively.
• Example: A 1-second audio clip might be represented as a sequence of 50 acoustic tokens, each corresponding to a ~20ms segment of sound.

Vision Transformers (ViTs) apply the transformer architecture to images by treating patches of an image as tokens.

• Process: An input image is divided into a grid of fixed-size, non-overlapping patches (e.g., 16x16 pixels). Each patch is linearly projected (flattened and passed through a trainable linear layer) into a patch embedding, a vector in the model's dimensionality. A learnable [CLS] token embedding is prepended to this sequence for classification tasks. Position embeddings are added to retain spatial information.
• Key Insight: This transforms a 2D image with (H x W x C) dimensions into a 1D sequence of N patch tokens, where N = (H * W) / (Patch Size²). This sequence can then be processed by a standard transformer encoder.
• Example: A 224x224 RGB image, split into 16x16 patches, yields (224/16) * (224/16) = 196 visual tokens. Each 16x16x3=768-value patch is projected into a 768-dimensional embedding vector.

Video tokenization extends image tokenization by incorporating the temporal dimension, treating a video as a sequence of frames, each frame as a sequence of patches.

• Common Strategies:
  • Uniform Frame Sampling: Select frames at a fixed interval (e.g., 1 frame per second). Each frame is tokenized independently using a ViT-like approach, and the resulting token sequences are concatenated or interleaved.
  • 3D Patches: Extract spatio-temporal volumes directly (e.g., 2x16x16 pixels covering 2 frames and a 16x16 spatial region). These 3D patches are then linearly embedded into tokens.
  • Factorized Encoding: Use separate encoders for spatial (image) and temporal (motion) features, then fuse the tokens.
• Challenge: The token sequence length grows rapidly (frames * patches per frame), requiring efficient attention mechanisms or model architectures designed for long sequences.
• Example: A 10-second video at 30fps has 300 frames. Sampling 1 fps yields 10 frames. With 196 patches per frame, this creates a sequence of 1,960 visual tokens.

Structured data, like database rows or sensor readings, is tokenized by treating each feature or time step as a discrete input.

• Tabular Data: Each column (feature) is processed based on its type.
  • Categorical Features: Converted to tokens via an embedding layer (similar to word embeddings).
  • Numerical Features: Often normalized, then either binned into discrete ranges (which become tokens) or projected directly via a linear layer into a continuous embedding.
  • The row becomes a sequence of these feature tokens, often with a special [CLS] token for aggregation.
• Time-Series Data: A sequence of readings (e.g., stock price, heart rate) is segmented into fixed-length windows. Each window can be:
  • Treated as a single multi-dimensional token.
  • Or, each time step within the window is treated as an individual token, with the feature values at that step projected into an embedding.
• Example: A patient record with features [Age=45, Diagnosis='Hypertension', HeartRate=72] is tokenized into three separate embeddings, one for each field.

Special tokens are reserved vocabulary items that provide structural and control signals to the model, crucial for formatting and task execution.

• Common Special Tokens & Their Functions:
  • [CLS] (Classification): Preprended to sequences for classification tasks; its final hidden state is used as the aggregate sequence representation.
  • [SEP] (Separator): Demarcates the end of a single sequence or separates two distinct segments (e.g., question and context in QA).
  • [PAD]: Used to bring sequences within a batch to the same length. Padding masks prevent the model from attending to these positions.
  • [MASK]: Used in masked language modeling (MLM) pre-training (e.g., BERT), where random tokens are replaced, and the model must predict the original.
  • [BOS] / [EOS] (Begin/End of Sequence): Signal the start and end of a generated sequence in autoregressive models like GPT.
• Importance: These tokens give the model meta-instructions, enabling it to understand boundaries, handle variable-length input via batching, and perform specific training objectives.

A subword tokenization algorithm that iteratively merges the most frequent pairs of characters or character sequences in a training corpus. This creates a vocabulary of variable-length subword units, effectively balancing vocabulary size with the ability to handle rare or out-of-vocabulary words.

• Core Mechanism: Starts with a base vocabulary of individual characters, then repeatedly merges the most adjacent frequent pair into a new token.
• Key Benefit: Enables open-vocabulary modeling by constructing tokens for unseen words from learned subword pieces (e.g., 'unfamiliar' → 'un', 'fam', 'iliar').
• Primary Use: Foundational for modern LLM tokenizers like those used in GPT and BERT families.

A family of text segmentation methods that decompose words into frequently occurring linguistic subunits—such as prefixes, stems, and suffixes—rather than treating whole words or individual characters as atomic units.

• Purpose: Mitigates the vocabulary explosion problem of word-level tokenization while providing more linguistic meaning than character-level tokenization.
• Common Algorithms: Includes Byte-Pair Encoding (BPE), WordPiece (used in BERT), and Unigram Language Model (used in SentencePiece).
• Output: A sequence of subword tokens that can reconstruct any word in the corpus, improving model efficiency and generalization.

A trainable neural network layer that maps discrete, high-dimensional token indices into dense, continuous, lower-dimensional vector representations. These vectors, or embeddings, are optimized during training to capture semantic and syntactic relationships between tokens.

• Function: Acts as a lookup table, where each token ID in the vocabulary is associated with a unique, tunable vector.
• Key Property: The geometric relationships in the resulting embedding space (e.g., cosine similarity) encode learned meaning (e.g., 'king' - 'man' + 'woman' ≈ 'queen').
• Downstream Impact: The quality of these initial embeddings directly influences all subsequent transformer layers in models like BERT or GPT.

A systematic approach to dividing continuous or large data streams into manageable, semantically coherent segments for processing by models with fixed context windows. This is critical for long-form text, audio, or video data.

• Objective: To preserve meaningful context boundaries (e.g., sentences, paragraphs, or scenes) while adhering to model token limits.
• Methods: Can be fixed-size (simple sliding window), semantic (split at sentence/paragraph boundaries), or recursive (hierarchical splitting).
• Challenge: Avoiding the severing of critical contextual relationships, which can degrade model performance on tasks requiring long-range coherence.

The process of temporally, spatially, or semantically synchronizing data points from different modalities so they correspond to the same real-world event or entity. This creates coherent, paired examples for multimodal model training.

• Temporal Alignment: Syncing audio waveforms with video frames or transcript timestamps.
• Semantic Alignment: Linking an image region with a descriptive text phrase (as in image-text datasets like COCO).
• Technical Implementation: Often involves dynamic time warping for sequences, bounding box annotation for spatial data, or heuristic matching based on metadata.

A binary tensor applied in sequence models (notably Transformers) to indicate which positions in a batched input sequence contain valid data and which are padding. This prevents the model's attention mechanism from attending to these irrelevant, artificial positions.

• Creation: Generated automatically when sequences in a batch are padded to a uniform length.
• Usage: The mask is applied during the attention score calculation, typically by adding a large negative value (e.g., -1e9) to the padded positions before the softmax, zeroing out their influence.
• Critical Role: Ensures computational efficiency through batching without corrupting the model's learned representations with padding noise.