spaCy

spaCy's tokenizer segments text into Doc objects while preserving perfect alignment with the original string. Unlike destructive tokenizers, every token retains its idx and idx_end character offsets.

• Whitespace pre-tokenization splits on spaces before applying language-specific rules
• Exception lists handle contractions (don't → do, n't) and special cases
• Prefix/suffix/infix splitting uses regex patterns to separate punctuation
• The resulting Doc object is a view over the original text, never a modified copy

spaCy executes a configurable sequence of pipeline components in a single pass over the Doc. Each component modifies or annotates the Doc, with later components able to consume predictions from earlier ones.

• Tok2Vec generates dense vector representations for tokens
• Tagger predicts part-of-speech tags and morphological features
• Parser constructs dependency trees using a transition-based algorithm
• NER identifies and labels named entities using a transition-based or span-based model
• Components can be disabled or replaced at runtime for speed-critical applications

The core Doc, Span, and Token objects are implemented in Cython with carefully managed memory layouts. This eliminates Python interpreter overhead during traversal.

• Tokens are stored as contiguous C arrays for cache-friendly iteration
• String attributes use a global StringStore with integer hashing for O(1) lookups
• Lexical attributes (is_alpha, is_stop, like_num) are pre-computed and stored as bit flags
• The result: processing speeds exceeding 100,000 tokens per second on a single CPU core

spaCy supports multiple embedding strategies within a unified API, allowing models to scale from lightweight CPU deployments to GPU-accelerated transformer architectures.

• Static word vectors (GloVe, fastText) loaded as numpy arrays into the Vocab
• Tok2Vec trains task-specific embeddings using a CNN with hash-embedded subword features
• Transformer pipeline wraps HuggingFace models (BERT, RoBERTa) with token-to-span alignment
• Switching from en_core_web_sm (12 MB) to en_core_web_trf (400 MB) requires no code changes

The Matcher and PhraseMatcher classes provide efficient pattern matching over Doc objects, operating on token attributes rather than raw strings. This enables linguistically aware extraction without regex.

• Token patterns match on any combination of text, lemma, POS, dependency label, or custom attributes
• PhraseMatcher efficiently matches large terminology lists using a trie-like Aho-Corasick automaton
• EntityRuler combines pattern matching with NER to add domain-specific entities
• Matchers operate in linear time relative to document length, suitable for high-throughput pipelines

spaCy models are self-contained Python packages with versioned dependencies, enabling reproducible deployments across environments.

• nlp.to_disk() and nlp.from_disk() serialize the full pipeline including weights, config, and vocabulary
• Models are distributed as pip-installable packages (e.g., pip install en_core_web_sm)
• The config.cfg file defines every pipeline component, training hyperparameter, and data path
• This design supports CI/CD integration and model versioning in production ML systems

spaCy's config-driven training system uses Thinc, its lightweight deep learning library, to fine-tune transformer-based or CNN-based pipelines on domain-specific corpora. The spacy train CLI reads a single .cfg file defining the entire training loop.

• Transformer support via spacy-transformers for BERT, RoBERTa, and custom models
• Data augmentation with spacy project templates for common NLP tasks
• Evaluation metrics including precision, recall, and F1 per entity type
• Model packaging with spacy package for pip-installable distribution