Morphological Analysis

The core mechanism of breaking a word into its stem (base meaning) and affixes (prefixes, suffixes, infixes, circumfixes).

• Example: "unhappiness" → un- (negation prefix) + happy (stem) + -ness (nominalization suffix)
• Stemming uses heuristic rules to strip affixes, often producing non-words ("running" → "runn")
• Lemmatization uses vocabulary lookup and morphological rules to return the dictionary form ("running" → "run")
• Critical for languages with rich inflectional morphology like Finnish, Turkish, and Arabic

Morphological analysis distinguishes between two fundamental types of affixation:

• Inflectional morphology: Modifies a word's grammatical properties without changing its core meaning or part of speech
  • Tense: walk → walked
  • Number: cat → cats
  • Case: he → him
• Derivational morphology: Creates a new word with altered meaning or part of speech
  • Verb→Noun: teach → teacher
  • Adjective→Adverb: quick → quickly
• Search systems use this distinction to determine when to conflate terms vs. treat them as distinct concepts

The dominant computational framework for implementing morphological analyzers, particularly for languages with complex agglutinative structures.

• An FST maps between two regular languages: the surface form (actual word) and the lexical form (stem + morphological tags)
• Example mapping: cats ↔ cat +N +PL
• Bidirectional operation enables both analysis (word → morphemes) and generation (morphemes → word)
• Widely used in search engines for query expansion: a search for "running" can automatically match documents containing "ran" or "run"
• Tools like Helsinki Finite-State Technology (HFST) and Xerox Finite-State Tool (XFST) provide production-grade implementations

Heuristic, rule-based approaches that strip affixes without requiring a complete dictionary, trading linguistic accuracy for computational speed.

• Porter Stemmer: The classic English stemmer using a sequence of suffix-stripping rules in five phases
  • "generalizations" → "general" (strips -izations)
  • "running" → "run" (handles doubled consonants)
• Lancaster Stemmer: More aggressive, producing shorter stems at the cost of over-stemming
• Snowball (Porter2): A framework for building stemmers across multiple languages with improved rule notation
• Krovetz Stemmer (KSTEM): A hybrid approach that uses a dictionary to validate stems, reducing over-stemming errors
• Stemming remains widely used in information retrieval systems where recall is prioritized over precision

A more sophisticated approach that resolves a word to its lemma (canonical dictionary form) using part-of-speech tagging and vocabulary lookup.

• Pipeline steps:
  1. POS tagging: Determine the word's grammatical role ("saw" as verb vs. noun)
  2. Morphological analysis: Identify the stem and inflectional features
  3. Dictionary lookup: Map to the canonical lemma
• Example: "better" → good (comparative adjective) vs. "bettering" → better (verb)
• WordNet Lemmatizer: Uses the WordNet lexical database for English
• spaCy: Provides neural lemmatization models with context-aware disambiguation
• Essential for semantic search where exact word form should not affect retrieval relevance

A specialized morphological operation for languages like German, Dutch, and Finnish that form long compound nouns by concatenating multiple words.

• Example: German "Donaudampfschifffahrtsgesellschaftskapitän" splits into:
  • Donau (Danube) + Dampf (steam) + Schiff (ship) + Fahrt (journey) + Gesellschaft (company) + Kapitän (captain)
• Decompounding algorithms use:
  • Dictionary-based segmentation with frequency weighting
  • Statistical models trained on annotated compound corpora
  • Neural sequence-to-sequence models for low-resource languages
• Critical for search relevance: a query for "shipping company" should match documents containing the compound "Schifffahrtsgesellschaft"
• Improves machine translation by enabling translation of constituent parts when the full compound is out-of-vocabulary

A heuristic, rule-based process that crudely chops off affixes to reduce words to a base form. Unlike morphological analysis, stemming does not consider context or parts of speech, often producing non-linguistic stems.

• Algorithm: Porter Stemmer iteratively applies suffix-stripping rules
• Output: 'running' → 'run', 'happily' → 'happili'
• Trade-off: High speed and recall, but low precision; stems are not guaranteed to be valid dictionary words
• Use Case: Optimized for high-recall information retrieval where linguistic accuracy is secondary

The process of reducing a word to its canonical dictionary form (lemma) using full morphological analysis and a vocabulary lookup. It requires part-of-speech tagging to correctly handle ambiguous forms.

• Mechanism: Uses WordNet or similar lexical databases to map inflected forms to lemmas
• Output: 'better' → 'good', 'running' → 'run', 'mice' → 'mouse'
• Advantage: Always produces valid, linguistically correct words
• Computational Cost: Significantly slower than stemming due to POS tagging and dictionary traversal

The smallest meaning-bearing unit in a language, serving as the atomic building block of morphological analysis. Morphemes cannot be further divided without losing or altering their semantic content.

• Free Morphemes: Can stand alone as words ('cat', 'run', 'happy')
• Bound Morphemes: Must attach to other morphemes ('un-', '-ness', '-ed', '-s')
• Derivational: Change word class or meaning ('teach' → 'teacher')
• Inflectional: Modify grammatical function without changing class ('walk' → 'walked')

A prerequisite for accurate lemmatization that assigns grammatical categories to each token in a sentence. Morphological analysis relies on POS tags to disambiguate words with identical surface forms.

• Ambiguity Resolution: 'leaves' as NOUN (plural of leaf) vs. VERB (third-person singular of leave)
• Tag Sets: Penn Treebank (45 tags), Universal Dependencies (17 tags)
• Modern Approach: Transformer-based sequence labeling models achieve >97% accuracy
• Downstream Impact: Incorrect POS tags cascade into erroneous lemmatization and parsing

The study of how words change form to express grammatical features such as tense, number, case, and gender without altering the core lexical meaning or part of speech.

• English Examples: Plural '-s', past tense '-ed', comparative '-er', possessive ''s'
• Agglutinative Languages: Turkish can stack multiple inflectional morphemes ('evlerimden' = from my houses)
• Fusional Languages: Russian morphemes often encode multiple grammatical features simultaneously
• NLP Challenge: Morphologically rich languages require sophisticated analysis to normalize surface forms

The process of creating new words by adding affixes that change the lexical category or semantic meaning of a base form. This contrasts with inflection, which only modifies grammatical function.

• Nominalization: Verb → Noun ('decide' → 'decision', 'happy' → 'happiness')
• Negation: 'possible' → 'impossible', 'agree' → 'disagree'
• Adjectivization: Noun → Adjective ('beauty' → 'beautiful', 'danger' → 'dangerous')
• Productivity: Some derivational affixes are highly productive ('-ness'), while others are fossilized