Glossary
Preemptive Algorithmic Cybersecurity
Adversarial Machine Learning Defenses
Terms related to hardening models against deceptive inputs designed to cause misclassification. Target: Security Engineers and CTOs.
Adversarial Perturbation
A carefully crafted, often imperceptible modification to input data designed to cause a machine learning model to make an incorrect prediction.
Fast Gradient Sign Method (FGSM)
A white-box adversarial attack that generates perturbations by taking a single step in the direction of the gradient of the loss function with respect to the input, maximizing the loss under an L-infinity norm constraint.
Projected Gradient Descent (PGD) Attack
An iterative, multi-step variant of the FGSM attack that projects the adversarial example back onto an epsilon-ball around the original input at each step, representing a powerful first-order adversary.
Data Poisoning
An attack on model integrity where an adversary injects malicious samples into the training dataset to corrupt the learned model's behavior, creating a backdoor or degrading overall performance.
Model Inversion
A privacy attack that reconstructs representative features or exact samples of the private training data by exploiting a model's confidence scores or internal representations.
Membership Inference Attack
A privacy attack that determines whether a specific data record was part of a machine learning model's training set by analyzing the model's prediction outputs.
Differential Privacy (DP)
A mathematical framework that provides a provable guarantee against privacy leakage by injecting calibrated statistical noise into computations, ensuring that the output of an analysis is nearly indistinguishable whether or not any single individual's data is included.
DP-SGD
Differentially Private Stochastic Gradient Descent, the standard training algorithm that clips per-sample gradients and adds Gaussian noise to the aggregated gradient during each training step to achieve differential privacy guarantees.
Certified Robustness
A formal guarantee that a model's prediction will remain constant for any input perturbation within a mathematically proven bound, providing a lower bound on adversarial robustness.
Randomized Smoothing
A technique for building a certifiably robust classifier from any base model by adding random Gaussian noise to inputs and returning the most probable prediction under that noise distribution.
Adversarial Training
A defensive methodology that augments the training dataset with adversarial examples generated on-the-fly, forcing the model to learn a more robust decision boundary.
TRADES
TRadeoff-inspired Adversarial Defense via Surrogate-loss minimization, a training objective that explicitly balances the trade-off between standard accuracy and adversarial robustness by regularizing the model's output stability.
Gradient Masking
A brittle and often ineffective defense mechanism where a model's gradients are intentionally or unintentionally obfuscated, preventing gradient-based attacks from functioning but remaining vulnerable to black-box or transfer attacks.
Out-of-Distribution (OOD) Detection
The task of identifying inputs that are semantically or statistically different from the model's training distribution, allowing the system to abstain from making unreliable predictions.
AutoAttack
A standardized, parameter-free ensemble of diverse white-box and black-box attacks used as a reliable benchmark for evaluating a model's empirical adversarial robustness.
Model Extraction
An attack where an adversary queries a victim model's API to steal its functionality by training a functionally equivalent substitute model, violating intellectual property.
Model Watermarking
A technique for embedding a secret, verifiable identifier into a neural network's weights or behavior to prove ownership and protect intellectual property against theft.
Neural Cleanse
A backdoor defense technique that reverse-engineers potential triggers by finding the minimal perturbation required to cause misclassification for every label, then applying anomaly detection to identify compromised classes.
Secure Multi-Party Computation (SMPC)
A cryptographic protocol that allows multiple parties to jointly compute a function over their private inputs while revealing nothing but the final output, enabling privacy-preserving inference.
Homomorphic Encryption (HE)
A cryptographic scheme that enables computation directly on encrypted data, producing an encrypted result that, when decrypted, matches the result of the operations as if they were performed on the plaintext.
Trusted Execution Environment (TEE)
A hardware-enforced secure area of a main processor that guarantees the confidentiality and integrity of code and data loaded inside it, isolating sensitive model processing from the host operating system.
Federated Learning (FL)
A decentralized machine learning paradigm where a shared global model is trained across multiple client devices holding local data samples, without the raw data ever leaving the device.
Secure Aggregation
A cryptographic protocol in federated learning that allows a central server to compute the sum of model updates from multiple clients without inspecting any individual client's contribution.
Byzantine-Robust Aggregation
A class of aggregation rules in distributed learning designed to tolerate the presence of malicious or faulty nodes that send arbitrary or adversarial updates to derail the training process.
Adversarial Patch
A physically realizable, localized perturbation that can be printed and placed in a scene to cause a visual classifier to ignore the rest of the image and output a targeted misclassification.
Threat Model
A formal characterization of an adversary's capabilities, goals, and knowledge, defining the assumptions under which a system's security is evaluated.
Lp Norm
A mathematical distance metric used to constrain the magnitude of an adversarial perturbation, with common variants including L0, L2, and L-infinity norms.
RobustBench
A standardized benchmark and leaderboard for adversarial robustness that provides a curated set of model checkpoints and attack evaluations to ensure reproducible and comparable results.
Adversarial Robustness Toolbox (ART)
An open-source Python library from IBM providing a unified interface for implementing, defending against, and evaluating adversarial attacks and defenses for machine learning models.
Backdoor Trigger
A specific, secret pattern or perturbation inserted into training data that, when present at inference time, causes a poisoned model to produce a predetermined malicious output.
Data Poisoning Prevention
Terms related to protecting training pipelines from malicious data injection that corrupts model integrity. Target: ML Engineers and Data Architects.
Data Poisoning Attack
An attack vector where an adversary contaminates a model's training data to deliberately degrade performance, introduce backdoors, or skew predictions toward a malicious objective.
Backdoor Attack
A training-time attack that implants a hidden trigger-response pattern in a model, causing it to misclassify inputs containing a specific trigger while performing normally on clean data.
Label Flipping
A data poisoning technique where an attacker intentionally changes the labels of training examples to confuse the model's learned decision boundary.
Clean-Label Attack
A stealthy poisoning method where the attacker injects correctly labeled but visually perturbed training samples that cause the model to associate the perturbation with the target class.
Trigger Injection
The process of embedding a specific visual pattern, watermark, or signal into training data that later serves as the activation key for a backdoor during inference.
Data Sanitization
The defensive process of filtering, transforming, or removing suspicious or anomalous training samples to neutralize potential poisoning threats before model training begins.
Anomaly Scoring
A detection technique that assigns a numerical score to each data point based on its deviation from the expected distribution, enabling the identification of potential poisoned samples.
Spectral Signatures
A defense method that identifies poisoned data by analyzing the singular value decomposition of feature representations, revealing the latent separability of corrupted samples from clean ones.
Data Provenance
The documented chronology of a dataset's origin, transformations, and chain of custody, used to verify the trustworthiness and integrity of training data sources.
Lineage Tracking
The systematic recording of data transformations and dependencies across a pipeline, enabling forensic analysis to pinpoint the source of contamination when a model exhibits unexpected behavior.
Schema Validation
An automated gatekeeping mechanism that rejects training data that violates predefined structural rules, type constraints, or expected value ranges before ingestion.
Distributional Shift
A statistical divergence between the data a model was trained on and the data it encounters in production, which can mask or mimic the effects of a poisoning attack.
Drift Detection
The continuous monitoring of statistical properties in feature distributions to alert engineers when incoming data deviates from the training baseline, signaling potential data integrity issues.
Robust Aggregation
A class of algorithms designed to combine model updates from multiple sources while remaining resilient to a minority of malicious or corrupted contributions, such as in federated learning.
Byzantine Resilience
The property of a distributed learning system that guarantees convergence to a correct model even when an arbitrary subset of worker nodes behaves adversarially or sends arbitrary faulty updates.
Krum Aggregation
A Byzantine-resilient aggregation rule that selects the single gradient vector from a set of client updates that minimizes the sum of squared distances to its closest neighbors, ignoring outliers.
Trimmed Mean
A robust statistical aggregation technique that discards a fixed percentage of the most extreme values for each coordinate before averaging, mitigating the impact of outlier gradients.
Gradient Clipping
A defensive technique that caps the magnitude of individual gradients during training to prevent maliciously large updates from dominating the learning process and destabilizing the model.
Differential Privacy
A mathematical framework that provides a provable guarantee against information leakage by injecting calibrated noise into computations, limiting an adversary's ability to infer the presence of any single record.
Privacy Budget (Epsilon Parameter)
A quantifiable metric representing the cumulative privacy loss allowed in a differentially private system, where a lower epsilon value indicates a stronger, more restrictive privacy guarantee.
Homomorphic Encryption
A cryptographic scheme that allows computations to be performed directly on encrypted data without requiring decryption, ensuring data remains confidential even during processing.
Secure Multi-Party Computation (SMPC)
A cryptographic protocol that enables multiple parties to jointly compute a function over their private inputs while revealing nothing to each other beyond the final output.
Federated Sanitization
The application of data sanitization and anomaly detection techniques at the edge or client level in a federated network to prevent poisoned data from ever being included in local model updates.
Data Quality Score
A composite metric that quantifies the completeness, consistency, accuracy, and timeliness of a dataset, serving as a leading indicator of vulnerability to low-quality or malicious data.
Concept Drift
A specific type of distributional shift where the statistical relationship between the input features and the target variable changes over time, invalidating the original learned mapping.
Training Set Integrity
The assurance that the data used to train a model has not been subject to unauthorized modification, tampering, or corruption, maintained through cryptographic verification and access controls.
Data Versioning
The practice of creating immutable snapshots of datasets at specific points in time, enabling reproducibility of model training and forensic rollback to a clean state if poisoning is detected.
Cryptographic Hashing
A one-way function that generates a unique fixed-size fingerprint for a dataset or model artifact, allowing any subsequent modification to be instantly detected through a mismatched checksum.
Immutable Audit Logs
A tamper-proof, append-only record of all data access, transformation, and ingestion events, providing a forensic trail to identify the root cause and blast radius of a poisoning incident.
Influence Function
A robust statistical tool that quantifies the impact of removing or modifying a specific training point on a model's learned parameters, used to identify the most harmful poisoned samples.
Model Inversion Protection
Terms related to preventing attackers from reconstructing sensitive training data by querying a model. Target: Privacy Officers and Security Architects.
Model Inversion Attack
An attack that reconstructs sensitive training data or statistical features by exploiting a model's confidence scores or internal representations.
Membership Inference Attack
An attack that determines whether a specific data record was present in a model's training dataset by analyzing its output behavior.
Differential Privacy
A mathematical framework that provides provable privacy guarantees by injecting calibrated statistical noise into data or model outputs.
Privacy Budget
A finite resource quantified by the parameter epsilon that controls the total allowable privacy loss over a series of queries or computations.
Differentially Private Stochastic Gradient Descent (DP-SGD)
A training algorithm that clips per-sample gradients and adds Gaussian noise to protect the privacy of individual training records during optimization.
Homomorphic Encryption
A cryptographic scheme that allows computation directly on encrypted data without requiring decryption, producing an encrypted result that matches the plaintext computation.
Trusted Execution Environment (TEE)
A hardware-enforced secure enclave that isolates code and data from the host operating system to protect confidentiality and integrity during processing.
Federated Learning
A decentralized machine learning paradigm where a shared model is trained across multiple edge devices or servers holding local data without exchanging the raw data.
Gradient Leakage
An attack that reconstructs private training inputs from publicly shared model gradients during distributed or collaborative learning.
Private Aggregation of Teacher Ensembles (PATE)
A framework that transfers knowledge from an ensemble of teacher models trained on disjoint data to a student model via noisy voting to guarantee differential privacy.
Knowledge Distillation
A compression technique where a smaller student model is trained to replicate the softened output distribution of a larger, pre-trained teacher model.
Synthetic Data Generation
The process of creating artificial datasets using statistical models or generative neural networks that mimic the statistical properties of real data without exposing actual records.
Confidence Score Masking
A defense mechanism that truncates or suppresses the full prediction vector returned by an API to prevent attackers from exploiting fine-grained confidence values.
Query Auditing
A security process that logs and analyzes incoming inference requests to detect and block suspicious query patterns indicative of extraction or inversion attacks.
Information Bottleneck
A training objective that compresses input representations to retain only the mutual information necessary for the prediction task, naturally limiting data leakage.
Secure Multi-Party Computation (SMPC)
A cryptographic protocol that allows multiple parties to jointly compute a function over their private inputs while keeping those inputs completely hidden from one another.
K-Anonymity
A data privacy property ensuring that each released record is indistinguishable from at least k-1 other records with respect to quasi-identifiers.
Training Data Extraction
An attack that verbatim recovers specific sequences or images from a model's training set by exploiting unintended memorization in large generative models.
Gradient Clipping
A technique that bounds the L2 norm of individual sample gradients during training to limit the influence of any single data point and bound sensitivity.
Secure Aggregation
A protocol in federated learning that computes the sum of model updates from multiple clients without revealing any individual client's contribution to the central server.
Privacy-Utility Trade-off
The fundamental balancing act between the strength of a privacy guarantee and the resulting accuracy or performance degradation of the machine learning model.
Defensive Distillation
A training strategy that uses soft labels generated by a first model to train a second model, smoothing the decision surface to resist adversarial and inversion attacks.
Overfitting
A modeling error where a neural network memorizes noise and specific details of the training data rather than learning generalizable patterns, increasing vulnerability to privacy attacks.
Data Minimization
A privacy engineering principle dictating that only the minimum amount of personal data necessary for a specific purpose should be collected and processed.
Pseudonymization
A data management procedure that replaces direct identifiers with artificial pseudonyms, reducing linkability while preserving data utility for analysis.
Prediction Vector Truncation
A defense that returns only the top-k predicted classes instead of the full probability distribution to reduce the information leakage available to inversion attacks.
Attribute Inference
An attack that infers sensitive demographic or personal attributes about individuals in the training data by analyzing a model's statistical outputs.
Differential Privacy in Federated Learning (DP-FedAvg)
The integration of differential privacy mechanisms into the federated averaging algorithm to protect client data from both the server and external observers.
Privacy by Design
A systems engineering framework that embeds privacy controls into the architecture and lifecycle of a technology from the initial design phase rather than as a retrofit.
Memorization
The phenomenon where a neural network encodes exact copies of training data within its weights, making it possible for attackers to extract rare or unique sequences.
Membership Inference Defense
Terms related to blocking attacks that determine if a specific record was used in a model's training set. Target: Compliance Officers and ML Engineers.
Membership Inference Attack
A privacy attack that determines whether a specific data record was part of a machine learning model's training dataset by analyzing the model's outputs.
Differential Privacy
A mathematical framework that provides provable privacy guarantees by injecting calibrated statistical noise into computations, ensuring the output does not reveal the presence of any single individual in the dataset.
Differentially Private Stochastic Gradient Descent (DP-SGD)
A training algorithm that modifies standard stochastic gradient descent by clipping per-sample gradients and adding Gaussian noise to limit the influence of individual training examples and provide differential privacy guarantees.
Privacy Budget (Epsilon Budget)
A quantifiable limit on the total privacy loss permitted over a series of differentially private queries or training steps, parameterized by the privacy loss parameter epsilon.
Privacy Loss Distribution
A probabilistic characterization of the divergence between the output distributions of a mechanism run on two adjacent datasets, used for precise privacy accounting.
Rényi Differential Privacy
A relaxation of standard differential privacy based on Rényi divergence that provides tighter composition bounds and is commonly used for tracking privacy loss in DP-SGD.
Privacy Accounting
The process of tracking the cumulative privacy loss expenditure over multiple operations to ensure the total loss remains within a predefined privacy budget.
Moments Accountant
A specific privacy accounting technique that tracks the moment-generating function of the privacy loss random variable to compute tight bounds on the overall privacy cost of a composed mechanism.
Subsampling Amplification
A privacy amplification phenomenon where randomly sampling a subset of data before applying a differentially private mechanism provides a stronger overall privacy guarantee than processing the full dataset.
Shadow Model Training
An attack methodology where an adversary trains multiple local models on datasets synthetically generated from the target model's output distribution to train a classifier that distinguishes members from non-members.
Attack Model
A binary classifier trained on the outputs of shadow models to infer whether a given input record was present in the training set of the target model under attack.
Per-Sample Gradient Clipping
A technique in DP-SGD that bounds the L2 norm of the gradient computed for each individual training example to limit the maximum influence any single record can have on the model update.
Gaussian Mechanism
A fundamental differential privacy mechanism that achieves privacy by adding noise drawn from a Gaussian distribution calibrated to the L2 sensitivity of the query function.
Laplace Mechanism
A fundamental differential privacy mechanism that achieves privacy by adding noise drawn from a Laplace distribution calibrated to the L1 sensitivity of the query function.
Sensitivity Analysis
The determination of the maximum change in a function's output caused by adding or removing a single record from the input dataset, a critical parameter for calibrating noise in differential privacy.
Exposure Metric
A quantitative measure of the degree to which a model has memorized a specific secret or training data point, often computed using canary insertion and likelihood ratio tests.
Canary Gradient
A technique for auditing unintended memorization where a unique, random string is inserted into a training dataset, and the model's gradient with respect to that string is analyzed to detect exposure.
Influence Function
A robust statistical method that approximates the effect of removing a specific training point on a model's learned parameters or predictions, used to identify highly memorized examples.
Black-Box Attack
A membership inference attack variant where the adversary can only query the target model and observe its final output predictions or confidence scores without access to internal parameters.
White-Box Attack
A membership inference attack variant where the adversary has full access to the target model's internal architecture, parameters, and gradients, enabling more powerful privacy violation tests.
Label-Only Attack
A membership inference attack that requires only the predicted class label from the model, exploiting the observation that models are often more confident and robust to perturbations on training data.
Gap Attack
A membership inference technique that measures the difference between a model's confidence on a true label and its next highest confidence to distinguish between training and non-training data.
Likelihood Ratio Attack
A sophisticated membership inference method that uses likelihood ratio tests on model output distributions, often leveraging reference models trained on population data to compute a calibrated membership score.
Overfitting Detection
The process of identifying when a model has memorized specific training examples rather than learning generalizable patterns, a primary vulnerability exploited by membership inference attacks.
Memorization Score
A metric quantifying the extent to which a model has encoded verbatim or near-verbatim training data, often measured by comparing generation likelihoods under the model versus a reference distribution.
Training Data Extraction
A related privacy attack that goes beyond membership inference to actively reconstruct verbatim text strings or images from a model's training set by prompting or querying the model.
Model Inversion
An attack that reconstructs representative features or prototypes of a specific class from the training data by inverting the model's learned mapping from inputs to outputs.
Attribute Inference
A privacy attack that infers sensitive attributes of individuals in the training data that are correlated with the model's learned features, even if the attribute was not a direct input feature.
Output Perturbation
A defense mechanism that adds calibrated noise directly to a model's output predictions or confidence vectors to mask the subtle statistical differences exploited by membership inference attacks.
Confidence Masking
A defense that truncates or rounds the model's output confidence scores to a limited precision or only reveals the top-K predictions, reducing the information leakage exploited by membership inference.
Differential Privacy Implementation
Terms related to the mathematical frameworks that inject calibrated noise to provide provable privacy guarantees. Target: Cryptographers and Data Scientists.
Epsilon-Differential Privacy (ε-DP)
A mathematical definition of privacy that bounds the probability of any output differing by more than a factor of e^ε between two neighboring datasets, providing a quantifiable privacy guarantee.
(ε, δ)-Differential Privacy
An approximate relaxation of pure differential privacy that allows a small failure probability δ, enabling more efficient mechanisms like the Gaussian mechanism while maintaining strong privacy guarantees.
Privacy Budget
The finite, quantifiable limit on total privacy loss allocated across multiple queries or analyses on a sensitive dataset, after which further access must be denied to prevent reconstruction.
Sensitivity
The maximum change in a query's output caused by adding or removing a single record from the dataset, which directly determines the amount of noise required to achieve differential privacy.
Laplace Mechanism
The foundational differential privacy mechanism that adds noise drawn from a Laplace distribution calibrated to the L1 sensitivity of a query to achieve pure ε-differential privacy.
Gaussian Mechanism
A differential privacy mechanism that adds noise drawn from a Gaussian distribution calibrated to the L2 sensitivity, typically used to achieve approximate (ε, δ)-differential privacy.
Rényi Differential Privacy (RDP)
A relaxation of differential privacy based on the Rényi divergence that provides tighter composition bounds for tracking cumulative privacy loss across iterative algorithms like DP-SGD.
Privacy Amplification by Subsampling
The phenomenon where randomly sampling a subset of data before applying a differentially private mechanism results in a stronger overall privacy guarantee than processing the full dataset.
Composition Theorem
A formal rule that quantifies how the total privacy budget degrades when multiple differentially private mechanisms are applied sequentially or in parallel to the same dataset.
Moments Accountant
A privacy accounting technique that tracks higher-order moments of the privacy loss random variable to provide significantly tighter bounds on cumulative privacy loss during training.
Gradient Clipping
A preprocessing step in DP-SGD that bounds the L2 norm of individual per-example gradients to a fixed threshold, limiting the sensitivity of the training update before noise is added.
DP-SGD
Differentially Private Stochastic Gradient Descent, the core training algorithm that clips per-example gradients and injects calibrated Gaussian noise to train deep learning models with provable privacy guarantees.
Local Differential Privacy (LDP)
A trust model where noise is added to data on the user's device before collection, ensuring the central aggregator never sees raw records and privacy is protected against a compromised server.
Randomized Response
A classic survey technique and the simplest local differential privacy mechanism where a user flips a biased coin to decide whether to answer a sensitive question truthfully or randomly.
Private Aggregation of Teacher Ensembles (PATE)
A knowledge distillation framework where an ensemble of teacher models trained on disjoint data partitions votes on labels, and noisy aggregation protects the privacy of the training data.
Differentially Private Federated Learning
The integration of differential privacy mechanisms into federated averaging to protect individual client contributions from being inferred from the aggregated global model updates.
Post-Processing Immunity
A resilience property of differential privacy guaranteeing that any arbitrary computation applied to the output of a differentially private mechanism cannot weaken the privacy guarantee.
User-Level Privacy
A stronger privacy granularity that protects all records belonging to a single user, ensuring the presence or absence of an individual's entire contribution in the dataset remains indistinguishable.
Differentially Private Synthetic Data
Artificially generated data produced by a model trained with differential privacy, designed to preserve the statistical properties of the sensitive source data without exposing individual records.
Differentially Private Generative Adversarial Network (DP-GAN)
A generative adversarial network trained with differential privacy constraints on the discriminator's gradients, enabling the generation of realistic synthetic data with formal privacy guarantees.
Differentially Private Heavy Hitters
Algorithms for identifying the most frequent items in a dataset under differential privacy, crucial for discovering popular trends in sensitive streams without revealing individual contributions.
Differentially Private Empirical Risk Minimization (DP-ERM)
The framework for training machine learning models by minimizing a loss function under differential privacy constraints, typically via objective perturbation, output perturbation, or gradient perturbation.
Differentially Private A/B Testing
The application of differential privacy to controlled online experiments, allowing analysts to compute test statistics and confidence intervals while protecting individual user behavior in the control and treatment groups.
Differentially Private Recommendation Systems
Collaborative filtering and matrix factorization algorithms adapted with noise injection to learn user preferences and item embeddings without memorizing individual rating or interaction histories.
Differentially Private Natural Language Processing
The application of DP-SGD and related mechanisms to text models, enabling fine-tuning of language models on sensitive text corpora while preventing memorization of specific documents or phrases.
Differentially Private Computer Vision
The adaptation of image classification, object detection, and segmentation models to train with differential privacy, protecting the visual features of individuals in sensitive image datasets.
Differentially Private Reinforcement Learning
The integration of differential privacy into Q-learning and policy gradient methods to protect the privacy of reward signals and state transitions experienced by an agent during exploration.
Differentially Private Statistical Disclosure Control
The broader field of applying differential privacy and related techniques to protect official statistics, census data, and survey releases from revealing information about specific respondents.
Differentially Private Genomic Data
Specialized protocols for releasing summary statistics, performing genome-wide association studies, and sharing biomedical data with differential privacy to protect against re-identification of genetic information.
Differentially Private Data Publishing
The one-time release of a sanitized dataset or statistical summary with a fixed privacy budget, designed to enable downstream analysis by untrusted third parties without compromising individual privacy.
Confidential AI Computing
Terms related to hardware-enforced Trusted Execution Environments that isolate model processing from the host OS. Target: Infrastructure Architects and CISOs.
Confidential Computing
A hardware-based security paradigm that protects data in use by performing computation within a hardware-enforced Trusted Execution Environment (TEE), isolating it from the host operating system, hypervisor, and other privileged processes.
Trusted Execution Environment (TEE)
A secure area within a main processor that guarantees the confidentiality and integrity of code and data loaded inside it, protecting against unauthorized access even from a compromised operating system.
Intel SGX
A set of security-related instruction codes built into Intel processors that allow user-level code to allocate private regions of memory, called enclaves, which are protected from processes running at higher privilege levels.
AMD SEV
A hardware feature embedded in AMD processors that encrypts the memory of a virtual machine with a unique key, isolating it from the hypervisor and other VMs to protect data in use.
AWS Nitro Enclaves
An isolated compute environment on Amazon EC2 instances that provides a hardened and highly constrained environment for processing sensitive data, with no persistent storage or external network access by default.
Enclave
A private, isolated execution region in memory that protects code and data from being read or modified by any process outside the enclave itself, including the operating system.
Attestation
The process by which a Trusted Execution Environment generates a cryptographically signed report to prove its identity, integrity, and that it is running specific code on a genuine hardware platform to a remote relying party.
Remote Attestation
A mechanism that enables a remote client to verify the integrity and authenticity of a software stack running inside a TEE on an untrusted host, establishing a hardware-rooted chain of trust.
Trusted Computing Base (TCB)
The set of all hardware, firmware, and software components that are critical to a system's security, where a single vulnerability or misconfiguration could compromise the entire system's security guarantees.
Side-Channel Attack
A security exploit that gathers information from the physical implementation of a computer system, such as timing information, power consumption, or electromagnetic leaks, rather than weaknesses in the implemented algorithm itself.
Memory Encryption
A hardware mechanism that transparently encrypts and decrypts data as it moves between the processor and main memory, preventing physical attackers from reading sensitive data via DRAM probing or cold boot attacks.
Hardware Root of Trust
A physically immutable, inherently trusted source within a computing platform that serves as the foundation for all subsequent security operations, such as secure boot and cryptographic key generation.
Hardware Security Module (HSM)
A dedicated physical computing device that safeguards and manages digital keys for strong authentication and provides cryptoprocessing, designed to be tamper-resistant and certified to high security standards.
Sealing
A TEE-specific operation that encrypts data and binds it to the specific enclave identity and platform that generated it, ensuring the data can only be decrypted by the exact same enclave on the exact same hardware.
Data-in-Use Protection
The practice of securing data while it is actively being processed by the CPU, as opposed to data-at-rest on a disk or data-in-transit across a network, typically achieved through confidential computing.
Confidential AI
The application of confidential computing principles to artificial intelligence workloads, ensuring that model weights, training data, and inference queries remain encrypted and isolated during computation.
Confidential Virtual Machine
A virtual machine instance whose memory is encrypted with a hardware-generated key, making it inaccessible to the cloud provider's hypervisor and host operating system.
Confidential Container
A containerized application that runs inside a hardware-backed Trusted Execution Environment, combining the agility of containers with the strong isolation guarantees of confidential computing.
Secure Boot
A security standard that ensures a device boots using only software that is trusted by the Original Equipment Manufacturer, verifying the digital signature of each piece of boot software against a database of authorized keys.
Measured Boot
A process that cryptographically measures and logs each component of the boot chain into a Platform Configuration Register (PCR), allowing a remote party to verify the exact state of the booted system.
Virtual TPM (vTPM)
A software-based representation of a physical Trusted Platform Module that provides cryptographic functions and secure storage for virtual machines, enabling measured boot and key management in virtualized environments.
Model Provenance
The verifiable record of the origin, training methodology, and transformation history of a machine learning model, ensuring its integrity and authenticity throughout its lifecycle.
Supply Chain Attestation
The process of cryptographically verifying the integrity and origin of every software component and dependency in a build pipeline, from source code to the final deployed artifact.
Code Transparency
A security property that allows a relying party to verify that the code running inside a TEE is exactly the code they expect, typically achieved by publishing a hash of the code and including it in the attestation report.
Secure Provisioning
The process of securely injecting secrets, configuration data, and cryptographic keys into a Trusted Execution Environment only after its identity and integrity have been verified through remote attestation.
Key Management Service (KMS)
A centralized cloud service that manages the lifecycle of cryptographic keys, integrating with hardware security modules to securely generate, store, rotate, and audit access to keys.
Chain of Trust
A hierarchical sequence of validation where each component in a system is authenticated by the preceding component, beginning with an immutable hardware root of trust and extending to the application layer.
Enclave TLS
A standard Transport Layer Security protocol that terminates the encrypted channel inside a TEE, ensuring that data is decrypted only within the protected memory region and not in the host operating system.
Confidential Orchestration
The automated management and scheduling of confidential containers and VMs across a cluster, ensuring that workloads requiring TEE protection are placed on nodes with the appropriate hardware capabilities.
Zero-Knowledge Proof
A cryptographic method by which one party can prove to another that a statement is true without revealing any information beyond the validity of the statement itself.
Federated Learning Security
Terms related to securing decentralized training against gradient leakage and malicious nodes. Target: Distributed Systems Engineers.
Gradient Leakage
An attack reconstructing private training data from publicly shared model gradients in distributed learning.
Secure Aggregation
A cryptographic protocol that allows a server to compute the sum of model updates from multiple clients without inspecting individual contributions.
Differential Privacy
A mathematical framework that injects calibrated statistical noise into queries or outputs to provide provable guarantees against the leakage of individual records.
Homomorphic Encryption
A cryptographic scheme enabling computation directly on encrypted data without requiring decryption, producing an encrypted result that matches the computation on plaintext.
Secure Multi-Party Computation (SMPC)
A cryptographic protocol enabling multiple parties to jointly compute a function over their private inputs while keeping those inputs mutually secret.
Byzantine Fault Tolerance
The resilience of a distributed system to arbitrary node failures or malicious actors sending conflicting information to corrupt the consensus.
Model Poisoning
An attack where a malicious participant manipulates local model updates to corrupt the global model's performance or introduce a backdoor.
Data Poisoning
An attack that injects maliciously crafted samples into the training dataset to compromise the integrity of the resulting machine learning model.
Backdoor Attack
A targeted attack embedding a hidden trigger pattern into a model during training, causing misclassification only when the specific trigger is present at inference.
Gradient Inversion
An iterative optimization technique that reconstructs high-fidelity private input data from the corresponding gradients of a neural network.
Trusted Execution Environment (TEE)
A hardware-enforced isolated enclave within a processor that protects code and data from the host operating system and other applications.
Confidential Computing
A hardware-based security paradigm that protects data in use by performing computation within a Trusted Execution Environment inaccessible to the cloud provider.
Zero-Knowledge Proof (ZKP)
A cryptographic method allowing one party to prove to another that a statement is true without revealing any information beyond the validity of the statement itself.
Secret Sharing
A cryptographic method for distributing a secret among a group of participants, where the secret can only be reconstructed by combining a sufficient number of shares.
Federated Distillation
A privacy-preserving technique where clients share model predictions on a public dataset instead of model parameters to transfer knowledge to a central model.
Split Learning
A distributed training paradigm where a neural network is partitioned between a client and server, with only intermediate activations and gradients exchanged.
Model Inversion
An attack that exploits access to a trained model's confidence scores to reconstruct representative samples of the private training data.
Membership Inference
An attack determining whether a specific data record was included in a model's training dataset by analyzing the model's prediction behavior.
Noise Injection
A defense mechanism that deliberately adds random perturbations to model parameters, gradients, or outputs to mask sensitive patterns and enhance privacy.
Privacy Budget
A quantifiable limit on the total privacy loss allowed over a series of queries on a sensitive dataset, defined by the epsilon parameter in differential privacy.
DP-SGD
Differentially Private Stochastic Gradient Descent, an optimization algorithm that clips per-sample gradients and adds Gaussian noise to provide formal privacy guarantees during training.
Non-IID Data
Non-Independently and Identically Distributed data, a common challenge in federated learning where local datasets have heterogeneous statistical distributions.
Federated Averaging (FedAvg)
The foundational federated learning algorithm that combines locally trained model weights on a central server by computing a weighted average.
Client Selection
The scheduling mechanism in federated learning that determines which subset of available devices participates in a given round of training.
Remote Attestation
A security process verifying the integrity and trustworthiness of a remote client's software and hardware environment before granting access to data or computation.
Federated Unlearning
A technique to efficiently remove the influence of a specific client's data from a trained global federated model without full retraining.
Personalized Federated Learning
A variant of federated learning that creates tailored local models for individual clients to handle statistical heterogeneity rather than relying on a single global model.
Krum
A Byzantine-resilient aggregation rule that selects the single local update with the smallest sum of squared distances to its nearest neighbors to filter out malicious gradients.
Norm Clipping
A technique bounding the L2 norm of individual model updates or per-sample gradients to limit the influence of any single data point or malicious outlier.
Concept Drift
The phenomenon where the statistical properties of the target variable change over time in unforeseen ways, degrading model performance in continuous learning settings.
Model Watermarking Techniques
Terms related to embedding imperceptible identifiers into neural networks to prove intellectual property ownership. Target: IP Lawyers and MLOps Leads.
Digital Watermarking
The process of embedding an imperceptible, verifiable identifier into a neural network's weights, structure, or outputs to assert intellectual property ownership.
White-Box Watermarking
A watermarking technique that embeds a signature directly into the internal parameters or weights of a model, requiring full access to the model's architecture for extraction.
Black-Box Watermarking
A watermarking technique that embeds a signature detectable solely through the model's input-output behavior, enabling ownership verification via remote API queries without internal access.
Trigger-Set Watermarking
A black-box method that trains a model to produce specific, pre-defined incorrect outputs for a secret set of crafted inputs, serving as a statistical proof of ownership.
Backdoor Watermarking
A technique synonymous with trigger-set watermarking where a model learns a hidden mapping from a trigger pattern to a target label, acting as a covert ownership identifier.
Parameter Encoding
A white-box method that directly embeds a bit string into the least significant bits or statistical distribution of a model's trainable parameters.
Weight Regularization
A watermark embedding strategy that adds an auxiliary loss term during training to constrain model weights to carry a specific statistical signature without degrading primary task performance.
Watermark Embedding
The training-phase procedure of injecting an ownership identifier into a host model, balancing the trade-off between watermark detectability and model fidelity.
Watermark Extraction
The process of retrieving or detecting an embedded identifier from a model, requiring either white-box access to parameters or black-box access to query outputs.
Watermark Verification
The cryptographic or statistical protocol used to confirm the presence of a specific watermark, typically involving a secret detection key and a null hypothesis test to prevent false claims.
Fidelity Preservation
The constraint that a watermarking algorithm must not cause a statistically significant degradation in the host model's performance on its original task.
Robustness to Fine-Tuning
The property of a watermark to survive transfer learning or domain adaptation processes where an adversary retrains the model on a new dataset to overwrite the ownership signature.
Robustness to Distillation
The resilience of a watermark against model extraction attacks where a student model is trained to mimic the outputs of the watermarked teacher model.
Overwriting Resistance
The ability of a watermark to prevent an adversary from embedding a new, conflicting ownership signature on top of the original without destroying model utility.
Payload Capacity
The maximum length of the identifying bit string that can be reliably embedded and extracted from a model without violating fidelity constraints.
Bit Error Rate
The fraction of incorrectly decoded bits during watermark extraction, used as a metric to quantify the reliability of the embedded payload under model modifications.
False Positive Rate
The probability that a watermark detection algorithm incorrectly claims ownership of a non-watermarked model, a critical metric for legal admissibility in IP disputes.
Statistical Uniqueness
The requirement that a watermark signature is sufficiently improbable to occur by random chance, providing a rigorous mathematical basis for asserting model ownership.
Ownership Verification
The complete protocol by which a legitimate owner proves model provenance to a third-party arbiter using the embedded watermark and a secret extraction key.
IP Provenance
The establishment of a verifiable chain of custody and creation history for a model artifact, using watermarking to link a deployed model to its original training run and owner.
Steganographic Embedding
A covert communication technique adapted for model watermarking, hiding the ownership payload within the noise-tolerant redundancy of over-parameterized neural network weights.
Entanglement Watermarking
A method that entangles the watermark extraction process with the model's learned feature representations, making the signature intrinsically difficult to remove without damaging the model.
Dynamic Watermarking
A technique where the watermark verification trigger set is generated on-the-fly using a cryptographic function of the input, preventing attackers from reverse-engineering static triggers.
Static Watermarking
A technique using a fixed, pre-generated set of trigger samples for embedding and verification, which is more vulnerable to collusion and reverse-engineering attacks.
Adversarial Robustness
The resistance of a watermark to deliberate removal attacks, including parameter pruning, fine-tuning, distillation, and input perturbation designed to erase the ownership signature.
Collusion Resistance
The property that an attacker cannot successfully remove a watermark by comparing multiple independently watermarked copies of the same base model.
Ambiguity Attack
An adversarial strategy where an attacker forges a fake watermark to create a conflicting ownership claim, exploiting a lack of statistical uniqueness in the original embedding.
Watermark Detection Key
The secret cryptographic material required to extract or verify a watermark, ensuring that only the legitimate owner can prove model provenance.
Digital Fingerprinting
A distinct but related technique that embeds a unique, user-specific identifier into each distributed copy of a model to trace the source of unauthorized redistribution.
Passport Layer
A dedicated, parametric layer inserted into a neural network architecture whose weights are designed to encode a digital watermark, offering a standardized embedding target.
AI Supply Chain Security
Terms related to verifying the provenance and integrity of model artifacts and dependencies. Target: DevSecOps Engineers.
Software Bill of Materials (SBOM)
A formal, machine-readable inventory of all components, libraries, and dependencies that constitute a software artifact, enabling precise vulnerability and license management.
SLSA Framework
A security framework (Supply-chain Levels for Software Artifacts) providing a graded checklist of controls to prevent tampering and improve the integrity of software packages and build pipelines.
Model Provenance
The verifiable chronology of the origin, ownership, and transformations applied to a machine learning model, ensuring its integrity and authenticity throughout the development lifecycle.
Sigstore
An open-source project enabling automated, free digital signing and verification of software artifacts using short-lived certificates issued via OpenID Connect identities.
Dependency Confusion
A supply chain attack vector where a malicious package with a higher version number is uploaded to a public registry, tricking a build system into downloading it instead of the intended private dependency.
Reproducible Build
A deterministic compilation process that allows independent parties to recreate a bit-for-bit identical software artifact from the same source code, verifying that no tampering occurred during the build.
Binary Authorization
A deploy-time security control that enforces strict policy checks, requiring a valid cryptographic signature from a trusted authority before a container image or artifact can be executed in a production environment.
Trusted Execution Environment (TEE)
A secure, isolated area within a main processor that guarantees the confidentiality and integrity of code and data loaded inside it, protecting them from the host operating system.
Software Composition Analysis (SCA)
An automated process for identifying and cataloging open-source components within a codebase to manage security vulnerabilities, licensing risks, and code quality.
Supply-chain Levels for Software Artifacts (SLSA)
An end-to-end framework for ensuring the integrity of software artifacts throughout the supply chain, protecting against source, build, and dependency tampering.
Policy as Code
The practice of writing and managing infrastructure and security rules as versioned, executable code, enabling automated enforcement and auditability of compliance requirements.
Open Policy Agent (OPA)
A general-purpose policy engine that decouples policy decision-making from application logic, providing a unified framework for enforcing authorization and compliance across the cloud-native stack.
The Update Framework (TUF)
A specification designed to secure software update systems by providing a robust defense against key compromises and various repository attacks through a defined role-based trust model.
In-toto
A framework that cryptographically attests to the integrity of every step in a software supply chain, creating a verifiable, end-to-end record of who performed what action and what materials were used.
Chain of Custody
A chronological, auditable trail that documents the sequence of custody, control, transfer, and analysis of a digital artifact, ensuring its integrity and non-repudiation.
Code Signing
The cryptographic process of digitally signing executables and scripts to confirm the software author's identity and guarantee that the code has not been altered or corrupted since it was signed.
Vulnerability Exploitability eXchange (VEX)
A standardized security advisory that allows a software supplier to declare the exploitability status of a specific vulnerability in a specific product, reducing false positives for end-users.
CycloneDX
A lightweight, standards-based Bill of Materials specification designed for use in application security contexts and supply chain component analysis, supporting SBOM, SaaSBOM, and OBOM.
SPDX
An open standard (Software Package Data Exchange) for communicating software bill of material information, including components, licenses, copyrights, and security references.
Dependency Pinning
The practice of locking a software dependency to an exact, immutable version or cryptographic hash to guarantee reproducible builds and prevent unexpected breakage from upstream changes.
Transparency Log
An append-only, cryptographically verifiable public ledger that records digital events, such as code signatures, making the issuance of certificates auditable and detectable if compromised.
Cosign
A tool under the Sigstore project that enables signing and verification of container images, blobs, and other artifacts using a keyless workflow tied to an OIDC identity.
Workload Identity
A method of assigning a verifiable, short-lived identity to a non-human process or service running in a cloud-native environment, enabling secure, credential-free authentication to other resources.
Immutable Artifact
A software artifact, such as a container image, that is never modified after creation; any change requires building and deploying a completely new artifact with a unique identifier.
Secure Software Development Framework (SSDF)
A set of fundamental, sound practices for secure software development defined by NIST SP 800-218, organized into four groups: Prepare the Organization, Protect the Software, Produce Well-Secured Software, and Respond to Vulnerabilities.
Zero Trust Supply Chain
A security model that applies the principle of 'never trust, always verify' to every element of the software delivery pipeline, requiring continuous validation of artifacts, identities, and build integrity.
Artifact Registry
A centralized, managed repository for storing, managing, and securing build artifacts and dependencies, often providing vulnerability scanning and access control policies.
Dependency Graph
A directed graph representing the relationships between all software packages in a project, mapping out which components depend on others to enable transitive vulnerability analysis.
Container Scanning
The automated process of analyzing container images layer-by-layer to identify embedded operating system packages, libraries, and known vulnerabilities before deployment.
GitOps
An operational framework that uses a Git repository as the single source of truth for declarative infrastructure and application configurations, with automated processes ensuring the live state matches the desired state.
Prompt Injection Defense
Terms related to mitigating attacks that override system instructions via crafted user inputs in LLMs. Target: Application Security Engineers.
Prompt Injection
A vulnerability where an attacker overrides a model's system instructions by embedding malicious commands within user-provided input.
Indirect Prompt Injection
An attack where malicious instructions are injected into data sources a model retrieves, such as web pages or documents, rather than the direct user query.
System Prompt Hardening
The practice of designing robust system-level instructions that are resistant to override attempts by malicious user inputs.
Input Sanitization
The process of cleaning and normalizing user-provided text to remove or neutralize potentially malicious control sequences before model processing.
Delimiter-Based Defense
A mitigation technique that uses special character sequences to clearly separate untrusted user input from trusted system instructions.
Guard Model
A secondary, often smaller, model that screens inputs and outputs of a primary model to detect and block policy violations or injection attacks.
Structured Output Enforcement
Constraining a model to generate responses in a specific, machine-readable format like JSON to prevent the execution of injected free-form instructions.
Canonicalization
The process of converting input data into a single, standard, unambiguous representation to prevent attackers from bypassing filters using encoding tricks.
Multi-Modal Injection
An attack vector that embeds malicious prompts in non-text modalities, such as images or audio files, processed by a multi-modal model.
Context Window Exhaustion
An attack that floods a model's context window with filler content to displace or dilute system instructions and safety guardrails.
Prompt Leaking
An attack designed to extract a model's confidential system prompt or internal instructions, often the first stage of a more complex injection.
Adversarial Prompt Detection
The use of classifiers, heuristics, or perplexity analysis to identify user inputs that are likely crafted to manipulate a model.
Prompt Injection WAF
A Web Application Firewall-like layer that inspects and blocks malicious prompts at the API gateway before they reach the language model.
Tool Authorization Gate
A security checkpoint that validates and authorizes any function call or API request a model attempts to make, preventing unauthorized actions from injections.
Retrieval-Augmented Generation (RAG) Injection
A specific form of indirect prompt injection where an attacker poisons a vector database or knowledge base to manipulate a RAG system's outputs.
Data Source Poisoning
The act of inserting malicious content into a model's external data sources, such as a website or document store, to influence its behavior upon retrieval.
Multi-Turn Injection
An attack distributed across multiple conversational turns, where seemingly benign messages gradually steer a model toward a malicious goal.
Homoglyph Attack
A technique that uses visually similar characters from different alphabets to bypass text-based filters while appearing normal to a human reviewer.
Zero-Width Character Injection
An attack that inserts invisible Unicode characters into text to break tokenization or keyword filtering without being visible to the user.
Fine-Tuning Data Injection
A supply chain attack where malicious examples are inserted into a model's fine-tuning dataset to create a backdoor or override safety training.
Refusal Training
A safety technique that fine-tunes a model to explicitly reject requests that violate its usage policies, making it harder to jailbreak via injection.
Chain-of-Thought Hijacking
An injection that manipulates a model's step-by-step reasoning process to arrive at an attacker-chosen, harmful conclusion.
Prompt Normalization
The process of rewriting or restructuring user prompts into a safe, canonical form before they are combined with system instructions.
Egress Content Guard
A filter applied to a model's output to redact sensitive data, block malicious URLs, or prevent the leakage of system instructions.
Code Execution Sandboxing
Isolating any code generated or executed by a model within a restricted, ephemeral environment to prevent system compromise from an injection.
Human-in-the-Loop (HITL) Approval
A security pattern requiring a human operator to manually approve sensitive actions, such as tool calls, triggered by a model to prevent autonomous exploitation.
Red Teaming for Prompt Injection
The adversarial practice of systematically probing a language model application with creative injection attacks to discover vulnerabilities before deployment.
Instructional Hierarchy
A safety framework that prioritizes system-level instructions over user-level or tool-level instructions to prevent lower-privilege inputs from overriding core directives.
Context Boundary Enforcement
A defensive technique that strictly segregates different information sources within a prompt to prevent cross-contamination and privilege escalation.
Prompt Injection Kill Chain
A model of the sequential stages of a prompt injection attack, from initial reconnaissance to achieving a malicious objective, used to design layered defenses.
AI Guardrail Architectures
Terms related to the safety layers that filter toxic, biased, or harmful model outputs. Target: Responsible AI Leads.
Constitutional AI
A training methodology where an AI model is supervised by a set of principles (a 'constitution') to evaluate and revise its own outputs, reducing reliance on human feedback for harmlessness alignment.
Reinforcement Learning from Human Feedback (RLHF)
A fine-tuning technique that uses human preferences as a reward signal to align language model outputs with complex qualitative goals like helpfulness and harmlessness.
Red Teaming
A structured adversarial process where security experts systematically probe an AI system to discover vulnerabilities, biases, and potential harmful outputs before deployment.
Safety Classifier
A specialized model or layer that evaluates an input prompt or generated output to assign a risk score, triggering a refusal or sanitization action if a toxicity or policy threshold is breached.
Jailbreak Detection
The real-time identification and blocking of adversarial prompts specifically engineered to bypass an LLM's safety guardrails and system instructions.
System Prompt Hardening
The practice of engineering robust system-level instructions that are resistant to extraction, leaking, or override attempts via prompt injection attacks.
Refusal Training
A fine-tuning process that explicitly teaches a model to decline compliance with harmful, toxic, or out-of-policy requests by generating a safe refusal string.
Self-Critique
A prompting architecture where an LLM generates an initial response and then evaluates its own output for factual consistency, safety, or policy adherence before presenting it to the user.
Moderation API
A dedicated, stateless endpoint provided by platforms like OpenAI or Google that scores text or images for categories such as hate speech, violence, and sexual content without generating a response.
Prompt Injection Classifier
A detection model trained to distinguish between legitimate user instructions and malicious payloads attempting to override the system prompt or exfiltrate data.
Indirect Injection Guard
A defensive filter that sanitizes external data sources (web pages, emails) before they are ingested into the LLM context window to prevent embedded adversarial instructions from executing.
Hallucination Filter
A post-processing mechanism that verifies generated statements against a retrieval corpus or knowledge graph to suppress non-factual or ungrounded content.
Guard Model
A lightweight, independent classifier (often a fine-tuned BERT or DeBERTa variant) that runs in parallel with the main LLM to intercept and block unsafe prompts or outputs with low latency.
Circuit Breaker
An automated operational safeguard that immediately halts model inference or revokes API access when a critical volume of policy violations or anomalous queries is detected within a time window.
Over-Refusal
A safety alignment failure mode where the model incorrectly rejects benign or legitimate requests due to overly aggressive safety training, degrading the user experience.
Representation Engineering
A safety technique that directly manipulates the internal activations of a neural network to control high-level cognitive states like honesty or harmlessness without prompt-based instructions.
Safety Vector
A specific direction in a model's latent space identified via activation steering that, when added to the forward pass, reliably induces safer behavior or triggers a refusal.
Machine Unlearning
The algorithmic process of efficiently removing the influence of specific, problematic data points (e.g., toxic text, PII) from a trained model's weights without full retraining.
Constrained Decoding
A runtime inference technique that applies a logit bias or token mask to force the LLM to generate outputs that strictly adhere to a predefined grammar, schema, or vocabulary restriction.
LLM-as-a-Judge
An evaluation paradigm where a strong, general-purpose language model is used to score the quality, safety, or factual accuracy of another model's outputs, replacing human evaluators for scalable oversight.
Ensemble Guard
A defense-in-depth architecture that combines multiple heterogeneous safety classifiers (e.g., regex, semantic embedding, neural classifier) via a voting or cascading mechanism to reduce false negatives.
Selective Prediction
A risk-management strategy where the model is equipped with a 'reject option' to abstain from making a prediction when its confidence score falls below a calibrated threshold.
Adversarial Debiasing
A training technique using a gradient reversal layer and an adversarial domain classifier to learn data representations that are maximally predictive of the task but minimally predictive of protected attributes.
Model Card
A structured transparency document (pioneered by Google) that details a model's intended use, evaluation results, and ethical limitations to standardize responsible disclosure.
Direct Preference Optimization (DPO)
A stable alignment algorithm that directly optimizes a policy on human preference data using a binary cross-entropy loss, bypassing the need for a separate reward model in the RLHF pipeline.
Canary Token
A unique, decoy data string embedded in system prompts or training data that triggers an alert if it appears in an external output, serving as a tripwire for prompt extraction or data leakage.
PII Redaction
A pre-processing or post-processing guard that uses named entity recognition to detect and mask personally identifiable information (names, SSNs, emails) before data reaches the core model or the end user.
Context Distillation
A training process where a large, safe 'teacher' model generates refined responses to adversarial prompts, and a smaller 'student' model is fine-tuned on this curated data to internalize the safety guardrails.
Audit Trail
An immutable, chronological log of all prompts, model decisions, guardrail interventions, and human overrides, providing forensic traceability for compliance and incident response.
Safety Gym
A suite of reinforcement learning environments designed by OpenAI to benchmark the ability of agents to optimize for a goal while satisfying complex safety constraints.
Adversarial Robustness Training
Terms related to training methodologies that improve model resilience against perturbed or manipulated inputs. Target: Research Scientists.
Adversarial Training
A defensive technique that augments training datasets with adversarial examples to improve a model's resilience against maliciously perturbed inputs.
Projected Gradient Descent (PGD)
A powerful iterative white-box attack method that generates adversarial examples by taking multiple small steps and projecting the perturbation back onto an epsilon-ball constraint.
Fast Gradient Sign Method (FGSM)
A single-step, white-box attack that creates adversarial examples by adding a perturbation in the direction of the gradient of the loss function with respect to the input.
Carlini-Wagner Attack (C&W)
An optimization-based adversarial attack that finds minimal perturbations necessary to cause misclassification by minimizing a distance metric subject to a misclassification constraint.
DeepFool
An iterative, untargeted attack that computes minimal perturbations by iteratively projecting an input onto the closest separating hyperplane of a linearized classifier.
Virtual Adversarial Training (VAT)
A regularization method that smooths the model's output distribution by minimizing the KL divergence between predictions on clean data and locally perturbed data.
Mixup Training
A data augmentation technique that trains a model on convex combinations of input pairs and their corresponding labels, encouraging linear behavior between training examples.
TRADES
A training algorithm that optimizes a trade-off between natural accuracy and adversarial robustness by decomposing the prediction error into natural and boundary errors.
Adversarial Weight Perturbation (AWP)
A technique that injects worst-case perturbations directly into model weights during training to flatten the loss landscape and improve adversarial robustness.
Randomized Smoothing
A probabilistic defense that constructs a certifiably robust classifier by adding Gaussian noise to inputs and returning the most probable prediction under that noise distribution.
Certified Robustness
A property of a model that provides a mathematical guarantee that its prediction will not change for any input within a specified Lp-norm radius.
Interval Bound Propagation (IBP)
A verification method that propagates symbolic bounds through a neural network to compute a guaranteed range of possible outputs for a given input perturbation set.
Input Gradient Regularization
A defensive technique that penalizes the magnitude of the gradient of the loss with respect to the input, encouraging smoother model decision boundaries.
Spectral Normalization
A weight normalization technique that constrains the Lipschitz constant of a neural network layer by dividing its weight matrices by their spectral norm.
Feature Squeezing
A lightweight detection method that compares a model's prediction on an original input with its prediction on a squeezed version to identify adversarial examples.
Defensive Distillation
A training procedure where a second model is trained on the softmax probability vectors of a first model to reduce the amplitude of adversarial gradients.
Ensemble Adversarial Training
A robustness strategy that augments training data with adversarial examples generated from multiple surrogate models to improve black-box attack resilience.
Adversarial Patch Training
A method for hardening models against physical-world attacks by training on images containing localized, highly salient, and arbitrarily shaped perturbations.
Expectation over Transformation (EoT)
A technique for generating robust adversarial examples in the physical world by optimizing perturbations over a distribution of environmental transformations like rotation and scale.
Boundary Attack
A decision-based black-box attack that starts from a large adversarial perturbation and iteratively reduces its magnitude while staying on the adversarial side of the decision boundary.
Robust Self-Training
A semi-supervised learning approach that generates pseudo-labels for unlabeled data using a teacher model and incorporates adversarial objectives to improve robustness.
Adversarial Robustness Toolbox (ART)
An open-source Python library, primarily maintained by IBM, providing standardized implementations of adversarial attacks, defenses, and robustness metrics for machine learning models.
Gradient Masking
A phenomenon where a defense gives a false sense of security by producing obfuscated or useless gradients, preventing gradient-based attacks but remaining vulnerable to other methods.
Lipschitz Constant Constraint
A regularization approach that enforces an upper bound on the rate of change of a model's output relative to its input, directly limiting sensitivity to perturbations.
Min-Max Optimization
The mathematical framework for adversarial robustness that formulates training as minimizing the empirical risk against a worst-case adversary that maximizes the loss.
Out-of-Distribution Detection
Terms related to identifying inputs that differ significantly from the training data distribution. Target: ML Reliability Engineers.
Out-of-Distribution (OOD) Detection
The task of identifying inputs that are semantically or statistically different from a model's training data, preventing unpredictable predictions on unknown concepts.
Anomaly Detection
The identification of rare items, events, or observations which deviate significantly from the majority of the data, often used synonymously with novelty or outlier detection in unsupervised settings.
Open Set Recognition
A classification framework that requires a model to accurately classify known classes while simultaneously rejecting samples from unknown classes not seen during training.
Novelty Detection
The process of recognizing that a test input originates from a previously unseen pattern or class that was entirely absent from the training distribution.
Maximum Softmax Probability (MSP)
A baseline OOD detection method that uses the highest softmax output score as a confidence measure, rejecting inputs that fall below a set threshold.
Energy-Based Model (EBM)
A probabilistic framework that assigns low energy values to in-distribution data and high energy to OOD data, using the Helmholtz free energy as a discriminative score.
ODIN Detector
An OOD detection technique that combines temperature scaling with small input perturbations to widen the separation gap between in-distribution and out-of-distribution softmax scores.
Mahalanobis Distance Score
A parametric OOD detection method that computes the distance of a feature representation to the nearest class-conditional Gaussian distribution, capturing covariance structure.
Deep SVDD
Deep One-Class Classification that trains a neural network to map normal data into a minimal hypersphere, flagging points far from the center as anomalies.
Isolation Forest
An ensemble-based anomaly detection algorithm that isolates observations by randomly selecting a feature and split value, exploiting the fact that anomalies are easier to isolate.
Local Outlier Factor (LOF)
A density-based anomaly detection method that measures the local deviation of a data point's density relative to its neighbors, identifying points in substantially lower-density regions.
Epistemic Uncertainty
The model uncertainty arising from a lack of knowledge or data, which is theoretically reducible with more training samples and is crucial for identifying unfamiliar inputs.
Aleatoric Uncertainty
The irreducible statistical uncertainty inherent in the data itself, such as sensor noise or class overlap, which cannot be reduced by collecting more data.
Monte Carlo Dropout
A Bayesian approximation technique that performs multiple forward passes with dropout enabled at inference time to estimate predictive uncertainty for OOD detection.
Deep Ensembles
A method that trains multiple independent models with different initializations and averages their predictions to provide robust uncertainty estimates and disagreement metrics for OOD detection.
OpenMax
An open set recognition algorithm that replaces the standard softmax layer with a Weibull-calibrated activation vector, explicitly modeling the probability of an 'unknown' class.
Contrastive Training for OOD
A self-supervised learning approach that pulls augmented views of the same sample together while pushing others apart, improving feature separability for downstream OOD detection.
Likelihood Regret
A metric that compares the likelihood of a test sample under a trained model to the likelihood under a background model, correcting for input complexity to detect OOD data.
Typicality Test
A statistical evaluation that rejects inputs not only for low likelihood but also for failing to reside in the typical set of the model's learned distribution.
Virtual Adversarial Training (VAT)
A regularization technique that smooths the model's output distribution against local perturbations, improving the uniformity of softmax scores for OOD inputs.
Outlier Exposure
A training strategy that leverages an auxiliary dataset of outliers to teach the model heuristics for detecting unknown inputs, significantly improving generalization to unseen OOD distributions.
GradNorm
An OOD detection method based on the observation that the gradient magnitude of the KL divergence, with respect to model parameters, is typically higher for in-distribution data.
ReAct
A post-hoc OOD detection method that rectifies activations by clipping extremely high values, reducing the overconfidence of neural networks on out-of-distribution inputs.
Virtual Logit Matching (ViM)
A technique that combines class-agnostic feature norms with standard logits by projecting features onto the residual space of principal components to detect OOD inputs.
KNN Distance for OOD
A non-parametric detection method that uses the distance to the k-th nearest neighbor in the training feature space as a normality score, rejecting inputs far from the manifold.
Extreme Value Theory (EVT)
A statistical framework for modeling the tail behavior of distributions, used in OOD detection to calibrate the probability of extreme activation values for open-set rejection.
Normalizing Flows
A generative modeling approach that transforms a simple base distribution into a complex one through a sequence of invertible mappings, enabling exact likelihood computation for OOD detection.
Diffusion Model for OOD
The use of denoising diffusion probabilistic models to reconstruct inputs, where high reconstruction error on the noised input serves as a signal for out-of-distribution detection.
Self-Supervised OOD Detection
A paradigm that trains models on pretext tasks like rotation prediction or jigsaw puzzles to learn robust feature representations without class labels, enhancing OOD sensitivity.
Hyperspherical Embedding
A technique that constrains feature vectors to lie on the surface of a unit sphere, improving OOD detection by aligning class directions and reducing feature collapse.
Model Obfuscation Techniques
Terms related to protecting model architecture and weights from extraction or reverse engineering. Target: Embedded Systems Engineers.
Model Obfuscation
A set of techniques that transform a trained machine learning model into a functionally equivalent but unintelligible form to protect its architecture, weights, and logic from extraction or reverse engineering.
Gradient Masking
A defensive technique that hides or distorts the true gradient information of a model to prevent attackers from using gradient-based methods for adversarial example generation or model extraction.
Model Sharding
The process of partitioning a neural network's computational graph and parameters across multiple isolated devices or secure enclaves so that no single node possesses the complete model.
Homomorphic Encryption Inference
A cryptographic method that allows computation to be performed directly on encrypted data, enabling a model to generate predictions without ever decrypting the user's input or the model's output.
Secure Multi-Party Computation (SMPC)
A cryptographic protocol that distributes a model's computation across multiple parties who jointly compute an inference result without revealing their private input shares to one another.
Trusted Execution Environment (TEE)
A hardware-enforced secure area within a main processor that guarantees the confidentiality and integrity of the code and data loaded inside it, isolating model inference from the host operating system.
Binary Obfuscation
The practice of modifying a compiled model binary or executable to make its underlying logic difficult for decompilers and disassemblers to analyze, while preserving its original runtime behavior.
Control Flow Flattening
An obfuscation technique that removes the natural conditional branching structure of a program's control flow and replaces it with a single flat dispatch loop, obscuring the logical sequence of operations.
Opaque Predicates
Conditional statements inserted into code whose outcome is known at obfuscation time but is computationally difficult for a static analyzer to determine, creating dead paths that confuse reverse engineers.
Virtualization Obfuscation
A technique that translates native machine code into a custom, randomized bytecode executed by an embedded virtual machine, hiding the original instructions from static analysis.
Anti-Tampering
A set of integrity-checking mechanisms embedded within a model or its runtime that detect unauthorized modifications and trigger defensive responses such as shutdown or self-destruction.
White-Box Cryptography
A cryptographic implementation designed to protect secret keys even when an attacker has full visibility into the execution environment and can observe or alter memory and internal operations.
Model Encryption
The process of cryptographically securing a stored model artifact so that it can only be loaded and executed by an authorized runtime possessing the correct decryption key.
Layer Fusion
An optimization and obfuscation technique that merges multiple consecutive neural network layers into a single computational kernel, making it harder to isolate and extract individual layer parameters.
Adversarial Training Obfuscation
A defensive strategy that combines adversarial training with obfuscation techniques to produce a model that is both robust to input perturbations and resistant to architecture extraction.
Noise Injection
The deliberate addition of calibrated random perturbations to model weights, gradients, or outputs to degrade the signal available to an attacker attempting model stealing or inversion.
Latent Space Obfuscation
A technique that applies transformations to a model's internal representations to prevent an attacker from interpreting the learned features or reconstructing training data from intermediate activations.
Model Steganography
The practice of covertly embedding a secret payload, such as a watermark or identifier, within the parameters of a neural network without noticeably affecting its primary task performance.
Side-Channel Attack Mitigation
A class of defenses that eliminate or mask the physical information leakage—such as timing, power consumption, or electromagnetic emanations—from a processor running model inference.
Timing Attack Mitigation
Specific countermeasures that ensure the execution time of a cryptographic operation or model inference is constant and independent of secret data, preventing attackers from inferring information from latency variations.
Memory Access Obfuscation
A technique that randomizes the pattern of memory reads and writes during computation to prevent an attacker from deducing the model's data flow or parameters by monitoring the memory bus.
Bus Encryption
The on-the-fly encryption of data traveling between a processor and external memory, ensuring that an attacker physically probing the memory bus cannot capture plaintext model weights or inputs.
Physically Unclonable Function (PUF)
A physical hardware security primitive that derives a unique, unclonable cryptographic key from the inherent microscopic manufacturing variations in a silicon chip, used to bind a model to a specific device.
Logic Locking
A hardware security technique that inserts additional key-gates into an integrated circuit's design, rendering the chip non-functional unless the correct secret key is applied, protecting the hardware-accelerated model.
Bitstream Encryption
The process of encrypting the configuration file for an FPGA so that the implemented model architecture and weights cannot be read or cloned by intercepting the configuration data.
Zeroization
An active defense mechanism that immediately and irrevocably erases cryptographic keys, model weights, and sensitive data from memory upon detection of a physical tampering event.
Secure Element
A dedicated, tamper-resistant hardware chip designed to securely store cryptographic keys and execute sensitive operations, such as model decryption, in an isolated environment separate from the main application processor.
Remote Attestation
A cryptographic protocol that allows a remote party to verify the integrity and trusted state of a device's software and hardware environment before provisioning a decryption key or allowing model execution.
Model Fingerprinting
A technique that extracts a unique, verifiable identifier from a model's decision boundary or weight distribution to detect unauthorized copies or verify the integrity of a deployed model.
Control Flow Integrity (CFI)
A runtime security policy that restricts the execution paths of a program to a pre-computed control flow graph, preventing attackers from hijacking the model's execution flow via code-reuse attacks.
Secure Model Serving
Terms related to the authentication, authorization, and audit logging of inference API endpoints. Target: API Architects.
Mutual TLS (mTLS)
A mutual authentication protocol where both the client and server present and verify X.509 digital certificates to establish a highly trusted, encrypted connection for model serving endpoints.
JSON Web Token (JWT)
A compact, URL-safe token format used to transmit claims securely between parties, commonly used for stateless authorization in machine learning inference APIs.
OAuth 2.0 Client Credentials Flow
An OAuth 2.0 grant type designed for machine-to-machine authorization where a client application authenticates directly with the authorization server to obtain an access token without user interaction.
Role-Based Access Control (RBAC)
An authorization model that restricts system access based on a user's assigned roles within an organization, simplifying permission management for model serving platforms.
Policy Enforcement Point (PEP)
A component in a zero-trust architecture that intercepts requests to a protected resource, such as an inference endpoint, and enforces the access decision made by an external policy engine.
Rate Limiting
A defensive technique that restricts the number of API requests a client can make within a specific timeframe to prevent abuse, denial-of-service, and model extraction attacks.
Immutable Audit Trail
A chronological, tamper-proof record of all access and query events against a model serving system, stored in WORM-compliant storage to ensure non-repudiation for compliance reporting.
HMAC Signature
A mechanism for verifying both the data integrity and authenticity of an API request by computing a cryptographic hash using a shared secret key, preventing tampering during transit.
Payload Encryption
The practice of encrypting the request and response bodies of an inference API call at the application layer, providing defense-in-depth beyond transport-level TLS.
Input Sanitization
The process of cleaning and validating all data supplied to a model inference endpoint to neutralize malicious content, such as code injection or adversarial perturbations, before processing.
Schema Validation
The enforcement of a strict data contract, often defined by an OpenAPI Specification, on all inference requests to reject malformed or unexpected inputs that could trigger undefined model behavior.
SPIFFE
The Secure Production Identity Framework for Everyone, a set of open-source standards for securely identifying software services in dynamic, heterogeneous environments like Kubernetes-based model serving.
Token Introspection
A mechanism defined by RFC 7662 that allows a resource server, such as a model serving API, to query the authorization server to determine the active state and metadata of a presented access token.
Least Privilege Principle
The security concept that a user, program, or process should be granted only the minimum permissions necessary to perform its function, limiting the blast radius of a compromised inference credential.
Zero Trust Architecture
A security model that eliminates implicit trust and requires continuous verification of every access request to a model serving resource, regardless of the network origin of the request.
Proof-of-Possession Token
A security token, such as a DPoP token, that cryptographically binds an access token to a specific client, preventing a stolen token from being replayed by an unauthorized attacker.
Confidential Inference
The execution of model inference within a hardware-based Trusted Execution Environment (TEE) that isolates the model and data from the underlying host operating system and cloud provider.
Inference Data Masking
A technique for dynamically obfuscating or redacting sensitive data elements, such as PII, within the input and output of a model inference request to prevent unauthorized exposure in logs or responses.
Web Application Firewall (WAF) for ML
A specialized WAF configured with rulesets to inspect and filter HTTP traffic to machine learning APIs, blocking common web exploits and model-specific attacks like prompt injection.
IP Allowlisting
A network access control mechanism that restricts connections to a model inference endpoint to only pre-approved, trusted source IP address ranges, reducing the external attack surface.
Model Access Control List (ACL)
A granular permission set attached directly to a specific model artifact or version, defining which users, groups, or service accounts are authorized to invoke it for inference.
Runtime Application Self-Protection (RASP)
A security technology that is embedded within the model serving runtime to detect and block attacks in real-time by analyzing the application's internal state and execution context.
API Discovery
The automated process of identifying and cataloging all active, deprecated, and undocumented inference APIs within an environment to eliminate shadow and zombie API security risks.
GraphQL Query Depth Limiting
A security control that restricts the maximum nesting level of a GraphQL query sent to a model serving endpoint to prevent resource exhaustion and denial-of-service attacks.
Just-In-Time (JIT) Access
An access provisioning protocol that grants temporary, time-bound elevated privileges to a model serving system only when needed for a specific task, eliminating standing privileges.
Open Policy Agent (OPA)
A general-purpose policy engine that decouples authorization logic from the model serving application, allowing unified, fine-grained access control decisions written in the Rego policy language.
Policy as Code
The practice of defining and managing authorization and security rules for model serving infrastructure using machine-readable definition files, enabling version control and automated compliance testing.
User and Entity Behavior Analytics (UEBA)
A security solution that uses machine learning to establish baselines of normal API query behavior and detect anomalous activities indicative of model extraction or credential compromise.
Model Stealing Detection
A defensive mechanism that analyzes sequential query patterns to identify and block black-box extraction attacks attempting to replicate a proprietary model's functionality.
FIDO2
An open authentication standard that enables passwordless, phishing-resistant multi-factor authentication for administrative access to model serving infrastructure using cryptographic credentials.
AI Red Teaming Automation
Terms related to automated tools for probing model vulnerabilities and simulating adversarial attacks. Target: Offensive Security Teams.
Automated Red Teaming (ART)
A systematic process using specialized software to continuously probe AI models for vulnerabilities, simulating multi-turn adversarial attacks to identify safety failures before deployment.
Adversarial Prompting
The technique of crafting malicious inputs designed to override a language model's system instructions, causing unintended outputs such as jailbreaks or system prompt leakage.
Jailbreak Automation
The use of algorithms to automatically discover and chain prompt sequences that bypass a model's safety guardrails and refusal training.
Greedy Coordinate Gradient (GCG)
An optimization algorithm that automatically generates adversarial suffixes by iteratively computing token-level gradients to maximize the probability of a harmful target response.
Tree of Attacks with Pruning (TAP)
An automated black-box method that uses a tree-search structure with an attacker LLM to iteratively refine and prune prompt candidates until a jailbreak is achieved.
Black-Box Query Attack
An attack methodology that probes a model's vulnerabilities using only input-output pairs without any access to the internal architecture, gradients, or parameters.
White-Box Penetration Testing
A security assessment performed with full access to a model's weights, gradients, and architecture to identify vulnerabilities like gradient leakage or backdoor triggers.
Payload Splitting
An evasion technique where a malicious instruction is fragmented across multiple separate inputs or prompts to bypass safety filters that scan for complete harmful strings.
Crescendo Attack
A multi-turn jailbreak strategy that gradually escalates benign-seeming dialogue to manipulate the model into generating policy-violating content over successive interactions.
Many-Shot Jailbreaking
An attack that exploits long context windows by prepending hundreds of fabricated harmful dialogue examples to override the model's safety training through in-context learning.
Prompt Injection Suite
A collection of automated tools designed to test LLM applications for indirect, direct, and stored injection vulnerabilities that can hijack agent tool calls or exfiltrate data.
Attack Surface Mapping
The automated process of enumerating all input channels, APIs, plugins, and data retrieval endpoints of an AI system to identify potential vectors for adversarial exploitation.
Continuous Automated Red Teaming (CART)
A DevSecOps practice that integrates persistent, automated adversarial probes into the CI/CD pipeline to detect model regressions and new vulnerabilities with every code update.
Breach and Attack Simulation (BAS)
A platform that continuously simulates specific adversarial attack chains against production AI infrastructure to validate the effectiveness of security controls and guardrails.
Guardrail Bypass Detection
The automated process of stress-testing content safety classifiers and input/output filters to identify edge cases where toxic or disallowed content passes through undetected.
RLHF Weakness Probing
The systematic testing of models fine-tuned with Reinforcement Learning from Human Feedback to discover specific scenarios where the alignment training fails or produces sycophantic behavior.
Refusal Suppression
An attack technique that adds specific tokens or instructions to a prompt to inhibit the model's trained tendency to decline answering harmful or restricted queries.
Universal Adversarial Trigger
A specific input sequence or token pattern discovered algorithmically that causes a high rate of misclassification or harmful output across many different inputs to a target model.
Transfer Attack
An adversarial example generated against one surrogate model that successfully fools a different, black-box target model due to the transferability of adversarial perturbations.
Gradient-Based Attack
A white-box attack that computes the gradient of the model's loss function with respect to the input to create minimally perturbed adversarial examples that cause misclassification.
Fuzzing
An automated software testing technique that injects massive amounts of random, unexpected, or malformed data into model inputs to discover crashes, leaks, or undefined behavior.
Model Extraction
An attack that steals the functionality of a proprietary model by training a clone on carefully selected queries and their corresponding predictions from the target API.
Membership Inference Attack
An attack that determines whether a specific data record was part of a model's training set by analyzing statistical differences in the model's confidence scores or loss values.
Data Reconstruction Attack
An attack that recovers sensitive training data features, such as faces or text, by inverting the gradients or exploiting the generative capabilities of a trained model.
Token Smuggling
An obfuscation technique that encodes malicious instructions using invisible characters, Unicode normalization tricks, or split tokenization to evade string-matching safety filters.
Indirect Prompt Injection
An attack where malicious instructions are hidden in external data sources retrieved by the LLM, such as websites or PDFs, causing the model to execute them during the retrieval process.
System Prompt Leakage
An automated technique that uses role-playing or translation requests to trick the model into revealing its confidential initial system instructions and operational constraints.
Attack Success Rate (ASR)
The key performance indicator measuring the percentage of adversarial attempts that successfully bypass safety filters or cause a model to generate the attacker's intended harmful output.
Adversarial Drift Monitoring
The continuous tracking of model behavior and input distributions in production to detect when the system becomes more susceptible to known attack patterns due to data drift.
Model Resilience Scoring
A quantitative benchmark that aggregates performance across a suite of adversarial tests to provide a single metric representing a model's overall robustness to red team attacks.
ML Pipeline Security Hardening
Terms related to securing the CI/CD pipeline for machine learning, including container scanning and dependency management. Target: Platform Engineers.
Software Bill of Materials (SBOM)
A formal, machine-readable inventory of all components, libraries, and dependencies that constitute a software artifact, enabling vulnerability tracking and license compliance.
SLSA Framework
A security framework (Supply-chain Levels for Software Artifacts) providing a graduated checklist of controls to prevent tampering and improve the integrity of software packages throughout the build and distribution process.
Sigstore
An open-source project enabling automated, keyless signing and verification of software artifacts using short-lived ephemeral certificates issued via OpenID Connect-based identity tokens.
In-Toto Attestation
A metadata specification that cryptographically verifies the steps and materials used in a software supply chain, providing non-repudiable evidence of the build process integrity.
Provenance Metadata
Verifiable information about the origin, build steps, and source materials that produced a specific software artifact, allowing consumers to assess its trustworthiness.
Binary Authorization
A deploy-time security control that enforces signature validation, ensuring only trusted and verified container images are allowed to run in a production environment.
Dependency Confusion
A supply chain attack where a malicious package with a higher version number is uploaded to a public registry, tricking a build system into downloading it instead of the intended private dependency.
Vulnerability Exploitability eXchange (VEX)
A standardized security advisory format that communicates the exploitability status of a known vulnerability within a specific product context, reducing false positives in scanning.
Common Vulnerability Scoring System (CVSS)
An open industry standard for assessing the principal technical severity of software vulnerabilities, producing a numerical score reflecting their characteristics and impact.
Reproducible Builds
A software development practice where compiling the same source code with the same environment always produces a bit-for-bit identical output, enabling independent verification of binary integrity.
Trusted Execution Environment (TEE)
A secure, isolated area within a main processor that guarantees the confidentiality and integrity of code and data loaded inside it, protecting sensitive computation from the host operating system.
Open Policy Agent (OPA)
A general-purpose policy engine that decouples policy decision-making from application logic, using the Rego language to evaluate structured data against defined rules.
Admission Controller
A Kubernetes-native plug-in that intercepts authenticated API requests to the cluster, enforcing custom security policies and validations before objects are persisted and executed.
eBPF Security
The use of the extended Berkeley Packet Filter to run sandboxed programs in the Linux kernel without changing kernel source code, enabling deep network and runtime observability for threat detection.
Distroless Base Image
A minimal container image that contains only the application and its runtime dependencies, excluding package managers, shells, and other standard OS utilities to reduce the attack surface.
Digest Pinning
The practice of referencing a container image by its immutable content-addressable SHA256 hash rather than a mutable tag, guaranteeing that the exact same artifact is deployed every time.
Workload Identity Federation
A mechanism that allows a software workload running outside a cloud provider to securely impersonate a service account using an external identity token, eliminating the need for long-lived service account keys.
SPIFFE
The Secure Production Identity Framework for Everyone, a set of open-source standards for securely identifying software workloads in dynamic and heterogeneous environments via a universal identity control plane.
Software Composition Analysis (SCA)
An automated process for identifying and cataloging open-source components in a codebase to manage security vulnerabilities, license compliance, and code quality risks.
Container Breakout Prevention
A set of defensive configurations—including seccomp profiles, user namespace remapping, and capability dropping—designed to prevent a process from escaping the container isolation boundary to the host.
User Namespace Remapping
A Linux kernel feature that maps a container's root user to an unprivileged user on the host system, significantly mitigating the impact of a container breakout vulnerability.
Seccomp Default Deny
A security profile that blocks all system calls by default for a containerized process, explicitly allowing only a minimal set required for the application to function, reducing the kernel attack surface.
Immutable Infrastructure
A deployment paradigm where servers and containers are never modified after provisioning; any configuration change requires destroying the existing component and replacing it with a new, validated version.
Policy as Code
The practice of writing security and compliance rules in a high-level programming language, storing them in version control, and applying automated testing and deployment pipelines to them.
GitOps Security
The application of security controls to a GitOps workflow, ensuring that the Git repository serves as the single source of truth for both desired infrastructure state and the cryptographic verification of that state.
Compliance as Code
The methodology of translating regulatory framework requirements into executable, automated tests and policies that can be continuously validated against cloud infrastructure to prove audit readiness.
Cloud Security Posture Management (CSPM)
A class of security tools that continuously monitor and remediate misconfigurations and compliance risks across cloud infrastructure, IaaS, and PaaS environments through automated assessment.
Kubernetes Security Posture Management (KSPM)
A specialized security practice focused on automating the detection and remediation of misconfigurations, risky RBAC permissions, and compliance violations specifically within Kubernetes clusters.
Infrastructure as Code (IaC) Scanning
The automated analysis of declarative configuration files to identify security misconfigurations, policy violations, and exposed secrets before the infrastructure is provisioned in a live environment.
Continuous Authorization to Operate (cATO)
A modernized risk management framework that uses real-time security telemetry and automated control validation to maintain a system's ongoing authority to operate, replacing static, point-in-time assessments.
Model Extraction Prevention
Terms related to thwarting attacks that aim to steal model functionality through black-box querying. Target: API Security Specialists.
Model Extraction Attack
An attack where an adversary queries a black-box model to reconstruct a functionally equivalent surrogate model, effectively stealing intellectual property.
API Rate Limiting
A defensive mechanism that restricts the number of API requests a client can make within a specific time window to prevent automated model extraction.
Query Throttling
The intentional slowing down of API response times for suspicious or high-frequency clients to increase the cost and time required for model stealing.
Differential Privacy
A mathematical framework that injects calibrated noise into query responses to provide a provable guarantee that individual training records cannot be inferred.
Output Perturbation
The technique of adding statistical noise directly to a model's predictions or confidence scores to obscure the precise decision boundary from an attacker.
Prediction Truncation
Limiting the number of output classes or top-k predictions returned by an API to reduce the information leakage per query.
Confidence Score Masking
The practice of hiding or rounding the raw confidence probabilities returned by a model, often returning only the final class label to harden extraction.
Decision Boundary Hardening
Training models to have smoother or more complex decision boundaries that are difficult for a surrogate model to approximate through querying.
Model Distillation Resistance
Techniques designed to prevent a student model from effectively learning the behavior of a teacher model through black-box query access.
Ensemble Obfuscation
Using a diverse ensemble of models to serve predictions, making the aggregate decision function inconsistent and harder to steal than a single model.
Model Watermarking
Embedding a unique, verifiable identifier into a model's weights or behavior to prove ownership if the model is stolen and deployed elsewhere.
Honeypot Model
A decoy model deployed to attract attackers, allowing security teams to study extraction techniques and trigger alerts without exposing the production model.
Query Pattern Analysis
Monitoring API query sequences to detect the systematic, non-random access patterns indicative of an ongoing model extraction attack.
Sequential Query Detection
Identifying extraction attempts by analyzing the temporal and spatial correlation between consecutive queries designed to map the input space.
Feature Space Distortion
Applying a non-linear, secret transformation to input features before processing, so that stolen queries cannot train a useful surrogate model.
Gradient Masking
A defense that hides or obfuscates the true gradient of the loss function, making it difficult for an attacker to use gradient-based optimization for extraction.
Response Randomization
Introducing controlled randomness into the model's output logic so that identical queries do not always return the exact same result.
Information Gain Limiting
Capping the amount of new information an attacker can derive from a single query, often measured by mutual information or entropy reduction.
Entropy Thresholding
Blocking or flagging queries where the model's prediction entropy is high, as these boundary-probing queries are most valuable for extraction.
API Tokenization
Requiring authenticated, rotating tokens for API access to enable per-session tracking, rate limiting, and attribution of extraction attempts.
Session Fingerprinting
Building a unique profile of a client's querying behavior and device characteristics to link anonymous sessions and detect coordinated extraction campaigns.
Proof-of-Work Challenge
Requiring a client to solve a computationally expensive cryptographic puzzle before serving an inference request, increasing the cost of automated extraction.
Cost-Based Querying
A pricing model where each API call has a monetary or credit cost, creating a direct economic disincentive for large-scale model extraction.
Model Inversion Defense
Countermeasures specifically designed to prevent an attacker from reconstructing representative training data from model outputs, often overlapping with extraction prevention.
Membership Inference Defense
Techniques that prevent an attacker from determining whether a specific data record was part of the model's training set, a common precursor to extraction.
Surrogate Model Detection
The process of identifying unauthorized copies of a model by comparing their behavior on a set of proprietary trigger inputs to the original model's behavior.
Query Fuzzing
A defensive technique that injects slight, imperceptible variations into the input received by the model to degrade the quality of a stolen surrogate model.
API Schema Obfuscation
Hiding or randomizing the structure, field names, and data types of the inference API to make automated reverse engineering of the interface more difficult.
Response Delay Injection
Artificially adding a variable time delay to API responses to disrupt the timing-based analysis used in some side-channel extraction attacks.
Decoy Output
A deliberately incorrect or misleading prediction served to clients exhibiting high-risk behavior to poison the training data of a potential surrogate model.
Partnered with leading AI, data, and software stack.
How We Work
Custom AI workflows for your Business
One-fit-all AI don't work for modern businesses. At Inferensys, we aim to understand your business & custom requirements; which we use to define most efficient agentic workflows, the data, and the tools for your business.
01
Review the use case
We understand the task, the users, and where AI can actually help.
Read more02
Pick the right approach
We define what needs search, automation, or product integration.
Read more03
Build the first useful version
We implement the part that proves the value first.
Read more04
Improve from there
We add the checks and visibility needed to keep it useful.
Read moreThe first call is a practical review of your use case and the right next step.
Talk to Us