Blog
Architecture review before implementation
Implementation scope and rollout planning
Clear next-step recommendation
Encryption at-rest and in-transit is standard, but data must be decrypted for the CPU to process it, creating a critical vulnerability window during AI inference and training.
Data is vulnerable during computation. Standard encryption protects data on disk (at-rest) and over the network (in-transit), but it must be decrypted into plaintext for the CPU to perform operations. This creates a critical exposure window where sensitive information is unprotected during AI model inference or training, the exact moment it is most valuable to an attacker.
Hardware TEEs are the core solution. The promise of Confidential Computing is fulfilled by Trusted Execution Environments (TEEs) like Intel SGX or AMD SEV. These are hardware-isolated enclaves where data is processed in encrypted memory, invisible even to the cloud provider's hypervisor or root user, closing the data-in-use gap.
AI workloads demand this protection. When fine-tuning a model on proprietary code or running RAG queries against a vector database like Pinecone or Weaviate containing customer PII, the data is actively in memory. Without TEEs, a privileged insider or compromised host OS can exfiltrate this plaintext data, turning your AI pipeline into a data breach.
Evidence: A 2023 study by the Confidential Computing Consortium found that over 70% of organizations consider data-in-use protection a top priority for AI/ML workloads, yet fewer than 20% have implemented it, highlighting a massive security debt.
Encryption for data-at-rest is table stakes; the trillion-dollar risk lies in unprotected data during active AI processing.
The EU AI Act classifies high-risk AI systems and mandates strict data governance. Non-compliance triggers fines of up to 7% of global turnover. Processing data in the wrong jurisdiction under GDPR creates a double liability. Confidential Computing with Trusted Execution Environments (TEEs) provides a technical basis for compliance by ensuring data remains protected during computation, regardless of physical location.
Enterprises routinely send sensitive data to external AI APIs from providers like OpenAI, Anthropic Claude, and Google Gemini. This creates an ungoverned data exfiltration vector. Standard API encryption only protects data in transit, not in-use on the vendor's infrastructure. A Policy-Aware Connector architecture, integrated with Confidential Computing, can redact PII and enforce usage policies before data leaves the corporate boundary.
Breakthroughs in healthcare and finance require pooling sensitive datasets across organizations. Traditional methods force data centralization, creating trust and legal barriers. Secure Multi-Party Computation (SMPC) and Federated Learning architectures, underpinned by Confidential Computing, allow joint model training without exposing raw data. This turns data collaboration from a legal nightmare into a competitive advantage.
This table compares the security posture of sensitive data across its three primary states, highlighting why encryption at-rest and in-transit is insufficient for modern AI workloads.
| Vulnerability / Protection | Data At-Rest (e.g., on disk) | Data In-Transit (e.g., over network) | Data In-Use (e.g., in CPU memory) |
|---|---|---|---|
Primary Threat Vector | Physical theft, unauthorized access | Network interception, man-in-the-middle | Memory scraping, side-channel attacks |
Standard Protection | AES-256 encryption | TLS 1.3 encryption | None (cleartext processing) |
Attack Complexity | Low (bypasses require key compromise) | Medium (requires network access) | High (exploits CPU microarchitecture) |
Example Attack | Stealing an unencrypted database backup | Packet sniffing on an unsecured Wi-Fi network | Spectre/Meltdown, Rowhammer, cold boot attack |
Protection Maturity |
|
| <10 years (emerging via TEEs) |
AI Workload Risk | Low (data is static) | Medium (data is moving) | Critical (data is actively processed) |
Confidential Computing Required | |||
Typical Latency Impact | < 1 ms (decryption) | 1-10 ms (handshake/encryption) | 5-20% (TEE overhead) |
TEEs create hardware-isolated secure enclaves where data is processed in encrypted form, protecting it from the host OS, hypervisor, and cloud provider.
Trusted Execution Environments (TEEs) protect data during computation by creating a hardware-isolated secure enclave within the CPU. This is the core mechanism for Confidential Computing, which secures data-in-use, not just at-rest or in-transit.
The CPU enforces a hardware root of trust. Modern processors from Intel (SGX), AMD (SEV), and ARM (TrustZone) dedicate secure memory regions and cryptographic instructions. Code and data loaded into the enclave are encrypted in RAM and are only decrypted inside the CPU's protected execution circuits.
Remote attestation proves enclave integrity. Before sending sensitive data, a client cryptographically verifies the enclave is running unaltered code on genuine hardware. This process prevents malicious cloud providers or compromised hypervisors from tampering with the workload.
Evidence: Azure Confidential VMs leverage AMD SEV-SNP. These instances demonstrate a 5-15% performance overhead for TEE-protected inference, a viable trade-off for processing regulated data in platforms like Databricks or Hugging Face.
Encrypting data at-rest is table stakes; the critical vulnerability is the unencrypted data in CPU memory during AI processing. These are the scenarios where that exposure is unacceptable.
Processing EU citizen data on a US cloud server for real-time personalization violates GDPR's data residency requirements. A standard VM offers zero protection for data-in-use during inference.
Model inversion and membership inference attacks can reconstruct sensitive samples from your proprietary training data. Standard GPU clusters process this data in cleartext.
Hardware enclaves like Intel SGX or AMD SEV are necessary but insufficient. A layered PET architecture combines them with software-based runtime encryption.
A wearable ECG monitor streaming data to an on-device AI model cannot have patient health information (PHI) exposed in device memory, even for milliseconds.
Intelligent data connectors enforce residency and redaction policies before data reaches an LLM API, preventing violations at the source.
Banks cannot share raw transaction data to collaboratively train a fraud model. Secure Multi-Party Computation (SMPC) is required, but it breaks without foundational data-in-use protection.
Software-based encryption cannot protect data during active computation without crippling performance for real-time AI.
Homomorphic encryption (HE) fails for enterprise AI because it protects data by performing computations on ciphertext, which introduces massive computational overhead incompatible with real-time inference. This makes it impractical for latency-sensitive applications like live customer service chatbots or real-time fraud detection.
The performance tax is prohibitive. A simple inference task on HE-encrypted data can be 100 to 10,000 times slower than on plaintext, stalling production AI pipelines. This overhead directly conflicts with the economics of scalable model serving using platforms like vLLM or NVIDIA Triton.
HE creates integration complexity. Its specialized libraries and data formats do not integrate natively with standard AI toolchains like PyTorch, TensorFlow, or Hugging Face Transformers. This forces teams into complex, bespoke engineering, increasing development time and technical debt.
Evidence: A 2023 study by Microsoft Research showed that performing a single transformer layer inference on HE-encrypted data required ~1.5 seconds, versus milliseconds on plaintext—a >1000x slowdown that makes interactive AI applications impossible. For true data-in-use protection, you need the hardware-rooted security of Trusted Execution Environments (TEEs).
Common questions about why Confidential Computing must protect data while it's being processed, not just when it's stored or in transit.
Data-in-use refers to information actively being processed by a CPU or GPU in memory. This is the most vulnerable state, as traditional encryption only protects data at-rest (on disk) and in-transit (over a network). Confidential computing uses Trusted Execution Environments (TEEs) like Intel SGX or AMD SEV to encrypt data during computation.
Encryption for data-at-rest and in-transit is table stakes; the critical attack surface is unencrypted data in server memory during AI processing.
When a CPU processes data, it must decrypt it into system memory, creating a window of vulnerability measured in milliseconds to minutes.\n- Adversaries with cloud admin access or compromised hypervisors can dump this memory.\n- Side-channel attacks like Spectre can infer data patterns from CPU cache behavior.\n- This renders perimeter security and disk encryption completely ineffective for active workloads.
Confidential Computing uses CPU-based enclaves (e.g., Intel SGX, AMD SEV, ARM CCA) to create cryptographically isolated memory regions.\n- Data and code are encrypted within the CPU and inaccessible to the host OS, hypervisor, or cloud provider.\n- Remote attestation cryptographically verifies the integrity of the enclave before releasing sensitive data.\n- This shifts the trust boundary from the entire infrastructure down to the silicon itself.
Laws like GDPR, the EU AI Act, and sector-specific regulations (HIPAA, PCI DSS) impose strict data processing obligations.\n- Article 32 of GDPR explicitly requires protection of data during processing.\n- Data residency rules demand processing occur within geographic borders; TEEs enable secure processing in compliant zones.\n- Failure to protect data-in-use creates massive compliance and audit liabilities, turning AI projects into legal risks.
Data-in-use protection is not just a security cost; it's a strategic business enabler.\n- Allows multi-party collaboration on sensitive datasets (e.g., healthcare, finance) without sharing raw data.\n- Enables privacy-preserving AI on regulated data, turning previously unusable information into a competitive asset.\n- Facilitates migration to hybrid and public cloud by protecting 'crown jewel' data during inference and training.
Hardware enclaves are a foundational layer, but modern AI workloads demand a layered PET architecture.\n- TEEs protect isolated workloads but not data during pre-processing, feature engineering, or post-processing.\n- They require integration with policy-aware connectors for PII redaction and software guards for runtime encryption.\n- A comprehensive approach is needed for true end-to-end confidential pipelines, a concept explored in our pillar on Confidential Computing and Privacy-Enhancing Tech (PET).
Next-generation protection combines hardware with software-based distributed trust models for scalable AI.\n- Hybrid TEEs will orchestrate enclaves across CPUs, GPUs, and accelerators.\n- AI-Native PET Frameworks will embed protection into vector databases, embedding models, and the MLOps lifecycle.\n- This evolution is critical for governing data flows to third-party models (OpenAI, Anthropic) and is a core focus of our discussion on AI TRiSM: Trust, Risk, and Security Management.
Enabling Efficiency, Speed & Accuracy
We build AI systems for teams that need search across company data, workflow automation across tools, or AI features inside products and internal software.
Encrypting data at-rest is standard; the critical vulnerability is the moment it's decrypted for processing in memory, which is what Confidential Computing solves.
Confidential Computing protects data-in-use by isolating it within hardware-based Trusted Execution Environments (TEEs) during computation, closing the final gap where data is exposed in plaintext. This is non-negotiable for processing sensitive data in AI pipelines involving models like OpenAI GPT-4 or Anthropic Claude.
At-rest encryption is insufficient because data must be decrypted to be processed by the CPU, creating a window of exposure for memory scraping attacks or privileged insider threats. This gap renders compliance with regulations like the EU AI Act or GDPR incomplete.
The attack surface is expanding with complex AI workflows that chain multiple services. Data flowing between a vector database like Pinecone, an embedding model, and an LLM API is vulnerable at each handoff unless the entire pipeline is confidential. This is why a layered PET architecture is essential.
Evidence: A 2023 study by the Confidential Computing Consortium found that over 70% of data breaches involve the exploitation of data in memory. For AI systems handling PII, this risk is existential.
About the author
CEO & MD, Inference Systems
Prasad Kumkar is the CEO & MD of Inference Systems and writes about AI systems architecture, LLM infrastructure, model serving, evaluation, and production deployment. Over 5+ years, he has worked across computer vision models, L5 autonomous vehicle systems, and LLM research, with a focus on taking complex AI ideas into real-world engineering systems.
His work and writing cover AI systems, large language models, AI agents, multimodal systems, autonomous systems, inference optimization, RAG, evaluation, and production AI engineering.
How We Work
One-fit-all AI don't work for modern businesses. At Inferensys, we aim to understand your business & custom requirements; which we use to define most efficient agentic workflows, the data, and the tools for your business.
The first call is a practical review of your use case and the right next step.
