Data Retention Policy

The specific duration for which different classes of audit logs must be preserved. This is driven by regulatory mandates, business needs, and legal risk.

• Regulatory Mapping: Align periods with specific laws (e.g., GDPR's storage limitation, HIPAA's 6-year requirement, SOC 2's 90-day minimum).
• Data Classification: Tie retention to data sensitivity. Model access logs containing PII may have shorter lifespans than system-level security events.
• Granularity: Define periods per log type, not a single blanket rule. A chain of custody log may be kept longer than a performance metric.

A multi-layered storage architecture that balances access speed with cost efficiency over the log's lifecycle.

• Hot Storage: Real-time indexed logs (e.g., Elasticsearch) for immediate continuous auditing and alerting. Retention: days to weeks.
• Warm Storage: Searchable object stores (e.g., S3) for periodic analysis and e-discovery. Retention: months to a year.
• Cold Storage: Immutable archives (e.g., AWS Glacier, tape) for long-term compliance. Access latency is high, but cost is minimal. Often uses WORM technology.

The verifiable process of permanently deleting logs once their retention period expires, ensuring data minimization.

• Cryptographic Erasure: Destroy the encryption keys for a specific dataset, rendering it irrecoverable without needing to overwrite every bit.
• Physical Destruction: For end-of-life cold storage media, use degaussing or shredding.
• Certificate of Destruction: Generate a cryptographically signed, time-stamped record proving the deletion occurred, maintaining non-repudiation for the destruction event itself.

A mechanism to suspend the normal retention and destruction policy for specific logs when litigation or an investigation is reasonably anticipated.

• Automated Flagging: Integrate with legal systems to apply holds via metadata tags, not manual file copies.
• Immutability Override: A legal hold must prevent any automated deletion job from removing the targeted immutable audit trail entries.
• Chain of Custody Preservation: The hold process itself must be logged, documenting who initiated it, the scope, and the authorization, to withstand judicial scrutiny.

Implementing the retention policy through automated scripts and infrastructure-as-code, not manual runbooks.

• Lifecycle Policies: Use cloud-native lifecycle rules to automatically transition logs between storage tiers and trigger deletion.
• Policy as Code: Define retention rules in version-controlled files (e.g., Open Policy Agent) that are tested and deployed via CI/CD pipelines.
• Drift Detection: Continuously monitor configurations to ensure no bucket or database has been manually modified to violate the codified policy, a core tenet of compliance as code.

Techniques applied to retained logs to enable analysis while protecting sensitive personal data.

• Data Masking: Apply static or dynamic masking to fields like IP addresses or user IDs in warm storage copies used for analytics.
• Differential Privacy: Inject calibrated noise into aggregate queries run against audit logs, ensuring individual user behavior cannot be re-identified.
• Pseudonymization: Replace direct identifiers with pseudonyms, storing the mapping key separately with a stricter retention policy and tighter access control.