Amazon S3 (Simple Storage Service)

S3 stores data as objects within buckets. Each object consists of the data itself, a globally unique key, and customizable metadata. This is distinct from file systems (hierarchical directories) or block storage (raw disk volumes).

• Key: The unique identifier for the object within its bucket (e.g., projects/agent-memory/embeddings.json).
• Data: The actual content of the file, from 0 bytes to 5 TB.
• Metadata: Key-value pairs (system-defined like Content-Type, or user-defined) describing the object.
• Version ID: A unique string assigned when an object is created or overwritten, enabling object versioning.

Buckets are the fundamental containers for objects. Their names are globally unique across all of AWS and form part of the object's URL (https://<bucket-name>.s3.<region>.amazonaws.com/<key>).

• Region-Specific: A bucket is created in a specific AWS Region, governing its physical data location for latency and compliance.
• Flat Structure: While keys can include prefixes (like /), S3 has a flat namespace. Tools and consoles simulate folders using these key prefixes.
• Policies & Configuration: Bucket-level settings control access (via IAM and bucket policies), versioning, logging, lifecycle rules, and event notifications.

S3 is engineered for 11 9's (99.999999999%) durability and 99.99% availability over a given year for Standard storage.

• Durability: Protection against data loss. S3 automatically replicates data across a minimum of three physically separated Availability Zones (AZs) within an AWS Region.
• Availability: The proportion of time the service is accessible. S3's multi-AZ design ensures resilience to the failure of an entire data center.
• Storage Classes: Different classes optimize for cost vs. access frequency (e.g., S3 Standard for frequent access, S3 Glacier for archival), all while maintaining high durability.

S3 provides two read-after-write consistency models, a critical consideration for application design.

• Strong Consistency: For PUTS of new objects and DELETES, S3 now provides strong read-after-write consistency. After a successful write, any subsequent read request will immediately return the latest data.
• Eventual Consistency: Historically the default model. For overwrites (PUTS to an existing key) and listings, changes are propagated across all AZs and become visible to all clients eventually, typically within milliseconds. Applications must be designed to handle potential temporary stale reads in these cases.

All interactions with S3 occur via a secure, standardized HTTP/REST API. Every operation (GET, PUT, DELETE, LIST) is an HTTP request signed with AWS credentials.

• Authentication & Authorization: Controlled through a combination of:
  • IAM Policies: Attached to AWS users/roles for fine-grained permissions.
  • Bucket Policies: JSON-based resource policies attached directly to the bucket.
  • Access Control Lists (ACLs): A legacy, less granular permission mechanism.
  • Pre-signed URLs: Time-limited URLs granting temporary access to a specific object, useful for secure sharing.
• Encryption: Supports server-side encryption (SSE-S3, SSE-KMS, SSE-C) and client-side encryption for data at rest.

S3 Lifecycle configuration rules automate moving objects between cost-optimized storage classes or expiring them, which is essential for managing agent memory logs and historical data.

• Transition Actions: Automatically move objects to a different storage class after a defined period (e.g., move from S3 Standard to S3 Standard-IA after 30 days, then to S3 Glacier after 90 days).
• Expiration Actions: Permanently delete objects or non-current versions after a set time.
• Key Storage Classes:
  • S3 Standard: General purpose, high durability/availability/frequent access.
  • S3 Intelligent-Tiering: Automatically moves data between access tiers based on changing access patterns.
  • S3 Glacier Instant Retrieval: Archival with millisecond retrieval.
  • S3 Glacier Deep Archive: Lowest-cost storage for data accessed once or twice a year.

A data integrity protocol where all modifications are written to a durable log file before they are applied to the main data structures. This ensures recoverability in case of a crash.

• Analogy: Similar to a database transaction log.
• S3 Parallel: While S3 offers strong consistency for new object PUTs, architecting an agentic system often involves using S3 as the durable destination for checkpoints and state snapshots that are generated after a WAL process in the agent's operational memory.