Serverless

Serverless functions are invoked in response to specific events, not by continuously running processes. This is the core of the Function-as-a-Service (FaaS) model.

• Triggers: HTTP requests (via API Gateway), file uploads to object storage, database changes, message queue events, or scheduled cron jobs.
• Statelessness: Each function invocation is typically stateless; any required state must be stored externally in a database or cache.
• Example: An image upload to an S3 bucket triggers a Lambda function to generate thumbnails, which are then stored back in S3.

Cost is directly tied to resource consumption at a granular level, rather than paying for pre-allocated, idle capacity.

• Pay-Per-Use: Billing is typically calculated per millisecond of execution time and per gigabyte of memory allocated.
• No Idle Cost: There is no charge when the function is not running, eliminating the cost of idle servers.
• Example: AWS Lambda charges are based on the number of requests and the duration of code execution, measured in 1ms increments.

The cloud provider automatically scales the number of function instances up or down to match the incoming event rate, with no manual intervention required.

• Concurrency: Each event can trigger a new, isolated instance of the function, allowing for massive parallelization.
• Scale to Zero: When there is no traffic, the platform scales down to zero running instances.
• Limits: Providers impose concurrency and burst limits, which must be considered for high-traffic applications.

The cloud provider is responsible for all underlying server management, including provisioning, patching, operating system maintenance, and capacity planning.

• Abstraction Layer: Developers deploy code (functions or containers); the provider handles servers, virtual machines, and hypervisors.
• Shared Responsibility Model: Security responsibility is shared; the provider secures the cloud infrastructure, while the developer secures their code and data.
• Trade-off: This abstraction reduces operational overhead but can limit low-level control and debugging capabilities.

Cold start refers to the latency incurred when a function instance must be initialized from a dormant state to handle its first request. This is a critical performance consideration.

• Causes: Loading the function's runtime, dependencies, and model weights into memory.
• Mitigations: Provisioned concurrency (keeping instances warm), optimizing package size, and using lighter-weight runtimes.
• Impact: Particularly significant for large language model inference, where loading multi-gigabyte models can cause delays of several seconds.

Serverless function instances are transient and may be terminated by the provider at any time after execution. They are not designed for long-running processes or maintaining in-memory state.

• Design Implication: Applications must be designed for stateless execution. Session data, connection pools, and caches must be externalized to services like Redis or DynamoDB.
• Maximum Duration: Functions have strict maximum execution time limits (e.g., 15 minutes on AWS Lambda).
• Use Case Fit: Ideal for short-lived, bursty workloads like data transformation, real-time file processing, and API backends.

Serverless functions are ideal for hosting inference endpoints that experience sporadic or unpredictable traffic. The cloud provider automatically scales the underlying compute to zero when idle and spins up instances to handle incoming requests, eliminating the cost of idle GPU instances.

• Key Benefit: Perfect for applications with bursty traffic patterns, like chatbots or internal tools used during business hours.
• Consideration: Cold start latency can be significant for large models, as the function must load the model weights into memory. Techniques like provisioned concurrency or using smaller, optimized models (e.g., via quantization) mitigate this.

Serverless functions act as glue logic in ML pipelines, triggered by events in cloud storage or message queues. They perform stateless transformations on data before inference or on model outputs afterward.

• Common Triggers: A new file uploaded to cloud storage (e.g., for image resizing), a message in a queue (e.g., for text chunking), or a database change.
• Typical Tasks: Data validation, format conversion (JSON to Protobuf), feature engineering, embedding generation for Retrieval-Augmented Generation (RAG), or filtering/sanitizing model outputs.
• Architecture: Enables a clean separation of concerns, where the core model serving can be optimized separately from data wrangling logic.

Cron-like triggers can invoke serverless functions to run periodic batch jobs without managing a dedicated scheduler or compute cluster. This is cost-effective for workflows that run on a fixed schedule (e.g., nightly, weekly).

• Use Cases: Retraining a model on fresh data, running batch inference on a dataset, generating daily performance reports, or evaluating a new model version against a golden dataset.
• Execution: The function spins up, pulls code and data, executes the job, saves results (e.g., new model weights to a model registry, metrics to a database), and shuts down.
• Limitation: Jobs must fit within the provider's maximum execution time limit (often 15 minutes). Longer jobs require checkpointing or a different compute service.

In agentic architectures, a serverless function can serve as a secure, scalable router or orchestrator. It receives a natural language request, determines the necessary actions (tool calls), executes them sequentially or in parallel, and synthesizes a final response.

• Function as Agent Controller: The function maintains the execution loop, calling external APIs (e.g., database queries, web searches, software APIs) based on the LLM's reasoning. It enforces authentication, rate limiting, and error handling for all external calls.
• Security Boundary: The function provides a controlled environment for executing privileged operations, isolating the LLM from direct access to sensitive systems, which is a core principle of agentic threat modeling.

Lightweight serverless functions can be deployed as sidecars or stream processors to ingest telemetry data from live inference endpoints. They perform real-time analysis without impacting the primary serving path.

• Processing Streams: Ingest logs or metrics from an inference endpoint to calculate latency percentiles, track token usage, or detect hallucination patterns.
• Alerting: Evaluate incoming data against Service Level Objectives (SLOs) and trigger alerts (e.g., via PagerDuty or Slack) if error rates spike or latency degrades.
• Cost Tracking: Parse logs to attribute inference costs to specific users, projects, or API keys, feeding data into FinOps dashboards.

Serverless functions provide a simple, scalable way to expose public HTTP endpoints that act as webhook receivers. This is essential for integrating AI/ML systems with SaaS platforms and external event sources.

• Integration Points: Receive notifications from GitHub (code changes), Stripe (payments), Slack (user messages), or Twilio (SMS). The function can trigger an LLM pipeline in response—for example, summarizing a new pull request or generating a support ticket response.
• Characteristics: These endpoints must be highly available to not miss events, but traffic is often low-volume and sporadic, making serverless a cost-optimal choice. They handle authentication, payload parsing, and initial validation before delegating to core services.

The core process of deploying a trained machine learning model into a production environment where it can receive input data, perform inference, and return predictions via an API. For LLMs, this involves managing the autoregressive generation loop, handling prompt inputs, and streaming token outputs. It is the operational layer that serverless platforms abstract, providing the runtime for the model itself.

A hosted API, typically a URL, that exposes a machine learning model for making predictions. This is the primary interface for client applications to interact with a served model. In a serverless context, the cloud provider dynamically manages the scaling and availability of the infrastructure behind this endpoint.

• Key Feature: Abstracts the underlying compute (containers, GPUs).
• Serverless Benefit: Eliminates the need to provision and manage servers for the endpoint.

The latency incurred when a service, such as a serverless function or a model inference endpoint, must be initialized from a dormant state to handle a request. For LLMs, this is particularly impactful as it includes the time to:

• Load the multi-gigabyte model weights into GPU memory.
• Initialize the KV Cache.
• Warm up the inference engine. Serverless platforms use strategies like provisioned concurrency to keep a defined number of instances 'warm' to mitigate this penalty for latency-sensitive applications.

A critical inference optimization technique where incoming requests are dynamically grouped and processed together in a single forward pass through the model. This maximizes GPU utilization and throughput.

• How it works: Instead of processing requests one-by-one (static batching), the scheduler continuously adds new requests to the current batch as previous ones finish generation.
• Serverless Relevance: Essential for cost-effective serverless LLM serving, as it allows a single GPU instance to handle many concurrent users efficiently, directly impacting the cost per token.

A model compression technique that reduces the numerical precision of a model's weights and activations (e.g., from 32-bit floating-point FP32 to 8-bit integers INT8 or 4-bit NF4). This is a key enabler for serverless LLM deployment because it:

• Decreases memory footprint, allowing larger models to fit on less expensive GPUs.
• Reduces bandwidth for loading models, improving cold start times.
• Can accelerate inference on hardware that supports low-precision arithmetic. Post-Training Quantization (PTQ) is commonly used for serverless deployments as it doesn't require retraining.

A Kubernetes resource that automatically scales the number of pod replicas in a deployment based on observed metrics like CPU utilization, memory consumption, or custom application metrics (e.g., requests per second, queue length).

• Relation to Serverless: While serverless platforms (e.g., AWS Lambda, Google Cloud Run) provide built-in scaling, HPA is the fundamental scaling mechanism when deploying LLMs on Kubernetes using frameworks like KServe or Triton Inference Server. It brings serverless-like elasticity to containerized model deployments.