Prompt Engineering

These are foundational techniques for eliciting task performance without explicit training.

• Zero-Shot Prompting: Directly instructs the model to perform a task it wasn't explicitly trained for, relying on its broad pre-training (e.g., "Translate this to French: 'Hello world'.").
• Few-Shot Prompting: Provides the model with a small number of input-output examples (shots) within the prompt to demonstrate the desired task format and style before presenting the actual query. This is a form of in-context learning.

A technique that improves a model's performance on complex reasoning tasks by encouraging it to articulate its intermediate reasoning steps.

• Standard CoT: Append phrases like "Let's think step by step" to the prompt. The model generates a reasoning chain before the final answer.
• Manual CoT: The prompt designer provides a few examples of detailed reasoning within the few-shot examples.
• Automatic CoT: Uses LLMs to generate reasoning chains for examples automatically. This technique is critical for arithmetic, logical, and symbolic reasoning tasks, as it decomposes problems.

The practice of designing clear, structured commands and specifying exact output formats.

• Explicit Instructions: Use imperative verbs ("Summarize," "List," "Classify") and define scope ("in three bullet points").
• Role Assignment: Instruct the model to adopt a persona ("You are a senior software architect...") to bias its knowledge and tone.
• Structured Outputs: Demand specific formats like JSON, XML, or Markdown with defined keys. This is essential for deterministic parsing in automated pipelines. Example: "Return a JSON object with keys 'summary' and 'keywords'."

Techniques for specifying what the model should avoid generating or for enforcing hard constraints.

• Negative Prompts: Explicitly list undesired attributes, styles, or content (e.g., "no watermarks, no text, photorealistic"). Common in diffusion models.
• Constrained Decoding: Uses algorithms during token generation to enforce rules like keyword inclusion, grammar constraints, or preventing repetition. This moves beyond prompting into the sampling logic itself.
• Guardrails: Post-generation filters or validation steps that check outputs against a set of rules, rejecting or correcting non-compliant results.

A hybrid architecture that conditions a generative model on relevant information retrieved from an external knowledge source.

• Process: A user query triggers a retrieval system (e.g., a vector database) to fetch pertinent documents. These are injected into the prompt as context.
• Benefit: Drastically improves factual accuracy and reduces hallucinations by grounding generation in verifiable data.
• Prompt Role: The prompt must clearly delineate the context from the instruction (e.g., "Based on the following context: [retrieved docs]. Now answer: [user query]").

Advanced techniques where the model is prompted to analyze and improve its own outputs or processes.

• Self-Critique: The model is asked to evaluate its initial answer for errors, biases, or missing information, then produce a revised version.
• Automatic Prompt Optimization: Using the LLM itself to generate or refine prompts. For example, "Given this task description, write the most effective prompt for a language model."
• Program-Aided Language Models (PAL): The model is prompted to generate code (e.g., Python) that, when executed, yields the answer, offloading precise computation to an interpreter.

Effective prompts for image generation combine subject, style, and composition descriptors. Key elements include:

• Subject: The primary object, person, or scene (e.g., 'a cyberpunk cat').
• Style: Artistic medium or aesthetic (e.g., 'oil painting', 'photorealistic', 'in the style of Hayao Miyazaki').
• Composition: Camera angle, lighting, and framing (e.g., 'wide-angle lens', 'cinematic lighting', 'low-angle shot').
• Quality Modifiers: Terms like '4k', 'highly detailed', 'intricate' signal desired output fidelity. Precision reduces ambiguity, guiding the model's latent space more deterministically.

A technique to specify undesired elements, steering the model away from common failure modes or artifacts. By defining a negative prompt, the generation process is penalized for producing content matching those terms.

Common Uses:

• Avoiding distortions: 'disfigured hands', 'blurry', 'extra fingers'.
• Excluding styles: 'watermark', 'text', '3D render'.
• Removing unwanted objects: 'people', 'cars', 'buildings'.

This works by computing a weighted difference between the conditional (desired) and unconditional (negative) score estimates during sampling, effectively subtracting concepts from the latent trajectory.

A critical hyperparameter (often guidance_scale or cfg_scale) that controls the trade-off between creativity and adherence to the prompt. It amplifies the influence of the conditioning signal.

Mechanism: The model blends predictions from a conditional and an unconditional (no prompt) forward pass. A higher scale increases the weight of the conditional prediction.

Typical Ranges:

• Low (1-4): High diversity, low prompt fidelity. Often creative but ignores details.
• Medium (7-12): Standard range for Stable Diffusion, balancing creativity and control.
• High (12-20+): High prompt fidelity, but can reduce image quality and diversity, sometimes leading to over-saturated or 'burned' images.

Techniques for imposing structural control beyond text. ControlNet is a neural network architecture that adds spatial conditioning to pre-trained diffusion models (e.g., Stable Diffusion).

Conditioning Inputs:

• Canny Edge Maps: For precise shape control.
• Depth Maps: To enforce 3D structure and perspective.
• Human Pose Keypoints: For figure generation.
• Scribbles/Segmentation Maps: For rough layout.

ControlNet clones the model's encoder weights into a 'trainable copy' and a 'locked copy', learning conditional embeddings that are added to the main U-Net. Adapter layers (like T2I-Adapter) provide a lighter-weight alternative for similar control.

Syntax for fine-grained control over concept emphasis and blending.

Keyword Weighting:

• Emphasis: (keyword:1.5) increases attention; (keyword:0.8) decreases it.
• Alternate Syntax: keyword++ or keyword--.

Prompt Blending: Using the AND operator (in some implementations like SDXL) to fuse multiple concepts: a cat AND a dog attempts to blend both subjects.

Break & Composition: Using commas or line breaks to separate distinct clauses helps the model's cross-attention layers parse different aspects (subject, background, style) more effectively than a single run-on sentence.

Prompt engineering extends to other conditional generation tasks:

Image-to-Image: Using an input image as a conditioning signal, with a text prompt to guide the transformation. Strength parameters control how much of the original image is preserved.

Inpainting/Outpainting: Prompting guides content generation within a masked region or to extend an image beyond its borders.

Text-to-Video: Prompts must consider temporal consistency, often involving frames-per-second or motion descriptors (e.g., 'slow pan', 'smooth transition').

Audio Generation: Text prompts describe sound characteristics, genre, mood, and instrumentation (e.g., 'lo-fi hip hop beat with vinyl crackle').

A technique for controlling the output of conditional diffusion models by blending the predictions of a conditional and an unconditional model during the sampling process. This eliminates the need for a separate, pre-trained classifier.

• Mechanism: The model is trained to denoise with and without a condition (e.g., a class label or text prompt). During inference, the final denoising direction is extrapolated towards the conditional prediction and away from the unconditional one.
• Guidance Scale: A hyperparameter (often s) controls the strength of this extrapolation. Higher values increase adherence to the condition but can reduce sample diversity and quality.
• Primary Use: The standard method for controlling text-to-image models like Stable Diffusion, where the condition is a text embedding.

A neural network architecture that adds spatial conditioning controls to pre-trained text-to-image diffusion models. It enables precise structural control over generated images using inputs like edge maps, depth maps, or human poses.

• Architecture: A ControlNet clones the weights of a pre-trained diffusion model (e.g., Stable Diffusion's U-Net) into a "trainable copy" and a "locked copy." The trainable copy processes the conditioning image (e.g., a scribble) and its features are added to the locked copy via zero convolutions.
• Conditioning Types: Supports diverse inputs including Canny edges, HED boundaries, user scribbles, human pose keypoints, and depth maps.
• Impact: Allows for detailed compositional control, making diffusion models practical for tasks requiring specific layouts, such as product design or architectural visualization.

A neural network mechanism that allows a model to condition its generation process on a separate sequence of data by computing attention between two different sets of embeddings.

• Function: In a transformer or U-Net with transformer blocks, cross-attention layers enable the model to "attend to" a conditioning signal. For text-to-image models, image features (keys/values) attend to text token embeddings (queries).
• Role in Conditioning: It is the primary architectural component that allows diffusion models like Stable Diffusion to interpret and adhere to textual prompts. The strength of attention to specific tokens can be manipulated for prompt engineering.
• Broader Use: Found in all multimodal architectures where one modality (e.g., image, audio) conditions the generation of another (e.g., text).

A technique where users specify concepts or attributes they wish to avoid in the generated output, guiding the model away from undesired content.

• Implementation in CFG: In classifier-free guidance, negative prompting is implemented by using an "unconditional" embedding that represents the concept to avoid. The sampling process is guided away from this negative condition.
• Common Uses: Preventing common artifacts (e.g., "blurry," "deformed hands"), excluding unwanted stylistic elements, or ensuring the absence of specific objects or themes.
• Effectiveness: Highly dependent on the model's training data and the specificity of the negative terms. It is a fundamental tool for refining output in production text-to-image systems.

A parameter-efficient fine-tuning (PEFT) method that updates a pre-trained model by injecting trainable low-rank matrices into its layers, enabling adaptation to new tasks or styles with minimal new parameters.

• Mechanism: For a pre-trained weight matrix W, LoRA represents its update as W + BA, where B and A are low-rank matrices. Only B and A are trained, freezing the original W.
• Application to Prompting: While used for fine-tuning, LoRA's principles relate to prompt engineering by demonstrating how small, targeted interventions (like a well-crafted prompt) can steer a large, frozen model. Trained LoRAs can act as reusable "style prompts."
• Advantages: Dramatically reduces storage and computational cost compared to full fine-tuning, allowing many specialized adapters to be swapped in for a single base model.

A prompting technique for large language models that encourages the model to generate a step-by-step reasoning process before delivering a final answer, significantly improving performance on complex reasoning tasks.

• Prompting Style: Instead of asking "What is 12 * 15?", a CoT prompt would be "Calculate 12 * 15 step by step." The model is induced to output "12 * 10 = 120, and 12 * 5 = 60, so 120 + 60 = 180."
• Zero-Shot & Few-Shot: Can be elicited via a direct instruction ("Let's think step by step") or by providing few-shot examples that demonstrate the reasoning process.
• Connection to Engineering: Represents a high-level, structural prompt engineering strategy for decomposing problems. It is foundational for building reliable, transparent reasoning in agentic systems.