Prompt Engineering

The strategic placement of specific, high-signal nouns, adjectives, and verbs at the beginning of a prompt to establish the primary subject and its core attributes. This technique leverages the model's token-by-token processing to strongly bias the initial latent space representation.

• Example: "A majestic, photorealistic Siberian tiger, prowling through a misty pine forest at dawn, detailed fur, sharp focus" places the core subject (Siberian tiger) and key stylistic modifiers (majestic, photorealistic) first.
• Avoid vague or conflicting terms early in the prompt, as they can dilute the primary concept.

Adjectives and phrases that explicitly define the desired artistic style, medium, technical quality, and lighting of the output image. These modifiers directly condition the model's rendering pipeline.

• Artistic Styles: cinematic, impressionist oil painting, cyberpunk, Studio Ghibli style.
• Technical Quality: ultra detailed, sharp focus, 8k, professional photography.
• Lighting & Atmosphere: dramatic lighting, soft volumetric fog, golden hour, neon glow.
• Composition: wide angle shot, macro photography, low-angle view, rule of thirds.

A technique where you specify elements to avoid in the generated image. This provides negative conditioning, steering the model away from common artifacts, unwanted styles, or content. It is implemented by the model calculating a direction away from the concepts described in the negative prompt.

• Common Uses:
  • Artifacts: blurry, deformed hands, ugly, distorted, watermark, text.
  • Unwanted Styles: 3d render, cartoon, anime, low quality, pixelated.
  • Content Removal: people, buildings, cars (if generating a nature scene).

The use of special syntax to adjust the relative importance of different prompt components. This allows for fine-grained control over which concepts are emphasized during generation.

• Basic Weighting: (concept:1.3) increases emphasis; (concept:0.7) decreases it.
• Alternation Syntax: [concept A|concept B] can be used for the model to blend or choose between ideas.
• Break Symbol: :: or -- can be used to strongly separate distinct clauses, though its effect is model-dependent.
• Example: A (futuristic cityscape:1.2) at (night:1.1), [neon lights|holographic advertisements], (rain-slicked streets:0.9).

Organizing the prompt into logical, hierarchical sections to mimic the compositional process of an artist or photographer. This improves model comprehension of relationships between scene elements.

• General Structure: [Subject Description], [Action/Context], [Style & Medium], [Technical Quality], [Composition & Lighting].
• Example: An astronaut riding a horse, galloping on the surface of Mars, photorealistic NASA photography, ultra detailed, sharp focus, wide angle lens, Martian sunset in background.
• This format helps ensure all critical aspects of the desired image are explicitly addressed.

The process of using the output of one generation as the input or inspiration for the next, often combined with inpainting or img2img techniques. This allows for complex scene construction and error correction.

• Process:
  1. Generate a base image with a broad prompt.
  2. Use inpainting to modify specific regions with new, targeted prompts.
  3. Use img2img with a low denoising strength to refine details or adjust style while preserving composition.
• This technique is essential for achieving highly specific, multi-element scenes that are difficult to describe perfectly in a single prompt.

A critical hyperparameter in diffusion models that controls the strength of conditioning. It amplifies the influence of the text prompt on the generated image by scaling the difference between the conditional and unconditional noise predictions.

• Mechanism: At each denoising step, the model calculates a direction guided by the prompt. The CFG scale (e.g., 7.5) determines how far to move in that direction.
• Impact: Higher values increase adherence to the prompt but can reduce image diversity and sometimes introduce artifacts. Lower values yield more creative but less controlled outputs.

The neural mechanism that enables conditional generation by fusing textual guidance into the visual synthesis process. It allows the image generation pathway (e.g., a U-Net) to attend to the encoded text prompt embeddings at multiple layers.

• Function: At specific layers in the denoising network, a cross-attention block computes a weighted sum of the text embeddings, where the weights are determined by the current visual features. This injects semantic information from the prompt directly into the image formation process.
• Result: This is what allows prompts like "a red apple" to correctly associate the color attribute with the object, ensuring the generated apple is red.

A textual description of elements to avoid during image generation, providing negative conditioning to steer the model away from unwanted content.

• Usage: By specifying concepts like "blurry," "deformed hands," or "watermark," the model is guided to suppress the probability of those features appearing in the output.
• Technical Basis: During sampling, the model calculates a direction that moves away from the noise prediction associated with the negative prompt, effectively subtracting undesired concepts from the generation trajectory.

A parameter-efficient fine-tuning method for adapting large pre-trained models like Stable Diffusion to new styles or concepts without full retraining.

• Mechanism: LoRA injects trainable low-rank matrices into the cross-attention layers (and sometimes other layers) of a frozen base model. These small matrices are optimized on a small dataset of target images.
• Advantage: It creates very small checkpoint files (often 3-100 MB vs. several GB for a full model) that can be loaded and combined at inference time, enabling efficient personalization and style blending.

An automatic evaluation metric that quantifies the semantic alignment between a generated image and its text prompt.

• Calculation: The generated image and the text prompt are encoded into feature vectors using the CLIP model. The CLIP Score is the cosine similarity between these two vectors.
• Purpose: It provides a quantitative, model-based measure of how well the image matches the prompt's meaning, useful for benchmarking model improvements and tuning prompts. It correlates with human judgment of relevance.

Algorithms that define the iterative denoising process in diffusion models, controlling the trade-off between speed, quality, and determinism.

• Sampler: The core algorithm (e.g., DDIM, Euler, DPM++) that solves the reverse diffusion equation, determining how noise is removed at each step.
• Scheduler: Governs the noise schedule—the specific variance of noise added/removed at each step. It manages the progression from high to low noise levels.
• Combined Impact: The choice of sampler and scheduler directly affects inference speed, final image quality, and the number of denoising steps required.