Profile-Based Query Weighting Prompt for Hybrid Retrieval

This prompt is designed for infrastructure teams operating multi-stage, hybrid retrieval pipelines that must personalize search results at the backend routing layer. The core job-to-be-done is generating a weighted query strategy object that dynamically assigns priority to different retrieval backends—such as dense vector, sparse keyword (BM25), and graph traversal—based on a specific user's profile. This is not a query rewriting prompt; it assumes you already have candidate query variants and need to calibrate which retrieval path gets the most compute, the highest boost, or the first look for a given user and query combination. The ideal user is a search infrastructure engineer, a RAG platform architect, or an ML engineer responsible for retrieval orchestration who has access to user profile data, historical retrieval success metrics, and a clear map of available retrieval backends.

You should use this prompt when your system has heterogeneous retrieval backends with measurably different performance characteristics across query types and user segments. For example, a legal research platform might find that partners get better results from dense vector search for precedent queries, while associates benefit more from keyword search over internal memos. The prompt requires structured inputs: a user profile object containing role, department, and past retrieval success patterns; a set of candidate query variants; and a defined list of available backends with their known strengths. The output is a machine-readable weighting object that your retrieval orchestrator can consume directly to merge and rank results. Do not use this prompt for simple single-index retrieval, for users without profile history (cold-start scenarios), or when backend performance is uniform across query types—in those cases, static weights or a simpler routing heuristic will be more reliable and easier to debug.

Before implementing this prompt, ensure you have the evaluation infrastructure to measure whether the generated weights actually improve retrieval outcomes. The prompt includes eval criteria for weight calibration and backend performance alignment, but you must instrument your pipeline to track per-backend recall, latency, and user satisfaction. A common failure mode is generating weights that look plausible but don't reflect actual backend performance, leading to degraded results. Start by running this prompt in shadow mode against a sample of production traffic, compare the weighted results against a uniform baseline, and only promote to production when you have statistically significant improvement. If you lack per-user retrieval success data, invest in that instrumentation first—this prompt is ineffective without it.

This prompt template is designed to be placed directly into your orchestration layer. It instructs the model to analyze a user's profile, their raw query, and the available retrieval backends to produce a weighted query strategy object. The core job is to shift from a one-size-fits-all query to a personalized distribution of effort across dense, sparse, and other custom indexes. Before each call, replace every [PLACEHOLDER] with live data from your application context. The output is a structured JSON object that your retrieval router can consume directly to fan out requests with different weights.

textCopy
You are a retrieval strategist for a hybrid search system. Your task is to generate a weighted query strategy for a specific user query, optimized for the user's profile and our available retrieval backends.

## User Profile
- Role: [USER_ROLE]
- Department: [USER_DEPARTMENT]
- Recent Successful Backends (last 30 days): [RECENT_SUCCESSFUL_BACKENDS]
- Explicit Content Preferences: [USER_PREFERENCES]
- Access Scope Constraints: [ACCESS_SCOPE]

## Raw User Query
[USER_QUERY]

## Available Retrieval Backends
[AVAILABLE_BACKENDS]

## Instructions
1. Analyze the user's role, department, and preferences to understand their likely information needs and terminology.
2. Examine the raw query and determine which backends are most likely to return relevant, authorized results.
3. Generate a JSON object mapping each available backend to a weight between 0.0 and 1.0. The sum of all weights must equal 1.0.
4. For each backend with a weight > 0, produce a rewritten query variant optimized for that backend's retrieval mechanism (e.g., a natural language question for dense vector search, a keyword-rich string for sparse BM25).
5. If the user's access scope prohibits a backend from being useful, assign it a weight of 0.0 and provide a null variant.
6. Provide a brief `strategy_rationale` explaining the weighting decision.

## Output Schema
You must respond with a single valid JSON object conforming to this structure:
{
  "strategy_rationale": "string",
  "weighted_queries": [
    {
      "backend_name": "string",
      "weight": "number (0.0 to 1.0)",
      "query_variant": "string or null"
    }
  ]
}

To adapt this for production, ensure the [AVAILABLE_BACKENDS] placeholder is populated with a structured list of your actual index names and their types (e.g., dense_vector_v2, sparse_bm25_prod, graph_entity_index). The [RECENT_SUCCESSFUL_BACKENDS] field should be a pre-computed summary from your analytics pipeline, not raw logs. After receiving the output, validate the JSON schema strictly. If the weights do not sum to 1.0 (within a small epsilon like 0.01), reject the output and retry the call with a stronger constraint instruction. For high-stakes personalization, log the strategy_rationale alongside the final weights to enable offline auditing of why a particular backend was favored or suppressed for a given user.

This prompt is not a standalone query rewriter. It is a strategy generator that sits between user context assembly and retrieval execution. The prompt expects a structured user profile, a parsed query, and a list of available retrieval backends. Its output is a weighted strategy object that tells downstream orchestrators how to distribute the query across dense vector, sparse keyword, hybrid, or graph backends. The prompt should be called once per query turn, after user profile resolution and before any backend dispatch. It is a control-plane component, not a data-plane component, so latency tolerance is higher than for retrieval itself—budget 500–800 ms for the LLM call and plan to cache results for repeated queries within a session.

Integration pattern: Wrap the prompt in a lightweight service that accepts a UserProfile object, a QueryContext object, and a BackendRegistry list. The service should validate inputs before calling the model: confirm the user profile contains required fields (role, department, recent retrieval success rates, content preferences), confirm the query has been parsed into a normalized form, and confirm the backend list includes at least two backends with known performance characteristics. After receiving the model response, validate the output against a strict schema: the weighted_strategy object must contain a weights map where keys match backend IDs from the input, values are floats between 0.0 and 1.0, and the sum is within 0.95–1.05 tolerance. Reject strategies that assign zero weight to all backends or that assign disproportionate weight (>0.9) to a single backend without explicit justification in the rationale field. Log every strategy decision with the user profile hash, query fingerprint, and backend weights for offline calibration analysis.

Failure modes and recovery: The most common failure is stale profile data—when a user's role or preferences changed but the profile cache hasn't updated. Mitigate this by checking profile freshness before calling the prompt and forcing a profile refresh if the last update exceeds your configured TTL (typically 15–30 minutes for active users). The second failure mode is backend misalignment: the model assigns high weight to a backend that is currently degraded or returning empty results. Implement a circuit breaker that overrides the strategy if a backend's health check fails or its recent success rate drops below a threshold. The third failure mode is overfitting to historical success patterns—the model weights a backend heavily because it worked well in the past, even when the current query type is poorly suited to that backend. Mitigate this by including query-type classification in the input context and adding eval checks that compare strategy weights against a query-type-to-backend suitability matrix. For high-stakes deployments, route strategies with confidence scores below a threshold to a human review queue or fall back to a default equal-weight strategy.

Model selection and cost: This prompt requires strong instruction-following and structured output generation. Use a model with reliable JSON mode and low refusal rates on system-level tasks—GPT-4o, Claude 3.5 Sonnet, or Gemini 1.5 Pro are suitable. Avoid smaller models (under 7B parameters) for this task because they struggle with the multi-constraint reasoning required to balance profile signals, backend characteristics, and query features simultaneously. Cache identical (user_profile_hash, query_fingerprint, backend_list_hash) tuples for 5–10 minutes to reduce costs on repeated or similar queries. If cost is a primary concern, consider fine-tuning a smaller model on logged strategy decisions from a larger model, but only after collecting at least 10,000 validated strategy examples with backend performance outcomes.

Observability and eval loop: Log every strategy decision with the following fields: timestamp, user_id (hashed), query_fingerprint, backend_weights, rationale, model_version, prompt_version, and backend_performance (collected after retrieval completes). Use these logs to compute offline metrics: weight-to-performance correlation (do higher-weighted backends actually return better results?), strategy stability (do similar queries from the same user produce consistent weights?), and profile signal utilization (are all profile fields influencing the weights, or is the model ignoring some signals?). Run weekly eval suites that compare model-generated strategies against a golden set of human-annotated strategies for diverse user profiles and query types. If strategy accuracy drops below 85% agreement with human annotators, investigate prompt drift, profile schema changes, or backend performance shifts before the degradation affects users.

What to avoid: Do not call this prompt on every sub-query in a multi-hop retrieval plan—generate the strategy once per user turn and reuse it across sub-queries unless the user's intent clearly shifts. Do not skip output validation; malformed weights can silently degrade retrieval quality across all backends. Do not treat the model's weights as immutable—always allow the retrieval orchestrator to override weights based on real-time backend health signals. And do not deploy this prompt without a fallback strategy: if the model call fails, times out, or returns invalid output, fall back to a pre-configured default weight distribution based on query type classification alone.