URL Normalization

The scheme and host components of a URL are technically case-insensitive per RFC 3986, but servers may treat them differently. Standardization mandates lowercasing the entire scheme (HTTPS:// to https://) and host (Example.COM to example.com). The path, query, and fragment components are often case-sensitive depending on the server's file system, so lowercasing the path is a heuristic that must be applied with caution to avoid breaking Unix-based routes.

• Rule: HTTP://[email protected]/Path → http://[email protected]/Path
• Exception: Path case is preserved if the server is known to be case-sensitive.

Default transport-layer ports are removed to prevent duplication. An explicit default port is semantically equivalent to its omission. The standard defaults are port 80 for HTTP and port 443 for HTTPS. Non-default ports (e.g., 8080, 3000) are always retained as they signify a specific application server.

• Rule: https://example.com:443/page → https://example.com/page
• Rule: http://example.com:80/page → http://example.com/page
• Retain: https://example.com:8443/page remains unchanged.

A trailing slash indicates a directory index, while its absence suggests a file resource. Inconsistent usage creates two distinct URLs for the same content. The normalization choice—adding or removing—is a site-wide policy decision, not a universal rule. Apache often redirects directory paths to include the slash, while static file servers may strip it.

• Add: https://example.com/blog → https://example.com/blog/
• Remove: https://example.com/about/ → https://example.com/about
• Critical: Choose one policy and enforce it via 301 redirects.

Characters in URLs can be represented in their literal form or as percent-encoded triplets (e.g., %20 for a space). Unreserved characters (A-Z, a-z, 0-9, hyphen, period, underscore, tilde) should be decoded to their literal form. Reserved characters used without their special meaning should also be decoded. Additionally, percent-encoded triplets themselves should be uppercased for consistency.

• Decode: https://example.com/%7Euser → https://example.com/~user
• Uppercase: https://example.com/%2f → https://example.com/%2F
• Do not decode: Reserved characters acting as delimiters (e.g., %2F as a literal slash in a path segment).

The order of query parameters is functionally irrelevant on most application servers, yet ?a=1&b=2 and ?b=2&a=1 are distinct URLs. Normalization involves alphabetically sorting parameters by key while preserving duplicate keys and their original value order. This is a high-impact rule for e-commerce faceted navigation and tracking URLs.

• Sort: ?b=2&a=1&c=3 → ?a=1&b=2&c=3
• Preserve: ?a=1&a=2 remains in that order after sorting other keys.
• Remove: Empty parameters like ?a=&b=2 can be stripped to ?b=2.

The fragment component (everything after #) is never sent to the server in an HTTP request; it is purely a client-side directive for in-page navigation or JavaScript state. Consequently, all search engine crawlers ignore it for indexing purposes. Normalization strips the fragment entirely to prevent indexing multiple URLs that differ only by their hash.

• Strip: https://example.com/page#section-2 → https://example.com/page
• Exception: Single-page applications using hash-based routing (#!) require special handling via the AJAX crawling scheme, though this is now deprecated in favor of the History API.

A UI pattern allowing users to refine large product catalogs by applying multiple filters simultaneously (e.g., color, size, brand). Each filter combination generates a unique parameter-driven URL, creating exponential duplicate content risks. Normalization rules must define a canonical parameter order and strip irrelevant or default-value parameters to prevent millions of near-duplicate pages from being indexed.