Anatomy of a URL: Scheme, Host, Path, Query and Fragment

Every developer types, pastes and logs URLs all day, yet most of us only half understand them. We know the part before the first slash is "the domain" and the part after the question mark is "the parameters", and that is usually enough, until a redirect sends users to the wrong site, a query parameter vanishes in transit, a cache stores two copies of the same page, or a security review asks whether your allow-list check can be bypassed with a single @ sign. This guide takes a URL apart piece by piece. It is written for web developers, API authors, SEO practitioners and anyone who has ever stared at a long address and wondered which character means what. You will learn the vocabulary, the exact rules each component follows, how browsers resolve relative links, how origins and the same-origin policy are derived, where parsers disagree, and how to decompose URLs safely in JavaScript and Python. All the code in this article was run, and the outputs shown are the real ones.

URI, URL and URN: what are we talking about?

Three similar acronyms cause most of the confusion, so start with them.

A URI (Uniform Resource Identifier) is a string that identifies a resource. It follows a generic syntax defined in RFC 3986, and it always begins with a scheme, a short name followed by a colon. A URL (Uniform Resource Locator) is a URI that also tells you how to locate the resource, usually by naming a protocol and a network location. A URN (Uniform Resource Name) is a URI that names something persistently without saying where it lives, such as urn:isbn:9780141036144, which identifies a book regardless of which shop sells it.

So every URL is a URI, every URN is a URI, and the two are siblings. In everyday web work the distinction rarely matters, and modern specifications have largely stopped trying to enforce it. The WHATWG standard that browsers implement is simply called the URL Standard, and it treats "URL" as the umbrella term for all of these strings. This article follows that practice: when it says URL, it means anything you could pass to new URL().

There is one more term worth knowing. A URL reference is a string that may be either absolute, with a scheme, or relative, without one. The href attribute of a link accepts a reference, and the browser turns it into an absolute URL before using it. We cover that process in detail below.

A short history

The idea of a uniform, text-based address for anything on a network was part of the original design of the World Wide Web around 1990 to 1991, and early documents called these addresses "UDIs" before the names settled. The first formal specification of URLs came from the IETF in the mid 1990s, and the syntax was later consolidated and generalised into RFC 3986, published in 2005, which remains the reference for the generic URI grammar.

For years, browsers did not exactly follow that document. Real-world pages contained backslashes where slashes belonged, spaces in paths, missing slashes after the scheme and many other deviations, and browsers learned to accept them. Each browser accepted a slightly different set. To end the divergence, the WHATWG published the URL Standard, a "living standard" that describes, step by step, an algorithm for parsing any input string into a URL record, including how to recover from malformed input. It is the specification behind the URL object in browsers and in Node.js. The practical result is that the browser's behaviour is now well defined, but it is not identical to RFC 3986, a gap we examine later because it has real security consequences.

For a broader overview, the Wikipedia article on URLs is a good starting point, and the MDN reference for the URL API documents the object you will use in code.

The anatomy at a glance

Here is a URL that exercises every component:

https://user:pa55@shop.example.com:8443/a/c/d.html?q=url+parser&tag=a&tag=b#section-2
\___/   \__/ \__/ \_____________/ \__/ \________/ \____________________/ \_______/
scheme  user pass     host        port   path            query            fragment

The first three parts after the scheme, user, password, host and port, form the authority. Together with the scheme, the authority decides which server receives the request. The path, query and fragment then say what to ask for, how to refine the request, and where to look inside the response.

The punctuation is the grammar. A colon ends the scheme. Two slashes introduce the authority. An @ ends the userinfo. A colon inside the authority separates the host from the port. The first single slash after the authority starts the path, the first ? starts the query, and the first # starts the fragment. Each delimiter is meaningful only in its own context, which is why : can appear freely inside a query value but not inside a hostname.

Here is what a real parser reports for a similar URL. The JavaScript below was run on Node.js and the values are its actual output:

const u = new URL(
  "https://user:pa55@Shop.Example.com:8443/a/b/../c/./d.html?q=url+parser&tag=a&tag=b&empty=&x=%C3%A9#section-2"
);
for (const k of ["href", "protocol", "username", "password", "host",
                 "hostname", "port", "pathname", "search", "hash", "origin"]) {
  console.log(k.padEnd(9), JSON.stringify(u[k]));
}

Property	Value
href	https://user:pa55@shop.example.com:8443/a/c/d.html?q=url+parser&tag=a&tag=b&empty=&x=%C3%A9#section-2
protocol	https:
username	user
password	pa55
host	shop.example.com:8443
hostname	shop.example.com
port	8443
pathname	/a/c/d.html
search	?q=url+parser&tag=a&tag=b&empty=&x=%C3%A9
hash	#section-2
origin	https://shop.example.com:8443

Notice what the parser did without being asked. It lowercased the host, so Shop.Example.com became shop.example.com. It resolved the dot-segments in the path, so /a/b/../c/./d.html became /a/c/d.html. It left the query exactly as written, including the + and the percent-escape. The next sections explain each of those behaviours.

Scheme: the first decision

The scheme comes first and ends at the first colon. It consists of a letter followed by letters, digits, plus signs, hyphens and periods, and it is case-insensitive, so HTTP://EXAMPLE.COM/ and http://example.com/ are the same. The WHATWG parser normalises it to lowercase, which the code confirmed: HTTP://EXAMPLE.COM/%7euser parsed to http://example.com/%7euser.

The scheme tells the software how to interpret the rest of the string. It does not always mean a network request. Some schemes you meet constantly:

Scheme	Meaning	Notes
http	Hypertext transfer, unencrypted	Default port 80
https	Hypertext transfer over TLS	Default port 443
file	A path on a local file system	Usually no host; file:///etc/hosts
mailto	An email address	Opaque path plus optional query for subject and body
data	Content embedded in the URL itself	Type and encoded data follow the colon
javascript	A script to execute	Dangerous, see the security section
ws, wss	WebSocket, plain and over TLS	Default ports 80 and 443
ftp	File transfer	Listed as special by the standard, but browsers no longer support it

Special schemes and opaque paths

The WHATWG standard divides schemes into "special" ones, which have defined default ports and a hierarchical structure, and everything else. The special schemes are http, https, ws, wss, ftp and file. For these, the parser knows how to handle backslashes, requires a host (except for file), and applies host normalisation.

Other schemes, such as mailto, data and javascript, have what the standard calls an opaque path: after the colon there is no //, no authority, and the remainder is treated as a single unstructured string. Our test confirms what that means in practice. For mailto:ada@example.com?subject=Hi%20there the host is empty and the origin is the string null. The same is true for data:text/plain;base64,SGVsbG8= and javascript:alert(1).

A data: URL is worth a closer look because it shows how much a URL can carry. The text after the colon is a media type, an optional ;base64 marker, a comma, and then the data itself. The example above decodes to the five letters "Hello". Data URLs are handy for tiny inline images and test fixtures, but they bloat HTML and cannot be cached separately from the page that contains them.

Case and the colon

Scheme parsing is strict about the colon but lenient about everything after it. A string such as localhost:3000/path is a classic trap: a parser reads localhost as a scheme and 3000/path as an opaque path, because the grammar says anything before the first colon that looks like a scheme is a scheme. Browsers address this by treating bare text typed into the address bar differently from URL strings passed to APIs. If you pass localhost:3000 to new URL(), it does not throw; it parses as a URL with the scheme localhost:. Always include http:// when you build a URL in code.

Userinfo: credentials in the address

Between the // and the host there may be a userinfo component, written as username:password@. It dates from the days when ftp://user:pass@host/ was a common way to log in. The two halves are available in JavaScript as url.username and url.password.

Putting a password in a URL is a bad idea today. URLs end up in browser history, server access logs, proxy logs, analytics tools and the Referer header, so a secret in the userinfo leaks widely. RFC 3986 itself discourages the user:password form for exactly this reason. Browsers also strip or warn about credentials in many contexts. Treat userinfo as a legacy feature you need to understand mainly because attackers use it for spoofing, which we address in the security section.

One practical point: everything before the last @ in the authority belongs to userinfo, and everything after is the host. The URL https://example.com@evil.test/ therefore connects to evil.test, with the username example.com. We ran it:

new URL("https://example.com@evil.test/").host   // "evil.test"

Host: names, IPv4 and IPv6

The host says which machine to contact. It takes one of three forms: a registered name such as shop.example.com, an IPv4 address, or an IPv6 address in square brackets.

Domain names

Domain names are case-insensitive, so parsers lowercase them. Labels are separated by dots, and the DNS limits each label to 63 octets and the whole name to 253 characters in its text form, as defined by the DNS specifications rather than by the URL grammar.

Names containing non-ASCII characters are converted to an ASCII form called Punycode before they reach the network. Our run shows the conversion for a German example: https://bücher.example/ became https://xn--bcher-kva.example/. The xn-- prefix marks a label that has been encoded. Browsers often display the Unicode form in the address bar but always send the ASCII form on the wire. This is also the source of the "homograph" problem, where a name mixing look-alike letters from different alphabets can imitate a trusted site. Browsers apply their own display rules to reduce that risk, but you should not rely on eyeballing a hostname to judge it.

IPv4 addresses

A dotted quad such as 192.168.0.1 is the familiar form. The surprise is that the WHATWG parser also accepts several older notations and normalises them. Our test shows two:

http://0x7f.1/       ->  http://127.0.0.1/
http://2130706433/   ->  http://127.0.0.1/

The first uses a hexadecimal part and omits the middle octets, the second writes the whole address as a single decimal integer. Both mean the loopback address. This matters for security: a filter that looks for the literal text 127.0.0.1 in a hostname misses these variants, and a server-side request forgery check that compares strings instead of parsed addresses can be bypassed. Always normalise with a real parser, then compare the resulting address.

IPv6 literals

An IPv6 address contains colons, which would collide with the port separator, so URLs wrap it in square brackets. The port, if any, follows the closing bracket:

https://[2001:db8::1]:8443/x

The parser keeps the brackets in host, giving [2001:db8::1]:8443. The address 2001:db8:: belongs to a range reserved for documentation, which makes it safe to use in examples. In Python, urlsplit strips the brackets for you: hostname returned 2001:db8::1 and port returned the integer 8080 for the input http://[2001:db8::1]:8080/x. If you build URLs from addresses yourself, remember to add the brackets, or the result will be ambiguous.

Port: the default you never type

The port selects a service on the host. If omitted, the scheme's default applies: 80 for http and ws, 443 for https and wss, and 21 for ftp. The WHATWG parser goes one step further and removes a port that equals the default, so the redundant form collapses:

http://example.com:80/      ->  http://example.com/
https://example.com:443/    ->  https://example.com/
http://example.com:8080/    ->  http://example.com:8080/

This is why url.port is an empty string for most real-world URLs, and why you should not build a string by concatenating hostname and port yourself. Use host, which includes the port only when it is not the default. Ports must be integers from 0 to 65535. Python's urlsplit is lenient while parsing, but reading .port on http://example.com:99999/ raised ValueError: Port out of range 0-65535, so wrap the access in a try block when the input is untrusted.

Path: hierarchy, segments and dot-segments

The path begins at the first / after the authority and ends at the first ? or #. In a hierarchical URL it is a sequence of segments separated by slashes. The standard does not say a path maps to a file on disk. That is a convention of simple web servers. A path can name a route handled by application code, an object key in cloud storage, or an API resource.

Paths are case-sensitive as far as the URL standard is concerned, even though the scheme and host are not. /About and /about are different URLs, and whether the server treats them as the same page is a server decision. For SEO, you should pick one form and redirect the other, a point we return to later.

Dot-segments

The segments . and .. mean "this directory" and "the parent directory". Parsers remove them before the request goes out. In our example, /a/b/../c/./d.html became /a/c/d.html: the .. cancelled b, and the lone . vanished. Python's urljoin applies the same algorithm when it merges a relative reference, as shown by urljoin("https://example.com/a/b/../c/./d", "e"), which returned https://example.com/a/c/e.

Two details trip people up. First, removal happens inside the client, so the server never sees .. in a normal request. Percent-encoded variants such as %2e%2e are also treated as dot-segments by the WHATWG parser, which is one reason path-traversal filters must work on the decoded and normalised path rather than the raw text. Second, a .. that would climb above the root is simply ignored, so https://example.com/../x becomes https://example.com/x.

Empty segments and trailing slashes

A path may contain empty segments. /a//b/ stays exactly that: our test printed /a//b/ unchanged. Two slashes are not collapsed, and servers differ in whether they treat /a//b/ as /a/b/. A trailing slash is also significant to the URL itself. http://a.test/a and http://a.test/a/ compared as different strings in our test, which printed false. The path of http://a.test with no slash at all was normalised to /, because a URL with a special scheme always has at least that one slash.

Trailing slashes also change how relative links resolve, because the "directory" of a URL is everything up to its last slash. We explain that in the next section.

Percent-encoding in paths

Characters outside a safe set are written as a percent sign and two hexadecimal digits representing a byte, with non-ASCII characters first converted to UTF-8. Our run shows the parser encoding spaces automatically: https://example.com/a b?x=1 2#f g became https://example.com/a%20b?x=1%202#f%20g. Note that it did not touch existing escapes, which is why %7e remained lowercase in the earlier example. The parser does not decode %7e into ~, because decoding in general can change meaning: an encoded slash %2F is part of a segment, while a literal slash is a separator.

Python has no automatic encoding step in urlsplit, so you do it yourself with quote. The default keeps / safe, which is right for whole paths:

quote("a b/é?")              # 'a%20b/%C3%A9%3F'
quote("a b/é?", safe="")     # 'a%20b%2F%C3%A9%3F'

The second form also escapes the slash, which is what you want when a single value, such as a filename, goes into one segment.

Path parameters

Some old URLs carry parameters inside a path segment after a semicolon, such as /a/c/d.html;v=1. Python's urlsplit leaves these in path, as the earlier output showed, while urlparse splits them into a separate params field for the last segment. Most modern applications ignore the feature, but you will still see it in session identifiers added to old Java applications.

Query: the part after the question mark

The query starts at the first ? and ends at the first #. The URL standard treats it as an opaque string. The familiar key=value&key=value structure is a convention layered on top, specified for HTML form submissions as application/x-www-form-urlencoded. This distinction explains why different libraries behave differently on awkward input.

Parsing rules

The convention works like this. Split the string on &, split each piece at the first =, replace + with a space, and percent-decode both halves. Our JavaScript run applied it to q=url+parser&tag=a&tag=b&empty=&x=%C3%A9:

[...u.searchParams]
// [['q','url parser'],['tag','a'],['tag','b'],['empty',''],['x','é']]

Notice four behaviours. The + became a space in the value. The percent-escape %C3%A9 decoded to é. The key empty with nothing after the equals sign produced an empty string. And tag appears twice, with both entries kept in order.

Repeated keys and arrays

The query grammar has no concept of arrays or nested objects. Frameworks invent conventions on top of it:

Convention	Example	Used by
Repeated key	tag=a&tag=b	HTML forms with multiple selections, many APIs
Bracket suffix	tag[]=a&tag[]=b	PHP and Rails-style parsers
Indexed brackets	tag[0]=a&tag[1]=b	Some frameworks for nested data
Comma separated	tag=a,b	Compact APIs, but commas inside values need care

The URL API treats the brackets as plain characters. searchParams.get("tag") returned only the first value, a, while getAll("tag") returned ['a','b']. Python behaves in the same spirit: parse_qs returns a dictionary of lists, so every key maps to a list even when it appears once:

{'q': ['url parser'], 'tag': ['a', 'b'], 'x': ['é']}

Notice that empty is missing. By default, parse_qs discards parameters with blank values, which surprises people. Passing keep_blank_values=True includes 'empty': ['']. If you need order and duplicates exactly, parse_qsl returns a list of pairs.

Order matters sometimes

Order is preserved by both parsers, and there are cases where it matters. When a key repeats, the first-wins rule of get and the last-wins behaviour of many server frameworks can disagree, which is a classic source of bugs and of security bypasses known as HTTP parameter pollution. If two components of your system read the same query differently, one can be tricked into validating a value that the other does not use. Document which value wins, or reject duplicates for keys that should be unique.

For caching, order matters in the opposite direction. ?a=1&b=2 and ?b=2&a=1 mean the same thing to your application but are different URLs to a cache. URLSearchParams.sort() gives a canonical order. In our test, after appending a bracketed key, setting a to 9 and sorting, the output was a=9&b=2&c=+x+y&list%5B%5D=1. The set call replaced both original a entries with one, and the brackets were escaped as %5B%5D.

Encoding pitfalls: + versus %20

In a query string, a space is traditionally written as +. In a path it must be %20, and a literal + there means a plus sign. URLSearchParams always serialises spaces as +, while encodeURIComponent always uses %20. Our run shows the difference for the same input a&b=c d:

URLSearchParams  ->  q=a%26b%3Dc+d
encodeURIComponent -> a%26b%3Dc%20d
encodeURI          -> a&b=c%20d/%C3%A9

Both forms decode correctly on a form-aware server. Trouble starts when a server decodes a query with a path-style decoder: + stays a plus, and your data gains a stray symbol. The third line shows why encodeURI is the wrong tool for values. It leaves & and = untouched, because it is designed for whole URLs, and a value containing them would split into extra parameters. Use encodeURIComponent or URLSearchParams for values, never encodeURI.

Fragment: the part the server never sees

The fragment starts at the first # and runs to the end of the string. It identifies a secondary resource, typically an element with a matching id, a text range, or a client-side route.

The key property is that the fragment is not sent to the server. The browser removes it before building the request line. We verified it by running a tiny HTTP server that logs req.url and then requesting http://127.0.0.1:PORT/page?a=1&a=2#secret-section with fetch. The server printed:

server saw: /page?a=1&a=2

The #secret-section part never arrived. Three consequences follow.

First, a fragment is a poor place to put anything the server needs, such as a session identifier or a filter that must influence the response. Second, it is a reasonable place for data you want to keep away from server logs, because the browser does not transmit it, though scripts on the page, browser extensions and history can still read it, so it is not a secret store. Third, single-page applications use fragments for routing because changing only the fragment does not reload the page. Jumping from /docs#a to /docs#b triggers a hashchange event and scrolling, but no network request.

For SEO, search engines generally treat URLs that differ only by fragment as the same page, so do not use fragments to distinguish content that should be indexed separately.

Relative URLs and reference resolution

Links in HTML are mostly relative: href="setup.html", src="/img/logo.png". The browser resolves each against a base URL, usually the address of the current document or the value of a <base> element. The algorithm is defined in RFC 3986 and in the WHATWG standard, and it is mechanical enough to learn.

We resolved a set of references against https://example.com/docs/guide/intro.html?v=1#top in both languages. JavaScript's new URL(ref, base) printed:

Reference	Resolved URL
setup.html	https://example.com/docs/guide/setup.html
./setup.html	https://example.com/docs/guide/setup.html
../api/	https://example.com/docs/api/
/root.html	https://example.com/root.html
//cdn.test/x.js	https://cdn.test/x.js
?v=2	https://example.com/docs/guide/intro.html?v=2
#bottom	https://example.com/docs/guide/intro.html?v=1#bottom
(empty string)	https://example.com/docs/guide/intro.html?v=1
https://other.test/	https://other.test/

Python's urljoin produced identical results for every reference it was given. The rules these examples illustrate:

• A reference with a scheme is already absolute and replaces the base entirely.
• A reference starting with // is "scheme-relative" and inherits only the scheme. It was common for loading assets from a CDN, though with HTTPS everywhere you should write the full https:// form.
• A reference starting with / replaces the whole path, keeping the scheme and authority.
• A plain relative path replaces everything after the last slash of the base path, then dot-segments are removed.
• A reference starting with ? keeps the base path and replaces the query.
• A reference starting with # keeps everything and replaces only the fragment.
• An empty reference points to the base document itself, without its fragment.

Why the trailing slash changes everything

Consider a page at https://example.com/docs/guide, with no trailing slash. A link to setup.html resolves to https://example.com/docs/setup.html, because guide is treated as a file name and replaced. If the page were https://example.com/docs/guide/, the same link would resolve to https://example.com/docs/guide/setup.html. This is the most common reason that relative stylesheets and images break when a site changes its URL structure. Using root-relative paths such as /img/logo.png, or setting a <base href>, removes the dependency on the trailing slash. Be careful with <base>, because it also changes how fragment-only links like #top behave: they resolve against the base, not the current page, which can send users away from the page they are on.

Origin and the same-origin policy

The origin of a URL is the triple of scheme, host and port. It is the unit the browser uses to decide which pages may read each other's data. The same-origin policy on MDN explains the rules in depth. In short, script running on one origin cannot freely read responses or DOM content that belong to another.

Our parser output shows how it works. For the main example the origin was https://shop.example.com:8443: the userinfo, path, query and fragment are not part of it, and the default port disappears. Comparisons follow:

Page URL	Other URL	Same origin?	Reason
https://example.com/a	https://example.com/b?x=1	Yes	Same scheme, host and port
https://example.com	http://example.com	No	Different scheme
https://example.com	https://www.example.com	No	Different host
https://example.com	https://example.com:8443	No	Different port
https://example.com	https://example.com:443	Yes	443 is the default and is dropped

Origins for file, data, mailto and javascript URLs came out as null in our tests. In practice, that means an "opaque origin" that does not match anything else, including itself in comparisons. file URLs are an area where browsers have varied in strictness, so do not rely on a particular behaviour.

Also note that site is a looser concept than origin. A site groups hosts that share a registrable domain, so https://a.example.com and https://b.example.com are different origins but the same site. Cookies and several newer security features use the site notion, while scripting access uses the origin. Mixing them up is a common source of confusion when debugging CORS and cookie issues.

To compare origins in code, never compare raw strings. Parse both URLs and compare origin:

const sameOrigin = (a, b) => new URL(a).origin === new URL(b).origin;

WHATWG versus RFC 3986: where they differ

If both documents describe URLs, why do we need two? Their goals differ.

RFC 3986 defines a grammar. It describes what a valid URI looks like and how a conforming producer should write one. It says little about what to do when input is invalid. The WHATWG URL Standard defines an algorithm. It takes any string, valid or not, and defines a result, or a failure, so that every browser gives the same answer. It is concerned with compatibility with how the web actually behaves.

At a conceptual level, the differences you will notice are these:

• Error recovery. WHATWG tolerates spaces, backslashes, extra slashes and stray characters by encoding or rewriting them. RFC 3986 would call those strings invalid.
• Backslashes. For special schemes, WHATWG treats \ like /. In our run, https://example.com\@evil.test/ parsed with the host example.com and the path /@evil.test/. A strict RFC reading would treat the backslash as an illegal character in the authority.
• IPv4 forms. WHATWG accepts hexadecimal, octal and single-integer notations and turns them into dotted decimal. RFC 3986 only defines the dotted-decimal form for IPv4 literals, and anything else would be a registered name.
• Host handling. WHATWG lowercases hosts, converts Unicode to Punycode and rejects some forbidden characters. RFC 3986 leaves internationalisation to other documents.
• Default ports. WHATWG removes default ports when serialising.
• Query and fragment encoding. WHATWG defines exactly which characters get percent-encoded in each component, with different sets for path, query and fragment.
• Opaque and special schemes. WHATWG classifies schemes this way; RFC 3986 has no such split.
• Terminology. WHATWG calls the result a URL record, and does not distinguish URI from URL.

A live parser differential

The differences are not theoretical. We parsed the same hostile-looking string in two parsers. The string was https://example.com\@evil.test/. In JavaScript, the WHATWG host was example.com. In Python's urllib.parse.urlsplit, which follows the older RFC-style approach more closely, the hostname was evil.test:

urlsplit("https://example.com\\@evil.test/").hostname   # 'evil.test'

Both programs are "correct" by their own rules. The danger is when one component validates a URL with one parser and another component fetches it with a different one. If your front-end gateway, written in JavaScript, approves the host as example.com and a Python worker then requests it as evil.test, the check is bypassed. Defence: parse once, keep the parsed object, and pass the normalised string to downstream code. Better still, reject URLs containing characters like backslashes, control characters and userinfo before doing anything else.

Real-world use cases

Understanding the pieces pays off in routine work.

Routing and link building. Frameworks split paths into segments to find a handler, and fill in :id parameters. Knowing that an encoded slash is data, not a separator, prevents bugs in routers that decode too early.

Tracking and campaign links. Marketing URLs carry parameters that analytics tools read. Keeping them in the query, not the path, lets caches and canonical tags ignore them.

OAuth and sign-in flows. A redirect URI is compared against a registered value, usually as an exact string or an origin plus path. Subtle differences such as trailing slashes, default ports or case in the host can cause "redirect URI mismatch" errors.

API design. Resource identifiers belong in the path, filters and pagination in the query. Our JSON formatting guide and GraphQL formatting guide cover the payload side of the same conversation.

Caching and CDNs. A cache key is normally the URL, so the order of parameters, a trailing slash or a stray tracking parameter can fragment the cache.

Security tooling. Link scanners, allow-lists, proxies and webhook validators all parse URLs and must do so consistently.

Debugging. When a request misbehaves, decomposing the URL into parts often shows the answer in seconds: the wrong port, a doubled slash, an unencoded ampersand or a missing parameter.

Sharing state. Query strings and fragments store filters, search terms and map positions so that a link reproduces the view. This is both powerful and fragile, as the length section below explains.

Canonical URLs, trailing slashes and SEO

Search engines treat each distinct URL as a separate page unless told otherwise. Duplicate URLs split signals and waste crawl budget. The usual sources are:

• http and https versions of the same page
• www and non-www hosts
• Trailing slash and non-trailing slash forms
• Upper and lower case paths
• Parameter order and tracking parameters
• Index file names such as /index.html next to /

Three tools resolve duplicates, and a good setup uses more than one:

1. Redirects. Send a permanent redirect from every variant to the one preferred form. This is the strongest signal, and it also fixes bookmarks and links.
2. The canonical link element. Put <link rel="canonical" href="..."> in the page head with an absolute URL for the preferred version. It is a hint, which search engines usually follow but are not obliged to.
3. Consistency. Use the preferred form in internal links, sitemaps and structured data.

Which trailing-slash policy is better? Neither is inherently superior, and search engines do not rank one above the other. The rule that does hold is to choose one and apply it everywhere. A common convention is a trailing slash for pages that act as directories and none for files with extensions, and many static site generators default to the directory style because it avoids ambiguity with relative links. Remember that the root path is always /, so https://example.com and https://example.com/ are the same URL after parsing.

Keep URLs readable. Short, lowercase, hyphen-separated words are easy to share and easy to guess from. Avoid session identifiers in the path or query, because they create an endless supply of duplicate pages. Use fragments only for in-page navigation, and do not depend on them for content that needs its own indexable page.

How long can a URL be?

There is no single limit in the URL syntax. Limits come from the software on the path: browsers, proxies, CDNs, web servers and application frameworks each impose their own, and they differ between products and versions. Many servers reject request lines beyond a few kilobytes by default, and some intermediaries are stricter. Treat any specific number you read online as a configuration default, not a law, and check your own stack if you approach long URLs.

The practical advice is more useful than a number:

• Keep URLs short enough to share. Long URLs get truncated or wrapped in chat tools, emails and documents.
• Do not store large state in a query string. If you need to send a lot of data, use a POST body.
• Be careful with filters and arrays. A form with dozens of checkboxes can produce a long query string quickly. Our run of ten a=1& pairs produced a URL of 61 characters, so the growth is linear and easy to estimate for your own parameters.
• Test the failure mode. A URL that exceeds a limit often produces a generic error, a 414 response or a silent truncation, depending on the layer.

Security pitfalls

URL handling is a rich source of vulnerabilities because many problems come from small inconsistencies.

Userinfo spoofing

The URL https://example.com@evil.test/ looks like a link to example.com. It is a link to evil.test, as the parser output earlier confirmed: host evil.test, origin https://evil.test. Phishing messages use this trick because readers parse URLs left to right and stop at the familiar name. Browsers now often warn about or strip credentials in navigation URLs, but your own code should never assume that the text before the first slash is the host.

A related trick is a host that starts with a trusted name, such as app.example.com.evil.test, or that embeds it in a path or query: https://evil.test/?u=app.example.com. A check using startsWith or includes on the raw string accepts all of these. Compare the parsed hostname for exact equality, or check that it ends with a dot plus the trusted domain.

Open redirects

An open redirect is an endpoint like /login?next=... that forwards the user to whatever address the parameter contains. Attackers use it to make a phishing link appear to belong to your domain, and to steal OAuth codes when a redirect URI is validated loosely. A robust fix is to resolve the target against your own origin, then verify the result. This function was run against hostile inputs:

function safeRedirect(target, allowedHost, base = "https://app.example.com") {
  let u;
  try { u = new URL(target, base); } catch { return "/"; }
  if (u.protocol !== "https:" || u.hostname !== allowedHost) return "/";
  return u.pathname + u.search + u.hash;
}

The results for seven inputs:

Input	Output
/dashboard?x=1	/dashboard?x=1
//evil.test/x	/
https://app.example.com/ok#a	/ok#a
https://app.example.com@evil.test/	/
javascript:alert(1)	/
/\evil.test	/
https://app.example.com.evil.test/	/

The /\evil.test case deserves a note. A naive check that requires the string to begin with a single / and not // would accept it, but the WHATWG parser reads the backslash as a slash and resolves it to https://evil.test/, which we confirmed. Resolving first and checking the parsed host catches every variant. The function returns a relative path, never the absolute URL, which also removes the chance of the browser being sent somewhere unexpected.

Server-side request forgery

When your server fetches a URL supplied by a user, such as a webhook target or an image to import, an attacker may point it at internal services: the loopback address, private network ranges, or cloud metadata endpoints. String filters fail because of the alternative IPv4 forms shown earlier (http://0x7f.1/ and http://2130706433/ both parsed to 127.0.0.1), IPv6 literals, and DNS names that resolve to internal addresses. Good defences combine several steps: parse with a real parser, allow only http and https, resolve the hostname yourself and check the resulting address against a deny list of internal ranges, connect to that exact address to avoid a second lookup, and limit redirects, re-checking each hop. Where possible, use an allow-list of hosts rather than a deny-list of bad ones.

javascript: and data: URLs

A javascript: URL executes script when navigated to. If you take a user-supplied URL and place it in an href, an attacker can supply javascript:alert(1) to run code in your page, which is cross-site scripting. The scheme appeared in our test with the opaque path and a null origin, and nothing in the parser rejects it, because it is a valid URL. The defence belongs in your code: parse the value and allow only the schemes you expect, usually http, https and sometimes mailto. Do not try to block bad schemes with a deny-list, because case variants, embedded tabs or newlines and encoding tricks have historically bypassed such filters. The WHATWG parser removes tab and newline characters from the input, so the check must run on the parsed protocol, not on the raw text. data: URLs deserve similar caution, since browsers restrict how they may be navigated to, but allowing them in images or elsewhere can still create exposure.

Parser differentials

The backslash example above is one instance of a broader class: two components parse the same string and disagree. Others include differences in how they treat control characters, whether they strip whitespace, how many slashes follow the scheme, how they handle a missing scheme, and where they stop the host. The remedy is organisational as well as technical: use one parser in a validation path, pass around structured values rather than strings, and re-serialise from the parsed form instead of forwarding the original.

Sensitive data in URLs

Treat anything in a URL as public. It appears in history, bookmarks, access logs, browser sync, analytics and the Referer header sent to other sites, subject to the referrer policy you set. Do not put passwords, API keys, reset tokens that remain valid for long, or personal data in the path or query. If a one-time token must travel in a link, make it single-use and short-lived, and consider the Referrer-Policy header to limit its spread.

Step by step: take a URL apart with DevUtilX

When you want an answer without writing code, the DevUtilX URL Parser breaks any URL into its components right in your browser.

1. Open the URL Parser tool.
2. Paste the URL, including the scheme. A bare example.com/path has no scheme, so include https:// to see the full breakdown.
3. Read the component list: scheme, host, port, path, query, fragment, and the individual query parameters with their decoded values.
4. Check for surprises: a host that is not what you expected, a port you did not know about, a repeated parameter, or an @ hiding in the authority.
5. Edit the URL and parse again to test a fix, then copy the part you need.

The tool runs locally in your browser, and nothing is uploaded, which matters when the URL contains tokens, internal hostnames or customer identifiers. Even so, remember that a URL pasted anywhere may sit in clipboard history, so remove secrets before sharing a screenshot. If the parser reports an error, look for a missing scheme, a stray space, an unmatched bracket in an IPv6 address, or a port outside the valid range.

A good habit is to parse suspicious links before clicking them, which makes the userinfo trick in this article obvious. Another is to paste a long tracking URL and delete parameters one by one until you find the ones that matter.

Code: JavaScript

The URL and URLSearchParams classes are available in all modern browsers, Node.js and other runtimes. These snippets were run on Node.js 22.

Parsing a URL and reading its parts:

const u = new URL("https://shop.example.com:8443/a/c/d.html?tag=a&tag=b#x");
console.log(u.hostname);               // "shop.example.com"
console.log(u.searchParams.getAll("tag")); // [ 'a', 'b' ]

Building a URL safely, so encoding is handled for you:

const url = new URL("https://api.example.com/search");
url.searchParams.set("q", "a&b=c d");
url.searchParams.append("tag", "new");
url.searchParams.append("tag", "sale");
console.log(url.href);
// https://api.example.com/search?q=a%26b%3Dc+d&tag=new&tag=sale

Resolving relative references, as in the table earlier:

new URL("../api/", "https://example.com/docs/guide/intro.html").href
// "https://example.com/docs/api/"

Handling invalid input. new URL() throws a TypeError for strings that cannot be parsed, such as a relative reference with no base. Newer runtimes also provide URL.canParse(), but a try block works everywhere:

function parseOrNull(s, base) {
  try { return new URL(s, base); } catch { return null; }
}

Two reminders. URL objects are mutable, so setting url.pathname or url.search re-encodes the value. And searchParams and search stay in sync: changing one updates the other.

Code: Python

Python's standard library offers urllib.parse. Its behaviour is documented in the urllib.parse reference. This code ran on Python 3.13.

from urllib.parse import urlsplit, parse_qs, parse_qsl, urlencode, urljoin

url = ("https://user:pa55@shop.example.com:8443/a/c/d.html;v=1"
       "?q=url+parser&tag=a&tag=b&empty=&x=%C3%A9#section-2")
s = urlsplit(url)
for k in ("scheme", "netloc", "username", "password",
          "hostname", "port", "path", "query", "fragment"):
    print(k.ljust(9), repr(getattr(s, k)))

Its output:

scheme    'https'
netloc    'user:pa55@shop.example.com:8443'
username  'user'
password  'pa55'
hostname  'shop.example.com'
port      8443
path      '/a/c/d.html;v=1'
query     'q=url+parser&tag=a&tag=b&empty=&x=%C3%A9'
fragment  'section-2'

Compare this with the JavaScript table. Python gives the port as an integer, splits netloc as the raw authority, and leaves the path parameter in path. Its hostname attribute is lowercased, whereas netloc is the raw authority text.

Working with the query:

print(parse_qs(s.query))
# {'q': ['url parser'], 'tag': ['a', 'b'], 'x': ['é']}
print(parse_qsl(s.query, keep_blank_values=True))
# [('q', 'url parser'), ('tag', 'a'), ('tag', 'b'), ('empty', ''), ('x', 'é')]
print(urlencode({"q": "a&b=c d", "tag": ["a", "b"]}, doseq=True))
# q=a%26b%3Dc+d&tag=a&tag=b

Without doseq=True, a list value would be converted with str() and encoded as a single weird parameter, so use it whenever a value may be a list. Reference resolution uses urljoin:

base = "https://example.com/docs/guide/intro.html?v=1#top"
print(urljoin(base, "../api/"))   # https://example.com/docs/api/
print(urljoin(base, "?v=2"))      # https://example.com/docs/guide/intro.html?v=2

Python's parsers are lenient: urlsplit rarely raises, and mistakes show up later, for example when reading .port. Do not treat a successful call as proof that a URL is valid or safe. If you need strict validation, check the scheme and hostname explicitly after parsing, and remember the differential shown above.

Common mistakes

• Building URLs by string concatenation. Values containing &, =, #, spaces or non-ASCII text corrupt the result. Use URLSearchParams, URL or urlencode.
• Using encodeURI for values. It leaves &, =, ? and # alone. Use encodeURIComponent for individual values.
• Double encoding. Encoding an already encoded string turns %20 into %2520. Encode raw values once, at the boundary where you build the URL.
• Decoding too early. Splitting a path after decoding turns an encoded %2F into a separator.
• Expecting the fragment on the server. It never arrives in the request.
• Comparing URLs as strings. http://example.com:80/ and http://example.com/ are the same URL. Parse first.
• Assuming get returns all values. searchParams.get returns only the first value for repeated keys.
• Validating with startsWith or regular expressions. These miss userinfo, subdomain and backslash tricks. Compare the parsed host.
• Forgetting the scheme. Without it, localhost:3000 parses as a scheme and an opaque path.
• Hand-writing IPv6 URLs without brackets. The port becomes ambiguous.
• Putting secrets in URLs. They persist in logs and history.
• Ignoring trailing slashes in relative links. Pages move, and links break silently.
• Relying on origin null checks. Several unrelated URL types share the opaque origin, so it is not an identity.

Performance, accessibility and other considerations

URLs are small, but a few operational points are worth stating.

Performance. Every distinct origin a page contacts costs a DNS lookup, a TCP connection and a TLS handshake, unless the connection is reused. Counting origins in your page, using the parsed origin property, is a simple way to spot third-party overhead. Moving assets to the same origin or using hints such as preconnect for critical third parties can reduce this cost. Normalised, consistent URLs also improve cache hit rates, since the cache key is derived from the URL. Parsing itself is fast, but constructing thousands of URL objects in a hot loop is wasteful, so parse once and reuse the object.

Accessibility. The visible text of a link matters more than its address, but the address still reaches users. Screen readers often read a bare URL character by character, so avoid showing long raw URLs as link text. Use descriptive link text, and keep the URL itself readable and stable. A clean path such as /guides/url-anatomy is easier to dictate, remember and type than a string of parameters. Fragments with meaningful id values let people jump to a section with the keyboard and share a precise link. Make sure the target element can receive focus when needed, as in single-page applications that change only the fragment.

Internationalisation. Non-ASCII paths and hosts work, but they appear percent-encoded or in Punycode in logs and many tools. Test links containing accented characters in email clients and chat apps, which sometimes break them at the first non-ASCII character.

Privacy. Analytics parameters that identify individuals can leak through the Referer header. Strip them from links you place on other sites, and avoid embedding personal data in URLs.

Troubleshooting

When a URL behaves unexpectedly, work from symptom to cause.

Symptom: a query parameter arrives with a different value than you sent

Cause. An unencoded &, #, + or % inside the value. The & started a new parameter, the # cut the query off and made the rest a fragment, and + turned into a space.
Fix. Build the query with URLSearchParams or urlencode. Decode only once, in the server framework.

Symptom: the server never sees part of the URL

Cause. The part after # is the fragment, which the browser does not send.
Fix. Move the data into the query or the path, or read the fragment in client-side code and send it explicitly.

Symptom: relative images and links break on one page only

Cause. The page URL lacks (or has) a trailing slash, so the base directory differs.
Fix. Use root-relative paths, set one canonical trailing-slash policy and redirect the other form, or add a <base> element carefully.

Symptom: new URL(...) throws "Invalid URL"

Cause. The input is a relative reference with no base, has no scheme, contains an unmatched bracket, or has an out-of-range port.
Fix. Pass a base as the second argument, add the scheme, or catch the error and report a friendly message.

Symptom: two URLs that look the same are treated as different

Cause. A difference in case, default port, trailing slash, parameter order or percent-encoding case.
Fix. Normalise before comparing: parse, lowercase the host, drop default ports, sort parameters, and apply your trailing-slash policy.

Symptom: OAuth reports a redirect URI mismatch

Cause. The registered value and the sent value differ by a trailing slash, port, scheme or query string.
Fix. Register the exact string your application sends and compare it as a parsed value.

Symptom: a spaces-versus-plus bug in decoded values

Cause. One side encodes spaces as + and the other decodes with a path-style routine that keeps +.
Fix. Use matching form-style functions on both sides, or encode with %20 consistently.

Symptom: a link validator accepts a URL that goes to the wrong site

Cause. The check used string matching, and the URL used userinfo, a backslash, an alternate IP notation or a look-alike subdomain.
Fix. Parse, check the scheme against an allow-list, compare the hostname exactly, and use the parsed result downstream.

Symptom: the server returns an error for very long URLs

Cause. A proxy, server or framework limit on request line or header size.
Fix. Shorten the URL, move data to a POST body, or raise the limit deliberately where you control it.

Best practices checklist

• Use the platform parser. URL in JavaScript and urllib.parse in Python, never ad-hoc splitting.
• Build, don't concatenate. Add parameters through URLSearchParams or urlencode.
• Encode values once, at the point where you assemble the URL.
• Parse once, pass structured data. Re-serialise from the parsed form when you must pass a string.
• Allow-list schemes and hosts for any user-supplied URL, and compare parsed hosts exactly.
• Resolve before you validate redirect targets, and return relative paths.
• Resolve and check IP addresses for server-side fetches, and limit redirects.
• Never put secrets in URLs.
• Choose a canonical form: scheme, host, trailing-slash policy and case. Redirect everything else to it.
• Add canonical links with absolute URLs on pages that have duplicates.
• Keep URLs short, lowercase and readable.
• Sort or whitelist query parameters for cache keys.
• Use root-relative or absolute links in shared templates.
• Test with awkward input: spaces, non-ASCII text, repeated keys, empty values, IPv6 hosts and long strings.
• Document your parameter conventions for arrays and duplicates.

Comparison: ways to work with URLs

Approach	Strengths	Weaknesses	Best for
WHATWG URL class	Matches browser behaviour, mutable, built in	Throws on invalid input; differs from RFC parsers	Front-end code and Node.js services
Python urllib.parse	Standard library, simple functions	Lenient, RFC-flavoured, port errors surface late	Scripts, backends, data pipelines
Regular expressions	Quick for fixed patterns	Fragile, easily bypassed, hard to maintain	Almost never for validation
String splitting	Fast for trusted, fixed formats	Breaks on encoding, IPv6, userinfo and fragments	Logging of your own URLs
Framework routers and helpers	Handle routing, encoding and reverse URLs	Framework-specific behaviour	Application code
URL templates (RFC 6570 style)	Describe families of URLs declaratively	Needs a library, adds a concept	API clients and documentation
Online URL parser tool	No setup, instant view of the parts	Manual, not for automation	Debugging and learning

FAQ

What is the difference between a URI and a URL?
A URI identifies a resource. A URL is a URI that also says how to locate it. Every URL is a URI, but a URN is a URI that names without locating. In modern specifications, "URL" is used for both.

Are URLs case-sensitive?
Partly. The scheme and host are case-insensitive and are lowercased by the parser. The path, query and fragment are case-sensitive as far as the standard goes, though servers may treat paths more loosely.

Does the fragment get sent to the server?
No. The browser removes it before sending the request, which we confirmed with a local test server. Scripts running in the page can still read it.

How do I send an array in a query string?
There is no standard. Repeating the key, tag=a&tag=b, is the most portable choice. Use getAll in JavaScript or parse_qs in Python to read it. Check what your server framework expects before using bracket syntax.

Why does a space sometimes appear as + and sometimes as %20?
In query strings built by HTML forms, a space is +. In paths, + is a literal plus and the space must be %20. URLSearchParams produces +, and encodeURIComponent produces %20.

What is the default port for HTTPS?
443, and 80 for HTTP. The parser drops a port that matches the default, so https://example.com:443/ becomes https://example.com/.

What counts as the same origin?
The same scheme, host and port. Path, query, fragment and credentials are ignored. https://example.com and https://www.example.com are different origins.

Is https://example.com@evil.test a link to example.com?
No. The text before the @ is userinfo, and the host is evil.test. This is a common phishing trick.

Should my URLs end with a trailing slash?
Either is acceptable. What matters is consistency: choose a policy, redirect the other form, and use the chosen form in links, sitemaps and canonical tags.

How long can a URL be?
There is no limit in the syntax. Real limits come from browsers, servers and intermediaries, and they vary, so keep URLs short and use a request body for large data.

Why does localhost:3000 not parse as I expect?
A parser sees localhost as a scheme because it precedes a colon. Write http://localhost:3000 in code.

Can I use non-English characters in a URL?
Yes. In paths and queries they are percent-encoded as UTF-8 bytes, and in hostnames they are converted to Punycode. Both are transparent to users but visible in logs.

Is the WHATWG parser the same as RFC 3986?
No. The WHATWG standard defines an algorithm that accepts and repairs messy input, while RFC 3986 defines a grammar for well-formed URIs. They agree on valid inputs and differ on edge cases such as backslashes and alternate IP notations.

Is it safe to paste a URL into an online parser?
Only if it runs in your browser. DevUtilX processes input locally. Even so, remove tokens and secrets before pasting any URL into any tool.

Conclusion

A URL is a small grammar with large consequences. The scheme chooses the rules, the authority chooses the server, the path and query choose the resource and its refinements, and the fragment stays in the browser. Once you know where each delimiter ends one component and begins the next, many mysterious bugs become easy to read: lost parameters, broken relative links, duplicate pages and bypassed filters. Parse with a real parser, encode with the right function, compare parsed values rather than strings, and be suspicious wherever two components might read the same text differently. To inspect any URL right now, open the URL Parser and see it split into its parts. For related reading, see our guide to JSON minification and payload size, which covers what travels over these URLs, and our HTML formatting and semantic markup guide, where links, href values and <base> elements live.

Further reading

• URL – Wikipedia
• URL API – MDN
• URL Standard – WHATWG
• RFC 3986: Uniform Resource Identifier (URI) Generic Syntax
• Same-origin policy – MDN
• urllib.parse – Python documentation