Text Compression Algorithms: Huffman, LZ77 and DEFLATE

Every time a browser loads a page, an API returns a response, or a backup job writes an archive, some piece of software is shrinking text and then expanding it again without losing a single character. You rarely see it happen, yet it sits underneath gzip, ZIP files, PNG images, Git packfiles and nearly every HTTP response you have ever received. This guide explains how lossless text compression works from first principles. You will see why repeated text shrinks so well, how to build a Huffman tree by hand, how the LZ77 sliding window finds repeats, and how DEFLATE glues the two together. You will also learn how to measure compression ratios honestly, when compression does nothing useful, and why compressing secrets next to attacker-controlled text can leak them.

The article is written for developers who use compression every day but have never had to look inside it: front-end engineers tuning payloads, back-end engineers configuring proxies, and anyone who has wondered why a 40 KB file became 2 KB while a photo refused to shrink at all. All the numbers in this guide were produced by scripts that ran in a sandbox, and the scripts are shown so you can reproduce them. Your own results will differ with other data, but the patterns hold.

What lossless compression is, and a short history

Data compression means representing information using fewer bits than the original representation. There are two families. Lossy compression, such as JPEG for photos or MP3 for audio, throws away detail that people are unlikely to notice. Lossless compression keeps everything: decompression returns exactly the bytes you started with. Source code, JSON, logs, configuration, documents and databases all need the lossless kind, because a single changed character can break a program.

Lossless compression works because real data is not random. English text uses some letters far more often than others. A JSON array repeats the same keys in every element. A log file repeats timestamps, host names and message templates. A compressor finds these regularities and encodes them more cheaply. If your data had no regularities at all, no lossless compressor could shrink it, and we will prove that later with a simple counting argument.

A brief timeline

The ideas are older than most people expect.

• 1948. Claude Shannon published "A Mathematical Theory of Communication", which defined information entropy and showed there is a hard lower limit to how compactly a source can be encoded on average.
• 1952. David Huffman, then a graduate student, described an algorithm that builds the best possible prefix code for a known set of symbol frequencies. It is the method you will build by hand below.
• 1977. Abraham Lempel and Jacob Ziv published the algorithm now called LZ77, which replaces repeated strings with pointers back to an earlier copy. A year later they published a related dictionary method, LZ78.
• Late 1980s. Phil Katz combined LZ77-style matching with Huffman coding in the ZIP format, and the combination became known as DEFLATE.
• 1990s. DEFLATE was standardised in RFC 1951, with the zlib wrapper in RFC 1950 and the gzip file format in RFC 1952. PNG adopted it for image data, and HTTP servers adopted gzip for web content.

The Deflate article on Wikipedia gives a compact overview of the format and its history. Newer formats, such as Brotli and Zstandard, build on the same two ideas with better modelling and bigger dictionaries, and we compare them near the end. Once you understand DEFLATE, you understand the core of all of them.

Entropy: how small can text get?

Before building anything, it helps to know what "small" can mean. Entropy is the average number of bits of information per symbol, given how predictable the symbols are. If a source emits symbols with probabilities p1, p2, and so on, its entropy in bits per symbol is the sum of p times log2(1/p) over all symbols.

Two extremes make the idea concrete.

• A source that always emits the letter "a" has entropy zero. You learn nothing from each symbol, so you need no bits to describe it beyond its length.
• A source that emits one of 256 byte values uniformly at random has entropy eight bits per byte. Nothing can be saved.

Real text sits between those extremes. Take the word abracadabra, which has eleven letters. The letter a appears five times, b and r twice each, and c and d once each. The entropy of that distribution, computed by the script in the next section, is about 2.04 bits per letter. A plain ASCII encoding spends eight bits per letter, so there is a lot of room.

Entropy is a statement about averages and about a particular model of the data. Two things follow from that.

First, entropy depends on the model. If you only count single letters, you get one number. If you model pairs of letters, or whole words, or "the next character depends on the previous three", you get a lower number, because context makes symbols more predictable. This is why LZ77 helps: it exploits long-range repetition that a letter-frequency model cannot see.

Second, it gives a limit. A lossless code cannot beat the entropy of the true source on average. Practical compressors only try to approach it for the model they use.

The counting argument: why nothing compresses everything

There are 2^n bit strings of length n, but only 2^n - 1 strings that are shorter (all lengths from zero to n-1 combined). So any scheme that shortens some inputs must lengthen others, otherwise two different inputs would map to the same output and decompression could not tell them apart. The consequence is practical: every compressor has inputs it cannot shrink, and the usual designs make sure those inputs grow only slightly. Random data, already-compressed data and encrypted data are exactly such inputs, and we measure them below.

Huffman coding: short codes for common symbols

Huffman coding assigns each symbol a variable-length bit string. Common symbols get short codes, rare symbols get long ones. The codes are chosen so that no code is a prefix of another. This prefix property lets a decoder read bits one at a time and know exactly when a symbol ends, with no separators.

Why prefixes matter: suppose a is 0 and b is 01. When the decoder sees a zero, it cannot tell whether it has read a or the start of b. If no code is a prefix of another, there is never any doubt.

Building a tree by hand

Use the string abracadabra. The first step is to count symbol frequencies.

Symbol	Count
a	5
b	2
r	2
c	1
d	1

The algorithm repeatedly takes the two lowest-weight nodes and joins them under a new parent whose weight is the sum.

1. The two smallest weights are c (1) and d (1). Join them into a node of weight 2.
2. Now the smallest are b (2), r (2) and the new cd node (2). Ties are possible here, and any tie-break gives a valid optimal code. Joining b and r gives a node of weight 4.
3. The two smallest remaining are cd (2) and br (4). Join them into a node of weight 6.
4. Finally join a (5) with the weight-6 node to get the root, weight 11, which equals the number of letters.

Now label every left branch 0 and every right branch 1, and read the path from the root to each leaf. In the run that produced the output below, the tree gave these codes.

Symbol	Count	Code	Code length
a	5	0	1
b	2	110	3
r	2	111	3
c	1	100	3
d	1	101	3

The frequent letter a costs one bit. Everything else costs three. The encoded message is the concatenation of the codes for each letter in order:

a   b   r   a  c   a  d   a  b   r   a
0 110 111 0 100 0 101 0 110 111 0

That is 23 bits in total, written without spaces as 01101110100010101101110. Compare the alternatives: eight bits per letter in ASCII is 88 bits, and a fixed three-bit code, the smallest that can distinguish five symbols, is 33 bits. Huffman needs 23, which is 2.09 bits per letter, very close to the entropy of 2.04. Huffman codes always use a whole number of bits per symbol, so they can lose a little against entropy when probabilities are not powers of one half. Arithmetic coding and the newer asymmetric numeral systems avoid that loss, which is one reason modern formats use them.

A working Huffman demo

This Python script builds the tree, prints each merge, produces the codes, encodes the string and decodes it again to prove the round trip. It uses a counter as a tie-breaker so the output is deterministic.

import heapq, math, itertools
from collections import Counter

def huffman(text):
    cnt = Counter(text)
    c = itertools.count()
    heap = [(f, next(c), s) for s, f in sorted(cnt.items())]
    heapq.heapify(heap)
    while len(heap) > 1:
        a = heapq.heappop(heap)
        b = heapq.heappop(heap)
        print("merge", a[2], a[0], "+", b[2], b[0], "->", a[0] + b[0])
        heapq.heappush(heap, (a[0] + b[0], next(c), (a[2], b[2])))
    codes = {}
    def walk(node, prefix):
        if isinstance(node, tuple):
            walk(node[0], prefix + "0")
            walk(node[1], prefix + "1")
        else:
            codes[node] = prefix or "0"
    walk(heap[0][2], "")
    return cnt, codes

text = "abracadabra"
cnt, codes = huffman(text)
bits = "".join(codes[ch] for ch in text)
print(codes)
print(bits, len(bits), "bits versus", len(text) * 8)

n = len(text)
entropy = -sum(f / n * math.log2(f / n) for f in cnt.values())
print("entropy", round(entropy, 3), "average code length", round(len(bits) / n, 3))

inverse = {v: k for k, v in codes.items()}
out, cur = "", ""
for bit in bits:
    cur += bit
    if cur in inverse:
        out += inverse[cur]
        cur = ""
print("round trip ok:", out == text)

Running it printed the merges c 1 + d 1 -> 2, b 2 + r 2 -> 4, then (c,d) 2 + (b,r) 4 -> 6, then a 5 + ... 6 -> 11. The result was 23 bits versus 88, an entropy of 2.04, an average code length of 2.09, and round trip ok: True. The left and right labelling can differ between implementations, and so can the exact codes, but the code lengths and the total of 23 bits will not.

The catch: the decoder needs the tree

Huffman decoding requires the same code table the encoder used. For a tiny message like this one, storing the table would cost more than the savings. Real formats solve this in two ways. A static table is agreed in advance and never transmitted, which suits data with known statistics. A dynamic table is transmitted compactly at the start of each block. DEFLATE offers both, and it sends the dynamic table as a list of code lengths only, since a fixed rule can rebuild the actual bit patterns from lengths. Codes rebuilt this way are called canonical Huffman codes, and the trick saves a lot of header space.

LZ77: replacing repeats with pointers

Huffman coding looks at symbol frequencies one symbol at a time. It cannot see that the phrase the cat sat appeared a few words ago. LZ77 can. The encoder keeps a sliding window of recent output. As it reads input, it looks for the longest string that also occurs inside the window. If it finds one, it emits a pair: how far back the earlier copy starts (the distance) and how many characters match (the length). If it finds nothing worthwhile, it emits the raw character, called a literal.

The decoder is trivially simple. It copies literals straight to the output, and for each pair it copies length characters starting distance characters back. Decoding is much faster than encoding, because the encoder searches while the decoder only copies.

A worked example

Take the string the cat sat on the mat, the cat sat, which has 35 characters. The script below, run with a 32-character window and a minimum match length of three, produced this token stream.

Step	Token	Meaning
1 to 9	nine literals	t h e space c a t space s
10	match (distance 4, length 3)	copy at from four characters back
11 to 13	three literals	o n and a space
14	match (distance 15, length 4)	copy the from fifteen characters back
15 to 18	four literals	m a t ,
19	match (distance 9, length 5)	copy the
20	match (distance 24, length 7)	copy cat sat

The 35 characters became 20 tokens: 16 literals and 4 matches. Whether that is a win depends on how many bits each token costs, which is where Huffman comes back in. Notice also how the last match repeats a seven-character phrase for the price of two numbers.

A working LZ77 demo

def lz77(data, window=32, minlen=3, maxlen=258):
    i, out = 0, []
    while i < len(data):
        best = (0, 0)
        for j in range(max(0, i - window), i):
            n = 0
            while i + n < len(data) and n < maxlen and data[j + n] == data[i + n]:
                n += 1
            if n > best[0]:
                best = (n, i - j)
        if best[0] >= minlen:
            out.append(("M", best[1], best[0]))
            i += best[0]
        else:
            out.append(("L", data[i]))
            i += 1
    return out

def unlz(tokens):
    out = []
    for t in tokens:
        if t[0] == "L":
            out.append(t[1])
        else:
            _, dist, length = t
            for _ in range(length):
                out.append(out[-dist])
    return "".join(out)

s = "the cat sat on the mat, the cat sat"
tokens = lz77(s)
print(len(s), "characters ->", len(tokens), "tokens")
print("round trip ok:", unlz(tokens) == s)

This is the naive version, which compares against every position in the window and takes quadratic time in the window size. Production encoders use hash tables keyed on the next three bytes, chains of earlier positions with the same hash, and limits on how many candidates to check. Those limits are what the compression level setting controls.

How DEFLATE combines the two

DEFLATE, specified in RFC 1951, works in two stages. First LZ77 turns the input into a stream of literals and length-distance pairs. Then Huffman coding compresses that stream. The two stages complement each other: LZ77 removes repetition of whole strings, and Huffman removes the skew in symbol frequencies that remains.

Blocks

The output is a series of blocks. Each block starts with a header declaring its type.

• Stored blocks contain raw bytes with a small length header. The compressor falls back to these when compression would not help.
• Fixed Huffman blocks use a code table defined once in the specification, so no table needs to be sent. This suits short inputs.
• Dynamic Huffman blocks include custom code tables tuned to that block's statistics. The tables are sent as code lengths, themselves Huffman-compressed. This wins on larger inputs.

The ability to switch per block matters. An encoder can choose the cheapest representation for each section of the data, which is why DEFLATE rarely expands input by more than a few bytes per block.

Two alphabets

Instead of separate Huffman tables for literals and for lengths, DEFLATE merges them into a single alphabet. Values 0 to 255 stand for literal bytes, the value 256 marks the end of a block, and values from 257 upward stand for match lengths. A second alphabet encodes match distances. The decoder reads a symbol from the first alphabet. If it is a literal, it outputs the byte. If it is a length code, it reads any extra bits that refine the length, then reads a distance symbol and its extra bits, and copies the bytes.

Huffman code lengths in DEFLATE are limited to 15 bits, which bounds table sizes and makes decoding fast.

From DEFLATE to zlib, gzip and HTTP

DEFLATE is a raw stream with no framing, and three wrappers surround it.

Format	Specification	Overhead	Typical use
Raw DEFLATE	RFC 1951	none	Inside other containers such as ZIP
zlib	RFC 1950	2-byte header, 4-byte Adler-32 checksum	Library streams, PNG image data
gzip	RFC 1952	10-byte header, 8-byte trailer with CRC-32 and size	Files and HTTP Content-Encoding: gzip

The overhead in that table is easy to verify. This Node.js script compresses the 35-byte string hello hello hello hello hello hello in all three formats.

const zlib = require("zlib");
const input = Buffer.from("hello hello hello hello hello hello");
console.log("raw", input.length);
console.log("deflateRaw", zlib.deflateRawSync(input).length);
console.log("zlib", zlib.deflateSync(input).length);
console.log("gzip", zlib.gzipSync(input).length);
console.log(zlib.deflateSync(input).subarray(0, 2).toString("hex"));
console.log(zlib.gzipSync(input).subarray(0, 3).toString("hex"));

The measured output: raw 35 bytes, raw DEFLATE 11 bytes, zlib 17 bytes, gzip 29 bytes. The differences are exactly the wrapper sizes: 6 bytes for zlib and 18 for gzip. The zlib stream began with the bytes 789c, the familiar zlib signature, and the gzip stream began with 1f8b08, where 1f8b is the gzip magic number and 08 identifies DEFLATE.

That small experiment already carries a lesson. For tiny inputs the wrapper can outweigh the data, and a 29-byte gzip file of a 35-byte string is barely a win.

How HTTP uses it

In HTTP, a client advertises the encodings it accepts in the Accept-Encoding request header, and the server replies with Content-Encoding naming the one it chose. The MDN guide to HTTP compression describes this negotiation. In practice gzip is universal, and br (Brotli) is widely supported for secure connections. Servers should also send Vary: Accept-Encoding, so a shared cache stores one copy per encoding. Our guide to JSON minification and payload size shows how to check these headers with curl and why compression usually matters more than removing whitespace.

Why repeated text compresses so well

Everything above comes together in one observation: compression rewards repetition. Three kinds of redundancy stack up in typical text.

1. Skewed symbol frequencies. Spaces, vowels and quotation marks are common, and Huffman gives them short codes.
2. Repeated strings. Keys, tag names, identifiers and phrases recur, and LZ77 replaces later copies with pointers.
3. Structured whitespace. Indentation is the same run of spaces on many lines, which is just another repeated string.

The experiment below measures how three kinds of input behave. It uses Python's zlib at level 9, and the table shows measured sizes.

import zlib, os, json

def report(name, data):
    packed = zlib.compress(data, 9)
    print(name, len(data), len(packed), round(len(packed) / len(data), 3))

report("repeat", b"hello world. " * 1000)
rows = json.dumps([{"id": i, "name": "item %d" % i, "active": i % 2 == 0,
                    "tags": ["a", "b"]} for i in range(500)]).encode()
report("json", rows)
report("random", os.urandom(5000))

Input	Original bytes	Compressed bytes	Ratio
One phrase repeated 1,000 times	13,000	65	0.005
JSON array of 500 similar records	34,530	2,677	0.078
5,000 random bytes	5,000	5,011	1.002

The repeated phrase shrank to half a percent of its size, because LZ77 expresses almost the whole input as a handful of long overlapping matches. The JSON shrank by more than 92 percent, because the keys and punctuation repeat in every record while only the numbers change. The random bytes did not shrink at all, and grew by 11 bytes of block overhead. Those three rows cover the whole spectrum from perfectly predictable to perfectly unpredictable.

The ratio here is compressed size divided by original size, so smaller is better. Some tools report the opposite, the original divided by the compressed size, and some report the percentage saved. Always check which convention a number uses.

Real-world use cases

Lossless text compression is so common that it is easy to miss. Typical places it appears:

• HTTP responses. HTML, CSS, JavaScript, JSON and SVG are compressed at the server, proxy or CDN. This is usually the single biggest transfer-size saving available. See our guides to CSS minification and JavaScript minification for what to do before compression.
• Archives and packaging. ZIP files use DEFLATE per entry. Other archive tools and package formats use gzip or newer compressors for whole bundles.
• Images. PNG compresses filtered pixel data with DEFLATE, which is why PNG is lossless.
• Logs and metrics. Log lines are extremely repetitive, so rotated logs routinely shrink by large factors when gzipped.
• Backups and transfers. Compressing before sending over a slow link often beats sending raw data, as long as CPU is not the bottleneck.

Measuring compression honestly

Numbers about compression are easy to get wrong. A few habits keep them meaningful.

Measure bytes, not characters. A JavaScript string length counts UTF-16 code units. A character like é takes two bytes in UTF-8, and many emoji take four. Use Buffer.byteLength in Node or new TextEncoder().encode(text).length in the browser. Compare sizes of the encoded bytes.

Use representative data. A compressor looks great on a repeated sentence and terrible on random data. Test with real samples from your system, ideally a few of different sizes.

State the level and the tool. Compression level changes results. Here is a measurement on 300 JSON user records (17,106 bytes) with Node.js.

const zlib = require("zlib");
const text = Buffer.from(JSON.stringify(Array.from({ length: 300 }, (_, i) => ({
  id: i, name: "user" + i, role: i % 4 ? "viewer" : "admin", active: true
}))));
for (const level of [1, 6, 9]) {
  console.log("gzip level", level, zlib.gzipSync(text, { level }).length);
}
console.log("brotli default", zlib.brotliCompressSync(text).length);

Compressor	Output bytes
gzip level 1	1,694
gzip level 6	1,610
gzip level 9	1,611
Brotli, default settings	864

Two points stand out. The jump from level 1 to level 6 saved about 5 percent, while level 9 gained nothing over level 6 on this input and was one byte worse. Higher levels search harder, but they do not guarantee a smaller result, and the extra CPU often buys very little. And Brotli, with its larger built-in dictionary and better modelling, produced output roughly half the gzip size on this sample. That result is specific to this data, so treat it as an illustration and not a rule.

Mind the baseline. Savings against pretty-printed input look better than savings against already minified input. Say which one you started from.

When compression does not help

Compression is not magic, and knowing where it fails saves you from wasted CPU and confusing results. The counting argument from earlier predicts all of these cases.

Already compressed data

JPEG, PNG, MP4, WebP, ZIP, gzip, PDF with embedded images and most font and archive formats have already removed their redundancy. Running DEFLATE over them again finds little to work with, and the result is typically the same size or slightly larger. Servers therefore skip compression for these MIME types. If you see a transfer of a .jpg with Content-Encoding: gzip, someone configured the server to waste time.

Random and encrypted data

Good encryption output is designed to be indistinguishable from random bytes, so it has no patterns to exploit. This Node.js check compresses 2,000 random bytes with gzip:

const zlib = require("zlib");
const rnd = require("crypto").randomBytes(2000);
console.log(rnd.length, zlib.gzipSync(rnd).length);

It printed 2,000 in and 2,023 out. The extra 23 bytes are the gzip header, trailer and block overhead. The same logic applies to ciphertext, which leads to a rule: if you compress and encrypt, compress first. Compressing after encryption achieves nothing. The next section explains why compress-then-encrypt needs care.

Base64 deserves a mention too. Encoding those same 2,000 random bytes as Base64 produced a 2,668-character string, and gzip squeezed it to 2,050 bytes. Compression recovered most of the 33 percent Base64 added, because Base64 uses only 64 distinct symbols and Huffman coding can spend about six bits on each instead of eight, but it could not get below the size of the underlying random bytes. Sending binary as binary is always cheaper than Base64 plus gzip.

Tiny inputs

Every format has fixed overhead. Compressing the two-byte string hi with Python produced 10 bytes with zlib, 22 with gzip and 4 with raw DEFLATE. Five bytes, hello, became 13 bytes with zlib. For small inputs, the header and checksum dominate, and the compressed form is larger. This is why web servers apply a minimum size threshold before compressing, and why compressing individual tiny database fields rarely pays off. Compress larger units instead, such as whole pages or batches of records, so the compressor has repetition to find.

Stored blocks as a safety net

When DEFLATE determines that a block cannot be shrunk, it falls back to a stored block. Level 0 in zlib forces that behaviour. On the 17,106-byte JSON sample, a level 0 raw DEFLATE stream measured 17,111 bytes: the input plus five bytes of block header. That fixed, small penalty is the good news behind the counting argument. Incompressible data grows by a few bytes, not by a large fraction.

Compression and security: CRIME and BREACH, conceptually

Compression leaks information through size. That sentence is the whole idea behind a family of real attacks, so it is worth understanding even if you never implement a defence yourself.

Suppose an application compresses a response that contains two things: a secret the attacker wants, such as a session token or a CSRF token, and some text the attacker can influence, such as a query parameter reflected into the page. The server then encrypts the compressed bytes. Encryption hides the content, but it does not hide the length, and the attacker can see the length of each message on the network.

Now the compressor does what it always does: it replaces repeated strings with pointers. If the attacker's guess matches part of the secret, the guess repeats text already present, so the compressor encodes it cheaply and the response is slightly shorter. A wrong guess is a new string and costs more. By trying guesses and watching lengths, the attacker can confirm the secret one character at a time.

A small simulation shows the effect. The script builds a page containing a secret and then a reflected guess, and compresses it:

import zlib

secret = "token=SECRET123"
for guess in ["token=SECRET123", "token=XXXXXXXXX"]:
    page = ("<p>" + secret + "</p><p>" + guess + "</p>").encode()
    print(guess, len(zlib.compress(page, 9)))

The correct guess produced a 33-byte output and the wrong one produced 37 bytes. Four bytes of difference is a visible signal, and an attacker who can send many requests can amplify it. This is an illustration of the principle only. Real attacks need many requests, an attacker who can observe traffic and who can cause the victim's browser to send requests, and some careful statistics.

The real-world attacks

Two named attacks exploit this idea. CRIME, published in 2012, targeted TLS-level compression: the attacker injected guesses into requests and observed the compressed, encrypted request sizes to recover cookies. The CRIME article on Wikipedia describes it. TLS compression was disabled in practice, and TLS 1.3 removed support for it. BREACH, published in 2013, applied the same reasoning to HTTP response compression, which TLS cannot turn off for you because it happens at the application layer. HTTP/2 used a purpose-built header compression scheme, HPACK, designed to resist this kind of attack.

Practical defences

• Do not compress secrets together with attacker-influenced text. The problem needs both ingredients, so separating them is the strongest fix.
• Mask secrets per response. A CSRF token XORed with a fresh random value each time changes the bytes on every response, so guesses no longer line up from one request to the next.
• Disable compression on sensitive dynamic responses that reflect input, if masking is not feasible.
  Static files with no secrets in them, such as scripts and style sheets, are not affected, so compressing them is still the right default. If you encrypt data at rest, the same logic applies to your own protocols: compress before encrypting, and think about who can influence the plaintext.

Common mistakes and pitfalls

• Compressing already compressed files. It burns CPU and gains nothing. Exclude images, video, archives and fonts that are already compressed.
• Forgetting that minification is not compression. Removing whitespace is a different, narrower operation. Do both when they make sense, and see our guide to HTML minification for the trade-offs.
• Comparing characters instead of bytes. Multi-byte characters make character counts misleading.
• Compressing tiny payloads. The overhead can make the output larger.
• Using the highest level everywhere. On our sample, level 9 matched level 6 and cost more CPU. Per-request compression at maximum level can slow down a server for almost no gain.
• Precompressing and forgetting Vary. A cache that does not know the response varies by Accept-Encoding can serve gzip bytes to a client that cannot decode them.
• Double compression. A reverse proxy compressing a response that the application already compressed wastes work and can corrupt headers if Content-Encoding is mishandled.
• Mixing up formats. Raw DEFLATE, zlib and gzip look different. Passing a gzip stream to a zlib decoder, or the reverse, produces an error such as "incorrect header check".
• Compressing secrets alongside reflected input. See the previous section.
• Decompressing untrusted input without limits. A tiny compressed file can expand to gigabytes. Always cap the output size and the number of nested archives. This is often called a decompression bomb.

Step-by-step: shrink text with DevUtilX

DevUtilX has a Text Compressor that runs in your browser. It is worth being precise about what it does, because it is a different job from DEFLATE. The tool reduces text size by collapsing runs of whitespace, removing blank lines and trimming the ends. It is a text minifier, so it does not apply entropy coding. It is useful as a pre-processing step for text that has accumulated stray spaces, and as a quick way to see how much of a document is just formatting.

1. Open the Text Compressor tool.
2. Paste your text into the input editor.
3. Click the Minify button.
4. Read the statistics: original size, minified size, bytes saved and the percentage reduction.
5. Review the cleaned output in the right-hand panel.
6. Copy the result with the copy button.

Your text is processed locally in the browser and is not uploaded, which matters for logs, notes and drafts that contain private information. Because the tool works in the browser, you can use it for quick checks without installing anything.

Two caveats are worth knowing. First, whitespace collapsing is not lossless in the strict sense used in this article: the original line breaks and indentation are gone, and you cannot recover them from the output. That is fine for prose, logs you only search, and copied text, but not for Python source, YAML, Markdown with indented code blocks, or any format where whitespace carries meaning. Second, size is reported in bytes of the text, so you can compare it directly with what a real compressor produces. A good workflow is to minify with the tool, then compare against a gzip run in your terminal to see how much of the remaining size gzip removes. If you are working with structured data, the JSON Minifier is a safer choice, and the same idea for stylesheets and scripts is covered in our guide to JavaScript minification.

Compression in code

These examples use only standard libraries, and each was run before it appeared here.

Python

import gzip, zlib

data = ("order,status\n" * 50).encode()
packed = gzip.compress(data)
print(len(data), len(packed), gzip.decompress(packed) == data)

This printed 650 40 True: 650 bytes became 40 bytes and the round trip matched. The zlib module offers compress and decompress for the zlib format, and compressobj with a negative wbits value gives raw DEFLATE.

Node.js

const zlib = require("zlib");

const original = Buffer.from("hello hello hello hello hello hello");
const packed = zlib.gzipSync(original);
const restored = zlib.gunzipSync(packed);
console.log(original.length, packed.length, restored.equals(original));

Node's zlib module includes deflateSync, deflateRawSync, gzipSync and brotliCompressSync, plus streaming versions for large data. Prefer streams for big files so the whole input does not sit in memory.

Command line

# Compress, keeping the original, then compare sizes
gzip -k -9 access.log
ls -l access.log access.log.gz

# Check what a server is really doing
curl -s -o /dev/null -D - -H "Accept-Encoding: br, gzip" https://example.com/

gzip -k keeps the source file, and -9 asks for the highest level. As the measurements above showed, the highest level is not always meaningfully smaller than the default.

Performance, security and accessibility considerations

CPU against bandwidth. Compression trades processor time for fewer bytes. Decompression is cheap, but compression at high levels is not. Compress static assets once at build time at a high level, and compress dynamic responses at a moderate level. If your server is CPU-bound, a lower level or skipping compression for small responses can improve overall throughput.

Security. Beyond CRIME and BREACH, remember decompression bombs, and remember that a corrupt compressed stream is untrusted input to a parser. Validate length limits before and during decompression.

Accessibility. Compression does not change what screen readers see, because the browser decompresses the body before building the page, and smaller responses help people on slow or metered connections. The risk comes from aggressive whitespace collapsing in HTML, which can break <pre> blocks, so test those pages after minifying.

Troubleshooting guide

Work from the symptom to the cause.

Symptom: the compressed file is larger than the original

Cause. The input is tiny, already compressed, encrypted or random, so the format overhead outweighs any saving.
Fix. Skip compression for that data, or batch many small items into one larger unit before compressing. Set a minimum size threshold on the server.

Symptom: "incorrect header check" or "not in gzip format"

Cause. The decoder expects a different wrapper than the data uses: zlib versus gzip versus raw DEFLATE. A file starting with the bytes 1f 8b is gzip, and one starting with 78 is usually zlib.
Fix. Use the matching decoder, or configure the library's window bits to auto-detect the header. Check the first bytes with xxd or hexdump.

Symptom: JSON or CSS is not compressed on the live site

Cause. The server or CDN compresses only certain MIME types, and yours is not on the list, or the response is below the minimum size.
Fix. Add application/json, text/css and similar types to the compressible list. Confirm with a curl request that sends Accept-Encoding.

Symptom: the text compressor tool made my code stop working

Cause. The tool collapses whitespace, and the language or format depends on indentation or line breaks.
Fix. Use it only on prose and logs. For code, use a language-aware minifier, and keep the original source.

Best practices and checklist

• Enable gzip or Brotli at the edge for text types: HTML, CSS, JavaScript, JSON, SVG, XML and plain text.
• Do not compress images, video, fonts and archives that are already compressed.
• Choose a sensible level. Moderate for dynamic responses, high for assets compressed at build time.
• Set a minimum size threshold, because small payloads gain little or grow.
• Minify first, then compress, for formats where minification is safe, since the two steps remove different redundancy.
• Send Vary: Accept-Encoding and cache each variant.
• Never compress secrets next to reflected input without masking or another defence.
• Cap decompressed size whenever you decompress untrusted data.
• Measure real data in bytes, and report whether your ratio is compressed over original, or the percentage saved.
• Compress before you encrypt, never after.
• Keep original files. Compressed or whitespace-collapsed copies are derived artifacts.
• Test your pipeline with a round trip: decompress and compare to the original, byte for byte.

Comparison: compression approaches

Approach	Lossless	How it works	Typical strength	Notes
Whitespace minification	Not strictly	Collapses spaces and blank lines	Small to moderate on formatted text	Safe for prose, unsafe where whitespace matters
Huffman coding alone	Yes	Short codes for common symbols	Modest on text	Ignores repeats of whole strings
LZ77 alone	Yes	Pointers to earlier repeats	Good on repetitive data	Needs entropy coding to reach its potential
DEFLATE (gzip, zlib)	Yes	LZ77 plus Huffman	Strong, universal	32 KiB window, very widely supported
Brotli	Yes	LZ77-style matching, Huffman, larger window, built-in dictionary	Often smaller than gzip on web text	Slower at high levels, common for HTTP
Zstandard	Yes	LZ77-style matching with fast entropy coding	Wide speed and ratio range	Strong for archives and storage
Arithmetic and range coding	Yes	Fractional-bit entropy coding	Closer to entropy	Often used inside newer formats
Lossy codecs (JPEG, MP3)	No	Discard imperceptible detail	Very large on media	Never for text and code

The exact relative strength of the lossless methods depends on your data, so measure on your own samples, as the Brotli and gzip numbers earlier show.

FAQ

What is the difference between lossless and lossy compression?
Lossless compression restores the exact original bytes, so it suits text, code and data. Lossy compression discards detail people are unlikely to notice, so it suits photos, audio and video.

What is the difference between Huffman coding and LZ77?
Huffman coding gives short bit codes to frequent symbols. LZ77 replaces repeated strings with pointers to earlier copies. They remove different kinds of redundancy, which is why DEFLATE uses both.

Are gzip, zlib and DEFLATE the same thing?
They share the same compression algorithm but use different wrappers. Raw DEFLATE has no framing, zlib adds a 2-byte header and an Adler-32 checksum, and gzip adds a 10-byte header and a trailer with a CRC-32 and the original size.

Why does repeated text compress so well?
Repetition lets LZ77 replace later copies with short pointers, and skewed symbol frequencies let Huffman coding use fewer bits. On our test, a phrase repeated 1,000 times shrank to about half a percent of its size.

Why did my file get bigger after compression?
The input was probably tiny, random, encrypted or already compressed, so the fixed overhead exceeded the saving. A 2-byte input became 10 bytes with zlib in our test.

Can I compress encrypted data?
You can, but it will not shrink, because ciphertext looks random. Compress before encrypting, and be aware of length-based leaks when secrets and attacker-controlled text share a response.

What are CRIME and BREACH?
They are attacks that recover secrets by watching how the size of compressed, encrypted traffic changes when the attacker injects guesses. CRIME targeted TLS compression, and BREACH targeted HTTP response compression.

Does the DevUtilX Text Compressor use gzip?
No. It is a text minifier that collapses extra whitespace and blank lines, and it shows the size before and after. It processes your text in the browser. For entropy-level compression, use gzip or Brotli on your server or the command line.

Conclusion

Lossless compression is two simple ideas working together. Huffman coding spends fewer bits on common symbols, and LZ77 replaces repeated strings with pointers. DEFLATE combines them, and that combination powers gzip, zlib, ZIP, PNG and most of the web. Repetitive text shrinks dramatically, while tiny, random, encrypted and already compressed inputs do not, and a compressor that mixes secrets with attacker-influenced text can leak them through size alone. Measure real data in bytes, enable compression for text types at the edge, and choose levels by measurement.

To clean up whitespace in a block of text and see the byte savings straight away, try the Text Compressor. To keep going, read our guides to JSON minification and payload size and JavaScript minification, which show how to combine minification and compression on real assets.

Further reading

• Data compression – Wikipedia
• Deflate – Wikipedia
• RFC 1951: DEFLATE Compressed Data Format Specification
• RFC 1952: GZIP file format specification
• Huffman coding – Wikipedia
• LZ77 and LZ78 – Wikipedia
• HTTP compression – MDN
• CRIME – Wikipedia