Compressing FastDL Files with bzip2 (and Why It Matters)

FastDL already moves downloads off the slow game channel onto HTTP (setup here). bzip2 compression is the next win on top of that: the client fetches a compressed copy and decompresses it locally, cutting map and model transfer sizes by roughly half. For a 20 MB custom map that is the difference between a fast join and a player giving up. The mechanism is simple but has one strict naming rule that trips everyone up.

1. How the client uses .bz2

When the GoldSrc client needs a file over FastDL, it does not ask for the raw file first — it asks for the bzip2-compressed version, and only falls back to the uncompressed file if the compressed one is absent:

1. GET /cstrike/maps/de_rats.bsp.bz2    (tried first)
2. GET /cstrike/maps/de_rats.bsp        (fallback if .bz2 is missing)

So if you place a .bz2 next to every large file, clients transparently download the smaller version and unpack it into their own game folder. You do nothing on the client side; the engine handles decompression.

2. The naming rule (the part everyone breaks)

The compressed filename must be the full original filename plus .bz2 — keep the original extension in the middle:

Dropping the original extension (de_rats.bz2) is the classic mistake — the client requests de_rats.bsp.bz2, gets a 404, and silently falls back to the slow uncompressed download. The good news: bzip2 produces exactly the right name by default.

3. Compress a single file

Use bzip2 -k so it keeps the original — you need both copies (see step 5):

cd /var/www/fastdl/cstrike/maps
bzip2 -k de_rats.bsp        # creates de_rats.bsp.bz2, keeps de_rats.bsp

Without -k, bzip2 deletes the source after compressing, which breaks the fallback path and any client that asks for the raw file.

4. Compress a whole mirror at once

You will not do this by hand for hundreds of files. This script walks the mirror and compresses only the content types worth compressing, skipping anything that already has an up-to-date .bz2:

#!/bin/bash
# bz2-mirror.sh -- compress a FastDL tree
ROOT=/var/www/fastdl/cstrike
find "$ROOT" -type f \
  \( -name '*.bsp' -o -name '*.mdl' -o -name '*.wav' \
     -o -name '*.spr' -o -name '*.tga' \) \
  ! -name '*.bz2' -print0 |
while IFS= read -r -d '' f; do
  if [ ! -f "$f.bz2" ] || [ "$f" -nt "$f.bz2" ]; then
    bzip2 -kf "$f"
    echo "compressed $f"
  fi
done

The [ "$f" -nt "$f.bz2" ] test re-compresses only files newer than their existing archive, so re-running after adding a map is cheap. Run it from cron after every content sync, or right after you copy new maps to the mirror.

5. Why you keep the uncompressed originals

It is tempting to delete the raw .bsp once the .bz2 exists to save disk. Do not. Two reasons: the fallback path (some clients or edge cases still request the plain file), and the fact that your own server may serve the mirror path directly. Disk is cheap; a broken fallback is a stream of \"missing textures\" complaints. Keep both.

6. What not to bother compressing

bzip2 helps on large, compressible assets — maps, models, sounds, sprites. It does almost nothing for files that are already compressed (JPEG/PNG images, .mp3) and adds request overhead for tiny files. Compress the big stuff; leave already-compressed formats alone. The headline win is maps, which are large and highly compressible.

Common errors

• Downloads still slow despite .bz2 files — wrong name (de_rats.bz2 instead of de_rats.bsp.bz2), so the client 404s and falls back. Rename to include the original extension.
• Client gets missing content after you compressed — you ran bzip2 without -k and deleted the originals, breaking the fallback. Regenerate the raw files.
• Web server refuses .bz2 — the mirror does not serve the extension as bytes. Add .bz2 to the served types as application/octet-stream (see the FastDL setup nginx/apache config).
• .bz2 is bigger or same size — you compressed an already-compressed file (a JPEG). Skip those formats.
• New map not compressed — you added it to the rotation and the mirror but forgot to re-run the compress script. Automate it after every sync.

Verification

Confirm the compressed file exists at the exact path the client will request, and that it is meaningfully smaller:

ls -la /var/www/fastdl/cstrike/maps/de_rats.bsp*
# expect de_rats.bsp AND de_rats.bsp.bz2, the .bz2 roughly half the size
curl -sI http://dl.example.com/cs/cstrike/maps/de_rats.bsp.bz2 | head -1
# expect HTTP/1.1 200 OK

Then join from a client missing that map and watch it pull the .bz2 and decompress in a second or two. If you see the raw .bsp being fetched instead, the compressed name or the web server type mapping is wrong.