Monitoring Server Uptime and Getting Alerts

A CS 1.6 server that crashes at 3am and stays down until you wake up loses its regulars fast. There are two separate problems to solve: making the process come back on its own, and knowing when it didn't. This covers auto-restart with systemd, a real availability check that queries the game port (not just the process), and a small alerting script you can run from cron.

1. Auto-restart with systemd

Running HLDS under systemd gives you automatic restart on crash for free. A minimal unit:

// /etc/systemd/system/cs16.service
[Unit]
Description=CS 1.6 Server
After=network.target

[Service]
User=hlds
WorkingDirectory=/home/hlds/hlds
ExecStart=/home/hlds/hlds/hlds_run -game cstrike +ip 0.0.0.0 +port 27015 +map de_dust2 +maxplayers 20
Restart=always
RestartSec=5

[Install]
WantedBy=multi-user.target

systemctl daemon-reload
systemctl enable --now cs16

Restart=always brings the process back whether it crashed or exited cleanly. That handles most outages, but it does not catch a process that is alive yet frozen — which is why you still need an external check.

2. Check availability the right way

Checking that the process exists (pgrep hlds) is weak: a hung server still has a running process. The reliable test is to send it an A2S_INFO query on its UDP game port and require a reply. A tiny Python check:

#!/usr/bin/env python3
import socket, sys
HOST, PORT = "127.0.0.1", 27015
pkt = b"\xFF\xFF\xFF\xFFTSource Engine Query\x00"
s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
s.settimeout(3)
try:
    s.sendto(pkt, (HOST, PORT))
    s.recvfrom(1400)          # any reply means it is answering
    sys.exit(0)
except Exception:
    sys.exit(1)

Exit 0 means the server answered a real client query; exit 1 means it is down or frozen. This is the same query a client's server browser sends, so it tests what players actually experience. Make sure UDP 27015 is reachable — see which ports to open.

3. Send an alert on failure

Wrap the check so it posts to a webhook (Discord, Slack, a status page — anything that takes an HTTP POST) only when the server is down:

#!/bin/bash
if ! /usr/local/bin/cs_check.py; then
  curl -s -X POST "$WEBHOOK_URL" \
    -H 'Content-Type: application/json' \
    -d '{"content":"CS 1.6 server 27015 is DOWN"}'
fi

Run it from cron every minute:

* * * * * WEBHOOK_URL='https://your-webhook' /usr/local/bin/cs_alert.sh

To avoid a message every single minute during a long outage, have the script write a small state file and only alert on the transition from up to down (and again on recovery). Even the naive version above is a big improvement over finding out from an angry player.

4. Watch the things that predict a crash

Uptime is downstream of resource health. Two cheap habits:

• Log server FPS and player count periodically — a server FPS that collapses under load precedes choke and drops. The sys_ticrate guide explains what a healthy value looks like.
• Watch the AMXX logs for repeating run-time errors; a plugin looping on an error can drag the whole server down. Load a suspect plugin in debug mode to pin it.

It is also worth watching the host, not just the game: memory pressure, a full disk in cstrike/logs/, or a runaway process on the same box will take the server down in ways no in-game check predicts. A one-line cron that alerts on disk usage over 90% or free memory near zero catches the slow failures that a per-minute A2S probe only notices once it is already too late.

Troubleshooting

• systemd restarts in a tight loop — the server exits immediately (bad startup line, missing files). Read journalctl -u cs16 -n 50; RestartSec=5 keeps the loop from hammering the box while you fix it.
• Check reports down but players are on — you queried the wrong IP/port, or the query is blocked by the firewall while the game traffic is not. Point the check at 127.0.0.1 from the same host.
• Alerts never fire — the webhook URL is wrong or unreachable, or cron's environment lacks it. Test the curl line by hand first.
• Constant false alarms — a 3-second timeout is too tight for a loaded server; raise it, and only alert on state changes.

Verification

Stop the server deliberately and confirm the whole chain works:

systemctl stop cs16
/usr/local/bin/cs_check.py; echo "exit $?"   // expect exit 1

You should receive one alert, and with Restart=always the service should be back within seconds of a crash (a manual stop stays down, as intended). Start it again and confirm the check returns exit 0. Once the loop is proven, layer the resource checks on top so you are warned before the crash, not after.