Server security check
Confirmed safeWhy we built it, and what it checks
The job was to find anything on the whole server that shouldn't be there, hidden back doors, code that could be hijacked, passwords or keys left in the open, and settings left dangerously loose. A public tool checks a site from the outside, the way a visitor's browser sees it; that is blind to what happens behind the scenes on the machine itself, the files on disk, which programs run and with how much power, and how the server is logged into. So we built one program to walk the entire server and check it all in one pass. Rather than settle for the obvious, we kept adding to its checklist until it covered 25 separate kinds of problem across the code, the running programs, the secrets, the components and the wider machine.
The first run, and a fair challenge
It finished quickly and returned a long list. The speed was rightly questioned, a real check of a whole server shouldn't finish in a blink. Two things were true: reading through a folder of text files genuinely is fast, so that alone wasn't suspicious; but the scanner had only looked inside one folder, because of a mix-up between where we'd dropped the script and what we actually wanted it to scan. Pointed properly at the top of every site on the machine, it produced the real picture: 253 possible issues.
Sorting real problems from false alarms
The 253 split three ways, genuine problems, the scanner being over-cautious, and things already done right. What follows is every fix that came out of that: first the real changes made to the server, then the corrections to the scanner itself. Those tool corrections are the "generation issues" this page is named for, building an honest checker meant first fixing the checker.
The real server fixes
1 · Sensitive settings files locked down. Several files holding the site's most sensitive secrets, database passwords and private keys, were readable by any account on the machine, not just the website itself. We tightened access on every one so only the owner can read them, then checked each: the display flipped from "owner, group and others can read" to "owner only."
2 · Backups removed from public folders. Backup copies of pages and of the database were sitting inside folders anyone can reach by guessing the address. We listed every one first so it could be eyeballed, deleted exactly those, then re-ran the listing to confirm it came back empty. Nothing outside those folders and nothing that wasn't a backup was touched.
3 · Duplicate all-powerful programs removed. Four programs were running with full administrator power. Before removing anything we checked which programs were actually answering live visitor traffic, by matching each to the exact network "door" it was listening on. The site's real copies were already running safely under a restricted account and held the live doors; the powerful duplicates held none, dead leftovers. Only then did we remove the two duplicates, lock in the change so they wouldn't return after a restart, and confirm the live site still served correctly.
Fixing the scanner itself, the generation issues
4 · Database wrongly reported as open to the internet. The scanner flagged the database as reachable from the whole internet. It wasn't, it was correctly locked to the machine only. The scanner had been fooled by a column in the output that always shows a wildcard "any address" value regardless of the real setting. We fixed it to read only the column that actually states the bind address.
5 · Settings-file check too strict. The check insisted on one exact permission value, which meant it would have wrongly complained about an even stricter, safer setting. We changed it to simply ask "can anyone other than the owner read this?" and flag only if the answer is yes.
6 · Encryption result not explained. The encryption check ran correctly but printed the raw technical result without saying what it meant, so a genuinely good outcome (an outdated, insecure protocol being refused) read like an unexplained error. We made it interpret the result and label it clearly as a pass.
7 · Folder-listing check tripping on a switched-off line. The check for "can people browse folder contents" would trigger on a configuration line that had been commented out (switched off) rather than one actually active. We taught it to ignore switched-off lines.
8 · Telling a real use of a risky feature from writing about it. The scanner mistook a blog article that was about a risky programming feature for code actually using it. We fixed the detection to tell the two apart, and caught our own mistake doing so, because the first attempt wrongly cleared a genuine use that sat tucked inside a line with other code around it. We fixed that and re-tested against a set of real examples until every one was judged correctly.
9 · Ignoring third-party code we don't own. A lot of noise came from the internal workings of well-known third-party libraries and a bundled chat app, code we neither wrote nor can change. We taught the scanner to skip genuine third-party and machine-generated bundles for the code checks, which is the correct call (unlike skipping our own files, which would be dodging).
10 · Tightening what counts as a "leaked password". The scanner was matching ordinary words like "secret" or "token" buried inside compressed third-party code as if they were real credentials. We tightened it to only flag something with the actual shape of a credential, and to ignore values that are correctly pulled from a safe settings source rather than written into the code.
11 · Ranking backup files by real danger. Not every stray backup is equally risky. We changed the check so backups sitting in a public folder, or ones that look like they contain settings/credentials, are flagged as high priority, while the rest are noted as lower, so the truly dangerous ones stand out instead of drowning in the list.
The result
12 · The count coming down. As the real fixes and the accuracy corrections went in, the number of findings fell from 253 to about 197 once the biggest false-alarm category was fixed, and lower again in later parts as switched-off sites were skipped and scattered backups tidied away, eventually a handful that are nothing but our own deliberate backup archives, kept in a private folder the internet can't reach. The point was never only to fix the real issues, but to end up with a checker accurate enough to trust, so a real problem can never hide inside a pile of false alarms.
Three threads that began in this sweep get their own full sections: turning off the powerful account's password login and the file-access snag (Part 2); a real, brief outage while making restarts safe (Part 3); and a genuine weakness fixed in an admin-only tool (Part 4).
Key points
- The test runs on the machine itself, so it sees what no online scanner can, actual files, permissions, and which programs run with what power.
- Its first run raised 253 flags; most were false alarms from third-party code, not real problems.
- The genuine fixes were few but real: a world-readable secrets file, a public backup file, and a duplicate all-powerful program.
- Just as much work went into fixing the checker itself so it stopped crying wolf, ending at 7–8 accurate findings.
- The goal was a checker accurate enough to trust, so a real problem can never hide inside a pile of noise.
How to run a server-wide security scan
- Point the checker at your whole hosting folder, not one site, on shared hosting one site's weakness exposes the others.
- Run it with enough permission to read every file, so nothing is skipped silently.
- Let it finish a full first pass and save the raw results before changing anything.
How to triage what a security scan reports
- Expect a large first count, a big number is normal on a first run.
- Separate real problems from third-party noise such as plugins, libraries and vendor files you don't control.
- Set aside anything already correct, so your list is only genuine issues.
- Rank what's left by real risk, exposed secrets and world-readable credentials first.
How to fix the issues a security scan finds
- Tighten any secrets file so only its owner can read it.
- Remove stray backups and database dumps from public folders.
- Shut down any duplicate all-powerful program.
- Where the checker flagged something harmless, tune the checker so it stops crying wolf.
- Re-run until every remaining flag is something real you recognise and accept.
Questions & answers
Why not just use an online scanner? An online scanner only ever sees your site the way an outside visitor's browser does, the pages, the headers, the TLS certificate. It has no way to see file permissions, hidden backup files sitting in folders, or which background programs are running and with how much power, because none of that is exposed over HTTP. Those invisible layers are exactly where the real risks in this audit turned out to be. That's why the main checks were run by a script on the machine itself, which can read the actual filesystem and process list. The online scanner and the on-machine script answer two different questions, and you need both.
253 problems sounds alarming, was it? No, the headline number was mostly noise, not danger. The large majority of those findings came from other people's code bundled into the site: plugins, third-party libraries, and minified vendor files that aren't yours to fix and aren't actually risky in context. The genuinely fixable problems were a small handful once that noise was stripped out. Rather than leave you to mentally filter 253 items every time, we spent the effort making the checker itself more accurate so the count reflects real risk. A tool that cries wolf 250 times is a tool people stop reading.
Why fix the checker instead of just reading past the false alarms? Because a checker you have to mentally filter is a checker that will eventually let a real problem slip past unnoticed. Every false alarm you learn to ignore trains you to skim, and the one genuine issue buried in the noise gets skimmed too. Accuracy isn't a nice-to-have here, it's the entire point of having the tool. So the fix was to teach the checker to recognise vendor code and safe-by-context patterns and stop counting them, without hiding anything genuinely risky. The result is a shorter list you can actually trust.
Is anything still flagged? Yes, but only things we put there on purpose. A few backup archives we created during the cleanup are still flagged, and they're kept in a private folder that the internet can't reach. They show up because the checker is doing its job and reporting every archive it finds, not because they're a problem. They're expected, known about, and safe where they sit. Leaving them flagged-but-explained is more honest than suppressing them and pretending the folder is empty.
Technical detail
Objective
An on-host scanner surfacing what an external, passive web scan structurally cannot see: filesystem permissions, process ownership/privilege, SSH config, and in-code patterns (injection sinks, hardcoded secrets, missing headers). Read-only by design, it reports, it never edits. Run as python3 server_audit.py /var/www/vhosts.
Scope, 25 checks
- Code:
eval()/child_process/new Function()/execSync; hardcoded secrets not fromprocess.env; SQL concat into.query(/.execute(; unescapedinnerHTML; missing headers. - Process/privilege: pm2/node as root; ownership mismatches.
- Secrets/config:
.envpermissions/web-exposure; stray backup/dump files. - Dependencies:
npm audit; Node/PM2 version vs EOL; Node--inspectflags. - Infrastructure: listening ports; DB bind address; firewall; SSH hardening; TLS legacy protocols; directory listing.
First run, the speed challenge, the path mix-up
Report format is [SEVERITY] Category: message. The first run returned fast and was rightly questioned:
$ python3 server_audit.py /var/www/vhosts/example.com/tmp ================================================================================ SECURITY AUDIT REPORT ================================================================================ [HIGH] Database exposure: bound to a public interface: 127.0.0.1:3306 ... 0.0.0.0:* [HIGH] Dangerous code: .../app-with-stats.js:7526, child_process usage [HIGH] Process ownership: pm2 process running as root (x4) [HIGH] SSH hardening: PermitRootLogin is set to yes [HIGH] Exposed backup/dump file: ... inside a public/ folder [MEDIUM] Missing security header: Content-Security-Policy not set anywhere ...
File checks are grep/regex over text files, fast, not suspicious. The real issue was scope: only tmp had been scanned, because that was the sole path argument, the folder the script file lived in had been conflated with the folder to scan. Re-pointed at the whole tree (the sites cross-reference each other): python3 server_audit.py /var/www/vhosts → 253 findings.
Triage, real vs. noise vs. already-fine
Real: .env at 644; .bak/.sql/.zip in public/ (e.g. site-audit/public/*.html.bak, site-audit/tranco.zip, site-audit/blog-posts.sql, a loose live_db_backup_20260624-081451.sql); 4 pm2 as root; PermitRootLogin yes; child_process.execFile at app-with-stats.js:7526 (later confirmed safe, admin-gated, parseInt()-clamped 1–400). Noise: eval() in blog prose (## The eval() Problem); innerHTML/secret/token inside vendor bundles (jQuery, Leaflet, a chat app's webpack). Already correct: X-Content-Type-Options, X-Frame-Options, Referrer-Policy, X-XSS-Protection set; DB on 127.0.0.1:3306.
Real server fixes (1–3)
1 · .env permissions 644 → 600. Reported octal 00644. Applied and verified:
$ chmod 600 /var/www/vhosts/example.com/seo-subscriptions/.env # ... every .env $ ls -la /var/www/vhosts/example.com/seo-subscriptions/.env -rw------- 1 example.com_████████ psacln 2038 Jul 21 14:38 .../.env
2 · Backups in public/ deleted, preview, delete, verify:
$ find /var/www/vhosts -path "*/public/*" -name "*.bak" -print
$ find /var/www/vhosts -path "*/public/*" -name "*.bak" -delete
$ find /var/www/vhosts -path "*/public/*" -name "*.bak" -print
(empty)
3 · Root-owned pm2 duplicates removed, PIDs mapped to listening ports proved the non-root site daemon already ran the live copies; the root copies listened on no port:
$ sudo ss -tlnp | grep node # mapped against pm2 jlist PORT PID PROCESS OWNER 4002 3645936 example-audit-dev example.com_██████ (non-root) 3002 3330141 example-audit example.com_██████ (non-root) 3000 2134877 seo-tools (example.net) root 3001 2619755 seo-subscriptions (example.net) root # root copies of example-audit / -dev listened on NO port -> dead duplicates $ sudo pm2 delete example-audit && sudo pm2 delete example-audit-dev && sudo pm2 save
A live homepage fetch then confirmed it still served. (Honest note made at the time: a homepage fetch confirms the site is up, not that every route is unaffected, a stronger "nothing disrupted" claim was walked back.)
Tool-accuracy fixes, the generation issues (4–11)
4 · DB-bind false positive. Matched 0.0.0.0 anywhere on the ss line, but the peer column is always 0.0.0.0:*. Fixed to read only the local-address column (parts[4]) and flag only if it starts 0.0.0.0/*:/[::].
5 · .env exact-value check. Flagged anything ≠ 600, wrongly flagging stricter 400. Fixed to test the group/other bits directly:
group_other_bits = mode_bits & 0o077 # any group/other access?
if group_other_bits != 0: log('MEDIUM','Env file permissions', ...)
6 · TLS raw dump. Forced a TLSv1.0 handshake correctly but printed raw openssl output; a real pass (no peer certificate) read like an error. Fixed to interpret and label it a pass.
7 · Directory-listing check. Plain substring match tripped on a commented # autoindex on. Fixed to skip lines starting with #.
8 · Real eval( vs. prose. Rewrote detection to separate a real invocation from writing that names it. First attempt 4/5, wrongly cleared a real call nested in a block on the same line, fixed to 6/6:
# prose, must NOT flag ## The eval() Problem eval() executes JavaScript from strings at runtime # real calls, MUST flag eval(cmd); const x = eval(userInput); if (x) { eval(cmd); } // <- first version wrongly cleared this
The final looks_like_real_eval_call() (in the script below) skips comments, rejects the "eval() is/executes/runs…" prose shape, and accepts an eval( that is assigned, called, or terminates a statement/block.
9 · Skip vendor/generated code. Added is_vendor_or_generated(), skips /vendor/, .min.js, and webpack bundles for the code checks. Correct for third-party code; our own files are never skipped by name.
10 · Tighten the secrets pattern. Required real credential shape (a quoted value of 12+ non-space chars after password/secret/api_key/token) and ignored any line pulling from process.env, killing the token/secret substring noise inside minified bundles.
11 · Rank backups by real risk. Split severity: backups inside a public/ folder, or whose name implies credentials (env), are HIGH (web-reachable / credential exposure); the rest are MEDIUM.
Result (12)
253 → 197 once the vendor-skip and the real eval fix cleared the biggest false-alarm categories; then lower in later parts (suspension/stopped-service awareness, backup archiving) to a steady 7–8, all intentional archives in a non-web-accessible tmp/. Accuracy was itself the goal. Threads continued elsewhere: SSH/root → Part 2; systemd reboot fault + pm2 kill outage → Part 3; admin innerHTML XSS → Part 4.
The script in full
Read-only. Shown exactly as built (after the fixes above), with only the site address changed to example.com.
#!/usr/bin/env python3
"""
Server-wide security audit for example.com
Run directly on the server: python3 server_audit.py /var/www/vhosts/example.com
Produces a single readable report. Read-only, makes no changes to any file.
"""
import os
import re
import sys
import json
import subprocess
import stat
import pwd
REPORT = []
def log(severity, category, msg):
REPORT.append({'severity': severity, 'category': category, 'msg': msg})
def run(cmd, timeout=30):
try:
r = subprocess.run(cmd, shell=True, capture_output=True, text=True, timeout=timeout)
return r.stdout.strip(), r.stderr.strip(), r.returncode
except Exception as e:
return '', str(e), -1
def walk_js_files(root):
skip_dirs = {'node_modules', '.git'}
for dirpath, dirnames, filenames in os.walk(root):
dirnames[:] = [d for d in dirnames if d not in skip_dirs]
for fn in filenames:
if fn.endswith('.js'):
yield os.path.join(dirpath, fn)
def is_vendor_or_generated(path):
"""Skip third-party/minified/bundled files, these aren't code you wrote
or maintain, so flagging their internals isn't actionable for you."""
lower = path.lower()
if '/vendor/' in lower or lower.endswith('.min.js'):
return True
try:
with open(path, 'r', errors='ignore') as f:
head = f.read(2000)
if 'webpackchunk' in head.lower() or 'webpackjsonp' in head.lower():
return True
except Exception:
pass
return False
def looks_like_real_eval_call(line):
stripped = line.strip()
if stripped.startswith('#'):
return False
after = stripped.split('eval', 1)[1] if 'eval' in stripped else ''
if re.match(r'\s*\(\)\s+(executes|is|runs|can|allows)', after, re.I):
return False
if re.search(r'[=(]\s*eval\s*\(', stripped):
return True
m = re.search(r'\beval\s*\([^)]*\)\s*([;\}\s]*)$', stripped)
if m:
return True
return False
def check_dangerous_code(root):
patterns = [
(r'child_process', 'child_process usage'),
(r'new Function\s*\(', 'new Function() usage'),
(r'execSync\s*\(', 'execSync() usage'),
(r'\.innerHTML\s*=', 'unescaped innerHTML assignment'),
]
for path in walk_js_files(root):
if is_vendor_or_generated(path):
continue
try:
with open(path, 'r', errors='ignore') as f:
for i, line in enumerate(f, 1):
if re.search(r'\beval\s*\(', line) and looks_like_real_eval_call(line):
log('HIGH', 'Dangerous code', f'{path}:{i}, eval() usage: {line.strip()[:120]}')
for pat, desc in patterns:
if re.search(pat, line):
log('HIGH', 'Dangerous code', f'{path}:{i}, {desc}: {line.strip()[:120]}')
except Exception:
pass
def check_hardcoded_secrets(root):
pat = re.compile(r'(password|secret|api[_-]?key|token)\s*[:=]\s*[\'"]([^\'" ]{12,})[\'"]', re.I)
for path in walk_js_files(root):
if is_vendor_or_generated(path):
continue
try:
with open(path, 'r', errors='ignore') as f:
for i, line in enumerate(f, 1):
if pat.search(line) and 'process.env' not in line:
log('HIGH', 'Hardcoded secret', f'{path}:{i}, {line.strip()[:120]}')
except Exception:
pass
def check_sql_concat(root):
pat = re.compile(r'\.(execute|query)\s*\([^)]*\+')
for path in walk_js_files(root):
try:
with open(path, 'r', errors='ignore') as f:
content = f.read()
for m in re.finditer(r'\.(execute|query)\s*\(\s*[\'"`][^\'"`]*\+', content):
line_no = content[:m.start()].count('\n') + 1
log('MEDIUM', 'SQL string concat', f'{path}:{line_no}, review for injection risk (check if concatenated value is user-controlled)')
except Exception:
pass
def check_security_headers(root):
headers = ['Content-Security-Policy', 'Strict-Transport-Security', 'X-Frame-Options', 'X-Content-Type-Options']
found = {h: False for h in headers}
for path in walk_js_files(root):
try:
with open(path, 'r', errors='ignore') as f:
content = f.read()
for h in headers:
if f"setHeader('{h}'" in content or f'setHeader("{h}"' in content:
found[h] = True
except Exception:
pass
for h, ok in found.items():
if not ok:
log('MEDIUM', 'Missing security header', f'{h} not set anywhere in codebase')
def check_npm_audit(root):
out, err, code = run(f'npm audit --prefix {root} --json', timeout=60)
if out:
try:
data = json.loads(out)
vulns = data.get('metadata', {}).get('vulnerabilities', {})
for level, count in vulns.items():
if count and level != 'info':
log('HIGH' if level in ('high', 'critical') else 'MEDIUM', 'npm dependency', f'{count} {level}-severity vulnerable package(s)')
except Exception:
log('LOW', 'npm audit', 'Could not parse npm audit output, run manually to review')
else:
log('LOW', 'npm audit', f'npm audit produced no output ({err[:200]})')
def check_process_ownership():
out, err, code = run("pm2 jlist")
if out:
try:
procs = json.loads(out)
for p in procs:
pid = p.get('pid')
name = p.get('name')
if pid:
uid_out, _, _ = run(f"ps -o user= -p {pid}")
user = uid_out.strip()
if user == 'root':
log('HIGH', 'Process ownership', f'pm2 process "{name}" (pid {pid}) is running as root, should run as a dedicated non-root user')
except Exception:
log('LOW', 'Process ownership', 'Could not parse pm2 jlist output, run manually: pm2 jlist')
else:
log('LOW', 'Process ownership', 'pm2 not found or no processes listed')
def check_env_file(root):
for dirpath, dirnames, filenames in os.walk(root):
for fn in filenames:
if fn == '.env' or fn.startswith('.env.'):
path = os.path.join(dirpath, fn)
try:
mode_bits = os.stat(path).st_mode
mode = oct(mode_bits)[-3:]
group_other_bits = mode_bits & 0o077
if group_other_bits != 0:
log('MEDIUM', 'Env file permissions', f'{path} has permissions {mode}, group/other should have no access (expected 600 or 400)')
except Exception:
pass
def check_stray_backup_files(root):
exts = ('.bak', '.old', '.sql', '.zip', '.tar.gz', '.tar')
skip_markers = ('prepackaged_plugins', '.npm/_prebuilds', 'node_modules')
for dirpath, dirnames, filenames in os.walk(root):
if 'node_modules' in dirpath or '.git' in dirpath:
continue
if any(marker in dirpath for marker in skip_markers):
continue
for fn in filenames:
if fn.lower().endswith(exts):
full = os.path.join(dirpath, fn)
in_public = '/public/' in full or full.rstrip('/').endswith('/public')
if 'env' in fn.lower():
log('HIGH', 'Exposed backup/dump file', f'{full}, contains "env", may include credentials')
elif in_public:
log('HIGH', 'Exposed backup/dump file', f'{full}, inside a public/ folder, likely web-reachable by direct URL')
else:
log('MEDIUM', 'Exposed backup/dump file', full)
def check_node_flags():
out, err, code = run("ps aux | grep node | grep -v grep")
if '--inspect' in out:
log('HIGH', 'Node debug flag', 'A node process is running with --inspect/--inspect-brk exposed, remote debugger risk')
def check_node_version():
out, err, code = run("node --version")
log('INFO', 'Node version', f'Running {out.strip() or "unknown"}, verify against https://nodejs.org/en/about/previous-releases for EOL/CVE status')
def check_pm2_version():
out, err, code = run("pm2 --version")
log('INFO', 'PM2 version', f'Running {out.strip() or "unknown"}, verify this is current')
def check_listening_ports():
out, err, code = run("ss -tulpn 2>/dev/null || netstat -tulpn 2>/dev/null")
log('INFO', 'Listening ports', out or 'Could not determine, run: ss -tulpn')
def check_mysql_bind():
out, err, code = run("ss -tulpn 2>/dev/null | grep ':3306\\|:5432'")
if out:
exposed = False
for line in out.splitlines():
parts = line.split()
if len(parts) > 4:
local_addr = parts[4]
if local_addr.startswith('0.0.0.0') or local_addr.startswith('*:') or local_addr.startswith('[::]'):
exposed = True
if exposed:
log('HIGH', 'Database exposure', f'Database appears to be bound to a public interface: {out}')
else:
log('INFO', 'Database exposure', f'Database bound to loopback only (safe): {out}')
else:
log('INFO', 'Database exposure', 'No MySQL/Postgres port found listening, verify manually if unexpected')
def check_firewall():
out, err, code = run("ufw status 2>/dev/null || iptables -L 2>/dev/null")
log('INFO', 'Firewall status', out[:500] or 'Could not determine firewall status, check manually')
def check_ssh_config():
try:
with open('/etc/ssh/sshd_config', 'r', errors='ignore') as f:
content = f.read()
if re.search(r'^\s*PermitRootLogin\s+yes', content, re.M):
log('HIGH', 'SSH hardening', 'PermitRootLogin is set to yes, root SSH login should be disabled')
if re.search(r'^\s*PasswordAuthentication\s+yes', content, re.M):
log('MEDIUM', 'SSH hardening', 'PasswordAuthentication is yes, key-only auth recommended')
except Exception:
log('LOW', 'SSH hardening', 'Could not read /etc/ssh/sshd_config, check permissions/manually review')
def check_tls():
out, err, code = run("openssl s_client -connect example.com:443 -tls1 </dev/null 2>&1 | grep -i 'Cipher is\\|no peer certificate'")
if 'no peer certificate' in out.lower() or not out:
log('INFO', 'TLS check', 'TLSv1.0 handshake rejected (good), legacy protocol not supported')
elif 'cipher is (none)' in out.lower():
log('INFO', 'TLS check', 'TLSv1.0 handshake rejected (good), legacy protocol not supported')
else:
log('MEDIUM', 'TLS check', f'TLSv1.0 handshake succeeded, legacy protocol still supported: {out}')
def check_directory_listing(root):
for conf_path in ['/etc/nginx/sites-enabled', '/etc/nginx/conf.d', '/etc/apache2/sites-enabled']:
if os.path.isdir(conf_path):
for fn in os.listdir(conf_path):
full = os.path.join(conf_path, fn)
try:
with open(full, 'r', errors='ignore') as f:
for line in f:
stripped = line.strip()
if stripped.startswith('#'):
continue
if 'autoindex on' in stripped or 'Options +Indexes' in stripped:
log('MEDIUM', 'Directory listing', f'{full} has directory listing enabled')
except Exception:
pass
def main():
root = sys.argv[1] if len(sys.argv) > 1 else '.'
print(f"Running security audit against: {root}\n")
check_dangerous_code(root)
check_hardcoded_secrets(root)
check_sql_concat(root)
check_security_headers(root)
check_npm_audit(root)
check_process_ownership()
check_env_file(root)
check_stray_backup_files(root)
check_node_flags()
check_node_version()
check_pm2_version()
check_listening_ports()
check_mysql_bind()
check_firewall()
check_ssh_config()
check_tls()
check_directory_listing(root)
sev_order = {'HIGH': 0, 'MEDIUM': 1, 'LOW': 2, 'INFO': 3}
REPORT.sort(key=lambda x: sev_order.get(x['severity'], 9))
print("="*80)
print("SECURITY AUDIT REPORT")
print("="*80)
for item in REPORT:
print(f"[{item['severity']}] {item['category']}: {item['msg']}")
print("="*80)
print(f"Total findings: {len(REPORT)}")
if __name__ == '__main__':
main()
Key points
- A local filesystem/process audit surfaces what passive HTTP/TLS scanners structurally cannot: perms, ownership, process privilege, SSH policy.
- Scope grew to 25 checks across code, secrets, process privilege, config, network, SSH, TLS, and dependencies.
- First run: 253 findings → real fixes were
.env644→600, a public.bak, and a duplicate root-owned pm2 app. - The bulk of the work was false-positive suppression (vendor-skip, taint tracking, tightened regexes) → 253 → 197 → ~7–8.
- End state: findings that remain are intentional, owner-created backup archives in a non-web-reachable path.
How to do this yourself
$ sudo python3 server_audit.py /var/www/vhosts # scan the whole vhost tree, not one site # triage the output into three buckets: # real -> fix (perms, public backups, root-owned procs) # vendor -> ignore (plugin/library code you don't own) # fine -> accept (already-correct config) $ chmod 600 path/to/.env # secrets readable only by owner $ rm public/old-file.bak # remove stray backups from web-served dirs $ sudo pm2 delete <duplicate-root-app> && sudo pm2 save # then re-run until every remaining finding is one you recognise
Questions & answers
Why a local script over an external scanner? External scanners are limited to what travels over the network, HTTP responses, the TLS handshake, DNS records. File permissions, file ownership, how much privilege a running process holds, and the SSH login policy have zero visibility over HTTP; nothing about them is sent to a visitor. The only thing that can inspect those is a tool running on the filesystem itself, with the rights to read them directly. That's why the deep checks were a local script rather than a URL scan. Each approach sees a layer the other is blind to.
Why did 253 collapse to ~8? Most of the original 253 were third-party bundles, minified and vendor JavaScript, plus patterns that are only dangerous in certain contexts and were safe in this one. Adding vendor-skip filtering removed the code that isn't yours to fix, and taint tracking told the difference between input that could carry an attack and input that provably couldn't. Together those two changes stripped the noise without hiding a single genuine issue. What was left, around eight, is the real, actionable set. The drop is the checker getting more accurate, not less thorough.
Isn't suppressing findings dangerous? It would be if suppression worked by location, 'ignore anything in this file', because a real problem could move into that file and vanish from the report. This checker suppresses by reason instead: a finding is dropped because the code is in a vendor path, or because the input feeding it is provably untainted. That means a genuinely risky instance of the very same pattern still flags, because its reason is different. Suppressing the why, not the where, keeps the safety net intact. That distinction is the whole reason the collapse from 253 is trustworthy.
What's left in the report? What remains are the deliberate backup archives created during the cleanup, and they sit in a tmp/ folder that returns a 404, meaning it isn't reachable from the web at all. They're reported because the checker faithfully lists every archive it finds, not because they pose a risk. They're expected and accepted, not something to action. Keeping them visible-but-explained is deliberate: a report that hid them would be less honest than one that shows them with context. Nothing here is a live vulnerability.
Login security tightened
FixedWhat the scan flagged, and why it was the most urgent thing on the list
The server audit reported that the most powerful account on the machine, the "root" account, which can do absolutely anything, read or change any file, on any of the sites hosted there, could be logged into with just a password. On any server that is the single most attractive target there is, and it is worth being clear about why. A password is a secret that only has to be discovered once: it can be guessed by automated attacks that try thousands of combinations a second against an exposed login, around the clock, forever; it can be leaked in a breach of some completely unrelated website where the same password happened to be reused; or it can simply be worked out from information about the owner. And because this particular account can touch every other site on the box, not just its own, a single successful guess would not compromise one website, it would compromise all of them at once. That is why closing this door mattered more than almost anything else the audit found.
The plan, and why we deliberately did not just switch the login off
There were two moves to make, and the order of them mattered. First, create a new, limited administrator account for everyday work, so that routine tasks stop running with unlimited power by default, the principle being that you should only hold full power in your hands at the exact moment you need it, not all the time. Second, change the powerful account itself from "a password will let you in" to "only a cryptographic key will let you in".
The tempting shortcut would have been to simply turn the powerful account's remote login fully off. We deliberately did not do that, and the reason is important: the file-management tool used to move files on and off the server (FileZilla) reaches the machine through that same account. Switching its login fully off would not just close the door to attackers, it would lock out a tool actually in daily use. So the real target was the precise middle setting: refuse all passwords, but still accept a valid key. That keeps the legitimate tool working for whoever physically holds the key, while removing the password that anyone in the world could try to guess.
1 · The limited account, created, and proven before anything else was touched
We created the new limited account and gave it the ability to carry out full-power administrative tasks when it explicitly asks to, the normal, safer pattern where you work as an ordinary user and elevate to full power only for the specific command that needs it. Then, before touching the powerful account's login settings at all, we proved the new account actually worked: we had it perform a task that requires full power and confirmed it succeeded. This proof was the safety net for everything that followed. The single most dangerous moment in this whole job is the point where you change the login rules and restart the login service, if you have made a mistake, that is the moment it locks you out with no way back in. By confirming the new way in worked first, we made sure that even if the change to the powerful account went wrong, there was still a proven, working route onto the server.
2 · Turning off the password login, and the lockout it immediately caused
We switched off password login for the powerful account and restarted the login service to apply the change. Almost at once a real, unplanned problem surfaced: the file-management tool stopped connecting. The reason is a piece of plumbing that is easy to miss, the file tool does not use its own separate channel; it rides on the very same secure login system we had just tightened (file transfer here is a feature built on top of the secure shell, not a thing beside it). It had quietly been authenticating as the powerful account using that account's password. The moment we forbade passwords for that account, the file tool lost its way in along with the attackers we were trying to keep out. This was not a hypothetical side effect noticed in review, it was an immediate loss of a tool in active use, and it had to be solved before we could go any further, which is exactly what the next steps did.
3 · Setting up the key-based login in its place
We generated a cryptographic key pair on the local computer. A key pair is two matching halves: a private half, which is a secret file that never leaves your own machine and stands in for the password, and a public half, which is not secret and is placed on the server. The server, holding the public half, will only let in someone who can prove they hold the matching private half, and because the private half never travels across the network the way a typed password does, there is nothing for an eavesdropper to capture and nothing for a guessing attack to grind against. We placed the public half on the server for the powerful account and set that account to the middle position described earlier: no passwords accepted, but a valid key accepted. That single change restores the file tool's way in, but now only for the person physically holding the private key.
4 · The fiddly part, getting the key into the right format, and the real mistakes made doing it
The file-management tool does not accept the key in the standard format it was created in; it needs the key converted into its own particular format first. This conversion is where genuine trial and error happened, and it is worth being candid about rather than presenting it as smooth. The conversion tool has a "Save public key" button and a "Save private key" button sitting right next to each other, and the wrong one was clicked more than once, which produced a file that looked for all the world like the converted private key the tool needed, but actually contained the public half, which is useless for logging in. The confusion was made considerably worse by the computer being set to hide file extensions: with the endings hidden, the original private key, its public counterpart, and the converted file all displayed under the same name, giving no visible way to tell three quite different files apart. We caught the mistake by opening the suspect file and simply reading it, it plainly announced itself, in its very first line, as a public key, then switched the computer's setting so that file extensions were visible again, which made the three files distinguishable at a glance, and repeated the export with the correct button. It was a small, human error, but a genuine one, and it is part of the honest record of how this was done.
5 · Confirming the whole thing actually works
We then confirmed the two facts that matter, rather than assuming them. First, that a password login to the powerful account is now refused outright, proving the door we set out to close is genuinely closed. Second, that a login using the key succeeds, proving we did not lock ourselves out in the process. Finally, the file-management tool was pointed at the newly converted key and reconnected cleanly, confirming the tool that broke in step 2 is fully restored on the safer footing.
6 · The one lasting consequence, stated plainly
There is a real, permanent trade-off to be honest about. Because everyday work now runs through the limited account rather than the all-powerful one, that account can only reach the files it has been given permission to reach. In practice this means that trying to browse a different site's folders through the everyday account is now refused, which is exactly correct and safe behaviour, and precisely the isolation we wanted, but it does change the daily routine: for work on other sites you now use either the powerful account's key or that site's own separate login, rather than reaching everything from one all-powerful account. Less convenient, deliberately, in exchange for a great deal less risk. This same trade-off surfaces again later in the audit.
Key points
- The most powerful account on the machine could be reached with just a password, the single most attacked door there is.
- Because that account can touch every site on the server, one guessed password would compromise all of them, not one.
- We didn't switch its login off, that would lock out the file tool that uses it. We set it to refuse passwords but accept a key.
- A limited everyday account was created and proven to work first, so we could never lock ourselves out making the change.
- Turning off the password briefly broke the file tool, because file transfer rides on the same login system, fixed by switching it to the key.
How to set up a safe everyday login on your server
- Create a separate, limited everyday account and give it powerful tasks only when explicitly requested.
- Prove the new account works by having it perform a task needing full power, before you change anything else.
- Only once that's confirmed, move on to hardening the powerful account.
How to switch your server to key-only login
- Switch the powerful account to "no passwords, key only" rather than disabling its login entirely.
- Generate a key pair; keep the private half on your own computer and place the public half on the server.
- If your file tool needs the key in its own format, convert it, and make file extensions visible so you pick the right one.
- Confirm a password login is now refused and a key login succeeds, then point your file tool at the key.
Questions & answers
Why not just turn off the powerful account's remote login completely? Because a file-management tool you use every day connects through that exact account, and turning its remote login off entirely would lock the tool out along with any attacker. The goal was to shut the door on attackers without breaking your own workflow. So instead of disabling the account, passwords were refused while key-based access was kept, an attacker has no password to guess, but your tool still gets in with its key. That's the difference between locking the door and bricking it shut. One keeps you out; the other keeps everyone out.
Why did the file tool suddenly stop working? File transfer runs over the same secure login system as everything else, and your file tool had been getting in using the account's password. The moment password logins were refused, the tool lost its way in, nothing was broken, it simply no longer had valid credentials. Switching it over to use the key instead restored access immediately. It's the same reason a door key stops working the day the lock is changed: expected, and fixed by handing over the new key. No data or settings were lost in the gap.
Why is a key safer than a password? A password is a shared secret, it can be guessed, reused across sites, phished, or leaked in a breach, and it physically travels across the network each time you log in, where it could be intercepted. A key works differently: its secret half never leaves your own computer, and only a matching public half sits on the server. Because the secret never travels, there's nothing to intercept in transit, and because it's long and random, there's nothing practical to guess. That combination is why key-based login is the standard for anything that matters. A password is a secret you send; a key is a secret you keep.
What changed day-to-day? Day-to-day work now runs through a limited account that can only reach the files it's specifically been given, rather than an all-powerful one. Reaching another site's files now requires either the key or that site's own login, instead of one account quietly having access to everything. It's deliberately a little less convenient, an extra step to cross between sites, and that friction is the point. The trade is a small amount of daily convenience for a large reduction in what a single compromised login could touch. Nothing you normally do is blocked; it's just properly scoped.
Technical detail
The finding
The scanner's SSH check reported PermitRootLogin yes in /etc/ssh/sshd_config, root reachable over SSH with password authentication. This is the highest-privilege account on the box, and it is directly exposed to credential-stuffing and brute-force attempts from the open internet. Because root's authority spans every vhost on the machine, a single compromise here is a whole-server compromise, not a single-site one, which is what makes it the top-priority finding.
The plan, and the reasoning behind the exact setting chosen
Standard hardening is a named sudo user for interactive work plus key-only root. The specific directive matters: we chose PermitRootLogin prohibit-password rather than the blunter PermitRootLogin no. no would forbid all direct root SSH, which would also kill the key-based root SFTP the file workflow depends on; prohibit-password refuses password auth while still permitting key auth, which is exactly the middle ground the workflow needs. The sequencing was chosen to be lockout-proof: establish and verify the alternative route (sudo user) before restarting sshd with the stricter root policy.
1 · Create the sudo user, and prove it before the risk moment
$ grep PermitRootLogin /etc/ssh/sshd_config PermitRootLogin yes $ sudo adduser serveradmin $ sudo usermod -aG sudo serveradmin $ sudo whoami root
The sudo whoami → root check is the safety net, and it was run before the sshd restart on purpose. An sshd restart is the exact point at which a bad config or a missing alternative route locks you out permanently; confirming the sudo path elevates correctly first guarantees a proven way back in even if the root-policy change had gone wrong.
2 · Disable root password login, and the SFTP lockout it caused
SFTP is not a separate service, it is a subsystem of SSH (Subsystem sftp in the daemon), so any change to the SSH auth policy applies to FileZilla's transfers too. FileZilla had been authenticating as root over SFTP using root's password; the moment password auth for root was refused, FileZilla lost its route in. Also worth recording for anyone reproducing this: on this box the service unit is named ssh, not sshd.
$ sudo systemctl restart ssh
# root terminal login AND FileZilla (root over SFTP) both now refused
3 · Key-based root via prohibit-password
Generated an ed25519 key pair locally, installed the public half into root's authorized_keys, locked that file's permissions, and moved the directive from yes to prohibit-password:
# on the local machine (PowerShell) $ ssh-keygen -t ed25519 -f C:\Users\██████\.ssh\example_key # example_key (private, stays local) + example_key.pub (public, goes to server) $ sudo sh -c 'cat >> /root/.ssh/authorized_keys' < example_key.pub $ sudo chmod 600 /root/.ssh/authorized_keys $ sudo sed -i 's/^PermitRootLogin yes/PermitRootLogin prohibit-password/' /etc/ssh/sshd_config $ sudo systemctl restart ssh
Key auth beats password auth here precisely because the private key never crosses the wire, there is no transmitted secret to intercept and no low-entropy target to brute-force, unlike a password prompt exposed to the internet.
4 · The .ppk conversion, the real diagnosis
FileZilla needs PuTTY's .ppk format, so the OpenSSH private key had to be converted in PuTTYgen (Load → Save private key). "Save public key" was clicked instead of "Save private key" more than once, producing a .ppk that actually held the public key, unusable for auth. Windows was hiding file extensions, so the extensionless OpenSSH private key, the .pub, and the .ppk were visually indistinguishable in the folder. We diagnosed it by dumping the suspect file and reading its header:
$ type $env:USERPROFILE\.ssh\example_key.ppk ---- BEGIN SSH2 PUBLIC KEY ---- Comment: "████████" ████████████████████████████████████████ ---- END SSH2 PUBLIC KEY ----
BEGIN SSH2 PUBLIC KEY is the giveaway, a real PuTTY private key would begin PuTTY-User-Key-File-. We enabled File name extensions in Explorer so the three files were distinguishable, then re-exported correctly and confirmed the set:
$ dir $env:USERPROFILE\.ssh
example_key (private, OpenSSH)
example_key.pub (public)
example_key.ppk (private, PuTTY format, the correct one, header: PuTTY-User-Key-File-3)
known_hosts
5 · Verification
$ ssh root@████████████ # refused, no password accepted for root $ ssh -i example_key root@████████████ # succeeds via key # FileZilla: Logon Type = Key file, Key = example_key.ppk, User = root -> reconnects cleanly
Both facts confirmed independently: password path closed, key path open, SFTP restored on the key.
6 · Lasting consequence
Interactive and day-to-day work now runs as serveradmin with sudo. Because serveradmin is an ordinary user and SFTP rides SSH, it only reaches files its own Unix permissions allow, attempting to browse another vhost's tree returns SSH_FX_PERMISSION_DENIED, which is the correct, intended isolation. Cross-site file access now requires sudo, root's key, or that site's own credentials, rather than one omnipotent account reaching everything. This trade-off recurs later in the audit.
Key points
- Finding:
PermitRootLogin yes, root reachable over SSH with password auth, the top brute-force/credential-stuffing target. - Root's authority spans every vhost, so a single compromise is whole-server, not single-site.
- Chose
prohibit-passwordoverno, refuses passwords but keeps key-based root SFTP working. - Lockout-proof sequencing: create + verify the sudo user (
sudo whoami → root) before the sshd restart. - SFTP rides SSH (
Subsystem sftp), so disabling root password auth broke FileZilla until it moved to the key.
How to do this yourself
$ sudo adduser serveradmin && sudo usermod -aG sudo serveradmin $ sudo whoami # -> root : prove elevation BEFORE touching sshd $ ssh-keygen -t ed25519 -f ~/.ssh/example_key # private stays local, .pub goes to server $ sudo sh -c 'cat >> /root/.ssh/authorized_keys' < example_key.pub $ sudo chmod 600 /root/.ssh/authorized_keys $ sudo sed -i 's/^PermitRootLogin yes/PermitRootLogin prohibit-password/' /etc/ssh/sshd_config $ sudo systemctl restart ssh # note: unit is 'ssh', not 'sshd', on this box $ ssh root@host # refused (no password) $ ssh -i example_key root@host # succeeds (key) # FileZilla: convert key to .ppk (PuTTYgen: Load -> Save PRIVATE key), then Logon Type = Key file
Questions & answers
Why prohibit-password and not no? Setting the option to no forbids all direct root login over SSH, and that would kill key-based root file transfer too, which the workflow relies on. prohibit-password is the precise middle setting: it refuses password logins while still allowing key logins. That's exactly the shape needed here, attackers get no password door, but the key-based tool keeps working. Choosing no would have been blunter than the problem required and would have broken a legitimate use. The narrower setting solves the risk without the collateral damage.
Why did FileZilla break? SFTP isn't a separate service, it's a subsystem that runs inside SSH, so it obeys the same login rules. FileZilla had been authenticating as root over SFTP using root's password, so the instant password authentication was refused, its login stopped working. Nothing in FileZilla itself broke; it simply presented a credential the server no longer accepts. Switching it to authenticate with the key restored the connection. It's the same login policy change surfacing in a second tool.
Why verify the sudo user before restarting sshd? Restarting sshd is the single moment a mistake in the config can lock you out of the server for good, because the new rules take effect and there may be no second way in. Confirming that sudo whoami returns root first proves you still have a working, privileged route back before you pull the trigger. It's a cheap check that turns an irreversible risk into a safe one. If that check had failed, you'd stop and fix access before touching sshd. Verify the escape hatch works, then change the lock.
What was the .ppk trap? The mistake was clicking 'Save public key' instead of 'Save private key', which produced a .ppk file containing only the public half, useless for logging in. The giveaway is the file's first line: a real PuTTY private key starts with PuTTY-User-Key-File-, while this bad file started with BEGIN SSH2 PUBLIC KEY. Hidden file extensions on the system made the two files look identical in the browser, which is how the wrong one got picked. Once the header was read directly, the mix-up was obvious. The fix was simply exporting the private key correctly.
A brief, real moment of downtime
FixedWhy we were looking at this at all
After the earlier clean-up work, removing duplicate all-powerful programs, tightening logins, the sensible next step was to make sure the whole server could survive being switched off and on again. A reboot is meant to bring every website back up automatically, without anyone having to log in and start things by hand. The trouble is that this only actually works if the machine has been told, correctly, which programs to bring back and how. If that instruction is missing or broken, a reboot doesn't warn you, the server simply comes back with the site dead, and stays that way until a human notices and intervenes. We wanted to confirm the automatic recovery genuinely worked before ever triggering a real reboot, precisely so we would never find out the hard way.
What we found, a safety net that wasn't actually holding anything
On the server there is a supervisor system whose entire job is to start the website's background programs and keep them running, including bringing them back after a reboot. When we inspected it, we found something quietly wrong: the part of it responsible for the main site was switched off and doing nothing, yet it was marked as enabled, so at a glance everything looked fine. The website's programs were in fact still running, but only because a copy of the supervisor had been started by hand in an earlier session and was holding them up. Nothing was actually relying on the automatic system that is supposed to take over after a reboot.
The plain-English version: the safety net existed on paper and looked to be in place, but nothing was resting on it. Everything was working at that moment purely because of a hand-started process that would not survive a restart. Reboot the machine in that state and the hand-started copy vanishes, the automatic system fails to take over, and the site does not come back. That is exactly the trap we were trying to find before it sprang.
1 · Saving the current state first
Before changing anything, we did the cautious thing: we saved an up-to-date snapshot of exactly which programs were currently running and how. This matters because the automatic recovery system restores from a saved list, and if that saved list is stale or empty, "recovery" brings back the wrong things or nothing at all. Capturing the correct current state first meant that whatever we did next, the system had an accurate record to rebuild from.
2 · Testing the recovery, and why it didn't work
Rather than assume the automatic system would work once switched on, we tested it in a controlled way, deliberately, while we were watching, not by gambling on a real reboot. It failed. The system reported that it could not complete its start-up steps and fell into a loop, trying and failing over and over. Digging into the cause, we found a specific setting that instructed it to look for a particular little "status file" as proof that start-up had succeeded, but the software in question never actually creates that file. So the check could never pass: every start-up looked like a failure to the supervisor, even when the programs themselves were fine, which is what drove the endless retry loop.
3 · The fix attempt that caused the outage
This is the honest part, reported in full rather than smoothed over. Our first attempt at a fix backfired and briefly took the live site offline. The logic seemed sound: remove the faulty "look for a status file that never exists" setting, then reload the supervisor so the change takes effect. But reloading it did something we had not anticipated. Buried in the supervisor's configuration was a separate instruction that said, in effect, "whenever you stop, for any reason at all, shut down every one of the website's programs as you go." A reload counts as a stop-and-start. So the instant we reloaded to apply our harmless-looking fix, that shutdown instruction fired, and it took down all of the site's running programs with it. Every part of the live website went offline as a direct, immediate result of our change. This was a genuine outage that we caused, not a near-miss, and it is part of the honest record.
4 · Getting back online
We brought everything back within a few minutes using the supervisor's own built-in "restore whatever was running" command, feeding it the accurate snapshot we had saved in step 1, which is exactly why saving that state first mattered. There was one awkward wrinkle worth explaining. For security, the website's own account is deliberately configured so that it cannot open a normal interactive command session, a sensible hardening measure that stops that account being used as a way in. But that same restriction blocks the straightforward way of starting the programs back up under that account. So the restore had to be run through a specific, explicit workaround that launches the recovery correctly as that locked-down account despite the restriction. Once that went through, all of the site's programs were back and the site was fully restored.
5 · Finding and removing the real cause
With the site safe again, we went back and traced the true culprit properly, rather than papering over it. The real problem was never the missing "status file" setting on its own, it was that buried "shut everything down on any stop" instruction. That kind of instruction has a legitimate use for a genuine, final shutdown of the machine, but here it was written to fire on every stop and restart, including entirely routine ones. That is what turned an ordinary reload into a site-wide outage, and it would have done the same on any future maintenance restart too. So we removed that instruction outright, eliminating the actual mechanism of the outage rather than just avoiding the one action that had triggered it.
6 · Testing a real restart, no interruption this time
Having removed the real cause, we did not simply declare it fixed, we proved it. We deliberately triggered a genuine restart of the supervisor, the very kind of action that had caused the outage the first time. This time every one of the website's programs stayed running straight through it, with no gap and no interruption at all. That is the difference between a fix that was merely worked around and one that is genuinely gone: the same action that broke the site before now runs cleanly.
7 · Outcome
The fault is permanently fixed and tested. Two real gains came out of it: the automatic recovery system now starts cleanly and will bring the site back by itself after a reboot instead of leaving it dead, and the specific hidden instruction that caused the outage is gone, so routine maintenance restarts can no longer take the site down. The brief downtime along the way was real, was caused by our own first attempt, and was resolved within minutes, and the end state is meaningfully safer than where we started, which is the whole point of having found this before a real reboot did.
Key points
- A boot service can be enabled but dead, it looks fine while nothing actually relies on it.
- The live programs were being held up only by a hand-started copy that would not survive a reboot.
- A hidden "shut everything down on any stop" instruction turned a routine reload into a full outage.
- Recovery needed a workaround because the site's account is deliberately barred from opening a normal session.
- Removing that instruction and then testing a real restart made the fix genuine, not merely worked around.
How to check your services survive a reboot
- Check the boot service is truly running, not just enabled, an "enabled" service showing "dead" won't restart anything.
- Save the current list of running programs first, so recovery has an accurate record to rebuild from.
- Test recovery deliberately, while watching, never gamble on a real reboot to find out.
How to fix services that die on restart
- If a restart takes everything down, look for a "kill everything on stop" instruction in the service definition.
- Remove that instruction, it belongs only to a genuine shutdown, then reload the service definition.
- Trigger a real restart and confirm every program stays up, with unbroken running time.
Questions & answers
Isn't "enabled" the same as "running"? No, they're two separate states, and both have to be true for the safety net to actually hold. 'Enabled' only means a service is set to start automatically at the next boot; it says nothing about whether it's running right now. A service can be enabled yet switched off and doing nothing at this moment, or running yet not enabled and therefore gone after a reboot. The check confirmed both: set to start at boot, and actually up. Assuming one implies the other is exactly how a reboot surprise happens.
Why did simply reloading take the site down? The supervisor that manages the site's programs had a hidden instruction to shut down every program whenever it itself stops. A reload counts as a stop-then-start, so that shutdown instruction fired and took the live programs down with it. Nobody expected a routine reload to be destructive, which is exactly why it was dangerous. The instruction belonged only to a genuine, deliberate shutdown, not to a reload. Removing it was the real fix.
Why not just restart normally to bring it back? The site's account is deliberately locked down so it can't open an ordinary interactive session, a security measure that also happens to block the normal restart path. So bringing the programs back couldn't use the usual command; it had to go through a specific workaround that forces a working environment for that locked account. That's why recovery looked unusual rather than a simple 'start it again'. The lockdown is a feature, not a fault, it just meant recovery needed the right technique. Once used, everything came back.
Is the site reboot-safe now? Yes. The dangerous instruction that tore everything down on a stop has been removed, so the trigger no longer exists. More importantly, we didn't just assume it was fixed, we ran a real restart, the exact action that caused the original outage, and watched every program stay up. Same trigger, no teardown, proven rather than hoped. That's the difference between a fix and a fix you can trust.
Technical detail
Context and objective
Following the pm2 clean-up in the server audit, we needed to confirm reboot-safety before any actual reboot: on boot, would the site's Node processes come back automatically? The processes are managed by pm2 under two systemd units, pm2-root.service and the site user's pm2-example.com_██████.service. The goal was to verify recovery in a controlled test rather than discover a broken boot path during a real restart.
What we found
Root's unit was enabled and clean (after the earlier pm2 save). The site user's unit, the one managing the live, traffic-serving apps, reported inactive (dead) despite being enabled. That combination is the tell: the live processes were being held up by a manually-started pm2 daemon from an earlier session, not by the systemd unit that is supposed to resurrect them on boot. Nothing was actually depending on the boot-time path.
$ systemctl status pm2-example.com_██████.service Loaded: loaded (/etc/systemd/system/pm2-example.com_██████.service; enabled) Active: inactive (dead)
1 · Save current process state before touching anything
pm2's resurrect path restores from a saved dump; a stale or empty dump means recovery brings back the wrong set or nothing. So we captured the accurate current list first:
$ sudo -u example.com_██████ pm2 save [PM2] Saving current process list... [PM2] Successfully saved in /var/www/vhosts/example.com/.pm2/dump.pm2
2 · Testing the unit, it fails to start cleanly
Restarting the unit to exercise the boot path surfaced a Type=forking + PIDFile= mismatch: the unit was configured to wait for a PID file that pm2 in this setup never writes, so systemd always concluded start-up had failed and looped.
$ sudo journalctl -xeu pm2-example.com_██████.service --no-pager | tail Can't open PID file /var/www/vhosts/example.com/.pm2/pm2.pid (yet?) after start: No such file or directory pm2-example.com_██████.service: Failed with result 'protocol'. # pm2 itself was alive throughout, the daemon answered: $ sudo -u example.com_██████ pm2 ping { msg: 'pong' }
On a real reboot there would be no pre-existing manual daemon to attach to, so this unit failing to start cleanly meant the live apps could genuinely have failed to come back on boot.
3 · The fix attempt that caused an outage
The obvious fix was to remove the broken PIDFile= directive and reload. But the unit also carried ExecStop=pm2 kill. A systemctl restart runs ExecStop before ExecStart, and pm2 kill tears down the entire pm2 daemon and every process under it, so the restart stopped all 6 live Node processes at once.
# removed the broken PIDFile= line, then: $ sudo systemctl restart pm2-example.com_██████.service # ExecStop=pm2 kill ran first -> daemon + all 6 processes killed -> site DOWN
4 · Recovery
Recovered by resurrecting from the saved dump. The wrinkle: the site user's login shell is /bin/false (a deliberate hardening choice), so a plain sudo -u <user> cannot spawn a working process environment. The restore had to be run through an explicitly forced shell:
$ sudo su -s /bin/bash example.com_██████ -c "pm2 resurrect" # all 6 processes restored from dump.pm2, site back up within minutes
5 · The real root cause, removed
The actual culprit was not PIDFile=, it was ExecStop=pm2 kill, which fires on any stop or restart, not only a genuine shutdown, converting every routine restart into a full teardown. pm2 kill is appropriate only for a real system halt, never for a service restart. We removed the directive entirely and reloaded the unit definition:
# unit before (dangerous): ExecStop=/usr/lib/node_modules/pm2/bin/pm2 kill # unit after: ExecStop line removed entirely, then: $ sudo systemctl daemon-reload
6 · Testing a real restart, zero interruption
We then re-ran the exact action that had caused the outage, and watched the processes across it:
$ sudo systemctl restart pm2-example.com_██████.service $ sudo -u example.com_██████ pm2 list # all 6 processes online, uptime unbroken through the restart, no ExecStop kill fired
7 · Outcome
Reboot-safety restored and proven: the unit now starts cleanly and resurrects the saved process list on boot, and the ExecStop=pm2 kill that caused the outage is gone, so a routine systemctl restart no longer stops the apps. The downtime was real, was caused by the first attempt, and was resolved within minutes; the verified end state is genuinely reboot-safe rather than merely worked around.
Key points
enabled+inactive (dead)= the boot path is not what is keeping the apps alive.Type=forkingwith aPIDFile=pm2 never writes = perpetualprotocolfailure and retry loop.ExecStop=pm2 killruns on everyrestart(ExecStop precedes ExecStart) → full daemon teardown.- Site user's
/bin/falseshell blockssudo -u; recover withsu -s /bin/bash … -c "pm2 resurrect". - Fix = delete
ExecStop,daemon-reload, then verify with a real restart.
How to do this yourself
$ systemctl status pm2-<user>.service # enabled but inactive(dead)? red flag $ sudo -u <user> pm2 save # snapshot before any change $ sudo journalctl -xeu pm2-<user>.service # look for PIDFile / 'protocol' errors # remove the ExecStop=pm2 kill line from the unit, then: $ sudo systemctl daemon-reload $ sudo systemctl restart pm2-<user>.service $ sudo -u <user> pm2 list # confirm uptime unbroken # if recovery is needed and the user's shell is /bin/false: $ sudo su -s /bin/bash <user> -c "pm2 resurrect"
Questions & answers
Why did restart kill everything? systemd runs the ExecStop step before ExecStart on a restart, and this unit's ExecStop was pm2 kill, which tears down the entire process manager rather than a single app. So a 'restart' first killed the whole daemon and every process under it, then tried to bring things back, an outage by design, not by accident. The stop half was doing far more than stopping one service. Fixing it meant changing what the stop step does, not the start step. Restart is only as safe as its ExecStop.
Why not just fix the PIDFile? The PIDFile mismatch was real, and it caused the failed-start loop where systemd kept thinking the service hadn't come up. But it wasn't what caused the outage, the outage came from ExecStop=pm2 kill tearing down the whole daemon. Fixing only the PIDFile would have quietened the loop while leaving the actual teardown in place, ready to fire again. So the PIDFile got corrected, but removing the destructive ExecStop was the fix that mattered. Treat the cause, not just the symptom.
Why su -s /bin/bash instead of sudo -u? The site's user account is configured with /bin/false as its shell, which is a deliberate lockdown that stops it opening an interactive session. sudo -u relies on that user being able to spawn a normal shell environment, so it can't do the job here. Forcing the shell with su -s /bin/bash overrides the /bin/false setting for that one command and gives the process a working environment. It's the specific technique that reaches a deliberately shell-less account. Without it, the recovery command simply wouldn't run.
How do you know it's reboot-safe now, not just fixed once? Because we re-ran the exact action that caused the original outage, a real systemctl restart, rather than reasoning that it should be fine. All six processes stayed up with unbroken uptime through it, meaning the teardown genuinely didn't happen this time. Same trigger, different outcome, observed live. That turns 'we think it's fixed' into 'we watched it not break'. Proving it under the real condition is the only way to know it's gone rather than merely worked around.
A real security weakness found and fixed
FixedWhy this one mattered more than its size suggests
Most of what the server scan turned up was housekeeping, permissions to tighten, stray files to clear. This one was different in kind: an actual way that malicious code could, under the wrong circumstances, be made to run. It was small, it was narrow, and it sat inside a tool only administrators can reach, but "small and narrow" is precisely the sort of hole that gets left alone until the day it doesn't, so it was worth understanding fully and closing properly rather than waving through.
What the scan flagged
Inside an admin-only diagnostic tool, the scanner found that some text, error messages pulled straight from the server's own internal logs, was being placed onto the page without being "cleaned" first. Cleaning, here, means neutralising the handful of characters that a web browser would otherwise treat as instructions rather than as ordinary text to display. The telling detail: a very similar piece of information sitting right next to it, on the very same page, was being cleaned correctly. So this wasn't a tool with no idea about safety, it was a single inconsistent spot, one place doing it right and one place, a few lines away, forgetting to.
1 · Reading the actual code, why it was a real risk, not a theoretical one
When text is dropped onto a page without cleaning, and that text ever happens to contain something shaped like a web instruction, the browser can be fooled into carrying it out instead of simply showing it. Here, two values were going onto the page unclean, the name of a running program and an error message, and both were coming from the server's internal error logs. Those logs are not something an ordinary visitor can freely write into, which is what kept the risk narrow. But narrow is not the same as none: internal logs routinely capture fragments of text that originally came from less trusted places, and once such a fragment lands on the page unclean, the browser has no way of knowing it was ever meant to be plain text.
2 · Why it mattered, who would actually be hit
The concrete danger runs like this. If something unusual, or something deliberately crafted by an attacker, ever found its way into one of those logged error messages, then the next time an administrator opened this diagnostic page, that crafted content could execute inside their browser, carrying all the access and privilege an administrator has. It is the classic shape of a quiet, high-value hole: it doesn't threaten ordinary visitors, it threatens the one person whose account can change everything. That is exactly the kind worth closing on sight, especially when the fix costs almost nothing.
3 · The fix, reusing what was already proven
We deliberately did not invent a new safety mechanism. The same page was already cleaning a neighbouring piece of data correctly, so we applied that identical, already-working method to the two spots that had been missed. After the change, the risky characters in a program name or an error message are converted into their harmless "just display this literally" form, so nothing arriving from a log can ever be mistaken by the browser for an instruction. Matching the existing approach, rather than bolting on a different one, also means the code stays consistent and easy for a future reader to understand.
4 · How we applied it carefully
We were deliberately cautious about the mechanics of the change, not just its content. The edit was made with a method that only alters the file if it finds the exact text it expects, character for character, and does nothing at all if anything fails to match, which removes any chance of a half-applied or corrupted edit silently breaking the tool. We also saved a backup copy of the file before touching it, so the previous version could be restored instantly if anything looked wrong afterwards.
5 · Outcome
Found and fixed the same day, and confirmed correct immediately. Because this is a file the browser loads and runs directly, rather than something running on the server, no restart of any service was needed; the corrected version simply takes effect the next time the page is opened. The inconsistency is gone: both spots now clean their text the same way the neighbouring code always did, and there is no longer a path by which logged text can reach the page as a live instruction.
Key points
- The one finding in the whole sweep that was a genuine way for malicious code to run, not just housekeeping.
- Two pieces of log text were placed on an admin-only page without being cleaned, while identical neighbouring text was cleaned.
- The risk is narrow (admin-only, log-sourced) but real: crafted text in a log could run in an administrator's browser.
- Fixed by reusing the cleaning method already working a few lines away, no new mechanism invented.
- Applied with an exact-match edit and a backup; it's a browser-loaded file, so no restart was needed.
How to find an injection weakness in your admin tools
- Look through admin tools for places that build page content out of text and drop it straight onto the page.
- Ask where each piece of that text comes from, anything from logs, errors or user input is untrusted.
- Flag every spot where untrusted text reaches the page without being cleaned first.
How to fix an injection weakness safely
- Make sure every untrusted value is cleaned, its special characters turned into harmless display form.
- If one spot already cleans its text correctly, reuse that exact method for the spots that don't.
- Make the change with a method that only edits on an exact match, and keep a backup, so a mistake can't spread.
Questions & answers
If only admins can see this page, why bother fixing it? Because an administrator account is the highest-value target on the whole system, it can change anything, so anything that can influence what an admin sees is worth locking down. On top of that, text that starts out from an untrusted source can quietly end up captured in logs, and those logs get viewed later in exactly that admin context. So the risk isn't hypothetical even on an admin-only page. The fix was tiny and the potential impact large, which is precisely the combination you close rather than accept. 'Only admins see it' lowers the odds, not the stakes.
What does "cleaning" the text actually do? Cleaning converts the handful of characters a browser treats as instructions, the ones that start and end tags, into their harmless 'display this literally' form. After that, a piece of text like an error message can only ever be shown on screen, never interpreted as code to run. It doesn't change what the message says; it changes whether the browser can be tricked by it. That single conversion is what turns a potential injection point into plain, safe output. It's the standard defence for any text that reaches a page.
Why reuse the existing method instead of writing a new one? The same page already cleaned a neighbouring value correctly, so a proven, working method was sitting right there. Reusing it keeps the code consistent, avoids introducing a new and untested approach, and means the next person reading the file sees one pattern rather than two. New code is where new bugs come from; reused, already-correct code isn't. It also made the change smaller and easier to review. When a safe method already exists a few lines away, copying it is the disciplined choice.
Did the site need to be restarted? No, the change was to a file the browser loads directly on the client side, not to a running server program. That means there's nothing to restart; the corrected version simply takes effect the next time someone opens the page. No downtime, no deploy step, no service bounce. It's one of the quieter kinds of fix. The moment the file is in place, the next page load is already using it.
Technical detail
Classification
This is the one finding in the sweep that is a genuine code-execution vector rather than hygiene: a stored-DOM-XSS sink on an admin-only surface. Low probability, high impact, narrow reachability, but admin-context execution if reached, which is why it's closed rather than risk-accepted.
The finding
In debug-tool.js, two values, p.proc and p.error, sourced from an admin-only /api/admin/recent-errors endpoint (real server process names and error-message strings), were interpolated into the DOM via innerHTML without escaping, while the adjacent line value on the same page was escaped. A hardened sink sitting inches from an unhardened one, an inconsistency, not a tool with no notion of escaping.
1 · The vulnerable code
html += `<div class="proc">${p.proc}</div>`;
if (p.error) { html += `<div class="l l-err">${p.error}</div>`; continue; }
Both interpolations write attacker-influenceable-in-principle strings straight into innerHTML. The data path is /api/admin/recent-errors → log-captured proc/error fields → template literal → innerHTML, with no encoding anywhere along it.
2 · Why it mattered
Stored-XSS-against-admin shape: if any crafted value is ever persisted into one of those log fields, it executes in the admin's browser on the next render of this page, in admin context. Reachability is limited (admin-only route, log-sourced data), but the sink is real and the remediation is trivial, so it is fixed rather than rationalised, the same disposition applied to every real finding in the sweep.
3 · The fix
Reused the escaping already present elsewhere on the page, a minimal HTML-entity encoder over the three structural characters, rather than introducing a new dependency or pattern:
const _esc = s => String(s).replace(/[<>&]/g, c => ({'<':'<','>':'>','&':'&'})[c]);
html += `<div class="proc">${_esc(p.proc)}</div>`;
if (p.error) { html += `<div class="l l-err">${_esc(p.error)}</div>`; continue; }
Encoding <, > and & is sufficient here because the values land in element text context, not attribute or URL context, so there is no need for quote or URL encoding, and the fix matches the neighbouring line handling exactly.
4 · How it was applied
Applied via an exact-match-only string replacement: it rewrites the target only if the expected substring matches verbatim and no-ops otherwise, eliminating any partial/corrupt-edit risk. A backup of the file was taken before the edit. As a static client-side asset, no service restart was required, the corrected file is effective on next page load.
5 · Outcome
Same-day remediation, verified immediately. p.proc and p.error are now entity-encoded before reaching innerHTML, matching the escaping already applied to the adjacent line data. The sink is closed and no log-sourced string can reach the DOM as markup.
Key points
- The only true code-execution finding in the sweep, everything else was hygiene.
p.proc/p.errorfrom/api/admin/recent-errorshitinnerHTMLunescaped; adjacentlinewas escaped, an inconsistency.- Shape: stored-DOM-XSS against admin, low reachability, admin-context impact.
- Fix reused the page's existing entity encoder; element-text context means
<>&encoding suffices. - Exact-match edit + backup; static asset, no restart.
How to do this yourself
# find unescaped interpolation into innerHTML $ grep -nE "innerHTML|\.html\(|\$\{[^}]+\}" debug-tool.js # confirm the source is untrusted/log-derived, then wrap every sink value: const _esc = s => String(s).replace(/[<>&]/g, c => ({'<':'<','>':'>','&':'&'})[c]); # static client asset -> effective on next load, no restart
Questions & answers
If it's admin-only, why fix it? Because admin context is the highest-value target there is, and log fields can capture attacker-influenced text that later gets viewed by an admin. The probability is low, but the impact if it landed would be high, and the fix was trivial. That specific combination, low odds, high stakes, cheap remedy, is one you close rather than accept. Waving it off because 'only admins see it' confuses low likelihood with low consequence. Cheap insurance against a serious outcome is worth taking.
Why only encode <>& and not quotes? Because the values are rendered in element-text context, between tags, not inside an attribute or a URL, where quotes would matter. In that context the three structural characters, less-than, greater-than and ampersand, are exactly the ones that can change how the browser reads the page, so encoding those is sufficient. It also matches how the adjacent, already-correct line handles its text, keeping the file consistent. Encoding more than needed wouldn't add safety here, just inconsistency. Right context, right characters.
Why not a framework/library escaper? Because the page already had a working encoder a few lines away, so pulling in a framework or library to do the same job would add a dependency for no gain. Reusing the existing function kept the change minimal, dependency-free, and consistent for whoever reads the code next. Fewer moving parts means fewer things to break or keep updated. The disciplined fix uses what's already proven in the file rather than importing something new. Small, local, consistent beats large and general here.
Did it need a deploy/restart? No, it's a static client-side file, so there's no server process to bounce and no deploy pipeline to run. The corrected version applies the next time the page is loaded in a browser. That makes it one of the lowest-risk changes possible: nothing to schedule, nothing to interrupt. Once the file is in place, the fix is simply live on the next request. No restart, no downtime.
Old suspended site being flagged as a risk
Confirmed safeWhat the scanner flagged, and why we didn't just accept it
Because the server test looks at every website hosted on the machine, not only the main one, it swept up some old, risky-looking code and flagged it. The easy thing would have been to treat that as one more problem to fix. But a flag is a starting point, not a verdict: the honest step is to find out what the code actually belongs to and whether it is genuinely live, rather than reacting to the label. On investigation, the code turned out not to belong to the main business at all. It was part of a completely separate website that the owner had deliberately suspended roughly a year earlier, switched off on purpose, not abandoned by accident.
1 · Confirming it really was suspended, two independent checks
We did not take "it looks suspended" on trust; we confirmed it two different ways so a single misleading signal couldn't fool us. First, we asked the hosting control panel directly what state it considered the site to be in, and it reported the site as suspended by the administrator, an authoritative answer from the system that actually manages the hosting, not a guess. Second, we tried to connect to the domain the way any visitor's browser would, and it did not respond at all, no page, no redirect, nothing. Two independent facts pointing the same way: the control panel says off, and the live internet agrees it is unreachable.
2 · The spammy content in search results, chased down, not hand-waved
A general web search for that old domain did turn up some unpleasant, spammy-looking content, which is exactly the sort of thing worth taking seriously rather than dismissing. So we chased it down properly. It proved to be leftover cached content, old copies held by search engines from before the site was switched off, not anything currently being served. Since the site itself no longer responds at all, there is nothing live behind those stale search entries; they are echoes of a site that is already gone, and they fade from search results over time on their own.
3 · A real decision, should a dead site even be in the scan?
This raised a genuine judgement call worth being open about. The tempting move was to simply tell the scanner to ignore that site entirely and be done with it. We actually tried that, and then deliberately reversed it. The reasoning matters: this is a whole-server security test, and the whole point of that is that it sees everything on the machine. Quietly dropping a site because it happens to look unrelated is exactly the kind of silent assumption that lets a real problem hide. If the same test were run for anyone else, it shouldn't secretly skip sites based on a guess about what's "in scope". So the rule we settled on is the honest one: every site stays visible in the scan, but a suspended site's findings are clearly labelled as suspended and set aside from the live-risk count, visible, accounted for, but not mixed in with problems that can actually affect someone today. Nothing is hidden; it is just correctly categorised.
4 · The fiddly part, getting the right command to ask about suspension
Teaching the scanner to recognise a suspended site meant asking the hosting control panel, in code, which sites are suspended, and that was not smooth first time, which is worth recording honestly given this report is partly about exactly these generation issues. The obvious-looking command to list suspended sites returned nothing at all, not because there were none, but because that particular version of the control panel's tool didn't accept that phrasing. Rather than guess at variations, we asked the tool itself for its real list of accepted options, found the correct form, and then had it report each site's status one by one. Only once we could see the real, confirmed status for every site did we build that exact query into the scanner. It was a small detour, but a real one, and doing it by checking rather than guessing is what stopped a wrong assumption being baked into the tool.
5 · Fixing the real problem, the scanner's blind spot
The genuine lesson was never about that one old site; it was that the scanner had no concept of a site being suspended at all. It treated dead code on a switched-off site exactly as seriously as live code on a running one, which is misleading twice over, it inflates the problem count with issues that cannot affect anyone, and in doing so it buries the findings that actually matter under noise. So we built in the check we'd just worked out: at the start of every run, the scanner now asks the control panel which sites are officially suspended and automatically sets those aside, replacing a pile of per-file flags with a single clear line noting the site was skipped and why.
6 · Outcome
This particular false alarm is resolved, and, more importantly, it can't recur. The scanner now recognises a suspended site for what it is and sets it aside automatically, for this site and for any site suspended in future, while still keeping it visible in the report rather than silently dropping it. The result is a report that's both cleaner and more honest: the live-risk count reflects real, reachable problems, and dead code on unreachable sites is accounted for separately instead of masquerading as urgent.
Key points
- The server test checks every site on the machine, so it flagged old code on a completely separate website.
- That site had been deliberately switched off by its owner about a year earlier, not live, not reachable.
- We confirmed it two independent ways: the hosting control panel says suspended, and the domain doesn't respond at all.
- Spammy content in search results was old leftover cache from before shutdown, not anything live.
- We kept the site visible in the scan but set aside from the live-risk count, never silently dropped, and taught the scanner to do this automatically.
How to check whether a flagged site is actually live
- When a scan flags a site, first find out whether it's even live before treating it as a real risk.
- Ask your hosting control panel directly what state it considers the site to be in, that's the source of truth.
- Also load the domain as a visitor would; if it doesn't respond at all, it isn't serving anything.
How to handle an old suspended site safely
- If old spammy content shows in search, check whether it's stale cache from before shutdown rather than live pages.
- Don't silently drop the site, keep it visible but set it aside, so dead code stops inflating your risk view.
Questions & answers
Why not just fix the risky-looking code? Because that code lives on a website that was deliberately switched off and can't be reached by anyone on the internet. Fixing code on a dead, unreachable site achieves nothing real, there's no live path for anyone to exploit it. The actual problem wasn't the code; it was that the scanner was counting a suspended site as if it were live and adding to the risk total. So the fix belonged in the scanner's understanding of site status, not in the dead code. Treating the symptom would have wasted effort on something already harmless.
Why not just tell the scanner to ignore that site completely? Because a whole-server audit that quietly skips sites is exactly how a genuine problem slips through unnoticed one day. If a site can vanish from the scan by assumption, so can a real risk hiding on it. So instead of dropping the suspended site, it stays visible in the report and is simply set aside from the live-risk count, clearly labelled as suspended. You get the honesty of seeing every site, without a dead one inflating the numbers. Excluding from the tally isn't the same as hiding from view.
Should the spammy search results worry you? No, those results are old copies held by search engines from before the site was shut down, not anything being served now. The live site returns nothing, so there's no active content behind those listings for anyone to reach. Cached entries like these naturally drop out of search over time as engines re-crawl and find the pages gone. There's nothing to clean up at the source because the source is already off. It looks alarming in a search box but it's stale, not live.
Why check the control panel and try loading the site? Because two independent checks are stronger than either one alone. The hosting control panel confirms the site was suspended deliberately, that's the authoritative record of intent. Trying to actually load the domain confirms it really is unreachable in the real world, not just marked that way in a database. One check tells you what should be true; the other tells you what is true. Agreeing on both is how you rule out a stale or mistaken status.
Technical detail
The finding
The whole-server sweep flagged new Function() and unescaped innerHTML inside a suspended domain's bundled JavaScript (third-party theme/plugin code). Flagged at the same severity as live-site findings, which is misleading for a site that is switched off.
1 · Independent verification of suspension
Confirmed two ways rather than trusting the appearance:
$ sudo plesk bin site --info <domain> | grep -i status Domain status: suspended by the administrator # and from the outside, as a browser would see it: $ curl -s -o /dev/null -w '%{http_code}' -m 5 http://<domain>/ 000 # no connection established at all, not a 200, not a redirect
Control-panel state (authoritative for intent) and live reachability (000) agree: the domain is off.
2 · Triaging the spam content in SERPs
A general web search surfaced gambling/spam content for the domain. Traced to stale search-engine cache predating the suspension, nothing is served live, since the origin returns 000. No live exposure; cached entries age out over time.
3 · Scope decision, exclude vs. label
Initially added an outright exclusion for the domain, then reverted it. Rationale: this is a whole-server audit; silently dropping a vhost by assumption is precisely the failure mode that lets real issues hide, and would behave wrongly for any other operator running the same tool. Settled policy: every vhost stays in scope and visible; suspended domains are excluded only from the live-risk code checks and surfaced as an explicit INFO line, not silently skipped.
4 · Getting the Plesk query right, the generation detour
The first attempt at a "list suspended" query returned empty on this Plesk version, a flag-syntax mismatch, not an empty result set:
$ sudo plesk bin site --list -status suspended 2>/dev/null # empty, wrong flag form for this version, not "none suspended" # rather than guess variants, ask the tool for its real interface: $ sudo plesk bin site --help 2>&1 | head # correct form is --info per domain; loop it to get authoritative status: $ for d in $(sudo plesk bin site --list); do \ echo -n "$d: "; sudo plesk bin site --info "$d" | grep "Domain status"; done <domain>: Domain status: suspended by the administrator ...all others: Domain status: OK
Only the verified-correct query was then wired into the scanner, checked, not guessed.
5 · The real fix, suspension-awareness in the scanner
Root cause was a scanner blind spot: it treated every vhost under /var/www/vhosts/ identically, with no notion of suspension, inflating the count with unreachable code and burying live findings. Added a startup step that builds the suspended-domain set from Plesk and excludes those domains from all code-scanning checks, emitting one INFO line instead of per-file findings.
6 · Outcome
False alarm resolved and structurally prevented: the scanner queries real suspension status each run and excludes suspended domains from code scans, replacing their per-file findings with one clear exclusion note while keeping them visible in the report. The reported live-risk count now reflects reachable risk only.
Key points
- A whole-server scan sweeps in every vhost, including a domain suspended ~a year earlier, flagged code was third-party, on a dead site.
- Suspension confirmed two ways: Plesk
Domain status: suspended+curlreturning000(unreachable). - Spam in search results was stale pre-suspension cache, not live content.
- Scope policy: keep suspended vhosts visible but excluded from the live-risk count, never silently dropped.
- Correct Plesk query was found via
--help, not guessed, then wired into a startup suspension check.
How to do this yourself
# confirm a flagged domain's real state before treating it as live risk $ sudo plesk bin site --info <domain> | grep -i status # authoritative $ curl -s -o /dev/null -w '%{http_code}' -m 5 http://<domain>/ # 000 = unreachable # build a suspended set at startup and skip those trees (label, don't drop): $ for d in $(sudo plesk bin site --list); do \ sudo plesk bin site --info "$d" | grep -q "suspended" && echo "skip: $d"; done
Questions & answers
Why not just fix the flagged code? Because it sits on a deliberately-suspended, unreachable site, and it's third-party bundle code on top of that, not something you'd hand-edit anyway. Fixing dead code on a dead site is wasted effort; nothing can reach it to exploit it. The real issue was upstream: the scanner was treating a suspended site as live and letting its findings inflate the risk count. So the correction went into how the scanner classifies site status. The code itself never needed touching.
Why not just exclude the site from the scan entirely? Because a whole-server audit that silently drops vhosts by assumption is precisely how real issues end up hidden. The safer design keeps suspended domains visible in the output and labels them, excluding them only from the live-risk tally rather than from the scan itself. That way nothing disappears quietly, and a dead site stops distorting the numbers at the same time. Visibility and accurate counting aren't in conflict, you can have both. Dropping a site outright trades one problem for a worse, quieter one.
Why loop --info instead of a one-shot status filter? Because the tidy one-shot form, listing domains filtered by suspended status, returned empty on this particular Plesk version. That was a syntax mismatch, not an absence of data: the command interface differed from what was expected. Checking --help revealed the real interface, and querying each domain individually with --info proved to be the authoritative, version-safe way to get the true status. It's more verbose but it doesn't silently return nothing on a version quirk. Correct-but-wordy beats neat-but-wrong.
Is the spam in search results a problem? No, it's stale cache left over from before the site was suspended, not anything live. The origin now returns a 000 response, meaning nothing is actually served, so those search entries point at pages that no longer exist. Cached listings like these age out of search on their own as engines re-crawl and find nothing there. There's no live content to secure or remove. It's a leftover impression, not an active exposure.
Unused chat app found and removed
Confirmed safeWhat we found, and why a running program you're not using is a risk
The server test turned up something nobody was expecting: a large team-chat application, around 605 megabytes of software, still actively running in the background on one of the hosted sites. The oddity is that the actual live website for that domain is just two simple static pages that have nothing whatsoever to do with a chat app. So here was a substantial, complex piece of software quietly running and consuming memory, that the live site did not need and nobody was using, almost certainly the leftovers of something set up once and long forgotten.
That matters for security, not just tidiness, and it's worth being clear about why. Every program left running is part of what an attacker could try to reach, the more software you have running, the more separate doors there are into the machine, and a large chat application you're not even using is a great deal of extra door for zero benefit. Worse, software that sits unused tends to go unpatched, because nobody is watching it or updating it; over months and years that quietly turns it into one of the most attractive ways in, since it accumulates known weaknesses that no-one is fixing. The safest possible state for something you genuinely don't use is switched off, and then gone entirely.
1 · Confirming it genuinely wasn't part of the live site
Before removing anything, we made sure the chat app really was disconnected from the live website and not quietly serving some part of it behind the scenes. We checked how the site's traffic is actually routed, where a visitor's request goes when they load the page, and confirmed it goes only to those two simple static pages. Nothing was ever being handed off to the chat application. It was running, but running in isolation, wired into nothing that a real visitor ever touches. That confirmation is what made it safe to remove: we weren't guessing it was unused, we had checked the actual traffic path and seen that it was.
2 · Switching it off, safely, and for good
We switched the chat application off completely, and, just as importantly, made sure it could not automatically turn itself back on later, including after a server reboot. This is a distinction that's easy to get wrong: stopping something without also stopping it from restarting is only half a fix, because a program set to auto-start would simply come back to life the next time the machine restarted, quietly undoing the work. So we did both, stopped it now, and removed its permission to start itself in future, then confirmed directly that it was genuinely marked as "will not start on its own" rather than assuming the command had taken.
3 · Backing it up before deleting a single file
Even though nothing was using it, we did not delete 605 megabytes of software on the assumption that it was worthless, that is exactly the kind of assumption that occasionally turns out to be expensively wrong. Instead we made a full, compressed backup of the entire installation first, then verified that backup genuinely contained everything, confirming it held the complete set of files, correctly and readably, rather than trusting that the backup command had simply worked. Only once we could see for ourselves that the archive was complete and sound did we delete the original folder. The backup was placed in a private location that is not reachable from the public internet, so it isn't itself an exposure. If it ever turns out something in there was wanted after all, it can be restored.
4 · Confirming the real site was never affected
Throughout the stop, the backup, and the deletion, we repeatedly confirmed the one thing that actually matters: that the live website, those two simple pages, stayed completely intact and reachable. It did, at every step, not just once at the end. The removal touched only the unused chat software, and nothing the public ever sees was altered.
5 · Clearing up the leftover reference the removal left behind
Removing the files was not quite the whole job, and this is the sort of detail that's easy to miss. The system that had been keeping the chat app running still held a leftover reference to it, a pointer to a service that no longer existed anywhere on disk. Left in place, that kind of dangling reference is untidy at best and, at worst, causes confusing errors later when something tries to act on a service that's gone. So we cleaned up that stale reference too, so the system's own record matches reality: the service isn't just stopped and deleted, it's no longer referred to at all.
6 · A real bug we caught in our own scanner along the way
This is the honest, on-theme part. While doing this, we found that our own scanner's check for "is this program actually switched off?" had a genuine flaw. To decide which program was which, it had been reading a field that can contain several different path-like pieces of information jumbled together, and because of that, it briefly matched a completely unrelated site instead of the chat application we were actually checking. In plain terms, the tool could have told us the wrong thing about whether something was really off, a serious kind of error in a tool whose whole job is to report accurately. We traced exactly why it happened, and fixed it to read instead from a single, unambiguous field that always points at the one right answer, so the "is it off?" check cannot be misled that way again.
7 · Outcome
The unused chat application is stopped, prevented from restarting, backed up, verified, deleted, and its leftover system reference cleared, 605 megabytes of unnecessary running software and attack surface gone, with the live site untouched throughout. And the scanner itself is more trustworthy than before, because a real flaw in how it judged "switched off or not" was found and corrected in the process.
Key points
- A large team-chat app (about 605MB) was running in the background on a domain whose real site is just two static pages.
- Unused running software is extra attack surface that tends to go unpatched, a growing liability for zero benefit.
- We confirmed it wasn't part of the live site, then switched it off and stopped it restarting, and checked that took.
- We made a verified backup in a non-public location before deleting, and confirmed the live pages stayed intact throughout.
- We cleared the leftover system reference the removal left behind, and fixed a real flaw in our scanner's "is it off?" check.
How to audit what's running on your server
- Look at what's actually running in the background, and ask whether the live site truly needs each thing.
- For anything unexpected, confirm it isn't quietly serving part of your site before you touch it.
How to disable an unused service safely
- Switch off what isn't needed, and also stop it auto-starting again.
- Check the change held and that your live site still works without it.
How to remove a service completely
- Make a full backup, keep it somewhere the public can't reach, and check it's genuinely complete.
- After deleting, clear any leftover system reference to the thing you removed.
- Confirm your site and the rest of the server are unaffected.
Questions & answers
If it wasn't doing any harm, why remove it? Because unused software is still a way in, even while it sits idle. It can be exploited if a flaw is found, or restarted by accident or attacker, and because nobody's actually using it, nobody keeps it patched, so its risk quietly grows over time. Removing it takes away that whole class of risk in exchange for something that was providing no benefit. Idle isn't the same as safe. The safest component is the one that isn't installed at all.
Why not just switch it off and leave the files there? Because switched-off software is still installed code that can be restarted or exploited if a vulnerability turns up in it. 'Stopped' only changes its current state, not its presence on disk. The genuinely safe state is backed-up and removed, so there's nothing left to restart and nothing left to find a flaw in. Keeping a checked backup first means nothing is lost if it's ever wanted again. Off is temporary; gone is permanent.
Why bother backing up something you're deleting? Because deleting on the assumption that something is worthless is exactly how things get lost for good. A checked backup, kept out of public reach, means that if anyone ever turns out to need something from it, it can be brought back. It costs almost nothing to keep and removes the only real downside of deletion, irreversibility. So the removal is safe precisely because it isn't actually irreversible. Delete confidently only when you've made undo possible.
Was the live website ever at risk during this? No, the two real, live pages were confirmed intact at every step of the work. The removal only ever touched the unused chat software, which nothing depended on. Checking the live pages before, during and after meant there was no moment where a visitor would have seen anything wrong. The change was isolated to the thing being removed. Careful scoping is why a cleanup didn't become an outage.
Technical detail
The finding
The server sweep found a Mattermost installation, 605MB, running as an active systemd service on a domain whose live site is two static HTML files. Unused, unrelated to live traffic, and a large standing attack surface: an unpatched, unused service is a classic lateral-movement target once any CVE lands against its version.
1 · Confirm it's outside the live traffic path
Checked the vhost's nginx config before touching anything: location / served the static docroot directly; no proxy_pass routed to the Mattermost service port. Running, but wired into nothing visitor-facing, verified, not assumed.
2 · Stop and disable
$ sudo systemctl stop mattermost $ sudo systemctl disable mattermost # stop AND disable, a stop alone comes back on reboot $ systemctl is-enabled mattermost disabled # confirmed, not assumed
3 · Archive, verify, then delete
Full compressed backup, integrity-verified by listing its contents before removing the original, and parked in a non-web-reachable path:
$ sudo tar -czf /var/www/vhosts/<host>/tmp/recycle-mm-backup-$(date +%Y%m%d).tar.gz \
-C /var/www/vhosts/<domain> mattermost
# 605MB -> 446MB compressed
$ sudo tar -tzf /var/www/vhosts/<host>/tmp/recycle-mm-backup-*.tar.gz | wc -l
4276 # full file listing readable, archive is sound
# tmp/ returns 404 externally, the archive is not itself an exposure
$ sudo rm -rf /var/www/vhosts/<domain>/mattermost
$ ls /var/www/vhosts/<domain>/httpdocs/
# the two real static HTML files, untouched
4 · Live-site integrity
The two static pages were confirmed present and served before and after removal. The deletion touched only the Mattermost tree.
5 · Clean up the orphaned unit reference
After deleting the files, systemd still carried a dangling instance-unit reference (a mattermost@<id> template instance) pointing at a service that no longer existed on disk. Left in place it produces confusing failures on future daemon-reload/boot. Removed the stale unit reference and reloaded so the manager's state matches reality:
$ sudo systemctl reset-failed mattermost@<id> 2>/dev/null
# remove the leftover unit/symlink, then:
$ sudo systemctl daemon-reload
6 · Scanner bug found and fixed
The stopped-service check matched services by scanning both ExecStart and WorkingDirectory for a vhost path. ExecStart's argument vector can contain several path-like tokens (binary path, flags, args), so the match landed on an unrelated vhost instead of the Mattermost unit, i.e. the "is it stopped?" check could report against the wrong service. Fixed to key off WorkingDirectory only, a single, unambiguous field.
# before: ambiguous, ExecStart can hold multiple path-like tokens match = vhost_path in unit['ExecStart'] or vhost_path in unit['WorkingDirectory']
# after: WorkingDirectory is a single canonical path
match = vhost_path in unit.get('WorkingDirectory','')
7 · Outcome
Mattermost stopped, disabled, archived (446MB, 4,276 files verified, non-web-reachable), deleted, and its orphaned unit reference cleared; live static site intact throughout. Scanner's stopped-service detection corrected to a single unambiguous field, removing a real false-match path.
Key points
- 605MB Mattermost running as an active
systemdservice on a static-HTML domain, unused standing attack surface. - Confirmed outside the live path via nginx config (no
proxy_passto it) before any change. stop+disable+is-enabledcheck, a stop alone would restart on reboot.- Archived (605→446MB) and integrity-verified (
tar -tzf … | wc -l→ 4276) in a 404-only path beforerm -rf. - Cleared the orphaned
mattermost@<id>unit reference; fixed scanner stopped-check fromExecStart(multi-token) toWorkingDirectory.
How to do this yourself
$ systemctl list-units --type=service --state=running # what's actually running # confirm it's not in the live path (grep the vhost's nginx/apache config for its port) $ sudo systemctl stop <svc> && sudo systemctl disable <svc> $ systemctl is-enabled <svc> # verify: disabled $ sudo tar -czf backup.tar.gz -C /path parent_dir && sudo tar -tzf backup.tar.gz | wc -l $ sudo rm -rf /path/parent_dir/unused_app $ sudo systemctl daemon-reload # clear any orphaned unit reference
Questions & answers
Why remove it instead of leaving it stopped? Because a stopped service is still installed code that can be restarted or exploited the moment a flaw is found in it. Unused software you'll never get around to patching is pure liability, all downside, no upside. Archiving it first and then removing it reaches the genuinely safe end state: nothing left running, nothing left to attack, nothing lost. Stopped is a pause; removed is a resolution. The idle component you forget to patch is the one that bites you.
Why disable as well as stop? Because stop only ends the current run, without disable, the unit would start itself again on the next reboot, quietly undoing the work. Stopping and disabling together means it's off now and stays off after a restart. Running is-enabled afterwards confirms the change actually took, rather than assuming it did. Both halves are needed for the same reason 'enabled' and 'running' are different states. Skip disable and the problem reappears the next time the box reboots.
Why clean up the leftover unit reference? Because a dangling reference, an instance still pointing at files that have been deleted, produces confusing failures on later reloads or boots, where systemd tries to act on something that's no longer there. Clearing it keeps systemd's idea of the system matching what's actually on disk. That saves a future headache chasing an error for a service that doesn't exist any more. Tidy state now prevents mysterious noise later. Leaving it would be leaving a small trap for future-you.
What was wrong with the scanner check? It identified services by searching the ExecStart line, whose argument list contains several path-like tokens, so it could match against the wrong vhost by picking up an incidental path. That made its results ambiguous. WorkingDirectory, by contrast, is a single canonical path with no competing tokens, so matching on that is unambiguous. Switching the check to use it removed the misidentification. Match on the one definite field, not the line that happens to contain several.
Unnecessary admin-level programs on a second site
Confirmed safeWhat we found, and why "full-power and unused" is the worst combination
On a second website belonging to the same owner, the server test found two background programs running with full administrator-level power, the highest level of access on the machine. That level of power is meant to be held only by things that genuinely need it, and held as briefly as possible, because of what it can reach. A program with full power isn't limited to its own website's files; it can, in principle, touch every site and every account on the whole server. So if one of these programs ever had a flaw, or were ever compromised, the damage wouldn't be contained to that one site, it could spread sideways to every other site sharing the machine. That "reaches everything" quality is exactly why full-power programs are the ones you most want to be sure are both necessary and watched.
And these two were neither. They were running with the maximum possible power, and, as we went on to confirm, doing nothing the live website actually needed. Full power plus no purpose is the worst of both worlds: all of the risk, none of the benefit. It's the kind of thing that tends to accumulate quietly over time: a program set up for some earlier purpose, never switched off when that purpose passed, still running months or years later with power nobody remembers granting it.
1 · Tracing how the site is really served, before assuming anything
The critical question was whether these two programs were secretly doing something important, because switching off something the site quietly depends on would break it. So rather than assume, we traced the actual path a visitor's request takes, step by step, from the moment it arrives to the moment a page is sent back. We confirmed two things in turn. First, where the site's files actually live and are served from: its content is delivered directly from a specific folder of finished files by the ordinary web-server software, not generated on the fly by either of the two programs. Second, that the front door of the server, the part that first receives every incoming request, hands those requests straight to that ordinary file-serving software, and not to either of the two full-power programs. At no point in that real path is either program involved.
2 · The thorough version of that check, not just the obvious route
It would have been easy to check only the main rule, "where do ordinary page requests go?", see that it doesn't touch these programs, and stop there. But that could miss a subtler arrangement, where most of a site is served normally while certain specific addresses (say, anything under a particular sub-path) are quietly routed to a background program. So we deliberately looked for that too: we searched the server's routing rules specifically for any instruction, on any path, that would send traffic to either of these two programs. There were none. Not the main route, and not any side route either, nothing anywhere pointed at them.
3 · Why the absence of a rule was strong evidence, not just an empty result
This is the part that turned "we found nothing" into real confidence rather than a shrug. On this same owner's main site, background programs genuinely are used, and there, the server's routing rules contain exactly the kind of instruction that sends specific traffic to a background program, because that site really depends on one. So we already knew precisely what a "this program is genuinely needed" rule looks like when it exists. Finding that this second site had no such rule anywhere wasn't an ambiguous blank; it was the confident absence of the one specific thing that would have to be present if these programs mattered. We weren't reading nothing into silence, we were noting that a signature we know well, and would expect if they were needed, simply wasn't there.
4 · Switching them off, stopped, not deleted, and made to stick
We switched both programs off. We deliberately chose to stop them rather than delete them outright, stopping removes the running risk immediately while keeping the option to bring them back easily if some need for them ever emerges, which is the more cautious, reversible choice for something that isn't actively harming anything the moment it's stopped. We then made sure that "off" state would survive a server restart, so they wouldn't quietly start themselves up again after the next reboot and undo the fix. Saving that state is what turns a temporary stop into a lasting one.
5 · Reconciling the saved state carefully
There was a subtle point worth being honest about here. The record of "which programs should be running" had last been saved at a moment when these two were still running, so simply trusting the old saved record would have brought them straight back on the next restart. Switching them off is only genuinely permanent once that saved record is re-written to reflect the new reality. So after stopping them, we saved the updated state deliberately, making the new "these two stay off" the version the server will restore from, rather than assuming the stop alone was enough. It's a small step that's easy to skip, and skipping it would have quietly undone the whole fix at the next reboot.
6 · Confirming the real site kept working
With both programs stopped, we checked the actual live website and confirmed it kept loading and working perfectly, exactly as the traffic-path trace predicted it would, since the site never relied on those programs in the first place. That final check is what closed the loop: not just "we think it's safe" but "we stopped them and watched the site carry on unaffected."
7 · Outcome
Both unnecessary, over-powered programs are stopped and kept stopped across restarts, with zero impact on the actual working website. A pair of full-power programs that could have reached every site on the server, for no benefit to anyone, are no longer running, which meaningfully shrinks what an attacker could ever take advantage of on that machine.
Key points
- Two background programs ran with full administrator power on a second site, power that can reach every site on the server.
- Full power plus no purpose is the worst combination: all of the risk, none of the benefit.
- We traced how the site is really served and checked both the main route and any side route, nothing pointed at these programs.
- The absence was strong evidence because the owner's main site does use such a route, so we knew exactly what to look for.
- We stopped (not deleted) both, saved the change so it survives a restart, and confirmed the live site kept working.
How to find over-privileged programs on your server
- List what's running in the background and, crucially, what power level each thing runs with.
- For anything running with full administrator power, ask hard whether it truly needs it.
How to safely remove an unnecessary admin-level program
- Trace how your site is actually served, checking both the main route and any per-path routing rules.
- Switch off what isn't needed; prefer stopping (reversible) over deleting, and save the state.
- Check the live site still works with the program off, rather than assuming it will.
Questions & answers
Why does it matter that they had "full power"? Because a full-power program isn't confined to its own site, it can reach every site on the server. That means a single flaw or compromise in one such program could spread to everything on the machine, not just one corner of it. So an unused program running with that much privilege is a large blast radius for zero benefit. That combination is why full-power programs you don't need are worth removing on sight. Least privilege isn't bureaucracy; it's containing the damage when something goes wrong.
How did you know the site didn't need them? Because we traced the actual path a visitor's request takes through the server and read the routing rules directly, both the main route and any side routes. Nothing in that path sent traffic to these programs, so removing them couldn't affect what visitors see. We also know what a genuinely 'needed' rule looks like, because the owner's main site has one to compare against. That's evidence, not assumption. Trace the real traffic, then you can remove with confidence.
Why stop them rather than delete them? Stopping removes the risk straight away and can be undone easily if a need ever appears. It's the more cautious choice for something that isn't harming anything the moment it's off.
Could they come back on after a restart? Not now, we saved the new "off" state so the server restores that, rather than the older record that still had them running.
Technical detail
The finding
Two Node processes, seo-tools (port 3000) and seo-subscriptions (port 3001), running as root under root's pm2 on a separate domain. Root-owned processes are the high-value case: a compromise isn't vhost-scoped, it's whole-box, reaching every site and user on the machine. Unused and root is the worst quadrant, likely legacy processes that outlived their purpose.
1 · Establish the real serve path
Confirmed the docroot and the front-door handoff from the vhost's nginx config:
$ sudo grep -E "root|proxy_pass|location" \ /var/www/vhosts/system/<domain>/conf/nginx.conf root "/var/www/vhosts/<domain>/site-audit/public"; location / { proxy_pass "https://127.0.0.1:7081"; ... }
location / proxies to Apache on :7081 (Plesk's standard nginx→Apache handoff); Apache serves the static docroot directly. Node on 3000/3001 is not in that path.
2 · Rule out path-scoped proxies too
A location / check alone can miss a path-scoped upstream (e.g. only /api/… proxied to Node). Explicitly searched for any route to either port:
$ sudo grep -B2 -A3 "127.0.0.1:3000\|127.0.0.1:3001" \ /var/www/vhosts/system/<domain>/conf/nginx.conf # empty, no proxy_pass, no upstream, on any path, to either port
3 · Why the empty result was positive evidence
The owner's main site legitimately proxies specific paths to a Node backend (a known proxyTo<port> pattern is present in its config). So the signature of "Node is genuinely wired in" was known and expected-if-needed. Its complete absence here, no location, no upstream, no proxy_pass to 3000/3001 anywhere, is a confident negative, not an ambiguous blank.
4 · Stop (not delete) and persist
$ sudo pm2 stop seo-tools
$ sudo pm2 stop seo-subscriptions
$ sudo pm2 save # stop is reversible; save makes "off" survive reboot
5 · Reconcile the saved dump
Root's pm2 dump had last been written while these two were running, so a bare reboot would resurrect them. The stop is durable only once pm2 save re-writes the dump to the new state, which is why the save matters as much as the stop. Skipping it would silently undo the fix on next boot.
6 · Verify live-site integrity
Fetched the live site with both processes stopped, full content served correctly, matching the routing trace exactly. Node was never in the request path.
7 · Outcome
Both root-owned Node processes stopped and persisted-off across reboots, zero live-site impact. A whole-box-reachable attack surface (two root processes serving nothing) is removed.
Key points
- Two Node processes running as root on a second domain, a compromise there is whole-server, not single-site.
- Live path is nginx → Apache (
:7081) → static docroot; Node on:3000/:3001isn't in it. - Checked both the
location /route and any path-scoped proxy, nothing routes to those ports. - Empty result was positive evidence: the main site's config shows the
proxyTo<port>signature that would be present if Node were needed here, and it isn't. pm2 stop(reversible) +pm2 saveto reconcile the stale dump; live site verified serving.
How to do this yourself
$ sudo pm2 list # what's running, and as which user # rule out BOTH a top-level and a path-scoped route before stopping: $ sudo grep -rnE "proxy_pass|127.0.0.1:<port>" /var/www/vhosts/system/<domain>/conf/ $ sudo pm2 stop <app> && sudo pm2 save # stop, then persist the new state $ curl -sI https://<domain>/ | head -1 # confirm site still serves
Questions & answers
Why is root-owned worse than an ordinary process? A root process can touch every site and account on the box, so a flaw in it is a whole-server compromise, not one site. Least-privilege says nothing should run as root unless it truly must.
How do you know the site doesn't need them? The live path is nginx → Apache → static files, and no rule on any path routes to the Node ports. We know what a "Node is needed" rule looks like, it's present on the main site, and it's absent here. Evidence, not assumption.
Why check for path-scoped proxies as well as the main route? A site can serve most content statically while quietly proxying one sub-path (e.g. /api) to a backend. Checking only location / would miss that, so we grepped for any route to the ports.
Why did save matter as much as stop? pm2 restores its last saved dump on reboot. That dump was written while these ran, so without a fresh save they'd come back. save makes "off" the state that persists.
New feature, planned, not yet built
Confirmed safeWhat the feature was meant to be
Not everything in this write-up is a fix, this part is about a feature that was planned but deliberately not built this time, and it's included for the sake of a complete, honest record rather than to claim credit for work that didn't happen. The idea was a public tool where someone types in something awkward or rude that was said to them, picks a tone, funny, blunt, classy, and so on, and gets back a set of AI-generated comebacks. The results would be stored in a way that search engines and AI assistants could later find and reference, so the feature would also bring in search visibility over time, not just serve the one person using it in the moment.
1 · Doing the homework first, looking at what already exists
Before writing a line of it, we looked at four existing tools that already do something similar. The point of that wasn't to copy them; it was to learn from what they get right and, more importantly, to spot the ways they go wrong so we could design against those from the very start rather than discovering them the hard way after launch. Studying the failure modes of comparable tools is cheaper and safer than making the same mistakes yourself in public.
2 · The genuinely concerning thing we found
One of those existing tools, during our testing, did something that matters a great deal for a feature like this: when a real person's name appeared in the test input, it generated a fabricated, made-up personal insult aimed at that named individual. That is a serious failure mode. There's a clear and important line between helping someone respond wittily to a remark that was actually made to them, and a tool that will invent damaging personal claims about a real, named person on request, the latter is a harassment risk, and it's the kind of thing that turns a light-hearted feature into something genuinely harmful.
3 · The design rule that came out of it
So this shaped a firm design rule for any version we would build, decided up front rather than bolted on later: the tool must always respond to what was actually said, the remark itself, and must never invent personal claims or attacks about a real, named individual. Respond to the words, never target the person. Building that boundary in from the first line of the design, rather than trying to patch it in after something goes wrong, is the difference between a feature that's fun and one that's a liability. Finding this in someone else's tool, before building our own, is exactly what the homework was for.
4 · How it would be controlled and paid for
We also worked out how the feature would be gated, so it isn't relevant to note it wasn't a loose end left unconsidered. It would use the same credit-based system already used elsewhere on the site, the existing mechanism that controls access to the other tools, rather than inventing a new one. Reusing the established pattern keeps the feature consistent with everything else and avoids a whole new set of things that could go wrong.
5 · Why it wasn't built, and why that's the right call
In the end, this feature was deliberately set aside this time in favour of the security work that makes up the rest of this write-up. That was a conscious priority decision, not something forgotten: with real, concrete security issues found on the server, fixing those came first, ahead of building a new public-facing feature. It's an honest reflection of where the effort actually went, and putting known security problems ahead of a nice-to-have new feature is the right order to do things in.
6 · Status
Planned, researched, and scoped, including its most important safety boundary, but not built this session. It's recorded here so the write-up reflects everything that was actually worked on, including the things that were thought through and then, sensibly, deferred.
Key points
- A planned public feature, type in a rude remark, pick a tone, get AI comebacks, recorded here as planned, not built.
- We studied four existing tools first, to learn their mistakes rather than repeat them.
- One of them invented a personal insult about a real named person, a serious harassment failure mode.
- That set a firm rule up front: respond to the remark, never invent claims about a real individual.
- It was deliberately set aside this session so the security work could come first.
How to plan a risky new feature responsibly
- Before building anything like this, study the tools that already exist, especially how they fail.
- Decide the safety boundary first: respond only to what was said, never target a real, named person.
- Build that boundary in from the start, rather than trying to patch it after something goes wrong.
How to prioritise a new feature against existing work
- Reuse the access and payment controls you already have rather than inventing new ones.
- Be honest about priority, put fixing known problems ahead of building nice-to-have features.
Questions & answers
Why look at other tools before building your own? To learn from their mistakes cheaply. Spotting a harmful failure in someone else's tool beforehand is far better than discovering the same failure in your own after it's live.
What was the concerning behaviour you found? One tool, given a real person's name, made up a personal insult aimed at that named person. That's the exact line a feature like this must never cross.
What rule did that lead to? Respond to the remark that was actually made, and never invent personal claims or attacks about a real, named individual, decided before any building began.
Why wasn't it built? Because real security issues on the server took priority. Fixing those came first, ahead of adding a new public feature, a deliberate choice.
Technical detail
Scope
Planned public feature ("Retort Q&A"): user submits a remark and selects a tone from a set of options; receives AI-generated retort candidates; results persisted as searchable Q:/A: text pairs for AI-search citability. Included here as a not-built item for a complete record.
1 · Competitor research before build
Reviewed four existing comparable tools to learn desirable behaviour and, critically, to surface failure modes to design against pre-emptively rather than post-launch.
2 · The harassment failure mode found
One competitor, given test input containing a real person's name, generated a fabricated targeted personal attack on that named individual. This is the key risk for a generative "comeback" feature: the gap between responding to a remark and generating defamatory/harassing content about a named real person.
3 · The resulting prompt-design constraint
Hard rule for any implementation, fixed at design time: the generator must respond only to the submitted remark and never fabricate character claims or personal attacks about a named real individual. Enforced in the system prompt and reinforced with input handling around detected names, designed in from the start, not retrofitted.
4 · Gating / cost model
Access gated through the site's existing credit-based middleware (the same requireTool() / requireApiCall() pattern used by other tools), rather than a bespoke mechanism, consistent access control and billing with the rest of the platform.
5 · Reusable storage pattern noted
During the earlier code review, a public GET /api/qa endpoint backed by a qa_items table was found, a close structural match to what this feature needs, meaning the storage layer could likely be reused rather than built from scratch.
6 · Status
Researched, scoped, and safety-bounded, but deprioritised this session in favour of the security audit work. Not built.
Key points
- Planned public "Retort Q&A" feature: remark + tone → AI retorts, stored as searchable Q:/A: pairs.
- Competitor research surfaced a real harassment failure mode: fabricated attacks on a named real person.
- Design rule fixed up front: respond to the remark, never fabricate claims about a named individual.
- Gating via existing credit middleware; storage likely reusable from an existing
qa_itemstable. - Not built this session, deliberately deprioritised behind the security work.
How you'd approach this yourself
# 1. study comparable tools for failure modes BEFORE building # 2. encode the safety boundary in the system prompt from line one: # "Respond only to the submitted remark. Never fabricate claims # or attacks about any named real individual." # 3. reuse existing access control rather than inventing new gating # 4. reuse existing storage (qa_items) if its shape already fits
Questions & answers
Why research competitors before building? To inherit their hard-won lessons cheaply, especially failure modes. Finding a harassment flaw in someone else's tool is far better than shipping the same flaw in your own.
What exactly was the risky behaviour? A competitor, given a real name, invented a personal attack on that named person. A comeback tool must respond to the remark, not manufacture defamatory content about an individual.
Why reuse existing gating and storage? Consistency and safety: the credit middleware and qa_items table are already proven on the site, so reusing them avoids a new surface of bugs and keeps behaviour uniform.
Why wasn't it built? Concrete server-security findings took priority. Fixing real, live security issues comes before shipping a new public feature, a deliberate ordering, not an oversight.
Testing our own security checker against real sites
Confirmed safeWhy a tool that always says "fine" is worse than useless
A security-checking tool is only worth anything if it tells the truth even when the truth is uncomfortable. The failure to worry about isn't a tool that's occasionally too harsh, it's one that quietly rubber-stamps everything as fine regardless of what it's actually looking at, because that gives false confidence exactly where it's most dangerous. So before relying on our own public tool for anything, we needed to prove it doesn't do that: that it genuinely inspects a site and reports real, varying results rather than always returning a comfortable pass.
1 · Choosing a fair, real-world test
The honest way to test that is to point the tool at real websites that we don't control and have no reason to score in any particular way, and ideally ones where you'd expect a serious security posture, so a weak result would clearly mean something. We chose two real, well-known businesses' public websites for exactly this. Running the tool against real sites, rather than only our own, removes any suspicion that it's tuned to flatter a site it already knows.
2 · What "a working tool" would actually look like
Before seeing the results, it's worth being clear about what would count as the tool working versus failing, because otherwise any result can be rationalised after the fact. A broken tool tends to fail in one of two obvious ways: it scores everything as perfect (so it isn't really checking anything), or it fails everything (so it's just throwing errors and flagging problems that aren't there). A genuinely working tool does neither: it produces different, believable scores for different sites, reflecting the real differences in how well each one is actually protected.
3 · The results, real, and meaningfully different
That's exactly what happened. The two sites scored clearly differently from each other, and both scores were plausible for what they were. One site was missing several important protections and scored notably lower as a result. The other had a number of protections properly in place, but was still missing some others, and scored better, though not perfect. Two real sites, two different, sensible scores, each one traceable to concrete things the site did or didn't have. Neither an all-pass nor an all-fail.
4 · Why that specific pattern is the reassuring one
The differentiation is the whole point, and it's worth spelling out why. If both real sites had scored a perfect 100, that would have been a warning sign that the tool wasn't genuinely testing anything, a rubber stamp. If both had scored zero, that would have suggested it was broken and flagging phantom problems. Two different, defensible scores, each tied to real, nameable gaps, is precisely the signature of a tool that is actually inspecting each site on its own merits and reporting honestly. It's the good outcome, not just an outcome.
5 · Outcome
The test confirmed the tool produces credible, real-world-plausible results rather than a fixed or broken score, which is what gave us the confidence to go on and use it for the broader testing that follows in this write-up. A checker you haven't proven is honest is one you can't safely rely on; this is the step that earned that reliance.
Key points
- A checker that always says "fine" is worse than useless, it gives false confidence where it's most dangerous.
- We tested our tool against two real, well-known businesses' sites, ones we don't control.
- A working tool gives different, believable scores for different sites, not all-pass, not all-fail.
- The two sites scored clearly differently, each traceable to real, nameable protections they had or lacked.
- That pattern is the reassuring one: it proves the tool genuinely inspects each site rather than rubber-stamping.
How to test a security checker against real sites
- Point any checker you rely on at real sites you don't control, not just your own.
- Decide in advance what "working" looks like: different, sensible scores, not everything perfect.
- Run it against at least two different sites and compare.
How to confirm a checker's scores are trustworthy
- For each score, check it traces back to real, concrete things the site does or doesn't have.
- Only trust the tool for real decisions once it's shown it can tell good and bad apart honestly.
Questions & answers
Why not just test it on your own site? A tool might be unconsciously set up to give your own site a good result. Testing real sites you don't control is a fairer, more honest check of whether it really works.
Why would two perfect scores be a bad sign? Because real websites genuinely differ in how well they're protected. If everything scores perfectly, the tool probably isn't really checking anything.
What made the results trustworthy? The two sites got clearly different scores, and each score matched real, specific protections the site had or was missing, not a random or fixed number.
Why do this before the other testing? Everything that follows depends on the tool being honest. This is the step that proved it, before we relied on it further.
Technical detail
Purpose
Validation of the live public security-audit tool: confirm it produces genuine, differentiated scoring rather than a constant pass (untested) or constant fail (broken). Tested against two real third-party sites we don't control.
1 · Method
Ran the live tool against two real, public business websites, chosen as independent, uncontrolled targets where a weak result would be meaningful. Passive HTTP/TLS/DNS checks only, identical to any user-initiated scan.
2 · Results
# Site A 63/100, missing Content-Security-Policy, HSTS, X-Frame-Options, X-Content-Type-Options, Referrer-Policy, Permissions-Policy; no DMARC/DKIM # Site B 82/100, HSTS present, full mail security (SPF/DMARC/DKIM); still missing CSP, Referrer-Policy, Permissions-Policy, cookie SameSite
3 · Interpretation
Each score is traceable to concrete, nameable header/DNS facts, not an opaque number. The two sites differ by 19 points on real differences (mail security and HSTS present on B, absent on A). This is the signature of genuine per-site inspection.
4 · Why the pattern validates the tool
Two failure signatures were being ruled out: uniform 100 (not actually testing) and uniform 0/errors (broken, false positives). Differentiated, defensible scores tied to specific missing controls rule out both.
5 · Outcome
Confirmed genuine, differentiated, plausible scoring. This validation is what justified using the tool for the subsequent multi-platform testing.
Key points
- Goal: prove the tool isn't a rubber stamp (constant pass) or broken (constant fail).
- Tested against two real, uncontrolled third-party sites, passive checks only.
- Results: 63/100 and 82/100, a 19-point spread on real differences (HSTS + mail security).
- Every point of difference traces to concrete missing headers/DNS records, not an opaque number.
- Differentiated, defensible scores rule out both failure signatures, the tool inspects genuinely.
How to do this yourself
# validate any scanner against real, uncontrolled targets before trusting it: # 1. run it against 2+ real sites you don't own # 2. expect DIFFERENT, defensible scores, not all-100, not all-0 # 3. trace each score to concrete facts, e.g.: $ curl -sI https://<site>/ | grep -iE "strict-transport|content-security|x-frame" $ dig +short TXT _dmarc.<domain> # mail security present or not
Questions & answers
Why test against sites you don't own? A tool can be accidentally tuned to flatter sites it knows. Uncontrolled real targets remove that bias, the scores reflect the sites, not the tool's expectations.
Why is a 100/100 on a real site a bad sign here? Uniform perfect scores across different real sites suggest the tool isn't actually inspecting anything. Real sites vary, so honest results should vary too.
What makes the two scores "believable"? Each one decomposes into concrete facts, B has HSTS and full mail security, A has neither, so the 19-point gap is explained by real, checkable differences, not a random number.
Why does this matter before the rest of the testing? Everything downstream relies on the tool being honest. Proving it produces differentiated, defensible results is what earns the right to trust its later verdicts.
Built a tool to test 10 major website platforms
Confirmed safeWhy build a second tool at all
Our public security audit checks one website at a time, the way a visitor's browser would. That's exactly right for a member of the public checking their own site, but it makes testing the checking logic across many sites slow and manual: you'd have to sit and run it by hand, one site after another, and copy down each result. So we built a separate companion tool that runs the same kind of safe, passive checks, the ones that only look at a page the way any browser sees it, but automatically down a whole list of websites in one pass, straight from our own server. Same checks, many sites, no manual clicking.
It's worth being clear this is a genuinely separate program from the public audit, not a change to it. The public tool on the website was never touched; this is a standalone script we wrote for our own testing, so we could put the checking logic through its paces against a wide, varied range of real sites quickly.
1 · Keeping it strictly to safe, passive checks
The tool does only passive checks, the exact same category of thing a normal web browser does when it loads a page and reads the response. It sends no attacks, tries no break-ins, and does nothing a site owner could object to; it simply looks at what every visitor's browser already sees: the security-related settings a site sends back with its pages. That restraint is deliberate and important, it's what makes running it against websites we don't own completely legitimate, exactly as looking at any public page in a browser is legitimate.
2 · Choosing a deliberately varied set of test sites
To really put the checking logic through its paces, we needed sites that are genuinely different from one another, not ten near-identical sites that would all pass or fail together and prove nothing. So we chose the official websites of 10 of the biggest website-building platforms in the world. These are run by large, sophisticated companies with real security teams, and they're built on very different underlying technology from one another, which makes them an excellent, demanding, and fair stress test: if the logic handles this much variety correctly, it's robust.
3 · What the results showed, real variation, not uniformity
The results varied genuinely and sensibly from platform to platform, which is exactly what you want to see. Some scored very well, with barely any issues. Others, even these big, capable companies, had real, specific, identifiable gaps in their security settings: things like missing protections on the small pieces of data a site stores in your browser, or missing security-related instructions in the pages they serve. A few even openly advertised, in their pages, exactly which platform they were built on, a small information leak in itself. The point isn't to single any of them out; it's that the tool found real, different, nameable things on different sites, which is precisely how you know it's genuinely inspecting each one rather than producing a canned answer.
4 · Why this was a meaningful test of the logic
This mattered because it proved the checking logic behaves correctly and consistently across a wide range of real, very different websites, not just the one site we started from. It's easy for a checker to work on the site it was built and tested against, and then behave strangely the first time it meets a site built differently. Running it across ten genuinely different, professionally-built platforms and getting sensible, varied, defensible results from each is strong evidence that the logic is sound in general, not just narrowly tuned to our own site.
5 · Outcome
The companion tool confirmed the underlying checking logic is correct and consistent across a broad, diverse set of real sites, giving real confidence in the checks themselves. And it did so using only the same safe, passive approach as the public audit, run entirely from our own machine against publicly-available pages.
Key points
- A separate companion tool runs the same safe, passive checks as the public audit, but down a whole list of sites in one pass.
- It does only what a browser does on page load, no attacks, which makes checking sites we don't own legitimate.
- We chose 10 of the biggest, most different website platforms on purpose, as a demanding, fair stress test.
- Results varied sensibly, some near-perfect, others with real, nameable gaps, proving genuine per-site inspection.
- It confirmed the checking logic works across many very different real sites, not just our own.
How to test a checker across many platforms
- If you rely on a checker, test its logic against many sites, not just the one it was built on.
- Keep the checks strictly passive (only what a browser does on page load), so testing others' public pages is safe.
- Deliberately pick sites that differ from one another, so a real bug has a chance to show.
How to read multi-site test results
- Look for varied, sensible results, some good, some with specific gaps, rather than everything identical.
- Treat that variation as the good sign: it means the tool is genuinely inspecting each site.
Questions & answers
Is this your public audit tool? No, it's a separate program we built just for testing. The public tool on the website was never changed; this one simply reuses the same safe checks to try them across many sites at once.
Is it OK to run this against sites you don't own? Yes, it only looks at a page the way any visitor's browser does. It sends nothing harmful and tries no break-ins, so it's the same as viewing a public page.
Why test 10 different platforms instead of 10 similar sites? Because difference is the whole point. Ten near-identical sites would all behave the same and prove little; ten very different ones that each give sensible results show the tool really works in general.
What did "varied results" actually look like? Some platforms scored very well; others had real, specific gaps, and a few even advertised which platform they were built on. Different, nameable findings on different sites is exactly how you know it's inspecting each one properly.
Technical detail
Purpose
Build a server-run, multi-target harness that exercises the live tool's passive check categories across many domains in one pass, to validate the check logic against a diverse real-world sample without manual, one-at-a-time runs. Distinct program from the public audit; the live product was untouched.
1 · What it is
A standalone script (site_check.py) replicating the live tool's passive categories, HTTP security headers, TLS config, cookie flags, mixed content, server fingerprint, CMS detection, mail DNS (SPF/DMARC/DKIM), robots/security.txt, looped over a domain list, driven by curl / openssl / dig. Purely outbound, read-only requests; nothing touches local files, no sudo needed.
$ python3 site_check.py # loops the domain list, passive checks only # per site: curl -sI (headers) · openssl s_client (TLS) · dig TXT (SPF/DMARC/DKIM)
2 · Passive-only, by design
Every check is what a browser already performs on page load, no active payloads, no port scans, no auth attempts. That scoping is what makes running it against third-party public sites legitimate (identical to viewing the page and reading response headers).
3 · Target set, chosen for diversity
Ran against 10 major CMS/site-builder platforms' own official sites: WordPress, Shopify, Wix, Squarespace, Webflow, Drupal, Ghost, BigCommerce, HubSpot, Joomla. Large, professionally-run, and built on very different stacks, a demanding correctness sample.
4 · Representative results
Ghost / Webflow near-perfect , strong headers, cookie flags, TLS Wix / BigCommerce / Squarespace real gaps , missing cookie SameSite / security headers Drupal / Joomla / Wixinfo leak , CMS generator meta tag exposed (platform advertised)
5 · Why it validates the logic
Uniform results across a diverse set would signal the logic is insensitive (not really inspecting). Instead, results decomposed into concrete, per-site, per-category facts, varied and defensible, confirming the check logic generalises beyond the site it was built on.
6 · Outcome
Check logic verified correct and consistent across a diverse real-world sample, via passive-only, server-run requests. Public audit untouched.
Key points
- Separate server-run harness (
site_check.py), same passive categories as the live tool, many domains per pass. - Passive-only (curl/openssl/dig), legitimate against third-party public sites; live product untouched.
- Target set chosen for diversity: 10 major platforms on very different stacks.
- Results varied per site/category (near-perfect → real cookie/header gaps → exposed generator meta), not uniform.
- Varied, decomposable results prove the check logic generalises beyond the origin site.
How to do this yourself
# passive, per-domain, in a loop, same shape as the harness:
$ for d in site1 site2 site3; do
echo "== $d =="
curl -sI "https://$d/" | grep -iE "strict-transport|content-security|x-frame|set-cookie"
dig +short TXT "_dmarc.$d"
done
Questions & answers
Is this the same as the public audit tool? No, it's a separate script that reuses the same passive check categories. The live product was never modified; this exists purely to test the logic at scale.
Is it legal to run against sites you don't own? Yes, it only does what a browser does on page load (reads response headers, checks TLS, looks up public DNS). No active payloads, no port scans, no auth attempts.
Why 10 different platforms rather than 10 similar sites? Diversity is the test. Ten near-identical sites would pass or fail together and prove little; ten different stacks that each yield sensible, distinct results show the logic generalises.
What does an "exposed generator meta tag" mean? Some sites include a tag in their HTML naming the platform they're built on. It's a minor information leak, it tells an attacker exactly what software (and therefore what known weaknesses) to target.
Real bug found in our own security tool
FixedThe moment of doubt, and why we welcomed it
Our own security-checking tool gave the main site a perfect 100 out of 100. That should feel good, but a perfect score is exactly the kind of result that deserves suspicion rather than celebration, because the most dangerous failure a checking tool can have is telling you everything's fine when it isn't. So instead of taking the win, we did the honest thing and cross-checked it: we ran the very same site through a completely independent, well-known security checker built by a major technology company, to see whether it agreed.
It didn't. The independent checker scored the site 80 out of 100, and it pointed to one real, specific weakness in a particular security setting, a weakness our own tool had missed entirely and papered over with full marks. That gap between "100 from us" and "80 from them, with a named problem" is the whole story of this section: a real bug in our own tool, caught precisely because we didn't trust our own perfect score.
1 · Not dismissing the disagreement
The easy, comfortable response to that disagreement would have been to wave it away, to assume the other tool was being fussy and ours was right. We deliberately didn't. When a trusted independent tool disagrees with yours and names a concrete problem, the honest assumption is that yours might be wrong, and the only way to know is to look. So rather than defend our score, we went and read our own tool's actual underlying code to find out exactly why it had given full marks where a respected checker saw a real fault.
2 · What the code was actually doing
Reading the code revealed the bug plainly, and it was a genuinely instructive one. Our check for this particular security setting was only checking whether the setting was present at all, not what it actually said. It's the difference between checking that a door has a lock fitted and checking whether that lock is any good or left wide open. Because of that, a site could have the weakest, loosest, most permissive possible version of this setting, one that effectively cancels out the protection the setting is supposed to provide, and still sail through our check with full marks, purely because something was there. Present-but-useless was being scored identically to present-and-strong.
3 · Fixing it against the real standard
We rewrote the check properly. Instead of merely asking "does this setting exist?", it now inspects what the setting actually contains and judges whether it's genuinely strong, using the real, well-established rule for what makes this setting strong (that it must not include the self-defeating options that cancel its protection). To be clear about what we did and didn't do: we did not change our tool to produce the same score as the independent one. The two use completely different scoring scales, and we're not claiming an 80 or trying to reproduce it. The independent tool didn't set our score, it simply helped us spot that our own check was wrong. All we changed was the check itself, so it now judges this setting correctly instead of rubber-stamping it.
4 · Testing the fix three ways before trusting it
A fix to a checking tool has to be tested as carefully as anything else, because a check that's wrong in a new direction is no better than the old one. So before putting it live we tested it three ways: we confirmed it correctly failed the site's real, currently-weak setup (the thing the independent tool caught); confirmed it correctly passed a properly strict, secure setup (so it isn't just failing everything now); and confirmed it correctly failed when the setting was missing entirely. Fails the weak case, passes the strong case, fails the absent case, all three, so we know it's discriminating correctly rather than just flipped to always-fail.
5 · Outcome
Once deployed and re-tested on the real live site, the tool now correctly and honestly reports this as a genuine issue needing attention, with a clear, accurate explanation of exactly what's wrong, instead of hiding a real problem behind a false, reassuring 100/100. The tool is now more honest than it was, which for a security checker is the most important quality it can have. And it was our own willingness to distrust a flattering result that surfaced the bug in the first place.
Key points
- Our tool scored the site a perfect 100; an independent, trusted checker scored it 80 and named a real weakness ours missed.
- We cross-checked rather than celebrated, a perfect score is the one that most deserves suspicion.
- The bug: our check only asked whether a setting existed, never whether it was any good.
- We rewrote it to judge the setting's real content, not to change our score to match the other tool (different scales); it just helped us find the bug.
- We tested the fix three ways (weak fails, strong passes, missing fails) before putting it live.
How to catch a bug in your own security tool
- Treat a perfect score from your own tool as a reason to double-check, not to relax.
- Cross-check the same thing with an independent, well-regarded tool and see if it agrees.
- If it disagrees and names a problem, assume yours might be wrong and go read your own logic.
How to fix and verify a checker bug
- Make sure your checks judge what a setting actually says, not just whether it's present.
- Test any fix on a weak case, a strong case, and a missing case before trusting it.
Questions & answers
Why not trust your own 100/100? Because a perfect score is exactly what a broken checker would also produce. The safest response to your own tool saying "all perfect" is to have an independent tool check the same thing.
What was actually wrong? Our check only looked at whether a security setting was there at all, not whether it was strong. So a site with the weakest possible version of that setting still got full marks.
How did you fix it? We rewrote the check to look at what the setting actually says and decide whether it's genuinely strong. We didn't change our tool to output the other tool's number, the scales differ, we just stopped it scoring a self-defeating setting as safe.
How do you know the fix is right? We tested it three ways: it now correctly fails the weak real setup, correctly passes a properly strict one, and correctly fails when the setting is missing, then verified it on the live site.
Technical detail
The trigger
Our tool scored the main site 100/100. Cross-checked against an independent tool (Mozilla HTTP Observatory), which scored it 80/100 and flagged 'unsafe-inline' / 'unsafe-eval' in the Content-Security-Policy header's script-src, both of which defeat CSP's core XSS protection. A perfect score from our tool alongside a named real fault from a trusted one meant our tool was wrong.
1 · Root cause, presence-only check
Read the live tool's backend (audit-handler.js). The CSP check was:
csp: { pass: !!h['content-security-policy'], weight: 8 }
Presence-only: any CSP header at all scored a pass, regardless of content. A policy with 'unsafe-inline' 'unsafe-eval', i.e. one that neuters its own protection, passed identically to a strict one.
2 · The fix, evaluate the directive against the real CSP rule
Added a strictness check using the actual CSP rule (a policy is only strong if the relevant directive contains neither 'unsafe-inline' nor 'unsafe-eval'). This does not tune our score to the independent tool's number, the two scales differ; it fixes what our check evaluates:
function _saCspIsStrict(csp) {
if (!csp) return false;
const scriptSrc = csp.match(/script-src\s+([^;]+)/i);
const defaultSrc = csp.match(/default-src\s+([^;]+)/i);
const relevant = scriptSrc ? scriptSrc[1] : (defaultSrc ? defaultSrc[1] : '');
if (!relevant) return false;
if (/'unsafe-inline'/i.test(relevant)) return false;
if (/'unsafe-eval'/i.test(relevant)) return false;
return true;
}
Falls back to default-src when script-src is absent (as the spec does), and fails on either unsafe token.
3 · Three-case validation before deploy
real live CSP (has unsafe-inline/eval) -> FAIL # the real fault, now caught genuinely strict CSP -> PASS # not just failing everything empty / missing CSP -> FAIL # absence still fails
4 · Deploy
Uploaded, reset ownership/permissions (chown/chmod), pm2 restart. Verified live: site now reports 21/22 checks passed, CSP correctly failing with an accurate custom message. Note this is the on-page audit's own header score, separate from the site's AI-readiness score, which lives elsewhere in the codebase.
5 · Outcome
A false 100/100 replaced by an honest, accurate result driven by real directive analysis and validated across fail/pass/absent cases. Our tool keeps its own scoring (checks passed / total), we did not re-style it to emit the independent tool's 80. The checker is now correct where it was previously blind.
Key points
- Our tool said
100/100; independent Mozilla HTTP Observatory said 80/100 and named a real CSP fault. - Root cause: CSP check was presence-only (
!!h['content-security-policy']), never read the directive. - A policy with
'unsafe-inline'/'unsafe-eval'(self-defeating) passed identically to a strict one. - Fix:
_saCspIsStrict()inspectsscript-src/default-srcand fails on unsafe tokens, matching the reference tool's logic. - Validated FAIL/PASS/FAIL across weak/strict/absent, then deployed; live now
21/22with CSP correctly failing.
How to do this yourself
# never trust a self-scored 100, cross-check against an independent tool # and evaluate directive CONTENT, not mere presence: $ curl -sI https://<site>/ | grep -i content-security-policy # a 'pass' should require: header present AND no 'unsafe-inline'/'unsafe-eval' # in script-src (or default-src as fallback)
Questions & answers
Why cross-check a perfect score? A 100 from your own tool is the least trustworthy result it can give, it's indistinguishable from a tool that isn't checking properly. An independent second opinion is how you catch that.
What was the actual bug? The CSP check used !!h['content-security-policy'], true if any CSP header exists. It never parsed the policy, so a self-defeating 'unsafe-inline' policy passed exactly like a strict one.
Why use the real CSP rule rather than invent your own? Because "strong CSP = no 'unsafe-inline'/'unsafe-eval'" is a well-established rule, so encoding it is simply correct. We did not tune our tool to reproduce the other tool's 80, the scales differ, we just stopped scoring a self-defeating policy as safe.
Why test the strict-CSP pass case, not just the fail? To prove the fix discriminates rather than merely flipping to always-fail. A check that fails everything is as useless as one that passes everything.
Added a check that was missing entirely
FixedA gap, not a bug, something we simply weren't checking
The previous section was about a check that was wrong. This one is about a check that was missing entirely, a genuine blind spot rather than a mistake. When we cross-checked our tool against the independent one, it tested one thing ours didn't look at all: whether the scripts and stylesheets a website loads from other websites carry a special safety verification. Being honest about a missing check is just as important as fixing a broken one; an incomplete checker gives false confidence in a quieter way.
Here's the real-world risk this addresses, in plain terms. Most websites pull in some code from outside sources, a font here, a script from another company there. Normally your browser just trusts and runs whatever those outside sources send. The safety verification in question lets a website say, in advance, "this outside file should look exactly like this, and if it doesn't, don't run it." That protects visitors if one of those outside sources is ever hacked or tampered with: instead of every visitor's browser silently running the altered, malicious code, the browser notices it doesn't match and refuses. It's a genuinely valuable protection, and our tool wasn't checking for it.
1 · Building the missing check
So we built and added it ourselves. The check reads through a page, finds every script and stylesheet that's loaded from outside the site's own address, and works out whether each one carries that safety verification or not, flagging the ones that don't. We built it on top of the same reliable page-reading component the tool already used elsewhere, rather than bolting on something new and untested, so it fits cleanly with how the rest of the tool works.
2 · The second, separate problem, it worked but didn't show
Once the check was working correctly behind the scenes, a second and quite different problem appeared: the results page that visitors and admins actually see didn't display the new check anywhere, even though it was running and scoring correctly out of sight. This is a good example of how two parts of a system have to agree with each other. The part that does the checking now knew about the new check, but the part that displays the results keeps its own separate list of checks it knows how to show, and that list hadn't been told the new check existed. So the check ran, scored, and then vanished before anyone could see it.
3 · Fixing the display side too
The fix was to add the new check to that display list as well, matching exactly the format every other check already used, so it slots in looking and behaving like a native part of the page rather than an afterthought. It's a small change, but it's the difference between a check that quietly works where nobody can see it and one that actually informs the person reading the report. A check that runs but never surfaces is, from the reader's point of view, not really there.
4 · The honest final score
With both parts fixed and live, the check itself, and its display, the new check now appears correctly on the results page with a clear explanation, and the main site now scores 93 out of 100, correctly passing 22 of 23 checks in total. The one remaining issue is the real CSP weakness described in the previous section, now accurately reported rather than hidden. That's the honest picture: a genuine, nearly-complete score with the one real outstanding item shown plainly, which is far more useful than a flattering but false perfect mark.
5 · Outcome
A real protective check that was entirely absent is now present, working, and visible, closing the last gap between our tool and the trusted independent one, and giving anyone who runs the audit a fuller, more honest picture of a site's security than before.
Key points
- This was a missing check, not a broken one, a genuine blind spot the independent tool exposed.
- It checks whether outside scripts/styles carry a safety verification so a tampered file won't run in a visitor's browser.
- We built the check on the tool's existing, proven page-reading component.
- A second problem: it worked but didn't show, the display side kept its own list and didn't know the new check existed.
- Fixing both parts, the site now scores 93/100 (22/23), with only the real CSP issue correctly still flagged.
How to find a missing check in your tool
- Compare your checker against a trusted independent one and note anything it tests that yours doesn't.
- For a genuine gap, build the missing check rather than ignoring it, a missing check is false confidence.
How to add a new check correctly
- Reuse your tool's existing, proven building blocks rather than bolting on something new.
- Remember the display side: a new check that scores correctly but never appears is no use.
- Prefer an honest, slightly-imperfect score that shows real issues over a flattering perfect one.
Questions & answers
What does this new check protect against? If a website loads code from an outside source and that source is ever hacked, this protection lets the browser notice the code has changed and refuse to run it, protecting every visitor instead of silently running the tampered version.
Why did the check work but not show up? The part that runs checks and the part that displays them keep separate lists. The checking part knew about the new one; the display part didn't, so it ran but never appeared until we updated the display side too.
Why does the score drop from 100 to 93? Because 93 is the honest number. The old 100 hid a real problem; 93 reflects a near-complete, genuine result with the one real outstanding issue shown clearly.
Is one remaining issue a bad sign? No, it's an honest one, and it's the CSP weakness from the previous section, now correctly reported. A tool that shows real remaining issues is doing its job.
Technical detail
The gap
Added Subresource Integrity (SRI) checking, previously absent entirely, and the one remaining real gap versus the independent Mozilla checker. SRI lets a page pin a cryptographic hash on cross-origin <script>/<link> resources so the browser refuses to execute a tampered file (e.g. a compromised CDN).
1 · Backend check
Implemented _saCheckSri() using the cheerio parser already present in the file, parse the HTML, find cross-origin script/stylesheet references lacking an integrity attribute, and report them:
function _saCheckSri(html, baseUrl) {
const $ = cheerio.load(html);
const missing = [];
$('script[src]').each((_i, el) => {
const src = $(el).attr('src');
if (isCrossOrigin(src) && !$(el).attr('integrity'))
missing.push({ tag: 'script', url: src });
});
// same pattern for <link rel="stylesheet">
return { applicable: true, missing };
}
2 · The frontend disconnect
The check scored correctly server-side but rendered nothing on the results page. Cause: the frontend keeps a separate hardcoded CHECK_META map of every known check's label/description, and it had no sri entry, so a correctly-scored result had no display metadata and was silently dropped from the rendered list.
# CHECK_META had no 'sri' key -> result computed but never displayed
3 · Frontend fix
Added the sri entry to CHECK_META, matching the existing label/description formatting exactly so it renders identically to native checks:
sri: { label: 'Subresource Integrity',
desc: 'Cross-origin scripts/styles pinned with integrity hashes' }
4 · Deploy & verify
Deployed both changes; verified live: the SRI row now displays with a PASS status. Combined with the CSP fix from the previous section, the site scores 93/100, 22/23 checks passed, the single remaining fail being the (correctly-reported) CSP.
5 · Outcome
SRI coverage added end-to-end (compute + display), closing the last gap against the reference checker. Backend and frontend check registries are now consistent for this check.
Key points
- Added SRI checking, previously missing; the last real gap vs the independent checker.
- SRI pins cross-origin scripts/styles so the browser rejects a tampered file (e.g. hacked CDN).
- Backend
_saCheckSri()reused the existingcheerioparser, flags cross-origin refs lackingintegrity. - Second bug: correct score didn't render, frontend
CHECK_METAhad nosrientry, so it was dropped. - Added the display entry; live now
93/100,22/23, only the CSP correctly still failing.
How to do this yourself
# find cross-origin scripts/styles missing an integrity attribute: $ curl -s https://<site>/ \ | grep -oE '<(script|link)[^>]+(src|href)="https?://[^"]+"[^>]*' \ | grep -v 'integrity=' # and remember: a new backend check also needs a frontend display entry
Questions & answers
What does SRI actually protect against? A compromised third-party source (e.g. a hacked CDN) serving altered code. With an integrity hash pinned, the browser refuses to run a file that doesn't match, so tampering fails closed instead of silently executing.
Why did a correct check show nothing on the page? The frontend has its own list of known checks with display labels. The new check scored fine but had no entry there, so the renderer had nothing to show and dropped it.
Why reuse cheerio rather than a new parser? It was already in the file and proven; reusing it keeps the new check consistent with existing ones and avoids adding an untested dependency for the same job.
Why is 93/100 a better result than the old 100? Because it's true. It reflects a real, near-complete posture with the one genuine outstanding item (CSP) shown plainly, instead of a false perfect that hid a real problem.
Tool built to fix the underlying weakness (not yet applied)
Confirmed safeDetecting a problem is not the same as fixing it
The earlier sections found and correctly reported the security-setting weakness. This section is about actually fixing it properly, and being completely honest that the fix is built but not yet fully applied. There's an important, tempting shortcut here that we deliberately did not take: you can make that weakness "go away" instantly by simply loosening the setting and moving on. But that's not a fix, it's hiding the problem, and worse, blindly tightening or loosening this kind of setting risks breaking real, legitimate features on the site, like analytics or visitor-tracking scripts that may depend on the current arrangement. A change that silently breaks working features is not an improvement. So we chose the slower, correct path: understand exactly what's there first, then fix it in a way that can't break anything.
1 · Investigating before changing anything
We started by looking closely at the homepage to understand what the setting is actually protecting. We found it has only a small handful of blocks of code written directly into the page itself (rather than loaded from separate files). Crucially, we checked whether any of them use a particularly risky kind of on-the-fly code execution, the kind that would genuinely require the very loosest, most permissive version of this security setting. None of them did. That's a significant finding, because it means the most dangerous part of the current permissive setting isn't actually needed at all on that page and can very likely be removed outright, tightening security with no loss of function.
2 · Building a tool to do it properly, at scale
A website isn't one page, though, so a real fix has to work across the whole site. So we built a tool that does this properly and at scale. It scans every single page on the site, finds every one of those directly-written code blocks, automatically removes exact duplicates, since many pages reuse the same few blocks, like a shared navigation-menu script that appears on every page, and then calculates a unique digital "fingerprint" for each genuinely different one. The point of that fingerprint list is what makes the fix safe: instead of telling the site's security settings "allow any directly-written code to run" (the loose, risky current state), you can tell them "only ever allow exactly these specific, known, pre-approved blocks, and nothing else, from anywhere." That's a dramatically safer posture, and because it's built from the real code already on the site, it does it without breaking any of the legitimate code that's supposed to be there.
3 · Why fingerprinting is the safe way to tighten this
It's worth being clear why this approach is the careful one. Loosening the setting weakens everyone's protection. Tightening it carelessly breaks the site. Fingerprinting threads the needle: it tightens the setting all the way down to "only the exact code we already know and trust", which is as strict as it can possibly be, while guaranteeing that every piece of legitimate code currently on the site is on the approved list and keeps working. You get maximum strictness and zero breakage at the same time, but only if the fingerprint list genuinely covers every page, which is exactly why the remaining step matters.
4 · Where this honestly stands
Here's the honest status. We successfully tested the tool on the homepage alone, and it correctly found and fingerprinted the handful of code blocks there. But the website as a whole has just over 1,000 separate pages. The tool is fully built and handed over, but running it across the entire site, and then actually updating the live security settings with the complete, site-wide fingerprint list, has not been done yet. Until that happens, the fingerprint list would be incomplete, and applying a partial list would risk blocking legitimate code on the pages not yet scanned. So the safe thing is to complete the full scan first.
5 · Outcome (open)
The proper fix is designed, built, and proven on a single page, but not yet rolled out site-wide, so the underlying weakness is still present on the live site as things stand. This is included precisely because an honest write-up records the things that are built-but-not-finished, not only the completed wins. The tool to do it right exists; the site-wide run and the live settings update are the remaining work.
Key points
- Detecting the weakness isn't fixing it, this is the proper fix, built but honestly not yet fully applied.
- We refused the shortcut of just loosening the setting (hides the problem) or tightening blindly (breaks real features).
- Homepage check found only a few directly-written code blocks and none needing the most dangerous permission.
- We built a tool that scans every page, dedupes shared blocks, and "fingerprints" each unique one for an exact allow-list.
- Open: the site has ~1,000 pages; the full scan and the live settings update aren't done, so the weakness is still live.
How to investigate a security setting before changing it
- Before changing a security setting, investigate what it's actually protecting so you don't break things.
- Check whether the most permissive, most dangerous part of the setting is even needed, often it isn't.
How to tighten a setting with an allow-list
- Prefer an exact allow-list of the specific code you already trust over a blanket "allow anything".
- Build the allow-list from a scan of every page, not just the homepage, or you'll block legitimate code.
- Be honest about a fix that's built but not fully rolled out, the weakness stays live until it's applied.
Questions & answers
Why not just switch the weak setting off now? Because blindly tightening it would break legitimate code on pages not yet accounted for. The safe fix needs the full list of trusted code from every page first.
What is the "fingerprint" actually for? It lets the site say "only allow exactly these known, trusted blocks of code to run." That's far safer than "allow anything", and it's built from the real code already on the site, so nothing legitimate breaks.
Why does having ~1,000 pages matter? The trusted-code list has to cover every page. Applying it after scanning only the homepage would block legitimate code on the other pages, so the whole site must be scanned first.
So is the site fixed? Not yet, honestly. The tool that does it right is built and proven on one page, but the full run and the live update remain, so the weakness is still present for now.
Technical detail
Goal
Properly remediate the CSP weakness ('unsafe-inline'/'unsafe-eval') rather than mask it, by moving to a hash-based CSP that allow-lists only the exact inline scripts actually present, so 'unsafe-inline' can be dropped without breaking legitimate inline code.
1 · Homepage investigation
$ grep -c "<script" index.html 6 # 6 total script tags $ grep -oE '<script[^>]*src=' index.html | wc -l 1 # 1 external -> 5 inline; one of those is JSON-LD, so 4 real inline JS $ grep -nE "eval\(|new Function" index.html # zero -> 'unsafe-eval' is not needed on this page at all
Key result: no dynamic-eval constructs, so 'unsafe-eval' is removable outright; the 4 inline scripts can be pinned by hash to remove 'unsafe-inline'.
2 · The tool
script_hasher.py walks every HTML file, extracts inline <script> bodies (excluding src= scripts and JSON-LD), deduplicates identical scripts site-wide, and computes a SHA-256 CSP hash per unique script:
SCRIPT_RE = re.compile(
r'<script(?![^>]*\bsrc=)(?![^>]*type="application/ld\+json")[^>]*>(.*?)</script>',
re.S | re.I
)
# for each unique body: "sha256-" + base64(sha256(body)) -> CSP script-src hash
3 · Why hashing is the safe remediation
Loosening CSP weakens protection; tightening blindly breaks inline scripts. Hash-pinning is exact: script-src 'sha256-…' 'sha256-…' allows precisely the known scripts and nothing else, dropping 'unsafe-inline' with zero breakage, provided the hash set covers every inline script on every page.
4 · Verified at single-page scale
Ran on the homepage: 4 unique inline scripts identified; tool-computed hashes matched hashes calculated manually. Logic confirmed correct.
5 · Remaining job (open)
Site has ~1,000 HTML files. Tool is built and validated at single-page scale but not yet run site-wide, and the nginx CSP header is not yet updated with the resulting hash set (7 identical CSP lines in the nginx config still carry 'unsafe-inline'/'unsafe-eval' for GTM/Cloudflare compatibility). A partial hash set would block legitimate scripts on un-scanned pages, so the full crawl must precede the header change. The weakness remains live.
Key points
- Goal: hash-based CSP that allow-lists exact inline scripts, so
'unsafe-inline'/'unsafe-eval'can be dropped safely. - Homepage: 6 script tags → 1 external, 4 real inline (1 JSON-LD excluded); zero
eval/new Function→'unsafe-eval'removable. script_hasher.pywalks HTML, excludessrc=and JSON-LD, dedupes, computes SHA-256 CSP hashes.- Hash-pinning = maximum strictness + zero breakage, but only if the hash set covers every page.
- Open: ~1,000 files not yet crawled; nginx CSP not yet updated, weakness still live.
How to do this yourself
# check whether unsafe-eval is even needed: $ grep -rnE "eval\(|new Function" *.html # hash one inline script for a CSP script-src allow-list entry: $ printf '%s' "<exact inline script body>" | openssl dgst -sha256 -binary | openssl base64 # -> add "sha256-…" to script-src, then remove 'unsafe-inline'
Questions & answers
Why not just remove 'unsafe-inline' now? It would break every legitimate inline script that isn't yet hash-pinned. The hash set must cover all ~1,000 pages first, or real functionality breaks.
How can you be sure 'unsafe-eval' is removable? The homepage has zero eval()/new Function constructs. If a site-wide scan confirms none elsewhere, 'unsafe-eval' can be dropped outright with no effect.
Why deduplicate scripts across pages? Shared blocks (nav, analytics) repeat on many pages. Deduping yields a small set of unique hashes instead of thousands of redundant ones, keeping the CSP header manageable.
Why isn't this done yet? The tool is built and proven on one page, but the full ~1,000-page crawl and the live nginx CSP update haven't been run. Applying a partial hash set would block legitimate scripts, so the complete crawl comes first.
The 'award winning' search phrase investigation
Confirmed safeNot a security issue, a mystery worth ruling out properly
This last item isn't a security fix at all; it's an honest investigation into something that looked odd and turned out to be harmless, included because ruling things out carefully is part of the same disciplined mindset as fixing real problems. A search-performance report showed a specific phrase describing the business favourably suddenly appearing in Google's results far more often than the week before, a genuinely large jump. The phrase didn't sound familiar, and an unfamiliar claim about your own business showing up in search is exactly the sort of thing you should chase down rather than either celebrate or ignore. Where was it coming from? Had something been added to the site without our knowledge? We investigated properly rather than assuming.
1 · Searching every file on the whole server
We started with the most thorough possible check: we searched every single file on the entire server, every domain, every file type, no exceptions, for that exact phrase. The only place it appeared was inside files belonging to completely unrelated third-party software that the site uses behind the scenes (marketing text in another product's bundled documentation), with nothing to do with this business's own content at all. So it wasn't in anything we'd written or published.
2 · Checking the files meant for AI and search engines
Next we checked the files specifically meant to be read directly by AI tools and search engines, the ones that describe a site to automated systems. If the phrase had been planted somewhere to influence how machines describe the business, this is where it would be. It wasn't there either. Nothing in any of those files contained it.
3 · Widening the net, and following the trail to its real source
We then broadened the search to catch near-variations, different spacing, hyphens, word order, in case it was present in a slightly different form. That turned up some old, leftover working files sitting in a private folder that isn't reachable from the internet at all. Digging into those, we traced the actual source: the phrase had been accidentally captured as sample test data by a separate tool used for checking lots of different websites at once, and it originally belonged to a completely different, unrelated real company. It had nothing to do with this business, it was just incidental data left over in a working folder. We also confirmed that folder returns a "not found" response from the internet, so nothing in it can be reached or read by any search engine or visitor.
4 · Checking everywhere else the business appears
To be genuinely thorough, we also checked everywhere off the site that the business appears. Every business directory listing was inspected, all of them plain factual entries (address, phone, category), with no promotional wording. Every social-media profile was checked too; several of those platforms deliberately block automated checking tools, so those were checked personally, by hand, rather than skipped. Nowhere did the phrase appear.
5 · The actual, boring, reassuring explanation
So the phrase genuinely does not exist anywhere connected to this business, not on the website, not in any file meant for AI tools, not in any directory listing, not on any social profile. The explanation turns out to be completely normal, well-documented behaviour: search engines can show a page as relevant to a search without that search's exact wording appearing anywhere on the page. They increasingly match on what a page is broadly about, its general topic and meaning, rather than on literal word-for-word matching. In other words, Google decided the site was relevant to that favourable phrase based on the site's overall subject matter, not because anyone wrote those words anywhere. We confirmed this is standard behaviour via the search engine's own official help community and several independent industry sources.
6 · Conclusion, nothing to fix, and that's the finding
There was nothing to fix, because there was genuinely nothing wrong or hidden anywhere to begin with. That is itself a worthwhile result: a surprising signal was chased all the way to the ground and shown to be benign, with certainty rather than a shrug. Sometimes the honest, valuable outcome of an investigation is a confident "there is no problem here", and being able to say that, having actually checked everything rather than assumed, is worth just as much as a fix.
Key points
- A favourable phrase spiked in Google's results; it didn't sound familiar, so we investigated rather than assumed.
- A whole-server search (exact and near-variants) found it only in unrelated third-party files and private, unreachable test data, never in the site's own content.
- Files meant for AI/search engines, every directory listing, and every social profile were all checked, some by hand.
- The cause is normal, documented behaviour: search engines rank a page for a phrase's topic, not its exact words.
- Nothing to fix, a confident "no problem here", reached by actually checking everything.
How to trace a strange phrase across your server
- When a strange phrase appears in your search stats, search your whole server for it before assuming anything.
- Search for near-variations too, different spacing, hyphens and word order.
- Check the files meant for AI tools and search engines specifically, not just your normal pages.
How to confirm whether a found phrase is exposed
- For any match, confirm whether it's actually reachable from the internet or just a private leftover.
- Check off-site too, directory listings and social profiles, by hand where automated tools can't reach.
Questions & answers
How can Google show a page for a phrase that isn't on it? Search engines increasingly match on what a page is broadly about, not just its exact words. So a page can be shown for a phrase it doesn't literally contain, based on its overall topic.
Was something added to the site without your knowledge? No. We searched every file on the whole server and the phrase only appeared in unrelated third-party software and in private files that aren't reachable from the internet, nothing in the business's own content.
Why check social profiles and directories by hand? Some of those platforms block automated checking tools, so an automated-only search would miss them. Checking personally makes sure nothing was overlooked.
If nothing was wrong, why does this count as a result? Because a surprising signal was chased all the way down and shown to be harmless with certainty. A confident "there's genuinely nothing wrong here" is a real and valuable outcome.
Technical detail
The signal
Google Search Console showed a query (a favourable "award winning" phrase) jumping 11 → 61 impressions week-on-week (+455%), all attributed to the homepage. The phrase wasn't known to be anywhere on the site, investigated rather than assumed.
1 · Exact-phrase sweep, whole server
$ grep -rlin "award winning" /var/www/vhosts/ # only match: an unrelated third-party npm package's own README.md # (marketing copy for a different product), nothing in the site's content
2 · AI/crawler-facing files
Checked llms.txt, JSON, and .md across all public folders specifically, empty result. Nothing planted where machines read.
3 · Broadened variant search → traced to source
$ grep -rlinE "award[- ]?winning|winning.{0,10}award|award.{0,15}win" /var/www/vhosts/ # 8 CSV files in a PRIVATE, non-web-accessible working folder # traced matched text -> an unrelated real company's own site content, # captured as incidental test data by a bulk domain-crawling tool $ curl -s -o /dev/null -w '%{http_code}' https://<site>/<that-folder>/ 404 # folder not web-accessible -> no path onto the live site, no crawler can read it
4 · Off-site presence
Directory listings (Hotfrog, FreeIndex, find-open.co.uk) fetched directly, bare factual entries, no promotional copy. Social profiles: several blocked automated fetching (robots/bot-detection), so were checked directly by the site owner and confirmed clean. General web search tying the phrase to the business/domain, no connecting results.
5 · Root cause
Confirmed via Google's Search Central community and independent SEO sources: this is standard, documented Search Console behaviour, impressions for a query do not require that query's literal wording on the ranked page. Google's ranking matches on broader topical/semantic relevance, so a page can accrue impressions for a phrase it never contains.
6 · Outcome
No fix required or possible: nothing associated with the site or business produces the phrase. A surprising metric was run to ground and shown benign with certainty.
Key points
- GSC query jumped 11 → 61 impressions (+455%) on the homepage for a phrase not known to be on the site.
- Whole-server exact
grep: only match was an unrelated npm package's README, not the site's content. - AI-facing files (
llms.txt/JSON/.md): nothing. - Variant regex found the phrase only in private, 404 (non-web-accessible) CSV test data from an unrelated company.
- Root cause: normal GSC semantic matching, impressions don't require the literal phrase on the page. No fix needed.
How to do this yourself
# 1. exact sweep across every vhost, case-insensitive: $ grep -rlin "the phrase" /var/www/vhosts/ # 2. broaden to spacing/hyphen/order variants: $ grep -rlinE "award[- ]?winning|winning.{0,10}award" /var/www/vhosts/ # 3. prove any hit is unreachable from the web: $ curl -s -o /dev/null -w '%{http_code}' https://<site>/<path>/ # expect 404
Questions & answers
How can a page rank for a phrase it doesn't contain? Modern search matches on topic and meaning, not just literal words. Google can judge a page relevant to a phrase based on what it's broadly about, so impressions don't require the exact wording on the page.
Was the phrase planted or injected anywhere? No. A whole-server sweep (exact and variant) found it only in unrelated third-party files and in private, non-web-accessible test data, never in the site's own served content.
Why did the 404 check matter? It proves the private folder holding the incidental match can't be reached from the internet, so no crawler or visitor can read it, ruling it out as the cause.
Why check social/directory listings by hand? Some platforms block automated fetching, so an automated-only check would have blind spots. Checking them directly closes those gaps and confirms the phrase is genuinely nowhere.
Rebuilt the audit to 54 checks, then tested it on 200 real sites
Confirmed safeMaking the audit more complete, then proving it holds up
The earlier testing showed the security audit was sound, but sound isn't the same as complete. A checklist that only covers some of what matters can still miss real things, so we went through the audit properly and added every meaningful check it was lacking, taking it from forty-four checks to fifty-four. Then, crucially, we didn't just trust that the new, bigger version worked, we put it through the hardest test we could: running it against two hundred different real websites and inspecting not only the results it calculated, but what it actually displayed on the page, to be certain the two always agreed.
1 · Adding the missing checks, honestly
We added a full set of checks the audit didn't previously make: modern browser-isolation settings, a safe test for an old and risky web method, certificate breadth, mail-branding records, and several others. The guiding rule throughout was honesty, not box-ticking. Many of the new items are genuinely optional, a site is not insecure for lacking them, so those are shown as neutral information, never as a red failure. A deprecated setting that modern browsers ignore is labelled as exactly that, with a note that any scanner still demanding it is out of date. The point of a bigger audit is to see more, not to invent more things to fail people on.
2 · Testing the results and the display together
A subtle but important point: a security tool can calculate the right answer and still show the wrong thing. The number it works out internally and the list of results a person actually reads on screen have to match exactly, or the report misleads even when the underlying logic is correct. So we tested both layers at once, the calculated result and the rendered page, and checked on every single site that the totals reconciled, that every result shown was accounted for, and that nothing extra was quietly displayed that the summary didn't count.
3 · The first hundred sites, and a real bug caught
We ran the audit against one hundred major websites and it behaved correctly on ninety-nine of them. The hundredth, a large government site, revealed a genuine display fault: because that site had a number of known software vulnerabilities, the page listed each one as if it were a separate check, inflating the visible list beyond the number the summary counted. The calculation was right, the vulnerabilities were real and correctly scored once, but the way they were shown didn't line up with the count. This is exactly the kind of fault that only appears on a site with that specific characteristic, which is precisely why testing across many varied real sites matters.
4 · Fixing it the honest way
The fix kept every vulnerability fully visible, hiding them would have been dishonest and defeated the point, but showed them as supporting detail beneath the single check that counts them, rather than as separate checks in their own right. The finding is reported once and counted once; the detail of exactly which vulnerabilities were found is shown in full underneath. Visible and honest, but no longer distorting the count.
5 · A second hundred, completely different sites
To be sure the fix worked and that the tool wasn't merely tuned to the first hundred, we ran it again against a different hundred sites, deliberately chosen to be varied: international sites, publishing sites built on common software (the exact kind likely to have the vulnerability pattern that caught us the first time), online shops, developer services, forums and government sites. This time all one hundred passed cleanly, results and display in perfect agreement on every one, including the sites that did have known vulnerabilities, which now displayed correctly. Two hundred different real sites in total, both the calculation and the on-screen report checked on each, every one reconciling.
6 · Outcome
The audit is now both broader and proven: fifty-four honest checks, validated on two hundred real websites at both the calculation and display level, with the one fault that testing uncovered found and fixed. A bigger checklist is only an improvement if it's still accurate; this is the work that confirmed it is.
Key points
- The audit was expanded from 44 to 54 checks, adding every meaningful check it lacked.
- New optional items are shown as neutral information, never as failures, a bigger audit should see more, not fail people on more.
- We tested the calculated result and the displayed page together on every site, so the two can never disagree.
- Testing on 100 sites caught a real display fault (a site's known vulnerabilities inflating the visible list); it was fixed so they show as detail, fully visible but counted once.
- A second, different 100 sites then passed cleanly, 200 real sites in total, every one reconciling at both layers.
How to properly test a tool you rely on
- Test the result it calculates and the result it displays together, they must always match.
- Run it across many genuinely varied real sites, not a handful of similar ones, some faults only show on specific kinds of site.
- When a fault appears, fix it without hiding information, keep everything visible, just show it correctly.
How to add checks honestly
- Mark genuinely optional items as neutral information, never as a red failure.
- Label deprecated settings as deprecated, don't demand things modern browsers ignore.
- Count each finding once; show supporting detail beneath it, not as extra checks.
Questions & answers
Does adding more checks just mean failing sites on more things? No, that would be the wrong kind of tool. Many of the added checks are genuinely optional and are shown as neutral information, not failures. The aim is to see more accurately, not to manufacture problems.
Why test what's displayed as well as what's calculated? Because a tool can work out the right answer and still show a misleading list on screen. If the number it counts and the results a person reads don't match, the report misleads even when the logic is sound, so both have to be checked.
Was hiding the vulnerabilities the fix? No. Hiding real findings would be dishonest and defeat the point of an audit. Every vulnerability stays fully visible; the fix simply shows them as detail under the single check that counts them, so the finding is reported once and counted once.
Why run a second, different hundred sites? To prove the tool isn't merely tuned to the first set and that the fix genuinely worked, especially on the kinds of site that had triggered the fault. Passing a fresh, varied hundred is far stronger evidence than re-running the same list.
Technical detail
Purpose
Expand the passive security audit from 44 to 54 checks, then validate the larger tool at scale across both layers, the API response (backend scoring/reconciliation) and the rendered DOM (frontend), against 200 unique real sites in two independent 100-site runs.
1 · The 54-check expansion
Added: coop, coep, x_dns_prefetch, clear_site_data, expect_ct (all weight 0, informational, rendered as a distinct INFO state, never a fail), http_methods (a safe TLS-socket HTTP TRACE probe for Cross-Site Tracing, weight 5), cert_scope (SAN breadth), bimi + DMARC-policy capture, cms_detected (informational), and refined referrer to grade policy strength and dkim to probe 20 selectors and warn (not hard-fail) on a miss, since a custom selector can't be detected passively. A new info status was added so weight-0 optional checks never render as red failures on any site.
2 · Both-layer test harness
Backend: POST each URL to /api/security-audit-free, assert pass+warn+fail+na+info === total and that any na check also has pass:true (so the score loop can't wrongly deduct). Frontend: render the live page in headless Chromium (Playwright), read the DOM, and assert rendered .check-row count equals the summary total, with no orphan rows, then cross-check API total === DOM rows.
3 · Run 1, 100 sites, one real fault
# 99/100 clean. The one failure: irs.gov: 74 rows vs summary 54 # 20 phantom rows # cause: the frontend injected one .check-row per NVD CVE match # (irs.gov matched 20 CVEs) -> visible but NOT in the counted set. # backend was correct: cve_none already scores the finding ONCE (weight 20).
4 · The fix
Rewrote the CVE render: each CVE now emits a .cve-detail-row (not a .check-row), fully HTML-escaped (NVD text is external), shown as detail beneath the single counted cve_none check. All CVEs stay visible; the check count is unaffected.
# re-verified live on irs.gov, headless browser: check-rows: 54 | cve-detail-rows: 20 | summary: 54 checks · 19 passed · 13 failed · 9 warnings · 7 N/A · 6 info
5 · Run 2, a different 100 sites, both layers
# fresh list: international, WordPress/CMS news, e-comm, dev/SaaS, forums, gov # each site: API reconcile + headless render + cross-check (api_total == dom_rows) === 100 tested, 100 fully clean (backend+frontend), 0 with issues === # CVE fix confirmed on new sites: techcrunch.com API:54/YES DOM:54(+20cve)/YES # 20 CVEs, still reconciles engadget.com API:54/YES DOM:54(+6cve)/YES
6 · Outcome
Two independent 100-site validations (200 unique sites), both API and rendered DOM checked on each, all clean; the single fault found in run 1 was fixed and re-verified in run 2 on the exact class of site (CVE-matching) that had triggered it. Calibration held across both populations (top scores gov/security-focused sites, bottom the header-bare giants), so the expanded tool is broad, consistent, and not overfit.
Key points
- Expanded 44 → 54 checks; added an
infostatus so optional/weight-0 checks never render as a red fail. - Tested both layers: API reconciliation (
pass+warn+fail+na+info===total,naimpliespass) and rendered DOM (rows === summary, no orphans, API total === DOM rows). - Run 1 (100 sites): 99 clean;
irs.govexposed a CVE-injection display fault (20 phantom.check-rows), backend was correct. - Fix: CVEs render as escaped
.cve-detail-rowdetail under the single countedcve_nonecheck, fully visible, count unaffected. - Run 2 (a different 100 sites): 100/100 clean both layers; CVE-matching sites (techcrunch +20, engadget +6) confirmed the fix. 200 unique sites total.
How to do this yourself
# 1. assert the backend reconciles, per site: # pass + warn + fail + na + info === total, and every na check is also pass # 2. render the real page headless and assert the DOM matches: $ node -e "const{chromium}=require('playwright'); ..." # rows === summary total, no orphans # 3. cross-check api_total === dom_rows, and run a SECOND, different list to rule out overfit
Questions & answers
Why does na have to imply pass:true? The score loop only skips a check when pass is true. An na ("not applicable") check that wasn't also pass:true would be silently deducted from the score, so every na must carry pass:true to be score-safe.
Why render in a real browser rather than just checking the API? The CVE fault was invisible at the API level, the JSON was correct. It only appeared in the rendered DOM, where extra rows were injected. Testing the actual rendered page is the only way to catch that class of bug.
Why is a weight-0 "info" check not a failure? Because the audit deducts by real risk. Genuinely optional headers (cross-origin isolation, DNS prefetch control) aren't vulnerabilities, so their absence is shown as neutral information, never a red fail, on any site scanned.
What does the second run prove that the first didn't? That the tool generalises. Passing a fresh, deliberately varied 100 sites, including the CMS/CVE-heavy kind that caught us, shows the result isn't tuned to one list and the fix works in the wild, not just on the site where the bug was found.