Back to catalog
Skills / Software Development / hermes-weekly-audit

hermes-weekly-audit

Use for weekly Hermes config audits. Archive to wiki.

Category 💻 Software Development
Version v1.1.0
hermesauditweeklyconfigurationhealth-check

Hermes Weekly Audit

Run a structured health check of the Hermes installation and archive the findings to the wiki.

When to Use

  • Joseph asks for a "Hermes audit", "setup audit", "config check", or "health check"
  • Weekly recurring review (runs as the auto audit cron, Sun 11:30am IST)
  • After any major config change, provider switch, or Hermes upgrade

Output Location

/opt/data/wiki/produce/hermes/hermes-audit/YYYY-MM-DD.md

Create the directory if it doesn't exist. Use today's date in ISO format.

Standard Prompt Template

Use this as the default audit prompt unless Joseph provides a custom one. Quote it in the Original Prompt section of the report.

Audit my Hermes setup for anything that is missing, misconfigured, or unnecessarily limiting the quality and reliability of your work.

Compare against the previous audit in wiki/produce/hermes/hermes-audit/ and highlight:
- New issues since last week
- Issues resolved since last week
- Issues persisting week-over-week

Inspect the current configuration, enabled tools, credentials, integrations, environment variables, scheduled tasks, and recent logs or recurring errors.

Look especially for:
- Important capabilities that are unavailable or silently falling back because they are not configured
- Tools that technically work but are using a weaker default when a better option is available
- Missing API keys, dependencies, permissions, browser settings, or search/extraction providers
- Repeated errors, failed tool calls, timeouts, or unnecessary friction in recent runs
- Basic configuration changes that would materially improve research, browser use, coding, file handling, memory, or automation
- Silent quality degradations: fallback models in use, cache hit rates dropping, compression triggering early, token costs trending up
- Tools that are configured but unused in the past 7 days — flag unused spend or missed capability
- Patterns in your own recent sessions: repeated tool failures, approval loops, clarification requests, multi-retry tasks

Give me a prioritized list divided into:
1. Fix now — meaningful impact or broken capability
2. Worth improving — likely to produce better results
3. Optional — useful, but probably not worth the complexity yet
4. Recently fixed or confirmed healthy — previously problematic items now verified working

For every recommendation, explain the practical benefit, show the evidence that led you to it, and give me the exact steps or commands needed to fix it.

Do not change anything automatically. Audit first and let me review the recommendations.

Audit Checklist

Run these in order. Batch independent calls.

0. Load Previous Audit (delta detection)

ls -t /opt/data/wiki/produce/hermes/hermes-audit/*.md | head -2

Read the most recent prior audit. Extract:

  • List of issues from sections 1-3 (Fix Now / Worth Improving / Optional)
  • Note their status this week: resolved / persisting / new

1. Version & Doctor

/opt/hermes/.venv/bin/hermes --version
/opt/hermes/.venv/bin/hermes doctor

2. Config Snapshot

Read /opt/data/config.yaml — look for:

  • default model validity (check against recent agent.log for actual model used)
  • fallback_providers chain — any models that 403/401?
  • web.backend / search_backend / extract_backend — empty means web tools disabled
  • agent.reasoning_effort, verify_on_stop, compression settings
  • auxiliary.* models — any paid OpenRouter models in fallback lane?

3. Environment & Credentials

# Key names only — never print values
grep -oE '^[A-Z_]+=' /opt/data/.env | sort
env | grep -iE 'API_KEY|TOKEN|SECRET' | sed 's/=./=**/' | sort

4. Recent Errors

# Error patterns and counts
grep -oE 'ERROR [a-z_.]+' /opt/data/logs/errors.log | sort | uniq -c | sort -rn | head -15

Last 24h summarized

grep "ERROR" /opt/data/logs/errors.log | tail -50 | grep -oE '(summary=[^,]+|Error code: [0-9]+ - .{0,80})' | sort | uniq -c | sort -rn | head -20

5. Tool Availability

grep "check_fn.*returned False" /opt/data/logs/agent.log | tail -30 | sort -u

Key gates to check:

  • check_web_api_key — web_search / web_extract
  • _browser_cdp_check — browser CDP tools
  • check_bfl_requirements — Flux image gen
  • check_computer_use_requirements — desktop automation

6. Cron & Scheduled Jobs

cat /opt/data/cron/jobs.json
ls /opt/data/cron/output/

7. Model Usage & Silent Degradation

python3 -c "
import sqlite3
conn = sqlite3.connect('/opt/data/state.db')
rows = conn.execute('SELECT model, COUNT(*) as calls, SUM(total_tokens) FROM session_model_usage GROUP BY model ORDER BY calls DESC LIMIT 10').fetchall()
for r in rows: print(r)
"

Check for:

  • Fallback models serving instead of default
  • Cache hit rates (look for cache=NNN/NNN (NN%) in agent.log)
  • Compression frequency vs. previous weeks
  • Token cost trend (compare session_model_usage totals)

7b. Memory Audit (file + sqlite)

Audit BOTH memory stores for health, staleness, bloat, and duplicates.

File memory (legacy memory tool):

ls -la /opt/data/memories/MEMORY.md /opt/data/memories/USER.md
wc -c /opt/data/memories/MEMORY.md /opt/data/memories/USER.md

Compare MEMORY.md char count against memory.memory_char_limit in config.yaml (currently 2200). Flag entries near/over capacity (>90%). Read the file and flag: duplicate entries, stale facts (superseded decisions, outdated numbers), and verbose prose that could be compacted or moved to wiki/skills.

Sqlite memory (Mnemosyne primary + session store):

ls -la /opt/data/mnemosyne/data/mnemosyne.db /opt/data/state.db
python3 -c "import sqlite3; c=sqlite3.connect('/opt/data/mnemosyne/data/mnemosyne.db'); print(c.execute('PRAGMA integrity_check').fetchall()); print(c.execute('SELECT COUNT(*) FROM memory_working_memory').fetchone())" 2>/dev/null

Check: DB file sizes growing out of control, PRAGMA integrity_check failures, orphaned/invalid rows, and recall health (run mnemosyne_recall_diagnostics / mnemosyne_stats if available). Flag oversized DBs, integrity failures, or if Mnemosyne and file memory have diverged on the same facts.

Flag cross-store inconsistencies in the report and recommend consolidation (e.g. mnemosyne_sleep, or folding file-memory entries into Mnemosyne and freeing MEMORY.md).

8. Capability Utilization

Identify configured-but-unused tools:

  • Cross-reference tool availability (step 5) with actual tool calls in recent sessions
  • Flag API keys with zero usage in 7 days
  • Flag available tools never invoked

9. Operational Friction

Reflect on recent sessions:

  • Repeated tool failures for the same operation
  • Approval loops (same approval requested multiple times)
  • Clarification requests that config could eliminate
  • Tasks requiring multiple retries

10. Slack/Platform Health

grep -iE "channel_not_found|SlackApiError|Send error" /opt/data/logs/errors.log | tail -10

Report Format

Start with an Original Prompt section quoting the exact request that triggered the audit. This preserves context for future reference.

Then add a Delta Summary section:

  • 🆕 New issues this week
  • ✅ Resolved since last week
  • ⏳ Persisting week-over-week

Then use the 4-bucket structure:

1. Fix Now — broken capability, meaningful impact

2. Worth Improving — better results, low effort

3. Optional — useful but not urgent

4. Recently Fixed or Confirmed Healthy — previously problematic items now verified working (do NOT list stable configs that haven't changed)

For each finding:

  • Evidence — log line, config key, or doctor output
  • Practical benefit — what improves
  • Fix — exact command(s)

End with a Priority Order list.

Pitfalls

  • Never print API key values — names only, sed 's/=./=**/'
  • Don't run hermes doctor --fix without explicit approval — audit only, no changes
  • Don't restart the gateway — the audit runs inside the gateway; restarting kills the session
  • Check agent.log for actual model used — config default may be silently falling back
  • check_web_api_key false means web tools are gone — even if plugins register, the gate requires a backend config or legacy key
  • Always diff against previous audit — the delta is the most valuable signal in a recurring audit
  • Skip "Already configured well" boilerplate — only include items that were previously broken or at risk

After the Audit

Present the report to Joseph. Do NOT execute fixes without explicit go-ahead. If he approves, apply fixes one at a time and verify each.

Update the folder README

Write (or overwrite) /opt/data/wiki/produce/hermes/hermes-audit/README.md with the current audit system documentation. This keeps the folder self-describing — anyone opening it sees what the audit covers, when it runs, and how to read the reports. Regenerate from the skill contents each time so it stays current if the skill changes. Include: schedule, delivery, output format, checklist sections (including 7b memory audit), conventions (no auto-fixes, no API key values), and the memory audit specifics (file + sqlite paths).