my-project-public

repository

loading code, commits, and activity

repositories

loading repo index

#1	# Clawd Memory Docs Audit - Reusable Workflow Checklist
#2
#3	This workflow audits Clawd Memory documentation against the current codebase. Historical references to Mnemosyne refer to the underlying engine or the pre-rebrand docs site.
#4
#5	Purpose: Bi-weekly cross-reference audit of docs site against codebase.
#6	Cadence: Every 2 weeks (or after any version bump / major feature merge).
#7	Audience: Hermes Agent (send this document as context to re-run the audit).
#8	Checkpoint file: `.audit-state.json` in `mnemosyne-docs` repo — tracks what was audited when.
#9
#10	---
#11
#12	## Phase 0: Load Checkpoint + Determine Scope (2 min)
#13
#14	This is the key to avoiding redundant work. The checkpoint file at `/root/.hermes/projects/mnemosyne-docs/.audit-state.json` tracks every page's last audit hash. Only re-audit pages whose content hash changed since then.
#15
#16	### Step 0.1: Load the checkpoint
#17	```python
#18	import json
#19	with open("/root/.hermes/projects/mnemosyne-docs/.audit-state.json") as f:
#20	state = json.load(f)
#21	```
#22
#23	### Step 0.2: Find changed files
#24	```bash
#25	cd /root/.hermes/projects/mnemosyne-docs
#26	# For each tracked file, compare current hash to audited hash
#27	git diff --name-only HEAD~10..HEAD -- 'content/' 'src/app/(docs)/'
#28	```
#29
#30	### Step 0.3: Build audit scope
#31
#32	Three categories:
#33
#34	\| Category \| Condition \| Action \|
#35	\|----------\|-----------\|--------\|
#36	\| Skip \| File hash matches `audit_hash` in checkpoint AND codebase version hasn't changed \| Skip entirely. This page was verified against this exact code and content. \|
#37	\| Re-audit \| File hash differs from `audit_hash` \| The page was edited since last audit. Verify it against current codebase. \|
#38	\| First audit \| File not in checkpoint at all \| Never audited. Full audit needed. \|
#39
#40	Also check:
#41	- [ ] Codebase version changed? → Re-audit ALL previously audited pages (API may have changed)
#42	- [ ] New pages added to docs site? → They won't be in checkpoint, add to audit scope
#43	- [ ] New tools/importers/features in codebase? → Check relevant doc pages exist
#44
#45	### Step 0.4: Report scope
#46	Before starting, report: "Skipping X pages (unchanged), auditing Y pages (changed), first-auditing Z pages (new)." This transparency is how you know you're not burning tokens.
#47
#48	---
#49
#50	## Phase 1: Codebase Surface Map (only if version changed)
#51
#52	Generate a fresh codebase map. This tells you what actually exists.
#53
#54	Method: Delegate to a subagent with:
#55	```
#56	Map the entire Mnemosyne codebase surface in /root/.hermes/projects/mnemosyne.
#57	Catalog every class, method signature, CLI command, configuration option,
#58	tool schema (MCP + Hermes plugin), API endpoint, and importer.
#59	Output structured JSON to mnemosyne_codebase_surface.json.
#60	```
#61
#62	Key files to check for changes since last audit:
#63	- `mnemosyne/core/beam.py` — BeamMemory (new methods, changed defaults)
#64	- `mnemosyne/core/memory.py` — Mnemosyne class (new methods, changed signatures)
#65	- `mnemosyne/mcp_tools.py` — MCP tool definitions (new tools, changed params)
#66	- `mnemosyne/core/importers/` — New import providers
#67	- Plugin yamls — `plugin.yaml`, `hermes_plugin/plugin.yaml` — tool counts, hook names
#68	- `pyproject.toml` — dependencies, entry points
#69
#70	Verify key numbers from code:
#71	- [ ] Default `top_k` in `recall()` — check `beam.py` line with `def recall`
#72	- [ ] MCP tool count — check `mcp_tools.py` for `get_tool_definitions()`
#73	- [ ] Hermes plugin tool count — check plugin.yaml `tools:` field
#74	- [ ] Hook names — check `hermes_plugin/__init__.py` for hook registration
#75	- [ ] Config env vars — check `mnemosyne/core/beam.py` and `memory.py` for `os.getenv`
#76	- [ ] config.yaml keys — check `hermes_plugin/__init__.py` for config reads
#77
#78	---
#79
#80	## Phase 2: Critical Page Audit (30 min)
#81
#82	These 10 pages are the ones most likely to rot. Check them every time.
#83
#84	### 1. `getting-started/configuration.mdx`
#85	- [ ] Every env var listed EXISTS in the actual codebase (grep for `os.getenv`)
#86	- [ ] Defaults match code defaults
#87	- [ ] Config file is correctly `config.yaml` (not `mnemosyne.yaml`)
#88	- [ ] Class name is `Mnemosyne` (not `Memory`)
#89	- [ ] Embedding model matches `MNEMOSYNE_EMBEDDING_MODEL` default
#90	- [ ] config.yaml keys match actual `memory.mnemosyne.*` structure
#91
#92	### 2. `api/python-sdk.mdx`
#93	- [ ] Constructor signature matches `Mnemosyne.__init__` exactly
#94	- [ ] `recall()` default `top_k` matches code default
#95	- [ ] All methods listed in docs exist in code (grep for `def method_name`)
#96	- [ ] All public methods from code are documented in docs
#97	- [ ] V2 Properties table matches actual properties on Mnemosyne class
#98	- [ ] Stream API methods match `MemoryStream` class
#99	- [ ] DeltaSync methods match `DeltaSync` class
#100	- [ ] Hermes Plugin Tools table lists ALL tools from plugin.yaml
#101
#102	### 3. `api/tool-schema.mdx`
#103	- [ ] Every tool definition's `required` params match MCP tool code
#104	- [ ] Every tool definition's `properties` match MCP tool code
#105	- [ ] No fictional parameters (like `tags` that was here before)
#106	- [ ] Number of tools matches actual tool count
#107
#108	### 4. `api/hermes-plugin.mdx`
#109	- [ ] Hooks table names match actual hook registration in `hermes_plugin/__init__.py`
#110	- [ ] Hook descriptions are accurate
#111	- [ ] Tool list matches plugin.yaml
#112	- [ ] No fictional configuration options
#113
#114	### 5. `api/mcp-server.mdx`
#115	- [ ] Tool names match `mcp_tools.py` exactly
#116	- [ ] Tool count matches
#117	- [ ] Transport options (stdio, SSE) match CLI
#118
#119	### 6. `architecture/beam-overview.mdx`
#120	- [ ] Number of memory tiers is correct (3: working, episodic, scratchpad)
#121	- [ ] TripleStore is correctly described as separate, not a 4th tier
#122	- [ ] Capacity numbers match env var defaults
#123	- [ ] Latency claims match benchmark data
#124
#125	### 7. `architecture/system-design.mdx`
#126	- [ ] No fictional components (like the "REST API" box that was here)
#127	- [ ] Component names match actual classes/modules
#128	- [ ] Technology stack table is accurate
#129
#130	### 8. `operations/performance.mdx`
#131	- [ ] Memory tier names match actual tables
#132	- [ ] Memory usage numbers are realistic
#133	- [ ] No fictional tiers (like "Semantic Memory")
#134	- [ ] Benchmark numbers are recent
#135
#136	### 9. `architecture/streaming.mdx`
#137	- [ ] API method names match actual `MemoryStream` class
#138	- [ ] API method names match actual `DeltaSync` class
#139	- [ ] `compute_delta()` shows required `peer_id` param
#140
#141	### 10. `architecture/plugin-system.mdx`
#142	- [ ] Registration method names match `PluginManager` class
#143	- [ ] Hook signatures match `MnemosynePlugin` abstract class
#144	- [ ] Built-in plugin names match actual plugins
#145
#146	---
#147
#148	## Phase 3: Comparison Pages (10 min)
#149
#150	These should match current version and feature set.
#151
#152	- [ ] `comparisons/*.mdx` — all say v2.5.0 (or current version)
#153	- [ ] `comparisons/*.mdx` — "Last updated" dates are recent
#154	- [ ] Tool counts referenced match actual counts
#155	- [ ] Provider counts match actual importers list
#156
#157	### Comparison pages:
#158	- [ ] `comparisons/honcho.mdx`
#159	- [ ] `comparisons/zep.mdx`
#160	- [ ] `comparisons/mem0.mdx`
#161	- [ ] `comparisons/letta.mdx`
#162	- [ ] `comparisons/cognee.mdx`
#163	- [ ] `comparisons/supermemory.mdx`
#164	- [ ] `comparisons/hindsight.mdx`
#165
#166	---
#167
#168	## Phase 4: Landing/Quick-Start Pages (5 min)
#169
#170	- [ ] `getting-started/quick-start.mdx` — version number, code snippets work
#171	- [ ] `getting-started/installation.mdx` — pip install command correct
#172	- [ ] `getting-started/first-steps.mdx` — API usage matches current signatures
#173	- [ ] `migration/overview.mdx` — provider count accurate, version current
#174
#175	---
#176
#177	## Phase 5: Fix and Commit (15-30 min)
#178
#179	### Fixing approach:
#180	1. Use patch tool for targeted edits — never rewrite entire files with sed/read_file
#181	2. Fix `content/` files first, then mirror to `src/app/(docs)/` copies
#182	3. Verify with: `grep -rn 'old_string' content/ src/` before declaring done
#183
#184	### Mirroring script:
#185	```python
#186	import os, shutil
#187	content_dir = "/root/.hermes/projects/mnemosyne-docs/content"
#188	app_dir = "/root/.hermes/projects/mnemosyne-docs/src/app/(docs)"
#189	for rel_path in modified_files:
#190	src = os.path.join(content_dir, rel_path)
#191	dir_part = os.path.dirname(rel_path)
#192	name_part = os.path.splitext(os.path.basename(rel_path))[0]
#193	dst = os.path.join(app_dir, dir_part, name_part, "page.mdx")
#194	shutil.copy2(src, dst)
#195	```
#196
#197	### Commit template:
#198	```
#199	fix(docs): bi-weekly audit — [brief summary of what changed]
#200	```
#201
#202	---
#203
#204	## Phase 7: Update Checkpoint + Report (5 min)
#205
#206	### 7.1: Update the checkpoint file
#207	After fixing everything, update `.audit-state.json`:
#208
#209	```python
#210	import json, subprocess
#211	from datetime import datetime, timezone
#212
#213	with open("/root/.hermes/projects/mnemosyne-docs/.audit-state.json") as f:
#214	state = json.load(f)
#215
#216	now = datetime.now(timezone.utc).isoformat()
#217
#218	# Update timestamps
#219	state["last_full_audit"] = now
#220	state["codebase_version"] = CURRENT_VERSION # from Phase 1
#221
#222	# Update each audited file's hash and status
#223	for filepath in audited_files:
#224	h = subprocess.check_output(["git", "ls-tree", "HEAD", filepath]).decode().split()[2]
#225	if filepath in state["files"]:
#226	state["files"][filepath]["last_audited"] = now[:10]
#227	state["files"][filepath]["audit_hash"] = h
#228	state["files"][filepath]["status"] = "clean"
#229	else:
#230	state["files"][filepath] = {
#231	"last_audited": now[:10],
#232	"audit_hash": h,
#233	"status": "clean",
#234	"category": "source" if filepath.startswith("content/") else "mirror"
#235	}
#236
#237	# Append to audit history
#238	state["audit_history"].append({
#239	"date": now[:10],
#240	"codebase_version": CURRENT_VERSION,
#241	"pages_audited": len(audited_files),
#242	"issues_found": N_ISSUES,
#243	"issues_fixed": N_FIXED,
#244	"commit": GIT_COMMIT_HASH,
#245	"summary": "Brief description of what changed"
#246	})
#247
#248	with open("/root/.hermes/projects/mnemosyne-docs/.audit-state.json", "w") as f:
#249	json.dump(state, f, indent=2)
#250	```
#251
#252	### 7.2: Write executive report
#253	- Write to `docs/audit-report-YYYY-MM-DD.md` in the main mnemosyne repo
#254	- Include: pages audited, skipped, issues found, fixes applied, remaining risks
#255	- Reference the checkpoint for full state
#256
#257	### 7.3: Commit both repos
#258	```bash
#259	cd /root/.hermes/projects/mnemosyne-docs
#260	git add .audit-state.json && git commit -m "chore: update audit checkpoint [date]"
#261	git push
#262
#263	cd /root/.hermes/projects/mnemosyne
#264	git add docs/audit-report-*.md docs/audit-workflow.md && git commit -m "docs: audit report [date]"
#265	git push
#266	```
#267
#268	---
#269
#270	## Checkpoint File Schema
#271
#272	The `.audit-state.json` file follows this structure:
#273
#274	```json
#275	{
#276	"_schema": "mnemosyne-docs-audit-checkpoint-v1",
#277	"last_full_audit": "ISO timestamp",
#278	"codebase_version": "2.5.0",
#279	"audit_history": [
#280	{
#281	"date": "YYYY-MM-DD",
#282	"codebase_version": "X.Y.Z",
#283	"pages_audited": N,
#284	"issues_found": N,
#285	"issues_fixed": N,
#286	"commit": "git hash",
#287	"summary": "text"
#288	}
#289	],
#290	"files": {
#291	"content/path/to/page.mdx": {
#292	"last_audited": "YYYY-MM-DD",
#293	"audit_hash": "git blob hash",
#294	"status": "clean\|issues_pending",
#295	"category": "source\|mirror",
#296	"note": "optional context"
#297	}
#298	}
#299	}
#300	```
#301
#302	Key invariant: A page's `audit_hash` is the git blob hash of the file AT THE TIME of the audit. On next audit, compare current blob hash to `audit_hash`. If they match and codebase version hasn't changed, skip the page.
#303
#304	---
#305
#306	## Edge Cases and Failure Modes
#307
#308	The checkpoint system handles these edge cases automatically. When any are detected during Phase 0, the health check reports them and adjusts the audit scope.
#309
#310	### 1. Git History Rewrite (Rebase/Squash)
#311	What happens: All git blob hashes change simultaneously after a rebase or squash merge.
#312	Detection: If ALL tracked file hashes differ from audited hashes at once, the checkpoint flags `_rebase_detected: true`.
#313	Action: Full re-audit of all pages. The old audit data is preserved as historical context but all pages are re-verified.
#314
#315	### 2. Checkpoint File Missing or Corrupted
#316	What happens: `.audit-state.json` is deleted, has malformed JSON, or is missing required fields.
#317	Detection: JSON parse fails or `_schema` field is missing/wrong version.
#318	Action: Full audit of all pages. New checkpoint generated from scratch.
#319
#320	### 3. Page Renamed or Moved
#321	What happens: A file is renamed (e.g., `old-name.mdx` → `new-name.mdx`). The old path disappears and a new path appears.
#322	Detection: Old path shows `status: gone`. New path shows `status: unaudited`.
#323	Action: The old entry is kept with `status: gone` for historical tracking. The new path gets a full audit. If git detected the rename, the agent can optionally carry forward the audit status.
#324
#325	### 4. Mirror Drift (Source and Mirror Out of Sync)
#326	What happens: Someone edits `content/foo.mdx` but forgets to sync `src/app/(docs)/foo/page.mdx`. The hashes diverge.
#327	Detection: Mirror hash differs from its source hash while both are marked `clean`.
#328	Action: Mark mirror as `status: drifted`. Sync it from source before auditing. The drifted status prevents skipping a stale mirror.
#329
#330	### 5. Codebase Version Bump
#331	What happens: Mnemosyne upgrades from v2.5.0 to v2.6.0. Doc pages were audited against v2.5.0.
#332	Detection: `codebase_version` in checkpoint differs from current `mnemosyne.__version__`.
#333	Action: All source pages marked `status: stale_version`. They need re-verification even if content hasn't changed — API signatures, tool counts, or config keys may have changed.
#334
#335	### 6. Concurrent Edits During Audit
#336	What happens: Someone pushes a doc change while the audit is running.
#337	Detection: At end of Phase 5 (before commit), re-check git hashes of all audited files. If any differ from what was audited, flag as `possibly_stale`.
#338	Action: Re-audit those specific files before finalizing the checkpoint.
#339
#340	### 7. Partial Audits (Only Some Pages Audited This Round)
#341	What happens: User requests audit of only specific pages, not the full site.
#342	Detection: Some pages have newer `last_audited` dates than others.
#343	Action: The checkpoint tracks per-page status independently. Each page's `audit_hash` is its own truth. No global consistency issue — this is by design.
#344
#345	### 8. Pages Deleted from Docs Site
#346	What happens: A page is removed from the repo entirely.
#347	Detection: File path exists in checkpoint but not in current `git ls-tree`.
#348	Action: Mark as `status: gone` with a note about when it disappeared. Kept for audit trail. Remove from checkpoint after 3 audit cycles (manual cleanup or automatic).
#349
#350	### 9. Untracked Pages (Never Audited)
#351	What happens: Pages exist in the repo but were never added to the checkpoint.
#352	Detection: File exists in `git ls-tree` but not in checkpoint's `files` dict.
#353	Action: Added with `status: unaudited`. Prioritized for audit on next run. The health check reports how many remain.
#354
#355	### 10. Agent Fixes Introduce New Issues
#356	What happens: During Phase 4, fixing one discrepancy accidentally breaks something else (e.g., wrong method name propagated to multiple places).
#357	Detection: Phase 5 verification step — after all fixes, re-run key cross-reference checks on the fixed pages.
#358	Action: If new issues found, fix them in the same audit cycle. The checkpoint only gets updated after all issues are resolved.
#359
#360	---
#361
#362	## Phase 6: Website Cross-Check (5 min)
#363
#364	- [ ] `mnemosyne-website/src/components/HomePage.tsx` — BEAM labels still correct
#365	- [ ] `mnemosyne-website/src/data/changelog.json` — last sync date is recent
#366	- [ ] Website version matches codebase version
#367
#368	---
#369
#370	## Pain Points Log (Lessons Learned)
#371
#372	### From May 11, 2026 Audit:
#373
#374	#### Tooling Pain Points
#375
#376	1. Never use sed with special characters in search strings. The `\|` in markdown tables conflicts with sed's `\|` delimiter. Parentheses in file paths like `src/app/(docs)/` break shell parsing. Solution: Use Python's `str.replace()` for multi-file batch edits, or the `patch` tool for single-file targeted edits. `patch` does fuzzy matching and handles special characters natively.
#377
#378	2. The patch tool is the safest edit method. It does fuzzy matching (9 strategies) and won't corrupt files. `sed` can silently fail or corrupt files when special characters, newlines, or unicode are involved. Rule: Always reach for `patch` first, Python `str.replace` second, `sed` never.
#379
#380	3. Mirror files are a trap. `content/` and `src/app/(docs)/` are separate copies with no build-time sync. If you fix one and not the other, the live site shows stale content. 4 mirrors were found drifted during this audit. Solution: Always sync mirrors after content edits. Checkpoint now detects mirror drift automatically.
#381
#382	#### Process Pain Points
#383
#384	4. The configuration page was the worst rot. It had zero correspondence with actual code — fictional env vars (`MNEMOSYNE_DB_PATH`), wrong class name (`Memory`), wrong config file (`mnemosyne.yaml`), wrong embedding model (`text-embedding-3-small`). Root cause: Config systems have no type checking and vary across environments. They're the hardest to keep in sync. Solution: The audit now prioritizes config pages. Verify every env var name against `grep os.getenv` in source.
#385
#386	5. Subagent timeouts on large scans. A single subagent cataloging 67 pages timed out at 600 seconds (18 API calls). Solution: Break work into chunks — one subagent for codebase mapping, separate subagents for page groups of ~15-20 pages each. 3 concurrent max.
#387
#388	6. Don't assume documentation is accurate. Some pages (configuration, system design) were clearly generated from assumptions, not from reading source code. Solution: Always verify claims against source code, not against other documentation pages. Trust `grep` over prose.
#389
#390	7. subagent `read_file` can drop data. When subagents read files with `read_file` and rewrite with `write_file`/`open()`, frontmatter export blocks and metadata can be lost. Solution: Use the `patch` tool for all edits. For full rewrites, verify content integrity by checking line counts before and after.
#391
#392	8. The checkpoint file eliminates redundant work. Without it, every audit is a full scan of 67+ pages (~2 hours, ~$3 in tokens). With it, only changed pages get re-audited. Solution: The `.audit-state.json` checkpoint stores per-page git blob hashes. Phase 0 diffs and skips unchanged pages.
#393
#394	9. Shell escaping in execute_code is fragile. Using `terminal()` from inside `execute_code` to run sed commands with dynamic strings caused repeated failures from unescaped characters. Solution: When inside `execute_code`, use Python's native `open()/read()/write()` for file operations. Reserve `terminal()` for git commands and system calls.
#395
#396	10. Gitignored directories block saving artifacts. The `.planning/` directory was gitignored in mnemosyne-docs, so the audit report couldn't be committed there. Solution: Save reports to the main mnemosyne repo's `docs/` directory, which is tracked.
#397
#398	11. Version drift across comparison pages. All 8 comparison pages referenced v2.3.0 while codebase was v2.5.0. These are high-visibility marketing pages. Solution: Comparison pages are now part of the critical audit list, checked every cycle.
#399
#400	12. The docs-mapping subagent timed out. 18 API calls across 67 pages in 600s was too much. Solution: For Phase 2 (page cataloging), split into 4 subagents by section (API pages, Architecture pages, Comparison pages, Everything else). Each gets ~15-20 pages.
#401
#402	#### Fixes-Introducing-Bugs Pain Points
#403
#404	13. sed corrupted comparison files. Running sed on files with pipe characters in table rows produced massive diffs (1355 insertions, 1361 deletions) from line ending changes. Required `git checkout` to restore and redo with `patch`. Solution: Never use sed for markdown files. Use `patch` tool exclusively.
#405
#406	14. Mirror drift went undetected for 4 files. During the main audit pass, `content/` pages were fixed but their `src/app/(docs)/` mirrors weren't synced. Caught during edge case hardening. Solution: The mirror sync step is now mandatory between Phase 4 and Phase 5.
#407
#408	---
#409
#410	To re-run this audit: send this document to Hermes Agent with the message "Run the bi-weekly docs audit using AUDIT-WORKFLOW.md"
#411

z6Mkq5mY3JWtxoxUobWcfNHm7AkRubgSWEZTkBVqZXJviFZ5/my-project-public