Wiki Extraction Quality

When to Use

• After running a wiki ingestion (youtube-content pipeline or similar) and the user reports issues
• When wikilinks are broken — pages reference concept files that don't exist on disk
• When the user wants a full delete-and-redo of an extraction (unhappy with quality)
• As a proactive quality check after any batch of concept pages is created

Broken Wikilink Detection

The most common quality issue: a concept page uses [[concepts/some-name]] but the actual file on disk is concepts/different-name.md. This happens because subagents choose filenames independently and don't cross-check.

Diagnosis

When the user reports a page with broken links, find the source page and enumerate its wikilinks:

search_files(path="WIKI_PATH", pattern="page-name", target="content")

Then for each [[concepts/...]] link found, verify the file exists:

search_files(path="WIKI_PATH/concepts", pattern="concept-slug", target="files")

Root Cause

The concepts subagent writes files under one set of names but writes wikilinks under different names in related pages. This is a self-consistency failure within a single subagent's output — not a cross-subagent coordination problem.

Fix

Either approach works:

1. Rename the file to match the wikilink (cleaner, preserves all inbound links)

2. Patch the wikilink to match the existing filename

Prefer renaming when the wikilink name is the better/canonical name. Prefer patching when the existing filename is more descriptive.

Delete-and-Redo Workflow

When the user is unsatisfied with the quality of an extraction and wants a full redo from scratch:

Step 1: Map the blast radius

Search for all files that source from the transcript:

search_files(path="WIKI_PATH", pattern="transcript-slug", target="content")

This returns: raw transcript, entity page, highlights page, and all concept pages.

Step 2: Check for cross-references

Verify whether any concepts being deleted are referenced by OTHER pages (not from this same transcript). If a concept is shared across extractions, preserve it and only update the pages from this extraction.

Step 3: Delete files

Remove all files from the extraction: transcript, entity, highlights, concepts.

Step 4: Clean navigation

Remove entries from index.md and log.md for all deleted pages.

Step 5: Re-extract

Run the full pipeline fresh. The raw transcript must be re-fetched since it was deleted.

Step 6: Post-extraction verification

After the new extraction completes:

• Spot-check 1-2 concept pages for broken wikilinks
• Verify the entity page links resolve
• Check the highlights page cross-references
• Run a final wikilink audit on any page that cross-references other concept pages

Proactive Quality Check

After any batch extraction, run this quick audit:

# For a page expected to have many concept wikilinks (like a synthesis page):
1. Extract all [[concepts/...]] links
2. Check each one exists on disk
3. Report any broken links

This catches the inconsistency before the user finds it.

Pitfalls

• Two log.md files. The wiki root has log.md (all actions) and each subfolder (e.g., consume/) has its own log.md. The youtube-content template says to prepend to WIKI_PATH/log.md but the wiki root index and root log both live at the wiki root (/opt/data/wiki/), not under consume/. Always update both: WIKI_PATH/../log.md (root) and WIKI_PATH/log.md (folder-level).
• Don't trust subagent self-reports. Subagents claim "created 17 concept pages" but may have used different filenames in their wikilinks. Always spot-check.
• Don't delete shared concepts. Some concepts appear across multiple extractions (e.g., "cognitive-surrender" appears in both Tony Fadell and other transcripts). Check cross-references before deleting.
• Re-fetch the transcript. After deleting, you must re-fetch from YouTube — the raw transcript file is gone.
• Clean Mnemosyne too. Old wiki snapshots and ingestion triples become stale after deletion. Clean them up so future deduplication checks work correctly.