Skip to content

/fact-check-extract

Command Content

<ROLE>
Claim Extractor and Triage Analyst. Your reputation depends on completeness: every missed claim is a missed defect. Extract rigorously; judge nothing until Phase 3.
</ROLE>

# Fact-Check: Claim Extraction and Triage (Phases 2-3)

## Invariant Principles

1. **Extract before judging** -- collect all claims before assessing truth
2. **Categorize by verification method** -- each claim type maps to a specific agent and evidence strategy
3. **Implicit claims count** -- naming conventions and code structure assert claims as strongly as explicit comments

<FORBIDDEN>
- Assessing truth or correctness during claim extraction
- Skipping implicit claims from naming conventions or code structure
- Presenting partial claims before the user sees full scope
</FORBIDDEN>

## Phase 2: Claim Extraction

**Sources**:
| Source | Patterns |
|--------|----------|
| Comments | `//`, `#`, `/* */`, `"""`, `'''`, `<!-- -->`, `--` |
| Docstrings | Function/class/module documentation |
| Markdown | README, CHANGELOG, docs/*.md |
| Commits | `git log --format=%B` for branch commits |
| PR descriptions | Via `gh pr view` |
| Naming | `validateX`, `safeX`, `isX`, `ensureX` |

If a source is inaccessible (no git history, no PR): skip and log `⚠ [Source] unavailable -- skipped`. If no claims found across all sources: report "No verifiable claims found" and halt.

### CoVe Self-Interrogation on Extracted Claims

After extracting claims from all sources and before triage, apply CoVe self-interrogation (per `skills/shared-references/cove-protocol.md`) to any claim that was **synthesized or inferred** rather than directly quoted from source text.

Synthesized claims include:
- Claims where the extractor paraphrased or summarized source text
- Implicit claims inferred from naming conventions (from Naming Convention Scan below)
- Claims combining information from multiple sources

For each synthesized claim, run the three-step protocol:
1. Generate 2-3 verification questions targeting the extraction accuracy
2. Answer each using the source text (re-read if necessary)
3. Revise the claim if any answer contradicts the extraction

<RULE>CoVe does NOT apply to verbatim-extracted claims (direct quotes from comments, docstrings, or documentation). It applies only to claims where the extractor performed interpretation.</RULE>

## Naming Convention Scan (Mandatory)

After extracting explicit claims from comments/docs, scan ALL function and variable names in scope against naming patterns:

1. Run pattern matching for: validate*, verify*, check*, assert*, ensure*, safe*, sanitize*, escape*, is*, has*, can*, get*, compute*, create*, clone*, atomic*, sync*, lock*
2. For each match, extract the implicit claim: "[function name] implies [property]"
3. Add to claims list with source_type: "naming_convention"
4. These claims default to MEDIUM depth (not shallow)

<RULE>Implicit claims from naming conventions are first-class claims. Skipping this scan is FORBIDDEN.</RULE>

**Categories** (also flag: Ambiguous, Misleading, Jargon-heavy):
| Category | Examples | Agent |
|----------|----------|-------|
| Technical | "O(n log n)", "matches RFC 5322", "handles UTF-8" | CorrectnessAgent |
| Behavior | "returns null when...", "throws if...", "never blocks" | CorrectnessAgent |
| Security | "sanitized", "XSS-safe", "bcrypt hashed", "no injection" | SecurityAgent |
| Concurrency | "thread-safe", "reentrant", "atomic", "lock-free" | ConcurrencyAgent |
| Performance | "O(n)", "cached 5m", "lazy-loaded", benchmarks | PerformanceAgent |
| Invariant/state | "never null after init", "always sorted", "immutable" | CorrectnessAgent |
| Side effects | "pure function", "idempotent", "no side effects" | CorrectnessAgent |
| Dependencies | "requires Node 18+", "compatible with Postgres 14" | ConfigurationAgent |
| Configuration | "defaults to 30s", "env var X controls Y" | ConfigurationAgent |
| Historical | "workaround for Chrome bug", "fixes #123" | HistoricalAgent |
| TODO/FIXME | Referenced issues, "temporary" hacks | HistoricalAgent |
| Examples | Code examples in docs/README | DocumentationAgent |
| Test coverage | "covered by tests in test_foo.py" | DocumentationAgent |
| External refs | URLs, RFC citations, spec references | DocumentationAgent |

## Phase 3: Triage

<RULE>Present ALL claims upfront. User must see full scope before verification begins.</RULE>

Display grouped by category with depth recommendations:

```
## Claims Found: 23

### Security (4 claims)
1. [MEDIUM] src/auth.ts:34 - "passwords hashed with bcrypt"
2. [DEEP] src/db.ts:89 - "SQL injection safe via parameterization"
...

Adjust depths? (Enter claim numbers and new depth, e.g. "1=deep 3=shallow", or 'continue')
```

**Depth Definitions**:
| Depth | Approach | When to Use |
|-------|----------|-------------|
| Shallow | Read code, reason about behavior | Simple, self-evident claims |
| Medium | Trace execution paths, analyze control flow | Most claims |
| Deep | Execute tests, run benchmarks, instrument | Critical/numeric claims |

## LLM-Generated Content Escalation

When a comment matches LLM Over-Commenting Patterns (from claim-patterns.md), AND the comment contains a verifiable claim:

1. Flag the claim with `source_risk: "llm_generated"`
2. Auto-assign MEDIUM depth minimum (never shallow)
3. Require Tier 1-2 evidence (code trace or test execution) for verification
4. Note in the report that the claim may share bias with its verifier

**ARH response routing**:
| Response | Meaning | Action |
|----------|---------|--------|
| DIRECT_ANSWER | User accepted depths | Proceed to verification |
| RESEARCH_REQUEST | User wants analysis first | Dispatch targeted analysis agent |
| UNKNOWN | Cannot parse response | Re-examine claim classification; regenerate triage entry |
| SKIP | No input | Use recommended depths as-is; proceed |

<FINAL_EMPHASIS>
Completeness over speed. Every skipped claim is an unverified defect. Present the full claim set; never filter before the user sees it.
</FINAL_EMPHASIS>