analyzing-skill-usage

Auto-invocation: Your coding assistant will automatically invoke this skill when it detects a matching trigger.

Use when evaluating skill effectiveness or comparing skill versions. Triggers: 'how are skills performing', 'skill metrics', 'which skills fire correctly', 'skill invocation analysis', 'compare skill versions', 'analyze skill usage'. NOT for: authoring or editing a skill (use writing-skills).

Skill Content

# Analyzing Skill Usage

<ROLE>Skill Performance Analyst. You parse session transcripts, extract skill usage events, score each invocation, and produce comparative metrics. Your analysis drives skill improvement decisions. Scores derive from observable events — never speculation.</ROLE>

<analysis>Before analysis: clarify session scope, skills of interest, and comparison criteria.</analysis>
<reflection>After analysis: summarize patterns observed, statistical confidence, and actionable findings.</reflection>

## Invariant Principles

1. **Evidence Over Intuition**: Scores derive from observable session events, not speculation
2. **Context Matters**: Correction after skill completion differs from mid-workflow abandonment
3. **Version Awareness**: Track skill variants for A/B comparison when version markers present
4. **Statistical Humility**: Small sample sizes warrant tentative conclusions

## Inputs / Outputs

| Input | Required | Description |
|-------|----------|-------------|
| `session_paths` | No | Specific sessions (defaults to recent project sessions) |
| `skills` | No | Filter to specific skills (defaults to all) |
| `compare_versions` | No | If true, group by version markers for A/B analysis |

| Output | Description |
|--------|-------------|
| `skill_report` | Per-skill metrics: invocations, completion rate, correction rate, avg tokens |
| `weak_skills` | Skills ranked by failure indicators |
| `version_comparison` | A/B results when versions detected |

---

## Extraction Protocol

### 1. Load Sessions

There is no spellbook library for this. Transcripts are plain JSONL on disk and
you parse them yourself — one JSON object per line, in message order:

```python
import json
from pathlib import Path

session_dir = Path.home() / ".claude" / "projects" / project_encoded
for path in sorted(session_dir.glob("*.jsonl")):
    messages = [json.loads(line) for line in path.read_text().splitlines() if line.strip()]
```

Harnesses other than Claude Code store transcripts elsewhere; locate the
directory for the harness under analysis before globbing. A skipped or
unreadable file is a gap in the sample, not a zero — record it and say so in the
report rather than letting it silently shrink the denominator.

### 2. Detect Skill Invocation Boundaries

**Start Event**: an assistant tool-use block where the tool `name == "Skill"`.
Walk `messages` in order and record, for each start event, the skill name (strip
any version marker), the message index, and the timestamp. Scan the assistant
message `content` list for entries whose `type == "tool_use"`.

**End Event** (first match): another Skill tool call (superseded), session end, or compact boundary (`type == "system"`, `subtype == "compact_boundary"`)

### 3. Score Each Invocation

**Success Signals** (+1 each):
- No user correction in skill window
- Skill ran to natural completion (not superseded)
- Artifact produced (Write/Edit tool after skill)
- User continued to new topic

**Failure Signals** (-1 each):
- User correction detected
- Same skill re-invoked within 5 messages (retry)
- Different skill invoked for apparent same task
- Skill abandoned mid-workflow (superseded without output)

**Correction Detection Patterns**:
```python
CORRECTION_PATTERNS = [
    r"\bno\b(?!t)",           # "no" but not "not"
    r"\bstop\b",
    r"\bwrong\b",
    r"\bactually\b",
    r"\bdon'?t\b",
    r"\binstead\b",
    r"\bthat'?s not\b",
]
```

### 4. Aggregate Metrics

Per skill, produce:
```python
{
    "skill": "develop",
    "version": "v1" | None,      # If version marker detected
    "invocations": 15,
    "completions": 12,           # Ran to end without supersede
    "corrections": 3,            # User corrected during
    "retries": 1,                # Same skill re-invoked
    "avg_tokens": 4500,          # Tokens in skill window
    "completion_rate": 0.80,
    "correction_rate": 0.20,
    "score": 0.60,               # Composite score
}
```

---

## Analysis Modes

### Mode 1: Identify Weak Skills

Rank all skills by composite failure score:

```
failure_score = (corrections + retries + abandonments) / invocations
```

Output format:
```markdown
## Weak Skills Report
| Rank | Skill | Invocations | Failure Rate | Top Failure Mode |
|------|-------|-------------|--------------|------------------|
| 1 | gathering-requirements | 8 | 0.50 | User corrections |
```

### Mode 2: A/B Testing Versions

When version markers detected (e.g., `skill:v2` or tagged in args):
```markdown
## A/B Comparison: develop
| Metric | v1 (n=10) | v2 (n=8) | Delta | Significant |
|--------|-----------|----------|-------|-------------|
| Completion Rate | 0.70 | 0.88 | +0.18 | Yes (p<0.05) |
| Correction Rate | 0.30 | 0.12 | -0.18 | Yes |
| Avg Tokens | 5200 | 4100 | -1100 | Yes |

**Recommendation**: v2 outperforms v1 across all metrics.
```

---

## Execution Steps

1. Enumerate sessions in target scope
2. Parse each session, extracting skill events
3. Score each invocation using signal detection
4. Aggregate by skill (and version if A/B)
5. Rank and report based on analysis mode
6. Surface actionable insights for skill improvement

---

## Version Detection

Look for version markers: skill name suffix (`develop:v2`), args containing version (`"--version v2"`, `"[v2]"`), or session date ranges.

<CRITICAL>
When comparing versions, require:
- Minimum 5 invocations per variant
- Similar task complexity (manual review recommended)
- Same time period when possible (avoid confounds)
</CRITICAL>

---

<FORBIDDEN>
- Drawing conclusions from <5 invocations
- Ignoring context (correction after success ≠ failure)
- Conflating skill issues with user errors
- Reporting without confidence intervals on small samples
</FORBIDDEN>

## Self-Check

- [ ] Sessions loaded and parsed successfully
- [ ] Skill invocation boundaries correctly identified
- [ ] Correction patterns detected in user messages
- [ ] Metrics aggregated per skill (and version if A/B)
- [ ] Statistical caveats noted for small samples
- [ ] Actionable recommendations provided

<FINAL_EMPHASIS>Skills improve through measurement. Extract events, score honestly, compare rigorously, recommend confidently.</FINAL_EMPHASIS>