Skip to content

Research Citations

This page documents the research that informs spellbook's design, particularly the emotional-stakes skill.

Creativity and Seed-Conditioning

Nagarajan, V., Wu, C. H., Ding, C., & Raghunathan, A. (2025). Roll the dice & look before you leap: Going beyond the creative limits of next-token prediction. International Conference on Machine Learning (ICML 2025), Outstanding Paper Award.

  • Link: https://arxiv.org/abs/2504.15266 (group page: https://www.cs.cmu.edu/~aditirag/icml2025.html)
  • Key finding: Injecting noise at the input layer ("seed-conditioning") works as well as, and sometimes better than, temperature sampling at the output layer for eliciting creative and diverse outputs from language models. The paper argues that standard next-token prediction has inherent limits for tasks requiring planning and novel pattern discovery.
  • Relevance: Informed spellbook's since-removed fun mode, whose random personas acted as semantic seeds intended to steer generation toward diverse solution pathways.

Persona Effects on Reasoning

Tan, F. A., et al. (2024). PHAnToM: Persona-based Prompting Has An Effect on Theory-of-Mind Reasoning in Large Language Models. arXiv preprint arXiv:2403.02246.

  • Link: https://arxiv.org/abs/2403.02246
  • Key finding: Personas significantly affect Theory of Mind (ToM) reasoning. Dark Triad personality traits have larger effects than Big Five traits. Models with higher variance across personas are more "controllable."
  • Relevance: Personas enhance social-cognitive reasoning, which is relevant to creative dialogue and collaboration.

Park, J. S., et al. (2023). Generative Agents: Interactive Simulacra of Human Behavior. 36th Annual ACM Symposium on User Interface Software and Technology (UIST '23).

  • Link: https://arxiv.org/abs/2304.03442
  • Key finding: Memory-augmented persona architectures enable emergent social behaviors. Agents in the "Smallville" simulation autonomously coordinated complex social events while maintaining consistent personalities.
  • Relevance: Demonstrates that persona consistency improves believability and emergent creative behaviors.

Emotional Prompts

Li, C., et al. (2023). Large Language Models Understand and Can be Enhanced by Emotional Stimuli. arXiv preprint arXiv:2307.11760.

  • Link: https://arxiv.org/abs/2307.11760
  • Key finding: Emotional prompts ("This is important to my career") improve LLM performance by 8% on Instruction Induction and 115% on BIG-Bench tasks.
  • Relevance: Emotional-stakes skill uses emotional framing to improve accuracy on critical tasks.

Wang, X., et al. (2024). NegativePrompt: Leveraging Psychology for Large Language Models Enhancement via Negative Emotional Stimuli. International Joint Conference on Artificial Intelligence (IJCAI 2024).

  • Link: https://www.ijcai.org/proceedings/2024/719
  • Key finding: Negative emotional stimuli ("If you fail, there will be consequences") improve performance by 12.89% on Instruction Induction and 46.25% on BIG-Bench.
  • Relevance: Consequence framing in emotional-stakes improves truthfulness and accuracy.

Theoretical Foundations

Janus. (2022). Simulators. LessWrong.

  • Link: https://www.lesswrong.com/posts/vJFdjigzmcXMhNTsx/simulators
  • Key finding: LLMs should be understood as "simulators" that can model any agent from their training data. Personas act as conditioning that steers generation to specific latent space regions corresponding to that agent type.
  • Relevance: Theoretical foundation for why personas affect output quality differently across domains.

Important Limitations

Zheng, M., et al. (2023). When "A Helpful Assistant" Is Not Really Helpful: Personas in System Prompts Do Not Improve Performances of Large Language Models. arXiv preprint arXiv:2311.10054.

  • Link: https://arxiv.org/abs/2311.10054
  • Key finding: Across 162 personas and 2,410 factual questions (MMLU), personas do not improve performance on factual tasks compared to neutral prompts. Effects are inconsistent and sometimes negative.
  • Relevance: Critical caveat - personas may help creative and social reasoning tasks but do not help factual question-answering. This finding is why the since-removed fun mode restricted personas to dialogue, never to code, commits, or documentation.

Gupta, S., et al. (2024). Bias Runs Deep: Implicit Reasoning Biases in Persona-Assigned LLMs. International Conference on Learning Representations (ICLR 2024).

  • Key finding: Persona-assigned LLMs can exhibit implicit reasoning biases that affect downstream task performance.
  • Relevance: Additional support for restricting personas to non-critical outputs.

Hallucination Prevention and Detection

Dhuliawala, S., Komeili, M., Xu, J., Raileanu, R., Li, X., Celikyilmaz, A., & Weston, J. (2023). Chain-of-Verification Reduces Hallucination in Large Language Models. arXiv preprint arXiv:2309.11495.

  • Link: https://arxiv.org/abs/2309.11495
  • Key finding: A multi-step self-correction protocol where the LLM generates verification questions about its own claims, answers them independently, and revises based on contradictions. Reduces hallucination across multiple tasks.
  • Relevance: Foundation for the CoVe self-interrogation protocol used in fact-checking, dehallucination, and verifying-hunches skills.

Min, S., Krishna, K., Lyu, X., Lewis, M., Yih, W., Koh, P. W., Iyyer, M., Zettlemoyer, L., & Hajishirzi, H. (2023). FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation. Proceedings of EMNLP 2023.

  • Link: https://aclanthology.org/2023.emnlp-main.741/
  • Key finding: Breaking text into atomic, independently verifiable claims enables fine-grained factual evaluation. The atomic decomposition approach reveals errors hidden in compound statements.
  • Relevance: Foundation for the /decompose-claims protocol command used during fact-checking extraction and design document review.

Tambon, F., Nikanjam, A., An, L., Khomh, F., & Bhatt, G. (2025). Detecting and Correcting Hallucinations in LLM-Generated Code via Deterministic AST Analysis. arXiv preprint arXiv:2601.19106.

  • Link: https://arxiv.org/abs/2601.19106
  • Key finding: AST-based analysis can deterministically detect code hallucinations including non-existent API calls, incorrect parameter usage, and fabricated import paths.
  • Relevance: Informs the API hallucination detection checklists in code-review and enforcing-code-quality skills.

Pomian, R., Santu, S. K. K., Guha, A., & Ahmed, T. (2025). HalluJudge: A Reference-Free Hallucination Detection for Context Misalignment in Code Review Automation. arXiv preprint arXiv:2601.19072.

  • Link: https://arxiv.org/abs/2601.19072
  • Key finding: Reference-free hallucination detection can identify context misalignment in automated code review, where review comments address code that does not exist or mischaracterize actual behavior.
  • Relevance: Supports the API hallucination detection approach in code-review audit mode, specifically checking that review findings reference real code structures.

De Langis, K., & Zheng, R. (2026). Reasoning Improves Accuracy but Hurts Recall at Strict False Positive Rates. Proceedings of EACL 2026.

  • Link: https://aclanthology.org/2026.eacl-long.190.pdf
  • Implementation note: This URL was provided from user source research and should be verified during implementation (confirm the ACL Anthology URL resolves correctly).
  • Key finding: Chain-of-thought reasoning improves accuracy but can reduce recall at strict false positive thresholds. Verification systems must balance thoroughness against over-flagging.
  • Relevance: Calibrates expectations for CoVe protocol: self-interrogation improves precision but should not be used to suppress genuine findings.

Gekhman, Z., Yona, G., Aharoni, R., Eyal, M., Feder, A., Reichart, R., & Herzig, J. (2025). The Law of Knowledge Overshadowing: Towards Understanding, Predicting, and Preventing LLM Hallucination. Findings of ACL 2025.

  • Link: https://aclanthology.org/2025.findings-acl.1199.pdf
  • Key finding: LLMs hallucinate when "popular" knowledge overshadows less common but correct facts. The model's confidence in a hallucination correlates with the dominance of the overshadowing knowledge.
  • Relevance: Explains why API hallucinations are common (popular API patterns overshadow less-used but correct alternatives) and supports the evidence hierarchy's prohibition on Tier 6 (parametric knowledge) as sole evidence.

Yaldiz, A., Su, H., Deshpande, A., Kumar, V., & Awadallah, A. H. (2025). Uncertainty Quantification for Hallucination Detection in Large Language Models. arXiv preprint arXiv:2510.12040.

  • Link: https://arxiv.org/abs/2510.12040
  • Key finding: Uncertainty quantification metrics can predict hallucination likelihood. High model uncertainty on a claim correlates with hallucination risk, suggesting verification effort should be concentrated on uncertain outputs.
  • Relevance: Theoretical support for the depth escalation protocol: claims where the model expresses uncertainty should be escalated to deeper verification, not accepted at face value.

Additional Reading

Kong, A., et al. (2024). Better Zero-Shot Reasoning with Role-Play Prompting. Proceedings of NAACL 2024, pages 4099-4113.

  • Role-play prompting can improve zero-shot reasoning in specific contexts.

Wang, Z., et al. (2024). Persona is a Double-edged Sword: Mitigating the Negative Impact of Role-playing Prompts in Zero-shot Reasoning Tasks. arXiv preprint arXiv:2408.08631.

Development ceremonies (research backing)

Spellbook's development ceremonies grew from practice, not from the literature, but independent research supports many of them. The five items below map each ceremony to the research that validates it.

The green mirage: tests that pass without verifying behavior

Spellbook's green-mirage audit asks of every test whether it would fail if the code were broken. This is the coding face of reward hacking / specification gaming, a documented failure mode where an agent satisfies the literal metric without the intended outcome. It is not hypothetical: audits find a material share of SWE-bench patches pass their tests yet are wrong or incomplete, and mutation analysis finds most SWE-bench Verified instances admit a surviving mutant.

Amodei, D., Olah, C., Steinhardt, J., Christiano, P., Schulman, J., & Mané, D. (2016). Concrete Problems in AI Safety. arXiv preprint arXiv:1606.06565.

  • Link: https://arxiv.org/abs/1606.06565
  • Key finding: Catalogues concrete AI safety failure modes, including reward hacking, where a system satisfies the literal specification of an objective without achieving the intended outcome.
  • Relevance: Frames the green-mirage audit as detection of the coding-domain form of reward hacking: a test that passes while the code is wrong.

Krakovna, V., et al. (2020). Specification gaming: the flip side of AI ingenuity. DeepMind blog.

Skalse, J., Howe, N. H. R., Krasheninnikov, D., & Krueger, D. (2022). Defining and Characterizing Reward Hacking. Advances in Neural Information Processing Systems 35 (NeurIPS 2022).

  • Link: https://arxiv.org/abs/2209.13085
  • Key finding: Provides a formal definition of reward hacking and characterizes when a proxy objective can be gamed relative to the true objective.
  • Relevance: Theoretical grounding for why a test (a proxy for correctness) can be satisfied without the code being correct.

Aleithan, R., et al. (2024). SWE-Bench+: Enhanced Coding Benchmark for LLMs. arXiv preprint arXiv:2410.06992.

  • Link: https://arxiv.org/abs/2410.06992
  • Key finding: An audit of SWE-bench patches finds many pass their tests yet are wrong or incomplete: 9.26% incorrect / 5.56% incomplete on SWE-bench Lite, and 12.5% / 9.82% on SWE-bench Verified.
  • Relevance: Empirical evidence that passing tests do not guarantee correct code, motivating the green-mirage audit.

Are Benchmark Tests Strong Enough? Mutation-Guided Diagnosis and Augmentation of Regression Suites (tool: STING; alt. "Probe to Generate", tool PROBE). arXiv preprint arXiv:2604.01518.

  • Link: https://arxiv.org/abs/2604.01518
  • Key finding: Mutation-guided analysis finds that 77% of SWE-bench Verified instances admit at least one surviving mutant, meaning the accompanying tests fail to detect a deliberately broken implementation.
  • Relevance: Quantifies test weakness in a widely used benchmark, the exact condition the green-mirage audit and mutation discipline target.

Mutation as the measure of a test's strength

Spellbook's TDD rule requires every assertion to name the mutation it kills, and no check counts until it has gone red on deliberately broken code. This is mutation testing, established since the late 1970s. A test that still passes when the code is broken proves nothing.

DeMillo, R. A., Lipton, R. J., & Sayward, F. G. (1978). Hints on Test Data Selection: Help for the Practicing Programmer. IEEE Computer, 11(4), 34-41.

  • Link: https://doi.org/10.1109/C-M.1978.218136
  • Key finding: Introduces mutation testing: deliberately altering a program to produce mutants, then measuring whether the test suite detects (kills) them as a measure of test adequacy.
  • Relevance: Origin of the mutation-as-check-strength discipline spellbook applies to TDD. Framework: Stryker.

Constrained beats unconstrained autonomy

Spellbook runs one orchestrator dispatching narrowly-scoped subagents, each with only its local context, and keeps the reviewer separate from the author. Research finds unconstrained multi-turn autonomy amplifies early mistakes, and that hierarchical orchestration with per-subagent context isolation manages exactly this.

Xia, C. S., Deng, Y., Dunn, S., & Zhang, L. (2025). Agentless: Demystifying LLM-based Software Engineering Agents. Proceedings of FSE 2025. arXiv:2407.01489.

  • Link: https://arxiv.org/abs/2407.01489
  • Key finding: A simple, constrained three-phase approach reaches 32.00% on SWE-bench Lite, outperforming more autonomous agents; unconstrained multi-turn autonomy compounds early errors and inflates context.
  • Relevance: Supports spellbook's constrained orchestration and its resistance to over-decomposition.

BOAD: Discovering Hierarchical Software Engineering Agents via Bandit Optimization (2026). Proceedings of ICLR 2026. arXiv:2512.23631.

  • Link: https://arxiv.org/abs/2512.23631
  • Key finding: Hierarchical agent structures discovered via bandit optimization improve software engineering task performance through structured decomposition and coordination.
  • Relevance: Supports the orchestrator-plus-subagents structure spellbook uses.

Confucius Code Agent: Scalable Agent Scaffolding for Real-World SWE (2026). arXiv preprint arXiv:2512.10398.

  • Link: https://arxiv.org/abs/2512.10398
  • Key finding: Agent scaffolding with per-subagent context isolation scales to real-world software engineering tasks by managing context per subagent rather than accumulating it in one loop.
  • Relevance: Supports spellbook's per-subagent context isolation and reviewer/author separation.

Granularity at the acceptance boundary

Spellbook's plan skills collapse work items that are only jointly acceptable, resisting over-decomposition. This is consistent with the Agentless finding that excess decomposition inflates context and compounds error (Xia et al., 2025, arXiv:2407.01489, cited above).

Specification before code

Spellbook's design-doc → plan → TDD flow, its discovery/clarify pass, and its loading of project governance (AGENTS.md) as binding constraints parallel spec-driven development: an executable contract before code, a "constitution" of principles, and a clarify step that interrogates the spec before building. This part of the literature argues the case rather than measuring it.

Piskala, D. B. (2026). Spec-Driven Development: From Code to Contract in the Age of AI Coding Assistants. arXiv preprint arXiv:2602.00180.

  • Link: https://arxiv.org/abs/2602.00180
  • Key finding: Argues (as a technical report, not an empirical measurement) for treating a specification as an executable contract that precedes and governs code generation.
  • Relevance: Parallels spellbook's design-doc → plan → TDD flow.

GitHub Spec Kit.

  • Link: https://github.com/github/spec-kit
  • Key finding: A toolkit for spec-driven development built around a project "constitution" of principles and a /clarify workflow that interrogates the spec before implementation.
  • Relevance: Parallels spellbook's clarify pass and its loading of project governance (AGENTS.md) as binding constraints.

Summary

Technique Research Support Domain Used In
Random personas Nagarajan et al. (ICML 2025), Tan (PHAnToM) Creative, social reasoning Not currently used (fun-mode skill removed)
Emotional framing Li (EmotionPrompt), Wang (NegativePrompt) All reasoning tasks emotional-stakes
Persona consistency Park (Generative Agents) Long-form interaction Not currently used (fun-mode skill removed)
CoVe self-interrogation Dhuliawala et al. (2023) Verification, claim checking fact-checking, dehallucination, verifying-hunches
Atomic claim decomposition Min et al. (FActScore, EMNLP 2023) Claim extraction, verification decompose-claims command
API hallucination detection Tambon et al. (2025), Pomian et al. (2025) Code review, quality code-review, enforcing-code-quality

Design principle: Where spellbook applied personas, it confined them to dialogue and kept them out of code and documentation, following Zheng et al.'s finding that personas do not improve objective task performance. The persona-driven modes have since been removed; the constraint stands for any future use.