Signal, Not Secrets
A Field Guide to AI Conversation Failure Modes
Evidence-first documentation for repeatable LLM failures without implying intent.
If you have ever thought, “This model keeps doing that thing,” but you cannot document it cleanly without turning it into a vibes-only argument, this is your system.
Signal, Not Secrets is a practical toolkit for capturing repeatable conversational failures in LLM systems using publishable, audit-ready evidence. No mind-reading. No intent claims. Just observable patterns, triggers, signatures, metrics, and mitigations.
What this helps you do
Â
- Document failures that happen again and again, not one-off weirdness
- Separate behavior from intent, so your analysis stays defensible
- Turn messy chat logs into structured evidence sheets and pattern maps
- Stress-test patterns with prompt families and simple scoring rubrics
- Produce work that is usable for audits, safety reviews, UX/policy iteration, and publishable analysis
What’s inside
The core framework
- Signal vs Secrets (what you can claim, what you cannot)
- Behavior vs Intent (how to stay evidence-first)
- Documentation scaffolding that is reviewer-friendlyThe 10 Pattern Map Library
The 10 Pattern Map Library
You get a full taxonomy of 10 repeatable failure modes, each structured as:
trigger → signature → impact → likely mechanism (non-intent) → mitigation → test prompts → metrics
5 Evidence-style case studies
Each case study is written to be publishable and includes:
behavioral signatures, triggers, mechanisms, mitigations, and reproduction notes.
Templates (copy/paste ready)
- Evidence sheet
- One-page pattern map template
- Stress test suite template
- Metrics and scoring rubric guidance Â
Choose your tier
 This is a 3-tier system, so you can match the depth to your use case.
Tier 1: Starter
Field Guide + Templates
Best for: quick, clean documentation.
Includes: core method, 10-pattern taxonomy (short), evidence + pattern map templates, lite stress-test, publishing checklist.
Deliverables: PDF + editable templates + 1 sample stress-test table.
Tier 2: Pro
Full eBook + Pattern Maps + Case Studies
Best for: repeatable analysis you can publish.
Includes: everything in Starter + 10 full pattern maps + scoring, prompt families + metrics, 5 case studies, practical mitigations, glossary.
Deliverables: PDF eBook + appendices (templates, scorecards, example logs).
Tier 3: Practioner
Toolkit for Audits + Client Work
Best for: consultants and internal teams.
Includes: everything in Pro + full stress-test suite + protocol, client-ready audit report template, case study expansion playbook, packaging assets (white paper/article/slide outlines).
Deliverables: PDF toolkit + prompt pack + report templates + kickoff checklist.
Who this is for
- Safety and evaluation teams
- UX and policy teams iterating on failure modes
- Researchers and auditors documenting repeatable behaviorsÂ
- Creators publishing rigorous analysis without speculative claims
Who this is not for
- Anyone looking for “secret internals” or hidden system claims
- Anyone who wants to argue intent instead of documenting behavior
- Anyone who wants vibes without evidence
About Vireya Institute
Educational Coaching & Integrative Consulting
Book a 1:1 educational coaching session focused on
ethical influence, agency, and applied consciousness studies.
