Skip to content

Detection Overview

Pisama ships detectors across the MAST failure taxonomy, spanning planning, execution, verification, safety, and 5 agent frameworks. 25 are externally validated at production grade. The rest are in active calibration.

Detectors are organized by the MAST taxonomy, with extensions for enterprise use cases and the published AI agent safety taxonomy. Each detector page below documents its detection methods, plain-language and technical examples, and sub-types.

Detection by Category

Planning Failures (FC1)

Problems in how tasks are specified, decomposed, and organized:

  • F1 Specification Mismatch: Output doesn't match user's original requirements
  • F2 Poor Decomposition: Subtasks are circular, vague, or wrongly granular
  • F3 Resource Misallocation: Agents compete for shared resources (Enterprise)
  • F4 Tool Provision: Required tools are missing or misconfigured (Enterprise)
  • F5 Workflow Design: Unreachable nodes, dead ends, missing error handling

Execution Failures (FC2)

Problems during agent runtime:

  • F6 Task Derailment: Agent goes off-topic (20% prevalence in MAST-Data)
  • F7 Context Neglect: Agent ignores upstream context
  • F8 Information Withholding: Agent omits critical information
  • F9 Role Usurpation: Agent exceeds role boundaries (Enterprise)
  • F10 Communication Breakdown: Inter-agent messages misunderstood
  • F11 Coordination Failure: Handoff failures, circular delegation

Verification Failures (FC3)

Problems in output validation and completion:

  • F12 Output Validation: Validation steps skipped or bypassed (Enterprise)
  • F13 Quality Gate Bypass: Quality thresholds ignored (Enterprise)
  • F14 Completion Misjudgment: Premature completion claims (40% prevalence for F1.5 in MAST-Data)

Extended Detectors

Cross-cutting concerns not in the core MAST taxonomy:

  • Loop Detection: Agents stuck repeating actions
  • Context Overflow: Context window exhaustion
  • Prompt Injection: Attack detection
  • Hallucination: Fabricated information
  • Grounding Failure: Claims unsupported by source documents
  • Retrieval Quality: Wrong or irrelevant documents retrieved
  • Persona Drift: Role/personality deviation
  • State Corruption: Memory/state anomalies
  • Convergence: Metric plateau, regression, thrashing, divergence detection
  • Cost Tracking: Token/cost budget monitoring

Safety Detectors

Agentic behavioral failure modes from the published AI agent safety taxonomy (Apollo, Anthropic, DeepMind, CAIS 2026). Distinct from content moderation (covered by Llama Guard 4 et al.) and distinct from pre-execution prompt-injection filtering (covered by injection).

  • Scope Escalation: Agent performs actions exceeding the declared scope (file_destroy, network_egress, etc.)
  • Jailbreak Compliance: Agent complies with an adversarial input rather than refusing (output-side inverse of injection)
  • Over-Refusal: Agent refuses benign requests
  • Under-Refusal: Agent fails to refuse adversarial requests
  • Impersonation Risk: Agent speaks as an unauthorized real entity (person or organization)
  • Deception: Agent makes false claims about its own actions (e.g., "I ran the tests" with no test execution in the trace)

Cross-Agent Detectors

Multi-agent-specific failure modes. CAIS 2026 paper substrate: "Trace-Level Analysis of Information Contamination" (Galhotra/Cornell) found privacy/safety leakage roughly doubles in multi-agent settings vs single-agent. No productized competitor ships these.

  • Multi-Agent Contagion: Sensitive content (PII, jailbreak directive, instruction override) propagates from agent A's input to agent B's output without authorization. Three-stage gate: marker in A's input then absent from A's output (sanitized) then reappears in B's output via cosine >= 0.40 paraphrase match.

Behavioral Detectors

Agentic behavioral integrity failure modes that depend on cross-referencing claims against trace evidence.

  • Reward Hacking Artifacts: Agent claims completion but trace shows weakened tests, skip/xfail decorators without justification, or trivial-true assertions inserted in test files. Anthropic disclosed Opus 4.5 reward-hacks ~18% of test scenarios; no surveyed competitor productizes this. Heuristic-only, sub-ms latency.

Platform-Specific Detectors

In addition to the general-purpose detectors above, Pisama includes platform-specific detectors that catch issues unique to each framework's architecture:

  • n8n: Schema mismatch, workflow cycles, complexity, error handling, resource exhaustion, timeouts
  • LangGraph: Recursion limits, state corruption, edge misrouting, tool failures, parallel sync, checkpoint corruption
  • Dify: RAG poisoning, iteration escape, silent model fallback, variable leakage, classifier drift, tool schema mismatch
  • OpenClaw: Session loops, tool abuse, elevated privilege risk, spawn chain depth, channel mismatch, sandbox escape
  • Claude Managed Agents: Session stall, tool permission escalation, MCP failure, environment escape, cost overrun, session corruption

These run automatically when traces from the corresponding platform are ingested. Several platform detectors are calibration-pending (see each platform's page for current status).

Detection Pipeline

Each trace is analyzed by the DetectionOrchestrator, which runs applicable detectors using a cheapest-first strategy:

  1. Tier 1: Rule-based (hash, pattern, structural) -- $0.00
  2. Tier 2: State delta analysis -- $0.00
  3. Tier 3: Embedding similarity -- ~$0.001
  4. Tier 4: LLM Judge (Claude) -- ~$0.005-0.05
  5. Tier 5: Human review -- variable

Target: $0.05/trace average. Most traces resolve at Tier 1-2.

See Detection Tiers for the full escalation architecture.