Skip to content

Calibration FAQ

Three questions we get constantly about detector thresholds and accuracy.

Why are OSS defaults uncalibrated?

The detector code in pisama-detectors ships with conservative default thresholds. That is deliberate: defaults are tuned to minimize false positives on a broad, generic trace mix, not to maximize F1 on a specific application.

Tuned thresholds require a labelled dataset — traces marked with ground-truth "this was a real loop", "this was a real injection". Without that data, any threshold shipped by default is a guess. Shipping guesses as "calibrated" would be worse than shipping conservative defaults: users would trust the number and be wrong.

So pisama-detectors gives you the detector code and reference thresholds that work out-of-the-box with low FP rates. What it does not give you is the curve of threshold → F1 for your application's traces. That curve is what Pisama Cloud produces.

How do Cloud thresholds differ?

Pisama Cloud runs each detector against a labelled golden dataset across easy/medium/hard difficulty and picks thresholds per-detector to maximise F1. 20 detectors are externally validated at production grade, certified by pooled k-fold out-of-fold scoring, so every certified row is scored under a threshold fit on folds that never contained it.

Full per-detector reference, including the detectors that ran and failed: Detection overview

The same detector code, with Cloud's tuned thresholds, recovers meaningful F1 gains over OSS defaults on real production traces. That delta is the product.

Cloud thresholds also refresh: each quality-gate run re-picks thresholds against the latest golden data, so drift (new model versions, new failure patterns) gets corrected without a code change.

Can I tune locally?

Yes. Supported pisama-detectors functions expose the options that apply to their detection method. For example, detect_loop accepts a recent-state window and a semantic similarity threshold:

from pisama_detectors import detect_loop

states = [
    {"step": 1, "status": "searching"},
    {"step": 2, "status": "searching"},
    {"step": 3, "status": "searching"},
]

default_result = detect_loop(states=states)
custom_result = detect_loop(
    states=states,
    window_size=3,
    similarity_threshold=0.9,
)

In version 0.3.2, window_size is the total recent-state window, including the current state. similarity_threshold must be between 0 and 1, inclusive. The same threshold is applied to pairwise and clustering-based semantic checks. Invalid values raise ValueError.

The detectors are deterministic on a given input, so a standard precision/recall sweep works. What Cloud gives you on top of that:

  • The golden dataset to sweep against, spanning the MAST failure modes and framework-specific cases.
  • The eval harness (calibrate.py) that runs the sweep, handles per-difficulty breakdown, and flags saturation.
  • Tiered escalation (Tier 1 hash → Tier 5 LLM judge) so expensive detectors only run when cheap ones are uncertain. Targets $0.05/trace.
  • Continuous re-calibration as you ship new agent versions.

If you have the labelled data and the time, local tuning works. If you don't, Pisama Cloud is the shortcut.