Skip to content

CI Regression Gate

pisama check turns a directory of saved traces into a pass/fail gate for your CI pipeline. It walks the path you give it, runs the built-in detectors against every trace it finds, and exits non-zero when any finding meets your severity threshold. When a production failure slips through, save its trace into your test set and the gate keeps it from regressing.

Everything runs offline and in-process: no server, no account, no API key.

pip install pisama
pisama check traces/ --fail-on error

What it does

  1. Discovers trace files (.json and .jsonl) under the path, skipping .git, node_modules, .venv, and other noise directories. OTEL, Langfuse, Phoenix, and raw JSON trace formats are auto-detected.
  2. Runs every built-in detector against each trace (loops, state corruption, coordination breakdown, hallucination, injection, and the rest). Run pisama detectors for the live list on your installed version.
  3. Reports each finding as a binary failure mode with a severity, and exits non-zero when any finding meets the --fail-on threshold.

Unparseable traces fail the gate rather than being skipped. A gate that silently ignores garbage inputs is a gate that false-greens.

Flags

Flag Default What it does
--fail-on {info,warning,error,critical,never} warning Severity threshold that flips the exit code. never reports findings without failing.
--detectors NAME[,NAME...] all Run only the named detectors.
--exclude-detectors NAME[,NAME...] none Run everything except the named detectors.
--json off Machine-readable output on stdout ({"schema_version": 1, "results": [...]}).
--quiet off Suppress per-file output; summary and exit code only.

Exit codes

Code Meaning
0 No finding met the threshold.
1 At least one finding met the threshold, or a trace file could not be parsed.
2 Usage error: unknown detector name, empty detector list, or both selection flags at once. The error lists the valid detector names, so a typo can never produce a silent green.

GitHub Actions

name: agent-trace-gate

on:
  pull_request:

jobs:
  trace-gate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.11"
      - run: pip install pisama
      - name: Gate on agent failures
        run: pisama check traces/ --fail-on error --quiet

To keep a machine-readable report as a build artifact:

      - name: Gate on agent failures
        run: pisama check traces/ --fail-on error --json > pisama-report.json
      - uses: actions/upload-artifact@v4
        if: always()
        with:
          name: pisama-report
          path: pisama-report.json

Scoping the gate

Gate on a subset when you only want structural failures blocking merges:

pisama check traces/ --detectors loop,corruption,coordination --fail-on warning

Or run everything but keep a noisy detector advisory-only by excluding it from the gate and reporting it in a second, non-blocking step:

pisama check traces/ --exclude-detectors cost --fail-on error
pisama check traces/ --detectors cost --fail-on never --json > cost-report.json

Where the traces come from

Any trace your agents already emit works: export from your OTEL collector, Langfuse, or Phoenix, or save traces from the Python SDK. A good starting set is 20 to 50 traces that cover your agent's main workflows, plus every trace that ever accompanied a production incident.

  • SDK Quickstart for producing traces locally.
  • Detection Reference for what each detector catches.
  • The hosted platform at pisama.ai runs the same detectors continuously on live traffic; the CI gate is the offline, pre-merge half of the same loop.