Harbor Integration¶

Harbor runs agents in reproducible environments and writes each trial as an Agent Trajectory Interchange Format (ATIF) trajectory. Pisama can analyze those trajectories after the run without adding a runtime dependency to Harbor.

Pisama supports ATIF v1.0 through v1.7 and can analyze:

• one trajectory.json file;
• a Harbor trial directory containing agent/trajectory.json; or
• a complete Harbor job directory containing multiple trials.

The file-aware clients discover exact trajectory roots, follow safe relative continuation references, and ignore job metadata such as config.json, lock.json, and result.json.

Run a Harbor job¶

Install Harbor and run a dataset with your agent and model:

uv tool install harbor
harbor run \
  --dataset harbor-index/harbor-index-1.0 \
  --agent "<agent>" \
  --model "<model>"

Harbor prints the job directory when the run completes. The examples below use jobs/<harbor-job> as that path.

Analyze locally¶

The open-source pisama package runs locally without an account or API key:

pip install pisama
pisama check jobs/<harbor-job> \
  --detectors hallucination \
  --fail-on never \
  --json > pisama-harbor.json

--fail-on never keeps the analysis observational. The JSON report includes every analyzed trial, clean trials, findings, analysis errors, detector counts, and an aggregate summary suitable for CI artifacts.

Check that every trajectory completed analysis:

jq -e '
  .summary.analysis_errors == 0 and
  ([.results[].status] | all(. == "clean" or . == "issues"))
' pisama-harbor.json

Analyze with the hosted API¶

Exchange an API key for a bearer token, then post one ATIF trajectory:

TOKEN=$(curl -sS https://api.pisama.ai/api/v1/auth/token \
  -H "Content-Type: application/json" \
  -d '{"api_key":"YOUR_API_KEY"}' |
  jq -r '.access_token')

jq '{trajectory: .}' jobs/<harbor-job>/<trial>/agent/trajectory.json |
  curl -sS https://api.pisama.ai/api/v1/atif/analyze \
    -H "Authorization: Bearer $TOKEN" \
    -H "Content-Type: application/json" \
    --data-binary @-

The endpoint parses the trajectory without persisting it and returns the diagnosis plus a trace summary. See the API reference.

Production status¶

Harbor support is ready for ATIF ingestion and shadow analysis. It is not currently approved as a blocking quality gate.

The latest production-comparable calibration used 51 real trajectories:

These results miss Pisama's production gate of F1 at least 80%, precision at least 70%, and a successful-run alert rate at most 10%. A runtime-error candidate improved precision to 100% and F1 to 50%, but recall remained 33.33%. It is therefore not enabled as a blocking profile.

Keep --fail-on never until a detector profile is calibrated on your workload. Harbor's verifier remains the source of task outcome and reward. Pisama provides process-level failure evidence, not a replacement for that verifier.