See a Failure in 5 Minutes

Run Failure Analysis on your existing AI data in three steps. No tracing, no instrumentation, no OpenAI key needed.

This quickstart uses dataset mode — you pass your AI inputs and outputs directly. If you already have tracing set up, see Failure Analysis from traces instead.

Step 1: Install the SDK

pip install valiqor

Step 2: Set your API key

export VALIQOR_API_KEY="vq_your_api_key_here"

Get your API key from the Valiqor Dashboard. You can also pass it directly via ValiqorClient(api_key="vq_...").

Step 3: Run Failure Analysis

Create a file called quickstart.py and paste:

quickstart.py

from valiqor import ValiqorClient

client = ValiqorClient(api_key="vq_your_api_key_here", project_name="quickstart")

# Run FA on existing data — no tracing required
result = client.failure_analysis.run(
    dataset=[
        {
            "input": "What is the capital of France?",
            "output": "The capital of France is Berlin.",
            "context": [
                "France is a country in Western Europe.",
                "The capital of France is Paris.",
                "Paris is known as the City of Light.",
            ],
        }
    ]
)

# Print the summary
print(f"Status: {result.status}")
print(f"Failures detected: {result.summary.total_failures_detected}")
print(f"Overall severity: {result.summary.overall_severity}/5")
print(f"Should alert: {result.summary.should_alert}")
print()

# Print each failure
for tag in result.failure_tags:
    if tag.decision == "fail":
        print(f"❌ [{tag.bucket_name}] {tag.subcategory_name}")
        print(f"   Severity: {tag.severity}/5  Confidence: {tag.confidence}")
        if tag.judge_rationale:
            print(f"   Rationale: {tag.judge_rationale}")
        print()

Run it:

python quickstart.py

What you’ll see

The output intentionally passes a wrong answer (“Berlin” instead of “Paris”) so you can see a real failure detected:

Status: completed
Failures detected: 1
Overall severity: 4/5
Should alert: True

❌ [Hallucination] Entity Fabrication
   Severity: 4/5  Confidence: 0.95
   Rationale: The output states Berlin is the capital of France,
   which directly contradicts the provided context that identifies
   Paris as the capital. This is a factual fabrication.

The exact bucket names, severity scores, and rationale text may vary slightly depending on your backend configuration and LLM judge version.

Understanding the result

The result object is an FARunResult with these key fields:

Field	Type	What it means
`result.summary.total_failures_detected`	`int`	Total number of failure tags with `decision == "fail"`
`result.summary.overall_severity`	`float`	Worst-case severity across all failures (0–5)
`result.summary.should_alert`	`bool`	Whether this result warrants an alert
`result.failure_tags`	`list[FailureTag]`	Individual failure classifications
`result.failure_tags[i].decision`	`str`	`"pass"`, `"fail"`, or `"unsure"`
`result.failure_tags[i].judge_rationale`	`Optional[str]`	Human-readable explanation of the failure (may be `None` for pass tags)

Next: Fix the failure →

Learn how to interpret the result, understand root causes, and fix your prompt.

Analyze multiple items at once

You can pass multiple input/output pairs in a single call:

result = client.failure_analysis.run(
    dataset=[
        {
            "input": "What is the capital of France?",
            "output": "The capital of France is Paris.",
            "context": ["The capital of France is Paris."],
        },
        {
            "input": "Summarize the article about climate change.",
            "output": "The article discusses economic policy in the 1990s.",
            "context": [
                "Climate change is causing rising sea levels worldwide.",
                "The Paris Agreement aims to limit warming to 1.5°C.",
            ],
        },
        {
            "input": "What medications interact with warfarin?",
            "output": "Warfarin interacts with aspirin and ibuprofen.",
            "context": [
                "Warfarin interacts with aspirin, ibuprofen, and many antibiotics.",
                "Always consult a healthcare provider for drug interactions.",
            ],
        },
    ]
)

print(f"Items analyzed: {result.summary.total_items}")
print(f"Items with failures: {result.summary.items_with_failures}")
print(f"Items all passed: {result.summary.items_all_passed}")

Optional: Add tracing later

Once you’re comfortable with dataset mode, you can add auto-tracing to capture LLM calls in production and run Failure Analysis on traces:

import valiqor.auto  # Auto-instruments OpenAI, Anthropic, LangChain

# Your normal LLM calls are now traced automatically
# Then run FA on the captured trace:
result = client.failure_analysis.run(trace_id="tr_abc123")

See the Tracing guide for full setup instructions.

Start Here

Core Workflows

Concepts

Integrations

Resources

See a Failure in 5 Minutes

Step 1: Install the SDK

Step 2: Set your API key

Step 3: Run Failure Analysis

What you’ll see

Understanding the result

Next: Fix the failure →

Analyze multiple items at once

Optional: Add tracing later

​Step 1: Install the SDK

​Step 2: Set your API key

​Step 3: Run Failure Analysis

​What you’ll see

​Understanding the result

Next: Fix the failure →

​Analyze multiple items at once

​Optional: Add tracing later

Step 1: Install the SDK

Step 2: Set your API key

Step 3: Run Failure Analysis

What you’ll see

Understanding the result

Analyze multiple items at once

Optional: Add tracing later