Skip to main content

Documentation Index

Fetch the complete documentation index at: https://docs.valiqor.com/llms.txt

Use this file to discover all available pages before exploring further.

Run Failure Analysis on your existing AI data in three steps. No tracing, no instrumentation, no OpenAI key needed.
This quickstart uses dataset mode — you pass your AI inputs and outputs directly. If you already have tracing set up, see Failure Analysis from traces instead.

​
Step 1: Install the SDK

pip install valiqor

​
Step 2: Set your API key

export VALIQOR_API_KEY="vq_your_api_key_here"
Get your API key from the Valiqor Dashboard. You can also pass it directly via ValiqorClient(api_key="vq_...").

​
Step 3: Run Failure Analysis

Create a file called quickstart.py and paste:
quickstart.py
from valiqor import ValiqorClient

client = ValiqorClient(api_key="vq_your_api_key_here", project_name="quickstart")

# Run FA on existing data — no tracing required
result = client.failure_analysis.run(
    dataset=[
        {
            "input": "What is the capital of France?",
            "output": "The capital of France is Berlin.",
            "context": [
                "France is a country in Western Europe.",
                "The capital of France is Paris.",
                "Paris is known as the City of Light.",
            ],
        }
    ]
)

# Print the summary
print(f"Status: {result.status}")
print(f"Failures detected: {result.summary.total_failures_detected}")
print(f"Overall severity: {result.summary.overall_severity}/5")
print(f"Should alert: {result.summary.should_alert}")
print()

# Print each failure
for tag in result.failure_tags:
    if tag.decision == "fail":
        print(f"❌ [{tag.bucket_name}] {tag.subcategory_name}")
        print(f"   Severity: {tag.severity}/5  Confidence: {tag.confidence}")
        if tag.judge_rationale:
            print(f"   Rationale: {tag.judge_rationale}")
        print()
Run it: 
python quickstart.py

​
What you’ll see

The output intentionally passes a wrong answer (“Berlin” instead of “Paris”) so you can see a real failure detected: 
Status: completed
Failures detected: 1
Overall severity: 4/5
Should alert: True

❌ [Hallucination] Entity Fabrication
   Severity: 4/5  Confidence: 0.95
   Rationale: The output states Berlin is the capital of France,
   which directly contradicts the provided context that identifies
   Paris as the capital. This is a factual fabrication.
The exact bucket names, severity scores, and rationale text may vary slightly depending on your backend configuration and LLM judge version.

​
Understanding the result

The result object is an FARunResult with these key fields:
FieldTypeWhat it means
result.summary.total_failures_detectedintTotal number of failure tags with decision == "fail"
result.summary.overall_severityfloatWorst-case severity across all failures (0–5)
result.summary.should_alertboolWhether this result warrants an alert
result.failure_tagslist[FailureTag]Individual failure classifications
result.failure_tags[i].decisionstr"pass", "fail", or "unsure"
result.failure_tags[i].judge_rationaleOptional[str]Human-readable explanation of the failure (may be None for pass tags)

Next: Fix the failure →

Learn how to interpret the result, understand root causes, and fix your prompt.

​
Analyze multiple items at once

You can pass multiple input/output pairs in a single call: 
result = client.failure_analysis.run(
    dataset=[
        {
            "input": "What is the capital of France?",
            "output": "The capital of France is Paris.",
            "context": ["The capital of France is Paris."],
        },
        {
            "input": "Summarize the article about climate change.",
            "output": "The article discusses economic policy in the 1990s.",
            "context": [
                "Climate change is causing rising sea levels worldwide.",
                "The Paris Agreement aims to limit warming to 1.5°C.",
            ],
        },
        {
            "input": "What medications interact with warfarin?",
            "output": "Warfarin interacts with aspirin and ibuprofen.",
            "context": [
                "Warfarin interacts with aspirin, ibuprofen, and many antibiotics.",
                "Always consult a healthcare provider for drug interactions.",
            ],
        },
    ]
)

print(f"Items analyzed: {result.summary.total_items}")
print(f"Items with failures: {result.summary.items_with_failures}")
print(f"Items all passed: {result.summary.items_all_passed}")

​
Optional: Add tracing later

Once you’re comfortable with dataset mode, you can add auto-tracing to capture LLM calls in production and run Failure Analysis on traces: 
import valiqor.auto  # Auto-instruments OpenAI, Anthropic, LangChain

# Your normal LLM calls are now traced automatically
# Then run FA on the captured trace:
result = client.failure_analysis.run(trace_id="tr_abc123")
See the Tracing guide for full setup instructions.