Skip to main content
Valiqor uses a state-of-the-art LLM as the default judge for all LLM-based evaluation metrics (e.g. hallucination, coherence, factual_accuracy).You can also bring your own OpenAI API key so that judge calls use your own quota. Pass it at any of these levels (highest priority wins):
  1. Method parameter — client.eval.evaluate(dataset=..., openai_api_key="sk-...")
  2. Client constructor — ValiqorClient(api_key="vq_...", openai_api_key="sk-...")
  3. Environment variable — VALIQOR_OPENAI_API_KEY
  4. Config file — openai_api_key in .valiqorrc
Your OpenAI key is never stored or persisted by Valiqor. It is only used for the duration of the API request.
  • Evaluation results, traces, and analysis data are stored in the Valiqor backend database for your team to review in the dashboard.
  • OpenAI API keys provided via BYOK (Bring Your Own Key) are never stored — they are used only for the duration of the request and then discarded.
Yes. Both the SDK and CLI support fully headless, non-interactive usage.Option 1 — CLI login with credentials:
valiqor login -e user@company.com -p $VALIQOR_PASSWORD
valiqor config set project_name=my-app environment=ci
valiqor eval run --dataset test_data.json --metrics factual_accuracy,coherence
Option 2 — Environment variables (no login needed):
export VALIQOR_API_KEY=vq_...
export VALIQOR_PROJECT_NAME=my-app
python run_evals.py
Option 3 — SDK with explicit key:
from valiqor import ValiqorClient

client = ValiqorClient(api_key="vq_...")
result = client.eval.evaluate(dataset=data, metrics=["factual_accuracy"])
All configuration can be set non-interactively with valiqor config set key=value or through environment variables (VALIQOR_API_KEY, VALIQOR_PROJECT_NAME, VALIQOR_TRACE_DIR, VALIQOR_SCAN_DIR, VALIQOR_BACKEND_URL, VALIQOR_ENVIRONMENT).
EvaluationFailure Analysis
PurposeScore LLM output quality on specific metricsFind why your AI app fails — root causes, severity, evidence
Accessclient.evalclient.failure_analysis
OutputPer-item metric scores (0–1)Failure buckets, root-cause evidence, severity (0–5), confidence (0–1)
Metricscoherence, factual_accuracy, hallucination, etc.Automatic — failure taxonomy with 30+ subcategories
ApproachLLM-as-judge per metricMulti-signal analysis
CLIvaliqor eval runvaliqor fa run
Use Evaluation when you want to measure quality over time. Use Failure Analysis when you want to understand what went wrong and fix it.
Three approaches, from zero-config to fully manual:Zero-config auto-instrumentation:
import valiqor.auto  # Patches all detected LLM providers automatically

# Your existing code works as-is — all LLM calls are traced
response = openai.chat.completions.create(...)
Decorator-based:
from valiqor.trace import trace_workflow, trace_function

@trace_workflow("my-agent")
def agent_pipeline(query):
    context = retrieve(query)
    return generate(query, context)

@trace_function()
def retrieve(query):
    ...

@trace_function()
def generate(query, context):
    ...
Manual spans:
from valiqor.trace import TracerV2

tracer = TracerV2()
with tracer.start_trace("agent-query") as trace:
    with tracer.start_span("retrieval"):
        docs = retriever.search(query)
    with tracer.start_span("generation"):
        answer = llm.generate(query, docs)
See Tracing AI Apps for the full guide.
Python 3.9+ — tested on Python 3.9, 3.10, 3.11, and 3.12.
The autolog() function (or import valiqor.auto) auto-instruments these providers:
ProviderMinimum Version
OpenAI≥ 1.0.0
Anthropic≥ 0.18.0
LangChain / LangGraph≥ 0.1.0
Ollama—
Agno—
You can also enable specific providers only:
from valiqor.trace.autolog import openai_autolog, anthropic_autolog

openai_autolog()       # Only instrument OpenAI
anthropic_autolog()    # Only instrument Anthropic
For providers without auto-instrumentation, use @trace_workflow and @trace_function decorators.
The base valiqor package requires:
  • requests >= 2.31.0
  • httpx >= 0.25.0
  • gitingest >= 0.1.0
Optional extras install provider-specific tracing support:
pip install valiqor[openai]      # + openai>=1.0.0
pip install valiqor[anthropic]   # + anthropic>=0.18.0
pip install valiqor[langchain]   # + langchain>=0.1.0, langchain-core>=0.1.0
pip install valiqor[trace]       # openai + anthropic + langchain
pip install valiqor[all]         # Everything
The SDK and CLI resolve configuration values in this order (highest priority first):
  1. Constructor / method parameters — ValiqorClient(api_key="vq_...")
  2. Environment variables — VALIQOR_API_KEY, VALIQOR_PROJECT_NAME, etc.
  3. Local config file — .valiqorrc in the project root
  4. Global credentials (CLI only) — ~/.valiqor/credentials.json
  5. Defaults — built-in defaults
Global credentials (~/.valiqor/credentials.json) are loaded by the CLI only. The SDK’s get_config() function resolves from environment variables and .valiqorrc but does not read global credentials.
Supported environment variables:
VariableConfig Key
VALIQOR_API_KEYapi_key
VALIQOR_PROJECT_NAMEproject_name
VALIQOR_OPENAI_API_KEYopenai_api_key
VALIQOR_TRACE_DIRtrace_dir
VALIQOR_SCAN_DIRscan_dir
VALIQOR_BACKEND_URLbackend_url
VALIQOR_ENVIRONMENTenvironment
1,000 items per request. This applies to evaluate(), audit(), and failure_analysis.run().If your dataset is larger, split it into batches or use the async API for better throughput.Trial users (email not verified) are additionally limited to 25 rows per run and 3 total runs. Verify your email with valiqor verify to remove trial limits.
Each user can have up to 5 API keys. Manage keys with:
valiqor keys list      # View your keys
valiqor keys create    # Create a new key
valiqor keys delete    # Delete a key