FAQ

What LLMs does Valiqor use for evaluation?

Valiqor uses a state-of-the-art LLM as the default judge for all LLM-based evaluation metrics (e.g. hallucination, coherence, factual_accuracy).You can also bring your own OpenAI API key so that judge calls use your own quota. Pass it at any of these levels (highest priority wins):

Method parameter — client.eval.evaluate(dataset=..., openai_api_key="sk-...")
Client constructor — ValiqorClient(api_key="vq_...", openai_api_key="sk-...")
Environment variable — VALIQOR_OPENAI_API_KEY
Config file — openai_api_key in .valiqorrc

Your OpenAI key is never stored or persisted by Valiqor. It is only used for the duration of the API request.

Is my data stored? For how long?

Evaluation results, traces, and analysis data are stored in the Valiqor backend database for your team to review in the dashboard.
OpenAI API keys provided via BYOK (Bring Your Own Key) are never stored — they are used only for the duration of the request and then discarded.

Can I use Valiqor in CI/CD pipelines?

Yes. Both the SDK and CLI support fully headless, non-interactive usage.Option 1 — CLI login with credentials:

valiqor login -e user@company.com -p $VALIQOR_PASSWORD
valiqor config set project_name=my-app environment=ci
valiqor eval run --dataset test_data.json --metrics factual_accuracy,coherence

Option 2 — Environment variables (no login needed):

export VALIQOR_API_KEY=vq_...
export VALIQOR_PROJECT_NAME=my-app
python run_evals.py

Option 3 — SDK with explicit key:

from valiqor import ValiqorClient

client = ValiqorClient(api_key="vq_...")
result = client.eval.evaluate(dataset=data, metrics=["factual_accuracy"])

All configuration can be set non-interactively with valiqor config set key=value or through environment variables (VALIQOR_API_KEY, VALIQOR_PROJECT_NAME, VALIQOR_TRACE_DIR, VALIQOR_SCAN_DIR, VALIQOR_BACKEND_URL, VALIQOR_ENVIRONMENT).

What's the difference between Evaluation and Failure Analysis?

	Evaluation	Failure Analysis
Purpose	Score LLM output quality on specific metrics	Find why your AI app fails — root causes, severity, evidence
Access	`client.eval`	`client.failure_analysis`
Output	Per-item metric scores (0–1)	Failure buckets, root-cause evidence, severity (0–5), confidence (0–1)
Metrics	`coherence`, `factual_accuracy`, `hallucination`, etc.	Automatic — failure taxonomy with 30+ subcategories
Approach	LLM-as-judge per metric	Multi-signal analysis
CLI	`valiqor eval run`	`valiqor fa run`

Use Evaluation when you want to measure quality over time. Use Failure Analysis when you want to understand what went wrong and fix it.

How do I trace a multi-step agent?

Three approaches, from zero-config to fully manual:Zero-config auto-instrumentation:

import valiqor.auto  # Patches all detected LLM providers automatically

# Your existing code works as-is — all LLM calls are traced
response = openai.chat.completions.create(...)

Decorator-based:

from valiqor.trace import trace_workflow, trace_function

@trace_workflow("my-agent")
def agent_pipeline(query):
    context = retrieve(query)
    return generate(query, context)

@trace_function()
def retrieve(query):
    ...

@trace_function()
def generate(query, context):
    ...

Manual spans:

from valiqor.trace import TracerV2

tracer = TracerV2()
with tracer.start_trace("agent-query") as trace:
    with tracer.start_span("retrieval"):
        docs = retriever.search(query)
    with tracer.start_span("generation"):
        answer = llm.generate(query, docs)

See Tracing AI Apps for the full guide.

Which Python versions are supported?

Python 3.9+ — tested on Python 3.9, 3.10, 3.11, and 3.12.

Which LLM providers does auto-instrumentation support?

The autolog() function (or import valiqor.auto) auto-instruments these providers:

Provider	Minimum Version
OpenAI	≥ 1.0.0
Anthropic	≥ 0.18.0
LangChain / LangGraph	≥ 0.1.0
Ollama	—
Agno	—

You can also enable specific providers only:

from valiqor.trace.autolog import openai_autolog, anthropic_autolog

openai_autolog()       # Only instrument OpenAI
anthropic_autolog()    # Only instrument Anthropic

For providers without auto-instrumentation, use @trace_workflow and @trace_function decorators.

What are the core dependencies?

The base valiqor package requires:

requests >= 2.31.0
httpx >= 0.25.0
gitingest >= 0.1.0

Optional extras install provider-specific tracing support:

pip install valiqor[openai]      # + openai>=1.0.0
pip install valiqor[anthropic]   # + anthropic>=0.18.0
pip install valiqor[langchain]   # + langchain>=0.1.0, langchain-core>=0.1.0
pip install valiqor[trace]       # openai + anthropic + langchain
pip install valiqor[all]         # Everything

How does config resolution work?

The SDK and CLI resolve configuration values in this order (highest priority first):

Constructor / method parameters — ValiqorClient(api_key="vq_...")
Environment variables — VALIQOR_API_KEY, VALIQOR_PROJECT_NAME, etc.
Local config file — .valiqorrc in the project root
Global credentials (CLI only) — ~/.valiqor/credentials.json
Defaults — built-in defaults

Global credentials (~/.valiqor/credentials.json) are loaded by the CLI only. The SDK’s get_config() function resolves from environment variables and .valiqorrc but does not read global credentials.

Supported environment variables:

Variable	Config Key
`VALIQOR_API_KEY`	`api_key`
`VALIQOR_PROJECT_NAME`	`project_name`
`VALIQOR_OPENAI_API_KEY`	`openai_api_key`
`VALIQOR_TRACE_DIR`	`trace_dir`
`VALIQOR_SCAN_DIR`	`scan_dir`
`VALIQOR_BACKEND_URL`	`backend_url`
`VALIQOR_ENVIRONMENT`	`environment`

What is the maximum dataset size?

1,000 items per request. This applies to evaluate(), audit(), and failure_analysis.run().If your dataset is larger, split it into batches or use the async API for better throughput.Trial users (email not verified) are additionally limited to 25 rows per run and 3 total runs. Verify your email with valiqor verify to remove trial limits.

How many API keys can I create?

Each user can have up to 5 API keys. Manage keys with:

valiqor keys list      # View your keys
valiqor keys create    # Create a new key
valiqor keys delete    # Delete a key

Start Here

Core Workflows

Concepts

Integrations

Resources