> ## Documentation Index
> Fetch the complete documentation index at: https://docs.valiqor.com/llms.txt
> Use this file to discover all available pages before exploring further.

# FAQ

> Frequently asked questions about Valiqor — architecture, data handling, integrations, and usage.

<AccordionGroup>
  <Accordion title="What LLMs does Valiqor use for evaluation?">
    Valiqor uses a **state-of-the-art LLM** as the default judge for all LLM-based evaluation metrics (e.g. `hallucination`, `coherence`, `factual_accuracy`).

    You can also **bring your own OpenAI API key** so that judge calls use your own quota. Pass it at any of these levels (highest priority wins):

    1. Method parameter — `client.eval.evaluate(dataset=..., openai_api_key="sk-...")`
    2. Client constructor — `ValiqorClient(api_key="vq_...", openai_api_key="sk-...")`
    3. Environment variable — `VALIQOR_OPENAI_API_KEY`
    4. Config file — `openai_api_key` in `.valiqorrc`

    Your OpenAI key is **never stored or persisted** by Valiqor. It is only used for the duration of the API request.
  </Accordion>

  <Accordion title="Is my data stored? For how long?">
    * **Evaluation results, traces, and analysis data** are stored in the Valiqor backend database for your team to review in the dashboard.
    * **OpenAI API keys** provided via BYOK (Bring Your Own Key) are **never stored** — they are used only for the duration of the request and then discarded.
  </Accordion>

  <Accordion title="Can I use Valiqor in CI/CD pipelines?">
    Yes. Both the SDK and CLI support fully headless, non-interactive usage.

    **Option 1 — CLI login with credentials:**

    ```bash theme={"system"}
    valiqor login -e user@company.com -p $VALIQOR_PASSWORD
    valiqor config set project_name=my-app environment=ci
    valiqor eval run --dataset test_data.json --metrics factual_accuracy,coherence
    ```

    **Option 2 — Environment variables (no login needed):**

    ```bash theme={"system"}
    export VALIQOR_API_KEY=vq_...
    export VALIQOR_PROJECT_NAME=my-app
    python run_evals.py
    ```

    **Option 3 — SDK with explicit key:**

    ```python theme={"system"}
    from valiqor import ValiqorClient

    client = ValiqorClient(api_key="vq_...")
    result = client.eval.evaluate(dataset=data, metrics=["factual_accuracy"])
    ```

    All configuration can be set non-interactively with `valiqor config set key=value` or through environment variables (`VALIQOR_API_KEY`, `VALIQOR_PROJECT_NAME`, `VALIQOR_TRACE_DIR`, `VALIQOR_SCAN_DIR`, `VALIQOR_BACKEND_URL`, `VALIQOR_ENVIRONMENT`).
  </Accordion>

  <Accordion title="What's the difference between Evaluation and Failure Analysis?">
    |              | Evaluation                                             | Failure Analysis                                                       |
    | ------------ | ------------------------------------------------------ | ---------------------------------------------------------------------- |
    | **Purpose**  | Score LLM output quality on specific metrics           | Find *why* your AI app fails — root causes, severity, evidence         |
    | **Access**   | `client.eval`                                          | `client.failure_analysis`                                              |
    | **Output**   | Per-item metric scores (0–1)                           | Failure buckets, root-cause evidence, severity (0–5), confidence (0–1) |
    | **Metrics**  | `coherence`, `factual_accuracy`, `hallucination`, etc. | Automatic — failure taxonomy with 30+ subcategories                    |
    | **Approach** | LLM-as-judge per metric                                | Multi-signal analysis                                                  |
    | **CLI**      | `valiqor eval run`                                     | `valiqor fa run`                                                       |

    **Use Evaluation** when you want to measure quality over time. **Use Failure Analysis** when you want to understand *what went wrong* and fix it.
  </Accordion>

  <Accordion title="How do I trace a multi-step agent?">
    Three approaches, from zero-config to fully manual:

    **Zero-config auto-instrumentation:**

    ```python theme={"system"}
    import valiqor.auto  # Patches all detected LLM providers automatically

    # Your existing code works as-is — all LLM calls are traced
    response = openai.chat.completions.create(...)
    ```

    **Decorator-based:**

    ```python theme={"system"}
    from valiqor.trace import trace_workflow, trace_function

    @trace_workflow("my-agent")
    def agent_pipeline(query):
        context = retrieve(query)
        return generate(query, context)

    @trace_function()
    def retrieve(query):
        ...

    @trace_function()
    def generate(query, context):
        ...
    ```

    **Manual spans:**

    ```python theme={"system"}
    from valiqor.trace import TracerV2

    tracer = TracerV2()
    with tracer.start_trace("agent-query") as trace:
        with tracer.start_span("retrieval"):
            docs = retriever.search(query)
        with tracer.start_span("generation"):
            answer = llm.generate(query, docs)
    ```

    See [Tracing AI Apps](/workflows/tracing) for the full guide.
  </Accordion>

  <Accordion title="Which Python versions are supported?">
    **Python 3.9+** — tested on Python 3.9, 3.10, 3.11, and 3.12.
  </Accordion>

  <Accordion title="Which LLM providers does auto-instrumentation support?">
    The `autolog()` function (or `import valiqor.auto`) auto-instruments these providers:

    | Provider              | Minimum Version |
    | --------------------- | --------------- |
    | OpenAI                | ≥ 1.0.0         |
    | Anthropic             | ≥ 0.18.0        |
    | LangChain / LangGraph | ≥ 0.1.0         |
    | Ollama                | —               |
    | Agno                  | —               |

    You can also enable specific providers only:

    ```python theme={"system"}
    from valiqor.trace.autolog import openai_autolog, anthropic_autolog

    openai_autolog()       # Only instrument OpenAI
    anthropic_autolog()    # Only instrument Anthropic
    ```

    For providers without auto-instrumentation, use `@trace_workflow` and `@trace_function` decorators.
  </Accordion>

  <Accordion title="What are the core dependencies?">
    The base `valiqor` package requires:

    * `requests >= 2.31.0`
    * `httpx >= 0.25.0`
    * `gitingest >= 0.1.0`

    Optional extras install provider-specific tracing support:

    ```bash theme={"system"}
    pip install valiqor[openai]      # + openai>=1.0.0
    pip install valiqor[anthropic]   # + anthropic>=0.18.0
    pip install valiqor[langchain]   # + langchain>=0.1.0, langchain-core>=0.1.0
    pip install valiqor[trace]       # openai + anthropic + langchain
    pip install valiqor[all]         # Everything
    ```
  </Accordion>

  <Accordion title="How does config resolution work?">
    The SDK and CLI resolve configuration values in this order (highest priority first):

    1. **Constructor / method parameters** — `ValiqorClient(api_key="vq_...")`
    2. **Environment variables** — `VALIQOR_API_KEY`, `VALIQOR_PROJECT_NAME`, etc.
    3. **Local config file** — `.valiqorrc` in the project root
    4. **Global credentials** (CLI only) — `~/.valiqor/credentials.json`
    5. **Defaults** — built-in defaults

    <Note>
      Global credentials (`~/.valiqor/credentials.json`) are loaded by the CLI only. The SDK's `get_config()` function resolves from environment variables and `.valiqorrc` but does not read global credentials.
    </Note>

    Supported environment variables:

    | Variable                 | Config Key       |
    | ------------------------ | ---------------- |
    | `VALIQOR_API_KEY`        | `api_key`        |
    | `VALIQOR_PROJECT_NAME`   | `project_name`   |
    | `VALIQOR_OPENAI_API_KEY` | `openai_api_key` |
    | `VALIQOR_TRACE_DIR`      | `trace_dir`      |
    | `VALIQOR_SCAN_DIR`       | `scan_dir`       |
    | `VALIQOR_BACKEND_URL`    | `backend_url`    |
    | `VALIQOR_ENVIRONMENT`    | `environment`    |
  </Accordion>

  <Accordion title="What is the maximum dataset size?">
    **1,000 items per request**. This applies to `evaluate()`, `audit()`, and `failure_analysis.run()`.

    If your dataset is larger, split it into batches or use the async API for better throughput.

    Trial users (email not verified) are additionally limited to **25 rows per run** and **3 total runs**. Verify your email with `valiqor verify` to remove trial limits.
  </Accordion>

  <Accordion title="How many API keys can I create?">
    Each user can have up to **5 API keys**. Manage keys with:

    ```bash theme={"system"}
    valiqor keys list      # View your keys
    valiqor keys create    # Create a new key
    valiqor keys delete    # Delete a key
    ```
  </Accordion>
</AccordionGroup>