# Run the STELQ public benchmark

```bash
curl -fsSLO https://stelq.com/bench/stelq-benchmark.mjs
node stelq-benchmark.mjs --list
node stelq-benchmark.mjs --full --providers stelq --dry-run
export STELQ_KEY="..."
node stelq-benchmark.mjs --full --providers stelq --allow-side-effects --strict
```

Keys stay in local environment variables. The website never receives them. Use
`--questions ./my-questions.json` to replace the answer set, or `--manifest` for all
contracts. Results go to a local `benchmark-results/` directory and deliberately have
no overall score.

`--full` exercises all six contracts. The current default STELQ corpus estimates
$0.243 and includes two potentially long-running Research jobs. It also creates,
inspects and deletes one temporary monitor; `--allow-side-effects` is required.
Use `--suite answers` when you intentionally want only the quick Answers audit.

## Run the current-fact answer comparison

This is the primary answer-quality comparison. It uses 24 facts frozen from
first-party GitHub Releases, npm, and PyPI records before any answer provider was
called. Correctness is exact and mechanical. Official evidence is a separate verdict.

```bash
curl -fsSLO https://stelq.com/bench/live-answers-24.json

# Preview the exact cases and spend.
node stelq-benchmark.mjs --suite live-answers \
  --providers stelq,exa,tavily,linkup --dry-run

# Set only the provider keys you want to test, then run the same frozen matrix.
export STELQ_KEY="..."
export EXA_API_KEY="..."
export TAVILY_API_KEY="..."
export LINKUP_API_KEY="..."
node stelq-benchmark.mjs --suite live-answers \
  --providers stelq,exa,tavily,linkup
```

The report keeps the expected version, asserted version, structured citations,
request receipts, latency, failures, input hash, and raw normalized response for
every call. A correct answer only receives the official-evidence verdict when the
expected version appears in a structured citation record from the pre-registered
registry host.

This contract measures current software-version resolution. It is not an overall
answer-quality score. Use `--live-answers-file` to replace the frozen pack with a
new one before evaluating a later snapshot.

## Run the historical Google FRAMES regression

FRAMES is a separate answer-quality pack because its free-form answers require an
autorater. It remains useful for regression testing, but it is no longer the primary
current answer comparison and is not blended into the six mechanical product contracts.

```bash
curl -fsSLO https://stelq.com/bench/frames-30.json
export STELQ_KEY="..."
export OPENROUTER_API_KEY="..."

# Preview the exact matrix and spend without making calls.
node stelq-benchmark.mjs --suite frames --providers stelq,exa \
  --exa-models exa,exa-pro,exa-research,exa-fast --dry-run

# Add EXA_API_KEY locally to include Exa. Omit exa from --providers otherwise.
export EXA_API_KEY="..."
node stelq-benchmark.mjs --suite frames --providers stelq,exa \
  --exa-models exa,exa-pro,exa-research,exa-fast
```

The bundled set contains the 30 questions selected with Python
`random.Random(75524).sample` from Google’s 824-question FRAMES test set for the
April 2026 comparison. Every provider receives the same frozen prompt in the same
run. Provider names are omitted from the judge prompt. The requested Exa model is
explicit and recorded, and each returned `costDollars` value stays in the artifact.

The runner preserves the original one-pass `openai/gpt-4o-mini` verdict. If a
rejected prediction contains the normalized gold phrase, a second blind adjudicator
checks whether the prediction actually asserts that answer rather than merely
mentioning it. The default adjudicator is `anthropic/claude-sonnet-4.6`; override it
with `--adjudicator-model`. Both verdicts and the adjudicated score remain visible.

This is a reproducible frozen-subset comparison, not a result on all 824 FRAMES
questions. See [FRAMES-NOTICE.md](./FRAMES-NOTICE.md) for attribution and scope.

The runner's `--help` and `--list` output document every supported contract.