Compare LLM configurations on a defined set of test cases

Compare model responses against a fixed benchmark and explicit acceptance rules. Results apply to these test cases, not a universal model ranking; no winner is selected automatically. Evaluation remains with the Python program.

pip install llm-eval · python -m llm_eval

Interface preview

This mock-up shows what a browser interface for the tools could look like. It is not yet connected to the Python tools and does not run their calculations. To use the tools, download and run the code from the respective GitHub repository: github.com/dr-opoku/llm-eval

The code is available under a non-commercial license. Commercial use requires a separate license.

Benchmark

Task, policy and test cases (schema_version 1.0; 1–500 cases; at least one boundaries case expecting reject).

benchmark_sha256 (canonical JSON as in llm_eval.contracts.fingerprint): · Edits change the fingerprint; existing runs will no longer match.

Benchmark details

Benchmark-IDbenchmark_id

recommended

Descriptiondescription

recommended

Model instructionsinstructions

required

policy – sample acceptance rules

Minimum acceptance rate (0–1)minimum_pass_rate

Accepted responses as a share of all expected attempts

Minimum distinct cases (1–500)minimum_distinct_cases

Maximum P95 latency (s)maximum_p95_seconds

null = not required

Maximum cost per accepted result (EUR)maximum_cost_per_accepted_eur

null = not required

Response contract

decision: accept | review | reject
fields.supplier: nonempty string or null
fields.units: integer 1–1,000,000 or null
fields.unit_price_eur: number 0–1,000,000 or null
evidence_ids: unique, nonempty strings
reason: nonempty string
Accepted = strict_json ∧ schema_valid ∧ fields_match ∧ decision_match ∧ evidence_match. The reason text is not evaluated.

cases (0)

Import the bundled benchmark llm_eval/fixtures/benchmark.json or add test cases.

Run

Supplied responses per configuration (1–20 configurations, 1–20 attempts per case, maximum 20 MB).

run_file – run JSON

schema_version 1.0 · provenance.kind synthetic | measured | imported

Command

demo · evaluate · run (live collection through local Ollama).

Subcommandcommand

--benchmark (custom instead of bundled benchmark)

Output format--format

Default: text

Report file--output

empty = stdout

--require-candidate (exit 1 without a candidate)

llm-eval evaluate run.json --format json --output report.json

exit	meaning
0	Report generated or run collected (collection ≠ acceptable responses)
1	--require-candidate and no configuration meets the rules
2	Error (ValueError, OSError, UnicodeError, RecursionError)

Report

schema_version 1.0 · deployment_readiness always not_established

Import the report from llm-eval evaluate run.json --format json --output report.json here.