Monitor prediction accuracy, costs and response times

Review predictions against actual outcomes: accuracy, outcome coverage, confidence, cost per correct result, latency and changes from a baseline. Calculations remain with the Python program.

pip install resqmonitor · python -m resqmonitor

Interface preview

This mock-up shows what a browser interface for the tools could look like. It is not yet connected to the Python tools and does not run their calculations. To use the tools, download and run the code from the respective GitHub repository: github.com/dr-opoku/resqmonitor

The code is available under a non-commercial license. Commercial use requires a separate license.

Inputs

Required current records and an optional baseline in the same format.

csv – current records

ID, prediction, actual outcome; optional confidence, cost, latency_ms, group

--baseline – comparison records

ID, prediction, actual outcome; optional confidence, cost, latency_ms, group

Configuration

Partial configurations are allowed; each role must use a distinct column.

✓ Configuration is valid.

Column roles

Report labeldataset_name

Default: Local classification results

Unique record IDid_column

Default: record_id

Predictionprediction_column

Default: predicted_label

Actual outcome (empty = pending)outcome_column

Default: actual_label

confidenceconfidence_column

null disables the role · Default: confidence

costcost_column

null disables the role · Default: cost

latencylatency_column

null disables the role · Default: latency_ms

groupgroup_column

null disables the role · Default: group

Accuracy & coverage

Minimum evaluated rowsmin_evaluated_rows

applies to accuracy, confidence, costs and baseline · Default: 30

Minimum accuracy (%)min_accuracy_pct

Default: 85

Assessment basisaccuracy_threshold_basis

wilson_lower_bound = lower 95% confidence bound · Default: point_estimate

Minimum outcome coverage (%)min_outcome_coverage_pct

Default: 90

Maximum confidence gap (pp)max_confidence_gap_pp

Default: 10

Comparison, costs & latency

Allowed accuracy drop from baseline (pp)allowed_accuracy_drop_pp

Default: 10

Maximum group distribution shift (%)max_case_mix_shift_pct

Total variation of group proportions · Default: 20

Cost unitcost_unit

no currency conversion · Default: EUR

Maximum cost per correct resultmax_cost_per_correct

Default: null

Maximum mean latency (ms)max_mean_latency_ms

Default: null

evidence_context – declared context

purposepurpose

≤ 2,000 characters or empty

populationpopulation

≤ 2,000 characters or empty

sampling_methodsampling_method

≤ 2,000 characters or empty

reference_basisreference_basis

≤ 2,000 characters or empty

config.json preview

{
  "dataset_name": "Local classification results",
  "id_column": "record_id",
  "prediction_column": "predicted_label",
  "outcome_column": "actual_label",
  "confidence_column": "confidence",
  "cost_column": "cost",
  "latency_column": "latency_ms",
  "group_column": "group",
  "cost_unit": "EUR",
  "min_evaluated_rows": 30,
  "min_accuracy_pct": 85,
  "accuracy_threshold_basis": "point_estimate",
  "evidence_context": {},
  "min_outcome_coverage_pct": 90,
  "max_confidence_gap_pp": 10,
  "allowed_accuracy_drop_pp": 10,
  "max_case_mix_shift_pct": 20,
  "max_cost_per_correct": null,
  "max_mean_latency_ms": null
}

Command

Prepare the command for the Python program.

Subcommandcommand

Evaluation time (ISO with time zone)--at

empty = current UTC time

Output format--format

Default: text

Output file--output

empty = stdout

--require-ok (exit 1 for attention/unknown)

resqmonitor check results.csv --config config.json --format json --output report.json

exit	meaning
0	Report generated
1	--require-ok and status ≠ ok
2	Invalid input (INVALID_INPUT)

Report

schema_version 2.0 · accuracy, outcome_coverage, confidence_alignment, cost_per_correct, latency, baseline_accuracy, case_mix

Import the report from resqmonitor … --format json --output report.json here.