Check CSV data for missing values, outdated records and rule violations

Review completeness, freshness, availability and consistency against your declared rules. Prepare inputs and settings, then import the report produced by the Python program.

pip install dataqmonitor · python -m dataqmonitor

Interface preview

This mock-up shows what a browser interface for the tools could look like. It is not yet connected to the Python tools and does not run their calculations. To use the tools, download and run the code from the respective GitHub repository: github.com/dr-opoku/dataqmonitor

The code is available under a non-commercial license. Commercial use requires a separate license.

Inputs

Required CSV data and an optional fetch log. Maximum 5 MiB / 50,000 records per file.

csv – data file

Unique, nonempty column names; timestamps with time zones

--attempts – fetch log

Columns: attempted_at, success (true/false/1/0)

Configuration

JSON object; unknown and duplicate keys are rejected.

✓ Configuration is valid.

Dataset & keys

Report labeldataset_name

Default: Local dataset

Expected recordsexpected_rows

Positive integer; when keys are configured, counts distinct keys · Default: null

Required columnsrequired_columns

null (all columns)

null = all columns; empty list = cell coverage disabled · Default: null

Composite keykey_columns

Duplicate keys violate consistency · Default: []

Freshness & availability

Timestamp columntimestamp_column

Default: null

Maximum age (hours)max_age_hours

Default: 24

Future tolerance (minutes)future_tolerance_minutes

Default: 5

Lookback window (hours)availability_window_hours

Default: 720

Minimum observationsmin_availability_attempts

null → availability assessment unknown · Default: null

Thresholds (minimum percentages)

completenessthresholds.completeness

Default: 95

freshnessthresholds.freshness

Default: 95

availabilitythresholds.availability

Default: 95

consistencythresholds.consistency

Default: 95

numeric_ranges – inclusive ranges

allowed_values – permitted categories

evidence_context – declared context

purposepurpose

≤ 2,000 characters or empty

populationpopulation

≤ 2,000 characters or empty

sampling_methodsampling_method

≤ 2,000 characters or empty

reference_basisreference_basis

≤ 2,000 characters or empty

config.json preview

{
  "dataset_name": "Local dataset",
  "expected_rows": null,
  "required_columns": null,
  "key_columns": [],
  "timestamp_column": null,
  "max_age_hours": 24,
  "future_tolerance_minutes": 5,
  "availability_window_hours": 720,
  "min_availability_attempts": null,
  "evidence_context": {},
  "numeric_ranges": {},
  "allowed_values": {},
  "thresholds": {
    "completeness": 95,
    "freshness": 95,
    "availability": 95,
    "consistency": 95
  }
}

Command

Prepare the command for the Python program.

Subcommandcommand

Reference time (ISO with time zone)--at

empty = current UTC time

Output format--format

Default: text

Output file--output

empty = stdout; must not overwrite an input file

--require-ok (exit 1 for attention/unknown)

dataqmonitor check data.csv --config config.json --format json --output report.json

exit	meaning
0	Report generated (always without --require-ok)
1	--require-ok and status ≠ ok
2	Invalid input (INVALID_INPUT); JSON error on stdout

serve starts the optional MCP server over stdio (tool output uses the same report schema).

Report

schema_version 2.0 · Checks completeness, freshness, availability, consistency

Import the report from dataqmonitor … --format json --output report.json here.