IDS²Eval audits network-intrusion datasets and evaluation conditions before classifier benchmarking.
Check duplication, label conflicts, shortcut features, split leakage, temporal artefacts and known dataset issues. Re-audit after configured deduplication, run the selected stress checks, benchmark classifiers, and keep the run artifacts.
A classifier score can reflect attack behaviour, dataset construction, or both. IDS²Eval checks the dataset and split conditions before benchmarking.
Exact duplicates and repeated flow identities can place effectively familiar traffic on both sides of a split; separate near-duplicate checks expose confusable samples.
Ports, addresses, timestamps, collection order and a single dominant feature can become class proxies even when the model appears accurate.
Some results move when the split, seed, feature set, scenario or deduplication policy changes. IDS²Eval can measure that sensitivity directly.
IDS²Eval focuses on NIDS dataset integrity, falsification checks, configured preprocessing, classifier benchmarking and the artifacts needed to trace a run.
Inspect sample leakage, label conflicts, identity and temporal shortcuts, scenario artefacts, data integrity, provenance and result stability.
When deduplication is enabled, IDS²Eval audits the raw split, deduplicates it, then reruns checks that depend on row content. Structural checks are reused.
Grouped resplits and opt-in stress checks compare results across stronger split, seed, scenario and feature-set conditions.
Run binary and optional attack-type benchmarks across configured scaling and sampling strategies, with per-class metrics and timing.
Checks with a defined config-level remedy can generate a patch. Other findings remain review items with their recorded details.
Keep scorecards, benchmark tables, fingerprints, environment details, resolved config and machine-readable evidence under a fresh run directory.
The shipped UNSW-NB15 scorecard reports 42.38% feature-space duplication in the configured training split before deduplication. The same run records the post-deduplication state.
Training feature-space duplication was flagged in the shipped validation run.
The run keeps both audit states alongside the deduplication statistics.
Deduplication disproportionately affected one class, so the benchmark should be interpreted in light of the changed distribution.
Each check returns a status (ok, warning, flag, or not applicable when it has no input to test), a summary and details. The scorecard links those findings to the run evidence and lists the checks a run left disabled. Nineteen checks are enabled by default and nine are opt-in.
Exact feature-space duplicates within and across train/test.
Identical feature vectors mapped to contradictory labels.
Declared flow identities that appear on both sides of the split; no-ops when no flow-ID columns are configured.
Fast model importance to expose suspiciously dominant features.
A depth-1 tree tests whether one simple rule already explains too much.
Standalone predictive power of declared IP-, port-, MAC- and similar identity columns.
When a timestamp column is declared, tests whether time alone acts as a class proxy for collection scenarios.
When group columns are available, re-evaluates under a split that blocks session-correlated leakage.
When a scenario column is declared, tests generalization with one collection scenario held out.
Compares performance across available split, dedup and feature-set conditions.
Missing labels or feature values, constant or near-constant features, and infinite values.
Curated dataset facts and feature-extractor signatures with documented defects.
The CLI and local dashboard use the same core pipeline. The dashboard opens with a pre-filled starter config.
Load CSV, Parquet, CSV-formatted .txt, compressed CSV, or a named CSV/Parquet member from ZIP. Then validate the schema and fingerprint the data.
Run enabled checks before deduplication and save the untouched evidence.
Deduplicate when enabled. Then rerun data-dependent checks and reuse structural findings.
Run the configured binary and optional attack-type benchmarks across the selected scaling and sampling strategies.
Write scorecards, benchmark detail, fingerprints, environment and run status.
Browse datasets and runs, open check evidence, apply supported recommendation patches to a new config, compare datasets, view classifier results, and inspect provenance and scorecards.
The registry contains 14 supported classifiers, including stacking and voting ensembles. Runs can include binary detection and, when configured, attack-type classification, with accuracy, weighted F1, macro F1, AUC, training time, inference time and per-class metrics.
When tuning is enabled, scaling and sampling stay inside the imbalanced-learn pipeline passed to GridSearchCV, so they are fitted independently inside each CV fold.
Benchmark outputs include confusion matrices, per-class precision/recall/F1, feature importance where supported, best hyperparameters when tuning runs, and class distributions before/after sampling.
The repository includes scorecards produced from real data for eight published NIDS datasets. They cover classic connection records, modern flow datasets, IoT traffic and DDoS-focused corpora.
Each run gets a fresh directory. Human-readable scorecards sit beside machine-readable findings, benchmark detail, configuration, environment and content fingerprints so the result can be traced back to the data and decisions that produced it.
# shipped example: examples/unsw-nb15-scorecard/scorecard.json # condensed below for the marketing view [audit:before] dedup_check ............... FLAG 74,301 train duplicates (42.38%) label_conflict_check ...... FLAG 1,758 rows in conflicting vectors row_order_leakage_check ... FLAG known_issue_lookup ........ WARN [dedup] train 175,341 → 101,040 test 82,332 → 52,644 [audit:after] dedup_check ............... OK 0.00% [scorecard] REVIEW REQUIRED · 9 ok · 2 warnings · 1 flag · 7 not applicable
IDS²Eval is AGPL-3.0-only free software. The licence permits use, study, modification and redistribution, including commercial use, subject to its terms. Modified versions offered over a network remain subject to the AGPL source-sharing requirements.
Review the audit findings, benchmark under the recorded configuration, and keep the artifacts from the run.