RigorMed
← Research resources

Computer science and AI

How to prevent data leakage in machine learning splits

Define who or what the model will predict and when the prediction is made. Multiple records from one device, crops from one image, and repeated observations need a split that reflects the evaluation target.

Computer science and AI

Gather sample IDs, source entity IDs, timestamps, label availability dates, and preprocessing code. Decide whether the score should describe new records, new devices, or future data.

Leakage can still occur after the split. Imputation, scaling, feature selection, or tuning can influence a test score if any of those decisions used test data.

How to do it

  1. Define the target population for the test

    Write one sentence such as “devices absent from training.” For forecasting on existing devices, record the prediction time, feature cutoff, and which historical data would be available at prediction time.

  2. Freeze groups or time boundaries

    For repeated device records, store device IDs, split by group, and check overlap. For forecasting, split chronologically. Add a gap based on the window length if feature or label windows can straddle the boundary.

  3. Fit preprocessing inside training folds

    Fit imputation, scaling, feature selection, and encoding on training folds. Apply the fitted transformations to validation and test data. Use group-aware or time-aware splits during cross-validation and tuning.

  4. Keep a split manifest

    Record input version, sample and group counts, group overlap, windows crossing the cutoff, class balance, and the locked final test set. Attach the split version to the result table.

Worked example

A synthetic dataset contains 6 devices with 20 records each. The goal is to predict faults on unseen devices. This is an example split manifest.

Worked example
CheckExample recordAcceptance check
Group identityTrain A–D, validate E, test FAll pairwise device-ID intersections are empty
CountsTrain 80, validate 20, test 20Report both sample and device counts
PreprocessingFit imputation and scaling on training rowsValidation and test reuse the fitted parameters
Final resultMetric binds to device F and a fixed configurationDisclose that the test has only 1 device

A single held-out device gives a score that can be sensitive to that device. Plan further group-based evaluation around the available devices and report fold-level results and their scope.

Common questions

Does a random seed prevent leakage?

It makes a split repeatable. You still need to check entity, original-file, and time-window overlap.

Can one device appear in both training and testing?

For a forecast on that same device, historical data may be valid. Enforce the time cutoff and interpret the score for that task.

Verified sources

The official sources below were checked on 2026-11-13. Access conditions and update dates are rechecked before publication.

  1. scikit-learn — Common pitfalls and data leakage
  2. scikit-learn — Cross-validation

RigorMed workspace

Audit my evaluation split

Bring your sample, group, and timestamp definitions and split script to draft an overlap, preprocessing, and evaluation audit.