Public medical data preparation guide
How should medical research data be cleaned?
Start by making cleaning actions into a reviewable rules table Requested outputs / delivery goal: A cleaning-rules table, a quality-issue summary and a change-log template.
Data cleaning is not simply deleting blank rows or replacing outliers with means. Researchers must first confirm field meanings, eligible populations and special codes, then freeze reviewable cleaning rules and preserve the originals and every change. How missing values enter statistical analysis is a separate decision for the study plan.
This guide concerns public or synthetic medical research data that you are entitled to use. It describes a conservative path from raw files to analysis-ready data. NHANES documentation illustrates variable-semantic checks, but this is not a universal automatic cleaning formula, nor does cleaned data mean that statistical or ethics review is complete.
A concrete task with RigorMed
Start by making cleaning actions into a reviewable rules table
These are suggested task inputs and outputs to request, not a new execution, a customer outcome or a guarantee of completion.
Materials to provide: A public or synthetic data dictionary you are entitled to use, coding documentation, deterministic quality summaries and candidate cleaning rules.
Requested outputs / delivery goal: A cleaning-rules table, a quality-issue summary and a change-log template.
An example request
For each item, record the original field, issue, proposed rule, evidence and impact. Distinguish blanks, special codes and genuine missingness, and draft a cleaning plan for my confirmation. Do not automatically impute or delete.
What still needs review
Executing cleaning requires separate confirmation. Models must not guess patient rows, missing values or data semantics.
Build the dictionary before deciding what needs cleaning
The same blank or numeric code can mean not asked, refusal, below detection limit or genuinely unrecorded. Establish its meaning from official documentation for the relevant version before changing it.
| Record item | Minimum checks | Role in cleaning |
|---|---|---|
| Source and version | Official files, cycle or version, download and documentation links, and access date | Avoid treating different releases as a single dataset |
| Variable definition | Original column name, label, type, unit, measurement method and permitted values | Prevent identical column names or changed units from being mistaken for equivalent variables |
| Eligible population | Eligibility criteria, questionnaire skip logic, subsamples and record granularity | Distinguish structural non-collection from genuine missingness and establish the denominator |
| Special codes | Variable-specific documentation of refusal, don't know, detection limits and other reserved codes | Map values only according to documented rules; do not treat special codes as continuous values |
| Transformation rules | Target field, original and new values, reason, execution version and affected row count | Make recoding, unit conversion and exclusions traceable |
NHANES Data Documentation, Codebook and Frequencies provide variable descriptions and frequencies. Still check the component, cycle and Analytic Notes. An example dictionary is only a field list; it cannot replace official definitions for the target data source.
Six steps from raw files to reviewable data
Preserve the raw files and state the research question
Confirm data-use terms, study population, main exposure or predictor, outcome and required cycles. Keep original files read-only and record the source version, acquisition date and permitted use. Do not modify the spreadsheet first and derive the study definition afterward.
Bind each variable to the official dictionary
Record types, units, values, eligible populations, questions or measurement methods, skip logic and special codes for core fields. Before joining files, check participant keys, record granularity and duplicates. Before pooling cycles, check measurement and coding comparability.
Describe quality before correcting values
Count raw rows, unique keys, duplicates, missing values and special codes by variable and key population, and values outside documented ranges. An unusual distribution is a signal to investigate, not automatic evidence of an entry error. Suppression, top-coding and detection limits in public data may be part of the design.
Freeze a conservative cleaning plan
Write explicit rules only for documented recoding, type parsing, unit conversion and prespecified derived variables. Specify triggers, original and replacement values, scope and how to stop when a decision cannot be made. Do not silently impute missing values, remove unusual rows or rewrite the raw data.
Execute on a copy and review changes
Output cleaned data, the confirmed dictionary and a change log. Compare before-and-after row counts, unique keys, special codes and main-variable distributions. Every exclusion and merge must explain participant flow. When a check fails, revisit the rules or documentation instead of manually editing a few cells to hide the issue.
Leave missing-data analysis decisions to the plan
Report missing counts and percentages by variable and population, distinguishing not asked, refusal, measurement failure and unknown causes. Complete-case analysis, imputation and sensitivity analyses depend on design, missingness assumptions, models and sampling structure—not an automatic missing-rate threshold. Specify the approach before formal analysis and report what was actually done.
Missingness is not a single category: do not fill every blank with 0 or delete whole rows
Someone not measured because of age, sex, skip logic or subsample design is not equivalent to someone asked but refusing to answer. A laboratory value below the detection limit is not an ordinary blank. Distinguish these cases using the official dictionary and study denominator. If the meaning cannot be established, retain an unknown status and stop automatic conversion.
Filling all missing values with 0 creates observations that never existed. Keeping only complete rows can change the target population and estimate. Imputation is an analysis method requiring advance justification, not a default cleaning patch. Subsequent analysis of complex surveys such as NHANES must also retain applicable weights, strata and PSUs.
Checklist before data enter analysis
Cleaning outputs can move to the next stage only if these questions can be answered:
- Are the raw files, versions, usage terms and official dictionaries traceable?
- Have eligible populations, units, values and special codes been checked for every core variable?
- Are structural non-collection, refusal, missing measurements and unknown causes recorded separately?
- Have join keys, record granularity, duplicates and cross-cycle comparability been checked?
- Does every transformation or exclusion have a rule, reason, affected count and change record?
- Are before-and-after participant flow, key-variable distributions and the dictionary consistent?
- Is there a separate prespecified plan for missing-data handling, sampling design and sensitivity analyses?
STROBE reporting checklists require missing-data handling to be described. A cleaning log establishes what was done; on its own, it cannot establish that the design, statistical methods or conclusions are correct.
Frequently asked questions
Are 7, 9, 77 and 99 always missing-value codes in public data?
Do not infer meaning from the number. Sources, variables and cycles can use different codes, and the same number can even be a valid observation. Open the relevant version's Codebook, Frequencies and questionnaire documentation and confirm field-specific meanings before converting them.
If less than 5% is missing, can rows simply be deleted?
There is no fixed proportion appropriate for all studies. The sample remaining after deletion may differ from the original population, affecting estimates and precision. Describe missingness patterns and causes first, then decide using the design, analysis plan and necessary sensitivity analyses.
Should outliers automatically be truncated or replaced with the mean?
Not automatically. Check units, measurement ranges, top-coding, detection limits and the original source first. Distinguish plausible extreme values from demonstrated errors. Any truncation or exclusion requires a prespecified rationale and change record.
Can analysis or manuscript writing begin immediately after cleaning?
No. The research question, data-use authorization, variable definitions, survey or cohort design, missing-data plan and applicable reporting guidelines still need to be confirmed. Cleaned data are only an input to subsequent controlled analysis.
Verified sources
The official sources below were checked on 2026-09-26 / RigorMed: 2026-10-13. Access conditions and update dates are rechecked before publication.
Start with a specific task
Start by making cleaning actions into a reviewable rules table
Bring legally usable materials and this page's example request to the workspace. Check the pre-execution material and credit notices before confirming, then review the resulting outputs.