Prompt
How do I check whether a dataset has missing values, duplicates, or bias before using it?
Latest observation
Before using a dataset, it’s a good idea to do a quick data quality and bias check. Here’s a practical checklist:
1) Check for missing values
Look for empty cells, NaN, NULL, blank strings, or placeholders like "?" or "unknown".
What to inspect
- Count missing values per column
- Calculate the percentage missing per column
- Check whether missingness is concentrated in certain groups or time periods
Example in Python
import pandas as pd
df.isna().sum() # missing count per column
df.isna().mean() * 100 # missing percentage per column
If missing values are represented by custom strings:
df.replace(["?", "unknown", ""], pd.NA).isna().sum()
2) Check for duplicates
Duplicates can mean:
- Exact duplicate rows
- Duplicate IDs or records that should be unique
- Near-duplicates in text or images
What to inspect
- Total duplicate rows
- Duplicates based on key columns like
id,email, ortransaction_id
Example in Python
df.duplicated().sum() # exact duplicate rows
df[df.duplicated()] # view duplicate rows
For a unique key:
df[df.duplicated(subset=["id"])]
3) Check for bias
Bias is harder to detect than missing values or duplicates. You want to see whether the dataset overrepresents or underrepresents certain groups, outcomes, or patterns.
Common bias checks
a) Representation bias
Are some groups underrepresented?
- Gender
- Age group
- Race/ethnicity
- Geography
- Device type
- Class labels
Example:
df["group"].value_counts(normalize=True)
Compare the dataset distribution to the real population if possible.
b) Missingness bias
Are missing values more common in some groups than others?
df.groupby("group")["target_column"].apply(lambda x: x.isna().mean())
c) Label imbalance
If you’re doing classification, check whether one class dominates.
df["label"].value_counts(normalize=True)
d) Sampling bias
Ask:
- How was the data collected?
- Was it from one source only?
- Does it exclude certain time periods, locations, or user types?
e) Outcome bias / target leakage
Check whether any features indirectly reveal the target in a way that wouldn’t be available in real use.
4) Check basic consistency and outliers
These often reveal hidden data issues:
- Impossible values, like negative ages
- Inconsistent categories, like
"Male","male","M" - Extreme outliers
df.describe(include="all")
df["age"].min(), df["age"].max()
df["category"].value_counts(dropna=False)
5) Use a quick automated report
Tools like these can save time:
- pandas profiling / ydata-profiling
- Great Expectations
- Sweetviz
Example:
from ydata_profiling import ProfileReport
ProfileReport(df).to_file("report.html")
6) Ask these questions before trusting the dataset
- Is the dataset complete enough for the task?
- Are missing values random or systematic?
- Are duplicates legitimate or accidental?
- Does the dataset reflect the population I care about?
- Could the data collection process have introduced bias?
Simple rule of thumb
A dataset is usually safer to use if:
- Missing values are low or well-understood
- Duplicates are removed or justified
- The sample is representative of the real-world population
- Biases are known and addressed
If you want, I can also give you:
- a Python script that runs all these checks automatically, or
- a checklist for Excel/CSV datasets without coding.