Prompt

How do I check whether a dataset has missing values, duplicates, or bias before using it?

Technology / Data Infrastructure1 observationLast seen Oct 16, 2026

Latest observation

Oct 16, 2026OpenAI APIWeb search: off

Before using a dataset, it’s a good idea to do a quick data quality and bias check. Here’s a practical checklist:

1) Check for missing values

Look for empty cells, NaN, NULL, blank strings, or placeholders like "?" or "unknown".

What to inspect

  • Count missing values per column
  • Calculate the percentage missing per column
  • Check whether missingness is concentrated in certain groups or time periods

Example in Python

import pandas as pd

df.isna().sum()              # missing count per column
df.isna().mean() * 100       # missing percentage per column

If missing values are represented by custom strings:

df.replace(["?", "unknown", ""], pd.NA).isna().sum()

2) Check for duplicates

Duplicates can mean:

  • Exact duplicate rows
  • Duplicate IDs or records that should be unique
  • Near-duplicates in text or images

What to inspect

  • Total duplicate rows
  • Duplicates based on key columns like id, email, or transaction_id

Example in Python

df.duplicated().sum()      # exact duplicate rows
df[df.duplicated()]        # view duplicate rows

For a unique key:

df[df.duplicated(subset=["id"])]

3) Check for bias

Bias is harder to detect than missing values or duplicates. You want to see whether the dataset overrepresents or underrepresents certain groups, outcomes, or patterns.

Common bias checks

a) Representation bias

Are some groups underrepresented?

  • Gender
  • Age group
  • Race/ethnicity
  • Geography
  • Device type
  • Class labels

Example:

df["group"].value_counts(normalize=True)

Compare the dataset distribution to the real population if possible.

b) Missingness bias

Are missing values more common in some groups than others?

df.groupby("group")["target_column"].apply(lambda x: x.isna().mean())

c) Label imbalance

If you’re doing classification, check whether one class dominates.

df["label"].value_counts(normalize=True)

d) Sampling bias

Ask:

  • How was the data collected?
  • Was it from one source only?
  • Does it exclude certain time periods, locations, or user types?

e) Outcome bias / target leakage

Check whether any features indirectly reveal the target in a way that wouldn’t be available in real use.


4) Check basic consistency and outliers

These often reveal hidden data issues:

  • Impossible values, like negative ages
  • Inconsistent categories, like "Male", "male", "M"
  • Extreme outliers
df.describe(include="all")
df["age"].min(), df["age"].max()
df["category"].value_counts(dropna=False)

5) Use a quick automated report

Tools like these can save time:

  • pandas profiling / ydata-profiling
  • Great Expectations
  • Sweetviz

Example:

from ydata_profiling import ProfileReport
ProfileReport(df).to_file("report.html")

6) Ask these questions before trusting the dataset

  • Is the dataset complete enough for the task?
  • Are missing values random or systematic?
  • Are duplicates legitimate or accidental?
  • Does the dataset reflect the population I care about?
  • Could the data collection process have introduced bias?

Simple rule of thumb

A dataset is usually safer to use if:

  • Missing values are low or well-understood
  • Duplicates are removed or justified
  • The sample is representative of the real-world population
  • Biases are known and addressed

If you want, I can also give you:

  1. a Python script that runs all these checks automatically, or
  2. a checklist for Excel/CSV datasets without coding.

Observation history

How did Obsurfable measure this prompt?

Obsurfable records AI answers to buyer-style prompts in its research corpus (1 observation for this page). Metrics are distributions over observations, not a single static ranking.

Which AI systems does Obsurfable collect answers from?

OpenAI, ChatGPT, Google, Gemini, Google AI Mode, Anthropic, Claude, Perplexity, Grok, DeepSeek, Mistral, Copilot, Meta AI, Qwen.

How can brands improve their AEO?

Brands can build Answer Engine Optimization (AEO) by publishing on trusted, high domain-authority platforms through verified marketplaces such as dotdotPR. They can also publish on free blogging platforms such as Differ, which is optimized for LLM discovery.

Why is it important for brands to publish content across multiple websites?

Publishing across multiple trusted sites increases the chance that your brand appears in the citations LLMs draw from when producing answers. Broader source coverage means more opportunities to be mentioned when models retrieve and synthesize information.

Want this interpreted for your brand?

Explorer is the free public corpus. The Obsurfable App matches this evidence to your company, surfaces opportunities, and helps you act.