Cleaning data

Drop missing values, remove duplicates, and fix column types in pandas — the three moves that fix most data quality problems.

Knowing the four data quality problems is not enough — you need to fix them. This lesson covers the three pandas methods that handle the most common cases: .dropna(), .drop_duplicates(), and .astype(). The code below creates a deliberately messy DataFrame and cleans it step by step.

Python — editable, runs in your browser

import pandas as pd
import numpy as np

# A dirty dataset: missing values, a duplicate row, and "amount" stored as strings
data = {
  "order_id": [1, 2, 3, 3, 4, 5],
  "customer": ["Alice", "Bob", None, "Charlie", "Diana", "Eve"],
  "amount":   ["120", "85", "200", "200", None, "45"],
  "region":   ["North", "South", "East", "East", "North", "West"],
}
df = pd.DataFrame(data)

print("=== Original ===")
print(df)
print("shape:", df.shape)
print("dtypes:\n", df.dtypes)

# Step 1: Drop rows where ANY column is missing
df_no_nulls = df.dropna()
print("\n=== After dropna() ===")
print(df_no_nulls)

# Step 2: Drop exact duplicate rows (keeps first occurrence)
df_no_dups = df_no_nulls.drop_duplicates()
print("\n=== After drop_duplicates() ===")
print(df_no_dups)

# Step 3: Convert "amount" from object (string) to float
df_clean = df_no_dups.copy()
df_clean["amount"] = df_clean["amount"].astype(float)
print("\n=== After astype(float) on amount ===")
print(df_clean)
print("dtypes:\n", df_clean.dtypes)
print("\nMean amount:", df_clean["amount"].mean())

What each step does

.dropna() removes rows where any value is NaN or None. By default it checks all columns — pass subset=["column_name"] to only check specific ones. The result here drops order 3's duplicate and the row with no customer name.

Wait — it also drops the row where amount is None. After .dropna() you are left with 4 rows. Then .drop_duplicates() removes the second occurrence of order 3 (both rows were identical), leaving 3 rows.

.drop_duplicates() compares entire rows by default. Pass subset=["order_id"] if you want to deduplicate by a specific key column (keeping the first occurrence).

.astype(float) converts the column in-place (on a copy). If any value cannot be converted — say the string "N/A" — pandas will raise a ValueError. That is usually what you want: the error tells you there is another problem to fix, rather than silently producing NaN.

pandas methods like .dropna() and .drop_duplicates() return a new DataFrame by default — they do not modify the original. Assign the result to a new name (as done above) or use inplace=True. Keeping the original around while you experiment is a good habit.

Where to go next

Now you can inspect and clean a dataset. The lab is next: an end-to-end practice session where you apply all of these skills to a new dataset without step-by-step instructions.

Finished reading? Mark it complete to track your progress.

What each step does

Where to go next

On this page