Data cleaning is unglamorous, time-consuming, and absolutely essential. Studies consistently show that data professionals spend 60–80% of their time cleaning and preparing data before any analysis can begin. Pandas makes this process faster, repeatable, and auditable — the three qualities that matter in professional data work.
Always Start With These Three Commands
Before touching any dataset, run:
df.info() — data types, non-null counts, memory usage
df.describe() — statistical summary of numeric columns
df.isnull().sum() — count of missing values per column
These three commands reveal 90% of data quality issues in under 30 seconds.
Handling Missing Values
Missing values are the most common data quality issue. Your strategy depends on the context:
df.dropna(subset=['critical_column']) — drop rows where a key column is null
df.fillna(0) — replace NULLs with zero (for numeric columns where 0 is meaningful)
df.fillna(df['column'].median()) — fill with median (better than mean for skewed data)
df.fillna(method='ffill') — forward fill (useful for time series data)
Fixing Data Types
Wrong data types cause silent errors — calculations that return NaN instead of numbers, date comparisons that fail silently. Always fix types early:
df['date'] = pd.to_datetime(df['date'], errors='coerce')
df['revenue'] = pd.to_numeric(df['revenue'], errors='coerce')
df['category'] = df['category'].astype('category')
The errors='coerce' argument converts unparseable values to NaN instead of crashing.
Standardizing Text
Inconsistent text destroys GROUP BY accuracy. 'Egypt', 'egypt', 'EGYPT', and ' Egypt ' will be counted as four different countries:
df['country'] = df['country'].str.strip().str.title()
df['email'] = df['email'].str.strip().str.lower()
Always strip whitespace and standardize case before any text-based grouping.
Removing Duplicates
df.duplicated().sum() shows how many duplicate rows exist.
df.drop_duplicates() removes exact duplicates.
df.drop_duplicates(subset=['customer_id', 'order_date']) removes duplicates based on specific columns — useful when you want to keep only the first or last occurrence.
Detecting and Handling Outliers
The IQR method identifies statistical outliers without assuming a distribution:
Q1 = df['sales'].quantile(0.25)
Q3 = df['sales'].quantile(0.75)
IQR = Q3 - Q1
df_clean = df[(df['sales'] >= Q1 - 1.5*IQR) & (df['sales'] <= Q3 + 1.5*IQR)]Always investigate outliers before removing them — they might be data entry errors or genuinely extreme events that need different treatment.
Build a Reusable Pipeline
The real efficiency gain comes from packaging all cleaning steps into a function:
def clean_sales_data(df):
df = df.copy()
df.drop_duplicates(inplace=True)
df['date'] = pd.to_datetime(df['date'], errors='coerce')
df['revenue'] = pd.to_numeric(df['revenue'], errors='coerce')
df['region'] = df['region'].str.strip().str.title()
df.dropna(subset=['date', 'revenue'], inplace=True)
return dfCall this function every time you load the dataset. Consistent, reproducible, auditable.
