Data wrangling

Overview

Data wrangling (also called data munging) is the umbrella term for the hands-on work of getting raw data into an analysis-ready shape. It typically includes parsing messy source formats, handling missing or malformed values, reshaping tables (pivoting, unpivoting, exploding nested fields), joining multiple sources together, and deriving new columns. It's less a single technique than a phase of work that combines data cleansing, data transformation, and exploratory analysis.

The term is often used interchangeably with "data preparation," though wrangling tends to imply a more interactive, iterative process — an analyst or engineer poking at the data, discovering a problem, fixing it, and re-checking, rather than a fixed, pre-defined pipeline.

Typical wrangling tasks

• Parsing inconsistent file formats (mixed date formats, nested JSON, malformed CSVs)
• Reshaping data between wide and long formats
• Joining and enriching data from multiple sources
• Deriving calculated fields and flags
• Filtering out irrelevant or invalid records
• Renaming and re-typing columns for consistency

Common tools

Data wrangling happens across a range of tools depending on the workflow: pandas or Polars in Python, dplyr in R, spreadsheet formulas for smaller datasets, and SQL for anything that fits in a database or query engine. dbt has also become a common home for wrangling logic once it needs to run repeatably in a pipeline rather than interactively.

Wrangling with SQL and DuckDB

Because DuckDB can query CSV, JSON, and Parquet files directly and supports a fairly complete SQL dialect (pivoting, list/struct types, regular expressions), it's well suited to interactive wrangling without a separate load step:

Copy code
-- inspect a messy CSV in place
SELECT *
FROM read_csv('raw/signups.csv', auto_detect = true)
LIMIT 20;

-- pivot long-format survey responses into wide format
PIVOT survey_responses
ON question
USING FIRST(answer)
GROUP BY respondent_id;

read_csv with auto_detect = true infers delimiters, headers, and types automatically, which removes a lot of the manual setup that wrangling raw files usually requires. Because DuckDB can also run directly against a pandas DataFrame or Arrow table, it's common to bounce between SQL and Python within the same wrangling session rather than committing to one tool.

Why it matters

Most real-world datasets aren't analysis-ready on arrival. Data wrangling is the practical, often underestimated work that determines whether downstream analysis, dashboards, or models are built on a solid foundation — poor wrangling produces subtly wrong results long before anyone notices.