CSV vs. Parquet: A 3-Line Python Snippet That Cut File Size by Over 40%

When building data-driven prototypes—such as my mood-based book recommendation application—it is common to start with a flat CSV file downloaded from an open data repository or exported from a database.

While CSVs are human-readable and universal, they are inefficient for production deployments and interactive apps. While preparing the book recommendation dataset for cloud hosting, the raw CSV was approximately 165 MB. For a free-tier hosting platform like Streamlit Cloud (which has strict RAM limits and git LFS constraints), storing and reading a 165 MB CSV on every container spin-up was uncomfortably slow.

Converting the dataset to Apache Parquet reduced the file size down to ~97 MB (a ~41% reduction) with zero data loss, while dramatically accelerating load times.

Here is the 3-line snippet, why it works, and why you should rarely use CSV for read-heavy workloads.

The 3-Line Conversion

Using Pandas (or Polars) and PyArrow:

import pandas as pd

# 1. Load CSV
df = pd.read_csv("books_dataset.csv")

# 2. Write to Parquet with Snappy compression (default)
df.to_parquet("books_dataset.parquet", engine="pyarrow", compression="snappy")

To read it back into your application:

# 3. Read back in a fraction of the time
df = pd.read_parquet("books_dataset.parquet")

Why Parquet Outperforms CSV

1. Columnar Storage vs. Row-Oriented Text

CSVs are row-oriented text files. Every single record is written out as a string of ASCII/UTF-8 characters separated by commas and newlines. Numbers, booleans, and floats are stored as literal character strings ("12345.67" takes 8 bytes instead of a 4-byte float32).

Parquet is a columnar binary format. Data is grouped by column rather than row. Because values in the same column share the same data type and often contain repeated patterns (e.g., categories, genres, authors), compression algorithms like Snappy or ZSTD can achieve much higher compression ratios than compressing raw row-based text.

2. Built-in Schema and Type Preservation

When you load a CSV into Pandas, the parsing engine has to inspect values line-by-line to infer data types (is "2024-05-12" a string or a datetime? Is "00123" an integer or a postal code?). This type inference step consumes significant CPU time and memory.

Parquet stores schema metadata directly in the file footer:

• Integer columns remain integers.
• Floats remain floats.
• Nulls are tracked via bitmasks rather than parsing empty strings ("" or "NaN").

When Pandas reads Parquet, it maps types directly without guesswork or expensive type casting.

3. Partial Column Reads (Projection Pushdown)

If your application only needs two columns (e.g., book_id and embedding_vector) out of twenty, a CSV parser still has to scan every single byte of every row across the entire 165 MB file.

With Parquet, you can read only the columns you need:

# Reads only the required columns from disk
df_subset = pd.read_parquet(
    "books_dataset.parquet", 
    columns=["book_title", "emotion_label", "rating"]
)

The reader inspects the footer metadata, seeks directly to the byte offsets for those three columns, and ignores the rest.

Benchmark Comparison (Approximate)

In the book recommender application context:

Summary

If you have static datasets, precomputed embeddings, or lookup tables packaged inside your repository or deployed to lightweight containers, convert them to Parquet before pushing to production. It takes less than thirty seconds, saves bandwidth, and reduces initial cold-start latency.