Parquet market data
Parquet stock data: columnar storage for market data
Parquet stock data is the standard answer when CSV stops working. Columnar, compressed and typed, partitioned by symbol and date, it lets you read one ticker's history without scanning the rest. Here is how to write and query it with pandas, pyarrow and DuckDB.
Why Parquet for market data
Market data is wide, append-only and queried by column: you want close and volume for a handful of symbols, almost never the whole row set. Parquet matches that shape. It stores data column by column, so reading two of six columns touches a fraction of the file. It compresses well because values repeat and change slowly within a column. It keeps types — timestamps stay timestamps, floats stay floats — so you stop re-parsing dates on every load.
The result is that parquet stock data is routinely an order of magnitude smaller and faster to query than the same rows in CSV, with no schema drift between files.
Partitioning by symbol and date
Partitioning turns one big file into a directory tree the reader can prune. For parquet stock data the two useful keys are symbol and date: partition by symbol when you usually fetch whole histories for a few tickers, and by year or month when you usually fetch a recent window across everything.
Do not over-partition. Millions of tiny files cost more in metadata and open calls than they save, and sub-daily data belongs inside files rather than in path names. A common layout is one directory per symbol, with files split by year inside it.
Write and read a partitioned dataset
import finzdata as yf
import pandas as pd
frames = []
for sym in ["AAPL", "MSFT", "SPY"]:
t = yf.Ticker(sym)
df = t.history(start="2020-01-01", interval="1d")
df["symbol"] = sym
frames.append(df)
panel = pd.concat(frames).reset_index()
panel.to_parquet("bars", partition_cols=["symbol"], engine="pyarrow")
aapl = pd.read_parquet("bars", filters=[("symbol", "=", "AAPL")])Querying with DuckDB
Once the data is partitioned, DuckDB queries it in place with SQL and no loading step:
SELECT symbol, date, close FROM read_parquet('bars/**/*.parquet') WHERE date >= '2024-01-01' ORDER BY date;That reads only the partitions and columns the query needs, which is the whole point of the format. The same files work from pandas, DuckDB, Spark and most warehouses, so you are not locked into one reader.
Sizing and compression tips
- Use Snappy for fast reads and zstd for the smallest files; both are widely supported.
- Sort within each file by timestamp so row-group statistics can skip ranges you do not need.
- Keep row groups in the tens of thousands to hundreds of thousands of rows.
- Store adjusted and unadjusted prices as separate columns rather than separate files, so both are one read away.
- Prefer a few large files per partition over many small ones; tiny files are the usual cause of slow Parquet.
FinzData bulk Parquet
If you would rather not build the pipeline, FinzData serves bulk Parquet files on paid plans: daily, hourly, 5-minute and 1-minute bars for 16,000+ US stocks since January 2000, including delisted names, plus 231 ETFs. Prices are stored as traded, with splits and dividends applied on read via adjust='none', 'splits' or 'all', so one file serves every adjustment convention.
Paid look-back bundles start at $195 for the first 5 years and $95 for each further 5 years, plus $19.95 per month for daily updates. The REST API returns JSON or CSV, and the Python client returns pandas DataFrames either way, so you can prototype small and switch to bulk files when volume demands it.
Frequently asked questions
Why use Parquet instead of CSV for stock data?
Parquet is columnar, compressed and typed, so you read only the columns you need, files are far smaller, and timestamps arrive already parsed. CSV must be read and parsed whole.
How should I partition market data?
By symbol when you fetch whole histories for a few tickers; by year or month when you fetch recent windows across many. Avoid millions of tiny files.
Is Parquet good for tick and intraday data?
Yes. Sub-daily bars compress and sort well inside files; keep time inside the file rather than in the directory path.
How much smaller is Parquet than CSV?
It depends on the data, but an order of magnitude is common for price bars thanks to columnar compression.
Can I query Parquet with SQL?
Yes. DuckDB reads partitioned Parquet directories directly and prunes partitions and columns based on the query.