How to Store Market Data PCAPs: Split, Compress

Storing market data PCAPs well comes down to two decisions: where each capture file begins and ends, and which compressor squeezes it. A PCAP is a packet capture, a file holding every network packet exactly as it arrived on the wire, stamped with the moment it was seen. Every command below runs in a fresh Ubuntu container and builds its own capture from a hex dump, so the whole page reproduces without downloading a single packet.

What is a market data PCAP?

An exchange publishes quotes and trades as a stream of small network packets, the feed. A capture card or a network tap writes those packets to disk before any software decodes them, and the result is a PCAP. It earns its disk space twice over: it carries the arrival timestamp of every packet, and it can be replayed years later through a decoder that did not exist when the day was recorded. The replay side lives in our guide to PCAP market data and replay; this page is about the disk.

Setting up a fresh Ubuntu container

Start from a current Ubuntu image and run everything as root inside it. DEBIAN_FRONTEND=noninteractive stops the Wireshark package from pausing on a question about who may capture packets, and a fixed time zone keeps every timestamp below reproducible.

export DEBIAN_FRONTEND=noninteractive TZ=UTC
apt-get update
apt-get install -y --no-install-recommends wireshark-common zstd xz-utils

wireshark-common carries the four command-line tools used here, text2pcap, capinfos, editcap and mergecap, without the graphical application. zstd and xz-utils are the compressors measured against the gzip already in the base image.

Build a deterministic PCAP with text2pcap

text2pcap turns a text hex dump into a capture file. Each packet begins at offset 000000, and a line of text placed before that offset is read as the packet's timestamp when you pass -t. The dump below holds four made-up quote messages of 32 bytes each: a sequence number, an eight-character symbol, bid and ask in cents, two sizes, and a nanosecond field standing in for the exchange's own clock.

cat > seed.hex <<'EOF'
2026-09-14 09:30:00
000000 00 00 00 01 53 50 59 20 20 20 20 20 00 00 af d2
000010 00 00 af d3 00 00 00 05 00 00 00 0c 0e e6 b2 80
2026-09-14 09:30:01
000000 00 00 00 02 41 41 50 4c 20 20 20 20 00 00 5a 6e
000010 00 00 5a 6f 00 00 00 03 00 00 00 08 1d cd 65 00
2026-09-14 09:30:02
000000 00 00 00 03 53 50 59 20 20 20 20 20 00 00 af d1
000010 00 00 af d2 00 00 00 14 00 00 00 01 2c b4 17 80
2026-09-14 09:30:03
000000 00 00 00 04 4d 53 46 54 20 20 20 20 00 00 c8 00
000010 00 00 c8 01 00 00 00 07 00 00 00 09 3b 9a c9 ff
EOF
text2pcap -q -t '%Y-%m-%d %H:%M:%S' -u 40000,40000 seed.hex seed.tmp
editcap -F pcap seed.tmp seed.pcap
capinfos -c -s -d -M seed.pcap

-u wraps each payload in dummy Ethernet, IP and UDP headers, 42 bytes in all, the shape of a real feed packet on the wire. The editcap step pins the output to classic pcap: older text2pcap releases write pcap by default and newer ones write pcapng, and the conversion makes the rest of this page behave the same on both. capinfos then reports 4 packets, 296 bytes of packet data (4 × 74) and a 384-byte file. That arithmetic is exact. A pcap file is a 24-byte header, then 16 bytes of header per packet, then the packet bytes, and nothing else.

A session-sized file to measure

Four packets say nothing about compression. The generator below writes the same layout for 30 minutes at 100 messages a second, 180,000 packets, with a fixed-seed random walk in the prices and sizes, so the file is identical on every run and every machine.

awk -v rate=100 -v seconds=1800 '
function u32(v) { return sprintf(" %02x %02x %02x %02x",
    int(v / 16777216) % 256, int(v / 65536) % 256, int(v / 256) % 256, v % 256) }
function sym8(str,  i, out) { str = sprintf("%-8s", str); out = ""
    for (i = 1; i <= 8; i++) out = out sprintf(" %02x", code[substr(str, i, 1)])
    return out }
BEGIN {
    split("SPY AAPL MSFT NVDA KO AMZN META TSLA", sym, " ")
    for (i = 32; i < 127; i++) code[sprintf("%c", i)] = i
    for (s = 0; s < 8; s++) px[s] = 5000 + s * 3000
    x = 20260914; seq = 0
    for (t = 0; t < seconds; t++) {
        ts = sprintf("2026-09-14 09:%02d:%02d", 30 + int(t / 60), t % 60)
        for (n = 0; n < rate; n++) {
            x = (x * 48271) % 2147483647; s = x % 8
            x = (x * 48271) % 2147483647; px[s] += (x % 3) - 1
            x = (x * 48271) % 2147483647; bsz = 1 + x % 40
            x = (x * 48271) % 2147483647; asz = 1 + x % 40
            x = (x * 48271) % 2147483647; ns = x % 1000000000
            seq++
            printf "%s\n000000%s%s%s%s%s%s%s\n", ts, u32(seq), sym8(sym[s + 1]),
                u32(px[s]), u32(px[s] + 1), u32(bsz), u32(asz), u32(ns)
        }
    }
}' > quotes.hex
text2pcap -q -t '%Y-%m-%d %H:%M:%S' -u 40000,40000 quotes.hex quotes.tmp
editcap -F pcap quotes.tmp quotes.pcap
capinfos -c -s -u -M quotes.pcap

By the same arithmetic, quotes.pcap holds 180,000 packets in 16,200,024 bytes and spans 1,799 seconds from first packet to last. Every step that follows reads it.

How do you split a large PCAP file?

editcap cuts a capture two ways: -c splits by packet count and -i by time interval in seconds. Each output file gets a running number and the timestamp of its first packet inserted into its name.

mkdir -p by-count by-time
editcap -F pcap -c 50000 quotes.pcap by-count/part.pcap
editcap -F pcap -i 300 quotes.pcap by-time/part.pcap
ls by-count by-time

The count split yields four files, three of 50,000 packets and one of 30,000. The time split yields six files of five minutes each. Time-based pieces are the useful ones for market data: a five-minute file has a known place in the session, while the 50,000th packet lands at a different clock time every day.

Merge captures with mergecap

mergecap reassembles pieces, ordering packets by timestamp across every input (-a concatenates in file order instead). Merging the six time slices gives back the original file, and pcap carries no per-file metadata beyond its 24-byte header, so the claim can be checked byte for byte.

mergecap -F pcap -w merged.pcap by-time/*.pcap
capinfos -c -M quotes.pcap merged.pcap
cmp quotes.pcap merged.pcap && echo identical

Both files report 180,000 packets, and cmp stays silent when the bytes match. That round trip, split then merge then compare, is the test to run before trusting any tool in a storage pipeline.

Which compression is best for PCAP files?

Compression decides the storage bill, so measure rather than assume. The commands below run gzip at its strongest level, xz at its default level and zstd at level 19 on the same 16 MB file; stat prints the resulting sizes in bytes, and two decompression timings follow.

time gzip -9 -c quotes.pcap > quotes.pcap.gz
time xz -c quotes.pcap > quotes.pcap.xz
time zstd -19 quotes.pcap
stat -c '%n %s' quotes.pcap quotes.pcap.gz quotes.pcap.xz quotes.pcap.zst
time zstd -dc quotes.pcap.zst > /dev/null
time xz -dc quotes.pcap.xz > /dev/null

Read the sizes as a ratio: raw bytes divided by compressed bytes. On packet data with repeated headers and a small symbol set, xz and zstd -19 typically land within a few percent of each other and well ahead of gzip -9; xz is usually the slowest to compress, and zstd decompresses several times faster than either. The decompression timing is the one that matters for replay: a compressor is paid once, a decompressor on every read. Byte counts shift slightly between compressor versions, so the manifest below records the tool and level beside each file.

How much storage does a full-day feed capture need?

The synthetic file has less variety than a real feed, so treat its ratio as an upper bound and repeat the three commands on one hour of your own capture before sizing anything. Our measurement of the options quote feed shows why the arithmetic bites: OPRA, the consolidated feed for US listed options, is the highest-volume market data feed in US markets, and a full-day raw capture of it runs to terabytes. A direct exchange feed is smaller and adds up the same way; the SIP versus direct feed comparison covers what each one carries.

The plan is a multiplication. Suppose a feed lands 2 TB of raw packets a day and your measured ratio is 5. That is 400 GB a day on disk, about 100 TB for a year of roughly 250 sessions, and about 200 TB for two years of retention. Each input moves the answer in proportion, which is why the ratio deserves a measurement rather than a guess.

The panel below runs the same multiplication at ratios of 3, 5 and 8 with the 2 TB day and 250 sessions held fixed. The inputs are illustrative, not a measurement of any feed; the middle row is the one quoted above, and the outer rows show how far a weaker or stronger compressor moves the bill.

SELECT
    concat('ratio ', toString(ratio)) AS scenario,
    round(raw_tb_per_day * 1000 / ratio) AS gb_per_day,
    round(raw_tb_per_day * sessions / ratio, 1) AS tb_per_year,
    round(raw_tb_per_day * sessions * 2 / ratio, 1) AS tb_two_years
FROM
(
    SELECT
        2 AS raw_tb_per_day,
        250 AS sessions,
        arrayJoin([3, 5, 8]) AS ratio
)
ORDER BY ratio

Should you store pcap or pcapng, and at what precision?

Classic pcap stores microsecond timestamps. A nanosecond variant of the same format exists, and pcapng declares its precision per interface, carries several interfaces in one file (line A and line B of a feed, say) and records dropped-packet counts. editcap -F converts between all three.

editcap -F nsecpcap quotes.pcap quotes-ns.pcap
editcap -F pcapng quotes.pcap quotes.pcapng
capinfos -t -s -M quotes.pcap quotes-ns.pcap quotes.pcapng

The nanosecond pcap is the same size as the microsecond one; only the magic number in the header differs. The pcapng copy is larger before compression: each packet block carries 32 bytes of framing plus padding to a four-byte boundary against pcap's 16, about a fifth more on 74-byte frames, most of which compression removes again. Precision is the choice to get right at capture time. A capture card stamps arrival in nanoseconds, and converting that file to classic pcap drops the last three digits. Keep the native resolution, and remember that the exchange's timestamp inside each message is a separate clock; our guide to market data timestamps lays out the difference.

How to store market data PCAPs: one file per feed per session

Cut captures at the session boundary and by feed, and write a manifest next to them. capinfos -T prints a header line and one machine-readable line per file, which is most of a manifest already.

capinfos -T -M -c -d -a -e quotes.pcap merged.pcap > manifest.tsv
sha256sum quotes.pcap.zst >> manifest.sha256

The reasons are operational. A truncated file loses one session rather than a year, and the manifest shows which day is missing. Retention becomes deleting the oldest directory instead of rewriting a multi-terabyte file. A replay of one session opens one file and decompresses nothing else. Compression runs one process per file with one file per core, and zstd -T0 adds threads inside a file when a single day is large. A layout such as opra/2026/09/14/opra-2026-09-14-A.pcap.zst turns all of that into a naming rule.

When to convert to a columnar format instead of keeping packets

Packets are the right store while the questions are about the wire: exact replay, or a decoder that might still change. Once the questions turn analytical ("every quote for AAPL between 10:00 and 10:05"), a columnar format wins. Column stores keep each field together, so a symbol column with a handful of values shrinks to a dictionary and a sequence column to small deltas, and a query reads only the columns it names. Whether the decoded messages are order-by-order or price-level updates decides how large that store gets; the MBO versus MBP explainer walks through the difference. A common arrangement keeps compressed packets for the retention window a replay or an audit demands and converts decoded messages to columns for research, so a decoder fix can always be re-run from the packets.

FAQ

Should I store market data as pcap or pcapng?

Both hold the same packets. pcapng adds per-interface timestamp precision, several interfaces per file and dropped-packet counters, at a cost of roughly 16 extra bytes per packet before compression. Classic pcap is smaller and read by every tool, and its nanosecond variant keeps full precision.

How much does a market data PCAP compress?

It depends on the feed, which is why this page measures instead of quoting one number. Binary feeds repeat headers and symbols heavily, so strong compressors such as zstd -19 and xz typically shrink them several-fold; the three commands above, run on one hour of your own capture, give a ratio you can plan with.

Is zstd or xz better for PCAP files?

xz usually produces the slightly smaller file. zstd -19 gets close and decompresses several times faster, which is the cost a replay pipeline pays on every read, and many storage pipelines settle on it for that reason.

How do I split a large PCAP file by time?

editcap -i 300 in.pcap out.pcap writes one file per five-minute interval, each numbered and stamped with its first packet's time. -c splits by packet count instead, and mergecap -w joins the pieces back in timestamp order.

Can I convert market data PCAPs to Parquet and delete the packets?

Conversion is common for research; deleting the packets gives up exact replay and the option to re-decode after a decoder fix. A frequent compromise keeps compressed packets for the retention window that replay or audit requires and derives the columns from them.

Every command above runs unchanged on any current Ubuntu release, which makes this page a script you can paste. To see the decoded side of a feed, the quotes and trades those packets carry, ask for a symbol and a minute on the Strasmore terminal.