By PCAP Market Data: How Market Replay Works
PCAP market data is the raw exchange packets with a capture timestamp per packet. How the file format works, and when a capture beats a normalized feed.
PCAP market data is the raw exchange traffic written to disk exactly as it arrived: every multicast packet, byte for byte, stamped by the capture host at the moment it hit the network card. Nothing in the file has been normalized or folded into a bar. That is why latency and sequencing research starts with a capture, and why one symbol on one busy day can fill gigabytes.
What is PCAP market data?
PCAP is short for packet capture. It is the file format libpcap and tcpdump have written since the 1990s, and it is what a capture host sitting on a network tap next to a feed handler produces. Exchange feeds are multicast UDP, so a tap sees the same packets the feed handler sees, and the capture host writes them down without interpreting them.
The layout is deliberately simple, which is what lets it be written at line rate. A file opens with one 24 byte global header. After that it repeats a 16 byte record header plus the captured bytes of one packet, over and over, until the file ends. There is no index and no packet count anywhere in it.
The global header carries the fields worth knowing:
- The magic number, four bytes. 0xa1b2c3d4 means microsecond timestamps, and 0xa1b23c4d means nanosecond timestamps. The byte order it appears in on disk also tells a reader whether the writing host was little endian.
- Version major and version minor, two bytes each. They have read 2 and 4 for decades.
- thiszone and sigfigs, four bytes each. Both are zero in practice.
- snaplen, four bytes. The maximum number of bytes the capture host keeps per packet.
- The link type, four bytes. 1 is Ethernet.
Every record header that follows holds four fields: ts_sec, whole seconds since the Unix epoch; ts_usec, the fractional part in microseconds or nanoseconds depending on the magic number; incl_len, the capture length, meaning how many bytes of this packet are actually in the file; and orig_len, the original length, meaning how many bytes were on the wire.
When orig_len is larger than incl_len, the capture host clipped the packet at its snaplen and the tail was never written. A replay of that file hands a feed handler a message that ends mid field. Comparing the two lengths is the first check worth running on a capture someone gives you.
Why the capture timestamp is the interesting one
A normalized feed gives you a record with a timestamp on it, and that stamp belongs to the vendor: it is applied once the vendor's parser has finished with the message. A capture timestamp is applied by the capture card or the kernel on arrival, usually within a microsecond or two of the packet landing, and it is written before anything parses the payload. One measures a pipeline. The other measures a wire.
The same idea is visible inside normalized data. Every consolidated US equity quote carries two clocks: the participant timestamp, written at the exchange, and the SIP timestamp, written when the consolidator processed the message. The distance between them is the step a direct capture skips. SIP and direct exchange feeds covers what that consolidation adds and what it drops.
The exact SQL behind every number
SELECT
formatDateTime(toStartOfInterval(toTimeZone(sip_timestamp, 'America/New_York'), INTERVAL 10 MINUTE), '%H:%i') AS et_time,
count() AS quote_count,
round(quantileDeterministic(0.5)(gap_us, det), 0) AS median_gap_us,
round(quantileDeterministic(0.9)(gap_us, det), 0) AS p90_gap_us
FROM
(
SELECT
sip_timestamp,
toUnixTimestamp64Micro(sip_timestamp) - toUnixTimestamp64Micro(participant_timestamp) AS gap_us,
toUInt64(toUnixTimestamp64Micro(sip_timestamp)) AS det
FROM global_markets.cache_stocks_quotes
WHERE ticker = 'AAPL'
AND sip_timestamp >= '2026-06-10 13:30:00'
AND sip_timestamp < '2026-06-10 15:00:00'
AND participant_timestamp > '2020-01-01 00:00:00'
)
WHERE gap_us BETWEEN 0 AND 1000000
GROUP BY et_time
ORDER BY et_timeOver the first ninety minutes of trading on June 10, 2026, split into ten minute buckets, the gap shows up plainly. In the 09:30 bucket, 93041 AAPL quotes carried both clocks with the exchange stamp at or before the consolidator stamp, and the middle message of that bucket sat 178 microseconds apart on the two. One message in ten sat at least 342 microseconds apart. Those distances are measured on a normalized record, after consolidation. A capture measures the same journey with that step taken out of the measurement.
Build a PCAP by hand, then read it back
The fastest way to stop treating the format as a black box is to build one. The script below runs on a stock Python 3 with no packages installed and no network access. Save it as pcap_demo.py and run it.
# pcap_demo.py (standard library only)
from binascii import unhexlify
import struct
GLOBAL_HEADER = unhexlify(
'd4c3b2a1' # magic 0xa1b2c3d4, written by a little endian host
'0200' # version_major = 2
'0400' # version_minor = 4
'00000000' # thiszone
'00000000' # sigfigs
'08000000' # snaplen = 8 bytes (tiny on purpose, see below)
'01000000' # link type 1 = Ethernet
)
RECORDS = [
unhexlify(
'80e14e68' # ts_sec = 1750000000
'a0860100' # ts_usec = 100000
'08000000' # incl_len = 8 bytes captured
'08000000' # orig_len = 8 bytes on the wire
'4d53472d30303031' # payload MSG-0001
),
unhexlify(
'80e14e68' # ts_sec = 1750000000
'90d00300' # ts_usec = 250000
'08000000'
'08000000'
'4d53472d30303032' # payload MSG-0002
),
unhexlify(
'80e14e68' # ts_sec = 1750000000
'801a0600' # ts_usec = 400000
'08000000' # incl_len = 8 bytes captured
'40000000' # orig_len = 64 bytes on the wire, clipped at snaplen
'4d53472d30303033' # payload MSG-0003
),
]
blob = GLOBAL_HEADER + b''.join(RECORDS)
magic, vmaj, vmin, tz, sigfigs, snaplen, link = struct.unpack('<IHHiIII', blob[:24])
assert magic == 0xa1b2c3d4, 'not a little endian microsecond pcap'
assert (vmaj, vmin) == (2, 4)
print('snaplen', snaplen, 'link type', link)
off = 24
while off < len(blob):
ts_sec, ts_usec, incl_len, orig_len = struct.unpack('<IIII', blob[off:off + 16])
payload = blob[off + 16:off + 16 + incl_len]
off += 16 + incl_len
state = 'clipped' if orig_len > incl_len else 'whole'
print(f'{ts_sec}.{ts_usec:06d} incl={incl_len} orig={orig_len} {state} {payload!r}')It prints four lines:
snaplen 8 link type 1
1750000000.100000 incl=8 orig=8 whole b'MSG-0001'
1750000000.250000 incl=8 orig=8 whole b'MSG-0002'
1750000000.400000 incl=8 orig=64 clipped b'MSG-0003'Three things to notice. The assert on the magic number is the entire file type check: those four bytes read back as a little endian unsigned integer identify both the flavour and the byte order. The version pair has read 2.4 since before most trading systems were written, so pinning it is safe across releases. And the third record declares 8 captured bytes against 64 on the wire, which is what a snaplen clip looks like from the inside. The snaplen here is set to 8 bytes, which no real capture host would do. It is set that small only so one record can show the clip.
Why a bad merge silently reorders a replay
Captures arrive in pieces. A capture host rolls a new file every few minutes or every gigabyte, so one session lands as capture-1.pcap through capture-40.pcap, and a replay needs them as a single ordered stream. Two failure modes live here, and neither raises an error.
The first is filename order. A shell glob sorts lexically, which puts capture-10.pcap ahead of capture-2.pcap. Vendor merge scripts lean on ls -1v for exactly that reason: the -v flag sorts the digits inside a name as numbers rather than as text. Filename order is still only a proxy for time order, and the proxy fails whenever a capture host restarts or two hosts contribute to the same session.
The second is concatenation. Running cat a.pcap b.pcap > merged.pcap looks like it works and produces a corrupt file. The second file's 24 byte global header lands where a record header belongs, and a parser reads its magic number as a packet stamped 2712847316 seconds after the epoch, somewhere in 2055. Mergecap, the merge tool from the Wireshark suite, strips the extra headers and writes the packets out in chronological order.
Append one out of order record to the file built above and the problem is visible in two lines of output.
# add this to the bottom of pcap_demo.py
import tempfile
LATE = unhexlify(
'80e14e68' # ts_sec = 1750000000
'f0490200' # ts_usec = 150000, earlier than the record before it
'08000000'
'08000000'
'4d53472d30303034' # payload MSG-0004
)
blob += LATE
records, off = [], 24
while off < len(blob):
ts_sec, ts_usec, incl_len, orig_len = struct.unpack('<IIII', blob[off:off + 16])
payload = blob[off + 16:off + 16 + incl_len]
records.append(((ts_sec, ts_usec), incl_len, orig_len, payload))
off += 16 + incl_len
stamps = [r[0] for r in records]
print('file order:', [f'{s}.{u:06d}' for s, u in stamps])
print('monotonic:', all(a <= b for a, b in zip(stamps, stamps[1:])))
records.sort(key=lambda r: r[0])
with tempfile.NamedTemporaryFile(suffix='.pcap', delete=False) as fh:
fh.write(GLOBAL_HEADER)
for (ts_sec, ts_usec), incl_len, orig_len, payload in records:
fh.write(struct.pack('<IIII', ts_sec, ts_usec, incl_len, orig_len))
fh.write(payload)
print('rewritten in timestamp order:', fh.name)The parse loop walks the records in file order and the monotonic check comes back false. Nothing else complains. A replay driven straight off that file feeds the 150000 microsecond message after the 400000 microsecond one, and any consumer holding state sees a quote update arrive after the update that already superseded it. On a real session spanning forty files, that defect shows up as a handful of misordered packets at each file boundary, which is precisely where book state gets stitched together. Sorting on the record timestamp before writing, as the second half of the script does, is the fix, and it is why timestamp order rather than filename order is the invariant to test.
A replay that reorders messages is a look ahead problem wearing different clothes: the consumer sees state that did not exist yet at the timestamp it believes it is at. Look ahead bias in backtesting covers the same failure at bar resolution.
How much data is actually in there
Sixty seconds of one symbol, second by second, gives a feel for the volume a capture absorbs.
The exact SQL behind every number
SELECT
formatDateTime(toStartOfSecond(toTimeZone(sip_timestamp, 'America/New_York')), '%H:%i:%S') AS et_time,
count() AS quote_count,
round(quantileDeterministic(0.5)(gap_us, det), 0) AS median_gap_us
FROM
(
SELECT
sip_timestamp,
toUnixTimestamp64Micro(sip_timestamp) - toUnixTimestamp64Micro(participant_timestamp) AS gap_us,
toUInt64(toUnixTimestamp64Micro(sip_timestamp)) AS det
FROM global_markets.cache_stocks_quotes
WHERE ticker = 'AAPL'
AND sip_timestamp >= '2026-06-10 13:30:00'
AND sip_timestamp < '2026-06-10 13:31:00'
AND participant_timestamp > '2020-01-01 00:00:00'
)
WHERE gap_us BETWEEN 0 AND 1000000
GROUP BY et_time
ORDER BY et_timeThe first second in the panel, 09:30:00 ET, carried 528 AAPL quote updates on the consolidated tape alone, with a median clock gap of 48 microseconds. The minute breaks into 60 seconds with quote activity. A direct feed capture of the same minute holds strictly more than that: every packet the exchange sent, including the updates a consolidator collapses, each wrapped in Ethernet, IP and UDP headers.
Quote traffic is what fills a capture disk, and the ratio is not the same across names.
The exact SQL behind every number
SELECT
q.symbol AS symbol,
q.quote_count AS quote_count,
round(q.quote_count / t.trade_count, 1) AS quotes_per_trade
FROM
(
SELECT ticker AS symbol, count() AS quote_count
FROM global_markets.cache_stocks_quotes
WHERE ticker IN ('AAPL', 'MSFT', 'NVDA', 'SPY', 'KO')
AND sip_timestamp >= '2026-06-10 13:30:00'
AND sip_timestamp < '2026-06-10 14:00:00'
GROUP BY ticker
) AS q
INNER JOIN
(
SELECT ticker AS symbol, count() AS trade_count
FROM global_markets.stocks_trades
WHERE ticker IN ('AAPL', 'MSFT', 'NVDA', 'SPY', 'KO')
AND sip_timestamp >= '2026-06-10 13:30:00'
AND sip_timestamp < '2026-06-10 14:00:00'
GROUP BY ticker
) AS t ON q.symbol = t.symbol
WHERE t.trade_count > 0
ORDER BY quotes_per_trade DESCAcross the 5 household names in the panel, over the same half hour, SPY printed 11.3 quote messages for every trade, the highest ratio of the group, on 1035147 quote updates. MSFT sat at the other end of the panel with 0.8. Quotes are the bulk of the bytes in any equity capture, which is also the reason depth of book costs so much more to store than the top of book. Level 1 and level 2 market data explains what those extra messages carry.
What a replay has to reproduce
A capture replays at whatever speed you drive it, and the per packet timestamps are the only thing that keeps the replay honest. Message rate across a session is nowhere near constant.
The exact SQL behind every number
SELECT
formatDateTime(toStartOfInterval(toTimeZone(window_start, 'America/New_York'), INTERVAL 15 MINUTE), '%H:%i') AS et_time,
round(sumIf(transactions, ticker = 'AAPL') / 900, 2) AS aapl_prints_per_sec,
round(sumIf(transactions, ticker = 'SPY') / 900, 2) AS spy_prints_per_sec
FROM global_markets.delayed_stocks_minute_aggs
WHERE ticker IN ('AAPL', 'SPY')
AND window_start >= '2026-06-10 08:00:00'
AND window_start < '2026-06-11 00:00:00'
GROUP BY et_time
ORDER BY et_timeA print is a completed trade report. The panel buckets every AAPL and SPY print into fifteen minute buckets on the New York clock, across the whole electronic day rather than the regular session alone. The earliest bucket with any activity, 04:00 ET, ran 10.45 AAPL prints per second against 3.12 for SPY. Read across the chart into the regular session and the shape does the teaching. Replay engines schedule off the per packet timestamp and sleep between packets instead of counting messages, and a replay clocked at a fixed rate reproduces no part of that curve.
When a capture is worth it
A capture earns its storage when the question is about time or order:
- Latency measurement, where the microseconds go between the wire, the feed handler and the order gateway. Only a capture holds a timestamp taken before your own software touched the packet.
- Gap and arbitration research, covering how often a feed drops packets and how quickly the B side of an A and B feed pair fills the hole.
- Feed handler testing, replaying a real session against a new parser to see whether it rebuilds the same book.
- Reconstruction disputes, proving what the book looked like at a given microsecond in a form both sides can parse independently.
A normalized feed is the better tool everywhere else. It carries one schema across venues, corporate actions applied, symbology cleaned up, and it fits in memory. Research at bar resolution never needs packets at all: how OHLCV bars are built walks through the aggregation step and where it hides detail. Captures also carry a real bill in tap ports and in storage that grows every session, on top of the data licences themselves, which what real time market data costs breaks down.
FAQ
What is a PCAP file in market data?
A PCAP file is a packet capture: the raw multicast packets of an exchange feed saved in arrival order, each preceded by a 16 byte header holding the capture timestamp, the number of bytes saved and the number of bytes that were on the wire. It contains the feed's own binary protocol, not a normalized quote or trade record.
Why are market data PCAP files so large?
Every packet is kept whole, including the Ethernet, IP and UDP headers, and quote updates outnumber trades by a wide margin on liquid names. The panel above measured that ratio for a handful of household symbols over a single half hour.
What is the difference between capture length and original length?
Capture length (incl_len) is how many bytes of the packet are in the file. Original length (orig_len) is how many bytes were on the wire. When the original length is larger, the capture host clipped the packet at its snaplen setting and the remainder was never written to disk.
Why must PCAP captures be merged in timestamp order?
A session usually arrives as many rolling files, and both filename sorting and plain concatenation can place packets in the wrong sequence. A consumer replaying an out of order stream sees updates arrive after the updates that replaced them, with no error raised anywhere in the chain.
Do I need PCAP data to backtest a trading strategy?
No. Bar data and normalized tick data answer most research questions and cost far less to store and query. Captures matter when the question is specifically about latency, packet loss, or the exact order in which a system saw its messages.
Every panel on this page ships with the SQL that produced it, expandable underneath the chart. To count the messages in a session yourself, or compare the two clocks on a single quote, ask the question in plain English on the Strasmore terminal.