Wetin be PCAP market data and how market replay dey work

PCAP market data na the raw exchange traffic wey dem write go disk exactly as e reach: every multicast packet, byte by byte, wey the capture host stamp the moment e hit the network card. Nothing inside the file don undergo normalization or bar aggregation. Na why latency and sequencing research dey start with capture, and why one symbol for one busy day fit fill gigabytes.

Wetin be PCAP market data?

PCAP na short form for packet capture. Na the file format wey libpcap and tcpdump don dey write since the 1990s, and na wetin capture host wey dey sit on top network tap near feed handler dey produce. Exchange feeds na multicast UDP, so tap dey see the same packets wey feed handler dey see, and capture host dey write dem down without interpreting dem.

The layout dey simple, wey be the reason e fit write at line rate. File dey open with one 24 byte global header. After that, e dey repeat 16 byte record header plus the captured bytes of one packet, over and over, until the file finish. No index or packet count dey inside.

The global header get the fields wey you suppose know:

• The magic number, four bytes. 0xa1b2c3d4 mean microsecond timestamps, and 0xa1b23c4d mean nanosecond timestamps. The byte order wey e take show for disk still tell reader weda the host wey write am na little endian.
• Version major and version minor, two bytes each. Dem don dey read 2 and 4 for decades.
• thiszone and sigfigs, four bytes each. Both na zero for practice.
• snaplen, four bytes. The maximum number of bytes wey capture host dey keep per packet.
• The link type, four bytes. 1 na Ethernet.

Every record header wey follow get four fields: ts_sec, whole seconds since Unix epoch; ts_usec, the fractional part for microseconds or nanoseconds depending on the magic number; incl_len, the capture length, wey mean how many bytes of this packet dey inside the file; and orig_len, the original length, wey mean how many bytes dey on the wire.

When orig_len pass incl_len, capture host don clip the packet at e snaplen and the tail no ever reach disk. Replay of that file go give feed handler message wey end mid-field. To compare the two lengths na the first check wey you suppose run on any capture wey person give you.

Why the capture timestamp na the one wey dey important

Normalized feed dey give you record with timestamp, and that stamp belong to the vendor: dem dey apply am once vendor parser don finish with the message. Capture timestamp, na capture card or kernel dey apply am upon arrival, usually within one or two microseconds of the packet landing, and dem dey write am before anything parse the payload. One dey measure pipeline. The other one dey measure wire.

The same idea dey visible inside normalized data. Every consolidated US equity quote get two clocks: participant timestamp, wey dem write for exchange, and SIP timestamp, wey dem write when consolidator process the message. The distance between dem na the step wey direct capture dey skip. SIP and direct exchange feeds cover wetin that consolidation add and wetin e drop.

SELECT
    formatDateTime(toStartOfInterval(toTimeZone(sip_timestamp, 'America/New_York'), INTERVAL 10 MINUTE), '%H:%i') AS et_time,
    count()                                            AS quote_count,
    round(quantileDeterministic(0.5)(gap_us, det), 0)  AS median_gap_us,
    round(quantileDeterministic(0.9)(gap_us, det), 0)  AS p90_gap_us
FROM
(
    SELECT
        sip_timestamp,
        toUnixTimestamp64Micro(sip_timestamp) - toUnixTimestamp64Micro(participant_timestamp) AS gap_us,
        toUInt64(toUnixTimestamp64Micro(sip_timestamp))                                       AS det
    FROM global_markets.cache_stocks_quotes
    WHERE ticker = 'AAPL'
      AND sip_timestamp >= '2026-06-10 13:30:00'
      AND sip_timestamp <  '2026-06-10 15:00:00'
      AND participant_timestamp > '2020-01-01 00:00:00'
)
WHERE gap_us BETWEEN 0 AND 1000000
GROUP BY et_time
ORDER BY et_time

Over the first ninety minutes of trading on June 10, 2026, wey dem split into ten minute buckets, the gap show clearly. For the 09:30 bucket, 93041 AAPL quotes carry both clocks with exchange stamp at or before consolidator stamp, and the middle message of that bucket sit 178 microseconds apart on the two. One message for every ten sit at least 342 microseconds apart. Those distances dem measure on top normalized record, after consolidation. Capture dey measure the same journey with that step removed from the measurement.

Build PCAP by hand, then read am back

The fastest way to stop treat the format like black box na to build one. The script below dey run on stock Python 3 without any package installed and no network access. Save am as pcap_demo.py and run am.

# pcap_demo.py (standard library only)
from binascii import unhexlify
import struct

GLOBAL_HEADER = unhexlify(
    'd4c3b2a1'   # magic 0xa1b2c3d4, written by a little endian host
    '0200'       # version_major = 2
    '0400'       # version_minor = 4
    '00000000'   # thiszone
    '00000000'   # sigfigs
    '08000000'   # snaplen = 8 bytes (tiny on purpose, see below)
    '01000000'   # link type 1 = Ethernet
)

RECORDS = [
    unhexlify(
        '80e14e68'          # ts_sec   = 1750000000
        'a0860100'          # ts_usec  = 100000
        '08000000'          # incl_len = 8 bytes captured
        '08000000'          # orig_len = 8 bytes on the wire
        '4d53472d30303031'  # payload MSG-0001
    ),
    unhexlify(
        '80e14e68'          # ts_sec   = 1750000000
        '90d00300'          # ts_usec  = 250000
        '08000000'
        '08000000'
        '4d53472d30303032'  # payload MSG-0002
    ),
    unhexlify(
        '80e14e68'          # ts_sec   = 1750000000
        '801a0600'          # ts_usec  = 400000
        '08000000'          # incl_len = 8 bytes captured
        '40000000'          # orig_len = 64 bytes on the wire, clipped at snaplen
        '4d53472d30303033'  # payload MSG-0003
    ),
]

blob = GLOBAL_HEADER + b''.join(RECORDS)

magic, vmaj, vmin, tz, sigfigs, snaplen, link = struct.unpack('<IHHiIII', blob[:24])
assert magic == 0xa1b2c3d4, 'not a little endian microsecond pcap'
assert (vmaj, vmin) == (2, 4)
print('snaplen', snaplen, 'link type', link)

off = 24
while off < len(blob):
    ts_sec, ts_usec, incl_len, orig_len = struct.unpack('<IIII', blob[off:off + 16])
    payload = blob[off + 16:off + 16 + incl_len]
    off += 16 + incl_len
    state = 'clipped' if orig_len > incl_len else 'whole'
    print(f'{ts_sec}.{ts_usec:06d}  incl={incl_len} orig={orig_len} {state}  {payload!r}')

E dey print four lines:

snaplen 8 link type 1
1750000000.100000  incl=8 orig=8 whole  b'MSG-0001'
1750000000.250000  incl=8 orig=8 whole  b'MSG-0002'
1750000000.400000  incl=8 orig=64 clipped  b'MSG-0003'

Three things to notice. The assert on the magic number na the full file type check: those four bytes wey you read back as little endian unsigned integer identify both the flavour and the byte order. The version pair don dey read 2.4 since before most trading systems dey exist, so to pin am dey safe across releases. And the third record declare 8 captured bytes against 64 on the wire, wey be how snaplen clip dey look from inside. The snaplen here na 8 bytes, wey no real capture host go do. E dey small like that just so one record fit show the clip.

Why bad merge dey silently reorder replay

Captures dey arrive in pieces. Capture host dey roll new file every few minutes or every gigabyte, so one session fit land as capture-1.pcap reach capture-40.pcap, and replay need dem as one single ordered stream. Two failure modes dey here, and none dey raise error.

The first na filename order. Shell glob dey sort lexically, wey dey put capture-10.pcap before capture-2.pcap. Vendor merge scripts dey rely on ls -1v for that reason: the -v flag dey sort digits inside name as numbers instead of text. Filename order still be just proxy for time order, and the proxy dey fail anytime capture host restart or two hosts contribute to the same session.

The second na concatenation. To run cat a.pcap b.pcap > merged.pcap fit look like e work but e go produce corrupt file. The second file 24 byte global header go land where record header suppose dey, and parser go read e magic number as packet wey dem stamp 2712847316 seconds after epoch, somewhere for 2055. Mergecap, the merge tool from Wireshark suite, dey strip extra headers and write packets out in chronological order.

Add one out-of-order record to the file wey you build above and the problem go show for two lines of output.

# add this to the bottom of pcap_demo.py
import tempfile

LATE = unhexlify(
    '80e14e68'          # ts_sec   = 1750000000
    'f0490200'          # ts_usec  = 150000, earlier than the record before it
    '08000000'
    '08000000'
    '4d53472d30303034'  # payload MSG-0004
)
blob += LATE

records, off = [], 24
while off < len(blob):
    ts_sec, ts_usec, incl_len, orig_len = struct.unpack('<IIII', blob[off:off + 16])
    payload = blob[off + 16:off + 16 + incl_len]
    records.append(((ts_sec, ts_usec), incl_len, orig_len, payload))
    off += 16 + incl_len

stamps = [r[0] for r in records]
print('file order:', [f'{s}.{u:06d}' for s, u in stamps])
print('monotonic:', all(a <= b for a, b in zip(stamps, stamps[1:])))

records.sort(key=lambda r: r[0])
with tempfile.NamedTemporaryFile(suffix='.pcap', delete=False) as fh:
    fh.write(GLOBAL_HEADER)
    for (ts_sec, ts_usec), incl_len, orig_len, payload in records:
        fh.write(struct.pack('<IIII', ts_sec, ts_usec, incl_len, orig_len))
        fh.write(payload)
    print('rewritten in timestamp order:', fh.name)

The parse loop dey walk records in file order and the monotonic check go return false. Nothing else go complain. Replay wey run straight from that file go feed the 150000 microsecond message after the 400000 microsecond one, and any consumer wey hold state go see quote update arrive after the update wey don already replace am. On real session wey span forty files, that defect go show as small misordered packets at every file boundary, wey be exactly where book state dey stitch together. To sort on record timestamp before writing, as the second half of the script do, na the fix, and na why timestamp order instead of filename order na the invariant wey you suppose test.

Replay wey dey reorder messages na look-ahead problem wey wear different cloth: consumer dey see state wey no exist yet at the timestamp wey e believe e dey. Look ahead bias in backtesting cover the same failure at bar resolution.

How much data dey inside

Sixty seconds of one symbol, second by second, go give you feel of the volume wey capture dey absorb.

SELECT
    formatDateTime(toStartOfSecond(toTimeZone(sip_timestamp, 'America/New_York')), '%H:%i:%S') AS et_time,
    count()                                            AS quote_count,
    round(quantileDeterministic(0.5)(gap_us, det), 0)  AS median_gap_us
FROM
(
    SELECT
        sip_timestamp,
        toUnixTimestamp64Micro(sip_timestamp) - toUnixTimestamp64Micro(participant_timestamp) AS gap_us,
        toUInt64(toUnixTimestamp64Micro(sip_timestamp))                                       AS det
    FROM global_markets.cache_stocks_quotes
    WHERE ticker = 'AAPL'
      AND sip_timestamp >= '2026-06-10 13:30:00'
      AND sip_timestamp <  '2026-06-10 13:31:00'
      AND participant_timestamp > '2020-01-01 00:00:00'
)
WHERE gap_us BETWEEN 0 AND 1000000
GROUP BY et_time
ORDER BY et_time

The first second for the panel, 09:30:00 ET, carry 528 AAPL quote updates on consolidated tape alone, with median clock gap of 48 microseconds. The minute break into 60 seconds with quote activity. Direct feed capture of the same minute hold pass that one: every packet wey exchange send, including updates wey consolidator collapse, each one wrap in Ethernet, IP and UDP headers.

Quote traffic na wetin dey fill capture disk, and the ratio no be the same across names.

SELECT
    q.symbol                                   AS symbol,
    q.quote_count                              AS quote_count,
    round(q.quote_count / t.trade_count, 1)    AS quotes_per_trade
FROM
(
    SELECT ticker AS symbol, count() AS quote_count
    FROM global_markets.cache_stocks_quotes
    WHERE ticker IN ('AAPL', 'MSFT', 'NVDA', 'SPY', 'KO')
      AND sip_timestamp >= '2026-06-10 13:30:00'
      AND sip_timestamp <  '2026-06-10 14:00:00'
    GROUP BY ticker
) AS q
INNER JOIN
(
    SELECT ticker AS symbol, count() AS trade_count
    FROM global_markets.stocks_trades
    WHERE ticker IN ('AAPL', 'MSFT', 'NVDA', 'SPY', 'KO')
      AND sip_timestamp >= '2026-06-10 13:30:00'
      AND sip_timestamp <  '2026-06-10 14:00:00'
    GROUP BY ticker
) AS t ON q.symbol = t.symbol
WHERE t.trade_count > 0
ORDER BY quotes_per_trade DESC

Across the 5 household names for the panel, over the same half hour, SPY print 11.3 quote messages for every trade, the highest ratio of the group, on 1035147 quote updates. MSFT sit for the other end of the panel with 0.8. Quotes na the bulk of bytes for any equity capture, wey also be the reason depth of book dey cost more to store than top of book. Level 1 and level 2 market data explain wetin those extra messages carry.

Wetin replay suppose reproduce

Capture dey replay at any speed wey you drive am, and the per-packet timestamps na the only thing wey dey keep replay honest. Message rate across session no dey constant at all.

SELECT
    formatDateTime(toStartOfInterval(toTimeZone(window_start, 'America/New_York'), INTERVAL 15 MINUTE), '%H:%i') AS et_time,
    round(sumIf(transactions, ticker = 'AAPL') / 900, 2) AS aapl_prints_per_sec,
    round(sumIf(transactions, ticker = 'SPY') / 900, 2)  AS spy_prints_per_sec
FROM global_markets.delayed_stocks_minute_aggs
WHERE ticker IN ('AAPL', 'SPY')
  AND window_start >= '2026-06-10 08:00:00'
  AND window_start <  '2026-06-11 00:00:00'
GROUP BY et_time
ORDER BY et_time

Print na completed trade report. The panel bucket every AAPL and SPY print into fifteen minute buckets on New York clock, across the whole electronic day instead of regular session alone. The earliest bucket with any activity, 04:00 ET, run 10.45 AAPL prints per second against 3.12 for SPY. Read across the chart into the regular session and the shape go teach you. Replay engines dey schedule off per-packet timestamp and sleep between packets instead of counting messages, and replay wey dem clock at fixed rate no go reproduce any part of that curve.

When capture dey worth am

Capture dey earn e storage when the question concern time or order:

• Latency measurement, where the microseconds dey go between wire, feed handler and order gateway. Only capture hold timestamp wey dem take before your own software touch the packet.
• Gap and arbitration research, wey cover how often feed dey drop packets and how quick the B side of A and B feed pair dey fill the hole.
• Feed handler testing, wey dey replay real session against new parser to see weda e go rebuild the same book.
• Reconstruction disputes, wey dey prove how the book look at given microsecond in form wey both sides fit parse independently.

Normalized feed na better tool for every other place. E dey carry one schema across venues, corporate actions applied, symbology cleaned up, and e fit enter memory. Research at bar resolution no ever need packets at all: how OHLCV bars are built walk through the aggregation step and where e dey hide detail. Captures also carry real bill for tap ports and storage wey dey grow every session, on top of data licences themselves, wey what real time market data costs break down.

FAQ

Wetin be PCAP file for market data?

PCAP file na packet capture: the raw multicast packets of exchange feed wey dem save in arrival order, each one get 16 byte header wey hold capture timestamp, number of bytes saved and number of bytes wey dey on the wire. E contain the feed own binary protocol, no be normalized quote or trade record.

Why market data PCAP files dey too large?

Every packet dey kept whole, including Ethernet, IP and UDP headers, and quote updates dey pass trades by wide margin on liquid names. The panel above measure that ratio for handful of household symbols over single half hour.

Wetin be the difference between capture length and original length?

Capture length (incl_len) na how many bytes of the packet dey inside the file. Original length (orig_len) na how many bytes dey on the wire. When original length pass capture length, capture host don clip the packet at e snaplen setting and the remainder no ever write go disk.

Why PCAP captures must merge in timestamp order?

Session usually arrive as many rolling files, and both filename sorting and plain concatenation fit place packets for wrong sequence. Consumer wey dey replay out-of-order stream go see updates arrive after the updates wey replace dem, without any error for the chain.

I need PCAP data to backtest trading strategy?

No. Bar data and normalized tick data dey answer most research questions and dey cost far less to store and query. Captures dey matter when the question specifically concern latency, packet loss, or the exact order wey system see e messages.

Every panel on this page ship with the SQL wey produce am, wey you fit expand under the chart. To count messages for session by yourself, or compare the two clocks on one quote, ask the question in plain English on Strasmore terminal.