BinaryLogReader

BinaryLogReader provides sequential access to market data stored in the binary log format. It handles segment discovery, time filtering, symbol filtering, and CRC verification.

struct ReaderConfig
{
  std::filesystem::path data_dir;
  std::optional<int64_t> from_ns;
  std::optional<int64_t> to_ns;
  std::set<uint32_t> symbols;
  bool verify_crc{true};
  int64_t reorder_window_ns{10'000'000'000};  // 10s
};

class BinaryLogReader
{
public:
  explicit BinaryLogReader(ReaderConfig config);

  // Static inspection (no event reading)
  static DatasetSummary inspect(const std::filesystem::path& data_dir);
  static DatasetSummary inspectWithSymbols(const std::filesystem::path& data_dir);

  // Instance methods
  DatasetSummary summary();
  uint64_t count();
  std::set<uint32_t> availableSymbols();

  // Iteration
  using EventCallback = std::function<bool(const ReplayEvent&)>;
  bool forEach(EventCallback callback);
  bool forEachFrom(int64_t start_ts_ns, EventCallback callback);

  // Progress reporting. The callback returns false to cancel the run; forEach
  // then returns false so the caller can tell a cancelled run from a completed
  // one. Returning true continues.
  using ProgressCallback = std::function<bool(double pct, int64_t cursor_ts_ns)>;
  void setProgressCallback(
      ProgressCallback cb,
      std::chrono::milliseconds interval = std::chrono::milliseconds(1000));
  void clearProgressCallback();

  // Metadata
  std::optional<std::pair<int64_t, int64_t>> timeRange() const;
  ReaderStats stats() const;
  std::vector<std::filesystem::path> segmentFiles() const;
  const std::vector<SegmentInfo>& segments() const;
};

Purpose

• Read market data from binary log segments in chronological order.
• Filter events by time range and symbol set.
• Support both full scans and timestamp-based seeks.

Configuration

Field	Type	Description
data_dir	filesystem::path	Directory containing .floxlog files
from_ns	optional<int64_t>	Start timestamp filter (inclusive)
to_ns	optional<int64_t>	End timestamp filter (inclusive)
symbols	set<uint32_t>	Symbol IDs to include (empty = all)
verify_crc	bool	Verify CRC32 checksums (default: true)
reorder_window_ns	int64_t	Bounded reorder buffer, default 10 s. Events arriving more than this far behind the emit cursor cannot be placed in order any more, so the reader throws FloxError with the observed delta. The default covers exchange-WS jitter and the 99th percentile of reconnect-induced cross-block inversions on real tapes. Memory bound is roughly reorder_window_ns x peak_event_rate x sizeof(ReplayEvent) — about 36 MB at 10 s and a 10k ev/s burst

Core Methods

Method	Description
inspect()	Static scan of directory, returns metadata without reading events
inspectWithSymbols()	Like inspect() but also collects symbol IDs
summary()	Returns dataset metadata after scanning
count()	Returns total event count across all segments
forEach()	Iterate all events matching filters
forEachFrom()	Iterate events starting from a timestamp
setProgressCallback()	Install a progress callback, invoked at most once per interval. Return false from it to cancel; forEach then returns false
clearProgressCallback()	Remove the progress callback
timeRange()	Returns (first_event_ns, last_event_ns) pair
stats()	Returns read statistics (events, bytes, errors)
segmentFiles()	Returns list of segment file paths
segments()	Returns detailed segment information

Data Structures

DatasetSummary

struct DatasetSummary
{
  std::filesystem::path data_dir;

  int64_t first_event_ns{0};
  int64_t last_event_ns{0};

  uint64_t total_events{0};
  uint32_t segment_count{0};
  uint64_t total_bytes{0};

  std::set<uint32_t> symbols;

  uint32_t segments_with_index{0};
  uint32_t segments_without_index{0};

  // Helper methods
  bool empty() const;
  std::chrono::nanoseconds duration() const;
  double durationSeconds() const;
  double durationMinutes() const;
  double durationHours() const;
  bool fullyIndexed() const;
};

ReplayEvent

struct ReplayEvent
{
  EventType type;           // Trade, BookSnapshot, BookDelta, OptionQuote, PoolState
  int64_t timestamp_ns;     // Event timestamp

  TradeRecord trade;        // Populated for Trade events

  BookRecordHeader book_header;  // Populated for Book events
  std::vector<BookLevel> bids;
  std::vector<BookLevel> asks;

  OptionQuoteRecord option_quote;  // Populated for OptionQuote events

  // Pool-state record: the fixed header plus the raw u256 payload bytes,
  // carried opaquely for the pool-state-tape layer to interpret.
  PoolStateRecordHeader pool_state_header;
  std::vector<std::byte> pool_state_payload;

  // Symbol id for the active record, whichever event type this is. Keeps symbol
  // filtering correct as new record types are added.
  uint32_t symbolId() const;
};

Read the symbol through symbolId() rather than reaching into a per-type record; it switches on type and stays correct as record types are added.

ReaderStats

struct ReaderStats
{
  uint64_t files_read{0};
  uint64_t events_read{0};
  uint64_t trades_read{0};
  uint64_t book_updates_read{0};
  uint64_t bytes_read{0};
  uint64_t crc_errors{0};
};

SegmentInfo

struct SegmentInfo
{
  std::filesystem::path path;
  int64_t first_event_ns{0};
  int64_t last_event_ns{0};
  uint32_t event_count{0};
  bool has_index{false};
  uint64_t index_offset{0};
};

Usage

replay::ReaderConfig config{
    .data_dir = "/data/market",
    .from_ns = start_timestamp,
    .to_ns = end_timestamp,
    .symbols = {1, 2, 3}
};

replay::BinaryLogReader reader(config);

reader.forEach([](const replay::ReplayEvent& event) {
    if (event.type == replay::EventType::Trade) {
        // Process trade
    } else {
        // Process book update
    }
    return true;  // Continue iteration
});

Time Utilities

The time_utils namespace provides helper functions:

namespace replay::time_utils
{
  int64_t toNanos(std::chrono::system_clock::time_point tp);
  std::chrono::system_clock::time_point fromNanos(int64_t ns);
  int64_t nowNanos();
  int64_t secondsToNanos(int64_t seconds);
  int64_t millisToNanos(int64_t millis);
  int64_t microsToNanos(int64_t micros);
  double nanosToSeconds(int64_t ns);
}

BinaryLogIterator

Low-level iterator for reading a single segment file:

class BinaryLogIterator
{
public:
  explicit BinaryLogIterator(const std::filesystem::path& segment_path);

  bool next(ReplayEvent& out);
  bool seekToTimestamp(int64_t target_ts_ns);
  bool loadIndex();

  const SegmentHeader& header() const;
  bool isValid() const;
  bool isCompressed() const;
  bool hasIndex() const;
};

Ordering guarantee

forEach and forEachFrom always deliver events in monotonically non-decreasing timestamp_ns order, regardless of how the underlying segment was recorded.

For segments written with the current writer (which sets SegmentFlags::Sorted), events are streamed directly — O(1) memory, early exit on false return is immediate.

For legacy segments without the flag, all events in the segment are buffered and sorted before delivery — O(segment events) memory, early exit stops delivery but the segment has already been read in full.

BinaryLogIterator is a low-level block-by-block streaming API with no ordering guarantee. Use BinaryLogReader when ordered output is required.

Notes

• Segments are automatically discovered and sorted by filename.
• Compressed segments (LZ4) are transparently decompressed.
• Seeking uses segment indexes when available for O(log n) lookup.
• The callback returning false stops iteration early.
• File extension is .floxlog.

Live-tail safety

Active segments — those whose writer is still appending — are safe to read. The writer fflushes after each compressed block and refreshes the segment header, so a snapshot of the file taken mid-run (e.g. via rsync) contains complete blocks and a non-zero SegmentHeader.

If the header is still zero-initialized at the time of read (event_count == 0 on a compressed segment), scanSegments() and inspect() recover the metadata: every CompressedBlockHeader is walked to sum event_count, and the first / last viable blocks are decompressed for first_event_ns / last_event_ns. The very last block is often truncated because the writer flushes its block header before the compressed payload is fully written, so the scan iterates backwards until one block decompresses successfully.