Ingesting Financial Tick Data Using a Time-Series Database

Financial tick data—granular, high-frequency records of every trade and quote in the market—is essential for algorithmic trading, risk modeling, and real-time analytics. Managing this data efficiently requires a database built for speed, scalability, and time-series optimization. QuestDB stands out as a next-generation time-series database designed specifically for market data workloads. With high-throughput ingestion, powerful SQL analytics, and hardware-efficient performance, QuestDB is ideal for handling crypto and traditional financial tick data.

This guide explores three robust methods to ingest financial tick data into QuestDB for real-time analysis and downstream applications.

Prerequisites

Before diving into data ingestion, ensure you have a local QuestDB instance running. Begin by creating a project directory and launching QuestDB via Docker:

mkdir cryptofeed-questdb
cd cryptofeed-questdb
docker run \
  -p 9000:9000 -p 9009:9009 -p 8812:8812 -p 9003:9003 \
  -v "$(pwd):/var/lib/questdb" \
  questdb/questdb:8.1.0

This command maps QuestDB’s web console (port 9000), InfluxDB Line Protocol port (9009), and other services to your local machine. Once running, access the web interface at http://localhost:9000 to monitor and query your data.

Method 1: Ingesting Data Using the Cryptofeed Library

One of the fastest ways to stream crypto market data into QuestDB is using Cryptofeed, a Python library that connects to multiple exchanges—including Binance, OKX, Gemini, and Kraken—via WebSocket APIs. It normalizes trade, book, and ticker data into a consistent format and supports direct integration with QuestDB through InfluxDB Line Protocol (ILP).

👉 Discover how real-time data pipelines can accelerate your financial analytics

Setting Up Cryptofeed

Start by creating a virtual environment and installing the library:

python3 -m venv cryptofeed
source cryptofeed/bin/activate
pip install cryptofeed

Create a file named questdb.py and add the following code to ingest BTC-USDT trade data from OKX and Gemini:

from cryptofeed import FeedHandler
from cryptofeed.backends.quest import TradeQuest
from cryptofeed.defines import TRADES
from cryptofeed.exchanges import OKX, Gemini

QUEST_HOST = '127.0.0.1'
QUEST_PORT = 9009

def main():
    f = FeedHandler()
    f.add_feed(OKX(channels=[TRADES], symbols=['BTC-USDT'], callbacks={TRADES: TradeQuest(host=QUEST_HOST, port=QUEST_PORT)}))
    f.add_feed(Gemini(channels=[TRADES], symbols=['BTC-USDT'], callbacks={TRADES: TradeQuest(host=QUEST_HOST, port=QUEST_PORT)}))
    f.run()

if __name__ == '__main__':
    main()

Run the script to establish WebSocket connections and stream data directly into QuestDB. You may notice a delay with Gemini due to lower trade frequency.

Customizing Data Ingestion

For greater control over schema and table structure, override the default callback. The example below customizes the table name and column formatting:

from cryptofeed import FeedHandler
from cryptofeed.backends.backend import BackendCallback
from cryptofeed.backends.socket import SocketCallback
from cryptofeed.defines import TRADES
from cryptofeed.exchanges import OKX

QUEST_HOST = '127.0.0.1'
QUEST_PORT = 9009

class QuestCallback(SocketCallback):
    def __init__(self, host='127.0.0.1', port=9009, **kwargs):
        super().__init__(f"tcp://{host}", port=port, **kwargs)
        self.numeric_type = float
        self.none_to = None

    async def writer(self):
        while True:
            try:
                await self.connect()
            except:
                exit(-1)
            async with self.read_queue() as update:
                update = "\n".join(update) + "\n"
                try:
                    self.conn.write(update.encode())
                except:
                    exit(-2)

class TradeQuest(QuestCallback, BackendCallback):
    default_key = 'trades'

    async def write(self, data):
        update = f'{self.key},symbol={data["symbol"]},side={data["side"]} price={data["price"]},amount={data["amount"]} {int(data["timestamp"]*1_000_000_000)}'
        await self.queue.put(update)

def main():
    handler = FeedHandler()
    handler.add_feed(OKX(channels=[TRADES], symbols=['BTC-USDT', 'ETH-USDT'], callbacks={TRADES: TradeQuest(host=QUEST_HOST, port=QUEST_PORT)}))
    handler.run()

if __name__ == '__main__':
    main()

This approach uses raw ILP strings for maximum flexibility. Cryptofeed’s built-in normalization reduces development time, making it ideal for rapid prototyping.

Method 2: Building a Custom Market Data Pipeline

When Cryptofeed doesn’t support your target exchange or you need preprocessing logic, building a custom ingestion pipeline gives full control. QuestDB supports ingestion via PostgreSQL wire protocol and ILP—however, ILP offers higher performance and schema flexibility.

Polling REST APIs with Python SDK

The following script fetches BTC price data from Binance and Gemini using REST APIs and writes it to QuestDB via the official Python SDK:

import requests
import time
from questdb.ingress import Sender, TimestampNanos

conf = 'http::addr=localhost:9000;'

def get_binance_data():
    url = 'https://api.binance.us/api/v3/ticker/price'
    params = {'symbol': 'BTCUSDT'}
    response = requests.get(url, params=params)
    data = response.json()
    btc_price = data['price']
    print(f"BTC Price on Binance: {btc_price}")
    publish_to_questdb('Binance', btc_price)

def get_gemini_data():
    url = 'https://api.gemini.com/v1/pubticker/btcusdt'
    response = requests.get(url)
    data = response.json()
    btc_price = data['last']
    print(f"BTC Price on Gemini: {btc_price}")
    publish_to_questdb('Gemini', btc_price)

def publish_to_questdb(exchange, price):
    print("Publishing BTC price to QuestDB...")
    with Sender.from_conf(conf) as sender:
        sender.row('prices', symbols={'pair': 'BTCUSDT'}, columns={'exchange': exchange, 'bid': price}, at=TimestampNanos.now())
        sender.flush()

def job():
    print("Fetching BTC price...")
    get_binance_data()
    get_gemini_data()

while True:
    job()
    time.sleep(5)

This script polls every 5 seconds and inserts data into a prices table. Use this pattern when you need to filter, enrich, or aggregate data before storage.

👉 Explore how high-performance databases transform financial data workflows

Method 3: Ingesting Market Data Using Change Data Capture (CDC)

For enterprise environments with existing data streams, Change Data Capture (CDC) enables real-time replication from source systems like Kafka or PostgreSQL into QuestDB. Instead of polling APIs, CDC listens for data changes and forwards them instantly.

A common architecture involves:

A service publishing price updates to Kafka (e.g., from Coinbase API).
QuestDB’s Kafka Connector consuming messages and writing them via ILP.

This decoupled design reduces latency and infrastructure overhead. For implementation details, refer to QuestDB’s guide on building a real-time crypto tracker with Kafka.

Core Keywords

tick data
time-series database
market data ingestion
cryptofeed
InfluxDB Line Protocol
Change Data Capture
financial data analytics
real-time data pipeline

Frequently Asked Questions

What is Level 1, Level 2, and Level 3 market data?
Level 1 provides basic pricing (last trade, best bid/ask). Level 2 includes market depth with multiple price levels. Level 3 offers full order book visibility with individual order details.

What is an order book?
An order book is a real-time list of buy and sell orders for an asset, organized by price level. It reflects supply and demand dynamics in the market.

What are candlestick charts used for?
Candlestick charts display price movements over time using open, high, low, and close values. They are widely used in technical analysis to identify trends and patterns.

How does InfluxDB Line Protocol work with QuestDB?
ILP is a text-based format that sends time-series data over TCP or HTTP. QuestDB uses it for high-speed ingestion without requiring predefined schemas.

Can QuestDB handle high-frequency trading data?
Yes. QuestDB is optimized for high-throughput ingestion and low-latency queries, making it suitable for tick-level trading data analysis.

Is CDC suitable for real-time analytics?
Absolutely. CDC enables near-instant data synchronization from source systems to analytical databases like QuestDB, supporting real-time dashboards and alerts.

QuestDB empowers developers and analysts to build scalable, low-latency financial data systems. Whether using prebuilt tools like Cryptofeed or designing custom pipelines with CDC, integrating tick data has never been more efficient.