QuestDB InfluxDB Line Protocol (ILP) High-Rate Ingestion Sizing Guide
TimescaleOps sizing guide for QuestDB SIMD columnar partitions, designated timestamps, ZFS zstd-3 storage, and ASOF JOIN queries alongside TimescaleDB.
When append-only sensor or market streams exceed 150,000 events per second on a single server and require time-aligned ASOF JOIN lookups without relational foreign keys, QuestDB memory-mapped columnar partitions paired with ZFS transparent compression offer a specialized alternative to relational hypertables.
| Telemetry Scale | Chunk / Partition Interval | Server DDR5 RAM | Retained Disk Footprint | Matched Infrastructure Tier |
|---|---|---|---|---|
| 25K events/s ILP | PARTITION BY DAY |
16 GB DDR5 | 384 GB Local NVMe | Vultr HF 16GB |
| 150K events/s ILP | PARTITION BY DAY / HOUR |
32–64 GB DDR5 | 768 GB Local NVMe | Vultr HF 32GB / UltaHost VDS 64GB |
| 500K events/s Firehose | PARTITION BY HOUR |
128 GB DDR5 | 4–10 TB ZFS LZ4/ZSTD | InterServer Bare-Metal Ryzen 128GB |
| Multi-Year Cold Replay | PARTITION BY DAY + ZFS |
64 GB DDR5 | 12 TB ZFS Pool | InterServer Price-Lock Storage |
-- Create partitioned columnar table with explicit SYMBOL dictionary capacity
CREATE TABLE l1_quotes (
ts TIMESTAMP,
venue SYMBOL CAPACITY 32 CACHE,
pair SYMBOL CAPACITY 2048 CACHE INDEX,
bid DOUBLE,
ask DOUBLE,
bid_size DOUBLE,
ask_size DOUBLE
) TIMESTAMP(ts) PARTITION BY DAY WAL DEDUP UPSERT KEYS(ts, venue, pair);
-- Align each event against the most recent prevailing reference quote via ASOF JOIN
SELECT t.ts, t.pair, t.price, q.bid, q.ask, (q.ask - q.bid) AS spread
FROM trades t ASOF JOIN l1_quotes q ON (venue, pair);1. Sizing OS Page Cache RAM vs JVM Heap in QuestDB Deployments
Per QuestDB memory architecture documentation, columnar vectors reside off-heap through OS memory-mapped files (mmap). Allocating 80% of system RAM to the Java heap starves mmap buffers; keep JVM -Xmx between 4 GB and 8 GB so 75%+ of DDR5 RAM backs Linux page cache.
2. Pairing Open-Source QuestDB Partitions with ZFS zstd-3 Filesystem Compression
Open-source QuestDB writes fixed-width binary column files per partition directory. Mounting its data root on an OpenZFS dataset configured with compression=zstd-3 and recordsize=128K reduces on-disk footprint substantially on dedicated Ryzen bare-metal storage.
3. QuestDB SYMBOL Capacity Planning Prevents Dictionary Re-Hash Pauses
Reserve QuestDB SYMBOL columns strictly for bounded-cardinality identifiers (exchange venues, sensor models, substation codes) and declare CAPACITY above your expected distinct key count.
TimescaleOps Engineering Q&A: TS-03 // QUESTDB-ILP Hypertable Operations
When should we pick QuestDB over a PostgreSQL TimescaleDB hypertable?
Choose QuestDB for pure high-rate append-only streams requiring vectorized scans and ASOF JOINs without relational foreign-key constraints, PostGIS spatial joins, or row-level UPDATE/DELETE transactions.
Does QuestDB support deduplication during collector reconnect replays?
Yes. QuestDB WAL tables support DEDUP UPSERT KEYS(ts, symbol) directly inside the commit pipeline so duplicate reconnect batches are filtered deterministically.
Which NVMe and ZFS filesystem block settings work best for QuestDB and TimescaleDB on TimescaleOps?
Mount ext4 or XFS with noatime for single-disk local NVMe setups, or configure OpenZFS with ashift=12, recordsize=128K, and compression=zstd-3 for multi-terabyte archival pools.