Skip to main content
Stories Worth Reading///Perspectives Worth Sharing
The Story Desk - Stories Worth Reading, Perspectives Worth Sharing.
2026-11-01•Editorial Week 44 • November 01, 2026•8 min read

The Telemetry Tax: Why High-Cardinality Logs Are Breaking Cloud Budgets

A systems teardown of modern APM billing models, the computational trade-offs of columnar stores like ClickHouse, and why OpenTelemetry is becoming the enterprise hedge against vendor lock-in.

By The Story Desk Editorial Desk
Verification Standards
Inside SigNoz: How an Indian Open-Source Engine Challenges Datadog

High-density telemetry dashboards and open-source query execution. Photo: Markus Spiske via Unsplash (Free License).

Executive Memo

In enterprise engineering teams, telemetry data volume has grown at nearly 3x the rate of revenue over the past four years. Microservices architecture, distributed tracing, and ephemeral Kubernetes pods generate gigabytes of diagnostic context every hour.

The economic crisis of modern observability is not collection; it is indexing. Traditional search engines and proprietary APM vendors (such as Datadog or legacy Elasticsearch clusters) rely on inverted indices. In an inverted index, every unique attribute—such as container ID, trace ID, user ID, or HTTP status code—must be mapped to an internal document list in memory.

When microservices emit high-cardinality tags, index sizes explode, driving RAM and SSD storage costs into an exponential curve. To manage bills, engineering teams are forced into aggressive sampling—discarding up to 90% of their traces and creating catastrophic blind spots during live production outages.

This week, we examine the structural shift toward vectorized columnar DBMS engines (specifically ClickHouse) and the vendor-neutral OpenTelemetry standard, which dismantle the legacy telemetry tax by separating compute from storage.

01.The Inverted Index Penalty vs. Vectorized Columnar Compression

Inverted indices excel at unstructured full-text search across arbitrary text corpora. However, 95% of machine-generated telemetry consists of structured attributes (timestamps, latency floats, IP addresses, service names). Storing structured telemetry inside an inverted index creates massive index fragmentation.

Columnar databases store data column-by-column rather than row-by-row. Because all values within a column share identical data types, compression algorithms (LZ4, ZSTD, Delta, DoubleDelta) achieve compression ratios between 5:1 and 8:1. More importantly, queries scanning 100 million logs for an error rate calculate aggregations using vectorized CPU SIMD instructions without touching unused string attributes.

02.The FinOps Reality: SaaS Margin Arbitrage vs. SRE Labor

Our empirical modeling indicates that for an enterprise ingesting 100 GB of telemetry per day, a tuned ClickHouse-backed platform (such as SigNoz Cloud) reduces monthly software fees by ~72% compared to Datadog. At 1 TB per day, the annual delta exceeds $120,000.

However, self-hosting introduces non-trivial operational costs. Operating distributed ClickHouse clusters with Zookeeper/Keeper replication, multi-disk volume tiering, and backup automation requires senior Site Reliability Engineering (SRE) attention. Unless an organization ingests more than 500 GB/day, managed open-source cloud offerings typically yield higher net ROI than internal infrastructure self-hosting.

Observability Ingestion & Architecture Model (100 GB/Day Telemetry)

Architecture ModelStorage & Index MechanismEstimated Monthly CostCompression RatioOperational Complexity
Proprietary SaaS (Datadog)Proprietary Inverted Index + Dynamic Retention~$4,200 / month~2.5:1 (Proprietary)Zero operational overhead; high pricing volatility
Managed Columnar (SigNoz Cloud)ClickHouse Columnar + OpenTelemetry Protocol~$1,150 / month6.5:1 (ZSTD / LZ4)Low; vendor handles scaling and backup automation
Self-Hosted ClickHouse ClusterBare-metal / AWS EBS Columnar Cluster~$580 infra + SRE labor7.2:1 (Tuned Codecs)High; requires cluster rebalancing, Keeper quorum maintenance
Self-Hosted ELK (Elasticsearch)Lucene Inverted Index on SSD NVMe~$1,950 infra + SRE labor2.0:1 – 2.8:1High; JVM heap tuning, garbage collection pauses

* Model assumes 100 GB/day ingestion (3 TB/month raw) with 30-day retention and 100 hosts, based on documented pricing calculators and infrastructure specs.

Executive Decision Framework

Practitioner playbooks and strategic risk evaluation for decision-makers.

Scenario

Ingesting <50 GB/day with small engineering team (<15 engineers)

Action: Stick with managed SaaS or lightweight OpenTelemetry collectors; internal operational overhead of cluster maintenance exceeds software savings.

Risk Vector: Unchecked high-cardinality tag additions causing end-of-month invoice shocks.

Scenario

Ingesting 100 GB to 1 TB/day with microservices generating high cardinality

Action: Standardize application instrumentation on OpenTelemetry (OTel); adopt managed ClickHouse-backed storage (SigNoz Cloud) to capture 70% savings.

Risk Vector: Migration friction when converting legacy proprietary alerting logic and dashboard templates.

Scenario

Ingesting >2 TB/day with strict data-custody and regulatory retention mandates

Action: Deploy self-hosted ClickHouse on dedicated NVMe instances with cold-tier S3 object storage archiving.

Risk Vector: Underestimating SRE hiring requirements for distributed database maintenance.

Supporting Field Investigation12 min read

Inside SigNoz: How an Indian Open-Source Engine Challenges Datadog

An architectural, commercial and operational teardown of an OpenTelemetry-native observability platform—with current published pricing, corrected product claims, a reproducible 100 GB/day model and explicit source notes.

Companion Coverage from the Desk

Weekly Digest

Primary Sourcing & Verification References

Official architectural specification for OTLP gRPC/HTTP protocols and semantic conventions.

Technical specifications on vectorized SIMD execution and columnar data compression.

Published list pricing for log ingestion ($0.10/GB) and 30-day index retention.