Real-Time Product Analytics at Scale: How We Deliver Sub-Second Insights on Billions of Events

Dashboards are slow. You know it, your PM knows it, and most importantly your users feel it. In today’s data-driven world, if your product analytics take more than a couple of seconds to load, you’ve already lost context, patience and probably the conversion. At Deuex we faced this head on. We needed to go from event firehoses to lightning-fast dashboards. So we rearchitected our analytics stack for sub-second latency at scale, and we didn’t just settle for fast. We engineered for real-time insights, high concurrency and cost efficiency.

This is the story of how we made it happen.

The Latency Problem: Why Traditional Warehouses Just Don’t Cut It

Let’s set the scene.

You have billions of events coming in like; user clicks, page views, button hovers, mobile app gestures, fraud alerts, operational logs, and you need insights now. So you load that into your favorite cloud data warehouse and start slicing.

Then you wait.
 Five seconds. Fifteen seconds. Sometimes minutes.

Why?
Because traditional warehouses were designed for deep, complex batch queries not for real-time, high-QPS (queries per second) product analytics. They shine for ETL pipelines, historical trends and BI reporting, but they’re not built to:

• Serve thousands of concurrent users querying simultaneously

• Handle streaming data ingestion with low freshness lag

• Provide sub-second response times for UI dashboards

• Retain granular event level data cost effectively over months

This gap, the “last-mile latency” is where most stacks crumble. Dashboards freeze. Users lose trust. Growth teams start flying blind.

At Deuex we needed something radically different.

Reimagining Analytics: What Powers Our Real-Time Stack

We rebuilt our analytics stack around one core idea:

“Speed isn't a luxury. It's the product.”

At the heart of our real-time engine is a columnar OLAP datastore optimized for ultra fast analytics. Without naming names, its purpose is built to serve analytics on fresh event streams with blazingly fast response times. Think:

• Millisecond latency for simple aggregates

• High throughput on massive datasets

• Real-time + batch ingestion from Kafka and beyond

• Smart indexing to prune billions of records instantly

• Concurrency at scale for user facing dashboards

It fits where traditional databases break. We call it our speed layer and it lives between raw event ingestion and the product analytics UI.

The Blueprint: Kafka → Fast OLAP Store → API Layer → Dashboard

Let’s break down our architecture.

Event Ingestion via Kafka

Everything starts with events. Our platform emits tens of millions of events daily clicks, fraud signals, backend API traces, even sensor data. All of this is streamed through Kafka topics organized by domain (e.g., user actions, telemetry, transactions).

We use compact, consistent Avro schemas for each event type, ensuring reliable ingestion.

Real-Time Store with Smart Indexes

Our OLAP store ingests directly from Kafka via stream ingestion jobs. But we don’t just dump the data, we optimize it.

• Star-tree indexes prune 95%+ of irrelevant data during queries.

• Rollups reduce cardinality without losing insight granularity.

• Time-based partitions help us query only the freshest data.

• Tiered storage lets us keep hot vs. cold data separately.

Each table is modeled carefully for its use case:

• User behavior analytics → row-level detail for funnels

• Ops telemetry → time-binned metrics for trends

• Growth dashboards → pre-aggregated rollups for rapid charts

API Layer for Query Federation

We expose a thin query layer via REST APIs that lets frontend teams fetch insights with ease.

Key features include:

• Query templates for funnel breakdowns, retention cohorts, conversion paths

• Built-in guards for timeouts, cardinality, and cost ceilings

• Support for upserts (correction events or late arriving data)

• Caching for repeated queries across sessions or views

UX Dashboards Built for Interactivity

On the frontend, dashboards use this API layer to serve dynamic views. Whether it’s a PM analyzing new feature adoption or a fraud analyst spotting spikes in suspicious activity, the experience is snappy and interactive.

Every click counts. Every query lands in under a second.

Benchmarks, Optimizations & Guardrails

Speed is table stakes. Sustainability is the real challenge. Here’s how we keep both.

Schema Design: Built for Queries, Not Storage

We treat schema design like API design. Each table schema is optimized for query access patterns:

• Flat schemas over deeply nested ones for scan efficiency

• Dictionary encoded dimensions for fast filtering

• Min/max and bloom filters on common fields

• Metric rollups to cut down cardinality

We also version schemas to support schema evolution without downtime.

Upserts & Change Data Capture (CDC)

Product data changes, users delete accounts, transactions get flagged. So our ingestion pipelines support:

• Upserts via primary keys (e.g., user_id + timestamp)

• Soft deletes using status flags

• Integration with CDC streams for fraud corrections and updates

This makes our analytics not just real-time, but accurate in real time.

Retention Tiers: Hot, Warm, Cold

Keeping months of data online is expensive. Not all data needs instant access, so we built retention tiers:

• Hot Tier (7 days): SSD-backed, instant querying

• Warm Tier (30–60 days): compressed store, slightly higher latency

• Cold Tier (older): offloaded to object storage (S3/Blob)

Older queries take longer but they still work. And our cloud bill stays sane.

Query Guards: No More Dashboard DDOS

One unbounded dashboard query can spike CPU, cost, and chaos. Guardrails prevent that:

• Max result rows and timeouts per query

• Query cost estimation before execution

• Pre-aggregated tables for common lookups

• Alerting on slow queries or dashboard abuse

This keeps the system fast and fair for everyone.

What We Gained: Speed, Scale, and Smiles

Our investment paid off. Here’s what we unlocked.

Sub-Second Queries (Even at Peak)

95% of queries now land under 400ms*, including:*

• Retention curves

• Daily active users by segment

• Conversion breakdowns

• Fraud spike heatmaps

Even during peak hours, query tail latency is minimal. The system handles hundreds of QPS with ease*.*

Cost Efficiency via Smart Tiering

By separating hot/warm/cold data, we reduced storage costs by 40%**, while still retaining 180+ days of queryable history.

No massive warehouse bills. No BI licensing nightmares.

Happier PMs, Analysts & Engineers

• PMs get instant feedback on feature launches.

• Analysts explore funnels interactively.

• Engineers build dashboards confidently, without slowdowns.

We went from “wait for the data” to “data waits for you.”

Final Thoughts: Building for Real-Time Thinking

The future of analytics is interactive, real-time, and embedded. Teams want data not just fast, but live. And while most traditional stacks break under that demand, we found a way to scale without compromise.

Here’s what we learned:

• Latency kills insight. Anything over 1s is friction.

• Real-time isn’t just freshness. It’s about experience.

• Speed changes behavior. When dashboards feel like apps, people use them like apps.

• Event streams are the new tables. Schema, ingestion, and storage must evolve with them.

And above all:

If your analytics layer isn’t as fast as your product, it’s not analytics, it’s archaeology.

5 Quick Wins for Teams Building Real-Time Analytics

If you're on the same path, here are our top tips:

1. Design for queries, not for data
   Start with what the product team needs and work backward into schema and ingestion.

2. Use time partitioning religiously
   It boosts freshness, speeds queries, and helps with retention strategies.

3. Build a query API layer
   Decouple frontend from datastore logic, add caching, guards, and metrics.

4. Roll up and index smartly
   Star-trees, bloom filters, and dictionary encoding save your compute budget.

5. Guard the last mile
   No open-ended filters. Put caps on rows, query time, and cost.

Closing: From Data to Delight

At Deuex, product analytics is not an afterthought. It’s a core feature of how we operate, iterate, and win. By rethinking the pipeline from event to insight, we transformed the user experience of data itself.

Dashboards are no longer slow.

They’re instant, delightful, and built for scale.