* Spyne uses ClickPipes to power its real-time CDC pipeline, streaming changes from MySQL and MongoDB sources directly into ClickHouse Cloud.  
* They migrated from a self-managed Debezium/Kafka stack to ClickPipes, eliminating 1.5+ hours of manual work per table onboarding and resolving schema drift issues.  
* Automated schema evolution in ClickPipes replaced a slow, error-prone manual process, drastically improving developer velocity and data consistency.

Summary

[Spyne](https://www.spyne.ai/) is an AI-native automotive retail tech company that helps dealerships sell faster and engage smarter. Its two flagship products handle opposite ends of the dealership workflow: [Studio AI](https://www.spyne.ai/virtual-car-photography-studio) transforms raw vehicle photos into studio-quality images, spins, and videos for digital storefronts, while [Vini AI](https://www.vini.mx/) acts as an always-on AI sales assistant, answering calls and chats, qualifying leads, and booking appointments.

Both products depend on data being accurate and up-to-date. But as the company grew and added new features, the team responsible for keeping that data flowing was spending more time maintaining the pipeline than analyzing what flowed through it.

At a [January 2026 ClickHouse meetup in Gurgaon](https://clickhouse.com/videos/202601-gurgaon-meetup-spyneai), DevOps engineer Abhash Solanki shared how Spyne simplified their CDC pipeline, migrating from a self-managed setup built on Debezium and Kafka to serverless [ClickPipes](https://clickhouse.com/cloud/clickpipes) and [ClickHouse Cloud](https://console.clickhouse.cloud/).

<iframe width="768" height="432" src="https://www.youtube.com/embed/gNJENprGhm0" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe>

## Infrastructure toil and schema drift

Spyne's old CDC pipeline was, in Abhash's words, a "complex beast, an intricate web of open-source tools and custom configurations."

Source databases in MySQL and MongoDB fed into Debezium connectors, which read changes from binary and operation logs in real time and published them as messages to a Kafka cluster. The Kafka cluster then fed those into a self-hosted ClickHouse instance, with a [Kafka engine table](https://clickhouse.com/docs/engines/table-engines/integrations/kafka) consuming the stream and flattening each message into a single JSON payload. From there, [materialized views](https://clickhouse.com/docs/materialized-views) (specific to the schema) transformed the flattened JSONs into final [ReplacingMergeTree tables](https://clickhouse.com/docs/engines/table-engines/mergetree-family/replacingmergetree), ClickHouse's mechanism for handling updates efficiently.

![Spyne AI Diagrams MDSN-54.png](/uploads/Spyne_AI_Diagrams_MDSN_54_af64f3e4a7.png)

*Spyne's old CDC pipeline: MySQL/MongoDB sources flowing through Debezium connectors, a Kafka cluster, and ClickHouse's Kafka engine table and materialized views into ReplacingMergeTree tables.*

The architecture, while functional, created what Abhash calls significant "operational toil," demanding constant attention and consuming valuable engineering hours.

The first major bottleneck was onboarding new tables. Every new table triggered a manual process: raise a DevOps ticket, map data types from MySQL or MongoDB to ClickHouse-compatible DDL (a manual, error-prone process Abhash calls the "type mismatch nightmare"), write the Kafka engine table, create the materialized view, update the Debezium connector config, and restart the pipeline, risking downtime.

"Adding a new table would take at least 1 to 1.5 hours of manual intervention and manual workload," Abhash says. "Whenever a new product release or a new analytical table has to be added, this whole cycle has to be performed."

That wasn't even the biggest pain point. The main problem, as Abhash describes it, was "schema drift nightmares." In a dynamic dev environment like Spyne's, source tables change frequently; new columns get added, data types get altered. When that happened, Debezium would faithfully capture the change and Kafka would deliver it, but the downstream materialized view, locked to its original DDL, would "silently ignore" the new data, leading to data inconsistencies and a lack of visibility into critical new features.

Identifying the gap could take days, after which the fix required stopping the pipeline, altering the materialized view and final table, restarting everything, and often resyncing the whole table from scratch. This manual intervention created major delays and increased the risk of errors, making every release, as Abhash puts it, a "high-stakes operation."

## Embracing simplicity with ClickPipes

When Abhash went looking for a more robust and manageable solution, he didn't have to look far. Spyne was already self-hosting ClickHouse, and he saw that the database's managed service, [ClickHouse Cloud](https://console.clickhouse.cloud/), offered a native ingestion engine, [ClickPipes](https://clickhouse.com/cloud/clickpipes). "The promise of a fully managed service, designed for seamless CDC, was compelling," he says.

With ClickHouse Cloud's efficient and easy-to-use UI, setting up a new data pipeline now takes "mere minutes," a welcome change from the meticulous Debezium configuration and schema mapping of the old system. "For a person who was manually adding each table and generating schemas for it, it felt like magic," Abhash says.

![Spyne AI Diagrams.png](/uploads/Spyne_AI_Diagrams_0f719848ca.png)

*Spyne's new CDC pipeline, with sources flowing directly into ClickHouse Cloud via ClickPipes.*

The biggest win, as he puts it, is "automated schema evolution." ClickPipes automatically detects upstream changes in sources like MongoDB and MySQL, then applies them to the corresponding ClickHouse tables instantly, with no manual intervention or pipeline restarts. This "zero-touch maintenance" has drastically improved developer velocity, allowing features to be deployed and data to be analyzed with "unprecedented speed and reliability."

## Solving challenges along the way

While ClickPipes immediately solved the team's main challenges, Abhash is quick to point out that "managed services aren't always magic… they also have some toils of their own." He notes that even with advanced platforms like ClickHouse Cloud, you still need to understand how things work under the hood and be willing to adapt how you operate.

As the team got to know the platform, they encountered specific architectural nuances and behaviors that required them to evolve their SQL patterns and operational awareness to fully leverage its benefits. "This was less about fixing flaws," Abhash says, "and more about optimizing our approach for the new paradigm."

Interested in seeing how ClickHouse works on your data? Get started with ClickHouse Cloud in minutes and receive $300 in free credits.

Get started today


## Challenge #1: The "snapshot" block

[Adding a new table](https://clickhouse.com/docs/integrations/clickpipes/postgres/add_table) to an active ClickPipe causes the entire pipe to pause its CDC streaming while it snapshots the new table. For Spyne, where new tables are added regularly, this created lag in analytics dashboards and interrupted real-time data flow.

The solution was careful ClickPipes planning. Abhash and the team isolated the 10 or so most critical large historical tables (i.e. those requiring real-time analytics at all times) into dedicated ClickPipes. New and lower-priority tables were grouped separately, so that when a snapshot pause occurs, production analytics remain unaffected.

## Challenge #2: The "consistency trap" with ReplacingMergeTree

ClickPipes uses ClickHouse's ReplacingMergeTree engine for efficient data ingestion and deduplication. While powerful, this introduces the concept of [eventual consistency](https://en.wikipedia.org/wiki/Eventual_consistency), which Abhash acknowledges caught the team off guard initially.

What it means is that when a row is updated, two versions briefly coexist (the old and the new) until a background merge resolves them. In Spyne's self-hosted setup, they had tuned this process to complete in less than a minute. But that level of control wasn't available in a managed environment like ClickHouse Cloud.

The team's solution was to add [`FINAL`](https://clickhouse.com/docs/sql-reference/statements/select/from#final-modifier) to their downstream queries, forcing deduplication at query time rather than waiting for the background merge. The trade-off, as Abhash notes, is increased CPU usage on the ClickHouse cluster due to the on-demand deduplication, but it guarantees accurate results in the dashboards that matter most.

## Challenge #3: The "all-or-nothing" recovery risk

When a source database outage (e.g. an AWS RDS failover) causes binary log loss, it creates what Abhash calls the "nightmare scenario": a binlog gap that breaks the CDC chain, invalidating the replication slot and halting data flow.

With self-managed Debezium, the team could perform a targeted partial resync, reingesting only the data from the window around the outage. But with ClickPipes, that granular recovery control isn't available. Recovery often requires a full table resync, which for Spyne can mean reingesting terabytes of data, driving up both ingress costs and latency.

For planned downtime, they can prevent the binlog gap from occurring by pausing the ClickPipe before the maintenance window begins. But for unplanned outages (which he notes are rare in managed services) there's no clean fix. "We have to adapt to it," he says.

## Less engineering toil, more time analyzing data

Throughout the migration, Abhash was clear about the tradeoffs. "Managing a self-managed pipeline gave us intricate control, but it had a very manual way of ingesting data into ClickHouse," he says. "Moving to ClickPipes has eliminated our operational toil."

With [ClickHouse Cloud](https://console.clickhouse.cloud/), Spyne has successfully removed the engineering toil of managing Debezium configurations, Kafka, and complex schema mappings. They've automated schema evolution with ClickPipes, dramatically improving velocity and data consistency. And they've learned to optimize queries for ReplacingMergeTree and strategically manage costs under ClickHouse Cloud's consumption-based pricing model.

As Abhash puts it, "The trade-offs were undeniably worth it for the enhanced stability and significant speed gains we achieved in our data pipelines at Spyne."

Spyne cut table onboarding from 1.5+ hours to minutes and eliminated schema drift by migrating their CDC pipeline from a self-managed Debezium/Kafka stack to ClickPipes and ClickHouse Cloud.

large_Spyne AI Customer Story Cover.jpg

medium_Spyne AI Customer Story Cover.jpg

small_Spyne AI Customer Story Cover.jpg

thumbnail_Spyne AI Customer Story Cover.jpg

Spyne AI Customer Story Cover.jpg

How Spyne simplified their CDC pipeline with ClickPipes and ClickHouse Cloud

- METRO Markets uses ClickHouse Cloud to power data warehousing, real-time seller analytics, credit risk modeling, observability, new AI use cases, and more.
- After outgrowing a self-hosted Hadoop-based stack, METRO Markets consolidated all company data into a single warehouse for the first time in its history.
- METRO Markets observed substantially faster query performance and lower storage costs compared to BigQuery and Snowflake, and adoption spread faster than anyone anticipated.

Across Europe, the big blue-and-yellow [METRO](https://www.metroag.de/en) wholesale stores are a familiar sight. For six decades, the cash-and-carry warehouses have supplied the continent's restaurants, hotels, and caterers, spanning over 600 locations across more than 20 countries, and cementing METRO's place as one of the largest retailers in the world.

But in 2018, a team inside the company looked at the B2B commerce landscape and saw a gap. Even a company of METRO's scale could only offer its customers a fraction of what they needed to run their businesses. An online marketplace, purpose-built for the hospitality industry, promised a far wider selection than any physical store could ever stock.

[METRO Markets](https://metro-markets.de/) launched in Germany in September 2019 and today operates across six European countries, with over a million products available to the hotels, restaurants, and catering businesses that form its customer base. Behind the scenes, a lean data team powers the intelligence that keeps the platform running, from seller analytics and credit risk scoring to behavioral data and operational logging, all of it flowing through [ClickHouse Cloud](https://clickhouse.com/cloud).

We caught up with the METRO Markets team to find out how their search for a better data warehouse led to something far more valuable and ambitious: a single platform that gives every team in the company a shared view of reality, in real time.

## Growing pains in the data layer

When Luis Jurado joined in 2020 as Data Engineering Team Lead, he arrived into a company growing faster than its infrastructure. Their self-hosted data stack was built on Impala and Kudu, tools from the Hadoop ecosystem that made sense at a certain scale but were showing their limits. "It grew out of hand as the company grew," he says. "There were a lot of in-house maintenance issues, and it wasn't easy to scale."

As the business scaled, data reliability became a growing concern. Engineering teams built parallel data stores to keep pace with demand, but business reporting remained dependent on Impala, which increasingly struggled to meet the reliability and consistency expectations of growing METRO Markets. "We had different teams looking at different numbers," says Hakan Dilaver, Head of Engineering Operations and Security. For a company making real-time decisions about pricing, logistics, and credit, that misalignment came with real costs.

There was a human cost, too. Despite the company's growth, the data team hadn't grown as fast as other parts of the business. Markus Nutz, Data Engineering and Science Lead, describes talented people spending too much time checking whether something had broken overnight rather than doing work that actually moved the needle. They needed something to make everyone's lives easier… or as Markus puts it, "something that just works."

## The road less traveled

When they began looking at alternatives, the obvious choice was Google BigQuery. It's what the broader METRO group uses, and, as Luis says, "Choosing tooling that differs from the rest of the group always requires some extra explanation and consensus."

Even so, the team ran a proper analysis, testing BigQuery alongside Snowflake and ClickHouse against the actual queries they run in production. On queries large and small, Luis says, "ClickHouse ran faster." On one particularly demanding query, we observed a significant difference in completion time between platforms. On another query, response time dropped from four seconds to under 200 milliseconds.

But performance alone wasn't going to win the argument. "From a business standpoint, there was also a need to predict our costs," Luis says. "Cost predictability was a challenge with alternatives. While controls were available to cap spending, we found they came with performance trade offs that didn't work for our use case." ClickHouse's [pricing model](https://clickhouse.com/pricing?plan=scale&provider=aws&region=us-east-1&hours=8&storageCompressed=false), by contrast, made it possible to forecast expenditure without sacrificing query speed.

"We needed the next step in our data journey," Luis says. "We did an analysis of a few tools, tested some of them, and landed on ClickHouse."

## Building the foundation

In the current architecture, Airbyte handles the majority of data movement, pulling from MySQL and Postgres sources via CDC connectors and writing into ClickHouse, though the team plans to test ClickHouse's native ingestion engine, [ClickPipes](https://clickhouse.com/cloud/clickpipes), once GCP support is available. For deduplication during background merges, they use ClickHouse's [ReplacingMergeTree engine](https://clickhouse.com/docs/engines/table-engines/mergetree-family/replacingmergetree) with an experimental cleanup feature. "We look forward to using ClickPipes when it is available for us on GCP to further simplify this infrastructure," Markus says.

One particular challenge was Google Analytics data, which lives in BigQuery and couldn't be moved through standard connectors. The team solved it by dumping the data into cloud storage buckets and loading it into ClickHouse via external tables. Markus calls it a "big step forward," adding, "We now finally have all our company data in one data warehouse."

On the consumption side, Luis says ClickHouse's broad compatibility has been as important as its ingestion capabilities. "When we were on Impala, the only drivers available were ODBC or JDBC," he explains. "Our services are mostly written in PHP. We couldn't use Grafana or Looker Studio." ClickHouse's MySQL-protocol endpoint changed that, allowing developers and business analysts to query the same data through whatever tool they prefer.

## One data platform, many stories

When METRO Markets adopted ClickHouse, nobody anticipated how quickly it would expand beyond its original mandate. What began as a data warehousing project has become a single, scalable platform spanning storage, analytics, machine learning, and observability.

Today, ClickHouse serves as the company's central [data warehouse](https://clickhouse.com/resources/engineering/choosing-cloud-data-warehouse), consolidating transactional data, behavioral data, and MongoDB collections in one place. "That's something we hadn't managed since METRO Markets started," Markus says.

Built atop that foundation is a growing layer of [real-time analytics](https://clickhouse.com/resources/engineering/what-is-real-time-analytics). Merchants who sell on the platform can see, in near real time, whether they're delivering on time, how competitive their pricing is, and how their products are performing. Because it directly affects how sellers run their businesses, the data needs to be fast and available. ClickHouse handles both.

ClickHouse's [Query API endpoints feature](https://clickhouse.com/docs/cloud/get-started/query-endpoints) (which Markus calls a "nice surprise… the documentation sells it a little bit short") have become central to several of these use cases. The team uses them to surface user characteristics like VIP status and account flags to downstream systems instantly. They also power a lightweight [feature store](https://clickhouse.com/blog/powering-featurestores-with-clickhouse) for the company's credit risk model. For every transaction, user features are fetched from ClickHouse and a prediction is generated end to end, without any intermediary systems. "If you think about how you'd normally build a feature store — the infrastructure, the ingestion pipelines — it was a bit of a gamble," Markus says. "But we took the bet, and it's working very well."

Teams are also starting to use ClickHouse for [observability](https://clickhouse.com/resources/engineering/observability). Engineers across the company use it to store and interrogate operational logs, particularly API logs from the seller-facing platform, where being able to run ad hoc counts and rate-limit queries without building a separate metrics layer saves significant time. "With ClickHouse, you can just go and query it," Luis says. "That would be really difficult with standard logs." Hakan adds that the team is exploring moving one of their largest log partitions from Datadog into ClickHouse.

And then there's AI, the use case nobody planned for. Several internal teams are exploring LLM-powered tools to help customer-facing staff look up order information and resolve issues faster. The path of least resistance to the relevant data almost always leads to ClickHouse. It's an example of something [ClickHouse is seeing accelerate across the industry](https://clickhouse.com/blog/ai-redrawing-database-market): as AI agents become the primary interface to data, the database underneath them needs to be fast, concurrent, and always on.

## Cheaper, faster, more reliable

ClickHouse's impact shows up first in storage, which, as Luis explains, leads directly to costs. "When I talk to Hakan, I always say things like, 'this terabyte of data in MySQL costs X amount of dollars to store, but move it to ClickHouse and now it's 100 megabytes, which is also cheaper to store per gigabyte,'" he says. "So it's a clear win."

The decision to go with a managed solution has also proven important. Having lived through the operational burden of running Impala and Kudu themselves, the team made a deliberate choice to let [ClickHouse Cloud](https://clickhouse.com/cloud) handle the infrastructure. As Luis puts it, "We found the pain of self-hosting in our previous solution and didn't want to take that risk again."

"The biggest benefit for me," Markus says, "is having all the data available and being able to query it fast." Benchmark testing against our production queries indicated that ClickHouse was better suited to our particular use case and query patterns.

Hakan agrees: "The most important part was being able to centralize our data operations in a solution that's fast and reliable." That reliability has resolved the misalignment that once plagued the company. Instead of different teams looking at different data sources and getting different versions of the truth, business and engineering draw from the same warehouse, through the tool of their choosing, and the platform stays up.

## The journey continues

Across METRO Markets, teams the data platform never even targeted have found reasons to build on ClickHouse. "The benefits and the number of things we've ended up using it for, because of the feature set, has been more than we anticipated," Luis says.

The journey, he adds, is far from over. "We want to incorporate a lot more things in the future," he says. Next on the roadmap is connecting their Kafka and Flink streaming infrastructure to ClickHouse, opening the door to real-time stream processing use cases that will take the platform further still.

With ClickHouse, what was initially scoped as a data warehousing project has become, quietly and organically, the do-it-all data infrastructure for the whole company.

How METRO Markets replaced a failing Hadoop-based stack with ClickHouse Cloud to build a single platform now powering data warehousing, real-time seller analytics, credit risk modeling, observability, and AI across the whole company.

large_Metro Markets Customer Story blog cover - yellow.jpg

medium_Metro Markets Customer Story blog cover - yellow.jpg

small_Metro Markets Customer Story blog cover - yellow.jpg

thumbnail_Metro Markets Customer Story blog cover - yellow.jpg

Metro Markets Customer Story blog cover - yellow.jpg

Beyond the warehouse: How METRO Markets built a do-it-all data platform on ClickHouse Cloud

There has been a lot of discussion recently around real-time analytics. For years, this was often dismissed as a small use case, relevant to a smaller set of operational or customer-facing workloads. We always believed otherwise. 

Customer-facing analytics, observability, fraud detection, operational applications, and AI agents all depend on fresh data and low-latency queries.

It seems others are now recognizing the category that ClickHouse and our customers have defined for years. It reflects the journey ClickHouse has been on, from an open-source project to serving more than 4,000 customers and over[ $250M in ARR](https://clickhouse.com/blog/thank-you-for-building-with-us). 

It feels like the right moment to revisit two principles we believe matter: openness and a clear definition of what a real-time analytics database actually is.

## Real-time claims deserve real benchmarks

The data industry has a long history of benefiting from open benchmarks. Whether it is [ClickBench](https://benchmark.clickhouse.com), [CostBench](https://clickhouse.com/benchmarks/costbench), [PostgresBench](https://postgresbench.clickhouse.com/), [TPC benchmarks](https://clickhouse.com/blog/tpc-h-clickhouse-cloud-vs-snowflake-databricks-bigquery-redshift), or community-led testing, customers are best served when performance claims can be independently verified.

A useful benchmark should be transparent about what is being measured, reproducible by anyone willing to run it, and clear about the conditions under which the results were achieved. Otherwise, it becomes difficult to distinguish between a meaningful measurement and a marketing claim.

![clickbench.png](/uploads/Blog_Costs_014_291362ced8.png)

> We believe benchmarks should be open, transparent, and reproducible. All of ClickHouse’s benchmarks are publicly available and reproducible, including [ClickBench](https://benchmark.clickhouse.com), [PostgresBench](https://postgresbench.clickhouse.com/), [JSONBench](https://jsonbench.com) and our recently announced [CostBench](https://clickhouse.com/benchmarks/costbench). You can explore the methodology, datasets, queries, and results on our dedicated benchmarks page and reproduce the results yourself.

This is particularly important in real-time analytics, where benchmark results can vary dramatically depending on what is actually being tested. Two systems can produce similar latency numbers under controlled conditions while behaving very differently in production. 

A benchmark showing sub-second queries on a static dataset is useful, but it does not necessarily tell us how a system behaves when new data is continuously arriving, when working sets exceed memory, or when hundreds or thousands of concurrent queries are competing for resources.

As more attention is given to real-time analytics, we hope the industry continues moving toward benchmarks that are open, reproducible, and explicit about what they are measuring. 

## What actually makes a real-time analytics database?

Our experience building systems for this workload has taught us that many of the properties that determine success in production are also the easiest things to omit from a benchmark. Freshness, continuous ingestion, concurrency, transformation latency, and efficiency at scale rarely fit neatly into a single chart on stage, yet they often determine whether a system succeeds or fails once it reaches production.

For this reason, it is worth asking not only how real-time analytics should be benchmarked, but what a real-time analytics database should actually be expected to do. Coincidentally, we recently explored this [question in depth](https://clickhouse.com/blog/selecting-a-real-time-analytical-database). In summary, any platform claiming real-time analytics capabilities should be able to answer a few simple questions:

### Does ingestion scale without degrading queries? 

Continuous writes should not pull query latency along with them. In practice, that means an ingestion path that can run on isolated resources from the read path, and a concurrency model that holds tail latencies bounded as writers grow. ClickHouse, for example, uses consensus-based coordination through Keeper rather than optimistic concurrency. It pays a small coordination cost on every write, but keeps tail latencies predictable at the high insert concurrency seen in real-time workloads.

### Is data queryable within a second or two of being written? 

At real-world scale, the realistic target is a delay measured in milliseconds to a couple of seconds, with no catalog or manifest refresh sitting in the middle.

### Do transformations update incrementally, and not on a schedule? 

Materialized views, rollups, and pre-aggregations (done using AggregatingMergeTree in ClickHouse) should update with each insert, not on an interval. This ensures that they're immediately applied and enforce a delay on data availability. 

### Can you bias the system towards read-time or write-time work? 

Different workloads want different trade-offs. For some, you want indexes and aggregations built at write time so the freshest queries are immediately accelerated. For others, you want to keep the ingest throughput maximal and let the background work catch up. A real-time platform should expose that choice without forcing it. And index maintenance should never block ingestion. New data should be queryable as soon as it is committed, with or without its indexes fully built.

### Does a single engine serve all workloads? 

Text search, vector similarity, JSON, structured analytics. These should live in the same storage and query engine, not in four loosely coupled systems with their own freshness floors. 

### Is the performance shown on hot data, not just on cached benchmarks? 

The realistic question to ask of any platform is how fast its queries are on data that arrived a second ago, not how fast its repeated queries are on yesterday's table. The first number is what production looks like.

### Does resource efficiency hold up at scale? 

Real-time systems need to remain efficient as ingest volume, query concurrency, and retention grow. The important question is whether performance scales linearly with workload, or whether latency and compute costs begin to degrade disproportionately as traffic increases.

The common thread across these requirements is that real-time analytics is not a feature you switch on. It is a set of design decisions about ingest, storage, transformation, and query that have to be made consistently across the whole system. Either the platform was designed for this workload, or it is working hard to approximate it.

## Real-time analytics comes of age

One question worth asking is why real-time analytics has suddenly moved to the forefront. While several trends are contributing, we believe the rise of agentic workloads is [a major factor](https://clickhouse.com/blog/ai-redrawing-database-market). Agents consume data differently from humans. They issue more queries, operate continuously, and depend on fresh context to make decisions. As organizations deploy more of them, the combination of low latency, high concurrency, and fresh data is becoming a prerequisite rather than a luxury.

As the category continues to evolve, we hope two things become standard. First, open and reproducible benchmarks that allow customers to independently evaluate performance claims under realistic conditions. Second, a shared understanding of what real-time analytics actually requires beyond a single latency number. 

As the yellow company, we have been leading in real-time analytics for years, and believe it to be a foundational workload.

It’s nice to see the other colors finally recognizing it too.



For years, real-time analytics was treated as a niche requirement. We believed otherwise. So why is everyone talking about it now?

large_Why everyone is talking about real-time analytics (and the color yellow).jpg

medium_Why everyone is talking about real-time analytics (and the color yellow).jpg

small_Why everyone is talking about real-time analytics (and the color yellow).jpg

thumbnail_Why everyone is talking about real-time analytics (and the color yellow).jpg

Why everyone is talking about real-time analytics (and the color yellow).jpg

Why everyone is talking about real-time analytics (a note from “the yellow company”)

- Appcues uses ClickHouse Cloud to support real-time analytics and segmentation across 1.31 PB of data and 410 billion events.
- While their previous technology stack, which included Airflow and Snowflake, worked well for batch reporting and dashboards, Appcues wanted to innovate and deliver real-time, customer-facing analytics.
- Now, Appcues cut P95 query times by 90% from 20+ seconds to <2 seconds, and reduced ingestion latency by 99% from 10+ minutes to 5 seconds.
- Even as workloads expanded to support new capabilities, Appcues reduced analytics spend by 23%.


[Appcues](https://www.appcues.com/), a platform that helps SaaS companies engage their customers to drive growth and lives inside its customers’ products, powering things like onboarding flows, tooltips, checklists, banners, emails, and push notifications. Customers send Appcues data from SDKs, Segment streams, CRMs, and public APIs so Appcues can help them, as Chris Brookins, VP of Engineering puts it, “deliver the right experience to the right customer at the right time.”

Today, Appcues manages 1.31 PB of data, has processed 410 billion events, and continuously updates over 1 billion user profiles. “We’re ingesting a ton of data from all these apps,” Chris says, “and there’s a lot of observability we need to provide.” Much of that data shows up in Appcues Studio, where customers track performance and outcomes. 

> “It’s very important that all this information is aggregated and delivered quickly for over a billion user profiles. Page load time has to be practically instant.”
>
> — Chris Brookins, VP of Engineering, Appcues

Appcues had spent years running its analytics platform on a stack that included Airflow and Snowflake. While Snowflake remains a strong solution for BI and reporting, delivering the real-time product analytics experience that customers increasingly expect requires a different approach. At our Boston meetup, Chris shared the journey that ultimately led them to [ClickHouse Cloud](https://clickhouse.com/cloud) on AWS.

## Challenges to delivering a delightful product experience

From the beginning, the platform ran on two main databases. DynamoDB handled the sub-second work of matching live visitors to segments, so Appcues could decide, before the next page load, which experiences to show. Snowflake powered analytics, including dashboards, performance graphs, NPS results, and the reporting customers rely on to tell whether their experiences are actually working. 

“Our aim is to create a delightful product experience for our customers,” Chris says. However, as Appcues scaled to ingest more and more data, query times went up to 20 seconds and costs became a challenge to manage as their previous stack wasn’t meant for low-latency analytics at scale. 

To increase query speeds, Appcues relied on Airflow. Every five to 10 minutes, jobs would roll up and pre-process data before writing it back. This delivered the product experience they were looking for, but it added costs and meant maintaining a growing web of jobs and servers. This setup also meant that customers were limited to looking at data in 10 minute windows.

When the team decided to introduce email, Chris' team knew they needed a new solution that was highly performant for real-time analytics and cost efficient. “That was a whole new channel with all sorts of new events and extra segmentation needs,” Chris says. Historically, Appcues was optimized to answer a direct, real-time question: *Which segments does this user belong to*? But with email, they needed to calculate, very quickly, *all the users in a segment*, across hundreds of segments per customer, with complex conditions based on historical behavior and even negative logic like “hasn’t been seen in 30 days.”

## Putting ClickHouse to the test

They started with a list of must-haves. As a small team, they wanted minimal operational overhead. They also wanted an open-source engine for transparency and optionality. After their experience with their previous cloud data warehouse, cost efficiency at scale was a huge priority. And any solution had to support HIPAA-compliant workloads and a future EU data center region. “It was a lot of requirements,” Chris admits. “We wanted to stress test everything possible.” He sent principal engineer Andy LeClair to a ClickHouse meetup to learn more, and those conversations became the start of an evaluation plan.

The first step was to export all Snowflake data to Parquet in S3 and load it into ClickHouse. Appcues also notified customers that ClickHouse would be used as a new sub-processor, since the platform stores a lot of personally identifiable information. 

The proof of concept focused on the email segmentation challenge. “A lot of these segments were very complicated,” Chris explains. “They had AND/OR conditions, nested conditions. They could refer to other segments of users or historical events, like, ‘Has this thing ever occurred in the last three months?’ It was a lot of work.”

In parallel, the team began laying the foundation for production. Events flowed through Kafka into a new “clickwriter” consumer for light deduplication and fast inserts. On the application side, the segment logic was formalized into a “clickclient” library, built on Elixir’s Ecto, giving the team a single, controlled way to generate complex ClickHouse queries.

Testing both their previous data warehouse and ClickHouse, they tuned each platform as far as they could. In ClickHouse’s case, features like [SharedReplacingMergeTree](https://clickhouse.com/docs/cloud/reference/shared-merge-tree) proved “huge in terms of improving performance,” Chris says.

By the end, the outcome was clear: ClickHouse ran the segment queries 73% faster. 

> “No matter how we tuned our previous cloud data warehouse, ClickHouse was faster. We were like, ‘Alright. This is a go.’”
>
> — Chris Brookins, VP of Engineering, Appcues

## Migrating to ClickHouse Cloud

From there, the project became a one-year, in-flight migration. “We were doing this while the plane was flying,” Chris says. “We had to ensure there was accuracy all through the project.”

With historical data already loaded into ClickHouse, the next phase was a steady loop of verification and iteration. The team refined their schema, learned how ClickHouse wanted data to be modeled, and worked closely with the ClickHouse team to choose the right [table engines](https://clickhouse.com/docs/engines/table-engines) and [materialized views](https://clickhouse.com/docs/materialized-views). Chris says. “The migration taught us a lot of positive practices. We developed a lot of discipline in terms of data management, moving to ClickHouse,” says Chris.

That discipline mattered, because Appcues’ data is hardly uniform. “Customers can send us anything,” Chris says. “It’s just key-value pairs.” Reports had to work across wildly different customer data shapes, from small startups to large enterprises. The team tested continuously across accounts, using edge cases to pressure-test the system before exposing it broadly.

As confidence grew, they began operationalizing ClickHouse Cloud in earnest. Different workloads were isolated into appropriately sized services, [autoscaling](https://clickhouse.com/docs/manage/scaling) rules were tuned, and the team “rinsed and repeated” until it was stable. Only then did they begin cutting customers over in waves. 

For those enterprise customers especially, the difference was night and day. “With our previous setup, there were queries that would take a week to complete, and other things they just couldn’t even do,” Chris says. “Our largest customers are thrilled.”


## Appcues’ new ClickHouse-based setup

By the time Chris took the stage in Boston, Appcues’ analytics system looked very different. At a high level, it now takes a noisy stream of key-value events and turns it into fast, queryable tables that match how the product actually needs to reason about users and experiences.

All events flow through Kafka into Appcues’ “clickwriter” consumer, which performs light deduplication before inserting data into ClickHouse. From there, data is split into two main families: experience analytics—impressions, completions, surveys, and outcomes tied to customer workflows—and user and group profiles, built from streams of property updates that must resolve into a clean, current-state view.

The deployment model reflects Appcues’ scale and compliance needs. Their primary ClickHouse warehouse runs five services with two to three replicas each, thirteen replicas total. Those services take advantage of what Chris calls “ClickHouse’s very fast upscaling and downscaling,” growing from modest instances with 16 GB of memory and 4 vCPUs to machines with over 1 TB of RAM and hundreds of cores during peak demand, then scaling back down to manage cost. A separate, fully isolated warehouse serves HIPAA customers.

[Materialized views](https://clickhouse.com/docs/materialized-views) are key to making this work at scale. Where Airflow once rolled up data every few minutes, ClickHouse now precomputes counts and sums at insert time. “That dramatically improved all our queries,” Chris says. “We were able to eliminate all of our Airflow infrastructure—all of that cost and, most importantly, all of that latency.”

For profile data, [SharedReplacingMergeTree](https://clickhouse.com/docs/cloud/reference/shared-merge-tree) sits at the heart of the design. Updates arrive as events, and ClickHouse resolves them into flattened profile tables. One important optimization was learning when *not* to use [FINAL](https://clickhouse.com/docs/sql-reference/statements/select/from#final-modifier). “We were trying to do all this in real time and not slow anything down,” Chris says. “We found that using [argMax](https://clickhouse.com/docs/sql-reference/aggregate-functions/reference/argmax) instead took stress off the system, while allowing us to get the latest user profile with minimal overhead.”

Late in the project, a final challenge emerged: deletes. Appcues provides self-serve APIs for GDPR and test-data deletions, and some customers generate a constant stream of them. Traditional deletes triggered expensive background merges, and as volume grew, they found themselves facing a growing backlog, with GDPR’s 30-day delete clock ticking.

The solution came with [ClickHouse 25.6](https://clickhouse.com/blog/clickhouse-release-25-06) and the introduction of [lightweight deletes](https://clickhouse.com/docs/guides/developer/lightweight-delete). “Its lightweight patch-part mechanism allowed us to blow through these deletes immediately,” Chris says. “All of a sudden, we were past our major challenge. That was awesome.”

## Faster queries, fresher data, lower cost

Previously, P95 query times hovered around 20 seconds. On ClickHouse, P95 is under two seconds—a 90% improvement—despite the fact that Appcues has added more workloads and features on top of the system. Even P99 and max latencies, Chris says, are “not that much higher… they’re very reasonable.”

The improvements in ingestion latency are even more impressive. Previously, thanks to those Airflow-based rollups, it could take 10 minutes or more for new events to show up in reports. But with ClickHouse, it’s closer to five seconds. “That’s 99% faster from the moment an event hits our servers to the moment you can query the results in ClickHouse,” Chris says.

Cost, meanwhile, has dropped significantly even as capabilities have expanded. Even with all the new segmentation and email workloads layered into ClickHouse, reducing the overall analytics stack by 23%, costs now scale more predictably as usage grows. 

Part of that comes down to the basic economics of [how cloud data warehouses bill for compute](https://clickhouse.com/blog/how-cloud-data-warehouses-bill-you) and [how different execution and scaling models translate performance into cost](https://clickhouse.com/blog/cloud-data-warehouses-cost-performance-comparison). In a provisioned “warehouse” model, scaling performance and concurrency often means scaling the meter right along with it. ClickHouse Cloud gives Appcues a more flexible way to size and autoscale compute to the workload they actually have.

For Chris and the Appcues team, this is only the beginning. “We didn’t stop there,” he says. “It wasn’t just a parity project.” The team has already started to push the platform further, rolling out new funnel reports that use ClickHouse [window functions](https://clickhouse.com/docs/sql-reference/window-functions) to show how users are progressing through workflow and messaging pipelines. They’ve also added AI-powered natural-language insights, with agents leveraging the “clickclient” query library to answer customer questions about product and user activity, benchmarks, and product adoption.

Now with ClickHouse Cloud, Appcues finally has an analytics platform that can keep up with real-time product decisions, support new channels and features, and scale efficiently without becoming a drag on the business. 

> “We’re thrilled with the results, and thrilled with the support we got from the ClickHouse team.”
>
> — Chris Brookins, VP of Engineering, Appcues


Appcues cut P95 query times 90%, ingestion latency 99%, and analytics spend 23% by migrating from Snowflake and Airflow to ClickHouse Cloud for real-time segmentation across 1.31 PB of data.

large_Appcues Customer Story Cover.png

medium_Appcues Customer Story Cover.png

small_Appcues Customer Story Cover.png

thumbnail_Appcues Customer Story Cover.png

Appcues Customer Story Cover.png

Appcues delivers personalized customer engagement with ClickHouse Cloud

* Open Electricity uses ClickHouse to power an open platform and public API serving live Australian energy data across \~1 billion records.  
* Migrating from Postgres and TimescaleDB to ClickHouse delivered a 92x query speed improvement, with a benchmark query dropping from over 5 minutes to just over 3 seconds—and 60 milliseconds via materialized view.  
* ClickHouse runs on significantly less hardware: 4 cores, 4GB RAM, and 200GB disk versus Postgres’s 16 cores, 32GB RAM, and 1.6TB, at a fraction of the cost.

Something remarkable is happening to Australia's electricity grid. In the midst of a massive transformation, it has become one of the most closely watched energy systems on the planet.

"Twenty years ago, the grid was pretty boring," says Nik Cubrilovic, a developer at [Open Electricity](https://openelectricity.org.au/), an open platform tracking Australia's electricity transition. Back then, the country had a hundred-odd generators—coal-fired power stations, the Snowy scheme, a handful of gas plants. "You might add or retire a facility each year," Nik says.

In 2025 alone, Australia registered 40 new facilities. And the grid isn't just growing. "The composition of the grid is also changing," Nik says, highlighting that more than half of those new facilities are battery storage. "The capacity of the coal-fired power stations is decreasing, whereas batteries, solar power, and wind have really taken off."

![](/uploads/openelectricity_jun2026_image1_a384eadde2.png)

*Australia's electricity generation capacity by fuel type since 1999. Coal (black and brown) has declined, while batteries (dark blue), solar (yellow), and wind (green) have surged in recent years.*

"This is what makes our grid interesting, not just in Australia but globally," Nik says. "There are a lot of regulators and researchers all around the world who are interested in Australia's energy data to see how we're adapting with some of the challenges that come out of it."

At a [February 2026 ClickHouse meetup in Melbourne](https://clickhouse.com/videos/202602-melbourne-meetup-openelectricity), Nik shared how Open Electricity is making that data accessible, what led them to migrate over a billion rows to ClickHouse, and why he believes ClickHouse plus Postgres is a "magical pairing."

## A billion rows and counting

Open Electricity pulls its data from two grids: the NEM, or National Electricity Market, covering the eastern states from Tasmania to north Queensland, and the WEM, the Wholesale Electricity Market in Western Australia. Every five minutes, each generating unit reports how much power it produced, and the regulator publishes that data in what Nik describes as "really awkward CSV files" that Open Electricity must then parse and enrich.

The grid, Nik explains, has a hierarchy. Each network is made up of facilities, each facility is made up of units, and each unit has its own fuel technology. "A coal-fired power station might have four different units," he says. "A wind farm might have 30 different wind generators."

The data breaks down into three types: market data (price, demand, generation, and forecasts per region), generation data recorded per unit per interval going back to 1999, and curated facility data built by Open Electricity on top of the raw four-digit regulator codes, covering over 100 fields per facility. Altogether that adds up to around 1 billion records, with the platform adding another 500,000 each day.

The scale is easy to see when you start digging into charts on the Open Electricity [website](https://openelectricity.org.au/). The default 7-day view draws on over 32,000 data points—12 five-minute intervals per hour, 24 hours a day, seven days, across 16 fuel technology groups. The all-time view, stretching back to 1999, requires aggregating around 45 million data points out of that billion-row table, which users can slice by region, fuel type, or renewables versus non-renewables. All of it is served live through a public API used by around 300 companies and institutions.

![](/uploads/openelectricity_jun2026_image3_238c75f05e.png)

*Open Electricity's default view, showing seven days of generation by fuel technology type.*

![](/uploads/openelectricity_jun2026_image2_e5175d3c0f.png)

*The all-time view, aggregated from billions of 5-minute data points stretching back to 1999.*

For Nik, who has worked across finance, crypto, and everywhere in between, nothing matches energy data for its intensity and complexity. "Electricity networks are extremely complex, and there is intricacy in parsing the data to tell its story."

That complexity doesn't go away just because you've put it in a database. For a small team serving live data to hundreds of institutions around the world, every five minutes of every day, getting the infrastructure right matters enormously.

## From TimescaleDB to ClickHouse

For years, Open Electricity ran on a combination of Postgres and TimescaleDB. But with a billion rows and counting, the setup was starting to strain, with some queries taking upwards of five minutes to complete. The worst offenders were joins between market data and generation data. As Nik says, "Doing a join based on an interval and then going back through a billion rows gets pretty expensive pretty quickly."

TimescaleDB did offer some relief. Its time-series functions, particularly time_bucket and time_bucket_gapfill, were useful for a dataset full of gaps, since generators don't run continuously, and filling nulls before joining the data was a constant requirement.

But it also had serious limitations. Continuous aggregates didn't support time_bucket_gapfill, and there was no timezone handling in materialized views—a problem given that, as Nik explains, "We're dealing with east coast time zones and summing it up with west coast time zones and having to show an Australia-wide view on top of that." Queries on continuous aggregates were still slow, and refreshing materialized views locked tables. The team ended up selecting out of large queries and writing results into separate tables manually.

Materialization was the right instinct. What they needed was a purpose-built OLAP engine. "I've been using ClickHouse for longer than ClickHouse has been a company," Nik says. "It's a column store. The compression is great. It's a lot easier to query… if you know SQL, you can pretty quickly pick up some of the custom ClickHouse functions."

It also offered [multiple engine types](https://clickhouse.com/docs/engines/table-engines) for different aggregation patterns, [clear performance benefits](https://clickhouse.com/docs/concepts/why-clickhouse-is-so-fast), and [materialized views](https://clickhouse.com/docs/materialized-views) that Nik and the team have put to full use. "We've got so many different ways of representing energy data that it's very cheap with ClickHouse to create materialized views once you've got your base tables," he says. "We run over 20 of them now, and we keep adding more and more all the time. If we see a hot spot with a query, we just create a new materialized view for it and then point the query to that."

## A new architecture that's 92x faster

With ClickHouse added to the stack, Open Electricity's architecture snapped into a clean division of labor that Nik calls "the best of both worlds—we've got the heavy relational data still in Postgres, but all the analytic workloads are happening on ClickHouse now. There aren't many workloads that you can throw at it where it's just not going to chew through it."

At the core of the ClickHouse setup are two tables: unit_intervals, which stores per-unit 5-minute generation, emissions, and market value data; and market_summary, which holds per-region price and demand. Both use [ReplacingMergeTree](https://clickhouse.com/docs/engines/table-engines/mergetree-family/replacingmergetree), one of ClickHouse's specialized engine types designed for deduplication. "ClickHouse's [documentation](https://clickhouse.com/docs/) and the [GitHub community](https://github.com/ClickHouse/ClickHouse/discussions) around ClickHouse are fantastic," Nik says. "You can learn all about the different engine types and when to apply each one."

On top of those two base tables sits a hierarchy of materialized views, each tuned to its aggregation pattern. "The queries are very fast," Nik says. "Some of those Postgres queries that were taking on the order of minutes are now returning in milliseconds."

To benchmark performance properly, the team ran the same query on both systems, aggregating all-time energy by fuel technology, grouped by month, going back to 1999. Postgres took 313 seconds. ClickHouse took 3.4 seconds. "So it's about 92 times faster," Nik says. "And it gets even better," he adds, noting that with the monthly materialized view, the same query comes back in just 60 milliseconds.

Nik is quick to point out that the benchmark isn't a hardware-for-hardware comparison. Postgres runs on 16 cores, 32 GB of RAM, and 1.6 TB of disk. ClickHouse runs on 4 cores, 4 GB of RAM, and 200 GB—a fraction of the hardware, at a fraction of the cost.

## Lessons learned from the migration

Nik and the team opted for a gradual migration, running the old and new systems in parallel and moving traffic over bit by bit. Feature flags let them toggle between the two systems per endpoint, with identical API output so users never knew which database they were hitting. They could bucket a percentage of users into one or the other, compare results, and flip the switch further when confident.

"ClickHouse is so cheap to run that you can run it in parallel for as long as you need to perfect it," Nik says. "There's no reason you can't go to the office, load up a ClickHouse schema, and start running it alongside your existing workloads and flick it over bit by bit."

While they're happy with where they landed, they had to overcome a few "gotchas." The first was what Nik calls the "FINAL trap." ClickHouse's ReplacingMergeTree engine doesn't deduplicate at query time unless you explicitly add [FINAL](https://clickhouse.com/docs/sql-reference/statements/select/from#final-modifier) to your query. "It's important to understand how ClickHouse is architected," Nik says. Otherwise you can "wake up one morning and all your data is doubled."

The second was a bit subtler. Combining auto-populating materialized views with ReplacingMergeTree can lead to, as Nik puts it, a "disaster," with partial daily aggregates winning the "max version lottery" over complete ones. The team came up with a clever hack: encode data completeness directly into the version number, so a full-day backfill with 288 intervals always outranks a partial auto-populated aggregate with 10.

Timezones caused headaches, too. With east and west coast data flowing through a Python backend, multiple drivers, and a JavaScript frontend, there are around 20 places in the pipeline where dates are touched. "If you alternate between non-timezone and timezone fields, it's going to get mangled," Nik says. He recommends standardizing [DateTime64](https://clickhouse.com/docs/sql-reference/data-types/datetime64) with UTC from day one.

Other gotchas included memory management during backfills (ClickHouse favors [bulk inserts](https://clickhouse.com/docs/optimize/bulk-inserts), and the team found that chunking data into 3-day windows with 20,000-record batches was the most efficient approach) and disk space, where "out of memory" errors often turned out to be ClickHouse running out of temporary disk space for large operations. Nik also recommends matching timeouts to the type of operation, with user-facing queries timing out in seconds and longer-running backfill and optimization tasks given much more room to breathe.

"Each one of these touchpoints I could go on for days about—there's a lot of learned experience in our codebase," he says with a smile.

## ClickHouse + Postgres = "A magical pairing"

Open Electricity's move to ClickHouse has enabled the team to query data in ways that weren't practical before—live, at scale, across a billion rows, in milliseconds.

Nik closed his presentation with two demos that show that speed and flexibility in action: a real-time status page for every coal-fired generator in Australia, displaying capacity and output at every 5-minute interval; and a solar generation heatmap built that same day by a frontend developer, tracking hour-by-hour solar output from 2016 to 2025. "We're talking millisecond-scale queries now," Nik says.

"Postgres combined with ClickHouse is a magical pairing," he continues. "You've got Postgres for all your relational workloads and ClickHouse for all your analytical workloads. It's been working really well for us, allowing us to query data in ways we couldn't before."

For a small nonprofit tracking one of the world's most closely watched energy transitions, that pairing is what lets researchers, journalists, and policymakers see what's happening to Australia's grid in real time, and share that picture with the rest of the world.


Open Electricity migrated ~1 billion rows of Australian energy data from Postgres and TimescaleDB to ClickHouse, achieving 92x faster queries on a fraction of the hardware.

large_Open Electricity Blog Upload.jpg

medium_Open Electricity Blog Upload.jpg

small_Open Electricity Blog Upload.jpg

thumbnail_Open Electricity Blog Upload.jpg

Open Electricity Blog Upload.jpg

92x faster queries: How Open Electricity uses ClickHouse to track Australia’s energy transition in real time

* Cisco Talos runs a threat intelligence reputation service on ClickHouse Cloud, classifying file hashes so researchers can make fast, accurate security decisions.  
* The team migrated a nearly 2-trillion-row, 2 PB dataset from self-managed OSS ClickHouse to ClickHouse Cloud with zero downtime and a 100% data match.  
* The move cut storage costs by \~90%, reduced TCO by \~75%, and reclaimed over 300 engineer hours a year for building new products.

[Cisco Talos](https://talosintelligence.com/) is one of the world's most trusted cybersecurity threat intelligence teams, made up of expert researchers, analysts, incident responders, and engineers. They defend [Cisco](https://www.cisco.com/site/us/en/index.html) customers and raise awareness of evolving threats within the cybersecurity community, partnering with industry and government organizations around the globe.

Senay Goitom is part of a small engineering team that provides the centralized data lake and enrichments behind that mission. One of its most important offerings is a reputation service built on a massive hash dataset, which classifies the disposition of every file artifact Talos sees, storing the verdicts for individual file hashes collected from Cisco product telemetry.

> "Our threat researchers use these data for guardrails that prevent false positive convictions of benign hashes, and our threat hunters use them to look for low prevalence hashes that might be signals for malware campaigns. In both cases, they, and by extension, our customers need fast, accurate answers." — Senay Goitom, Software Engineer, Cisco

But delivering those answers is, as Senay puts it, like searching for a needle in a haystack. In production, the dataset holds nearly 2 trillion rows, with 27 billion new rows ingested daily. On disk, that's around 192 TB of compressed data, or around 2 PB uncompressed. Because these systems power threat intelligence services for Cisco customers, latency and accuracy directly impact the company's security products.

At our [2026 Open House SF user conference](https://clickhouse.com/openhouse/san-francisco), Senay shared how he and the team moved this workload from OSS ClickHouse to [ClickHouse Cloud](https://clickhouse.com/cloud) on AWS and how it helped Cisco dramatically reduce costs with zero downtime.

## Outgrowing the old architecture

In the original design, data originated in a Databricks Delta table stored as Parquet files in S3. A custom, event-driven serverless ingestion pipeline fed a self-managed Kubernetes (EKS) cluster, where a staging table rolled up into an aggregated table behind an API endpoint. Under the old retention policy, the team kept about a petabyte of compressed, replicated data on 32 high-compute nodes, each with seven EBS volumes attached.

![](/uploads/ciscotalos_jun2026_image1_63d917b06b.jpg)

*Cisco Talos's original self-managed architecture: a custom ingestion pipeline feeds an EKS cluster, with data flowing from a staging table to an aggregated table behind an API endpoint.*

"It worked," Senay says, "but we started experiencing some challenges." As a small team inside a threat research organization, they found that running a cluster of that size pulled them away from their actual mission. "We were spending time on upgrades, backups, and troubleshooting," he explains, "instead of building new products for our threat researchers."

That setup also tied the team's storage to compute in a costly way. "Query performance needs meant we had to tie our ever-increasing storage needs to expensive compute nodes," Senay says. "Scaling up or down, depending on traffic, was a costly process and risked service disruption for our consumers. We have to ensure customer trust, so it ultimately meant that we had to over-provision."

He highlights two other challenges beyond the operational and cost burden. The first was SOC 2 compliance. "We had to be constantly vigilant about vulnerability management," Senay says, noting that this added even more overhead to an already lean team. The other was architectural complexity. Custom ingestion, custom aggregation, and custom optimizations created a system that was "hard to maintain and harder to evolve," requiring "additional engineer hours for maintenance and incident response."

## Choosing the right real-time data platform to support threat researchers

In their search for a better solution, the team weighed three options. They quickly ruled out Snowflake because of the storage costs and complexity it would have introduced. As Senay puts it, "Our query latency needs meant that we were going to have to go with a more complicated architecture than we would under ClickHouse."

They also looked at MongoDB, but its document model "wasn't a natural fit for our aggregation workload," Senay says, "and it wasn't cost-effective at the scale that we were operating."

That left one clear option: "ClickHouse Cloud was the natural choice," Senay says. The team already knew the technology, and its [columnar storage model](https://clickhouse.com/docs/faq/general/columnar-database) "was a perfect fit for our hash lookup pattern." [Separation of storage and compute](https://clickhouse.com/docs/guides/separation-storage-compute) promised to dramatically reduce storage costs, and conversations with a ClickHouse solutions architect convinced the team they could land on a far simpler architecture than the one they were maintaining themselves.

## Rebuilding around ClickHouse Cloud

Senay and the team ran the migration in four phases. The first laid the foundation: reusable Terraform modules for the ClickHouse service, [ClickPipes](https://clickhouse.com/cloud/clickpipes), and the database resources, plus SSO/SAML integration and the deployment of both dev and prod services. Next came the data pipeline, standing up the new ingestion flow and backfilling 12 months of historical data using sharded ClickPipes.

With the pipeline in place, the team moved to shadow traffic, routing live production queries to both backends in parallel for initial data validation and performance testing. The fourth and final phase was the cutover, a comprehensive data comparison against the source at scale, followed by stakeholder sign-off and the switch itself.

The result was a vastly simplified architecture. The new design keeps the Databricks Delta table as the source, but replaces the custom ingestion code with managed building blocks. Amazon EventBridge detects new objects as they land in S3 and routes them to SQS queues, and ClickPipes handles ingestion into ClickHouse Cloud. "All of that custom code, all of that orchestration is gone," Senay says.

![](/uploads/ciscotalos_jun2026_image2_8cb4fabb23.jpg)

*Cisco Talos's new architecture on ClickHouse Cloud: ClickPipes handles ingestion with no custom code, feeding a staging table that serves queries via AWS PrivateLink behind an API endpoint.*

Inside ClickHouse Cloud, the team uses the [SharedMergeTree table engine](https://clickhouse.com/docs/cloud/reference/shared-merge-tree), which stores data in object storage and is fully managed by ClickHouse. The Lambda function that routes queries to the backend connects over AWS PrivateLink, so traffic never goes through the public internet. Senay adds that a smart [partitioning](https://clickhouse.com/docs/partitions) strategy helped the team "drastically simplify our back end," letting them aggregate at query time rather than pre-aggregating in the cluster.

## A simpler, more cost performant backend

"From an operational perspective," Senay says, "the results speak for themselves." After a month in production, the new pipeline had ingested roughly 3,000 files a day with zero data loss and less than a minute of queue lag between a file landing in S3 and becoming queryable in ClickHouse Cloud. When the team compared API responses between the old and new backends across more than 1,000 randomly selected hashes, the match rate was 100%. All of this validation happened with zero consumer impact and zero downtime.

"And the cost story is just as dramatic," Senay adds. By breaking the dependency between storage and expensive compute, and by retiring the custom ingestion pipeline and the SOC 2 maintenance overhead that came with their old self-managed setup, the team cut storage costs by roughly 90% and reduced total cost of ownership (TCO) by about 75% across compute, storage, and engineering hours.

> "Moving to ClickHouse Cloud has allowed us to dramatically reduce our TCO by 75% and save over 300 engineer hours a year that can be spent on building new products for our threat intelligence teams." — Senay Goitom, Software Engineer, Cisco

For a small team inside one of the world's largest threat intelligence providers, that shift, from maintaining infrastructure to building on it, ultimately serves the researchers hunting threats, and the Cisco customers who depend on them.

Cisco Talos runs a threat intelligence reputation service on ClickHouse Cloud, classifying file hashes so researchers can make fast, accurate security decisions.

ClickHouse

Blog posts by ClickHouse

How Spyne simplified their CDC pipeline with ClickPipes and ClickHouse Cloud

Beyond the warehouse: How METRO Markets built a do-it-all data platform on ClickHouse Cloud

Why everyone is talking about real-time analytics (a note from “the yellow company”)

Appcues delivers personalized customer engagement with ClickHouse Cloud

92x faster queries: How Open Electricity uses ClickHouse to track Australia’s energy transition in real time

Cisco Talos empowers threat researchers while reducing TCO by 75% with ClickHouse Cloud