Summary

When [Langfuse](https://langfuse.com/) first shared its [journey from Postgres to ClickHouse](https://clickhouse.com/blog/langfuse-and-clickhouse-a-new-data-stack-for-modern-llm-applications) in March 2025, the challenge was clear. Their open-source LLM engineering platform had outgrown a simple, developer-friendly architecture, and the team needed a new data foundation to keep pace with rapidly scaling usage. ClickHouse became that foundation, allowing Langfuse to ingest and analyze orders of magnitude more data without sacrificing performance or flexibility.

By October, at our [Open House roadshow in Amsterdam](https://clickhouse.com/openhouse/amsterdam), the focus had naturally shifted. [ClickHouse Cloud](https://clickhouse.com/cloud) was running in production, and the questions were no longer about database choice. They were about how the shape of LLM applications themselves is changing, and what starts to break when systems that once handled millions of records are suddenly dealing with billions every month. As teams move from relatively simple RAG workflows to complex, agentic systems, observability must evolve along with them. The bottlenecks that matter most are no longer the obvious ones.

What followed was a field report from the frontier, tracing how Langfuse’s data model, ingestion patterns, and even SDK design have had to adapt as LLM applications continue to mature. As Langfuse co-founder and CTO Max Deichmann puts it, “What we’re building is observability for engineering teams working on the most complex and advanced agentic systems.”

## When LLMs got complicated

In the earliest days of production-grade LLM applications, the architecture was relatively straightforward. A user query went into a retrieval system, a handful of documents came back, and those documents were passed to a model to generate a response. These RAG workflows were simple enough to reason about, and simple enough to observe.

But that simplicity didn’t last long. “People realized this is not enough,” Max says. If you want better answers, more reliability, or richer behavior, the system has to do more.

As teams pushed for higher-quality outputs, they moved beyond single-shot prompts toward agentic systems that plan, iterate, and interact with external tools. Now, an agent might call a search API, query a database, invoke an internal service, reassess its plan, and repeat the process several times before producing an answer.

Once applications cross that line, observability becomes a different story. Engineers are no longer simply asking whether a request succeeded. They want to know whether the agent took the right tool call, passed the right parameters, or planned the right next step. And because, as Max puts it, “LLMs don’t have predictable outcomes,” the same input can lead to different execution paths.

Langfuse is built around that reality. “Tracing is at our core,” Max says. The platform started with familiar trace trees, but expanded into graph views that show how an agent moves through its turns and tool calls over time, along with specialized rendering for [large prompts](https://langfuse.com/docs/prompt-management/overview), completions, and business-level metadata.

From there, Langfuse moved beyond individual traces to analytics. Teams want to understand how latency, cost, and quality evolve across thousands or even millions of runs. They want to compare versions of an agent, test changes against production data, and see how behavior shifts over time. That’s why Langfuse added tools like datasets, which let users treat real traces as integration tests and [evaluate different agent versions](https://langfuse.com/docs/evaluation/overview) side by side.

Over the last two years, the product has grown into a full data platform that’s open by design, with [public APIs](https://langfuse.com/docs/api-and-data-platform/features/public-api) and export paths, supporting a growing ecosystem of teams building increasingly complex LLM systems. And all of that activity flows through one place: the database.

## From Postgres to ClickHouse

Langfuse’s previous architecture was intentionally simple: JavaScript and Python SDKs fed events into a single container backed by Postgres. Self-hosting was easy, and the team could move fast without much operational overhead. Even so, it was clear they would outgrow the setup someday. “We always knew this would happen,” Max says.

By June 2024, ingestion volumes had “skyrocketed,” along with IOPS costs. The team was running the largest available instance, and it still wasn’t enough. Analytical queries timed out. The ingestion API got so slow that users complained their SDKs couldn’t send data reliably. As a single-node database, Postgres was also an uptime risk. “We realized we needed to change Postgres,” he says, “or at least move the tracing data somewhere else.”

The team wrote a list of requirements: [columnar storage](https://clickhouse.com/docs/faq/general/columnar-database) for analytical queries, very high-volume [insert throughput](https://clickhouse.com/docs/best-practices/selecting-an-insert-strategy), [multiple write and read nodes](https://clickhouse.com/docs/academic_overview), and [SQL](https://clickhouse.com/docs/sql-reference) so users could slice and dice their data at low latency. Because Langfuse is open-source and widely self-hosted, the database also had to be deployable anywhere under an OSS license. ClickHouse checked all those boxes. 

They built a new pipeline with [SDKs](https://langfuse.com/docs/observability/sdk/overview) and an [OTel endpoint](https://langfuse.com/integrations/native/opentelemetry) feeding events into Redis queues and S3, async workers processing those events, and ClickHouse at the center as the analytical engine. “It worked surprisingly well at very high traffic,” Max says.

![Langfuse Issue 1225.jpg](/uploads/Langfuse_Issue_1225_f20eeb99d8.jpg)

Langfuse’s ingestion pipeline, moving SDK events through queues and storage into ClickHouse.

The turning point came quietly. “One Saturday afternoon, I decided to flip the switch and start reading data from ClickHouse,” Max recalls. The impact was immediate. “It was like minutes on Postgres, and then once I flipped it, latencies dropped right away.” The team wrote an extensive [blog post](https://langfuse.com/blog/2024-12-langfuse-v3-infrastructure-evolution) on that architecture transition.

Over the following year, the system continued to scale as adoption grew. Charts that once showed massive ingestion spikes now barely register. “What was big then is very small now,” Max says. “ClickHouse has served us really, really well.”

With the database no longer holding them back, new questions emerged: “What are our next steps?” Max asks. “Where are the bottlenecks we see today?”

## Lessons learned at scale

As Max shared at Open House, running Langfuse on ClickHouse surfaced a new class of problems that had less to do with raw throughput, and more to do with how data models and access patterns interact with real-world usage.

One early lesson was that performance doesn’t stop at the database boundary. The team learned the importance of passing [primary keys](https://clickhouse.com/docs/best-practices/choosing-a-primary-key), especially timestamps, through frontend state, ensuring that every query hitting ClickHouse is as selective as possible. Small choices in how queries are shaped often matter just as much as how fast the database itself can run.

Another lesson was skepticism. Rather than assuming a single data model would work for everyone, Langfuse now spins up alternative models in separate ClickHouse services and tests them side by side using real production data. With users free to ingest almost anything, cardinality and distribution vary widely. “Test in production,” Max advises. “What works for one user doesn’t work for another one.”

They’ve also learned the value of isolating workloads. “[Compute-compute separation](https://clickhouse.com/blog/introducing-warehouses-compute-compute-separation-in-clickhouse-cloud) is really helpful for us,” Max says. “If one ClickHouse cluster is under fire, the other one is still chill and can serve the UI for our users.” That separation makes performance more predictable and helps protect interactive queries from ingestion spikes.

The hardest lessons, though, came from mutability and joins. Langfuse’s original design allowed SDKs to emit multiple events per span: one capturing the model and input immediately, another arriving later with outputs and cost. That behavior matched how users expect LLM systems to work—calls can take time, and teams want visibility as soon as something happens—but it pushed Langfuse toward mutable tables and frequent deduplication.

[ReplacingMergeTree](https://clickhouse.com/docs/engines/table-engines/mergetree-family/replacingmergetree) can be a “great” solution for this, Max says, but it comes with tradeoffs. “If you want to have very accurate data, you need to deduplicate whenever you read the data.” At the same time, frequent joins across massive trace and span tables added pressure of their own, pushing query complexity and latency higher as usage grew.

All this led Max to step back and ask: “Why do we even have mutable data? Why do we update? Because in traditional observability systems, people don’t update.” Those questions marked a turning point that would reshape how Langfuse models agent behavior and observability at scale.

## One immutable path forward

For Langfuse, the answer lies in how user expectations have changed. While partial updates once mattered, “now people care more about the overall performance of agents rather than single calls,” Max says. That shift makes a new model possible.

Today, Langfuse is moving toward a wide, immutable observations table (called the “Events” table) that collapses trace- and span-level data into one structure. Metadata like user and session IDs are propagated directly into that table, making it much easier to query without joins.

Staying true to Max’s earlier advice, the team didn’t treat this as a theoretical improvement. “We tested this new table in production,” he says. Compared to the old approach, the new model delivered around three times less memory usage and up to 20 times faster queries.

The question now, Max says, is, “How do we get there?” Thousands of deployments still rely on older SDKs that emit multiple events per span. Langfuse can’t force everyone to upgrade. Instead, the team is running background jobs to backfill legacy data into the new schema, while newer SDKs write directly to the immutable table.

It’s an incremental transition, but a deliberate one. “This will help our users get very low latencies across the UI and also our APIs,” Max says.

A year ago, ClickHouse helped Langfuse escape the limits of a single-node database. What followed is more subtle: removing hidden taxes in the data model, aligning ingestion with how users actually reason about agents, and continuously reshaping the system as LLM applications evolve.

As Max made clear in Amsterdam, scaling is less a destination than a moving target. With ClickHouse Cloud, Langfuse has the data foundation to keep adapting as that target moves.


## Langfuse joins ClickHouse

In the time since this story was drafted, [we announced that ClickHouse has acquired Langfuse](https://clickhouse.com/blog/clickhouse-acquires-langfuse-open-source-llm-observability), deepening the integration between Langfuse's LLM observability platform and ClickHouse's analytical capabilities.

The story of scaling observability for agentic systems continues - now under one roof.

Interested in seeing how ClickHouse works on your data? Get started with ClickHouse Cloud in minutes and receive $300 in free credits.

Looking to scale your team’s data operations?

“We’re building observability for engineering teams working on the most complex and advanced agentic systems. ClickHouse has served us really, really well.”  Max Deichmann, Co-Founder and CTO

large_Langfuse Issue 1225.png

medium_Langfuse Issue 1225.png

small_Langfuse Issue 1225.png

thumbnail_Langfuse Issue 1225.png

Langfuse Issue 1225.png

How Langfuse is scaling LLM observability for the agentic era with ClickHouse Cloud

DoControl uses ClickHouse Cloud to power real-time analytics for SaaS security, governing file access, permissions, and third-party apps. After hitting the limits of Postgres, they migrated to ClickHouse Cloud for faster ingestion, complex filtering, and real-time queries at enterprise scale. They recently launched Dot, an AI-powered security assistant, using ClickHouse MCP to translate natural-language queries into real-time analytics workflows.


The rise of SaaS tools has made it easier than ever for teams to collaborate, but it’s also created a new kind of security challenge. Sensitive files can be shared with a click. Third-party apps proliferate without oversight. And while workplace productivity might boom as a result, organizations often have no idea how much risk they’ve inherited.

[DoControl](https://www.docontrol.io/) is all about identifying and addressing those blind spots. Founded in 2020 to bring control back to modern SaaS environments, the platform helps organizations govern access, detect misconfigurations, manage third-party integrations, and automate remediation across tools like Google Workspace, Microsoft 365, Slack, and more.

For many customers, the impact is immediate. “There’s a ‘wow moment’ when they’re finally able to get accurate visibility into their exposure,” says Amit Fidler, Director of Engineering. “They’re in shock when they see both the scale and risk of the exposure across millions of assets and applications—board materials, compensation data, competitive intelligence, former employee access, and more.”

DoControl empowers organizations to uncover, pinpoint, and remediate these risks within seconds, without compromising business productivity. But as adoption grew and the company started onboarding Fortune 500 customers with billions of assets, they ran into a new challenge: scale.

We caught up with Amit and the DoControl team to talk about outgrowing Postgres, migrating to [ClickHouse Cloud](https://clickhouse.com/cloud), and using [ClickHouse MCP](https://clickhouse.com/blog/integrating-clickhouse-mcp) to deliver AI-powered insights at scale.

## Growing pains with Postgres

Originally, DoControl’s analytics engine ran entirely on Postgres. It worked fine for most use cases, but as the company scaled and began ingesting tens of millions of assets from Google Drive and other SaaS platforms, the limitations quickly became apparent.

“Postgres is a pretty good database,” says Bar Ifrah, Technical Lead. “But the problem is, when you’re trying to build tens of indexes or run aggregations, it just can't do it.” 

Even with the indexes they built, the transactional database struggled to keep up with the volume of events and the complexity of the queries. “It’s not just about aggregation,” Bar adds. “Once we started onboarding enterprise-level customers, we couldn’t ingest all their data at the scale and pace we needed. We just didn’t have a good analytics system.”

Postgres remained the system of record for identity and filtering by ID, but it was no longer viable as the foundation for real-time analytics. The team began exploring alternatives, and eventually found their answer in [ClickHouse Cloud](https://clickhouse.com/cloud).

“We were super excited to start working with ClickHouse,” Amit says. “It has been a game-changer in how we serve the world’s largest companies, in a way that no competitive solution can match.”

## A new analytics engine with ClickHouse

For DoControl, the benefits of ClickHouse Cloud were clear right away.

“It’s much faster than Postgres,” Bar says. “When we added ClickHouse to the architecture, we could remove all the indexes, and ingestion worked much faster than before.”

“Now, we’re able to ingest tens of millions of files quickly and perform complex analytics in real time,” Amit adds. “Even querying a single asset across tens of millions of files and billions of permissions happens in real time, which is fantastic.”

Today, ClickHouse powers every analytics workflow in the product. From customer-facing dashboards to internal investigations, queries that once pushed Postgres to its limits now run smoothly. Whether it’s application data, audit logs, or sharing events, the team can query it live and surface insights instantly in the product UI. 

“This is something we couldn’t do with our previous database,” Amit says.

“The filters we’re allowing customers to run are very complex,” Bar adds. “ClickHouse’s ability to process a lot of data in a short period of time is a game-changer.”

DoControl currently operates ClickHouse instances across both AWS and GCP, an architecture that supports internal requirements and customer needs. ClickHouse Cloud’s multi-cloud support gives the team flexibility to run workloads where it makes the most sense.

With instances ranging from 100 GB to several terabytes across both clouds, DoControl has built a real-time analytics layer that’s sturdy and flexible enough to support its growing user base, while laying the foundation for what comes next.

## Agents, AI, and ClickHouse MCP

As the company looked to the future, the team felt convinced of one thing: the next wave of user interaction wouldn’t happen in a traditional UI. It would happen through conversation, with AI agents acting as the new front end.

So they built [Dot](https://www.docontrol.io/lp/dot), an AI-powered security assistant that helps users investigate events and answer questions about their environment. Dot uses [ClickHouse MCP](https://clickhouse.com/blog/integrating-clickhouse-mcp) (Model Context Protocol) to interact directly with ClickHouse, translating natural-language queries into SQL, performing “very interesting and complex aggregations,” and returning results in real time.

“It really opened a whole wide area of use cases and changed how we think about data,” Amit says. “Because once you take all the ClickHouse knowledge in the world, which we get from working with AI, and combine it with our data, you can achieve marvelous things.”

In one example, the team wanted to detect “impossible travel” events—logins from two distant locations within a short window. Rather than build custom logic, they let Dot try to solve the problem on its own. The agent discovered ClickHouse’s built-in [greatCircleDistance function](https://clickhouse.com/docs/sql-reference/functions/geo/coordinates), which calculates the distance between coordinates. It’s a perfect example of how ClickHouse MCP can open the door to powerful new workflows.

The architecture behind Dot is lightweight and scalable. A supervisor agent interfaces with the UI and routes queries to specialized sub-agents—one for events, one for identities. Each agent uses ClickHouse MCP to issue queries and return results.

“Working with a lot of indexes, like in Postgres, isn’t feasible for AI, since you don’t really know what filter or aggregation the user wants to run, and just guessing isn’t enough,” Bar explains. “You need a real-time analytics engine like ClickHouse with very fast filtering capabilities, because the user isn’t going to stay and watch the chat for 10 minutes.”

The team is now experimenting with multi-agent orchestration, cross-database querying, and even exposing the system through chat apps. As Amit says, “We believe that in the future, most user interactions will not be in the application itself, but within those chat applications.”

## What’s next for DoControl and ClickHouse

With AI use cases expanding and customer demand continuing to grow, the DoControl team is hard at work pushing their infrastructure forward. That includes rolling out [compute-storage separation](https://clickhouse.com/docs/guides/separation-storage-compute) in ClickHouse Cloud, giving them more flexibility to isolate workloads and support high-scale ingestion and querying. “It’s one of the best steps we’ve taken,” Amit says.

Looking ahead, they’re focused on enabling new interfaces, scaling to support more enterprise data, and deepening their AI capabilities. As an engineering-led culture, they’re proud to be early adopters of technologies like the [ClickHouse MCP server](https://clickhouse.com/blog/integrating-clickhouse-mcp).

“We’re always looking for ways to use the most advanced technology to achieve our business needs,” says Itai Birenshtok, DoControl’s VP of Research and Development. “That includes being one of the first companies to use ClickHouse MCP in production.”

The team is always exploring what’s next, always finding ways to combine cutting-edge infrastructure with business value. Whether it’s reducing security risk for customers, ingesting billions of events, or building the next generation of AI-powered assistants, DoControl is all-in on modern, real-time analytics. And ClickHouse is at the heart of it.

Want to see ClickHouse MCP in action?

“We were super excited to start working with ClickHouse. It has been a game-changer in how we serve the world’s largest companies, in a way that no competitive solution can match.”  Amit Fidler, Director of Engineering

large_DoControl User Story Issue 1224.png

medium_DoControl User Story Issue 1224.png

small_DoControl User Story Issue 1224.png

thumbnail_DoControl User Story Issue 1224.png

DoControl User Story Issue 1224.png

How DoControl delivers real-time, AI-driven SaaS security insights with ClickHouse Cloud and MCP

Lovable uses ClickHouse Cloud for observability and AI-powered debugging across millions of AI-generated apps and highly dynamic LLM workflows. ClickHouse MCP lets engineers investigate billions of logs with natural-language prompts, reducing debugging time and keeping development velocity high. ClickHouse also powers real-time analytics for millions of deployed apps, delivering sub-50ms queries with a simple pipeline built by one engineer in a week.

As growth stories go, it’s literally impossible to find one more impressive than [Lovable](https://lovable.dev/)’s. In just eight months after launching, the Swedish startup surpassed $100 million in subscription revenue, making it the [fastest-growing software company in history](https://www.forbes.com/sites/iainmartin/2025/07/23/vibe-coding-turned-this-swedish-ai-unicorn-into-the-fastest-growing-software-startup-ever/).

Their mission is simple but profound: to unlock human creativity by enabling anyone to create software. They do that with an AI-powered platform that lets users build full-stack apps from scratch using nothing more than natural-language prompts.

Today, more than 8 million people use the platform. Together they create over 100,000 projects and ship around 25,000 apps into production every day. The company recently crossed 3 million deployed apps, and that number is climbing fast. As technical staff member Tomas Nordström puts it, “This means we definitely need better observability.”

At our [Open House Roadshow in Amsterdam](https://clickhouse.com/openhouse/amsterdam), Tomas shared how Lovable uses [ClickHouse Cloud](https://clickhouse.com/cloud) to keep up with a system that evolves by the minute, including how engineers debug unpredictable AI behavior with [ClickHouse MCP](https://clickhouse.com/blog/integrating-clickhouse-mcp). He also walked through how Lovable powers web analytics for all of its deployed apps, and where the team is expanding next.

## Use case #1: Observability at scale

Lovable runs a deeply distributed, AI-powered system built from multiple distributed services that power millions of published apps. Every new project triggers a chain of internal API calls, external integrations, and LLM-driven actions. The result is a “very non-deterministic system,” Tomas says, in which understanding system behavior is uniquely challenging. 

“We have a lot of APIs that we’re exposing internally, and then we’re also communicating with a lot of APIs externally as well,” he adds. “It’s quite a stochastic system.”

To make sense of all that activity, the team built a full OpenTelemetry pipeline that streams directly into ClickHouse. Logs and traces from the Go API backend, Cloudflare Workers, their sandbox environment, Temporal workflows, and Kubernetes clusters all land in one unified ClickHouse schema. From there, the team visualizes everything in Grafana, creating a real-time view of how traffic moves across every endpoint—“both the ones we’re exposing,” Tomas says, “and also all of the endpoints we’re communicating with externally.”

ClickHouse gives them the high-volume write throughput they need, plus fast queries across billions of events. And because ClickHouse’s schema model is flexible, the team can evolve their observability pipelines as new services, new models, and new workflows come online, without slowing down development.

## Use case #2: AI-powered debugging

But even with full visibility into their microservices, Lovable still needed a way to understand what happens *between* those services—inside the LLM-driven logic that can change from one request to the next. When an outgoing request fails or behaves strangely, the root cause often lives in a chain of model-driven tool calls that didn’t exist a few minutes earlier. As Tomas puts it, “People are using Lovable in very weird ways that we weren’t expecting.” Those edge cases can be incredibly difficult to trace through billions of log entries.

This is where the [ClickHouse MCP server](https://clickhouse.com/blog/integrating-clickhouse-mcp) has been a “game-changer,” Tomas says. With 20-25 engineers supporting more than 8 million users, Lovable can’t rely on everyone being a SQL expert capable of querying hundreds of billions of log rows during an incident. MCP removes that barrier. Engineers simply describe the issue, and MCP fetches the right traces, recent Git commits, and the current log schema, with no SQL required. 

“MCP has really been the best help we could ask for,” Tomas says. “It’s one of the main use cases where we’ve been sticking to ClickHouse and started falling in love with it.”

## Use case #3: Analytics for 3 million apps

As more people began deploying apps with Lovable, another need emerged: giving users visibility into how their apps were performing in the real world. With millions of deployed apps and usage patterns ranging “from one visit to several million visits per day,” the team needed an analytics layer that could handle extreme multi-tenancy at scale.

After a quick conversation with the ClickHouse team, the solution was clear. “We understood, this is actually a great use case for just building this with ClickHouse,” Tomas says.

Today, every time an app renders an HTML page, a lightweight event is sent via Cloudflare Workers straight into ClickHouse. A small service built around four tables and four [materialized views](https://clickhouse.com/docs/materialized-views) handles the rest, powering real-time dashboards that track visitors, page views, sessions, bounce rates, referrers, devices, and more.

Despite the diversity in traffic, the system is fast and effortless to operate, with queries returning in 50 or so milliseconds. Just as impressive, the pipeline was “created by one engineer in one week,” Tomas says. “It was so easy and really a no-brainer to set up.”

## Use case #4: More ClickHouse applications

Lovable has continued to expand its ClickHouse usage with new features that benefit from its speed, reliability, and low operational overhead. “There’s a lot of use cases where we can continue using ClickHouse,” Tomas says.

A recent example is [Lovable Cloud](https://lovable.dev/cloud), the company’s native backend hosting environment. It came with a new AI gateway that tracks every LLM call—models, tokens, latency, cost—to support usage-based billing. Since the team already knew how to structure, materialize, and aggregate data efficiently in ClickHouse, Tomas says integrating it with the gateway was a natural fit.

Security scanning is another new application, driven by the rise of “vibe coding” and security concerns around the apps it produces. “We’ve taken a responsibility to make sure we try our absolute best to make these apps as secure as possible,” he says. Lovable now scans all deployed apps daily for leaked secrets, permission issues, and other vulnerabilities. Any findings are logged into ClickHouse, materialized, and re-checked every hour via [refreshable materialized views](https://clickhouse.com/docs/materialized-view/refreshable-materialized-view) to determine whether users should be alerted.

## Why ClickHouse matters for Lovable

Lovable’s vision of democratizing software comes with a second goal: democratizing the infrastructure behind it. As Tomas puts it, “We want to be able to move really, really fast—and we want to allow our users to move really, really fast as well.”

Part of that, he says, is exposing ClickHouse more and more to Lovable’s users. “You have a lot of options when using ClickHouse, but you can also take, in a sense, a lot of shortcuts as well. You don’t have to be a database expert.” With the right abstractions, both humans and AI can explore data, ask questions, and understand what’s happening without needing to understand every detail of the underlying schema.

That flexibility matters inside the company, too. With more than 200 commits landing in production each day, engineers need quick answers when something looks off. ClickHouse gives them that speed. Instead of digging through opaque logging pipelines, anyone on the team can jump into ClickHouse and instantly see how a change is behaving in the wild, without having to be a SQL expert. It shortens feedback loops, reduces operational drag, and keeps development velocity high.

Tomas closed his talk with a few words from Lovable itself: “At Lovable, we're making it possible for anyone to build software. ClickHouse makes it possible for anyone—human or AI—to understand and analyze that software at scale.”

Ready to scale your team’s data operations?

“At Lovable, we're making it possible for anyone to build software. ClickHouse makes it possible for anyone—human or AI—to understand and analyze that software at scale.”  Tomas Nordström, Member of Technical Staff

large_Lovable User Story Issue 1222.png

medium_Lovable User Story Issue 1222.png

small_Lovable User Story Issue 1222.png

thumbnail_Lovable User Story Issue 1222.png

Lovable User Story Issue 1222.png

Lovable <3 ClickHouse: AI-powered observability and analytics at the world’s fastest-growing software company

Chartmetric uses ClickHouse Cloud to deliver real-time analytics on billions of rows of music, playlist, and engagement data. They migrated from Postgres and Snowflake to improve speed and reduce storage; queries now run 10-15x faster at much lower cost. Today, ClickHouse powers a 5.5B-row playlist cache that ingests 15M+ new rows daily, plus LLM-facing queries using batched WHERE + IN filters and projections. 

[Chartmetric](https://chartmetric.com/) tracks the pulse of the music industry. Every day, the platform ingests millions of rows of data from streaming services, social media, and music charts, powering dashboards for A\&R teams, reports for record labels, and insights for artists navigating their next move. 

By 2024, that data footprint had ballooned into the billions and their systems weren’t scaling fast enough to keep up. [This past March](https://clickhouse.com/blog/chartmetric-uses-clickhouse-to-turn-artist-data-into-music-intelligence), lead engineer Peter Gomez shared how the team tackled the challenge by migrating time-series workloads from Postgres and Snowflake to [ClickHouse Cloud](https://clickhouse.com/cloud). The move sped up slow queries and cut RDS storage by 10 TB.

Since then, the team has gone even further. ClickHouse now powers production workloads across the company, from LLM-ready queries to a massive playlist cache pipeline that refreshes every five minutes. What began as a targeted performance fix has evolved into a foundational part of Chartmetric’s real-time data stack.

At a [September 2025 meetup in Toronto](https://clickhouse.com/videos/toronto-meetupSep1), lead data engineer Umang Sharaf shared what that evolution looks like today. He walked through the team’s architecture, new query patterns, and the tradeoffs of scaling a playlist cache that ingests over 15 million new rows a day.

## Chartmetric’s journey to ClickHouse

Today, Chartmetric handles around 300,000 requests per hour—roughly 80 per second—across a web app, external API, and small public API. The platform tracks more than 12 million artists, 140 million tracks, 45 million albums, and nearly 30 million playlists.

“These numbers are growing rapidly,” Umang says. “The number of artists grew by 13% in just a year as new artists join Spotify, Apple Music, Amazon, and other platforms.”

Before 2024, the team relied on a mix of Postgres, Snowflake, Elasticsearch, and Airflow. Postgres handled core application data and business logic, Snowflake supported analytics, Elasticsearch powered search and filtering, and Airflow orchestrated data pipelines. But none of these systems were built for the scale and flexibility Chartmetric needed.

“Storage was costly, with a lot of infrequent and cold-access data,” Umang says, noting their AWS Aurora RDS footprint included 28 TB of historical data. Snowflake, which ran 700+ cache and transform tables daily, was “very expensive and obviously not suitable for our API latency needs.” Elasticsearch, meanwhile, was fast and responsive, but “only for specific use cases—and writing JSON isn’t as easy as writing SQL,” he adds.

Chartmetric’s ClickHouse journey began with a small EC2 instance. “But as we started using it more,” Umang says, “our usage and our infrastructure needs increased.” In summer 2024, they migrated to [ClickHouse Cloud](https://clickhouse.com/cloud). Today, they use multiple warehouses tailored to specific workloads: “one for the application and the API (read-only), one for data science and analytics, and one for data engineering tasks like sync and ETL,” Umang says.

## The importance of projections

Back in March, Peter described how migrating time-series data from Postgres and Snowflake to ClickHouse led to a 10 TB storage reduction and major improvements in query speed. At the Toronto meetup, Umang dove deeper into the challenges and optimizations that followed.

As Umang explains, the issue they faced wasn’t ingest performance. “ClickHouse is very efficient for ingesting new data,” he says. “That part went off without a hitch.”

The real challenge was sync. The tables had been ordered for API access, not for update patterns. In Postgres, B-tree indexes handled it fine. But in ClickHouse, hourly sync jobs began running into timeouts and memory issues.

The solution was [projections](https://clickhouse.com/docs/sql-reference/statements/alter/projection), which Umang describes as “not quite clustering, not quite indexing… they’re essentially mini-tables that are maintained directly by ClickHouse—a good hands-off implementation, so that instead of creating, maintaining, and referencing multiple views from API code, it all just lives within the base table itself.” 

One projection surfaces the most recently modified rows. Another is sorted by ISRC, the music industry’s equivalent of a book’s ISBN. These allow precise queries without full scans. The team often uses multiple projections per table depending on workload. And while Umang notes that projections can complicate deletes, the tradeoff is worth it. “This provided huge flexibility compared to Snowflake’s single clustering key,” he says.

With this approach, migrating just eight time-series tables cut their RDS footprint from 28 TB to 18 TB—a 36% savings. And even queries scanning back to 2016, which frequently timed out after a minute in Postgres and Snowflake, now run in under three seconds.

## Rethinking JOINs with WHERE + IN

As Chartmetric expanded its use of ClickHouse beyond time-series workloads, they began creating [flattened, denormalized tables](https://clickhouse.com/docs/data-modeling/denormalization) to support an LLM-based interface for querying artist and track data. But joining large datasets—track metadata, metrics, playlist history—was easier said than done. “We frequently ran into out-of-memory issues,” Umang says.

Earlier this year, Peter shared a few initial lessons about limiting JOIN complexity: pre-join wherever possible, avoid joining more than one large table per query, and use projections to speed up scans. Still, the team saw room to improve.

The breakthrough came when they shifted to using [WHERE](https://clickhouse.com/docs/sql-reference/statements/select/where) and [IN](https://clickhouse.com/docs/sql-reference/operators/in) filters instead of joins. Rather than joining directly on artist and track tables, they’d first extract a list of relevant track IDs, then use that list to filter the main table. To reduce memory load, they split queries into smaller batches using [LIMIT](https://clickhouse.com/docs/sql-reference/statements/select/limit) and [OFFSET](https://clickhouse.com/docs/sql-reference/statements/select/offset).

The impact was huge. A query pulling track metadata for Taylor Swift’s songs initially scanned 17,000+ granules and attempted to use nearly 30 GB of memory, resulting in an error. After switching to WHERE \+ IN, the same query scanned fewer than 400 granules, ran in 98 milliseconds, and used just 300 MB of memory.

The team took it further by layering batching on top of those filters. For a query pulling all playlist history for Swift’s tracks since 2016, the raw join took 107 seconds and used 5 GB of memory. After batching, each subquery ran in under 0.5 seconds using just 44 MB of memory. 

“It took 50% less time and over 99% less memory,” Umang says. “In some cases, it was slower than a regular join, but these are overnight queries, so it wasn’t an issue. Reducing memory usage was an important goal for us, and this really helped a lot.”

## Perfecting the playlist cache

The playlist cache is one of Chartmetric’s most important and complex tables. It tracks which artists appear across nearly 30,000 playlists, how often, in what position, and across what types (e.g. editorial, algorithmic, branded). It supports filters by artist, track, album, or playlist ID, and stores data going back nearly a decade. Users rely on it to see near real-time changes, especially on Fridays, when platforms like Spotify refresh top playlists.

At the time of Peter’s talk in March, the table held around 2 billion rows, with more than a million rows added daily. The pipeline ran hourly, with deletion markers generated in Snowflake and synced into ClickHouse. To support this, the team used the [VersionedCollapsingMergeTree engine](https://clickhouse.com/docs/engines/table-engines/mergetree-family/versionedcollapsingmergetree), which let them “invalidate” old rows using a row\_active flag and insert updated versions, without relying on [FINAL](https://clickhouse.com/docs/sql-reference/statements/select/from#final-modifier) queries or background deduplication.

Since then, the system has scaled significantly, and the team has continued to refine it. In July, they moved the entire ingestion pipeline over to ClickHouse. “We’re now able to run it every five minutes, which has scaled very well for us,” Umang says. “In fact, we run it every five minutes, but it frequently takes less than two.”

Today, the playlist cache table holds 5.5 billion rows and ingests 15 million new records daily. It includes 31 columns covering everything from playlist type to position history, yet the entire compressed table size is just 340 GB. “Compare that to Postgres,” Umang says, “where the raw data with only seven columns took up half a terabyte.”

Query performance has also seen a major leap. Whereas in Postgres queries often timed out around the 60-second mark, and Snowflake took around 20 seconds for top-artist queries, those same queries now return in about 1.5 seconds in ClickHouse, even with complex filters. “That’s been great for our application and the API,” Umang says.

## Fast, flexible, and stackable

As Umang puts it, “ClickHouse works very well as part of a multi-system data stack.” Postgres remains essential for maintaining consistency and running complex relational queries, and Snowflake continues to support black-box analytics—but ClickHouse has proven to be “excellent for time series data,” at a much lower cost. 

He calls the VersionedCollapsingMergeTree engine a “game changer” and adds, “A speed-up from 20 seconds to 1.5 seconds was very significant.” While they initially struggled with large JOINs, they overcame those challenges using batching and projections. “ClickHouse was very flexible for us,” he says, adding a shoutout to the ClickHouse Cloud support team. 

For Chartmetric, the result is a system that’s fast, scalable, and perfectly suited to real-time analytics, while integrating cleanly alongside other tools in the stack.

Does your team’s data stack need a boost? Get started with ClickHouse Cloud in minutes and receive $300 in free credits.

Get started today

“ClickHouse works very well as part of our multi-system data stack. It’s excellent for time-series data, and the VersionedCollapsingMergeTree engine was a game-changer for us, speeding up queries from 20 seconds in Snowflake to 1.5 seconds in ClickHouse.”

large_Chartmetric User Story Issue 1213.png

medium_Chartmetric User Story Issue 1213.png

small_Chartmetric User Story Issue 1213.png

thumbnail_Chartmetric User Story Issue 1213.png

Chartmetric User Story Issue 1213.png

Behind the music: How Chartmetric is scaling music analytics with ClickHouse

お客様のデータで ClickHouse がどのように機能するか、ご確認になりませんか。ClickHouse Cloud は数分で利用を開始でき、300ドル分の無料クレジットをご提供いたします。

今すぐ始める

「ClickHouse MCPを使用することで、高度に複雑な分析や、データの監視、きめ細かなコントロールが可能になります。」- ソリューションアーキテクト、孫 セイ氏

large_User Story EMQ Issue 1209.png

medium_User Story EMQ Issue 1209.png

small_User Story EMQ Issue 1209.png

thumbnail_User Story EMQ Issue 1209.png

User Story EMQ Issue 1209.png

EMQとClickHouseの連携: 産業用エッジで機能する、AIを活用したリアルタイム分析

When you hear “industrial IoT,” what comes to mind? Probably sensors ticking away in factories, telemetry from smart vehicles, automated alerts from machines on the line. But what often goes unnoticed are the systems that make it all work—the message brokers, data pipelines, and analytics engines powering real-time intelligence behind the scenes.

For over a decade, [EMQ](https://www.emqx.com/en) has been at the forefront of that infrastructure. Its journey began in 2013 with the launch of [EMQX](https://www.emqx.com/en/platform), an open-source MQTT broker designed to connect devices in low-bandwidth, high-concurrency environments. It quickly became the world’s most scalable MQTT platform, trusted across IoT, IIoT, and connected vehicle applications alike.

Since then, EMQ has grown into a global leader, serving more than 1,000 enterprise customers and connecting over 250 million devices. Its technology powers everything from factory equipment and energy grids to autonomous vehicles and robotic systems, with customers including Geely, HPE, VMware, Verifone, SAIC Volkswagen, Lucid, and Ericsson.

As EMQ’s scale and reach have grown, so have the demands on its platform. Customers aren’t just routing messages between devices; they need to understand what those messages mean. They want real-time insights, flexible analysis, and the ability to act on data the moment it arrives. That shift, from connectivity to intelligence, called for something new.

Earlier this year, solution architect Cheng Sun joined us at a pair of ClickHouse meetups in Tokyo—in [April](https://clickhouse.com/videos/tokyo-meetup-EMQ-15apr25) and again in [June](https://clickhouse.com/videos/tokyo-meetup-emq-29jul25)—to share [how EMQ integrates with ClickHouse](https://clickhouse.com/docs/integrations/emqx) to deliver fast, flexible, end-to-end visibility from the edge to the cloud. From the messaging backbone to the AI-powered observability layer, he showed what it looks like when messaging and analytics are no longer separate systems, but part of the same real-time platform.

## Making MQTT work at scale

EMQ’s flagship product is an MQTT broker called [EMQX](https://www.emqx.com/en). “Whereas Kafka focuses more on data transaction capabilities, our focus is on parallelism and latency,” Cheng explains. “Our challenge is to deliver data from the edge to cloud services with minimal latency.”

[MQTT](https://en.wikipedia.org/wiki/MQTT) is a lightweight protocol originally created at IBM in 1999 and made publicly available in 2010. Unlike HTTP, which uses point-to-point communication, MQTT introduces a broker between publishers and subscribers. “The broker’s role is to route messages correctly, as well as manage the state of each message and connection,” Cheng says.

Developers can also fine-tune message delivery through MQTT’s Quality of Service settings. “You can use level 0 for lightweight, less important data—sending it just once and that’s it,” Cheng explains. “For critical data, you can use level 1 to ensure delivery at least once. And depending on the requirements, you can use level 2 to make sure data is sent exactly once.” 

That flexibility, combined with its ability to support high-concurrency environments, is a big reason MQTT has become the protocol of choice for IoT and industrial workloads. 

But running a large-scale MQTT cluster comes with challenges, particularly around state management. Every connection carries metadata: message headers, footers, connection status between publishers and subscribers. “There’s a lot of data that needs to be monitored,” Cheng says. “That’s why synchronizing data between servers in a cluster is a complex challenge.”

EMQX solves this with a cloud architecture that includes two node types: core nodes that manage session state, and replica nodes that scale out processing. “This allows you to build a cloud environment that can scale infinitely,” Cheng says.

In 2024, the team deployed a cluster capable of handling 100 million concurrent connections. “We achieved extremely high throughput on this cloud, reaching 2 million messages per second,” Cheng says. “At that time, we were running up to 23 nodes. More recently, we’ve been able to build a large-scale IoT cloud with up to 75 nodes.”

## Real-time analytics with ClickHouse

To turn raw device data into real-time insights, EMQ needed a setup that could support high ingestion rates, low-latency queries, and broad compatibility with both industrial protocols and modern analytics tools. For that, they turned to [ClickHouse Cloud](https://clickhouse.com/cloud).

![User Story EMQ Issue 1209 (2).png](/uploads/User_Story_EMQ_Issue_1209_2_8bbcc13f2d.png)

EMQ’s ClickHouse-based analytics setup connects edge devices to cloud dashboards.

ClickHouse sits at the heart of EMQ’s analytics stack, powering everything from ingestion to visualization. It pulls in data from a variety of sources, including relational databases like Postgres and MySQL, streaming systems like Kafka, and industrial brokers like EMQX.

In a typical data flow, industrial devices send telemetry through southbound protocols to [NeuronEX](https://www.emqx.com/en/products/neuronex), EMQ’s edge gateway for data normalization. From there, the data travels via MQTT to EMQX, then through Kafka and finally into ClickHouse Cloud. “With this kind of architecture, you can connect data from the edge all the way to the database,” Cheng says, noting that the setup delivers sub-second analytics, even across public networks.

![User Story EMQ Issue 1209 (1).png](/uploads/User_Story_EMQ_Issue_1209_1_d6d2b3001b.png)

End-to-end data pipeline from edge devices to ClickHouse-powered analytics.

Once inside ClickHouse, the data is immediately available for fast querying. Features like [dictionaries](https://clickhouse.com/docs/sql-reference/dictionaries), [projections](https://clickhouse.com/docs/sql-reference/statements/alter/projection), and [materialized views](https://clickhouse.com/docs/materialized-views) accelerate access and reduce resource usage. Engineers can explore the data using tools like Grafana, Power BI, or Tableau, or work with it directly in Python, Java, Rust, or Go. ClickHouse also integrates with external data lakes like [S3](https://clickhouse.com/docs/integrations/s3) and [Delta Lake](https://clickhouse.com/docs/integrations/deltalake), making it easy to combine real-time streams with historical data.

At the April meetup, Cheng shared a live demo using simulated factory data generated in Python. OPC-UA devices produced sensor readings like temperature and humidity, while Modbus devices sent things like production counts and error codes. The data was normalized using NeuronEX, then streamed via MQTT to EMQX. From there, EMQX’s sink feature routed it into ClickHouse using SQL-configured pipelines.

Once ingested, the data was immediately ready for querying and visualization. Cheng highlighted ClickHouse’s [AI-assisted query builder](https://clickhouse.com/docs/use-cases/AI/ai-powered-sql-generation), which translates natural-language prompts into SQL. “I find the AI features offered by ClickHouse to be very user-friendly,” he says. “The AI can analyze in a single sentence and instantly generate a SQL statement, allowing you to check the data in real time.” 

He also showed off ClickHouse Cloud’s built-in [dashboards](https://clickhouse.com/docs/cloud/manage/dashboards), setting the refresh interval to 10 seconds to display the latest data. “You can instantly see production numbers, defect counts, and more—all in real time,” he says.

## AI at the edge with ClickHouse MCP

<iframe width="768" height="432" src="https://www.youtube.com/embed/eY_kVWsOlOk?si=DVPWbPG224PuK88s" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" referrerpolicy="strict-origin-when-cross-origin" allowfullscreen></iframe>

At the July meetup, Cheng shared how EMQ has continued to expand its use of ClickHouse’s AI features to build real-time, AI-powered analytics tools for industrial IoT environments. To do that, they first needed to tackle three core challenges facing manufacturing clients.

The first is the complexity of managing diverse edge devices. “With so many different types of equipment and production sites,” Cheng says, “management becomes complicated.”

The second is tracking message latency. “We can’t track the delay of each individual message,” he explains. “We can only monitor average delays or delays for some messages.”

And the third is juggling multiple databases on the northbound (AKA cloud-facing) side of the architecture). “We need a time-series database for each message, a relational database to store detailed equipment information, and a vector database to record internal knowledge.”

The team’s goal, as Cheng puts it, was to “use ClickHouse as a unified data platform and combine the functions of all three databases into one.”

That’s where they turned to the [ClickHouse MCP server](https://clickhouse.com/blog/integrating-clickhouse-mcp). MCP (Model Context Protocol) is a new open protocol designed to let AI agents interact with third-party systems (e.g. databases, APIs, tools) via natural language. Each system is wrapped in an MCP server, enabling developers to trace, debug, query, and analyze events with AI-enhanced workflows. In EMQ’s case, that means giving operators the ability to ask questions like, “Why did the machine fail?” or “Where is the delay happening?” and get answers based on live data from ClickHouse.

During the demo, Cheng showed how MCP enables real-time observability, advanced analytics, and AI-powered troubleshooting. Using trace IDs and vector search, operators can instantly pinpoint the source of a machine error, retrieve the associated error code, and look up relevant solutions, all within the same system. “By using MCP, you can perform highly complex analytics, monitor data, and exercise fine-grained control,” Cheng says. “With this, you can detect changes in latency and errors in real time.”

He also demonstrated new developer tools that simulate sensor data, generate schema recommendations, and automatically configure EMQX rule engines using LLMs. “I’m not very good at SQL,” Cheng admits, “so having features like this is really helpful.” 

The system can stream and process data at scale, with even lightweight edge hardware feeding into cloud dashboards for cross-cluster visibility. And to support more constrained environments, the team is looking at ways to integrate MCP with MQTT. “Since it’s difficult to install MCP on various lightweight devices, we’re exploring new approaches,” Cheng says. “By installing the MCP server proxy on the broker, you can add AI capabilities to even traditional, standard devices. All devices can be enhanced with AI features.”

## Powering the factories of the future

EMQ’s integration of ClickHouse—from real-time pipelines to AI-powered observability with ClickHouse MCP—shows just how far industrial IoT infrastructure has come. Instead of stitching together separate systems for messaging, storage, and analytics, companies can now build a unified stack that’s fast, flexible, and AI-ready from day one.

By making it easier to query, monitor, and act on data, EMQ is helping customers run smarter, more responsive operations. And with seamless integration between EMQX and ClickHouse, plus AI features like MCP, ClickHouse is quickly becoming the platform of choice for the intelligent, data-driven factories of the future.

Learn more about [how EMQX integrates with ClickHouse](https://clickhouse.com/docs/integrations/emqx), and try both [EMQX Cloud](https://www.emqx.com/en/try) and [ClickHouse Cloud](https://clickhouse.com/cloud) for free.


“By using ClickHouse MCP, you can perform highly complex analytics, monitor data, and exercise fine-grained control.” - Cheng Sun, Solution Architect

The ClickHouse Team

Blog posts by The ClickHouse Team

How Langfuse is scaling LLM observability for the agentic era with ClickHouse Cloud

How DoControl delivers real-time, AI-driven SaaS security insights with ClickHouse Cloud and MCP

Lovable <3 ClickHouse: AI-powered observability and analytics at the world’s fastest-growing software company

Behind the music: How Chartmetric is scaling music analytics with ClickHouse

EMQとClickHouseの連携: 産業用エッジで機能する、AIを活用したリアルタイム分析

EMQ and ClickHouse: Real-time, AI-assisted analytics at the industrial edge