We used to rebuild thousands of tables on every pipeline run. Now we serve 15,000+ views over Iceberg on StarRocks and a data refresh rebuilds nothing. This post is adapted from our Next Query 2026 talk. All numbers are rounded and as of November 3, 2026.
At Carbon Arc, more than 500 data assets are mapped to one proprietary ontology, cleaned, modeled and served as aggregated views. Clients can reach them through an API and web platform, a Python SDK, bulk S3 delivery and an MCP server that AI agents call directly.
Until early 2026, serving meant rebuilding approximately 11,000 aggregated tables (and growing) on every pipeline run and answering queries from one shared Trino cluster. We replaced that with StarRocks views over Apache Iceberg tables on S3. In October 2026 our production StarRocks cluster answered 1.1 million queries. The median query returned in 0.2 seconds. Each data refresh now publishes new Iceberg snapshots atomically. The views over them update in place.
The challenge
Every dataset flows through the same stages. Spark on EMR cleans it, maps every record to the ontology and writes certified Iceberg tables. From those, we serve aggregated views at many grains: a brand by state by day, a company nationally by week, a ZIP code by day. Each grain covers a set of filter combinations for a given dataset that the API, the web platform and the MCP server can address by name.
The catalog is large and keeps growing. We serve 15,000+ generated views on the production cluster. Behind them sit petabytes of aggregated data. Two or three new datasets arrive each month and each one adds hundreds of views.
The Trino era: rebuild everything, every run
From 2023 to 2025, every one of those datasets was a physical table. Every pipeline run dropped and rebuilt each table on Trino from scratch. Spark built the heaviest ones. By the end we had over 11,000 denormalized Iceberg tables, and growing.
The queries against those tables were mostly simple. The platform API and its 100+ endpoints looked up one table at a time, filtered by entity and date. Our internal tools read dataset metadata, such as date ranges and coverage, through the same Trino cluster. The first versions of our MCP server sent agent questions there too. Bulk aggregation jobs built large tables on it. Our Trino client allowed a query to run for up to a day and retried failures three times with backoff, which tells you how long some of them took.
All of it ran on open-source Trino on EKS, deployed with Helm, as three clusters with one coordinator each. The serving cluster autoscaled between 3 and 50 workers of 8 vCPU and 64 GB, about 400 vCPU at its ceiling. The dev cluster scaled to 60 workers of 16 vCPU and 128 GB. A separate pipeline cluster scaled to 40 workers of 32 vCPU and 256 GB. Each worker took a whole node, scaled on CPU and spooled task exchange to S3 so failed tasks could retry. We ran the production cluster inside our dev EKS cluster to sit next to the Hive Metastore. The three clusters ran three different Trino versions.
That design broke down as the catalog grew:
- Building the denormalized tables took about 24 hours. For our largest card-spend dataset that was one day of a three-day refresh.
- Every new dataset added tables to rebuild and maintain, including ones nobody queried, so cost tracked the table count.
- By December 2025 the Trino cluster was under memory pressure. Research and MCP queries on the dev cluster were timing out.
- Open-source Trino has no warehouse isolation and we had configured no resource groups, so every client shared one queue. A heavy aggregation competed with API lookups. Every new user added to the bottleneck.
In August 2025 we had started planning a pipeline to run aggregations directly — query the certified Iceberg tables, apply views as filters over them, and delete the approximately 11,000 tables along with their ETL — but we still needed to fully scope it out. We estimated that dropping the rebuild layer alone would save about $20,000 a month.
First attempt: turn every table into a materialized view
While that plan was still taking shape, the obvious first move was to keep the table shape and change only the engine. In the Trino era every grain was its own physical table: card spend by brand by state by day was one table, by brand nationally by week was another, each a denormalized copy of the same certified data. StarRocks can maintain copies like that itself, as asynchronous materialized views, so that is what we built. In August 2025 we built the first StarRocks model as a materialized view. Over the next ten weeks we built every dataset the same way. By October 2025 we had thousands of them, each with a manual refresh.
The frontend (FE), which plans queries and holds the catalog, ran out of heap. Query memory on the compute nodes held up.
Two things in FE memory grow with partitions:
- Each materialized view records which snapshot of each base-table partition it last read, so it can refresh incrementally. A view over a table partitioned by day with five years of history tracks about 1,800 partitions. Every materialized view keeps its own copy of that map.
- The Iceberg metadata cache holds each base table’s partition list, which the FE reads to decide what to refresh. Every refresh check walks those partitions again.
The denormalized design multiplied both. Thousands of materialized views sat over the same few certified tables, so the same daily partitions were tracked thousands of times, once per view. FE memory grew with views times partitions and both kept rising: each new dataset added hundreds of views and each day added a partition to every base table. Refreshes slowed with it. Materializing every table of our largest card-spend dataset took six to eight hours after each certified build. Transaction errors piled up, so we wrote a script to convert failing models. QA waited for the cluster to free up.
Replacing copies with plain views
We kept one set of certified Iceberg tables per topic and expressed every grain as a plain view over them: a saved query that filters and aggregates when someone asks for it. A view holds no data and no refresh state. To the FE it is a few kilobytes of SQL text. FE metadata now scales with the certified tables and their partitions, which grow with the data alone.
In mid-October 2025 we concluded that materializing every grain was the wrong pattern for StarRocks. The FE could not hold metadata for thousands of materialized views without growing every month. We decided to serve the denormalized grains as plain views over the certified tables and made views the default for every new dataset. With views, the same card-spend data was live the moment its certified tables committed, six to eight hours sooner than before. That decision set the design this post describes.
We never went back to materializing everything. Partitions still cost FE memory after the switch, as the incident table below shows, but the count now grows only with base tables. In October 2026 we decided to return to materialized views for 28 hot query families, each one chosen because its measured query cost justified the refresh time and FE state it adds. StarRocks 4.0.16 had fixed a jemalloc issue where freed memory was not returned quickly enough, which had pushed nodes into the operating system’s out-of-memory killer. We also gave refreshes their own warehouse and made them all-or-nothing. That story is in Precomputing the Hard Queries.
What we needed and how we tested it
We wrote four requirements before committing to an engine:
- No vendor lock-in and no rebuild layer. Data stays in open Iceberg tables on S3 and nothing sits between the lake and the customer that must be regenerated.
- Fast on plain views. Interactive latency on views that aggregate at query time, without pre-building every table.
- Isolated workloads. Separate compute for API, agent and ad hoc traffic, so one cannot slow another.
- A bill that does not grow linearly with data.
We tested StarRocks on our own production workload: real platform queries against plain views over Iceberg at full scale. We drove API, MCP and ad hoc load at the same time on separate warehouses. In January 2026 our analytics team ran a full row-count and metric parity test between Trino and StarRocks. At cutover we compared every table against its replacement view across the stack. Those comparisons caught real issues before customers did. When we later moved production to a new StarRocks cluster, we copied the 15,000+ view definitions and re-created them in parallel because none depended on another view.
We run StarRocks on CelerData’s bring-your-own-cloud service in our AWS account, on Graviton instances, with an open-source StarRocks cluster on EKS for internal batch jobs.
We left Trino gradually so we could compare every table against its replacement view at each cutover. We moved the platform API in October 2025. In December 2025 we cut the dev Trino cluster to a fixed 10 workers and moved dev to StarRocks to reduce Trino spend, which halved it. We moved production serving in January 2026: the MCP server on January 13 and the bulk aggregator on January 20, keeping a Trino fallback while it settled. We moved pipeline reads over the next seven months and converted the last Trino SQL in the pipeline to Spark in August 2026.
Running StarRocks in production: what broke
The first months on StarRocks were not smooth. Most incidents traced back to memory limits that behaved differently from Trino’s, or to the FE doing more work than we expected:
| Date | What happened | Root cause | What we changed |
|---|---|---|---|
| Jan 2026 | Production outage: a compute node exceeded its process memory limit | Too many concurrent heavy queries; spill did not engage | Capped concurrency with a query queue |
| Feb 2026 | Production outage: the FE marked every compute node unavailable | Nodes unresponsive under memory pressure, during a flood of Iceberg metadata refreshes from expired snapshots | Recovered with CelerData support |
| Apr 2026 | Data-quality row counts failed | One count needed about 60 GB against a 40 GB per-query limit | Checked for any row in place of counting all rows; 80 GB per query, half the workers |
| May 2026 | A job hung for about 10 hours | The FE accepted TCP connections but never sent the handshake | A 30-second connect timeout so retries engage; worst case now about 105 seconds |
| May 2026 | Dev FE stopped accepting connections for days | Iceberg connector cleanup held the coordinator lock | Upstream fix in StarRocks 4.0.11 and a rule: no production release until a month of steady state on a new major version |
| Jul 2026 | FE slow to plan queries on one table | It loaded all 3,300 or so partitions of a raw transactions table eagerly | Dropped partitions before 2026, leaving about 1,400 |
| Jul 2026 | Repeated compute-node out-of-memory errors | The default resource group allowed 100% memory with a spill threshold of 1.0, so spill could only start at the point of failure | 80% memory and a 0.7 spill threshold; separate groups for the API and BI |
| Aug 2026 | About 1,000 HTTP 5xx errors in 3 minutes | An API resource group rejected every query over its concurrency limit of 5 | Queues in front of StarRocks, then a cluster-wide admission gate |
| Sep 2026 | A spill fix silently did nothing | One rejected setting silently skipped every setting after it | Apply each session variable separately and check it |
| Sep 2026 | Open-source dev FE ran out of heap | The Iceberg metadata cache held 1 to 2.5 million partition names per FE | Larger FE heap, on-demand nodes, no statistics collection triggered by queries |
| Oct 2026 | Nodes hit the out-of-memory killer | jemalloc did not release dirty memory in time | Upgrade to StarRocks 4.0.16 |
StarRocks enforces hard per-query and per-group memory limits where Trino would often spill and keep going, so every limit has to sit below the failure point and every queue has to make queries wait. Partition count is also an FE cost: it drives metadata memory, planning time and materialized-view state long before it matters for query speed.
How the new design works
Spark writes Iceberg tables, StarRocks reads them in place and every served dataset is a view generated from configuration.
Generate every view from typed config
With 15,000+ views, hand-written SQL was never an option. Each topic is a short config file. It lists the topic’s entities, locations and time grains and every combination becomes a typed, validated view definition.
Columns are typed the same way. Each one declares its role (metric, filter, entity, event or field), a data type and its tests. Validation runs when a view is generated, before anything reaches a cluster:
- A state-level view must have a state column. A daily view needs a date column and weekly or monthly views need start and end dates.
- Every view needs at least one entity column.
- Partition and dependency columns must exist and names must fit StarRocks’ 63-character limit.
Every view is generated from about 180 shared SQL templates, so a fix in one template reaches every view that uses it.
Tests come from the same models. Every generated view gets a test that it is not empty. The test checks for any row because counting every row can exhaust StarRocks memory on heavy views. On the Iceberg side, our data-quality checks read table metadata only: row counts come from the snapshot and we flag a change of more than 1.5 standard deviations from the last three loads.
Publish on a branch, promote in one commit
Our old full refreshes emptied a table and rewrote it chunk by chunk, so clients saw an empty and then a partial table for the whole run. Iceberg branches fix that. Writers never commit to main. Each write goes to a named branch, created automatically on first write and recorded in a tracking table.
Promotion is three steps:
- Check the tracking table holds exactly one latest branch per table.
- Fast-forward
mainto the branch in a single Iceberg metadata commit. - Refresh the table in StarRocks so its metadata cache sees the new snapshot.
If main moved after the branch was cut, a fast-forward is impossible. For client-facing tables, promotion runs in strict mode: it fails with the cause and leaves main where it was, so the previous data keeps serving. Strict mode also drops the branch after a successful promotion. Today this covers our bulk and sample catalogs and the data library; certified tables are next.
The metadata that describes these views, such as date ranges, coverage and lineage, gets the same guarantee a different way. Each metadata release snapshots every metadata table and mints a version. If any step fails, only the tables that changed are reset to the last stable version. We can also roll back on demand.
Refresh in place
A view is stored SQL over a live table. When a vendor reinstates data, the view’s definition does not change, so there is nothing to rebuild. Until August 2026, our pipeline still rebuilt all StarRocks views on every full refresh, schema change or vendor reinstatement, which meant rebuilding every view on all three clusters for no change in output. Since August 2026 we rebuild view definitions only when a config changes.
A routine run now does three things: it looks up which source tables changed from a lineage table, refreshes those tables on each cluster that serves them and publishes the updated metadata. For our largest card-spend dataset, materializing its tables used to take six to eight hours after each certified build; as views, the data is live as soon as the refresh finishes.
Isolate workloads by how they query
Once publishing was cheap, the hard part moved to the read path. Three clients with different query patterns hit the same views:
| Client | Share of queries | Pattern | How we run it |
|---|---|---|---|
| API and web platform | ~80% | Targeted: metadata names the exact view | Own warehouse, protected first |
| MCP (AI agents) | 12–20% | Bursty: agents fan out many queries at once | Own warehouse, sized for bursts |
| SQL editor (ad hoc) | 1–2% | Unpredictable and heavy | Own small warehouse, queues and limits |
The application enforces this too. The SQL editor opens its own short-lived connection, sets its warehouse, catalog and query timeout and runs queued jobs the same way, so an analyst’s exploratory join never lands on API capacity.
Agents fan out many queries at once, so the MCP server has a second layer of protection. Every tool has a concurrency class and a rate class, enforced before a query reaches StarRocks.
Heavier classes get fewer slots. At most two long-running tools run at once while lightweight lookups get ten. Rate limits range from 3 calls in a burst for the most expensive tools to 250 for an organization.
Rate limits are a Redis token bucket in one atomic Lua script, checked per client and then per tool. Concurrency permits are acquired in order: global weight, then the class lane, then the tool’s own cap. Heavy tools such as text-to-insight are capped at six in flight. We coordinate the StarRocks admission gate through Redis across the server’s 4 to 12 autoscaled replicas because an earlier per-process semaphore had silently multiplied its capacity by the replica count.
We sized the warehouses by measurement. In August 2026, 30 days of utilization showed the fleet was memory-bound: average CPU stayed under 25% while memory peaked above 80%. So we moved from general-purpose m7g to memory-optimized r7g nodes with the same 128 GB per node and half the vCPUs, which cut cluster compute by about 40%. Two weeks later, the API warehouse pinned at 99% CPU during a burst of framework-filter queries, each fanning out into heavy scans for distinct values. The API’s resource group also hit its memory limit. That warehouse averaged 8% CPU and the average had hidden the bursts, so we moved that warehouse to r7g.8xlarge, 32 vCPU and 256 GB per node. In September 2026 we cut it and the MCP warehouse from nine nodes to six, the next valid size down, because a multi-AZ warehouse grows in multiples of three.
Traffic also keeps office hours. Over 28 days, the hours between 23:00 and 05:00 Eastern saw up to about 450 queries an hour, against 600 to 3,700 during the day. We now drop the API and MCP warehouses to three nodes from 22:00 to 06:00 with a scheduled scaling policy in Terraform. The daytime size is the base and the night is the override, so a missed schedule leaves us over-provisioned.
The rest came from tuning:
- Our first query queue admitted 15 concurrent queries. Nine heavy distinct-value queries could pin three nodes at 99% CPU without reaching that count, so queries were cancelled outright. Now a query also waits when any node passes 80% CPU or 80% memory. We lowered the count to 8 and a queued query now gives up after 120 seconds, down from 300.
- API queries whose planner cost estimate exceeds 100,000 go to a tighter resource group that runs five at a time, because heavy framework queries had pinned the API warehouse. Point lookups score in the tens to hundreds and the heaviest framework queries around 500 million, so one threshold separates them by orders of magnitude.
- Each node’s cache for Iceberg data was about 200 GB. It sat at 100% full and kept refetching from S3, so we moved it to 1 TB on four 256 GB gp3 volumes, which doubled the aggregate baseline to 12,000 IOPS at no extra cost. Each node now holds about 500 GB of hot data.
Results
Volume grew while latency stayed low. Monthly production queries rose from about 900,000 in August 2026 to 1.1 million in October 2026. Daily volume over the last 30 days ranged from 30,000 to 60,000. Query errors fell from about 12,000 in August 2026 to about 200 in October 2026.
Latency for the seven days ending November 3, 2026, from a sample of about 15,000 audited queries:
| Client | p50 | p75 | p90 |
|---|---|---|---|
| All queries | 200 ms | 390 ms | 850 ms |
| API and web platform | 190 ms | 300 ms | 500 ms |
| MCP (AI agents) | 500 ms | 1.2 s | 2.4 s |
Over the same period the overall p99 was about 4 seconds. Agent queries are the slowest class, because agents ask broader questions over longer date ranges.
Carbon Arc runs StarRocks on CelerData. No endorsement implied.
- starrocks
- trino
- iceberg
- mcp
- data-platform
