Blog / Engineering

From Trino to StarRocks: Serving 15,000+ Views over Petabytes of Data

How we stopped rebuilding every table on every pipeline run and now serve 15,000+ generated views over Iceberg on StarRocks.

Harsh Sharma and Tara Khanna · November 4, 2026 · 15 min read

We used to rebuild thousands of tables on every pipeline run. Now we serve 15,000+ views over Iceberg on StarRocks and a data refresh rebuilds nothing. This post is adapted from our Next Query 2026 talk. All numbers are rounded and as of November 3, 2026.

At Carbon Arc, more than 500 data assets are mapped to one proprietary ontology, cleaned, modeled and served as aggregated views. Clients can reach them through an API and web platform, a Python SDK, bulk S3 delivery and an MCP server that AI agents call directly.

Until early 2026, serving meant rebuilding approximately 11,000 aggregated tables (and growing) on every pipeline run and answering queries from one shared Trino cluster. We replaced that with StarRocks views over Apache Iceberg tables on S3. In October 2026 our production StarRocks cluster answered 1.1 million queries. The median query returned in 0.2 seconds. Each data refresh now publishes new Iceberg snapshots atomically. The views over them update in place.

across about 800 topics
15,000+
views served in production, all from config
was 0.9M in August
1.1M
production queries in October 2026
p90 0.85s
0.2s
median query latency, week ending Nov 3, 2026
600 → 8,600 a day
14×
growth in MCP agent tool calls, April to October

The challenge

Every dataset flows through the same stages. Spark on EMR cleans it, maps every record to the ontology and writes certified Iceberg tables. From those, we serve aggregated views at many grains: a brand by state by day, a company nationally by week, a ZIP code by day. Each grain covers a set of filter combinations for a given dataset that the API, the web platform and the MCP server can address by name.

The catalog is large and keeps growing. We serve 15,000+ generated views on the production cluster. Behind them sit petabytes of aggregated data. Two or three new datasets arrive each month and each one adds hundreds of views.

The Trino era: rebuild everything, every run

From 2023 to 2025, every one of those datasets was a physical table. Every pipeline run dropped and rebuilt each table on Trino from scratch. Spark built the heaviest ones. By the end we had over 11,000 denormalized Iceberg tables, and growing.

From rebuilding every table to views over Iceberg BEFORE · cost grows with table count Trino pipeline rebuild every run Every table denormalized copies Trino one shared cluster All clients contend together ~24 hours to rebuild · slow API responses · no warehouse isolation AFTER · cost grows with usage Spark writes once Iceberg on S3 main + write branches StarRocks 15,000+ views Clients API · MCP · ad hoc Generated views configs generate every view
Before, Trino rebuilt every denormalized Iceberg table on every run, with Spark for the heaviest and one shared Trino cluster served them. Now Spark writes Iceberg once, a refresh updates metadata and StarRocks serves generated views over Iceberg main.

The queries against those tables were mostly simple. The platform API and its 100+ endpoints looked up one table at a time, filtered by entity and date. Our internal tools read dataset metadata, such as date ranges and coverage, through the same Trino cluster. The first versions of our MCP server sent agent questions there too. Bulk aggregation jobs built large tables on it. Our Trino client allowed a query to run for up to a day and retried failures three times with backoff, which tells you how long some of them took.

All of it ran on open-source Trino on EKS, deployed with Helm, as three clusters with one coordinator each. The serving cluster autoscaled between 3 and 50 workers of 8 vCPU and 64 GB, about 400 vCPU at its ceiling. The dev cluster scaled to 60 workers of 16 vCPU and 128 GB. A separate pipeline cluster scaled to 40 workers of 32 vCPU and 256 GB. Each worker took a whole node, scaled on CPU and spooled task exchange to S3 so failed tasks could retry. We ran the production cluster inside our dev EKS cluster to sit next to the Hive Metastore. The three clusters ran three different Trino versions.

That design broke down as the catalog grew:

  • Building the denormalized tables took about 24 hours. For our largest card-spend dataset that was one day of a three-day refresh.
  • Every new dataset added tables to rebuild and maintain, including ones nobody queried, so cost tracked the table count.
  • By December 2025 the Trino cluster was under memory pressure. Research and MCP queries on the dev cluster were timing out.
  • Open-source Trino has no warehouse isolation and we had configured no resource groups, so every client shared one queue. A heavy aggregation competed with API lookups. Every new user added to the bottleneck.

In August 2025 we had started planning a pipeline to run aggregations directly — query the certified Iceberg tables, apply views as filters over them, and delete the approximately 11,000 tables along with their ETL — but we still needed to fully scope it out. We estimated that dropping the rebuild layer alone would save about $20,000 a month.

First attempt: turn every table into a materialized view

While that plan was still taking shape, the obvious first move was to keep the table shape and change only the engine. In the Trino era every grain was its own physical table: card spend by brand by state by day was one table, by brand nationally by week was another, each a denormalized copy of the same certified data. StarRocks can maintain copies like that itself, as asynchronous materialized views, so that is what we built. In August 2025 we built the first StarRocks model as a materialized view. Over the next ten weeks we built every dataset the same way. By October 2025 we had thousands of them, each with a manual refresh.

The frontend (FE), which plans queries and holds the catalog, ran out of heap. Query memory on the compute nodes held up.

Two things in FE memory grow with partitions:

  • Each materialized view records which snapshot of each base-table partition it last read, so it can refresh incrementally. A view over a table partitioned by day with five years of history tracks about 1,800 partitions. Every materialized view keeps its own copy of that map.
  • The Iceberg metadata cache holds each base table’s partition list, which the FE reads to decide what to refresh. Every refresh check walks those partitions again.

The denormalized design multiplied both. Thousands of materialized views sat over the same few certified tables, so the same daily partitions were tracked thousands of times, once per view. FE memory grew with views times partitions and both kept rising: each new dataset added hundreds of views and each day added a partition to every base table. Refreshes slowed with it. Materializing every table of our largest card-spend dataset took six to eight hours after each certified build. Transaction errors piled up, so we wrote a script to convert failing models. QA waited for the cluster to free up.

Materialized views multiplied FE metadata; plain views add none BEFORE · denormalized, as materialized views Certified table MV 1 MV 2 MV n ⋮ FE state ≈ materialized views × partitions each view keeps its own partition map AFTER · plain views over certified tables Certified table view SQL text only view SQL text only 15,000+ views SQL text only ⋮ FE state ≈ base-table partitions adding views adds no metadata
As materialized views, every denormalized copy kept its own map of the base table's partitions in FE memory, so state grew with views times partitions. As plain views, the grains are SQL text and FE metadata grows only with the certified tables underneath.

Replacing copies with plain views

We kept one set of certified Iceberg tables per topic and expressed every grain as a plain view over them: a saved query that filters and aggregates when someone asks for it. A view holds no data and no refresh state. To the FE it is a few kilobytes of SQL text. FE metadata now scales with the certified tables and their partitions, which grow with the data alone.

In mid-October 2025 we concluded that materializing every grain was the wrong pattern for StarRocks. The FE could not hold metadata for thousands of materialized views without growing every month. We decided to serve the denormalized grains as plain views over the certified tables and made views the default for every new dataset. With views, the same card-spend data was live the moment its certified tables committed, six to eight hours sooner than before. That decision set the design this post describes.

We never went back to materializing everything. Partitions still cost FE memory after the switch, as the incident table below shows, but the count now grows only with base tables. In October 2026 we decided to return to materialized views for 28 hot query families, each one chosen because its measured query cost justified the refresh time and FE state it adds. StarRocks 4.0.16 had fixed a jemalloc issue where freed memory was not returned quickly enough, which had pushed nodes into the operating system’s out-of-memory killer. We also gave refreshes their own warehouse and made them all-or-nothing. That story is in Precomputing the Hard Queries.

What we needed and how we tested it

We wrote four requirements before committing to an engine:

  1. No vendor lock-in and no rebuild layer. Data stays in open Iceberg tables on S3 and nothing sits between the lake and the customer that must be regenerated.
  2. Fast on plain views. Interactive latency on views that aggregate at query time, without pre-building every table.
  3. Isolated workloads. Separate compute for API, agent and ad hoc traffic, so one cannot slow another.
  4. A bill that does not grow linearly with data.

We tested StarRocks on our own production workload: real platform queries against plain views over Iceberg at full scale. We drove API, MCP and ad hoc load at the same time on separate warehouses. In January 2026 our analytics team ran a full row-count and metric parity test between Trino and StarRocks. At cutover we compared every table against its replacement view across the stack. Those comparisons caught real issues before customers did. When we later moved production to a new StarRocks cluster, we copied the 15,000+ view definitions and re-created them in parallel because none depended on another view.

We run StarRocks on CelerData’s bring-your-own-cloud service in our AWS account, on Graviton instances, with an open-source StarRocks cluster on EKS for internal batch jobs.

The road from Trino to StarRocks Aug 2025 First StarRocks models all materialized views Oct 2025 Chose plain views smaller FE, faster refreshes Dec 2025 Dev leaves Trino Trino spend halved Jan 2026 Prod serves from StarRocks MCP, bulk; API in Oct 2025 Feb–Jun 2026 Stabilizing OOMs, FE hangs, upgrades Jul 2026 New prod cluster isolated warehouses Aug 2026 Last Trino SQL gone pipeline on Spark + Iceberg Oct 2026 MVs return, selectively on StarRocks 4.0.16
Milestones from git history, Slack and Linear. The first StarRocks design materialized every view; the one that shipped serves plain views and brings back materialized views only for a few hot query families.

We left Trino gradually so we could compare every table against its replacement view at each cutover. We moved the platform API in October 2025. In December 2025 we cut the dev Trino cluster to a fixed 10 workers and moved dev to StarRocks to reduce Trino spend, which halved it. We moved production serving in January 2026: the MCP server on January 13 and the bulk aggregator on January 20, keeping a Trino fallback while it settled. We moved pipeline reads over the next seven months and converted the last Trino SQL in the pipeline to Spark in August 2026.

Running StarRocks in production: what broke

The first months on StarRocks were not smooth. Most incidents traced back to memory limits that behaved differently from Trino’s, or to the FE doing more work than we expected:

DateWhat happenedRoot causeWhat we changed
Jan 2026Production outage: a compute node exceeded its process memory limitToo many concurrent heavy queries; spill did not engageCapped concurrency with a query queue
Feb 2026Production outage: the FE marked every compute node unavailableNodes unresponsive under memory pressure, during a flood of Iceberg metadata refreshes from expired snapshotsRecovered with CelerData support
Apr 2026Data-quality row counts failedOne count needed about 60 GB against a 40 GB per-query limitChecked for any row in place of counting all rows; 80 GB per query, half the workers
May 2026A job hung for about 10 hoursThe FE accepted TCP connections but never sent the handshakeA 30-second connect timeout so retries engage; worst case now about 105 seconds
May 2026Dev FE stopped accepting connections for daysIceberg connector cleanup held the coordinator lockUpstream fix in StarRocks 4.0.11 and a rule: no production release until a month of steady state on a new major version
Jul 2026FE slow to plan queries on one tableIt loaded all 3,300 or so partitions of a raw transactions table eagerlyDropped partitions before 2026, leaving about 1,400
Jul 2026Repeated compute-node out-of-memory errorsThe default resource group allowed 100% memory with a spill threshold of 1.0, so spill could only start at the point of failure80% memory and a 0.7 spill threshold; separate groups for the API and BI
Aug 2026About 1,000 HTTP 5xx errors in 3 minutesAn API resource group rejected every query over its concurrency limit of 5Queues in front of StarRocks, then a cluster-wide admission gate
Sep 2026A spill fix silently did nothingOne rejected setting silently skipped every setting after itApply each session variable separately and check it
Sep 2026Open-source dev FE ran out of heapThe Iceberg metadata cache held 1 to 2.5 million partition names per FELarger FE heap, on-demand nodes, no statistics collection triggered by queries
Oct 2026Nodes hit the out-of-memory killerjemalloc did not release dirty memory in timeUpgrade to StarRocks 4.0.16

StarRocks enforces hard per-query and per-group memory limits where Trino would often spill and keep going, so every limit has to sit below the failure point and every queue has to make queries wait. Partition count is also an FE cost: it drives metadata memory, planning time and materialized-view state long before it matters for query speed.

How the new design works

Spark writes Iceberg tables, StarRocks reads them in place and every served dataset is a view generated from configuration.

Generate every view from typed config

With 15,000+ views, hand-written SQL was never an option. Each topic is a short config file. It lists the topic’s entities, locations and time grains and every combination becomes a typed, validated view definition.

One config fans out into hundreds of validated views card.py ModelConfiguration 6 configured columns spend, avg spend, transactions, users, direction, method source: certified Iceberg tables Entity map locationbrand · goodsbrand · services+ representations × Locations usstatedmacbsazip × Dates day 250 views for the card topic 15,000+ across about 800 topics {entity}_{type}_{domain}_{topic}_{representation}_{location}_{date}
One topic config (the card-spend topic shown) crosses its entity map with its location and date grains. Each combination becomes a validated view definition. The card topic alone generates 250 views.

Columns are typed the same way. Each one declares its role (metric, filter, entity, event or field), a data type and its tests. Validation runs when a view is generated, before anything reaches a cluster:

  • A state-level view must have a state column. A daily view needs a date column and weekly or monthly views need start and end dates.
  • Every view needs at least one entity column.
  • Partition and dependency columns must exist and names must fit StarRocks’ 63-character limit.

Every view is generated from about 180 shared SQL templates, so a fix in one template reaches every view that uses it.

Tests come from the same models. Every generated view gets a test that it is not empty. The test checks for any row because counting every row can exhaust StarRocks memory on heavy views. On the Iceberg side, our data-quality checks read table metadata only: row counts come from the snapshot and we flag a change of more than 1.5 standard deviations from the last three loads.

Publish on a branch, promote in one commit

Our old full refreshes emptied a table and rewrote it chunk by chunk, so clients saw an empty and then a partial table for the whole run. Iceberg branches fix that. Writers never commit to main. Each write goes to a named branch, created automatically on first write and recorded in a tracking table.

Clients read main; new data lands on a branch and goes live in one commit Spark writer full or partial refresh branch_<name> built in isolation fast_forward main → branch, one commit REFRESH EXTERNAL StarRocks sees new main main moved underneath? strict mode fails; main unchanged What clients see previous main, for the whole write new main, complete
Writers commit to a named branch. Promotion fast-forwards main to that branch in one Iceberg commit, then refreshes StarRocks' cached table metadata. Readers see main.

Promotion is three steps:

  1. Check the tracking table holds exactly one latest branch per table.
  2. Fast-forward main to the branch in a single Iceberg metadata commit.
  3. Refresh the table in StarRocks so its metadata cache sees the new snapshot.

If main moved after the branch was cut, a fast-forward is impossible. For client-facing tables, promotion runs in strict mode: it fails with the cause and leaves main where it was, so the previous data keeps serving. Strict mode also drops the branch after a successful promotion. Today this covers our bulk and sample catalogs and the data library; certified tables are next.

The metadata that describes these views, such as date ranges, coverage and lineage, gets the same guarantee a different way. Each metadata release snapshots every metadata table and mints a version. If any step fails, only the tables that changed are reset to the last stable version. We can also roll back on demand.

Refresh in place

A view is stored SQL over a live table. When a vendor reinstates data, the view’s definition does not change, so there is nothing to rebuild. Until August 2026, our pipeline still rebuilt all StarRocks views on every full refresh, schema change or vendor reinstatement, which meant rebuilding every view on all three clusters for no change in output. Since August 2026 we rebuild view definitions only when a config changes.

A routine run now does three things: it looks up which source tables changed from a lineage table, refreshes those tables on each cluster that serves them and publishes the updated metadata. For our largest card-spend dataset, materializing its tables used to take six to eight hours after each certified build; as views, the data is live as soon as the refresh finishes.

Isolate workloads by how they query

Once publishing was cheap, the hard part moved to the read path. Three clients with different query patterns hit the same views:

ClientShare of queriesPatternHow we run it
API and web platform~80%Targeted: metadata names the exact viewOwn warehouse, protected first
MCP (AI agents)12–20%Bursty: agents fan out many queries at onceOwn warehouse, sized for bursts
SQL editor (ad hoc)1–2%Unpredictable and heavyOwn small warehouse, queues and limits
One engine, one warehouse per way of querying Production cluster · shared Iceberg data API default_warehouse Targeted, predictable 6 × r7g.8xlarge 3 overnight Protected first MCP mcp Bursty, high volume 6 × r7g.4xlarge 3 overnight Sized for bursts Ad hoc sql_editor Unpredictable, heavy 3 × r7g.2xlarge Queues and limits Pipeline mv_refresh Batch, scheduled 3 × r7g.4xlarge single AZ Off client warehouses Dev cluster: its own CelerData cluster, plus open-source StarRocks on EKS for internal batch jobs
Each client class gets its own warehouse in production, sized by how it queries. Development runs on its own cluster, so mass tests can't hit prod concurrency limits.

The application enforces this too. The SQL editor opens its own short-lived connection, sets its warehouse, catalog and query timeout and runs queued jobs the same way, so an analyst’s exploratory join never lands on API capacity.

Agents fan out many queries at once, so the MCP server has a second layer of protection. Every tool has a concurrency class and a rate class, enforced before a query reaches StarRocks.

Heavier classes get fewer slots. At most two long-running tools run at once while lightweight lookups get ten. Rate limits range from 3 calls in a burst for the most expensive tools to 250 for an organization.

Rate limits are a Redis token bucket in one atomic Lua script, checked per client and then per tool. Concurrency permits are acquired in order: global weight, then the class lane, then the tool’s own cap. Heavy tools such as text-to-insight are capped at six in flight. We coordinate the StarRocks admission gate through Redis across the server’s 4 to 12 autoscaled replicas because an earlier per-process semaphore had silently multiplied its capacity by the replica count.

We sized the warehouses by measurement. In August 2026, 30 days of utilization showed the fleet was memory-bound: average CPU stayed under 25% while memory peaked above 80%. So we moved from general-purpose m7g to memory-optimized r7g nodes with the same 128 GB per node and half the vCPUs, which cut cluster compute by about 40%. Two weeks later, the API warehouse pinned at 99% CPU during a burst of framework-filter queries, each fanning out into heavy scans for distinct values. The API’s resource group also hit its memory limit. That warehouse averaged 8% CPU and the average had hidden the bursts, so we moved that warehouse to r7g.8xlarge, 32 vCPU and 256 GB per node. In September 2026 we cut it and the MCP warehouse from nine nodes to six, the next valid size down, because a multi-AZ warehouse grows in multiples of three.

Traffic also keeps office hours. Over 28 days, the hours between 23:00 and 05:00 Eastern saw up to about 450 queries an hour, against 600 to 3,700 during the day. We now drop the API and MCP warehouses to three nodes from 22:00 to 06:00 with a scheduled scaling policy in Terraform. The daytime size is the base and the night is the override, so a missed schedule leaves us over-provisioned.

The rest came from tuning:

  • Our first query queue admitted 15 concurrent queries. Nine heavy distinct-value queries could pin three nodes at 99% CPU without reaching that count, so queries were cancelled outright. Now a query also waits when any node passes 80% CPU or 80% memory. We lowered the count to 8 and a queued query now gives up after 120 seconds, down from 300.
  • API queries whose planner cost estimate exceeds 100,000 go to a tighter resource group that runs five at a time, because heavy framework queries had pinned the API warehouse. Point lookups score in the tens to hundreds and the heaviest framework queries around 500 million, so one threshold separates them by orders of magnitude.
  • Each node’s cache for Iceberg data was about 200 GB. It sat at 100% full and kept refetching from S3, so we moved it to 1 TB on four 256 GB gp3 volumes, which doubled the aggregate baseline to 12,000 IOPS at no extra cost. Each node now holds about 500 GB of hot data.

Results

Volume grew while latency stayed low. Monthly production queries rose from about 900,000 in August 2026 to 1.1 million in October 2026. Daily volume over the last 30 days ranged from 30,000 to 60,000. Query errors fell from about 12,000 in August 2026 to about 200 in October 2026.

Latency for the seven days ending November 3, 2026, from a sample of about 15,000 audited queries:

Clientp50p75p90
All queries200 ms390 ms850 ms
API and web platform190 ms300 ms500 ms
MCP (AI agents)500 ms1.2 s2.4 s

Over the same period the overall p99 was about 4 seconds. Agent queries are the slowest class, because agents ask broader questions over longer date ranges.

Carbon Arc runs StarRocks on CelerData. No endorsement implied.

  • starrocks
  • trino
  • iceberg
  • mcp
  • data-platform

Share this post

Written by

Harsh Sharma

Written by

Tara Khanna

More from Engineering