Blog / Engineering

Arrow the Whole Way: Serving Query Results over Arrow Flight

Why we stopped turning columnar query results into JSON, and how a Flight data plane in front of StarRocks checks access, meters every delivery and keeps the browser endpoints working.

Myles Blodnick · November 6, 2026 · 11 min read

Carbon Arc builds data infrastructure for the AI economy. We take in third-party datasets, such as card transactions, point-of-sale data, clickstream and foot traffic, and map every record to a proprietary ontology of companies, brands, tickers and places. Clients buy a framework, a defined slice of that data (these companies, these measures, this date range), and then pull its results into whatever they analyze it with: our web platform, a notebook through our Python SDK, or their own pipeline.

Every one of those pulls ended the same way. Our API ran the framework’s SQL against StarRocks, read the rows back over the MySQL protocol, built a pandas DataFrame in the API process, and serialized it to JSON or CSV. The client then parsed that text back into a table. We replaced the transport with an Apache Arrow Flight data plane, so results stay in Arrow from the StarRocks scan to the edge of our API, and all the way to the client when the client asks for Arrow. It has served framework pulls from our web platform and API since July 23. In our tests, a warm pull of about 47,000 rows reached a client DataFrame 1.3 to 1.6 times faster. A small pull got slower.

2.90 s → 1.81 s
1.6×
faster to a client DataFrame on a 47,000-row framework
was 4
0
format conversions between the StarRocks scan and an Arrow client
first week
4,100
deliveries in production, behind the same HTTP endpoints
per pull
1
usage event for every delivery: completed, partial or failed

This post covers why the old path was slow, the five decisions behind the data plane, what we measured (including where Flight is slower), and what is left.

The challenge

StarRocks stores and scans data by column. Our clients analyze it by column too: a DataFrame in pandas or Polars, a table in a warehouse. Everything in between was rows and text.

Before: rows, then objects, then text, then back StarRocks columnar scan MySQL rows row by row pandas in the API JSON text names repeated Parse JSON on the client DataFrame for analysis Four conversions, and the API process holds the whole result before it sends a byte. After: Arrow the whole way StarRocks columnar scan Flight SQL Arrow batches Flight server Arrow batches Arrow table on the client DataFrame for analysis The same column buffers move from the scan to the client. Turning them into a DataFrame is close to a copy.
The legacy path turns columnar results into rows, then Python objects, then text, and the client turns them back. Compression makes the text smaller on the wire, but the server still builds it and the client still parses it. The Flight path keeps the data columnar from the scan to the client.

The legacy path converted each result four times. StarRocks sent rows over the MySQL wire protocol. The API collected them into a pandas DataFrame, then turned the DataFrame into a list of dictionaries and serialized that as JSON, so every column name was repeated on every row and every number became text. The client read the text and parsed it back into rows and then into a table. Brotli compression made the response smaller on the wire, but it didn’t remove any of that work, and it added one more step on each side.

The API process also held the whole result, twice, before it sent the first byte: once as a DataFrame and once as the response body. A large framework tied up that worker’s memory and CPU for as long as it took to build the response.

There was a second problem that had nothing to do with speed. We charge when a client buys a framework, and some of our supplier agreements share revenue based on how their data is used. A purchase doesn’t tell you what was delivered: how many rows, how many bytes, whether the pull finished or the client gave up halfway. We wanted a record of each delivery, written by the thing that delivered it.

Solution overview

Arrow Flight is a gRPC protocol for moving Arrow data. A client asks a Flight server about a dataset with GetFlightInfo and gets back one or more tickets, then redeems a ticket with DoGet and receives a stream of Arrow record batches. The batches arrive in the same column layout Arrow uses in memory, so the client does almost no work to use them.

We run a Flight server as a second process in the same container as our API. The web platform and API keep calling the HTTP endpoints they always used, and those endpoints now forward to the Flight server over localhost. Direct Arrow access for SDK clients comes next. Either way, one path runs the query, checks entitlement and records the delivery.

One path runs the query, checks access, caches and meters Python SDK Arrow over gRPC Web platform JSON and CSV over HTTP API container Flight server GetFlightInfo: token, entitlement, freshness, then a signed ticket for this user and this SQL DoGet: verify the ticket, check access again, wait for a query slot, read or scan, stream batches One usage event per delivery HTTP endpoints same /data and /export URLs Arrow to JSON or CSV here localhost StarRocks Flight SQL over ADBC Arrow straight from the scan Redis Arrow result cache keyed by the SQL's hash Postgres usage events: completed, partial or failed
The Flight server runs as a second process in the same container as the API. SDK clients talk to it directly over gRPC. Browser requests reach it through the existing HTTP endpoints, which forward the caller's token over localhost and shape Arrow into JSON or CSV only at the edge.

Five design decisions

1. Run our own Flight server instead of exposing StarRocks’

StarRocks already serves Arrow Flight SQL, so the obvious design was to put a proxy in front of it and let clients query it directly. We didn’t, for three reasons. Clients ask for a framework, not a table, and the SQL behind a framework is generated on our side and never shown to the client. Access depends on who bought what, which StarRocks knows nothing about. And we needed to meter each delivery, which means something of ours has to sit on the stream.

So clients send a small command, {"framework_id": "..."}, and our server does the rest. Building it was less work than it sounds, because our StarRocks client library already spoke Flight SQL through the ADBC driver. StarRocks still executes the scan and returns Arrow. Our server never converts it to pandas or anything else; it adds the checks, the cache and the meter around a stream that is Arrow from end to end.

2. Tickets that name the user, the framework and the exact SQL

GetFlightInfo authenticates the caller’s existing API token, checks that they own the framework (or bought it on our data exchange, or, for admins, that it belongs to their team), and checks that the data hasn’t been replaced upstream since they bought it. Then it issues a ticket: a small signed payload with the user, their client account, the framework, a SHA-256 hash of the SQL they are entitled to run, a random nonce and a five-minute expiry, signed with HMAC-SHA256.

body   = base64url({"sub": user, "cid": client, "fid": framework, "qfp": sha256(sql),
                    "nonce": ..., "iat": issued, "exp": issued + 300})
ticket = body + "." + hex(hmac_sha256(secret, body))

A ticket is not a bearer capability on its own. DoGet authenticates the token again, checks that the ticket was issued to that same user, and checks entitlement again, because access can be revoked between the two calls. If the framework’s SQL changed in the meantime, the hash no longer matches and the client is told to call GetFlightInfo again rather than receiving data for a query it wasn’t granted.

One detail matters more than it looks: the server compares the signature as raw bytes and never decodes the signature the client sent. Decoding first would let a signature with a non-ASCII byte raise a TypeError inside the comparison, which would escape as a server error instead of a rejected ticket.

3. Meter the delivery, not the request

Every DoGet that gets past the ticket, entitlement and freshness checks writes exactly one usage event to Postgres, with the rows, bytes and columns delivered and one of three outcomes:

OutcomeWhen
completedEvery batch was handed to the transport
partialThe client disconnected partway through
failedThe query failed, or it timed out waiting for a query slot

The server streams batches of up to 65,536 rows from a generator and writes the event in its finally block. When a client disconnects, gRPC closes the generator, which raises GeneratorExit at the yield. That is why the counters move before the yield, not after: counting after would drop the batch that had already been handed to the transport, and every partial event would undercount by one batch.

Before it scans, a pull goes through an admission gate that weighs queries with the same cost estimator as our other StarRocks admission gates, so an expensive framework takes more of the budget than a cheap one. If the wait runs out, the client gets UNAVAILABLE, which tells the SDK the pull is worth retrying, and the ledger still gets a failed event. Cache hits are metered too. Tickets can be redeemed more than once within their five minutes, so the nonce goes into each event, and duplicates can be removed downstream.

4. Keep the HTTP endpoints, and put them on the plane

Browsers can’t speak gRPC, and the web platform calls /data, /data?data_type=timeseries and /export directly. We kept those URLs. Behind a flag, they become a thin facade: they forward the caller’s own token to the Flight server over localhost, receive Arrow, and only then shape it into JSON records, a time series or CSV for the browser.

That keeps one data path. The facade doesn’t run queries, check entitlement or meter; the plane does all three, so a browser download lands in the same usage ledger as an SDK pull. The facade does keep a five-minute copy of each user’s Arrow table, so paging through a view and then exporting it counts as one metered pull, not several. The time series shaping reuses the legacy code, moved into a shared module, so the payload the web platform receives is identical by construction rather than by testing.

It also made turning the plane on for production a flag change. The flag has been on since July 23, and nothing on the frontend changed. In the first week, about 600 pulls asked for a framework whose data had been replaced upstream since purchase. The plane raised its own error, and the facade turned it into a 409, the status the web platform already handled for that case.

5. Tune the client, not the server

Arrow batches are large. A framework with 42 columns at about 620 bytes per row is roughly 40 MB in one of the plane’s 65,536-row batches. Two client defaults got in the way, one on each side of the plane.

The first was ours, as a client of StarRocks. The ADBC Flight SQL driver accepts messages of at most 16 MB by default, so our pulls of wide frameworks from StarRocks failed with Received message larger than max. We raised it to 256 MB, and made StarRocks’ chunk_size configurable, so a pull can ask for 16,384 rows per batch instead of 4,096 and pay the per-batch overhead a quarter as often.

The second is on our clients’ side. For SDK clients outside our network, the limit is gRPC flow control. The server advertises the default 64 KB initial window, which caps a single stream at roughly the window divided by the round-trip time until gRPC’s auto-tuning catches up. Raising the client’s stream window to 32 MB and its connection window to 64 MB is the biggest lever for pulls over long links, and it needs no server change, because the server accepts whatever the client advertises.

client = flight.connect(
    "grpc+tls://flight.example.com:443",
    generic_options=[
        ("grpc.initial_window_size", 32 * 1024 * 1024),
        ("grpc.initial_conn_window_size", 64 * 1024 * 1024),
        ("grpc.max_receive_message_length", 256 * 1024 * 1024),
    ],
)

Results

We pulled frameworks through both paths with the same account, from a laptop over VPN to our development cluster, three to five times per path, alternating which path went first. We report the median of the warm pulls, after both paths had the result cached, because that isolates what this change is about: getting a result from the API into a client DataFrame. The StarRocks query is the same on both paths. Three frameworks finished cleanly on both paths. A fourth, at about 71,000 rows, didn’t finish enough Flight runs to report. These numbers reflect our data, our network and a development cluster, so your results might vary.

Framework sizeJSON on the wire (Brotli)Arrow on the wireLegacy, warmFlight, warmChange
1,464 rows × 6 columns16 KB98 KB0.51 s0.94 s0.43 s slower
47,041 rows × 9 columns256 KB4.7 MB2.93 s2.28 s1.3× faster
47,184 rows × 9 columns257 KB4.7 MB2.90 s1.81 s1.6× faster

Most of the gain is on the server. On the 47,000-row frameworks, the legacy client spent about 0.06 seconds parsing JSON and 0.03 seconds building the DataFrame. Arrow cut both to almost nothing, but that is a tenth of a second out of nearly three. The rest came from the API no longer building a DataFrame, turning it into dictionaries, and serializing and compressing 10.8 MB of JSON before it could send anything.

Two results went against Flight. Small pulls got slower. A Flight pull is two calls, GetFlightInfo and then DoGet, and each one checks entitlement and freshness against the database, so a pull pays for two round trips and two sets of access checks. GetFlightInfo alone took 0.3 to 0.4 seconds, which is most of the gap on the 1,464-row framework. And Arrow is bigger on the wire. We send it uncompressed, so 47,000 rows were 4.7 MB of Arrow against 257 KB of Brotli JSON, about 18 times more. Over our VPN, Flight still came out ahead, because building and compressing the JSON cost more than moving the extra bytes. On a slower link, that trade can flip.

The same idea in our MCP server

Our MCP server reads StarRocks directly, so it doesn’t go through this plane, but we applied the same lesson to that path. pandas’ read_sql_query holds every row as a Python tuple next to the finished DataFrame. MCP now builds the frame from Arrow batches of 100,000 rows instead, over the MySQL connection it already had, and that runs in production today. On a 1.2-million-row fetch in our development environment, peak memory fell from 1,364 MB to 346 MB. It was also part of the groundwork for splitting the server into routers and horizontally scaled workers, which the first post in our MCP series, Adding Pods Didn’t Help, covers.

Lessons learned

  • The client format decides the server design. Our clients wanted a DataFrame. Every conversion between the scan and that DataFrame was work someone paid for, on the server or on the client. Moving the boundary to Arrow removed the work instead of making it faster.
  • Measure the small pulls too. Flight’s gain grows with the result. On a small one, the second round trip and the second access check cost more than the conversions we removed.
  • Arrow removes conversions, not bytes. Uncompressed Arrow was 18 times the size of Brotli JSON. Compression is a separate decision, and we haven’t made it yet.
  • Put the meter where the bytes leave. Metering in the handler that starts a pull can’t tell a finished download from an abandoned one. A generator’s finally block can, as long as the counters move before the yield.
  • Keep the old URLs. A facade let the browser use the new plane with no frontend change, made the production rollout a flag, and means one entitlement check and one meter instead of two that have to agree.
  • Defaults are sized for small messages. Both of our throughput problems were client defaults, 16 MB messages and 64 KB windows, not anything on the server.

What’s next

The plane has carried framework pulls from our web platform and API since July 23, behind the same HTTP endpoints. Four things are left:

  • Arrow for SDK clients. A gRPC route into the production cluster and an SDK release with get_dataset(...).to_arrow() and .to_polars(), so Python clients get Arrow instead of the JSON the facade makes for them.
  • Compression. Arrow IPC supports ZSTD-compressed record batches, which would shrink what goes over the wire.
  • Streaming straight through. Today the plane reads the full result from StarRocks as an Arrow table, because the result cache stores whole tables, and only then streams it out in batches. When we benchmarked our largest frameworks, back-to-back pulls ran our development API pods out of memory. Passing batches through as StarRocks produces them would cap the server’s memory per pull and get the first rows to the client sooner.
  • Retiring the old path. Frameworks bought on our data exchange already go through the plane, but a few internal callers, such as panel de-biasing and training data, still use the old serving code. Once they move, we can delete it.

Conclusion

Our results started columnar and ended columnar, and everything in between was rows and text. An Arrow Flight data plane removes those conversions, checks access with tickets bound to the exact SQL a user bought, and writes one usage event for every delivery, finished or not. It isn’t free: small pulls pay for an extra round trip, and Arrow needs compression before it is smaller on the wire than JSON. The browser keeps its JSON, produced at the very edge from the same Arrow stream.

If you serve analytical results to clients who will load them into a DataFrame, look at how many times your data changes shape on the way, and measure your small results as well as your large ones.

  • arrow
  • arrow-flight
  • starrocks
  • grpc
  • performance

Share this post

Written by

Myles Blodnick

More from Engineering