Blog / Engineering

Adding Pods Didn't Help

A primer on Python's event loop, the GIL, and how one blocked loop stalled our MCP server. Part 1 of our MCP re-architecture series.

Chris Blanton · November 5, 2026 · 5 min read

103 s → 7.0 s
15×
faster p95 for heavy data queries, load-tested at 64× production traffic
22.4 s → 1.7 s
13×
less event-loop stall per pod, per minute, in the same 64× load test
was 16×
65×
production traffic held with clean latency in load tests
with Claude Code
4 days
from first commit to a load-tested rewrite

The event loop and the GIL

An async Python server runs one event loop per process. While one request waits on the network, the loop runs another. Anything that holds the loop, such as a synchronous database call or a big serialization, makes every other request on the process wait, health checks included.

Moving the work to a thread doesn’t always help. The global interpreter lock (GIL) lets only one thread run Python at a time, and CPU-heavy library calls like pandas transforms can hold it, so the loop still can’t run. More pods don’t help either: each one is another single loop that stalls the same way.

One loop per pod: it either takes turns or it waits request running on the loop blocking work requests stuck waiting Healthy turns while each waits on I/O A B C D A B C D Blocking call A makes a sync database call A: sync database call B, C, D and health checks wait B C D Thread holds the GIL A hands a pandas job to a thread A loop thread can't run: no GIL worker thread: pandas holds the GIL B C D time →
Illustrative. Each bar is one pod's event loop; orange slices are requests A to D taking turns. A synchronous call holds the loop until it returns. A CPU-heavy library call in a thread can hold the GIL the same way, so the loop thread can't run either.

What it looked like

MCP usage is growing quickly as more clients connect AI agents straight to our data. Handling today’s traffic isn’t enough: to meet that demand, the server has to absorb many times it without slowing down. So we load-test well past today’s peak, stepping traffic up in multiples of our average production rate.

In those load tests, at 64× average production traffic, each old-build pod’s loop was stalled 22 seconds of every minute, with over a thousand tasks queued behind it, on 1 to 1.8 of its 4 cores. Quick lookups went from under 0.3 seconds to about 20, and our traces blamed the dependencies.

Our traces blamed the dependency; the loop was the culprit What the trace said client span for one lookup 18 s What actually happened same 18 seconds waiting for the event loop to come back the call itself, per the dependency's own metrics 0 s 5 s 10 s 15 s 20 s
The span is the old build's quick-lookup p95 at 64× traffic. The split inside it is illustrative: the call itself was quick, and the rest was time spent queued behind a blocked loop, which the span can't tell apart.

Blocking also chained across services. Every MCP request checked the caller with the auth and billing service, through a synchronous call on the loop, and that service’s own loop was blocked by synchronous database calls. It stalled under load in August, and every MCP pod waiting on it stalled too.

One slow auth lookup froze the whole MCP pod sync HTTP sync SQL MCP server pod loop blocked while it waits on the auth check every other request on the pod, health checks included, waits too Auth and billing service loop blocked by sync database calls inside async routes, so one slow query held every lookup behind it Postgres users, billing
Every MCP request checked the caller with the auth service. That check was a synchronous HTTP call on the MCP pod's loop, and the auth service's own loop was blocked by synchronous database calls. Any slowness at the end of the chain froze every request on the MCP pod.

Finding it

How we found it: measure the loop, not the spans 1 Loop-lag monitor Injected into every pod atstartup. Times how late the loopwakes and names the codethat held it. 2 Server-side check Compares each client span withthe dependency's own latencyfor the same window. 3 Dependency sweep Slows one dependency at a timethrough a proxy and flagsroutes whose loop lag rises.
Three checks, from production-like load down to a single route. Part 3 of this series covers them in detail.

Fixing it

  • Off the loop. Synchronous calls moved to threads, and clients are reused instead of rebuilt on every call.
  • Cached auth. A per-token cache cut auth service lookups from 39,540 to 934 while serving almost 3× the requests.

Rewriting it

Each fix helped, but what remained was CPU-heavy Python holding the GIL on a single loop, and no patch fixes that. In mid-October we stopped patching and redesigned the server. With Claude Code, the rewrite reached a load-tested build matching the old one in four days.

Six weeks of patches, then four days to a rewrite six weeks of fixes Sep Oct Nov Stall confirmed loop-lag monitor on MCP Stop patching first commit of the rewrite Load-tested matches the old build Merged to main after review Scorecard runs the results below
Six weeks of fixes on the old design, then four days from first commit to a load-tested rewrite, written with Claude Code.
of 175 commits
161
written with Claude Code
files
528
touched by the rewrite
−4.3k lines
+56k
added and removed
Oct 18 → Oct 28
10 days
of review before merging to main

Groundwork already in place made the split fast:

  • One tool runner. Admission, time budget, metrics and billing for every tool call live in one place, so the router only had to swap in a call to a worker there.
  • Leaner heavy queries. Arrow batches cut a 1.2-million-row fetch from 1,364 MB to 346 MB at peak, and aggregating in SQL took the heaviest framework from 1.4 GB to 7 MB.
  • Async query jobs. Long queries are submitted and retrieved later instead of holding a connection open.

The rewrite splits the server in two. Routers handle the protocol and auth and forward tool calls over gRPC. Workers run the tools and scale out on their own. CPU-only PyTorch and the split shrank the images too.

One event loop per pod became routers and workers BEFORE · EVERYTHING IN ONE PROCESS Clients AI agents over the MCP protocol MCP server protocol, auth and every tool one process, one event loop any blocking call stalls every request Auth and billing service user lookup on every request Databases and model APIs StarRocks, Neo4j, vector search, LLMs AFTER · ROUTERS AND WORKERS gRPC cached auth Clients same tools, same results Router MCP protocol, auth routes each tool call Worker runs the tools add more to scale Auth and billing far fewer lookups Databases, LLMs unchanged
Before, protocol, auth and every tool shared one event loop per pod. After, small routers handle the protocol and cached auth and send tool calls over gRPC to workers, which scale out on their own.
Smaller images, far fewer auth lookups Container image size Old server 10.55 GB New router 0.16 GB New worker 0.59 GB Auth service lookups per test run Old build 39,540 ~28,600 requests New build 934 ~79,000 requests
Image sizes are compressed, from our registry. Auth lookups are the auth service's own request counts over each full scorecard run; the new build served almost 3× the requests with under 3% of the lookups.

Results

Both builds ran on the same test stack on November 3 and 4, 2026, with load stepped every six minutes as a multiple of average production traffic. Old build: today’s production image on four pods. New build: two routers and four workers.

The new build kept latency flat as load grew Old build New build Heavy data query 0.1 s 1 s 10 s 100 s 1×8×16×32×64× load, × average production traffic Old build at 1×: 7.0 sOld build at 8×: 6.8 sOld build at 16×: 11 sOld build at 32×: 38 sOld build at 64×: 103 s 103 s New build at 1×: 7.9 sNew build at 8×: 6.4 sNew build at 16×: 6.7 sNew build at 32×: 6.9 sNew build at 64×: 7.0 s 7.0 s Quick lookup 0.1 s 1 s 10 s 100 s 1×8×16×32×64× load, × average production traffic Old build at 1×: 0.28 sOld build at 8×: 0.29 sOld build at 16×: 0.72 sOld build at 32×: 11 sOld build at 64×: 18 s 18 s New build at 1×: 0.09 sNew build at 8×: 0.09 sNew build at 16×: 0.09 sNew build at 32×: 0.14 sNew build at 64×: 0.26 s 0.26 s
p95 response time by load, log scale. Old build: production image on four pods. New build: two routers and four workers. Same test stack, November 3–4, 2026.
The event loop stopped stalling Old build pod New build worker New build router 0 s 5 s 10 s 15 s 20 s 25 s 1×2×4×8×16×32×64× load, × average production traffic Old build pod at 1×: 0.45 sOld build pod at 2×: 1.2 sOld build pod at 4×: 1.9 sOld build pod at 8×: 1.9 sOld build pod at 16×: 7.5 sOld build pod at 32×: 8.5 sOld build pod at 64×: 22.4 s 22.4 s New build worker at 1×: 0.02 sNew build worker at 2×: 0.15 sNew build worker at 4×: 0.11 sNew build worker at 8×: 0.03 sNew build worker at 16×: 0.33 sNew build worker at 32×: 0.80 sNew build worker at 64×: 1.7 s 1.7 s New build router at 1×: 0.09 sNew build router at 2×: 0.07 sNew build router at 4×: 0.09 sNew build router at 8×: 0.11 sNew build router at 16×: 0.15 sNew build router at 32×: 0.20 sNew build router at 64×: 0.43 s 0.43 s
Seconds per minute each pod's event loop was stalled, from the loop-lag monitor. Old build: four pods. New build: four workers and two routers.
At 64× production trafficOld buildNew build
Heavy data query, p95103 s7.0 s
Quick lookups, p9518–22 s0.26–0.31 s
Tool calls that failed12%1.9%
Loop stall per pod, per minute22.4 s1.7 s

The new build ran with settings tuned for it, such as more worker memory and lighter trace sampling. Above 64×, both builds hit the test environment’s smaller StarRocks warehouse, so 65× is a floor. For the database side, see Prathyush Shankar’s post on materialized views.

In this series

  1. Adding Pods Didn’t Help (this post)
  2. Our Spans Were Lying: tracing under a blocked loop and the GIL (coming soon)
  3. Measuring the Stall: the loop-lag monitor and dependency sweeps (coming soon)
  4. A Fair Fight: load tests you can compare (coming soon)
  5. Off the Loop: the fixes and the caching (coming soon)
  6. Routers and Workers: scaling out, memory and moving the encoder (coming soon)1

Footnotes

  1. Longer term, we are designing a data-processing service in Go with DuckDB, with no GIL. It is a design today, not a result. ↩

  • mcp
  • python
  • asyncio
  • gil
  • performance

Share this post

Written by

Chris Blanton

More from Engineering