Tiny Team, Billion-Token Workloads: What Durable and LinkedIn Tell Us About the New Economics of Agentic Apps
胡新宇
发布于 2026-08-25
Two teams, opposite ends of the size axis, made the same bet: stop running your own AI electricity. Durable runs 1.1B tokens/day on six engineers using a managed AI Gateway. LinkedIn ships custom kernels and prompt libraries for 1.3B users. The mid-market is the new battleground — and the wrong infrastructure choice now costs six to eighteen months.
Tiny Team, Billion-Token Workloads: What Durable and LinkedIn Tell Us About the New Economics of Agentic Apps
Durable runs an AI business builder for 3 million customers on six engineers and no DevOps team. LinkedIn rebuilt retrieval for 1.3 billion users with a single LLM-powered model. Same workload category, opposite end of the team-size axis. The interesting question isn't who's right — it's why both teams made the same kind of bet at radically different scales, and what that means for everyone in between.
The Two Teams
Durable's stack serves ~1.1 billion AI tokens per day for a user base that includes a 19-year-old selling chocolate-covered bananas at fairs and a laid-off tech executive launching a porch-decoration business in Texas (Vercel News, 2026). The team: six engineers, no DevOps. They run their entire platform on Vercel — Next.js, Functions, AI SDK, AI Gateway, Workflows, Blob, Firewall, the whole stack — and ship new production agents to customers in a single day.
LinkedIn's Feed team serves predictions in under 50 milliseconds for 1.3 billion users, after ripping out five specialized retrieval systems and replacing them with one LLM-powered retrieval model (ByteByteGo Newsletter, 2026). Thousands of engineers across LinkedIn's broader org, of course, but the Feed project itself was a focused team rebuilding a single critical path. They built a custom Flash Attention variant called GRMIS. They built their own prompt library. They wrote a Generative Recommender with late fusion and an MMoE head.
#AI Agent#AI工程#AI创业#上下文工程
These are not the same kind of team. They are not solving the same kind of problem. But both teams made a structural choice that the AI infrastructure market is now forcing on everyone: stop running your own electricity, or stop pretending you can.
The teams that get this wrong pay in margin. The teams that get it right ship faster than their competitors can refactor.
The line between "team that can ship an agentic product" and "team that can't" is no longer drawn by headcount. It's drawn by infrastructure choice.
What Durable Actually Built
Durable didn't start on managed infrastructure. They started self-hosting, and self-hosting taught them three things they didn't want to learn.
First: model orchestration is a moving target. Every few weeks a new model launches, a provider goes down, or pricing shifts. Self-hosting meant rewriting the orchestration layer every time. That's a tax a six-person team can't afford (Vercel News).
Second: multi-tenant context isolation is existential when you're serving millions of businesses with shared agent infrastructure. "Context spilling" — one customer's workflow leaking into another's — isn't a latency bug. It's a compliance failure. Self-hosting meant building the guardrails themselves.
Third: per-customer AI economics are not optional at scale. If one customer uses 100× more compute than another, you need tenant-level visibility into runs, models, and spend. Without that, you can't price the product correctly. Without that, your margin gets eaten by a power-law distribution you can't even see.
The migration pattern is worth pausing on. Durable's engineering lead described it this way: "We skinned the old product using an iframe, pushed to production on Vercel, then ripped out the self-hosted infrastructure entirely" (Vercel News). No refactor. No big-bang rewrite of the user-facing app. They wrapped the old product, moved the wrapping to the new platform, and pulled the old organs out one at a time once the new blood was flowing.
The "iframe skin, then rip out" pattern is the architectural equivalent of a heart transplant done without stopping the heartbeat.
The result: 3–4× lower infra cost than self-hosting, 10× leverage per engineer, and ship-cadence measured in days instead of quarters. The whole Durable Vercel stack is publicly itemized — Build (Next.js, Functions, Fluid Compute, Turborepo, Flags, Toolbar); AI (AI SDK, AI Gateway, Workflows, Skills); Edge (CDN, Domains, Cron); Security (Firewall, Bot Protection, BotID, Sandbox); Observability (Analytics, Speed Insights, OpenTelemetry); Data (Blob, Streamdown, json-renderer) — and that list is itself a statement. Every line is a managed component. Nothing on it is "we wrote a custom service for this" (Vercel News).
Durable is what happens when a six-person team stops running its own generators.
What LinkedIn Had to Build That Durable Didn't
LinkedIn is the opposite end of the spectrum. They had to build because at hyperscale, every managed primitive eventually breaks under the load.
The first problem was the most counter-intuitive: LLMs don't understand numbers. When LinkedIn's Feed team started putting structured features into prompts, they passed raw counts. A post with 12,345 views became "views:12345" in the text. The model treated those digits as a sequence of text tokens, not a quantity. The correlation between popularity features and embedding similarity came back at -0.004 — essentially zero (ByteByteGo Newsletter, 2026). Popularity is one of the strongest relevance signals in recommendation, and the model was ignoring it.
The fix was deceptively simple. They wrapped the count in a percentile bucket and a special token: <view_percentile>71</view_percentile>. Now the model learned a stable vocabulary for "how popular is this" instead of trying to parse arbitrary digit sequences. The popularity-to-similarity correlation jumped 30× (ByteByteGo). Recall@10 — whether the top 10 retrieved posts are actually relevant — improved 15%.
A managed AI Gateway wouldn't have caught that. That's an epistemic quirk of the model, not an infrastructure problem.
The second problem was training data. LinkedIn initially put every post a member was shown into the interaction history — engaged or not. The assumption was more data equals better model. It didn't. Scrolled-past posts are ambiguous: maybe irrelevant, maybe busy, maybe interesting-but-not-stop-worthy. Including them made the model worse and training more expensive, because transformer compute scales quadratically with sequence length.
Filtering to positively-engaged posts only:
Memory per sequence: −37%
Sequences per training batch: +40%
Training iteration speed: 2.6×
Two hard negatives per member added a further 3.6% recall improvement. Cleaner signal compounds (ByteByteGo Newsletter).
The third problem was serving. A sequential model processing 1,000+ historical interactions through multiple transformer layers is expensive. LinkedIn's solution was shared context batching — compute the user's history representation once, then score all candidates in parallel using custom attention masks. On top of the transformer, a Multi-gate Mixture-of-Experts (MMoE) head routes different engagement predictions (clicks, likes, comments, shares) through specialized gates while sharing the same sequential representations underneath. Late fusion handled the count and affinity features that didn't benefit from full self-attention. The custom Flash Attention variant GRMIS delivered an additional 2× speedup over PyTorch's standard implementation (ByteByteGo Newsletter).
Durable didn't have to do any of that. LinkedIn had to do all of it. Same workload category, different league.
The Asymmetry
This is where the interesting pattern lives. Small teams and large teams are making structurally similar decisions — don't run your own electricity — but they draw the line in different places.
A small team outsources everything they can. Durable's stack table is the proof: every line is a managed component. The team ships product. The platform handles the rest.
A large team is forced to build because the off-the-shelf components break at hyperscale. LinkedIn's GRMIS, prompt library, custom data loader — these aren't vanity engineering. They're the minimum required to make the model behave at LinkedIn's scale and latency budget.
Both are rational. The mistake is assuming one playbook works for both.
The AI infrastructure market is splitting into a managed tier for the SMB edge and a custom-primitives tier for the hyperscaler edge. The middle — mid-market AI startups running 10–100M tokens/day — is the new battleground.
This middle is where most founders actually live. You have a real product. You're spending real money on tokens. You're not LinkedIn-scale, so you can't justify custom kernels, but you're big enough that managed-gateway pricing starts to bite your margin. You have to decide: stay on managed and accept the markup, or commit to building your own serving layer and pay it back in engineering time.
The wrong answer in either direction costs six to eighteen months.
What This Means for Builders Today
Three rules from watching these two teams work.
Rule one: if you're below 100M tokens/day and not latency-bound on GPU kernels, default to a managed AI Gateway and spend the engineering on product. Durable's stack is the proof point. Six engineers, 1.1B tokens/day, ship in days. The infrastructure choice unlocked the team size. The team size unlocked the ship cadence. The ship cadence unlocked the customer growth.
Rule two: if you're above 1B tokens/day or have unique latency constraints, plan to build two custom components — your numerical feature encoder and your inference serving layer. LinkedIn's percentile-bucket trick and GRMIS are the proof points. You can ride managed gateways for orchestration, observability, and failover. You cannot ride them for the model's epistemic quirks (numerical understanding) or for the serving math (attention kernels at scale). Build those two things first; rent everything else.
Rule three: the math on "rewrite vs. migrate" has flipped. "Everyone tells you not to rewrite your product," Khan from Durable said. "We did exactly that, and it worked" (Vercel News). The iframe-skin pattern made the rewrite survivable. Managed infrastructure made it cheap. The old wisdom — never rewrite, only refactor — assumed infrastructure migration was a quarter-long distraction. When the migration is a deployment and the rewrite is funded by token-spend savings, the calculation changes.
Conclusion
The teams that win the next eighteen months won't be the ones with the most GPUs.
They'll be the ones who correctly decided which layer of the AI stack to own and which to rent.
Durable owns the product and rents everything else. LinkedIn owns the parts that break at hyperscale and rents everything else. Both teams are right. Both teams shipped. The middle-market teams that try to copy LinkedIn's custom-everything playbook will burn their margin on engineering they didn't need. The teams that copy Durable's rent-everything playbook at hyperscale will hit the percentile-bucket problem and the Flash Attention problem at the same time, with no one to call.
