3K Tokens/Sec Changes LLM Economics Overnight

Why this matters: inference speed directly determines whether your LLM application makes economic sense. Every millisecond of latency is either a better user experience or a cheaper infrastructure bill. Most founders have been stuck choosing between expensive A100 clusters that barely hit these speeds or settling for degraded performance on cheaper hardware. This breakthrough suggests a third path exists.

The real implication is simpler than the technical details: the infrastructure moat around LLM applications is eroding. When you can run capable models at scale on commodity hardware at reasonable speeds, you shift competition away from "who can afford the biggest GPUs" and toward "who can build something users actually want." That's healthier for the ecosystem and more favorable for the 99% of founders who aren't backed by mega-VCs.

Connecting the dots: we're seeing this pattern across multiple angles this week. Liquid AI's new 8B mixture-of-experts model demonstrates that smaller, efficiently-trained models can punch above their weight class—you don't need 70B parameters to ship something production-ready. Meanwhile, Tiny-vLLM is pushing the same philosophy into open-source tooling, making it easier for founders to optimize inference without proprietary frameworks.

The data generation problem is also cracking open in interesting ways. Shift's model—where robots generate training data through real-world work rather than expensive human annotation—is a clever inversion of the typical ML scaling problem. Instead of "how do we generate data cheaply," they're asking "how do we make data generation profitable." That's the kind of unit-economics thinking that separates sustainable AI companies from money-burning research labs.

On the application side, Boston Children's Hospital unlocking 40+ rare disease diagnoses through AI shows where the real value lives: problems with high stakes and high complexity where AI can genuinely reduce human error or latency. That's not flashy, but it's durable. Healthcare systems will pay for tools that solve diagnosis bottlenecks. That's different from consumer use cases where AI sometimes feels like feature creep.

One cautionary note: the CAPTCHA research shouldn't surprise anyone, but it's a useful reminder that security theater doesn't scale. If you're building systems that need to distinguish humans from AI agents, traditional friction isn't the answer. You're going to need behavioral or cryptographic solutions that actually scale.

The through-line here is commoditization. Inference speed improving, model efficiency improving, open-source tooling improving—these aren't independent trends. They're the natural arc of any technology moving from frontier research to infrastructure. For founders, that means the window for competing on infrastructure is closing. The founders winning in 2025 will be the ones who treat inference as a solved problem and focus on what you can build on top of it.