AI Code Translation Works—But Ships the Bugs Too

Here's what happened: the team tested LLMs on actual COBOL-to-Java migrations and found that while the models could handle the structural translation reasonably well, they either propagated existing bugs or introduced new ones at a meaningful rate. This isn't a knock against LLMs; it's evidence of a systematic gap. The models are trained on code that exists in the wild, which includes plenty of buggy code. They learn statistical patterns, not semantic correctness. When translating, they reproduce those patterns faithfully—which sometimes means faithfully reproducing the bugs.

Why should you care? Because there's a $600 billion+ legacy codebase sitting in enterprises right now, and modernization is a recurring pain point. AI-powered code migration tools are inevitable and potentially huge. But this research shows the current generation of "just run it through Claude" approaches are dangerous in production environments.

The market opportunity is real: companies would pay significant money to safely migrate COBOL systems to maintainable languages. But the winners won't be the tools that just translate code fastest—they'll be the ones that add proper validation layers. Think of it like AI-assisted coding but with guardrails: automated migration + static analysis + semantic verification + human review workflows. The friction matters because it catches bugs before they hit production.

This also highlights a broader pattern we're seeing across AI dev tools. Boris Cherny's recent experiment with having Claude rewrite itself showed both the promise and the brittleness of current code generation. The models can do real work, but they fail in subtle ways that require human judgment to catch. That's not a reason to abandon AI for development—it's a reason to build tools that treat AI as powerful but fallible.

For founders building in this space, the lesson is clear: the next wave of value isn't in raw generation speed, it's in confidence. Can you make developers trust AI-generated code enough to ship it? That requires not just good models, but good validation infrastructure. That could mean integration with static analyzers, test generation, type systems, or even formal verification in certain contexts. The teams that crack this—that turn "AI can help migrate code" into "AI can safely migrate code"—will own significant market share.

The other angle worth watching: this research will probably accelerate specialized fine-tuning efforts. Instead of generic LLMs, we'll see models specifically trained on migration tasks with correctness metrics baked in. JetBrains' new KotlinLLM plugin is a hint at this trend—narrower scope, tighter integration, better outcomes.

Bottom line: AI code translation is real and valuable. But it's not a substitute for verification. The real product opportunity isn't the AI—it's the system that makes the AI's output trustworthy.