From 5 to 87 percent: AI can now translate COBOL. What it means for your legacy systems

TL;DR. On October 19, OpenAI published two evaluations of old programming languages. When porting COBOL programs to modern languages, its new model GPT-6.1 Sol gets 86.6% of them right without a single error, its predecessor 5.4%. With more compute it reaches 97.9%. For the first time, AI is a serious tool for legacy code in COBOL, Fortran and Visual Basic. But an independent test from the same day shows the limit: for whole programs, a completely error-free port is still rare. If you run legacy systems, don't start by choosing the AI. Start by preparing the tests.

What OpenAI measured

OpenAI calls them low-resource languages. For COBOL, Fortran or Visual Basic there is far less code and documentation online than for Python, and coding models have struggled with them so far. Yet banks, insurers and weather services run on them. According to IBM, which OpenAI cites, COBOL handles 95% of ATM transactions.

The first of the two evaluations tests a migration. A model has to port COBOL programs to Java, Python, C++, Kotlin or Rust without changing how they behave. Grading uses hidden tests, and the output has to match byte for byte. A migration only counts if every test passes.

At the same setting, GPT-5.6 Sol reaches 5.4% and the new GPT-6.1 Sol 86.6%, at 39% lower cost. With higher reasoning effort the score rises to 97.9%. According to OpenAI, that is close to the maximum the evaluation allows.

The second evaluation is called MultiLangBench. Here the model has to build a database in a given language that behaves like SQLite, in a long coding session without internet access. GPT-6 Astra passes 87.4% of the tests on average across 20 languages, GPT-5.6 Sol only 26.8%. COBOL remains the hardest.

Why old code is so treacherous

OpenAI gives the best example itself. COBOL truncates extra decimal places in fixed-point numbers unless the programmer explicitly asked for rounding. A rewrite in Java or Python that rounds instead runs without errors and still calculates differently.

We worked out what that adds up to: monthly interest at 3.5% a year on one million amounts between €100 and €10,100, in one-cent steps. €1,000.00 earns €2.9166… in interest. COBOL books €2.91, the rewrite €2.92. Across all amounts, exactly every second booking is off by one cent, which adds up to €5,000 a month.

A test that only checks whether the new program runs will not find this error. A comparison with the old outputs finds it immediately.

The fine print

• The vendor grades itself. OpenAI built the tasks and the grading. The COBOL programs are not published, and OpenAI does not say how many there are or how large they are.
• The open test is well documented. MultiLangBench is freely available on GitHub. Of 24,000 converted SQLite tests, only 272 are meant to run for scoring. And the model gets the complete SQLite documentation. That is exactly what legacy code often lacks.
• Your language may be missing. The 20 languages include COBOL, Fortran and Visual Basic. ABAP, RPG for IBM i systems, PL/I and Delphi are not among them.
• OpenAI says it itself: production software still needs expert review and testing against your own business rules.

The cross-check: whole programs

On the same day, Vals AI, an independent evaluation firm, updated its code migration benchmark. There, models port 30 real open-source programs into four other languages each. That makes 120 migrations, each checked with 100 to 400 hidden behaviour tests. The programs are written in modern languages. A COBOL part is included, but Vals does not publish a separate score for it.

The best models pass roughly four in five tests. Claude Sonnet 5.5 leads with 86.9%, GPT-6 Astra reaches 80.6%. With the best model, almost every migration passes at least half of its tests. Passing all of them stays rare even for the best model.

The gap to OpenAI's 97.9% is not a contradiction. It shows that "error-free" depends on how large a program is and how many tests check it. The bigger the system, the more certain it is that some remainder is left for a human to find.

What this means for a mid-sized company

Many businesses run code that nobody likes to touch anymore: the inventory system on an IBM i machine, a costing tool in Visual Basic, a calculation program in Fortran from the engineering department. The rules inside it are often written down nowhere else, and the colleague who knew them is about to retire. Until now, porting was so expensive that it kept getting postponed. That is changing. AI is getting good at reading and translating, and the bottleneck is moving from programming to checking. Whoever collects the old inputs and outputs today will have the checker tomorrow that OpenAI also needed for its 97.9%.

What you should do now

1. Take stock. Which programs run in which language, who still knows them, what depends on them? One page is enough.
2. Secure the old outputs. Collect real inputs and the results that go with them: lists, receipts, exports. Those are your tests. Without them, no migration can be checked, no matter who does it.
3. Start small. Have the AI port one self-contained program, such as a nightly batch run, and compare the result byte for byte with the old one.
4. Write down the special rules. Rounding, truncation, empty fields, date boundaries: the business side confirms what is correct before the AI decides it.
5. Check your own language. If it is not in the vendors' tests, their numbers do not apply to you. Then measure with a small test of your own first.

For a long time, old code was a question of the people who can still read it. With the new models, it becomes a question of tests. If you would like to go through which of your legacy systems is a good fit, we are happy to talk.