AI Models Achieve Breakthrough Performance in Legacy Languages like COBOL and Fortran

The latest advancements in AI coding models are specifically designed to address the significant challenges posed by low-resource programming languages, such as COBOL, Fortran, and Visual Basic. These languages are crucial because they underpin much of the world's essential infrastructure, even though they lack the vast amounts of public code and documentation available for modern languages like Python or JavaScript. COBOL, for instance, is deeply embedded in global finance, supporting over 40% of online banking systems, 80% of in-person credit card transactions, and a staggering 95% of ATM transactions. Similarly, Fortran remains vital for complex scientific simulations, including weather forecasting and critical nuclear reactor safety analyses. The ability of AI to operate effectively in these legacy systems is paramount for maintaining and modernizing the services that keep global economies running.

This research was directly motivated by feedback from enterprise developers and customers who noted that existing coding models struggle immensely when dealing with the unique syntax, conventions, and limited training data of these older languages. The process of maintaining or migrating software built in these languages is inherently difficult. Teams often face incomplete documentation, the departure of original subject matter experts, and the critical business logic—which might include specific rules for financial calculations or record handling—is often only preserved within the code itself. Whether the goal is simply fixing a bug or undertaking a full migration to a modern stack, the absolute necessity is preserving the precise, intended behavior of the existing system.

To rigorously measure the progress of AI in this domain, the developers introduced two sophisticated evaluations. The first is MultiLangBench, which assesses a model's capability to build a functional, SQLite-compatible database engine from scratch across a diverse set of 20 programming languages. This test is comprehensive, requiring the model to not only write code but also to manage complex state, handle various runtime errors, and integrate new features while strictly preserving all existing functionality. The second evaluation focuses specifically on COBOL migration, challenging the model to translate existing COBOL programs into modern languages like C++, Java, Kotlin, Python, or Rust, all while guaranteeing that the program's original business logic and output remain perfectly intact.

Together, these benchmarks test two complementary and equally vital skills: the ability to build complex, new software systems (MultiLangBench) and the ability to accurately preserve and translate the functionality of decades-old, mission-critical software (COBOL migration). The results demonstrate substantial and measurable improvements across both fronts, achieving higher accuracy while simultaneously becoming more cost-efficient for developers.

Focusing on MultiLangBench, the new models show a dramatic leap in performance. GPT-6 Astra achieved an impressive average test pass rate of 87.4% across all 20 languages, representing a massive jump from the 26.8% recorded by its predecessor, GPT-5.6 Sol. This improvement was not isolated; every single language tested saw a significant boost. For example, in COBOL, the pass rate jumped from a mere 10.9% to 72.1%, and in Visual Basic, it rose from 27.5% to 89.4%. GPT-6 Astra scored above 83% in 19 out of the 20 languages, confirming its broad competency, though COBOL remains the most challenging at 72%. Furthermore, the model demonstrated exceptional capability in OCaml and Fortran, showing increases to 85.8% and 83.5%, respectively.

For those concerned with cost-effectiveness, GPT-6 Luna offers a compelling alternative. While achieving a lower average test pass rate of 40.3% on MultiLangBench, it still significantly outperforms GPT-5.6 Sol in every language tested. Crucially, these gains come with a massive reduction in operational expenditure, estimated at an average API cost of $0.77 per sample, which is approximately 78% lower than the $3.48 cost associated with GPT-5.6 Sol. This positions Luna as a highly viable, cost-optimized solution for enterprise adoption.

Turning to the COBOL migration evaluation, the performance gains are equally dramatic. GPT-6.1 Sol achieved an accuracy of 86.6% at low reasoning effort, a monumental improvement over the 5.4% recorded by GPT-5.6 Sol. This was achieved while costing 39% less. Even when demanding a higher reasoning effort, the accuracy reached an astounding 97.9%, nearing perfect saturation of the evaluation criteria. This demonstrates that the model can handle the most complex, nuanced business logic embedded in the code.

The technical depth of these evaluations is noteworthy. MultiLangBench requires the model to interact with a shared interface—a database file—receiving an initial database state and an SQL query, and then producing a modified database. The reference SQLite implementation is used to verify if the resulting database contains the expected, correct contents, allowing for a standardized test across all languages. The COBOL evaluation highlights the need for deep domain knowledge; COBOL programs frequently handle financial transactions, where specific details like fixed-point arithmetic (which dictates how decimal digits are handled and potentially discarded unless explicitly rounded) and complex missing-record handling are not merely programming details but reflect core business requirements. The overall findings confirm that modern AI models are rapidly becoming highly proficient, accurate, and cost-effective tools for working across a vast spectrum of programming languages, particularly those critical but historically underserved by AI development. The significant performance jump in COBOL migration, from single digits to over 86% accuracy, represents a major breakthrough for the financial and industrial sectors relying on legacy systems. The introduction of MultiLangBench and the COBOL migration test suite provides the developer community with transparent, measurable tools to assess AI capabilities in their specific, often niche, programming environments.