OpenAI's low-resource coding results: migrate legacy systems by behavior, not translation

OpenAI's new low-resource programming-language evaluations suggest that current coding models can do substantially more useful work in COBOL, Fortran, Pascal and other languages that are poorly represented in common benchmarks. The practical lesson is not to ask a model to “translate the repository.” It is to define observable behavior as the contract, build characterization tests around that contract, migrate in bounded slices, and require deterministic parity before any production cutover. This is most useful to teams modernizing legacy systems; it is not proof that an autonomous model can safely replace domain experts.

OpenAI published the results on 19 October 2026. The company introduced MultiLangBench, which asks models to build a SQLite-compatible database engine in 20 languages, and a separate COBOL migration evaluation that checks behavior-preserving migrations to C++, Java, Kotlin, Python and Rust. The benchmark figures below are OpenAI-reported results, not independent measurements or production guarantees.

What OpenAI actually measured

MultiLangBench tests whether a model can implement a substantial system in languages including lower-resource ones rather than solve isolated syntax puzzles. OpenAI reports an average score of 87.4% for GPT-6 Astra, compared with 26.8% for GPT-5.6 Sol. It also reports 40.3% for GPT-6 Luna at an estimated API cost about 78% lower than GPT-5.6 Sol on that evaluation.

The COBOL evaluation is more relevant to modernization programs. It checks whether a model can rewrite programs while preserving observable behavior. OpenAI reports 86.6% for GPT-6.1 Sol at low reasoning effort, compared with 5.4% for GPT-5.6 Sol, and 97.9% for GPT-6.1 Sol at higher effort. The company explicitly warns that production modernization still requires expert review and testing against business rules and dependencies.

Those results support two narrow conclusions. Model capability on underrepresented languages has improved, and evaluation must test program behavior rather than the plausibility of generated code. They do not establish that a model understands undocumented accounting policy, production batch schedules, regulatory obligations, operator workarounds or every external file contract around a legacy program.

Make behavior the migration contract

A source-to-source diff is a poor acceptance test because equivalent programs can look very different. A successful migration preserves the observable contract: output records, database mutations, return codes, reports, monetary calculations, ordering rules, error handling and side effects for the same inputs and initial state.

Start by recording behavior before generating replacement code. Build characterization tests from representative production cases after removing sensitive data. Add boundary cases for dates, empty files, maximum field widths, negative values, duplicate records, invalid encodings and interrupted batches. Capture every observable output byte when exact formatting is part of the interface.

COBOL numeric semantics deserve special attention. Fixed-point declarations, truncation, rounding, overflow and intermediate precision can change a financial result even when both implementations compile. Encoding, collating sequence, packed decimal fields, record layouts and end-of-file behavior can also create silent differences. A test should compare the business result and the serialized interface, not merely a value printed during a happy-path run.

Inventory the system before choosing a model

Legacy modernization begins with a dependency map. Identify entry points, copybooks or include files, database schemas, message formats, job-control scripts, schedulers, downstream reports, file transfers and operational recovery procedures. Classify components by consequence: a report formatter is not equivalent to a settlement calculation.

Use a stronger reasoning model for repository comprehension, cross-file dependency analysis and high-risk transformations. A lower-cost model can help inventory modules, propose tests or migrate repetitive low-risk units, but every output still passes the same deterministic gates. Model routing is a cost and latency decision inside the process; it never weakens the acceptance contract.

Version the complete request bundle: model snapshot, reasoning setting, system instructions, relevant source context, tool definitions and generated artifacts. Without that evidence, a team cannot reproduce a defect or compare a later model fairly. The GPT-4 API migration guide covers snapshot and compatibility planning, while the portable AI evaluation contract explains how to keep release criteria stable across providers.

Use a strangler workflow, not a big-bang rewrite

Divide the system by an observable seam: one batch step, report, transaction type or service boundary. Freeze the old program as an oracle for that slice, generate or hand-build the replacement, and run both implementations against the same sanitized input. A comparison service should classify every difference as expected, defect, ambiguous rule or test-fixture error.

A safe pipeline looks like this:

1. inventory dependencies and rank business risk; 2. capture golden inputs, outputs and state transitions; 3. generate tests before replacement code; 4. migrate one bounded component; 5. compile, scan and run deterministic parity tests; 6. execute the old and new paths in shadow mode; 7. review unexplained differences with domain owners; 8. canary a small workload with rollback available; 9. expand only while parity and operational objectives hold.

Keep generated code in ordinary review. Require human approval for changes to money, permissions, retention, external interfaces and recovery logic. Record why a difference was accepted. An undocumented exception that exists only in a chat transcript will become the next hidden legacy rule.

Treat the benchmark as a starting point, not an SLA

MultiLangBench is useful because it expands evaluation beyond popular languages and small functions. But its database-engine task is not the same distribution as a company's codebase. Repository size, proprietary frameworks, missing documentation, test quality and context limits will change outcomes. The reported average also hides language-level variance and failure severity.

Build a private evaluation from real migration units. Score compile success, functional parity, exact-output parity, security findings, reviewer corrections, escaped defects, time to accepted change and total model cost. Segment results by language, subsystem, risk tier and model configuration. A 95% pass rate is meaningless if the remaining 5% contains monetary or authorization defects.

Keep a holdout set that prompt authors cannot see. Re-run it when the model, reasoning effort, prompt, toolchain or compiler changes. A release gate should fail on any prohibited side effect, security regression or unexplained high-risk mismatch even when the aggregate score improves.

Secure the coding-agent boundary

Legacy repositories often contain credentials, customer identifiers, production endpoints and algorithms that are commercially sensitive. Remove secrets before model access, limit repository and network permissions, and run compilers and tests in isolated, disposable environments. Give the agent read access by default and expose writes through reviewed patches.

Tool output is untrusted input. A build log, README or source comment can contain instructions that conflict with the migration policy. The orchestrator must treat repository text as data, keep authorization outside the model and restrict commands by allowlist. For agent-style execution, the OpenAI Agents API architecture guide describes approval, isolation and trace controls.

Do not send regulated or confidential source code to a service until legal, security and data-governance owners approve the processing terms and retention path. Protect evaluation fixtures as production-like data, and avoid copying raw source or customer records into broad observability systems.

When AI-assisted migration is a good fit

Use it when the legacy behavior can be executed, representative inputs exist, outputs are observable, the replacement can be introduced behind a seam, and domain experts can resolve ambiguous cases. Repetitive transformations, test generation, dependency summaries and well-bounded modules are strong starting points.

Do not start with a system whose old environment cannot run, whose critical rules live only in operator memory, or whose side effects cannot be replayed safely. Do not use autonomous migration for a regulated cutover without independent validation. If the business can retire the workflow or replace it with a standard product, code translation may preserve complexity that should disappear.

The most important deliverable is not the generated replacement. It is the executable evidence that the replacement preserves the behavior the business relies on. OpenAI's new evaluations indicate that modern models can participate in more legacy-language work than earlier generations. A production program still succeeds through dependency discovery, characterization tests, risk-based routing, differential execution, expert review and reversible rollout—not through benchmark scores alone.

Prepared by: Noor Yasser