Challenges and Paths Towards AI for Software Engineering

Correspondence: papers@team.qeios.com — Qeios will forward to the authors

Naman Jain, Wen-Ding Li, and Manish Shetty equally contributed to this work.

1. Introduction

AI for software engineering has made remarkable progress recently, becoming a notable success within generative AI. Despite this, there are still many challenges that need to be addressed before automated software engineering reaches its full potential. With additional efforts, it should be possible to reach high levels of automation where humans can focus on the critical decisions of what to build and how to balance difficult tradeoffs while most routine development effort is automated away. Reaching this level of automation, however, will require substantial research and engineering efforts across academia and industry. This paper provides an opinionated view of the tasks, challenges, and promising directions towards achieving this goal.

Many existing surveys overlap with the topics that are discussed in this paper.^[1] and^[2] survey the successes and challenges of AI programming assistants,^[3] survey using LLMs for software testing, and^[4] survey using LLMs in low-resource and domain-specific languages, and^[5] focus on automated program repair, both with and without LLMs. Finally,^[6] is a roadmap for formal mathematical reasoning and has some overlap with our discussion on software verification.

In addition, many papers discuss the current state, challenges, and future of AI for software engineering^{[7][8][9][10][11][12][13][14]}. Our work draws inspiration from them, and we recommend that the reader consult with them for alternative perspectives.

In this paper, our goal is threefold. In Sec. 2, we provide a structured taxonomy of concrete tasks in AI for software engineering. In particular, we emphasize that there are many other tasks in software engineering beyond code generation and code completion, encouraging research in these areas. We provide three measures for categorizing concrete realizations of each task: the scale of the problem, the logical complexity, and the level of human intervention.

Moving forward to Sec. 3, we highlight nine challenges in the field that today’s models face, each cross-cutting and applicable to several tasks. In Sec. 4, we posit a set of promising research directions to tackle the challenges above, with Fig. 1 summarizing which directions correspond to each challenge. We hope that through our paper, the reader can appreciate the progress the field has made, understand the shortcomings of today’s state-of-the-art models, and take inspiration from our suggested future ideas for tackling these challenges.

2. Tasks in AI Software Engineering

We first provide a taxonomy of tasks in AI software engineering. To provide a structured way to consider concrete realizations of each task, we define three measures that apply across them: scope, logical complexity, and level of human intervention. To achieve an AI software engineer, we strive for AI to be capable across the board for all three measures.

Scope Measure: We define three levels of scope, the extent of changes that the AI makes to the codebase. Function-level scope refers to single, self-contained functions such as in HumanEval^[15] and MBPP^[16]. Self-contained unit scope goes beyond singular functions and to larger chunks of code such as entire files and classes, such as FullStackBench^[17] and BigCodeBench^[18]. Finally, project-level scope refers to larger codebases such as entire repositories, such as in Commit0^[19] and SWE-Bench^[20].

Logical Complexity Measure: Tasks require a wide range of reasoning abilities when it comes to devising algorithms to solve them. Low logical complexity tasks require little to no reasoning, such as writing CRUD (create, read, update, delete) applications or using APIs. Medium logical complexity tasks include most LeetCode problems, finding inputs to fuzz simple programs, and reasoning about execution behavior of multithreaded programs. High logical complexity tasks require meticulous and challenging levels of algorithmic and logical reasoning, either because the algorithm is complex or because the problem requires clever thinking and insights. This includes difficult competition programming problems, writing large thread-safe concurrent programs, cracking cryptographic ciphers, and solving SMT-like problems. Many popular coding benchmarks are function-level, medium-high logical complexity, such as APPS^[21], CodeContests^[22], and LiveCodeBench^[23].

Level of Human Intervention Measure: AI coding is a collaborative task.^[24] categorize interactions between developers and AI. Each interaction progresses through four phases: the trigger for the interaction, the AI response describing the system’s output, the developer response capturing how developers react to the AI response, and the output of the interaction, the exact result. They characterize these developer-AI interactions into eleven types, including autocomplete code suggestions, conversational assistance (e.g., asking a question about a codebase), selection-based enhancements (e.g., refactoring a selected chunk of code), comment-guided prompts (e.g., natural language to code), check correctness, and more.

We map these interactions to the autonomy taxonomy outlined in^[25]1 to define three levels of human intervention. We distill their six levels of autonomy into three levels: low (No AI and AI as a Tool), medium (AI as a Consultant and AI as a Collaborator), and high (AI as an Expert and AI as an Agent). Low autonomy is when the human has full control over the task and uses AI to automate simple sub-tasks. This might look like writing a codebase with tests while leaving small function-level snippets for the AI to fill in. Medium autonomy is when there is a similar amount of human-AI collaboration, with interactive coordination of goals and tasks. Here, the human and AI might both suggest refactorings, optimizations during the development cycle. High autonomy is when AI drives the interaction and tasks, identifying required changes and the changing demands of the user. The AI would defer to the human only when needed or for a final check, write the code and tests autonomously.

Next, with our taxonomy of measures in place, we turn to the set of tasks that are reflective of the tasks and capabilities of a human software engineer. We give a brief description of each task in this section, deferring a more extensive survey to Appendix A.

2.1. Code Generation

Code generation is the task of generating code from a specification. In code completion, the specification takes the form of a preexisting code snippet, and the goal is to complete the snippet. The most popular form of code completion is tab completion, where the user can press the tab key to complete a block of code (e.g. GitHub Copilot). Tab completion is often done at line-level or function-level scopes but needs to be fast to provide users with a seamless experience. Another paradigm is natural language to code, where the specification is a natural language description with requirements such as the task description, input-output examples, or libraries to use.

Recently, AI-driven IDEs, such as Cursor Composer and Codeium’s Windsurf Editor, have blurred the lines between the two paradigms. With the ultimate goal of decreasing the burden of human programmers, they aim to automatically infer the user’s intent from the code context and user behavior (e.g. keystrokes, user edits, file navigation patterns). However, when intent is vague, they allow users to specify desired functionality via chat interfaces. Depending on scope and logical complexity, code generation can vary highly in difficulty. Reliable code generation in large codebases is still a challenge for state-of-the-art AI systems today.

2.2. Code Transformation

2.2.1. Code Refactoring

In code refactoring, the goal is to take a working implementation of a piece of software and rewrite parts of it while maintaining correctness. One challenge with this task is that success extends beyond functional correctness or metrics. The goal is often to improve code maintainability, readability, or extensibility—qualities that can be inherently difficult to quantify and highly context-dependent.

For instance, extracting shared functionality into helper methods presents trade-offs between modularity and cognitive complexity^[26]. While there are no hard rules for when to extract functionality, one heuristic adopted by software engineers is the rule of three (“three strikes and you refactor”)–abstractions should only be used when a piece of code has been duplicated thrice. Because it can often be unclear what level of abstraction refactorings should be done at, completing a refactoring at a high autonomy level is also difficult. These challenges are further compounded by the need to understand implicit trade-offs customized to specific codebases, respect conventions, and reason about the long-term maintenance implications of structural changes. While code refactoring often has a low logical complexity, it can be laborious in practice due to scope, as seemingly small refactors can propagate across the entire codebase.

2.2.2. Code Migration and Translation

An incredibly resource-intensive (time and manual effort) task frequently affecting companies is migrating large amounts of code while preserving all the original functionality and semantics. Such high-value migrations present opportunities for AI-assisted automation to reduce cost and manual effort. Code migration often has a very high scope (many files and systems affected alongside their interdependencies) and high logical complexity (semantic depth of required transformations, constructs in different languages may be different). Current solutions may excel at migrations with high scope but modest logical demands (API migrations, type conversions) but struggle with changes across component boundaries^[27].

A special case of code migration is code translation (transpilation): rewriting code from a source language to a target language. In industry, this task can be motivated by several reasons, such as security and scalability concerns in legacy languages, avoiding the technical debt a project has accumulated over the years, and improving the performance of a codebase. Due to the safety-critical and cross-system nature of many migrations, this task often requires substantial human oversight in practice and cannot be done fully autonomously.

2.2.3. Code Optimization

Transforming programs to improve performance characteristics while maintaining functional correctness is a critical software task. Optimizing real-world systems poses significant challenges due to the large scope and high logical complexity of the task, as performance bottlenecks must be identified and new algorithms to mitigate them must be proposed. Code optimization often has a large search and solution space with competing objectives like speed, memory efficiency, and readability, for example when optimizing kernel code at the PTX level for GPU-based AI model optimization^[36][37]. In many scenarios, high levels of autonomy may not be desirable, as tradeoffs can depend heavily on external factors such as hardware, and the best optimizations may ultimately affect readability.

2.3. Software Testing and Program Analysis

In the process of software development, there will inevitably be bugs. The difficulty of detecting these bugs can vary depending on their scope and logical complexity. For LLMs, minor typos or correctness bugs (small scope, low logical complexity) are easier to spot^[39] while complex concurrency bugs and security vulnerabilities (large scope, high logical complexity) can be tricky because they can be hidden deep in the call stack, contain subtle logic errors, or be hard to isolate due to the large scope^[40].

2.3.1. Software Testing

Software testing is a practical approach to prevent bugs, both during development and production. There are several popular approaches to software testing, some short-term and others longer-term. Unit testing refers to using input-output style tests that exercise the functionality of a piece of code. Property-based testing is based on formal specifications and relies on specifying test cases that ensure that known properties of the code hold. Mutation testing modifies a program subtly and ensures that the test suite can detect errors in these mutations. Fuzzing refers to executing programs with unexpected inputs and monitoring for exceptions, usually over a more extended time period.

The goal of software testing is to design tests that can surface bugs reliably. This is evaluated through metrics such as code coverage–how much of the source code is executed when the test suite is run. An alternate to code coverage is mutation score, where mutants are generated, and the score is defined as the percentage of mutants causing the suite to fail. While practical, software testing faces challenges such as the scalability limits of traditional tools and the difficulty of manually designing tests with good coverage. As LLMs continue to improve at coding, they present a promising avenue for automatically generating high-quality tests.

2.3.2. Program Analysis

While testing catches bugs, the most challenging software issues are security vulnerabilities and zero-day exploits, from memory corruption to privilege escalation. This requires a deep program understanding, that testing/fuzzing often misses. For instance, a zero-day is a vulnerability unknown to the software developers that is found by an attacker, and there is no patch available from the vendor. In such cases, the only practical approach is offensive security research, manual source code audits, and root cause analysis of prior vulnerabilities to harden codebases.

Another example of a valuable but challenging analysis is invariant detection. A program invariant is a property of a piece of code that is guaranteed to be true at a specified program point, no matter what the input is. A simple example is that after the line int x = c * c; is executed, x must be nonnegative. Identifying invariants in a program can be useful when testing, debugging, and modifying code. This task can be challenging because it requires reasoning abstractly about code execution across many different potential inputs and execution paths to determine what properties must hold for all possible inputs.

2.3.3. Program Repair

Bug localization is a significant challenge in program repair, as pinpointing the exact site of a bug can be challenging, especially in large codebases. Issues like out-of-memory accesses often manifest themselves further downstream, making it difficult to identify the root cause. Once the bug is localized, the next step is to repair the bug. LLMs can be an ideal tool for this because they have seen a wide variety of bugs during training. Function-level, low-logical complexity bugs can often be easily fixed by feeding back error information to the model. It can be tricker to perform repair in larger scopes (e.g. repositories) where the code has higher logical complexity. This can often require several steps, including designing and implementing new algorithms or making complex refactorings across multiple files.

2.4. Software Maintenance

2.4.1. Code Documentation and Summarization

To ensure maintainability, readability, and ease of collaboration, code must be well documented. Good documentation needs to be written cleanly and crisply, describing what the function does and how the function works. It must also anticipate and address any misunderstandings that a programmer might have, such as potential side effects or special cases. Humans often see documentation as a chore and neglect it, leading to code and documentation frequently being out of sync. This has led to the concept of “self-documenting code”, code that clearly conveys its purpose. As documentation is generally a task that has a low logical complexity and does not require too much human intervention, LLMs can help ensure that documentation is a continuously updated artifact in sync with the code.

2.4.2. Pull Request (PR) Review

Reviewing pull requests is an integral aspect of the software development cycle. While the most essential requirement for PRs is that a new feature is implemented correctly, other important considerations include checking whether the repository’s style conventions are satisfied, ensuring that the PR does not introduce any new bugs, verifying that program invariants and guarantees still hold, and inspecting whether tests are robust. Generally, reviewing PRs is a task requiring low logical complexity and can be automated relatively easily.

2.4.3. Code Understanding, Navigation, and Question Answering

When encountering a codebase for the first time, developers often find it challenging to understand and develop a good mental model of the code. This can be due to many reasons: too many wrapper functions, excessive error-handling boilerplate, deep call stacks, or poor code cleanliness. One important challenge in code understanding is code navigation: finding where relevant functionality is implemented. Doing this well requires understanding the high-level layout of where every functionality lies in the codebase and the low-level understanding of which helper functions are used to implement each functionality.

Another challenge is code question answering: answering complex questions about a codebase, which requires sophisticated code understanding and reasoning abilities. Models should not hallucinate or give incorrect information that skews a developer’s mental model of the code. Beyond other tasks mentioned in this section, developers might commonly ask questions related to data flow (when and where data structures get mutated), code functionality (whether there are any side effects), performance characteristics (determining the runtime and memory complexity of a function), or error handling (whether certain corner cases are handled).

2.5. Scaffolding and Meta-Code

For a software system to work, the core logic must be written well, but that is not enough. Many infrastructural aspects must be in place to support the software. We group these into two main categories: scaffolding and meta-code. We define scaffolding as a task outside of the code that must be done to get the software running properly. Examples of scaffolding include setting up Google authentication, subscribing to APIs, managing file storage, and generating API tokens. In contrast, we define meta-code to be code that is important to make the system work but does not actually participate in the execution of its main logic. Examples of meta-code include test harnesses, configuration files, CI/CD code, Makefiles, Dockerfiles, sandbox databases, and preprocessors. Scaffolding and meta-code often are small in scope and have low logical complexity but can require a lot of domain-specific knowledge about the application, requiring human intervention.

Infrastructure-as-code and Security. A particularly challenging case is generating Infrastructure-as-code such as Terraform, where you specify the type of infrastructure specifications (such as AWS EC2 instances, Kubernetes clusters, S3 buckets, VPC buckets) as code and execute it to create the infrastructure. When generating such code, LLMs struggle with security configurations due to the complex interplay between service-level permissions (e.g., AWS resource access), resource-level permissions (e.g., specific allowed actions), and provider-specific security primitives like IAM roles, security groups, and network access controls.

2.6. Formal Verification

The task of formal verification involves generating checkable, mechanized proofs that can guarantee that a piece of code works as intended. There are two major classes of formal verification: full functional verification (FFV) and property verification (PV). In FFV, the goal is to design a complete and precise formal specification that captures the desired behavior of the implementation, such as fully verified data structures (mutable lists, trees, graphs, hash tables)^[46]. The main challenge in full functional verification is in correctly writing the specification so that all desired properties are specified. FFV generally has a high scope and medium logical complexity, as the properties to verify are often straightforward to write once the correct abstractions are found.

While FFV provides a complete set of guarantees, it is usually sufficient to opt for PV, where a few key properties of a system are proven correct. Examples include: ensuring that two threads do not simultaneously enter a critical section of a program, verifying that a complex program will always terminate, proving the absence of security vulnerabilities like buffer overflows, and guaranteeing memory safety. One challenge that makes PV difficult to use in practice is the issue of false positives, where functionally correct code often does not pass property checks. A prime example is Rust: while the powerful type system enforces many desired guarantees, code with correct semantics often does not pass type checks. Another challenge is that many standalone tools for PV are often semantics-dependent, which can make them hard to maintain as language semantics change.

While formal verification tools have begun to see adoption in industry, they has not yet become mainstream because of these challenges. Code LLMs could greatly ease this burden and make it more feasible to verify code at larger scales, especially verifying properties requiring lower logical complexity.

3. Challenges

While the field of AI for code has made fruitful progress, cutting-edge AI still struggles with SWE tasks, especially at larger scopes and higher levels of logical complexity. Next, we discuss ten key challenges in AI for code. Each challenge spans multiple tasks, and progress on any can lead to improvements on many tasks at once.

3.1. Evaluation and Benchmarks

Our taxonomy of tasks and measures highlights some of the shortcomings of today’s evaluations and benchmarks. For example, the majority of today’s coding evaluations have no level of human intervention, with a few, such as Copilot-Arena^[50], having low to medium autonomy. HumanEval, MBPP, APPS, CodeContests, and LiveCodeBench are all at function-level scope, with low to medium-high logical complexity. Commit0^[19], SWE-Bench^[20], TestGenEval^[51], RefactorBench^[52], SWE-Lancer^[53] are at project-level scope with low to medium logical complexity.

Task Diversity and Capability Isolation: Current coding evaluations primarily focus on the code generation task, while most of the tasks discussed in Section 2 are either not studied such as Code QA or only studied in limited scopes like EvalPerf^[54], vulnerability detection^[55], formal verification^[56]. As more agent-based approaches are introduced for software engineering (e.g. pairing a code generation model with a debugging model), these engineering-related capabilities beyond just code generation will be important towards designing a maximally performant system. Solely relying on end-to-end coding evaluations that focus on the overall correctness of a codebase makes it difficult to precisely measure progress and learn from the failure modes on individual tasks.

Contamination: Data contamination is a serious issue that, if not taken into account, can affect the soundness of various conclusions drawn from a set of benchmark results. In coding, the performance of LLMs on competitive programming^[57][23] and SWE-Bench^[58] tasks has been shown to degrade over time, indicating the possibility of older problems being contaminated due to public exposure on the internet. For simpler function-level HumanEval style problems,^[59] suggest three potential causes of contamination: direct data leakage (benchmarks are on GitHub), synthetic data leakage (there are only a limited number of interview problems), and overfitting to test sets (benchmark hacking). In addition, for code, contamination can be hard to detect, as semantically equivalent code that is syntactically distinct could be thought of as contamination^[60]. A recent benchmark, the Konwinski Prize², is a promising way to fairly evaluate SoTA LLM models by only using new GitHub issues.

Construct Validity: Construct validity refers to how closely a measurement reflects the underlying concept. Given the implications of rapid performance improvement in the AI for the code domain, it is essential to have high-construct validity benchmarks evaluating how well programming agents can perform. While benchmarks like SWE-Bench come close, user experiences do not currently match rapid performance gains obtained from them. This is partially because many desiderata in software engineering cannot be described cleanly via automated unit testing. Things like multi-turn code generation, designing an UI, and writing clean and idiomatic code are all difficult to quantitatively measure with precision. Designing reliable proxy metrics for these desired goals remains a challenge.

3.2. Effective Tool Usage

Software engineering has witnessed the development of various open and proprietary tooling support for programming, debugging, analysis, and code management over time. For example, program analysis tools provide static and dynamic assurances on code correctness. Print statements and debuggers are used for dynamically analyzing and debugging programs at a fine-grained level. Beyond programming, such tools are richly integrated into the entire software development lifecycle, e.g., code navigation or search, reviewing code, CI testing.

There have been efforts combining LLMs with tools such as calculators and search engines^[61][62]. However, effective integration of LLMs with software engineering tools is a more challenging problem. Several early works have incorporated such tool feedback in code generation in an automated fashion, for example, linter or execution feedback in^[63][64][65]. However, these works do not actively interact with tools. More recently, programming agents have started incorporating tool use within their workflows termed as Agent-Computer-Interface^[66]. These tools range from aiding in general search (grep), providing code editor for making changes^[67][68], language server for static analysis^[69], dependency analyzer^[70], terminal access for bash commands including code execution^[66], debugger^[71].

Dynamic and Effective Tool Usage: While many efforts combine LLMs and agents with tools, they do not achieve fully dynamic and effective software engineering tool usage. This involves an AI system seamlessly and proactively integrating appropriate tools depending on the task at hand. There are a few challenges towards achieving this goal. First, the AI system must identify which tools could potentially be useful for the task at hand. Second, the system then needs to decide when to invoke the tool. A complex debugging task might require the use of pdb or gdb to track intermediate program states, while looking at input-output pairs may be sufficient for simple debugging tasks. Third, the agent then must figure out how to invoke the tool. If the agent knows that a certain function in a program has an error, it may wish to walk through only that function instead of the entire code from start to finish. Finally, the agent needs to incorporate the output provided by the tool in order to inform its next steps, e.g. edit the code if a bug was uncovered or run the tool again otherwise.

3.3. Human-AI Collaboration

While AI coding systems are increasingly more powerful, the majority of them are still at a low to medium autonomy level, serving as engineer assistance rather than achieving high or full automation. We identify a few key challenges of today’s AI coding systems that prevent these systems from working with humans effectively at higher levels of autonomy.

Vague Specifications and User Misalignment: When using code LLMs or coding agents, we typically prompt them with a natural language specification. This can include a natural language description of the desired code, input-output examples, relevant code snippets, and other functional requirements. However, there is a gap in the level of abstraction between English and code, leading to incomplete or ambiguous specifications. This issue becomes more pronounced in longer programs, where the number of ambiguous decision points increases, and choices traditionally made by humans are instead implicitly embedded in the LLM’s generated code. Consequently, users often experience misalignment due to vague specifications. While many code LLMs support multi-turn interactions, it remains inherently challenging for users to articulate their thought processes into follow-up natural language instructions.

Specifications beyond text: While today’s specifications predominantly rely on text, there are many domains for which pure text is insufficient as a specification. In domains like robotics, virtual reality, embedded devices, and user interfaces, specifications often need to be multi-modal (e.g. showing the model a picture of an UI to create) and world-interfacing (e.g. providing simulation code describing a robot will interact with its environment).

Inherent trade-offs in software development: Designing large software systems always surfaces trade-offs between various desiderata such as readability, scalability, performance, maintainability, reliability, security, etc. These trade-offs are often context-dependent. A long-term and rapidly moving project may be willing to trade off some performance to have simplicity and readability. Performance-critical applications may completely sacrifice readability to eke out every millisecond of speed (such as using bit-twiddling hacks). Finding the sweet spot among these trade-offs can often involve extensive prototyping and benchmarking to understand the performance characteristics of different approaches. However, user specifications in the initial prompt rarely include details about these trade-offs, nor do models often take them into account.

Implicit constraints: Aside from functional/semantic correctness, there are also often implicit constraints in writing code. For example, many companies such as Jane Street and Google have style guides, and many GitHub repositories explicitly outline style elements that new code ought to follow. ^[73] find that GitHub pull requests that are more consistent with the style of the existing code get merged faster. Additionally, corporations may enforces codes of conduct or compliance requirements at the code level. Furthermore, codebases follow common programming patterns or system design patterns that are implicitly specified by the way the current code is written. However, when using code LLMs, these constraints are often inferred incorrectly^[74].

Lack of Controllability: When using AI coding systems, programmers often seek specific functionality, yet they lack reliable ways to steer LLMs toward generating precisely the desired code. Instead, they typically rely on a trial-and-error approach, repeatedly sampling outputs or providing feedback until the AI produces an acceptable solution. Consequently, significant human effort is still required to review and modify the code to ensure it meets the intended functionality^[75].

A way to improve controllability is for AI coding systems to recognize when human input is needed and communicate effectively—yet this remains the top-reported challenge in human-agent collaboration^[76]. LLMs rarely defer to humans for clarification, while developers often ask questions to clarify the description of a task provided by their peers. For example, when a product manager refines a requirements document, developers who are unclear about the scope or specifications ask questions and leave comments, which the manager resolves iteratively to disambiguate requirements^[77]. Based on its knowledge of existing software, AI should be able to incorporate implicit priors from a specification while keeping the user in the loop. For instance, when designing an academic website, certain expectations—such as including a list of publications and contact information—are implicit. However, whether to include a person’s GPA would require explicit clarification.

Restricted Human-AI Interface: Existing interfaces for code LLMs primarily manifest as intelligence features embedded within integrated development environments (IDEs).^[24] establishes a taxonomy of developer-AI tool interactions, emphasizing low-level support mechanisms such as auto-complete suggestions, selection-based enhancements, and conversational assistance within the codebase context. While this taxonomy comprehensively covers existing AI coding systems that function primarily as engineering assistants, its applicability becomes questionable as these systems advance toward higher levels of automation. For instance, the ubiquitous “Tab” interaction paradigm prevalent in intelligent IDEs may prove inadequate when AI systems transition from completing developer-scaffolded functions to authoring the majority of the codebase autonomously.

Current interfaces for coding agents, such as Devin, typically stream raw actions without adequate context or explanation. Given that these agents can execute numerous actions rapidly, developers face significant challenges in effectively monitoring the process, implementing timely interventions, or reasserting control when necessary. This lack of transparency can also undermine trust in AI-generated code^[78]. While human-AI interface design has received extensive attention in autonomous vehicle research^[79][80], similar consideration for AI coding systems remains notably absent.

3.4. Long-Horizon Code Planning

When working on large projects, engineers and tech leads often make complex decisions about how to design and structure the code to best support the various functionalities that will eventually be needed. To build a long-lasting software system, an engineer must know the potential paths that the system’s evolution might take. This requires domain expertise and experience in how different code structures require different forms of extension. We believe that today’s language models are unable to perform this level of sophisticated planning.

Designing Good Abstractions: One instance of long-horizon code planning is choosing the right abstractions from the outset. An API designed with good abstractions will allow new features to be implemented seamlessly with minimal user overhead, while an API designed with poor abstractions may lead to excessive code duplication, refactoring, or even debugging. We discuss two examples of this, library learning and data representation.

Library learning: Designing APIs and libraries are designed with useful abstractions often leads to more code reuse and more intuitive interfaces. The challenge of library learning is to derive a library of useful abstractions from a corpus of programs by abstracting out common reusable features^[81][82][83]. While the traditional library learning literature has focused primarily on code reuse, a truly effective library must also prioritize ease of use and maintainability, as well as be robust and adaptable to future extensions.

Data representation: The choice between data structures leads to a variety of trade-offs when it comes to performance aspects such as memory usage and processing speed. For example, database engineers need to decide between various data models, storage formats, and indexing methods to balance performance.

Modularity and Code Quality: LLMs are trained and optimized primarily for code correctness with insufficient focus on other aspects of code like quality and maintainability. This is further exacerbated with large scale reinforcement learning being performed using test cases which can lead to unintended consequences regarding code quality, as correct but poor quality code is still often given a high reward. Empirically, it has been observed that LLM written solutions are often more complex than human-written counterparts. For example,^[86] identified that LLMs prefer to repeat existing code instead of making use of abstractions in the existing code. One aspect of code quality is modularity, ensuring that code does not get duplicated too often. Here,^[87] identified that library or tool reuse is non-trivial for LLMs in coding and formal math domains.

3.5. Large Scope and Long Contexts

Large Scopes: At the repository level, the tasks in Sec. 2 become significantly more difficult and require many steps. In code generation, user alignment can be an issue because there are many decision points and tradeoffs that can compound. In code refactoring, modifications will touch multiple parts of the codebase, and it can be tricky to keep the repository consistent. In code debugging, functions can be large and bugs can be nested deeply within stacks of function calls. In code navigation, because there are so many functions interacting in various ways, it can be difficult to know where each piece of functionality is implemented and how the code is pieced together.

Another issue with large scopes is large context lengths. Software engineering often requires dealing with very large codebases–for example, Google has repositories with over a billion lines of code^[88]. As this is far too large for modern-day LLMs, choosing the correct context to include when using LLMs is important.

Limits of Retrieval-Augmented Generation (RAG): Retrieval-based algorithms have been the predominant way to deal with long-context coding issues. First, the retriever retrieves relevant functions. Then, the generator leverages the retrieval to improve generation. While RAG has proven effective in many NLP tasks such as question answering^[89][90], the code domain provides new challenges for these methods.

Retrieval: In most NLP tasks, the retrieval step can be done relatively well because keywords that are in the query are often keywords that need to be retrieved. Unlike answering NL questions, writing code often requires drawing inspiration from code snippets that may be completely different syntactically. This can include programs with similar semantics, algorithms, or API calls, all of which potentially have very little in common when it comes to syntax. For example, the implementation of Dijkstra’s algorithm in a GPS navigation application can guide the implementation of a shortest-path algorithm in a social media application. Because retrievers often rely on syntactic matching, these relevant programs can be hard to retrieve^[91][92].

When deciding what to retrieve, it is also necessary to have a sufficient awareness of other parts of the codebase so that you know which building blocks are necessary to construct the new function.

This can make the retrieval task relatively tricky, as shown by failure modes on two benchmarks, CodeRAGBench^[93] and BRIGHT^[94].

Generation: In NLP tasks, the generation step often is a straightforward application of the retrieved information. However, in code, writing a new function requires more than copy and paste. This is closely tied to the problem of code reuse: piecing together relevant snippets of code in a precise and productive way to fit the current context. Depending on what is retrieved, each piece of retrieved content provides different information. This can include information about the language’s syntax, documentation about the API, clues about the algorithm to be written, or examples of similar functionality being written.^[96] find that even when the oracle context is retrieved, LLMs tend to misuse it, highlighting a lack of semantic understanding, which we discuss in the next section.

3.6. Semantic Understanding of Codebases

A global and holistic semantic understanding of a codebase is important for performing almost all code-related tasks. For example, let’s say an engineer is asked to improve the runtime performance of a query. To do so, they must first understand the codebase’s structure well enough to know where all the pieces of the algorithm are implemented. Then, they need to understand the algorithm and implementation in detail. This includes both the high-level algorithm (including its time complexity) and the low-level implementation details to identify both algorithmic and implementation bottlenecks. Finally, after coming up with a solution, they must then return to their understanding of the global code structure so that they can integrate their new algorithm without introducing new bugs.

LLMs struggle at semantic understanding of codebases for several reasons. First, the way that code is pieced together can be relatively intricate, and understanding all these complex relationships can be difficult. Second, code can often have units with high logical complexity that contain custom algorithms that may never have appeared anywhere in the training data. Finally, because a disproportionately large number of LLM training tokens are spent on code generation rather than other coding tasks, models may lack a holistic awareness and world model of code.

One desiderata is that models can generalize knowledge across various coding tasks^[97]. However, this may not be straightforward as just training on more tasks:^[98] found that coding models fine-tuned on additional natural language/code pairs saw significant improvements on code generation but did not transfer to improving code understanding and execution reasoning. While there have been successful efforts to augment code LLM training with execution information to improve general coding capabilities^[99][100], imbuing code LLMs with a general and holistic understanding of code remains an important challenge today.

3.7. Low-Resource Languages and Specialized Libraries

As we adapt code LLMs to individual codebases, generating correct code in out of distribution (OOD) scenarios becomes crucial. Much of software development in business contexts revolves around proprietary codebases, which is a distribution shift from the open-source code that dominates LLM training data^[101]. These OOD scenarios include domain-specific languages (DSLs), custom internal libraries, low-resource APIs, and company-specific coding styles and conventions.

Syntactic Failures: Models have been shown to hallucinate constructs from higher resource languages when working in low-resource languages.^[102] remark that ”contemporary LLMs fail to follow Hazel’s syntax and semantics, often borrowing syntactic forms and library functions from [higher-resource languages like] OCaml and Elm“.

Poor Semantic Understanding: In low-resource languages, models have less exposure to the various language constructs. Therefore, they have a weaker semantic understanding of the language. Many studies reveal that code LLMs perform poorly when asked to write code in low-resource languages. Due to the lack of training data in these OOD domains, models may struggle to write common primitives or piece together functionality coherently. On HumanEval, Qwen 2.5 Coder Instruct (32B)^[103] has an accuracy of 83% in Python but only 27% in D.³

Library Usage Failures: In OOD scenarios, LLMs lack awareness of the libraries and functions available for use. In new codebases using custom libraries, many functions appear only a few times, providing limited training data for AI models to learn their usage. This scarcity can lead to overfitting, where models fail to recognize an effective use-case of these functions. Models also frequently hallucinate non-existent functions based on patterns that it infers.

3.8. Library and API Version Updates

Continual learning, the idea of training an AI system to take in new information continually, has been a long-standing challenge in AI and NLP^[104][105]. In software engineering, codebases are continuously changing as new features are supported and awkward design patterns are reworked. While backwards compatibility is often prioritized in software design, it inevitably becomes broken as codebases evolve further. Therefore, programming libraries have version releases, each release supporting and deprecating features in the last version.

There have been a few works exposing this issue. For example, CodeUpdateArena^[106] and GitChameleon^[107] are two benchmarks exploring the ability of LLMs to write version-specific code, examining this issue at the function and file level. They find that language models struggle to adapt to these changes even with this limited scope. In theorem proving (Lean),^[108] try to mitigate this by developing a lifelong learning framework that continuously learns and uses new theorems. In real-world engineering, the challenge of library and API versioning generally spans across an entire repository, as everything must be kept consistent. To our knowledge, there are no techniques that successfully deal with this challenge at such a large scale. This problem is difficult for a few reasons, which we discuss below.

Version Identification: In order to to successfully deal with version changes, a LLM must first identify which version of each library is being used in a codebase. This may often be quite difficult, because versioning information can be hidden deeply within a codebase. Sometimes, it can be found in comments or configuration files, but in the worst case, it must be inferred from the library calls being used. To make things worse, some code may be compatible across multiple versions, while other code will cause errors only in specific versions. Therefore, the model will often require a deep understanding of both the codebase and the nuances between different versions in order to infer the version at hand.

Version Adaptation: Many fast-changing libraries are not backward compatible as older features become deprecated. It can be difficult for LLMs to implicitly keep track of which constructs and patterns are associated with each version. Therefore, consistently using constructs from the right version can be difficult. As we will see in the examples below, LLMs often write code that mixes and matches API constructs from different versions of the same library.

Continuous Adaptation to Paradigms, Features, and APIs: New styles, patterns, and paradigms are often introduced to replace older, more cumbersome ways to write code. For example, React came out with its Hooks paradigm in version 16.8 (2019). Over the next few years, developers transitioned from the old class components paradigm to using hooks, as hooks made code cleaner and more maintainable. Only in early 2023, with the launch of react.dev, were Hooks the default paradigm in the documentation. For language models, incorporating these features can take a long time, because code in these new paradigms are initially completely absent in the training data and inherently in the low-resource regime. In^[109], the authors find that LLMs fail to utilize security features in compiler and toolkit updates (such as in Java 17), still relying on legacy methods such as insecure random number generation. While it is possible to use retrieved examples and documentation in order to coerce language models to write code using new and updated features, we should strive to create AI coding assistants that can quickly internalize new changes and be able to naturally incorporate new features and paradigms, even without an abundance of training data. For each task, the language model should be able to reason about the best way to write the code, independently of the number of occurrences seen in the training data.

3.9. High Logical Complexity and OOD Domains

Some programming tasks are challenging for even the best human programmers, requiring approaches with a very high logical complexity. Examples of tasks that fall into this category include superoptimizing programs, discovering attacks for purportedly secure code, writing performant compilers, optimizing GPU kernels^[37], and writing very error-prone and very technical code.

Limits of Symbolic Techniques: When it comes to applying symbolic techniques to these tasks, there are a few limiting factors that make them difficult to tackle. First, for synthesis-style tasks, the search space can be very large. Deductive and rewrite-based synthesis techniques are unable to explore a majority of the search space. Second, verifiers can be limited in power, such as when dealing with properties in concurrency or weak memory models. Third, many domains lack clean models to reason about properties, such as dealing with memory bandwidth in GPU kernels.

Because they are hard for humans, these tasks are very rarely in the training data of today’s language models. They have unique, domain-specific, challenges that making generalizing from existing data difficult. For these problems, language models rely heavily on feedback-driven search algorithms^[112], and it can be difficult to navigate the search space effectively. In addition, many of these tasks lack feedback mechanisms, which is crucial for AI to pick up learning signals. When designing a complex algorithm or data structure, it is often hard to know if you are on the right track until you get to the correct result. When writing code for a large multithreaded operation, it may be hard to know if the algorithm has concurrency issues until all the parts are fully fleshed out. Without feedback, incremental improvement is nearly impossible.

4. Paths Forward

4.1. Data Collection

4.1.1. Automatic Data Curation

Augmenting Data with Program Information: One challenge in enabling LLMs to develop a world model of code is that programs are often treated like text: as tokens with no semantic information. However, modern programming tools allow us to extract rich semantic and structural information about code. By leveraging these tools, we can augment training datasets with detailed annotations describing various properties of programs. We hypothesize that this augmentation will significantly improve a model’s understanding of code, leading to better generalization and stronger coding capabilities. Information can include:

• Static analysis: the syntactic structure of a program (abstract syntax trees, control flow graphs), information about the type of each variable, data flow analysis (reachability, liveness analysis)
• Program instrumentation: memory consumption, runtime analysis, aliasing, and code coverage (like statement or branch coverage)
• Dynamic analysis: program states at various points in the program, call stacks, dynamically curated properties (often relies on instrumentation)
• Formal verification: concurrency analysis, program invariants, loop invariants, memory safety

There have been a few examples of this in the literature: ^[37] leverage profiler feedback to improve GPU kernel generation, ^{[100][116][99]} incorporate execution trace information, ^[117] train with program invariants, GraphCodeBERT ^[118] incorporate data flow information, and ^[119] train on a dataset of performance-improving edits.

High-quality, Verifiable Synthetic Data: The advantage of code is it is possible to achieve strong, verifiable feedback with test cases, program execution engines, and other symbolic tools. This makes high-quality synthetic data generation viable, as it is possible to generate a large batch of data and filter out low-quality samples. For example, to generate code with interesting program invariants, we can sample a large batch of programs, run an invariant detection engine, and retain only programs with interesting invariants. While synthetic data in code has mostly been at the function-level scope, there are no fundamental bottlenecks to expanding to larger scopes. As code is quite compositional, individual building blocks can be combined to generate complex synthetic data at the repository-level scope, which can be very helpful in both training and evaluation.

While the importance of having high-quality data vs. high quantities of data is debated, using verified data has proven to be useful. For example, ^[120] shows that simply removing bugs in existing datasets such as TACO ^[121] can lead to significant boosts. KodCode ^[57] also showed that fine-tuning on verified synthetic data also leads to significant improvements. However, these works work with programs at the function-level scope with low to medium logical complexity, and we imagine that general SWE abilities can improve with synthetic data across scopes and logical complexities.

In DSLs, where programs can be cleanly described with semantics and rewrite rules, one can symbolically generate programs with desired properties via sampling, drawing on enumeration techniques from program synthesis^[122]. This technique has been successfully applied to make considerable progress in difficult reasoning tasks such as ARC-AGI^[123] and math olympiad problems^{[124][125][126]}.

4.1.2. Human-Centric Data Curation

Below, we list three classes of human-annotated data that would be invaluable for the next generation of coding LLMs.

Fine-Grained Data of the Developmental Process: Many code LMs are trained on datasets such as the Stack^[127][128], consisting of trillions of tokens sourced from GitHub. However, training on raw GitHub tokens omits many crucial human signals in the process of software development. For example, companies such as Google rely on internally captured logs of high-quality SWE data. This includes ”fine-grained code edits, build outcomes, edits to resolve build issues, code copy-paste actions, fixes of pasted code, code reviews, edits to fix reviewer issues, and change submissions to a repository“^[129]. Similarly, Meta and GitHub Copilot use telemetry with their AI coding assistants to track and leverage signals from AI-generated code^[130][131]. These tools, along with coding IDEs like Cursor, could provide a treasure trove of reward data for RL-based methods. With direct access to the full history and evolution of a codebase, they can track which suggestions are adopted over time. However, collecting data from human usage also raises critical privacy and intellectual property concerns.

Data for Diverse SWE Tasks: Most of today’s code LLM training recipes still focus primarily on code generation because large-scale datasets are mostly in a continuous, tokenized format. However, as described in Sec. 2), there are many tasks involved in software engineering which models lack exposure to. Training on a broader set of tasks would also incentivize models to learn general capabilities of programs beyond just generation (e.g. a better understanding of program semantics). As initial evidence, ^[132] find that training models on input-output prediction data leads to consistent improvements on reasoning tasks.

The lack of high-quality data on these tasks makes it hard to train on them. It can also be hard to automatically curate them on GitHub. For example, for code refactoring (Sec. 2.2.1), we need paired repositories before and after refactoring, ideally with the refactoring changes described. While some signal such as commit messages and version releases can be used, many repositories lack clean commit histories and releases conflate many features at once. Therefore, to mitigate this, we envision large community-based efforts curating task-specific data on these diverse challenges.

Human-Centric Data: Code LLMs are typically trained and evaluated on carefully curated datasets with clear instructions and verifiable test cases. However, as discussed in Sec. 3.3, these models are often deployed in real-world scenarios where users provide vague specifications or incomplete requirements in their queries. Collecting human-centric data that reflects real-world model usage is a promising approach to bridging the gap between model development and deployment. Recent efforts, such as Copilot Arena^[50] and WebDev Arena, have explored gamified arenas to gather data on human preferences, offering an alternative to purposefully curated datasets. However, such data collection methods may introduce noise, and arena-style approaches are not well-suited for long-horizon, interactive tasks. One potential approach is to leverage existing coding tools and environments, such as developing plugins for GitHub Copilot^[133] or open-source IDEs, to capture real-world interactions. Unlike static datasets, human-centric data can also be collected encompassing diverse interaction modalities, such as users providing sketches to AI coding systems for web development^[134]. As AI coding systems continue to emerge and evolve, launching data initiatives focused on human-centric SWE data is also a crucial direction for advancing human-AI collaboration in software development.

4.2. Training

4.2.1. Environment Design for Code RL

Collecting executable codebases: In recent months, RLVR has seen success in solving algorithmic programming problems through DeepSeek-R1^[135] and OpenAI o1. Recently, on SWE-Bench, SWE-RL^[136] use RL on a rule-based reward to improve performance on SWE-Bench. We find it promising to continue scaling the RL approach to problems collected from real-world software engineering repositories. Towards this, we believe that collecting execution-assisted gym-like reinforcement-learning environments will lead to further performance improvements. These environments can be used further to improve reasoning skills, environment-interaction capabilities and tool usage.

Several prior works^{[86][137][138][139]} curate executable environments for programming agents by supporting CI/heuristic-based repository installations. However, these works are at a relatively small scale and limited in scope, offering only a few thousand tasks from a maximum of a thousand repositories and more importantly, limited to the Python language. Scaling this up significantly requires solving several research and engineering problems. First, installing arbitrary repositories from Github, even using CI is challenging and we require smarter solutions potentially involving LLM-based installation agents. Next, setting up execution infrastructure would require storing installed repository images in something akin to docker for efficient storage and fast container startup times^[140]. Notably, combined docker images can grow massively large and often grow at hundreds of gigabytes even at a modest scale of a few hundred repositories. They require engineering support for efficient storage and serving of such images.

Sourcing task prompts and rewards: Beyond environments, performing large-scale reinforcement learning would require collecting diverse challenging problems with an appropriate way to compute the rewards. These task prompts can be collected from Github^[137] or generated synthetically from problems on Github. Moreover, assuming access to many executable repositories, we can source various end-to-end problems for tasks beyond bug-fixing such as optimization, fuzzing, etc. Access to pre-existing or generated test cases allows for measuring correctness and providing rewards.

However, we envision many practical challenges to remain. For example, longer-horizon tasks are usually more ambiguous and approaches may require multi-turn interactions beyond autonomous coding agents. This would pose a considerable challenge during reinforcement learning where ambiguity resolution might need to be modeled in the reinforcement learning process itself. We elaborate on human collaboration further in Section 4.2.3. Reward hacking^[141] poses another challenge as we build more real-world coding challenges. Test cases often suffer from coverage issues and can grade correct solutions as incorrect. For example, ^[142][143] identified that models attempt to bypass or cheat against the testing harness when optimized using reinforcement learning.

Rewards without execution: As setting up execution environments can lead to considerable overhead, another potential strategy is to use proxy metrics and trained language models to judge correctness. This was common in the pre-LLM era, researchers often used BLEU/CodeBLEU^[144][145] and BERTScore/CodeBERTScore^[146][147] to assess correctness of text and code. In code, semantic and structural properties can be used to improve similarity metrics. Two examples of this are Dolos^[148], an AST-aware plagiarism detector, and difflib.SequenceMatcher, which can be used to compute the similarity between two patches^[136][149]. Beyond rule-based rewards, LLMs-as-a-judge approaches can also be used as reward functions, possibly in conjunction with other execution-based or execution-free approaches.

4.2.2. Adapting to Specialized and Quickly Changing Codebases

Test-time training (TTT) to custom codebases: TTT is the recent paradigm of adapting to a specific problem instance by training on a narrow set of in-distribution examples^[150][151]. This can be used when working in a low-resource context, for example training on a specific codebase, new domain, or unseen API. One challenge in this setting is customizing the model to the particular codebase while retaining general coding knowledge, potentially by using algorithms that can induce controllable forgetting^[104]. To get data in specialized contexts, we envision two mitigation strategies: generating synthetic data and collecting trajectories. In-distribution synthetic data can be generated in large quantities and then filtered and annotated with symbolic (e.g. compiler) information to gain a more global understanding of the current environment and setting. To gather agentic trajectories, we can keep track of previous model attempts and failures to learn from past successes and avoid making repeated mistakes. This will steer the model closer to the desired distribution–for example, to generate code in the specified version of libraries being used in the current context.

Keeping an information bank of code information: For library and versioning issues, retrieval (Sec. 4.3.1) can be very effective for preventing hallucinations of wrong versions of libraries, which can inherently lead to better synthetic data and agentic trajectories. During the TTT process, we can also keep a large growing memory bank of code, documentation, synthetic code, and agentic trajectories in the specialized context. Retrieving from the memory bank would improve the success of generating code, which can then be augmented to the memory bank, and so on, continuously increasing the amount of data and knowledge available.

Prompt and prefix tuning for specialized code contexts: One issue that makes it difficult to continuously keep up with library updates is that doing full finetuning every time something changes is very expensive. Because only a small amount of knowledge needs to be learned compared to that of the pre-trained model, we believe less expensive approaches such as prompt tuning^[152] and prefix-tuning^[153] could suffice. Both these methods append a set of learned task-specific vectors to the input sequence in order to model a specified context, though prompt tuning only modifies the input and prefix-tuning modifies the input at each layer. These methods have also been shown to have good OOD performance, and we believe they present a promising approach to dealing with multiple library versions. A separate prompt/prefix can be trained for each version and then applied according to the context. When an API has new updates, the prompt/prefix can then be cheaply re-tuned to reflect the new updates without undergoing full fine-tuning. This approach also applies to adhering to specific coding styles, where codebase-specific prompts/prefixes can also be learned.

Learning on the fly: When humans are faced with a task they have never seen before, they are often able to draw from past experiences and quickly adapt and generalize to the new domain. This is one of the big unsolved challenges of today’s LLMs: given an OOD coding task, how can models get up to speed and productively work on the task with few samples? On toy domains, an example of this is DreamCoder^[81], a system that learns to solve problems by writing programs and automatically discovering domain concepts and composing concepts together. Designing such approaches for more practical applications is an exciting research direction that will have drastic implications for coding and reasoning.

4.2.3. Training Code LLMs to Collaborate with Humans

Learning to Leverage Specifications Beyond Natural Language: As discussed in Section 3.3, while natural language prompts offer intuitive and flexible ways to express requirements, they often suffer from ambiguity and incompleteness. One direction to address this limitation is to train models to leverage enhanced specifications with more precise and verifiable representations, such as formal specifications and test-based specifications.

Formal specifications: To mitigate underspecification issues, one solution is to develop systems that can translate user intent into formal specifications^[154][155]. While current autoformalization approaches face challenges in accurately capturing user intent (see example below), we envision next-generation systems that will iteratively refine formal specifications through interactive verification with human feedback. These systems would present intermediate formalizations in accessible notation, enabling non-expert users to verify correctness before code generation.

Tests as specifications: Another approach to specify software behavior is through tests. These range from input-output examples and assertions to property-based tests. However, in practice, hand-crafted test suites are often incomplete, failing to capture the full intended behavior, particularly edge cases. This can lead to misalignment, where AI-generated code passes tests but does not genuinely meet functional requirements, potentially misleading users. Moving forward, a direction is training models to generate high-quality test cases based on the user’s initial query, ensuring more comprehensive specification coverage.

Learning to Quantify Uncertainty and Communicate Proactively: As AI coding systems are increasingly deployed to complex software engineering tasks, they encounter more ambiguous and uncertain scenarios compared to traditional benchmarks for coding models. Ideally, in such situations, these systems should proactively communicate with users to clarify tasks and acknowledge its own limitations rather than becoming stuck in endless failure loops or generating buggy code. A key challenge is enabling models to distinguish between well-specified and ambiguous instructions while quantifying uncertainty in a robust manner. While early studies, such as^[157] and the example below, demonstrate that interactive LLMs can improve performance through clarification-seeking behavior, current models still struggle with uncertainty estimation. Equipping models with the ability to quantify uncertainty will likely require incorporating corresponding reasoning data into the post-training stage.

Besides uncertainty quantification,^[158] identify communication as a primary challenge in human-agent collaboration, highlighting the need for improving models’ proactive communication capability. Current models often fail to ask meaningful questions when user input is ambiguous or insufficient, and they struggle to provide progress updates or verify plans in interactive settings. Enhancing models’ proactive communication abilities requires innovative approaches to reward behaviors that yield benefits over multiple steps. Since communication with users does not immediately resolve the task at hand but may improve long-term outcomes, effective strategies must account for delayed rewards in training.

4.3. Inference Time Approaches

4.3.1. Semantic-Aware Embeddings and Retrieval

Semantic and execution aware code embeddings: When training LLMs, code is often treated as pure tokens (just like text) rather than explicitly incorporating code-specific information such as program execution and semantics. As a result, code that is close in embedding space is more often syntactically similar than semantically similar^[92][159], and there are few reliable methods today to retrieve semantically similar code. However, before the LLM era, there were a variety of efforts to incorporate code properties when training embeddings. For example,^[160] train neural modules to represent program operations, leading to compositional program representations that encode the semantics of the underlying programming language. Many other works^{[161][162][163]} attempt to learn execution-aware latent representations for partial and full programs, taking semantics into account.

We speculate that incorporating these techniques to train models to have better and more semantically aware representations may lead to models with a more general understanding of code (Sec. 3.6). For example, if correct and buggy programs could hypothetically be separated in embedding space, then models could be steered away from the incorrect program space. While such a clean separation might not be possible, we believe that training embeddings to have interesting semantic properties is worth exploring.

Better retrieval-augmented code generation: When retrieval-augmented language models were first introduced, they often relied on training the retriever and language model jointly, as in FiD^[164], RETRO^[165], and Atlas^[166]. As language models increased in size, the field shifted to a black-box setting^[167], where the retrieval module is tuned independently to adapt to the pretrained black-box LLM. This setting is much more cost-effective, but the language model is not explicitly trained on how to use its retrievals.

The black-box setting is ideal for challenges such as low-resource languages or specialized contexts. In these situations, the model has not seen enough training data to fully grasp the context, and the challenge is often syntactic rather than algorithmic. For example, when adapting to a domain or a codebase where the relevant API functionality or code style, retrievals can be very instructive. When using APIs with multiple versions, providing retrievals in the correct version can inform the model of how to use the API. When writing code in a completely new language, showing examples of for loops and while statements will teach the model the syntax of these constructs. Retrievals should be diverse and given in multiple forms, including documentation, function definitions of APIs that are used, and example use cases of target functions.

In many other cases, however, we believe that a black-box setting is insufficient. As described in Sec. 3.5, there are two challenges: 1) knowing what to retrieve and 2) using the retrieval. The first challenge relies on retrieving relevant examples, both syntactically and semantically. We believe that having more semantically aware embeddings, as mentioned above, will drastically improve this. For example, embeddings can be trained contrastively to minimize the distance between semantically similar programs. Another potential direction is to consider a diverse set of potential retrievals and then train the retriever to prefer samples that help during generation, as in Atlas^[166].

The second challenge, using the retrieval, is a code reuse task, which requires complex reasoning and code understanding. Algorithms provided in retrievals may often need to be modified and adapted significantly to adapt to the current setting. An example of this might be writing a C++ version of a shortest path algorithm when the retrieval is a Java version, a translation task that models may not have been trained for explicitly. Long chunks of retrieved documentation may need to be understood precisely so that correct hyperparameters and flags can be used. Yet, in a black-box setting, models have not been explicitly trained to leverage this information. Therefore, just as training on incorrect-correct code pairs can improve program repair, we believe that direct training can be very beneficial for code reuse and retrieval-augmented generation. Execution information could also be useful, as code reuse often requires understanding the situation well enough to identify subtle differences between the context of the retrieved code and the current context.

Retrieving via code navigation on the fly: Standard retrieval-augmented methods keep a large retrieval index containing millions of embeddings, which can require a high one-time cost to create. As the codebase evolves, these embeddings may also need to be continuously updated. Instead of keeping track of embeddings, another approach is to find retrievals on the fly by navigating the codebase. We can imagine an agent that learns to use command line functions such as cd, ls, and grep, as well as IDE functions such as jumping to function definitions or finding all references of a function. Static analysis tools can also be paired with the agent to improve code navigation, such as providing the abstract syntax tree (AST) or file structure of a codebase.

4.3.2. Integration with SWE Development Frameworks

Incorporating AI into the CI/CD process: In continuous integration and continuous deployment (CI/CD), automated pipelines are the backbone for building, testing, and deploying code changes. CI/CD accelerates feedback cycles and minimizes integration issues. AI offers several integration points within CI/CD. AI-powered code review tools can be incorporated into CI pipelines to automatically identify and flag style violations, potential security vulnerabilities, and code smells before human reviewers are involved. Furthermore, AI can provide intelligent deployment risk assessments. By analyzing code changes, test outcomes, and historical deployment data, AI can predict the likelihood of deployment issues, informing decisions about whether to proceed with automated deployment or mandate manual verification steps. Finally, AI can automate the generation of release notes by summarizing commit messages, issue tracker data, and relevant code modifications within the CI/CD process.

Steering away from software anti-patterns: In software engineering, certain anti-patterns frequently lead to bugs. For example, common weakness enumeration (CWE) is a categorization of software and hardware weaknesses often leading to vulnerabilities. Because publicly available GitHub code often contains code with anti-patterns, bugs, and CWE vulnerabilities, LLMs often write code susceptible to these issues^[168][169]. We hypothesize that explicitly steering models against these vulnerabilities will lead to more secure and correct code. One way to do this is to collect a large number of program samples violating each CWE (either synthetically or on GitHub) and then use these samples as negative signal during further supervised fine-tuning or RL stages.

4.3.3. Incorporating SWE Tools

Learning to use SWE Tools: As mentioned in Sec. 3.2, we believe SWE agents should understand the intricacies of programming tools and be able to autonomously invoke them as needed. There are three skills to learn: which tool to use, how to use the tool, and how to incorporate the results of the tool. Similar to how models learn to play complicated games, we believe that intelligent tool integration can be learned through repeated interactions with the tool in a RL-style manner. One way we envision this is as follows: first, the interface of the tool must be precisely specified. Next, data containing repeated interactions from the tool (with varying degrees of success) should be collected. Finally, multiple rounds of RL and expert iteration can be done to improve understanding of the tool and learn from misuses.

Evidence that learning higher-level strategies might be possible is that through test-time techniques, OpenAI’s o3 model learned to write brute-force solutions to verify the correctness of more complicated solutions^[170]. We envision that after learning to use tools, AI coding agents can autonomously invoke tools as needed to improve its overall world model of the code and hence its software engineering capabilities.

Neurosymbolic Approaches: Code is a unique domain because there is a vast body of techniques from programming languages (PL) research to build off of, but the majority of AI for code research today does not leverage the symbolic properties of code. Some of these PL techniques are as follows: abstract interpretation^[171] is a technique to compute over-approximations of program state in order to prove the soundness of program properties at points in the code. Concolic testing^[172][173] finds bugs in software by combining concrete and symbolic execution. Model checking^[174] is a way to prove properties of execution traces via temporal logic. Linting and type-checking^[175] provide a static check to ensure that variables, expressions, and functions adhere to a programming language’s rules. Finally, many other program analysis algorithms leveraging these tools have been designed to prevent bugs and ensure code correctness properties.

Traditional PL approaches have a few common shortcomings, which overlap with some of the issues mentioned in Sec. 2.6. First, they often require very complete and precise specifications. Many tools need to have specifications for all library functions, need to specialize to a precise version of the language, and need to specialize to the build system. Second, there is often a high computational cost due to the large search space. Third, there can be many false positives due to the limitations of the tool. We believe that deeply integrating these symbolic tools with LLMs can partially mitigate these challenges.

We provide a few examples of this potential integration. When generating code, program analysis techniques could be applied on shorter snippets of AI-generated code to surface potential bugs or prove properties of the generated code. To improve general code understanding, LLMs can be trained with information about program structure such as abstract syntax trees^[176]. When debugging a large codebase, when the scale is too large to directly apply PL techniques, AI could be first used to narrow down potentially problematic sections of the code which are then handed off to PL tools for debugging. During code generation in DSLs, LLMs can leverage the grammar of the programming language to do constrained decoding^{[177][178][179]} to mitigate syntactic errors. During code refactoring, abstract interpretation and static analysis can be used to identify whether new errors have been introduced and preemptively cut off unpromising search paths.

Deductive Synthesis and Intermediate Languages: Early program synthesis relied on deductive synthesis approaches^[180], where programmers would write a clean simple implementation and then apply transformation rules to convert it into a more efficient one. The appeal of deductive approaches is that because these rewrite rules are semantics preserving, there is a correct-by-construction guarantee. One success story of deductive synthesis is Spiral^[181], a DSL for signal processing kernels that takes advantage of domain-specific transformation rules to produce implementations beating expert hand-crafted code. Another example is Halide^[182], a DSL for high-performance image and array processing code. Due to the difficulty of writing optimized code, humans generally opt for writing code in these intermediate DSLs, and we find it promising for LLMs to do the same.

4.3.4. Scaffolding Human Supervision

At inference time, most machine-generated code will be presented to humans in a format shaped by the human-AI interface design. Since AI may be responsible for generating the majority of the code within a human-AI team, it is important to ensure human control and oversight. By scaffolding human supervision with techniques like summarization and interactive verification, we could potentially improve trust in AI-generated code.

Challenges addressed: 3.3 Once code LLMs are deployed for inference, it is crucial to scaffold human supervision of AI-generated code. This goes beyond merely enhancing the accuracy of AI-generated code, as humans often still need to make the final decision on whether to accept the code or understand it for future integration and maintenance. A study on Github Copilot usage^[184] revealed that programmers tend to allocate less visual attention to AI-generated code. While one solution is to train humans to better identify issues in AI-generated code^[185], a more desirable approach is to design AI systems that scaffold human supervision, reducing their cognitive load when reviewing generated code.

One way to achieve this is by enriching AI-generated content with additional contextual information. Modern LLM chatbots now routinely generate text with citations for knowledge-intensive queries. In Collaborative STORM^[186], researchers demonstrated that dynamically presenting hierarchical “mind maps” alongside the actual collected information significantly enhanced human-AI collaboration, particularly in long sessions. In software engineering specifically,^[187] highlighted the benefits of high-quality source code summarization in aiding software developers in understanding and maintaining machine-generated code. Second, interactive approaches can also enhance supervision. One example is Live Programming^[188], a continuous display of the runtime values of a program, as a means of lowering the cost of validating AI-generated code. However, these existing studies are largely limited to specific programming languages and small codebases. Finally, improving the readability and interpretability of AI-generated code itself presents a promising direction. For example,^[189] showed that modeling program synthesis as rational communication improved end-user interpretation and subsequent communication of code. Expanding on these ideas, future research should prioritize human interpretability in the design and optimization of AI coding systems, fostering greater trust and control in AI-assisted software development.

5. Limitations

We identify a few limitations below:

Speculative nature of future work: The ideas we list in the future work section are opinionated directions we believe have a high chance of success. Many draw upon insights from related work in the literature, but many lack strong and concrete evidence. We encourage further research validating or disproving the effectiveness of these ideas.

Limited scope of future work: We also do not include any novel moonshot ideas, and many of the directions we propose have their roots in existing code LLM literature. Our future work section is also relatively general and applies holistically to AI for code. However, the field has many tasks and challenges that can benefit from using domain-specific knowledge and insights, and we do not touch on these. Finally, this paper is written by people primarily in the academic community, who may not know the details of cutting-edge methods employed in frontier industry labs. We cater this paper towards areas we have more expertise in, and thus leave out many promising directions such as novel architectures.

Focus towards code-specific challenges: In this paper, we mostly focus on code-specific challenges and techniques. However, there are many techniques that apply to general LLM reasoning and development that could be directly applied to code. We believe many of these methods can be used in synergy with code-specific techniques.

Quickly changing nature of the field: The field of LLM for software engineering is progressing very rapidly, with new innovations released weekly. It is possible that a reader reading this paper a few months down the line will find that several of the mentioned challenges will have been partially or entirely resolved.

6. Conclusion

In this position paper, we have identified key tasks at the heart of AI for software engineering as well as a set of three measures to classify different realizations of these tasks. We have also highlighted critical cross-cutting challenges that permeate throughout many tasks. Finally, to drive progress in AI for code, we’ve pinpointed a set of exciting and promising research directions for alleviating these challenges and advancing AI towards being a more capable software engineer. We hope this work provides valuable insights about the current landscape of AI for software engineering and encourages future research in these directions. By building on these insights, we are optimistic that the community can work toward developing AI-driven solutions that better support software engineers in real-world settings.

Appendix A. Survey of Related Work: Tasks in AI Software Engineering

In this section, we briefly survey some of the relevant works for each of the tasks we mention in Sec. 2. These works are by no means complete, and we encourage the reader to check out the survey works mentioned in the introduction and in this section for further references.

A.1. Code Generation

Code Completion: Completion typically happens in conjunction with live programming or within an IDE, helping developers write code faster by suggesting relevant continuations. Traditional code completion systems rely heavily on syntactic and type-aware models (e.g., AST-based models), but recent advances leverage LLMs trained on code corpora to offer semantically rich and context-aware suggestions, naturally following the next-token prediction task in language modeling^[190]. Tools like GitHub Copilot and Codex exemplify this trend^[15], and are followed by commercial tools such as Cursor⁴ and Tabnine⁵. Recent advances in context-aware^[191], grammar-aligned^[192], and constraint-based decoding^[193] have improved the quality of local completions, particularly for shorter code snippets. For longer code snippets, the typical task formulation is method implementation synthesis given a function signature. This setup is commonly evaluated using benchmarks such as MBPP^[16] and HumanEval^[15].

Natural Language to Code Generation: Translating natural language into code has long been a central challenge in AI for programming. Early attempts at code generation involved semantic parsing^[194][195], where natural language is translated into logical forms or domain-specific languages. A prominent example is SQL query synthesis from natural language questions, as seen in systems like Seq2SQL^[196] and Spider^[197], where the target language is constrained, small, and domain-specific. Recent work demonstrates that large language models (LLMs) can generalize to general-purpose programming languages, enabling the generation of larger and more complex code snippets^[198]. When applied to code completion, users often begin with natural language instructions in the form of comments, which LLMs use as context for code synthesis. Beyond function-level code generation^[16][15], recent work has extended to class-level generation^[199], which targets classes in object-oriented programming, and even project-level code generation^[200][201], which involves generating or completing entire multi-file codebases.

Multimodal Code Generation: While text can describe most cases of code generation, certain instructions are better defined visually. For example, in graphics applications, visual context such as a trajectory or a 3D model is essential to synthesize the correct code. Demonstrations of GPT-4’s multi-modal capabilities have shown that models can generate functional webpage code directly from paper sketches, translating visual layouts into HTML and CSS^[202]. LogoMotion^[203] explores visually grounded code synthesis for animations and motion graphics in JavaScript. The system leverages vision-language models (VLMs) to incorporate both visual inputs and user instructions, enabling code generation that aligns with spatial and temporal visual cues. Other works, such as SynthesizeCAD^[204] and SGP-Bench^[205], explore how LLMs can interface with visual and 3D modalities by generating code in languages like SVG and CAD.

Code Generation in Low-Resource Languages: As discussed in Sec. 3.7, one major challenge is writing code in low-adoption general purposed language and domain specific languages (DSLs). Benchmarks for this include MultiPL-E^[206], McEval^[207], and VerilogEval^[208]. A popular method to improve performance is to train on manually curated and processing data in low-resource languages such as Coq^[209] and Verilog^[210]. Another line of work aims to achieve transfer between different low-resource languages^{[211][212][213]}. Finally, since the lack of data is a large bottleneck, another popular direction is using relevant retrievals such as useful functions and library documentation^{[214][215][214]}. For a recent survey of code generation for low-resource languages and DSLs, see^[4].

Security Concerns Surrounding Code Generation: Despite the growing power of LLMs for code generation, their outputs often remain insecure, incorrect, or misaligned with user intent. For instance, BaxBench^[216] evaluates LLMs on generating secure and correct back-ends, revealing that while the average functional correctness is already modest ($\sim 60\%$), the rate of secure outputs is even lower ($< 35\%$). To better understand and quantify these limitations, several benchmarks and evaluation suites have been proposed. SecurityEval^[217], SafeCoder^[218], CodeLMSec^[219], CWEval^[220], and CyberSecEval^[221][222] each provide distinct lenses on evaluating vulnerabilities, unsafe API usage, or compliance with common weakness enumerations (CWEs). In response, several approaches introduce human-in-the-loop guardrails, where developers can interactively guide, inspect, or constrain the generation process. Dynex^[223], for instance, supports dynamic, step-wise code synthesis with user feedback, enabling real-time correction and iterative refinement before errors can accumulate.

Human Interaction in Code Generation: Modern code LLMs typically support interactive code generation through conversational interfaces.^[224] conducted a quantitative analysis of developer-ChatGPT interactions using the DevGPT dataset^[225], examining how the quality of the initial prompt influences conversation length. Code LLMs can be further optimized for various interactive scenarios, including debugging environments^[226], educational settings^{[227][228][229][230]}, and use by non-professional programmers^[231]. Beyond human-driven interactions in chat-based setups, more advanced code generation systems such as coding agents can proactively ask clarifying questions^[157] or generate test cases for users to validate^[232][233] before generating the actual code, helping to resolve ambiguities.

A.2. Code Transformation

Code Refactoring: Code refactoring aims to simplify and remove repetitions in complex repositories without altering high-level program intent. While there have been traditional methods^[234] that refactor data structures, Aider AI introduces a refactoring benchmark⁶ evaluating LLM’s ability to output long chunks of code that simplify complex programs without changing its behavior. More recently, RefactorBench^[52] introduced a more complex benchmark with natural language refactor requests, as well as an LLM agent that can perform refactoring.

Code Migration: Compared to code refactoring, code migration typically refers to mid-scale modifications that affect a program’s interface, dependencies, or underlying architecture. Common examples include switching the back-end database from MySQL to PostgreSQL, migrating a machine learning model from TensorFlow to PyTorch, or upgrading the Java version from legacy Java 8 to a more modern Java 17. While recent work has introduced benchmark designed to evaluate library migrations^[235], works at Google^[27] and Amazon^[236] have explored LLM-driven solutions for simple but vast migrations. Google’s system identifies locations for changes, generates edits with LLMs fine-tuned on internal code, and automatically validates changes through compilation and test execution.

Code Translation (Transpilation): Moving beyond code migration, transpilation involves large-scale transformation of a program’s underlying programming language. Transpilation serves not only to modernize outdated codebases but also to eliminate classes of safety issues inherent to older languages. A particularly active area of research involves transpiling C-based systems to Rust, a systems-level language that offers strong memory and concurrency safety guarantees. This direction has garnered attention, including from the U.S. Department of Defense⁷, which maintains critical infrastructure built on aging C code. An end-to-end LLM-based approach, such as Flourine^[237], has been proposed for real-world code translation, but it has achieved only limited success due to frequent compilation errors. Recent efforts like Syzygy^[33], C2SaferRust^[34], and AlphaTrans^[35] have shown the potential for hybrid approaches combining LLMs with traditional program analysis techniques. However, some significant challenges remain, as identified by^[238], including ensuring correctness in large codebases while maintaining desirable attributes such as speed, reduced vulnerabilities, and idiomaticity. Specifically, We anticipate that the techniques discussed in Section A.3may help address these remaining challenges.

Code Optimization: Certain refactoring or transpilation tasks are specifically aimed at optimizing code performance. Prior work has explored the use of LLMs for optimizing standalone programs, such as PIE^[119], which targets C++ functions, and AlphaDev^[239], which discovers more efficient sorting algorithms at the assembly level. These tasks are particularly challenging due to the vast search space of possible code transformations. More recently, KernelBench^[37] introduced a benchmark focused on optimizing machine learning models written in high-level PyTorch code into low-level, high-performance CUDA GPU kernels. For a broader overview of language models applied to code optimization, see the survey by^[240].

A.3. Software Testing and Program Analysis

Short-horizon Testing: For short-horizon testing such as unit tests^[241] and property-based tests^[242], LLMs are employed to automatically generate targeted test cases^[243][244], and even hill-climb on code coverage to improve test effectiveness^[245]. At the granularity of individual functions, LLM-generated tests have also been employed to support downstream tasks such as filtering implementations based on behavioral correctness^[246][247], as well as assisting in program debugging by surfacing inputs that expose incorrect behavior^[248].

Long-horizon Testing: Long-horizon testing involves evaluating system behavior across extended executions, complex interactions, or multiple components, potentially embedded within a CI/CD (Continuous Integration or Delivery) pipeline. Fuzzing^[249] is a long-horizon testing approach that continuously generates novel random input. Recent works such as Fuzz4All^[250], KernelGPT^[251], and OSS-Fuzz^[252][41] have shown that LLMs can significantly improve effectiveness through better input generation and exploration strategies. Specificatlly, OSS-Fuzz-Gen^[253] employs diverse LLMs for fuzzing harness generation, helping to find novel and complex crashing interactions.

Static Analysis for Vulnerability Detection: Vulnerability Detection refers to the task of identifying weaknesses or flaws in software code that could be exploited to compromise the system’s security, stability, or correctness. A wide range of prior work leverages machine learning models such as Graph Neural Networks (GNNs) and Recurrent Neural Networks (RNNs) to detect software vulnerabilities^{[254][255][256][257][258]}. While some recent methods pre-train or fine-tune LLMs on code-specific datasets^{[259][260][261]} to improve vulnerability classification, several studies have highlighted the limitations of LLMs in real-world software^{[262][263][264]}. To combat such limitations, works like^[265], IRIS^[266], LLMDFA^[267], and InferROI^[268] explored augmenting static analysis tools (e.g., CodeQL) with LLMs for taint and resource leak analyses. More recently,^[71] demonstrated the potential of using LLMs at a much bigger scale by finding a real SQLite vulnerability through exploratory variant analysis.

Specialized Program Analysis: Beyond long-running analysis to identify vulnerabilities, several traditional program analyses have struggled to scale in practice despite their theoretical promise. For instance, inferring program invariants (properties deemed to always be true at a program point) has been challenging with traditional symbolic methods such as Daikon^[269][270] while being valuable for exposing bugs^[271] and aiding software evolution^[272]. Similarly, type inference for dynamically typed languages suffers from coverage limitations of rule-based approaches and requires specialized tools like ShapeIt^[273] for domain-specific challenges such as inferring symbolic tensor shapes.

Specification Inference: Specification inference is the task of automatically recovering formal description of a program’s expected behavior, including pre-conditions, post-conditions, or invariants. The availability of specification is at the core of establishing trust^[274], and existing works^[275][276] have shown that LLMs can help the inference of such specifications. For instance,^[277] presents a program structure aware technique for synthesizing pre-conditions for arbitrary code snippets, and have established a dataset of 18K LLM generated pre-conditions on real Java projects.

Invariant Inference: As a subtask of specification inference, invariant inference aims at inferring loop, function, or class invariants, which are greatly helpful in automatic program verification. There have been several LLM-based approaches for invariant identification. They enhance traditional approaches through structured representations^[278], LLM-based prompting^[279][117] and re-ranking^[280], and reinforcement learning^[281]. Similarly, works have used sequence-to-sequence models^[282], few-shot LLM approaches like TypeGen^[283], and generate-then-rank methods like TIGER^[284] for type inference. Consequently, we observe new benchmarks emerging in the space such as LIG-MM^[285] for loop-invariant detection.

Binary Analysis: While the aforementioned tasks primarily focus on human-readable programming languages, many can also be extended to operate on compiled machine code, or binaries. One prominent example is binary type inference, which aims to recover high-level type information from low-level binary code. It has seen significant improvements with deep learning models and LLMs^[286][287]. These advancements, alongside other LLM-based analyses, have enhanced the capabilities of decompilers, enabling them to synthesize human-readable code from binaries^[288]. Beyond decompilation, LLMs have also been applied to detect security vulnerabilities in binaries^[289] and to generate semantic summaries that capture the high-level intent of binary code^[290].

A.4. Software Maintenance

Code Navigation: Code navigation refers to the task of locating a specific position within a code repository based on either a natural language description^[291] or a programmatic specification^[292]. Common use cases include identifying where a particular functionality is implemented, tracing the origin of user input that leads to a vulnerability, or locating relevant files when starting work on a new feature. This capability underpins many downstream tasks such as software testing, vulnerability detection, program repair, and code question answering. Code navigation or code search modules are integral components of modern code agents^{[66][293][294]}, often implemented using find commands, embedding-based similarity search, or query-based tools like CodeQL and Semgrep.

Code Documentation and Summarization: Several works have used LLMs for code summarization invoking techniques like prompting^{[187][295][296][297]}. RepoAgent^[298] is a framework that analyzes global contextual relationships in source code to generate fine-grained documentation.^[299] show that LMs are capable of generating good natural language outlines – text descriptions alongside code to partition it into semantically coherent sections. One challenge is that the evaluation of this task is very tricky: the academic community currently lacks datasets and benchmarks that contain good documentation and the automatic evaluation metrics do not align well with human metrics^[300].

Pull Request (PR) Review: In industry, autonomous software agents such as OpenHands^[67] and Devin have been able to automatically review and even fix PRs. At ByteDance, BitsAI-CR^[301] is a code review system that identifies issues based on a manually crafted taxonomy of review rules. In the academic community, there have been several works studying the ability of AI systems to automatically review PRs^{[302][303][304][305]}. Recently, AutoCodeRover^[306] combines LLMs with code search to automatically fix GitHub issues.

Program Repair: Automated program repair has had a long history, with many benchmarks covering different scopes and languages. These include DroixBench^[307] for android apps; Defects4J^[308], GitBug-Java^[309], and growingBugs^{[310][311][312]} for real-world Java; Bugsinpy^[313] for Python; BugSwarm^[314] for multilingual; DebugBench^[315], LiveCodeBench^[23], and Codeflaws^[316] for LeetCode-style problems; and many more.

Historically, there have been many techniques for this task, including heuristic-based APR (using genetic programming to explore the search space of the correct patch), constraint-based APR (treating repair as a constraint-solving task), pattern-based APR (apply expert hand-crafted repair templates), and learning-based APR (using language models)^[317]. More recently, with LLMs, there have been agent-based approaches such as FixAgent^[318] using agents specializing in different aspects of debugging, and RepairAgent^[293] that invokes suitable tools. On the other hand, Agentless^[294] uses a three-phase process of localization, repair, and patch validation.

Finally, program repair has also been used as a tool to improve code generation, where error messages and incorrect test cases are fed back into the model to improve code generation^{[319][320][321][322][323][324]}. This is also known as self-repair or self-debugging. For a much more comprehensive survey of automated program repair, we recommend the reader check out this website⁸.

Code Understanding and Question Answering: Code understanding with language models has been studied for many years. In earlier days, researchers used the CodeXGLUE^[325] benchmark containing tasks such as clone detection, code search, code summarization, and so on.^[326] create an IDE plugin containing features that help users understand code through explaining highlighted sections of code and explaining domain-specific code.^[327] present a survey touching on reasoning-enhanced code intelligence.

A.5. Scaffolding and Meta-Code

Beyond code generation, the broader software engineering ecosystem includes DevOps workflows, CI/CD pipelines, and Infrastructure-as-Code (IaC). LLMs have shown particular promise in generating, debugging, and explaining CI/CD configurations (e.g., GitHub Actions, Jenkinsfiles), assisting with environment setup, test orchestration, and deployment logic. A case study at Ericsson^[328] demonstrates how an LLM-based chatbot can support CI/CD question answering, enabling engineers to better understand and manage deployment pipelines. LLMs are also being explored for automated testing across heterogeneous software environments. ExecutionAgent^[329] presents a language model-driven agent that autonomously installs, configures, and runs test suites for arbitrary projects.

Beyond CI/CD and testing, LLMs are increasingly used to reason about configuration logic and scaffolding code, which is a critical but often overlooked layer of modern software systems. For instance,^[330] conducted an empirical study of real-world configuration errors, identifying systemic causes of failure such as external dependencies, inter-parameter violations, and overlooked default parameters. Building on this line of work, Ciri^[44] confirms the feasibility of using LLMs for configuration validation. Further, in the domain of IaC, an empirical study of 812 open-source Terraform projects found that while access policies are commonly adopted, critical practices like encryption at rest are often neglected^[331]. This highlights the opportunity for LLMs to assist practitioners in detecting and enforcing security best practices in IaC configurations.

A.6. Formal Verification

There are a variety of programming languages designed with different principles to support formal verification. Some of the popular ones include TLA^[332], Coq^[48], Lean^[110], Dafny^[333], Isabelle^[334], and Verus^[335].

Formal software verification has seen a few great successes in the last few years: Astrée^[336] was able to completely verify that Airbus A340’s primary flight-control software had no run-time errors, verifying 132,000 lines of C code. More recently, formal methods have been applied to verify a cryptographic server^[337] and an IoT lightbulb at both a hardware and software level^[338]. CompCert^[47], a verified compiler and seL4^[339], a verified microkernel are demonstrations that formal methods could be promising for verifiable code. At Amazon, formal methods been used to verify and protect cryptographic software^[340], cloud resources^[341], and authorization^[342]. Notably, SV-COMP^[343] is an annual competition designed to evaluate program verifiers using a curated benchmark of verifiable C and Java code. It even includes samples from the Linux Driver Verification (LDV) project^[344], aiding the verification of Linux kernel device drivers. For more applications, we refer the reader to the survey in^[345].

Recently, the ability of LLMs to write formal verification code. Benchmarks like DafnyBench^[346] and miniCodeProps^[347] were designed to measure the ability of LLMs to write software proofs in Dafny and Lean, respectively. In Dafny,^[348] use a combination of search and prompting to create a synthetic dataset of annotations greatly improving performance on DafnyBench. Clover^[56] generates code alongside consistency checks (like Dafny annotations),^[349] employ Dafny as an intermediate language to improve code generation, and^[350] explore prompting and retrieval to generate Dafny. In Rust, Verus is a popular formal verification language, with AutoVerus^[351] and AlphaVerus^[352] generating verified specifications and proofs for Rust functions. There are also many IDE plugins designed to help humans to write code in formal languages such as Dafny and Lean such as^[353], Lean Copilot^[354], and llmstep^[355].

Finally, there is a growing interest of work in using formal languages like Lean for mathematical theorem proving, which is covered comprehensively in^[356] and^[6].

Acknowledgements

We thank Alex Polozov, Baptiste Roziere, Daya Guo, Jenny Liang, Jiawei Liu, Justin Chiu, Kexun Zhang, Leonardo Hernandez Cano, Li Zhong, Michael Wang, Silas Alberti, Theo Olausson, Valerie Chen, Xingyao Wang, Yangruibo Ding, Yuxiang Wei, Zhiruo Wang, and several anonymous workshop reviewers for providing valuable feedback regarding various stages of the draft.

We also thank the following people for bringing up illustrative examples mentioned in this paper: Silas Alberti (debugging cloud applications), Chuyue Sun (incomplete specification in Verus), MIT’s 6.172 Course (performance instrumentation), Theo Olausson (costly disasters), Songlin Yang (syntax error in Triton).

A. Gu is supported by the National Science Foundation (NSF) Graduate Research Fellowship under Grant No. 2141064. N. Jain is supported by NSF grants CCF:1900968, CCF:1908870, and by SKY Lab industrial sponsors and affiliates. A. Solar-Lezama is supported by the National Science Foundation (NSF) and Intel Corporation through NSF Grant CCF:2217064. D. Yang is supported by the ONR YIP Award N000142412532.

Footnotes

¹ We follow page 9, Table 2 from their paper

² https://www.kprize.ai/

³ As reported by the BigCode Models Leaderboard on the MultiPL-E benchmark^[206]

⁴ https://www.cursor.so

⁵ https://www.tabnine.com

⁶ https://github.com/Aider-AI/refactor-benchmark

⁷ https://www.darpa.mil/news/2024/memory-safety-vulnerabilities

⁸ https://program-repair.org/