Artificial Intelligence Developers at Microsoft Reveal CodePlan: Streamlining Intricate Software Development Duties through AI Advancements
Researchers at Microsoft have proposed a new system called CodePlan, designed to automate repository-level coding tasks such as migrating to a new database system, updating to a new API, or fixing bugs that span multiple files.
CodePlan utilises a large language model to orchestrate changes across a codebase, setting it apart from baseline methods like using LLM edits alone and purely reactive approaches. The system was evaluated on two challenging repository-level coding tasks: API migration and ORM framework migration, which require developers to make many intricate, interdependent code edits. Some tasks required edits to as many as 168 files.
In comparison to baseline methods, CodePlan identified 2-3 times more correct code blocks to modify. Furthermore, CodePlan's final code edits aligned significantly better with the ground truth changes. This demonstrates its ability to automate complex repository-level coding tasks where less sophisticated applications of LLMs fail.
CodePlan's success is due in part to its planning capabilities, which allow it to require far fewer overall edits to complete tasks. The system decomposes the overall repository task into incremental steps, guided by the LLM's localization strength but augmented with rigorous planning and analysis.
Repository-level coding tasks are complex, error-prone, and time-consuming to perform manually. CodePlan was tested on proprietary Microsoft repositories and open-source GitHub projects ranging from thousands to over 15,000 lines of code. The system produced valid final code repositories without build issues while baselines often had many errors.
While CodePlan offers a promising path to expand AI's advantages into the repository-level automation domain, there are several areas for future work. These include expanding the framework to more languages, artifacts, and software projects. Future improvements in large language models will likely focus on enhancing their ability to understand and modify complex codebases effectively. Integrating AI coding assistants with more development tools and workflows can further enhance their utility and ease of use.
Similar AI systems, such as GitHub Copilot and Claude Code, have already shown significant potential in automating and simplifying coding tasks, enhancing developer productivity. However, these systems still face challenges in handling complex, repository-level tasks and require careful integration with existing workflows and tools to be effective.
[1] GitHub Copilot: An AI-Powered Coding Assistant, [2] FEA-Bench: A Benchmark for Incremental Code Development,
In the realm of science and technology, CodePlan, an advanced AI system developed by Microsoft, has demonstrated its potential in automating complex repository-level coding tasks, outperforming baseline methods in tasks like API migration and ORM framework migration. The system's success can be attributed to its integration of data-and-cloud-computing technologies and the application of artificial intelligence, setting it apart from other systems in the field.
