A
Anson Horton
Guest
In 1907, during the heat and humidity of a northern hemisphere summer, Paris newspaper Le Martin and the New York Times announced they would be sponsoring a race so ambitious as to be known simply as, “The Great Race.” The planned route was from New York to Paris and the astute among you may note there isn’t any actual land passage from one to the other. There was a theory that the Bering strait would ice over in the winter and prove crossable; that ‘Bering Land Bridge’ had been, of course, gone for ~18k years (just missed it!). This was at a time when there was little infrastructure to support these vehicles and very few paved roads.
This is how it has felt integrating AI into Visual Studio. AI remains so nascent that there aren’t paved paths to success. At times, we were busy building out an experience and realized that there wasn’t a way to accomplish it, so we had to pivot entirely (as did the motorists who ended up having their vehicles shipped across the Pacific). What has our approach been to build out the Copilot support that we have today given that? Well, experimentation, iteration, and leveraging .NET have played large roles, so let’s talk about it.
Visual Studio started our AI journey back when we announced IntelliCode at Build 2018. IntelliCode offers several productivity features including starred suggestions, whole line completions (grey text insertions that can be accepted with tab), repeated edits, and API usage examples.
These are all built on top of local models trained on public repositories. These models are not large language models and are exceptionally small such that they execute well even on lower powered machines and can load in limited memory space. IntelliCode was built before open standards for leveraging AI models had gained traction so much of the integration is bespoke. However, IntelliCode helped inform UX paradigms that would be generally adopted for future AI interactions such as ‘ghost text’ for completions and ‘tab to accept’ for repeated edits. IntelliCode has both an ‘in process’ component that is loaded into the devenv.exe process as well as a ‘model host’ which is loaded into a ServiceHub process. The in process and ServiceHub components are built with netfx. The out-of-process service is named ServiceHub.IntellicodeModelService.exe. Communication between the processes is facilitated via vs-streamjsonrpc which implements the JSON-RPC protocol for .NET over various transports. Since all of this was built on .NET, it was easy to rapidly iterate and worked seamlessly within the VS ecosystem. This is due to .NET’s rich ecosystem for profiling, debugging, performance monitoring, and its broad usage within Microsoft that allows for reuse of ideas and code. It also allowed reuse of early tokenizers such as BlingFire. Understanding this architecture helps set the stage for what came next.
In 2021 GitHub announced a technical preview of completions (ghost text) functionality in VS Code. It was March 2022 when GitHub released a similar extension for Visual Studio. GitHub Copilot was initially built on the OpenAI Codex model which was a GPT-3 language model additionally trained on gigabytes of source code in a variety of programming languages. At this time, GitHub developed the extension to Visual Studio and VS Code, along with other clients. To facilitate rapid development between these clients a single ‘agent’ was created that every client uses. That agent is a Node process and is responsible for making web requests to GitHub, collating the context to be used to prompt the model, truncating/massaging responses, and additional functionality and features such as content exclusion. If you are using Copilot completions in VS, you’ll see a separate process named ‘copilot-language-server.exe’ (its name has changed a few times, but this is the most recent name). If you’ve ever been curious why you may need to set environment variables to enable proxy support for Copilot completions in VS in addition to modifying devenv.exe.config, this is why. The Node process uses environment variables while .NET will use the information in the config file (this is an area where we’d love to improve in the future as it’s non-obvious). The nice thing is that .NET makes it relatively trivial to interoperate via IPC, so having these separate processes is straightforward.
The reason the process is now called copilot-language-server is because it communicates with its host via the LSP protocol.
GitHub Copilot Completions has evolved since GitHub introduced the feature in 2021. For example, completions no longer uses the Codex model. In Visual Studio 17.10 the feature is now bundled with the installer and does not require acquiring a separate extension (it can be unchecked/uninstalled as a recommended component for anyone that does not want it). The high-level architecture is largely unchanged though, so you’ll still see a Node process created whenever you’re signed into GH Copilot and using completions. This architecture allows substantial sharing between the clients which was GitHub’s goal when initially developing the extension; however, running the copilot-language-server as a Node process is somewhat foreign to the overall architecture of Visual Studio, so when the GitHub Copilot X investments started, we reconsidered this approach and decided to align more directly with Visual Studio’s natural extensibility model.
In March 2023 GitHub announced Copilot X which introduced many new features, including chat integration into Visual Studio. This may be clear if you’ve gotten this far, but the separation of ‘chat’ and ‘completions’ is arbitrary and, at this point, largely the result of historical evolution rather than any technical need. The Visual Studio team set out to deeply integrate ‘chat’ functionality into Visual Studio. This started with basic features like a chat tool window and inline chat; however, many other features have subsequently been added which leverage this ‘chat’ functionality, such as rename suggestions, generating commit and PR messages, helping to diagnose exceptions, etc. The architecture is like IntelliCode and completions; however, instead of a .NET Framework or Node process, there are separate .NET 8 processes:
Unlike the Node process for completions these are all specific to VS and not shared with VS Code, JetBrains products, or other clients. We try to be very intentional about when choosing to build centralized features and a big part of that calculus has to do with user expectations and client specifics. We chose this architecture because although there are many overlapping concerns between the different clients, the prompt crafting, context retrieval, etc. is very client specific. Additionally, the VS team knew that by building the OOP components on .NET, we could both leverage and inform the new framework types in .NET 8/9 that speed up the creation of ‘intelligent apps’ (i.e., apps that are powered by AI). It also forced us to separate our logic so that as the AI landscape evolved, we could react quickly. Finally, it also aligned nicely with the new modern extensibility model for Visual Studio.
This architecture has allowed us to leverage/influence many .NET improvements, for example: server-sent events, System.Text.Json to JSON schema mapping, tokenizers, various aspects of semantic kernel, and others. This includes the recently announced Microsoft.Extensions.AI.Preview.
The indexing service is another component upon which much of the Copilot architecture depends. Providing good context as part of interacting with LLMs is critical to quality responses. The context needed differs depending on the scenario, but it’s frequently necessary to quickly answer questions about the source location for definitions or references. For example, imagine that VS is providing completions and you have the following:
// Person.cs
Then in program.cs you start editing Main. You might see a completion suggestion like this:
This would happen if the context to create the completion didn’t include information about the Person record. The LLM will do its best to guess and, it’s seen plenty of .NET Person classes and it just assumes that this one probably has an IsVillian property. However, with the semantic index, this would instead look like:
The semantic index is leveraged for completions, chat functionality, etc.
This is a bit of the history and high-level architecture of GitHub Copilot integration in Visual Studio. AI is progressing exceedingly rapidly and building on .NET is one way that the VS team can keep pace. There will be many enhancements, changes, benefits, and surprises in upcoming VS and GitHub Copilot releases, so keep an eye out! If you’re curious, the New York to Paris race was eventually completed by three of the initial six teams. The race led to the call for improved roads to be constructed across the world. This is how I see the partnership between Visual Studio, GitHub Copilot, and all the other AI application development happening with .NET. It is trailblazing for improved libraries, better tooling, and faster development. There may not be a clear destination, but we can certainly help pave the way.
The post How we build GitHub Copilot into Visual Studio appeared first on .NET Blog.
Continue reading...
This is how it has felt integrating AI into Visual Studio. AI remains so nascent that there aren’t paved paths to success. At times, we were busy building out an experience and realized that there wasn’t a way to accomplish it, so we had to pivot entirely (as did the motorists who ended up having their vehicles shipped across the Pacific). What has our approach been to build out the Copilot support that we have today given that? Well, experimentation, iteration, and leveraging .NET have played large roles, so let’s talk about it.
Visual Studio started our AI journey back when we announced IntelliCode at Build 2018. IntelliCode offers several productivity features including starred suggestions, whole line completions (grey text insertions that can be accepted with tab), repeated edits, and API usage examples.
These are all built on top of local models trained on public repositories. These models are not large language models and are exceptionally small such that they execute well even on lower powered machines and can load in limited memory space. IntelliCode was built before open standards for leveraging AI models had gained traction so much of the integration is bespoke. However, IntelliCode helped inform UX paradigms that would be generally adopted for future AI interactions such as ‘ghost text’ for completions and ‘tab to accept’ for repeated edits. IntelliCode has both an ‘in process’ component that is loaded into the devenv.exe process as well as a ‘model host’ which is loaded into a ServiceHub process. The in process and ServiceHub components are built with netfx. The out-of-process service is named ServiceHub.IntellicodeModelService.exe. Communication between the processes is facilitated via vs-streamjsonrpc which implements the JSON-RPC protocol for .NET over various transports. Since all of this was built on .NET, it was easy to rapidly iterate and worked seamlessly within the VS ecosystem. This is due to .NET’s rich ecosystem for profiling, debugging, performance monitoring, and its broad usage within Microsoft that allows for reuse of ideas and code. It also allowed reuse of early tokenizers such as BlingFire. Understanding this architecture helps set the stage for what came next.
In 2021 GitHub announced a technical preview of completions (ghost text) functionality in VS Code. It was March 2022 when GitHub released a similar extension for Visual Studio. GitHub Copilot was initially built on the OpenAI Codex model which was a GPT-3 language model additionally trained on gigabytes of source code in a variety of programming languages. At this time, GitHub developed the extension to Visual Studio and VS Code, along with other clients. To facilitate rapid development between these clients a single ‘agent’ was created that every client uses. That agent is a Node process and is responsible for making web requests to GitHub, collating the context to be used to prompt the model, truncating/massaging responses, and additional functionality and features such as content exclusion. If you are using Copilot completions in VS, you’ll see a separate process named ‘copilot-language-server.exe’ (its name has changed a few times, but this is the most recent name). If you’ve ever been curious why you may need to set environment variables to enable proxy support for Copilot completions in VS in addition to modifying devenv.exe.config, this is why. The Node process uses environment variables while .NET will use the information in the config file (this is an area where we’d love to improve in the future as it’s non-obvious). The nice thing is that .NET makes it relatively trivial to interoperate via IPC, so having these separate processes is straightforward.
The reason the process is now called copilot-language-server is because it communicates with its host via the LSP protocol.
GitHub Copilot Completions has evolved since GitHub introduced the feature in 2021. For example, completions no longer uses the Codex model. In Visual Studio 17.10 the feature is now bundled with the installer and does not require acquiring a separate extension (it can be unchecked/uninstalled as a recommended component for anyone that does not want it). The high-level architecture is largely unchanged though, so you’ll still see a Node process created whenever you’re signed into GH Copilot and using completions. This architecture allows substantial sharing between the clients which was GitHub’s goal when initially developing the extension; however, running the copilot-language-server as a Node process is somewhat foreign to the overall architecture of Visual Studio, so when the GitHub Copilot X investments started, we reconsidered this approach and decided to align more directly with Visual Studio’s natural extensibility model.
In March 2023 GitHub announced Copilot X which introduced many new features, including chat integration into Visual Studio. This may be clear if you’ve gotten this far, but the separation of ‘chat’ and ‘completions’ is arbitrary and, at this point, largely the result of historical evolution rather than any technical need. The Visual Studio team set out to deeply integrate ‘chat’ functionality into Visual Studio. This started with basic features like a chat tool window and inline chat; however, many other features have subsequently been added which leverage this ‘chat’ functionality, such as rename suggestions, generating commit and PR messages, helping to diagnose exceptions, etc. The architecture is like IntelliCode and completions; however, instead of a .NET Framework or Node process, there are separate .NET 8 processes:
Unlike the Node process for completions these are all specific to VS and not shared with VS Code, JetBrains products, or other clients. We try to be very intentional about when choosing to build centralized features and a big part of that calculus has to do with user expectations and client specifics. We chose this architecture because although there are many overlapping concerns between the different clients, the prompt crafting, context retrieval, etc. is very client specific. Additionally, the VS team knew that by building the OOP components on .NET, we could both leverage and inform the new framework types in .NET 8/9 that speed up the creation of ‘intelligent apps’ (i.e., apps that are powered by AI). It also forced us to separate our logic so that as the AI landscape evolved, we could react quickly. Finally, it also aligned nicely with the new modern extensibility model for Visual Studio.
This architecture has allowed us to leverage/influence many .NET improvements, for example: server-sent events, System.Text.Json to JSON schema mapping, tokenizers, various aspects of semantic kernel, and others. This includes the recently announced Microsoft.Extensions.AI.Preview.
The indexing service is another component upon which much of the Copilot architecture depends. Providing good context as part of interacting with LLMs is critical to quality responses. The context needed differs depending on the scenario, but it’s frequently necessary to quickly answer questions about the source location for definitions or references. For example, imagine that VS is providing completions and you have the following:
// Person.cs
public record Person(string Name, bool HasEvilLair);Then in program.cs you start editing Main. You might see a completion suggestion like this:
This would happen if the context to create the completion didn’t include information about the Person record. The LLM will do its best to guess and, it’s seen plenty of .NET Person classes and it just assumes that this one probably has an IsVillian property. However, with the semantic index, this would instead look like:
The semantic index is leveraged for completions, chat functionality, etc.
This is a bit of the history and high-level architecture of GitHub Copilot integration in Visual Studio. AI is progressing exceedingly rapidly and building on .NET is one way that the VS team can keep pace. There will be many enhancements, changes, benefits, and surprises in upcoming VS and GitHub Copilot releases, so keep an eye out! If you’re curious, the New York to Paris race was eventually completed by three of the initial six teams. The race led to the call for improved roads to be constructed across the world. This is how I see the partnership between Visual Studio, GitHub Copilot, and all the other AI application development happening with .NET. It is trailblazing for improved libraries, better tooling, and faster development. There may not be a clear destination, but we can certainly help pave the way.
The post How we build GitHub Copilot into Visual Studio appeared first on .NET Blog.
Continue reading...