Microsoft initiates trial of Copilot desktop sharing on Windows 11, enabling AI to scan and converse regarding screen contents
Microsoft has introduced an exciting update to its Copilot AI assistant on Windows 11. The new feature, known as Copilot Vision, now allows users to share their entire desktop with the AI, transforming the assistant into a proactive, context-aware tool that offers comprehensive help across all open apps and content.
This desktop sharing feature is opt-in and currently available to Windows 11 users in the Windows Insider Preview builds. To activate desktop sharing, users simply click the glasses icon in the Copilot composer and select either their whole desktop or a particular app window to share. Copilot Vision can then see exactly what the user sees in real-time on their screen.
With this expansion from app-window-level sharing to full desktop sharing, Copilot Vision becomes more capable and context-aware, enhancing its usefulness across a wide range of activities without the need for manually specifying each element Copilot should see.
The real-time analysis of screen content enables Copilot to offer more complete understanding of users' tasks and context. It can provide tips for improving creative projects, help with resume building, and guide users through navigating software or games. Copilot can also explain technical details, such as cryptic Windows error codes or help with complex tasks across multiple applications.
Moreover, Copilot can talk users through issues in real-time, coaching them based on what’s visible on their desktop. This feature is particularly useful for users who are not familiar with certain tasks or software.
The desktop sharing feature is optional and requires users to enter into a special mode in the Copilot app first before Copilot Vision becomes active. It is currently only available in the United States, but Microsoft has teased a customizable persona for Copilot, allowing users to create a character they can chat with.
The update for sharing the desktop with Copilot can be acquired via the Microsoft Store. Users can chat with Copilot in natural language as it views their shared screen. The Copilot key on Copilot+ PCs is set to open the Copilot app by default, offering fast access to features like Copilot Vision.
This expansion marks a significant step forward in AI assistance, elevating it beyond isolated app interactions to seamless, screen-wide support. With Copilot Vision, Microsoft is turning the Copilot app into a tool that works alongside the user, like a real copilot, providing highly contextual, interactive help across all their open apps and content on Windows 11.
[1] Microsoft (2022). Copilot Vision now allows users to share individual apps or windows with Copilot, and is being expanded to allow sharing of the entire desktop. Retrieved from https://news.microsoft.com/2022/05/25/copilot-vision-now-allows-users-to-share-individual-apps-or-windows-with-copilot-and-is-being-expanded-to-allow-sharing-of-the-entire-desktop/
[2] Microsoft (2022). The rollout of this feature will likely happen in waves. Retrieved from https://www.windowscentral.com/microsoft-testing-copilot-desktop-sharing-windows-11-across-all-insider-preview-channels
[3] Microsoft (2022). Copilot Memory, a new feature for Copilot, allows it to build a profile about the user and carry that information across devices. Retrieved from https://www.windowscentral.com/copilot-memory-new-feature-copilot-windows-11-lets-it-learn-your-habits-likes-and-dislikes
- The Copilot update on Windows 11, which includes the new feature Copilot Vision, now permits users to share their entire desktop with the AI, making the assistant more proactive and context-aware.
- To activate desktop sharing in Copilot, users simply click the glasses icon in the Copilot composer and select either their whole desktop or a particular app window to share.
- With full desktop sharing, Copilot Vision becomes more capable, enabling it to offer comprehensive help across all open apps and content on a user's PC without the need for manual specifications.
- The real-time analysis of screen content allows Copilot to provide tips for creative projects, help with resume building, guide users through navigating software or games, explain technical details, and coach users through tasks in real-time.
- Copilot can be accessed via the Microsoft Store and offers fast access to features like Copilot Vision via the Copilot key on Copilot+ PCs.
- Microsoft is turning the Copilot app into a tool that works alongside the user, providing highly contextual, interactive help, transforming it from an isolated app interaction to seamless, screen-wide support.
