Intelligent Visual Studio Search Service
In Visual Studio (VS) 2019 v16.0 release, the Visual Studio Search shortcut Ctrl+Q, as shown in the screenshot below, was introduced as a Quick Launch for VS feature searches and expanded in v16.3 to support code searches. With millions of developers using the VS IDE, more and more developers are using this hotkey to boost their productivity; we hear customer feedback at developer conferences, customer interviews, and user surveys regarding improving search relevancy and increasing feature discoverability.
Looking into the existing in-box Ctrl+Q search experience, there are a few limitations:
- The search results relevancy was built-in to VS releases and hard to improve over time.
- Relies on keywords manually added to each feature for keyword matching.
- Employed heuristics that were based on point-in-time data for ranking algorithm tuning.
To address these limitations, last year we started to explore a solution built upon AI/ML based Azure services and shipped the feature in v16.9 release – an intelligent Visual Studio feature search service powered by Azure services, currently focusing on VS feature searches, and integrated with the Ctrl+Q box. In this blog, we want to share our experience and learnings in building the feature search service.
Service Architecture
The system extracts Visual Studio metadata for all 14 locales of each release (e.g., v16.9 release), uploads the metadata to a secure store for further validating and processing; the processed data is fed into the index processing pipeline, which leverages MS language analyzers for different locales and MS text analytics to extract keywords for VS features. When a user performs a search in the Ctrl+Q box in the VS IDE, the request is sent through the load balancer to one of the nearest geo-redundant REST API endpoints, which processes the request based on user’s search context, and returns the top matched search results to the user.
There are two primary data flows in this system, each consisting of various components and Azure services. Currently the system supports all 14 Visual Studio locales and v16.9+ releases.
From top left: feature metadata generation and index creation
- An automated pipeline extracts Visual Studio features metadata for all locales of each release and uploads them into a secured endpoint.
- An Azure Functions app triggers, validates, and transforms the latest drop of the new or updated VS feature metadata, as well as a Synonyms mapping list, then stores the processed data into Azure Storage service’s Table storage.
- The indexers of the Azure Search service pick up the updates in storage to index all documents, while leveraging Azure Cognitive Services’ NLP text analytics for keywords extractions with language detection during indexing process and query result retrieval. For any updated metadata, we do incremental caching with the Azure Storage to only process those features with changes.
From the right: feature search and data retrieval
- When a Visual Studio user types in a search query in Ctrl+Q box, the VS client sends the anonymized request to the feature search service, going through the Azure Front Door that routes the call into the nearest region for the REST API endpoints.
- The REST API app receives the search request, checks to see if the search results were in local cache or in the Redis Cache, and quickly returns it. If not yet cached, it retrieves new results from one of the indexes in the Azure Search service, which in turn will be cached for future fast retrieval.
- Based on the user’s search context, such as whether any project was opened or while Visual Studio is in debugging mode, the top results are then compared to the contextual ranking data with same feature identifiers in the Re-ranking model. The contextual ranking data was pre-processed and cached to speed up the process.
- The top re-ranked results then return to the Visual Studio user.
- The anonymized user’s feature search click-through and contextual usage data are fed into the Contextual Usage store for continued contextual ranking updates in the re-ranking model.
Search Scoring and Ranking
To reach the parity of the existing in-box search experience and improve upon the search relevancy, we took a few calculated choices in the index design, ranking algorithm, and scoring tuning:
- Created a few composable cognitive skills to extract the key phrases from each VS feature with detected language for where feature keywords were left off.
- Added the Synonyms map based on historical search usage, which is supported in the English locale index in this release.
- Designed the scoring profile with selected fields and tag boosting based on whether the features were menus, commands, options, project and item templates, preview features, or setup items.
- Adapted MS Language Analyzers in our indexes for lemmatization and word decompounding that are more suitable for different VS locales.
- Leveraged Azure Search service’s feature to compute ranking scores of documents across all 12 shards in the Azure Search service for consistent search results of those features that are closely related but scattered in different shard.
- Used the BM25 Similarity ranking function with term frequency hyperparameters customization, instead of the default TF-IDF similarity. The latter is not desirable for us because certain categories of VS features that are infrequently used, such as Installer components, are biased towards due to their greater lengths.
- Built a Lucene Query function to support fuzzy, proximity, wildcard searches for greater search flexibility.
Service in Production
To achieve production level performance, scalability, geo-redundancy, and reliability, manage potential service throttling, and monitor service health, the system is plugged into our well-tested AI service infrastructure, which supports our service’s development, staging and deployments with Azure DevOps and Azure Resource Manager templates with consistency.
Results and Benefits
So how did the search relevancy from the search service perform compared to that of the inbox search experience? We conducted a few staged A/B experiments in v16.8 release and measured with MRR the scores of the Control group and the Treatment group, and statistically significant improvements in the Treatment group were consistently observed at different stages.
Below are a few examples of the top searched VS features, their relevant results without and with the search service.
When searching for “solution”, the topmost selected result is “Solution Explorer”. The left screenshot below without the service has the result at #5 in the result list, while it is at the top 1 with the service shown in the right screenshot below.
When searching for “keyboard”, we got frequent user feedback about wanting “keyboard” mapped to “Change keyboard settings”, with the service, the feature result is showing at the top 1 in the right screenshot.
With the release of the VS feature search service powered by Azure services, the system enables:
- Preliminary context aware search experience.
- Iterative, data-driven improvements to feature search rankings.
And we can help you with more and more accurately what you are looking for with faster iterations in improvements.
What’s Next
There are several ways that we can continue to improve the system. For example, the system can consider the current coding file’s language, the current project’s type, what workloads are currently installed in the VS IDE, to setup automatic feedback loop based on user behavior, and provide more focused search results.
Taking search a little further, the system can be scaled to provide results from additional data sources, such as VS online docs and tech blogs, Bing search, etc., for improved VS feature discoverability, to unify with AI-powered code search services, and to eventually offer personalized search experience in VS.
Help Us Improve with Your Feedback
User feedback is the best way for us to improve the results. We encourage you to install the latest v16.9 release to try Ctrl+Q with our search service, as shown in the screenshots in the Results section above, the little cloud icon at the end of each result indicates that the result was from the search service.
We invite you to share your feedback or ideas for improvements at Suggest a Feature for Visual Studio. On behalf of our team working on enabling this feature, thank you!
Happy coding!
Light
Dark
9 comments
Is it correct that everything I type into that search box is sent into the cloud?
If yes, first, you are not telling me that the search box won’t work if I’m offline?
And second, more important, this would be a huge privacy concern for me. Please don’t assume that the whole world has the American culture and the American way of thinking. You are probably going to assure that ‘everything will be sent anonymously’, but that’s not what I want. Please provide at least an option “don’t send any keystrokes into any cloud”.
I guess I will have to create some firewall rules. It’ll save me some time if you tell me which hosts to block.
The Search box works offline as well in version 16.9, you can try and see.
If you think searching for an option in Visual Studio through the Search box vulnerates your privacy, you can always use other search boxes in the Visual Studio UI. If you are not satisfied by this, you are free to open a new issue in the Developer Community portal and ask for adding a new option to disable the new search experience. Visual Studio 2019 updates every month so surely they will address this kind of feedback soon.
Hi, Jorge,
Thank you very much for reading the blog and for the comment! And thanks for using Visual Studio!
Hi Martin,
Thank you very much for reading the blog and for the comment!
If you are offline while using Visual Studio, the search box works as before by leveraging the Visual Studio feature metadata from your installation. If you would like to block this feature while connected to the internet, you can block the endpoint data-ai.microsoft.com/vssearch at port 443 (see more details listed here).
For your comment about privacy concern: your privacy is important to us. All requests to the cloud service are made over HTTPS. The connection is encrypted end-to-end for enhanced security. Only feature search queries, and no information associated with identity, are processed and sent to the cloud service to provide the service, surfacing the most relevant results directly to you. This data is required for the service to work. Any telemetry associated with your usage of this feature (and most others) in Visual Studio is not required for this service to work. This information is only collected if you participate in the Visual Studio Customer Experience Improvement Program (VSCEIP). You can opt out of VSCEIP at any time by following these instructions.
Please let us know additional question you might have for the service. Thanks again for using Visual Studio!
Please improve the search experience of localized users.
The searchable term of any entry should include their English form, and their form that I would type in IME. This matches VSCode and Windows Search behavior.
Hi, Yaoyuan,
Thank you very much for reading the blog and for the comment! For us to better understand your scenarios, could you please let us know which of following scenarios that you are referring to?
1. Use English characters to type words in your preferred language, OR
2. Be able to type in either English or your preferred language and search
Thanks again for using Visual Studio!
I use that search box rarely but for a few specific thing. Today, my usual search for “breakpoint” was very slow to produce results and when it did, the Breakpoint window was not in the list (Debug > Windows > Breakpoints) . I was baffled as it’s always been there before. This explains a lot. And I’m sorry to say I’ll never use that search box again.
Hi, John,
Thank you very much for reading the blog and for the comment!
Could you please help to try and repro the scenario once, then file a feedback item with Help > Send Feedback? Doing so will give us the diagnostics information to investigate further for product faults and errors.
Thank you again for using Visual Studio!
Sorry, I am unwilling to file a feedback item to diagnose this. I really dislike the direction that box has gone especially as it relates to privacy.
I see in the VS 2019 19.10 preview that there’s a “Replace Go To with VS Search” option – please tell me that this will never become the default search path. I do not want any of the interaction and messages outlined in this blog post to take place.