Google is giving its AI Mode a new upgrade by letting users search using images. Robby Stein, Google's VP of Product, says Google is bringing the powerful multimodal capabilities in Lens to AI Mode. “AI Mode builds on our years of work on visual search and takes it a step further. With Gemini’s multimodal capabilities, AI Mode can understand the entire scene in an image, including the context of how objects relate to one another and their unique materials, colors, shapes and arrangements,” he said in a post.

Users can now take a photo or upload an image and ask AI Mode a question about it. This works through the integration of Google Lens and Gemini, and it’s designed to bring the same kind of visual search that Lens has offered for years, but with the added context and intelligence of AI Mode.

Say a user is browsing a product like a tool or gadget, with this update, the user could take a picture of it, ask AI Mode what it is, and get not just a short answer but a full explanation about the image. Google will also provide links for users to access more information about their searches.

Google says it uses a “query fan-out” technique, which means AI Mode issues multiple queries at once. It looks at the entire image and specific elements within it. For instance, for an uploaded photo of a crowded dinner table, AI Mode might recognize the food, the cutlery, and the wine glass, and run separate searches for each to give users more results.

The experience puts together the power of Lens and Gemini’s AI capabilities. Users get the benefits of traditional Google Search, combined with AI-generated responses and more context. 

Google rolled out AI Mode in early February just for Google One AI Premium subscribers. But last month, it opened access to more users and confirmed it would keep updating it with new features.  Millions of more Lab users will now access the feature and including image-based queries is part of this broader rollout.

A response to rising competition

Adding AI Mode to Lens seems like a response to rising competition in generative AI search. Google is doing all it can to stay ahead in the search market, especially as other companies are ramping up their search efforts. Microsoft has introduced Copilot Search, which brings AI summaries to Bing like Google AI Overviews. Perplexity pitches itself as an AI-native search alternative, and OpenAI now allows users to search the web with SearchGPT.

Google’s goal is to ensure it doesn’t lose ground as the leader in Search. As people are searching with images, Google wants to be the platform where they do it. 

Previously, the company rolled out new updates in Lens that allow iPhone users to search what’s on their screen and also get AI summaries and links to more resources from a snapshot.

‍