Reddit has taken measures to control its data usage by blocking major search engines, including Microsoft’s Bing, from crawling its site unless they enter into formal agreements. 

According to Reddit's Chief Executive Officer, Steve Huffman, Microsoft has been using Reddit’s data to help improve its artificial intelligence models, such as those used in Bing's search engine, without Reddit's permission. Also, Microsoft has sold access to Reddit’s data through its Bing API to other search engines, further distributing Reddit's information without authorization. Huffman also listed Perplexity and Anthropic as part of companies scrapping Reddit’s data without permission. 

Reddit’s new policy 

This new policy requires search engines and other data users to sign agreements with Reddit to access and use its content. “Any crawler that we don’t have a formal agreement with, we’re now blocking,” Huffman said. 

Previously, Reddit allowed search engines to crawl its site freely. However, Huffman noted that this practice has led to issues such as using Reddit data for AI training and resale without proper attribution or compensation. 

"When it was used for simple search, to create simple links that would send us traffic from search engines, that was fine," Huffman said. "But now folks are using Reddit data for training, they’re reselling it, doing search summaries instead of linking to us."

Therefore, a month ago, the company changed its approach to data access, preventing companies from scraping its data without authorization. This new policy will mean that Reddit will demand payment from AI search engines for data access.

Why this is important to Reddit

Huffman explained that the importance of this new measure is to protect the platform’s data; because the lack of agreement means Reddit cannot dictate how its data appears in search results or how it is utilized. “Without these agreements, we don’t have any say or knowledge of how our data is displayed and what it’s used for, ” He said. 

The development since the new policy 

Perplexity and Anthropic, have also adjusted their practices in response to Reddit’s new policy. Perplexity has extended an invitation for Reddit to join its Publishers' Program. Perplexity announced that it will start sharing ad revenue with publishers. Anthropic has respected Reddit’s block since mid-May. 

Following Reddit's update to its robots.txt file, Microsoft Bing stopped collecting data from Reddit last month.