Some of the largest news publishers are weighing whether to block Google's web crawler, a step that would pull their content out of the search results that have sent them readers for two decades. The Wall Street Journal reports that executives at USA Today, People Inc., Reuters, and Reddit have discussed cutting off Google's access as AI Overviews and AI Mode answer more queries directly on the results page.

Why the Old Bargain Is Breaking

For years the arrangement between Google and publishers was straightforward. Publishers let Google crawl and index their articles, and Google sent readers back through search, where those visitors became subscribers and ad impressions.

AI-generated answers change that math. When Google resolves a query at the top of the page, the reader often never clicks through, so the traffic that funded the content no longer arrives. Citing Semrush data, the Journal reported that organic search traffic fell 20% at Politico and 31% each at CNN and Business Insider between June 2025 and June 2026, with steeper declines at Reuters and the Washington Post over the trailing year.

The Crawler Bind

One of the biggest problems for publishers is that Google generally uses the same web crawler for both traditional search indexing and AI-generated search features. That means a publisher cannot simply tell Google to stop using its content for AI responses while continuing to appear in regular search results. Blocking the crawler would also prevent Google from indexing new pages for Search.

A full noindex instruction, which removes a site from Google search altogether, is the only clean way out. For most publishers that trade is far steeper than the traffic they are losing. The tool starving them of clicks is still the one delivering the clicks they have left.

Who Is Ready to Walk

Reddit has gone furthest in public, discussing whether to shut off Google's AI access entirely even though Google pays it roughly $60 million a year under a 2024 content deal now up for renewal, according to the Journal.

Others are testing softer measures. Politico staff have reportedly discussed requiring users to log in before reading free articles, creating a registration wall that limits unrestricted access to content. Time is experimenting with text-only ads aimed at AI crawlers rather than human readers.

People Inc. also told The Wall Street Journal that Google now accounts for roughly one-quarter of its traffic, compared with more than half two years ago. With Google's share declining, the company reportedly believes blocking Google's crawler has become a more realistic option than it once was.

A Pattern Already in Motion

The debate follows earlier disputes over AI and publisher content. Last year, publisher referral traffic from Google began falling as AI-generated search results expanded. Since then, several publishers have looked for ways to regain control over how their content is used. Penske Media, the owner of Variety and Rolling Stone, sued Google over AI summaries in an antitrust case filed last September.

Earlier this year, Google agreed to provide publishers in the UK with an opt-out for AI Overviews following discussions with regulators. A similar option is still not available for publishers in the U.S.

What It Means for Marketers

For brands and agencies, the standoff signals that the open web's largest content sources are becoming less predictable as distribution channels. If more publishers restrict crawling or wall off content, the pool of freely indexable material behind both search results and AI answers shrinks, and referral traffic from Google grows less reliable to plan around.

Marketers who lean on earned media or SEO across news sites may find that reach harder to forecast as publishers rethink what they hand Google for free.

HOME
Related News