On October 15, 2026, Cloudflare and OpenAI published early results from a joint pilot that started in July. Cloudflare, which sits between visitors and millions of websites to speed them up and protect them, passes signals from its network to OpenAI's search crawler. Those signals help the crawler find pages it didn't know about. According to Cloudflare, the pilot ran on only about 1% of its network and brought hundreds of millions of new pages into ChatGPT search.
Pages nobody links to
Cloudflare's post gives a few examples: the manual for one specific washing machine model, the current requirements and fees for a passport, the menu and opening hours of a neighborhood restaurant. It says pages like these can be very useful even when hardly any other site links to them.
That is the interesting line in the post. Google, like other search engines, leans heavily on links between sites to discover pages and judge them. A local business site gets few links, so it often reaches Google's index late or not at all. A system that discovers pages from what happens on the network itself depends on links less.
What the crawler gets and what stays out
Before the crawler reaches a page, Cloudflare filters out addresses that aren't worth a visit: URLs that don't exist but bots keep requesting, sites that redirect every address to the homepage, and copies of the same content. It also tells the crawler when a page has changed and is worth another visit. On top of that it is testing signals that flag a single compromised page, such as a fake login form an attacker planted on an otherwise legitimate site, so that one page can be kept out of the index without penalizing the whole site.
OpenAI decides on its own which pages appear in answers and in what order. Cloudflare stresses that a signal about a page does not grant permission to access it. The pilot uses information about network traffic, and page content is not passed on for model training.
The edge Google doesn't have
Google has more than twenty years of data and experience, and its search algorithm is the most sophisticated one in use today. Cloudflare has something else: from where it sits, it sees at the server level what happens on websites, and Google has no access to that.
The job itself is different too. A person searching usually looks at the top three results, while a language model can read dozens of pages before it answers. Someone building search only for AI mostly has to find the good pages and worry less about ordering them precisely, and ordering is the hard part. Google has to get both right for as long as it keeps the regular list of results.
Three checks for site owners
A site that blocks OpenAI's crawler, OAI-SearchBot, in its robots.txt file won't appear in ChatGPT search answers. According to OpenAI, you can allow that crawler while blocking GPTBot, which collects content for model training, because each setting works independently. If your site runs on Cloudflare, also check its dashboard for which AI crawlers are blocked. Cloudflare has also offered a free setting called Crawler Hints since 2021, which tells search engines when a page on your site has changed, and it is turned on with one click.
What is still unknown is how often the new pages were cited in ChatGPT answers, and whether they sent visits to websites. According to Cloudflare, that is what the next phase of the pilot will measure.