October 6, 2026 · ChimpanSEO

How often do AI engines recrawl a site?

AI engines revisit a site on their own schedule, and the gap between visits usually runs from a few days to a few weeks. No single setting controls that rate. Google, OpenAI, Perplexity, and Anthropic each run separate crawlers with separate refresh logic, and each one reacts to different signals. What you actually control is signal quality: sitemap accuracy, visible update dates, internal links, and server speed. ChimpanSEO published more than 80 bilingual articles to observe how those signals behave on one domain over time.

How often do AI engines recrawl a site?

AI search engines revisit most sites on schedules that range from daily to weekly, and frequency tracks update signals rather than page count.

Google states in its Search Central documentation that crawl rate depends on two forces: crawl demand and crawl capacity. Demand grows when URLs are popular or change often. Capacity depends on how quickly your server responds and how many URLs you publish. AI search crawlers follow the same logic, because their index also has to stay useful.

The realistic range for a healthy site sits between a few hours and about a month. A newsroom can be visited within minutes of publishing. A small site with static pages may wait weeks. Server logs remain the only honest measurement of your own crawl pattern, since external trackers sample traffic instead of recording it.

Do different AI engines crawl at different speeds?

ChatGPT, Perplexity, and Google AI Overviews each run separate crawlers, so a page can appear in one engine weeks before the others pick it up.

Every operator publishes its own user agents, and they behave differently. Training crawlers collect pages in bulk and rarely return to the same URL. Search crawlers come back more often because citations expire and answers need current sources. A live fetch agent touches a page only when a person asks a question that points there.

That split changes how you write crawl rules. Blocking a training crawler in robots.txt does not stop the search crawlers from the same company, and blocking a search crawler removes your pages from citations entirely.

Crawler	Operator	What it does
Googlebot	Google	Feeds Google Search and AI Overviews
Google-Extended	Google	Controls Gemini and Vertex AI grounding
GPTBot	OpenAI	Collects content for model training
OAI-SearchBot	OpenAI	Builds the ChatGPT Search index
ChatGPT-User	OpenAI	Fetches a page during a live user question
PerplexityBot	Perplexity	Builds the Perplexity answer index
ClaudeBot	Anthropic	Collects content for Claude

What makes AI crawlers return sooner?

Freshness signals such as an accurate lastmod date in your sitemap and a visible update timestamp push AI crawlers to return sooner.

Freshness is a scheduling input, not a decoration. A timestamp that changes without real content change trains crawlers to ignore your dates. Google documents that sitemap lastmod values are trusted only when they stay consistently accurate, and every engine that reads the same file applies a similar test.

Six signals do most of the work:

Accurate lastmod values in your XML sitemap, one per URL
A visible update date on the page that matches dateModified in your JSON-LD
Internal links from hub pages you edit frequently
Stable URLs with a single canonical version
Fast responses with no 5xx errors and short redirect chains
Real content changes in the body, not just a new timestamp

Sitemaps cap out at 50,000 URLs and 50MB uncompressed, so large sites need a sitemap index. Splitting by section helps crawlers prioritise the pages that actually change.

How do you measure AI citation frequency?

AI citation tracking counts how often a URL appears inside generated answers, which is a different metric from ranking position in classic search.

Rankings tell you where you sit. Citations tell you whether the engine quoted you. You need both sources of data, and neither one can be bought.

Export server logs and filter requests by known AI user agents
Group hits by URL and count the unique days each URL was fetched
Run a fixed prompt set in each engine every week and record which URLs get cited
Track referral sessions from chatgpt.com and perplexity.ai in analytics
Map crawl gaps against your publishing dates to see which updates triggered a visit

After four to six weeks, the pattern becomes readable. Pages you update without changing the body keep their old crawl interval. Pages with genuine new sections get picked up faster and start appearing in answers within days.

What did the ChimpanSEO GEO experiment show?

ChimpanSEO ran a public GEO experiment by publishing over 80 blog articles as automatically paired Italian and English versions.

The setup is simple. Every article goes through an automatic generation pipeline and ships in two languages on the same domain. One topic therefore produces two crawlable URLs, two sets of structured data, and two chances to be cited in a generated answer.

The more useful finding is operational. The team runs its own corporate blog through ChimpanSEO every day, so the pipeline is tested on the product itself rather than on client accounts only. Bugs surface in production output, schema errors show up on live pages, and freshness signals get checked against real crawl behaviour.

Three habits came out of those 80 articles:

Publishing in pairs doubles the entry points without doubling the research
Keeping every section self contained makes extraction easier for RAG pipelines
Updating an old article with real new content restarts its crawl cycle

Frequently Asked Questions

Crawl frequency questions come up often, so the answers below cover the practical details that site owners ask about most.

Does blocking AI crawlers hurt my Google rankings?

Blocking Google-Extended stops Gemini and Vertex AI from using your content, but it does not affect Googlebot or your standard Search rankings. Blocking Googlebot itself removes you from Google Search and AI Overviews. The two controls are separate.

How long does a new page take to get cited by AI engines?

Most new pages reach AI answers within days to a few weeks. Speed depends on your sitemap, internal links from established pages, and how often competitors cover the same question. Pages on topics with no existing citations move faster.

Do I need an llms.txt file?

No engine requires llms.txt in 2026. The format is still a community proposal, not a standard adopted by Google, OpenAI, or Anthropic. A clean XML sitemap, working robots.txt rules, and accurate dates do more for crawl frequency.

Is crawl budget a problem for small sites?

Crawl budget matters mainly for sites with thousands of URLs. Small sites rarely hit a limit, because their constraint is crawl demand rather than capacity. If you publish fewer than a few hundred pages, focus on freshness signals instead.

If you want AI engines to visit more often, stop guessing and start logging. Pull your server logs, check which AI user agents arrive, and compare those visits against your publishing dates. Then fix the weakest signal first, which is usually an inaccurate lastmod value or a missing update date. ChimpanSEO applies that same loop to this blog, so the crawl patterns described here come from production work rather than theory.