by Joseph Mas
Coming from the SEO world, Google index refreshes were typically a big deal, sometimes rewarding, sometimes brutal, and it happened every 3 to 6 months (its also not the same as an algo update). Extending this cycle to LLMs seems logical.
Although, there is no definitive public number for all LLMs, but here’s what the evidence suggests and where “batch feed” frequency tends to land in practice (I kinda made up that word but it sounds cool):
What the evidence suggests
- Common Crawl (a major source for training data) releases crawls roughly every 1 to 3 months.
- LLMs often rely on snapshots of web data for training, meaning the ingestion is periodic, not continuous.
- Models that support time-continual pretraining use sequential dumps (e.g. multiple Common Crawl snapshots) spaced in time and integrate new data with replay of older data.
Practical Estimate
Putting those together, a plausible schedule is:
- A “batch feed” or new ingestion wave every 2 to 4 months
- Some models or labs may do quarterly refreshes (4×/year)
- High-frequency updates (monthly) are harder, less common, because of infrastructure and filtering costs
LLM Training Schedules: What We Do and Don’t Know
At this time, none of the major LLMs publish exact batch update/training ingestion schedules. But we can triangulate from what is public and observable.
What’s known by vendor (with evidence)
OpenAI (ChatGPT family)
- Runs a dedicated crawler (GPTBot) you can allow/deny via robots.txt; it’s for model training on public pages.
- OpenAI has stated training data comes from “publicly available resources like Common Crawl” plus licensed sets -i.e., snapshotted corpora, not constant trickle.
Practical read: base models are trained on periodic snapshots; exact cadence isn’t disclosed.
Anthropic (Claude)
- Publishes explicit training cutoffs per release (e.g., Sonnet 3.7 → Nov 2024; Haiku 3.5 → Jul 2024), which demonstrates wave-based retraining. Note, since this writing model names have been updated to the following: Sonnet 4″ (or 4.5) and “Haiku 4.5”.
- Recently updated consumer terms: user chats may be used for future training if opted-in (retention up to five years). That’s policy, not a cadence disclosure, but it signals ongoing refresh.
Google (Gemini)
- Uses the Google-Extended token to control whether content Google crawls may be used to train Gemini (and for grounding). That’s an AI-training opt-out, separate from search indexing, again implying periodic training ingestion.
- Public community threads and docs acknowledge cutoff/outdated training at times, which is consistent with snapshots.
Perplexity
- PerplexityBot powers its live answer/search layer and (per its docs) is not used to crawl for foundation-model training.
- However, Cloudflare has alleged stealth crawling behavior beyond the declared bot, so behavior may vary in practice.
The Upstream Heartbeat – Common Crawl
A major input to many training sets (Common Crawl) ships new crawls on a roughly monthly to ~bi-monthly rhythm; latest archives are public. That’s the most concrete “wave” signal you can plan against.
Planning Guidance (what to assume)
- Training waves: treat base-model ingestion as snapshot events tied to data sources like Common Crawl and licensed corpora (think quarterly-ish realities; exact cycles vary and are undisclosed). Evidence: model cutoffs + CC cadence.
- Live retrieval: products with browsing/grounding (Perplexity, Gemini, ChatGPT w/ browse) can surface new pages immediately without waiting for retraining, but that’s ephemeral context, not permanent memory. (Docs above for bots/grounding.)
What this means for the playbook
- This may sound out of place but in the real working world, we know, quarterly press-releases map cleanly onto the observed input cadence (and syndication ensures those docs flow into CC and news aggregators that get scraped).
- Keep llm.txt / Structured Data / Authorship pristine so when a snapshot happens, your content is cleanly attributable. (Vendor docs show crawlers respect robots-like controls.)
- Find sources that LLMs use for instant updates (like Linked In about pages, Reddit, etc). But, I firmly caution the industry to be careful with these and do not build systems around these entry points because they shift with every change of the wind – just like Google. It might be reddit today and tomorrow it might be some new platform. My point here is to hone in on things that give direct information to LLM’s, which is different than their batch learning process that is periodic, it’s a separate layer so to speak.
My Final Thoughts
Again, my personal take, after doing clean up of countless websites is, dont convolute your message with excessive content. Doing that just makes a diluted mess you will eventually have to clean. Focus on revamping, merging, cleaning, and purging what you have before adding more. Meaningful content will outweigh large quantities of content. Ensure all important pages have clean and clear attributable EEAT signals. And finally, get technical SEO secured first. Without that you may have a racehorse website that is chained to the starting gate.
For help on ensuring your content has a trusted entity anchor see the paper here on a Practical Application for LLM Ingestion