Learn / Documents and data
How does web crawling work for RAG?
Short answer
Web crawling for RAG means visiting a website, collecting the useful pages and the documents linked from them, cleaning the text and indexing it with its source. The craft is deciding what to collect, what to skip and how to keep it current and safe.
With troveGEN
troveGEN crawls a website together with its linked PDF, Word and Excel files, files each with its source, and keeps the crawl safe and tidy.
What a good crawl does
- Starts from a page or a sitemap and follows links to a chosen depth.
- Stays within the hosts you allow and respects robots.txt.
- Skips noise: login pages, carts, tracking links, images and endless archive pages.
- Removes duplicates, including the same page reached through different addresses.
- Downloads linked documents such as PDFs, Word files and spreadsheets, where much of the real content lives.
- Keeps the address and any details found, so every answer can cite its page.
Pages versus documents
Many organisations publish their important material as PDFs: reports, circulars, price lists, guidelines. A crawler that only reads web pages misses most of it. Following document links, within size limits, closes that gap.
Safety
A crawler fetches addresses it is given, so it must never be steered to internal network addresses. Redirects need to be checked at every hop. Limit size, time and depth, and treat fetched content as untrusted.
Freshness
Websites change. Recrawl on a schedule or when content changes, replace updated pages, and remove pages that disappear so old answers do not linger.
Respect and rights
Only crawl content you have the right to use. Identify your crawler, honour robots.txt and keep the request rate polite.
Key takeaways
- A good crawl is selective: collect what is useful, skip the noise.
- Follow links to documents, not only web pages.
- Protect against internal-address access and keep content fresh.
How troveGEN helps with web crawling for RAG
troveGEN can crawl a site and its linked PDF, Word and Excel files, skips the noise, removes duplicates, files each document with its source and details, and blocks private network addresses, including through redirects. Scanned PDFs found during a crawl are not read yet; scans you upload are read with OCR.
What troveGEN provides
- Crawl to a chosen depth, with sitemap support
- Linked PDFs, Word and Excel files collected and parsed
- Noise and duplicate pages skipped
- Private network addresses blocked, including through redirects
- Continuous sync from connectors as an alternative to recrawling
Frequently asked questions
Is crawling legal?
It depends on the site's terms and the content. Crawl only what you are entitled to use and follow robots.txt. Take advice if unsure.
How deep should a crawl go?
As deep as needed to reach the content that matters, usually two or three levels. Deeper crawls fetch more noise.
Can it keep a site in sync?
Yes, by recrawling on a schedule or using a feed or manifest, replacing changed pages and removing deleted ones.
How does troveGEN help with web crawling for RAG?
troveGEN crawls a website together with its linked PDF, Word and Excel files, files each with its source, and keeps the crawl safe and tidy. It provides: Crawl to a chosen depth, with sitemap support; Linked PDFs, Word and Excel files collected and parsed; Noise and duplicate pages skipped; Private network addresses blocked, including through redirects; Continuous sync from connectors as an alternative to recrawling.