Apify Website Content Crawler for RAG: Feed Sites to LLMs (2026)

Why this actor is the RAG standard

The hard part of RAG ingestion is not downloading pages — it is turning messy HTML into clean content. The Website Content Crawler solves the extraction: main-content detection, markdown conversion, metadata (title, URL, section) preserved per page.

It renders JavaScript properly — modern docs sites, SPAs and client-rendered content come back as full text, not empty shells.

Setup: site to clean text in 10 minutes

1. Apify Store → Website Content Crawler — input the start URL
2. Options crawl depth, include/exclude URL patterns (limit to /docs or /blog), max pages
3. Output format markdown for RAG (structure preserved) or plain text for raw corpus
4. Test on 20 pages — inspect the cleanliness of the extraction — Test on 20 pages — inspect the cleanliness of the extraction
5. Full run — sitemap-scale crawls run unattended — Full run — sitemap-scale crawls run unattended

The RAG pipeline end to end

The full path: Website Content Crawler → markdown per page → chunk (by heading or token window) → embed with your embedding model → vector store (Pinecone, Weaviate, pgvector) → retrieval in your chat/agent.

The crawler covers the extraction half — which is the half that usually breaks. Chunking and embedding are library calls; clean input is what makes them work.

Use cases beyond RAG

Where the same extraction earns its keep:

• Chatbot knowledge base: crawl your own docs/help center — the bot answers from your content, not hallucinations
• Competitor content analysis: crawl a rival's blog — their whole content library as searchable text for gap analysis
• Site migration: full-content export before replatforming — nothing lost in the move
• SEO content audit: your own site as structured text — thin pages and duplicate content surface in the data
• LLM fine-tuning corpus: domain-specific text at corpus scale

For the training-data angle and dataset-scale considerations, see Apify for AI training data.

Output formats and scheduling

Markdown preserves headings, lists and links — best for RAG where structure aids chunking. Plain text for language-model corpora where structure adds noise. HTML output exists when you need layout preserved.

Scheduled crawls keep the knowledge base fresh: re-crawl docs monthly, diff the dataset, re-embed changed pages only — a living RAG source instead of a frozen snapshot.

Respect and limits

Crawling your own properties: unrestricted use — it is your content. Crawling third-party sites for analysis: keep to public pages, reasonable volume, and never republish extracted content as your own.

For LLM training on third-party content specifically: the legal picture is unsettled and jurisdiction-dependent — the honest guidance in web scraping legality applies; RAG on docs and knowledge bases you own is the safe default.