Crawlee Guide: Build Production Web Scrapers with Apify's Framework (2026)
Every serious Apify actor runs on Crawlee — the open-source Node.js scraping library the Apify team built and maintains. If the marketplace covers your target you never need it. But the moment you need a custom scraper — your niche site, your unusual extraction, your protected target — Crawlee is how you build something production-grade instead of a fragile script.
I have written enough hand-rolled scrapers to know why Crawlee exists: retries, concurrency, session rotation, fingerprint management, request queues — the 80% of scraping that is not extraction logic. Crawlee handles all of it so your code is just "here is what to pull from this page." And because it is the same engine under Apify actors, code you write locally deploys to the Apify cloud with scheduling, datasets, and proxies included.
This guide covers the architecture, the three crawler classes, the request-queue pattern, a complete working example, proxies and anti-blocking, and deploying to the Apify platform. Related: [Apify web scraping tutorial](/blog/apify-web-scraping-tutorial), [Apify actors explained](/blog/apify-actors-explained), [Apify vs Crawl4AI](/blog/apify-vs-crawl4ai).
Quick Answer
Crawlee is Apify's open-source Node.js scraping framework. Choose Cheerio (fast) or Playwright (JS-heavy) crawlers, manage a RequestQueue, extract in requestHandler, push to Dataset. Handles retries, proxies, sessions. Develop locally, deploy as Apify actors.
Crawlee is Apify's open-source Node.js scraping framework. Choose Cheerio (fast) or Playwright (JS-heavy) crawlers, manage a RequestQueue, extract in requestHandler, push to Dataset. Handles retries, proxies, sessions. Develop locally, deploy as Apify actors.
- Crawlee is Apify's open-source Node.js scraping library — the engine under marketplace actors. Free, TypeScript-native, and production-grade.
- Three crawler classes: CheerioCrawler (fast HTML), PlaywrightCrawler (full browser), and PuppeteerCrawler — pick by how JS-heavy the target is.
- Core concepts: RequestQueue (the work list), requestHandler (your extraction logic), Dataset (structured output), and proxy/session management built-in.
- Crawlee handles the hard parts for you: retries, concurrency, session rotation, fingerprinting — the things hand-rolled scrapers get wrong.
- Local dev → Apify cloud: build/test locally with `npx crawlee create`, deploy as an actor, get scheduling + storage + proxies on the platform.
- The pattern that matters: enqueue links → extract data → pushData() to dataset → the queue drains and you get clean output.
1. What Crawlee is and when you need it
The engine under Apify actors:
- Crawlee: open-source (Apache 2.0) Node.js library for building reliable scrapers — free, MIT-adjacent permissive, TypeScript-first.
- When you need it: the marketplace has no actor for your target, or you need custom extraction/behavior a stock actor cannot do.
- When you do NOT: a marketplace actor already solves your target — run it, do not rebuild it.
- The stack position: Crawlee is the library; Apify platform is where you deploy it. You can also run Crawlee anywhere — it is not locked to Apify.
Crawlee = the open-source library powering Apify actors. Use it when no marketplace actor fits your target. It runs anywhere — Apify cloud is just the most convenient host.
2. The three crawler classes — pick by target
The first decision every project makes:
- CheerioCrawler: parses static HTML — blazing fast, tiny memory. Use for server-rendered pages where the data is in the HTML.
- PlaywrightCrawler: drives a real browser — for JS-rendered SPAs, infinite scroll, and interaction-heavy sites.
- PuppeteerCrawler: alternative browser driver — similar use case to Playwright, personal preference for most jobs.
- The rule: start with Cheerio (10-100x faster, cheaper). Escalate to Playwright only when the data is not in the initial HTML — check view-source first.
Start with Cheerio — it is 10-100x faster and cheaper than a browser. Only escalate to Playwright when view-source does not contain your data. Most over-engineered scrapers use a browser unnecessarily.
3. The core pattern — queue, handle, store
The mental model for every Crawlee scraper:
- RequestQueue: the work list — seed it with start URLs, enqueue more as you discover links.
- requestHandler: your function per page — extract data, call context.pushData(), enqueue follow-up requests.
- Dataset: structured output storage — pushData() writes records you export as JSON/CSV/Sheets.
- Concurrency/retry built-in: Crawlee parallelizes requests and retries failures automatically — you do not write that plumbing.
The pattern: seed queue → requestHandler extracts + enqueues more → pushData() stores → queue drains. Concurrency and retries are automatic — your code is just extraction logic.
4. A complete working example — a minimal Crawlee scraper
The shape of real code (sketch, not full listing):
- Install: `npm install crawlee` (or scaffold with `npx crawlee create my-scraper`).
- Shape: `new CheerioCrawler({ requestHandler: async ({ $, request, enqueueLinks, pushData }) => { ... } })` then `await crawler.run([startUrl])`.
- Inside the handler: use the Cheerio `$` to select fields → `pushData({ title, price, url })` → `enqueueLinks({ selector: "a.next" })` to paginate.
- For JS pages: swap to `PlaywrightCrawler` and use `page` (the browser page object) to `page.$$eval()` fields.
- Output: records land in `storage/datasets/default/` locally — on Apify they land in a dataset you can API/export.
requestHandler gives you Cheerio `$` or a Playwright `page`. Extract → pushData() → enqueueLinks() for pagination. Three calls cover 90% of scraper logic.
Apify
4.7Verified partnerWeb scraping and automation platform — Free / $49/mo Starter
5. Proxies, sessions, and not getting blocked
The production-grade parts Crawlee handles:
- ProxyConfiguration: rotate IPs — plug in Apify proxy groups or your own.
- SessionPool: persistent cookies/identity per session — survives login flows and reduces fingerprint flags.
- Fingerprinting: Playwright/Puppeteer crawlers get realistic browser fingerprints automatically via `browserPoolOptions`.
- Retry + blocked handling: failed requests retry; you can mark pages as blocked (`throw new SessionError`/`retryOnBlocked`) to rotate.
Crawlee gives you proxies, session pools, and browser fingerprints — use them on protected targets. Skipping proxy config on hard sites is how scrapers get instabanned.
6. Local dev → Apify actor — the deploy path
From laptop to scheduled cloud job:
- Develop locally: `npx crawlee create` scaffolds a project; run and iterate on your machine.
- Make it an actor: add an `INPUT_SCHEMA.json` so non-devs can configure it, plus the Apify SDK calls (Actor.init, Actor.pushData).
- Deploy: `apify push` uploads it to the platform → it becomes a runnable actor with its own API, schedule, and dataset.
- What you get free on Apify: cron scheduling, dataset storage, webhook triggers, proxy integration, logs, and monitoring — zero extra code.
Local Crawlee → `apify push` → a hosted actor with scheduling, datasets, webhooks, and proxies included. Your laptop code becomes a managed pipeline.
Apify
4.7Verified partnerWeb scraping and automation platform — Free / $49/mo Starter
7. Common pitfalls — what trips people up
The mistakes I see repeatedly:
- Using Playwright when Cheerio would do — 10x the cost/time for no benefit.
- Not enqueueing pagination — scraping page 1 and stopping.
- No proxy config on a protected target — instant blocks.
- Extracting in the wrong place — remember the handler runs per page; page-specific data belongs in requestHandler.
- Forgetting pushData — the most common "why is my dataset empty" cause.
Top mistakes: browser when Cheerio suffices, missing pagination enqueue, no proxies on protected sites, and forgetting pushData(). Check those first when output is empty or runs are slow.
8. Crawlee vs writing a raw Puppeteer/Cheerio script
Why not just hand-roll it:
- Raw scripts: you write retries, concurrency, sessions, queue management, fingerprinting — the plumbing that is 80% of the work.
- Crawlee: the plumbing is the library — you write only the extraction.
- When raw is fine: a 50-line one-off script for a trivial task. The moment it needs reliability, scale, or scheduling — Crawlee.
- The honest framing: Crawlee is the difference between a scraper that runs once and a scraper that runs every Monday for a year.
Crawlee is for scrapers that must run reliably forever, not scripts that run once. The moment you need retries/sessions/queues/scale, the library pays for itself.
Crawlee is how you build scrapers that survive production: request queues, retries, sessions, proxies, and fingerprints handled, so your code is just extraction. Develop locally, deploy to Apify for scheduling and storage. It is the difference between a script and a system.
Start with `npx crawlee create` and a CheerioCrawler on an easy target — get the queue/handle/store pattern down before reaching for Playwright. Deploy to Apify when you want it on a schedule. Related: Apify web scraping tutorial, Apify actors explained, Apify proxy guide.
Ready to get started?
Try the tools from this article risk-free. Free trials available.
FAQ
What is Crawlee?
Apify's open-source Node.js web scraping library — the framework under marketplace actors. Provides request queues, retries, concurrency, sessions, proxies, and datasets so you write only extraction logic. Free, TypeScript-native, runs anywhere.
Cheerio vs Playwright crawler — which do I use?
CheerioCrawler for server-rendered pages (data in initial HTML) — 10-100x faster. PlaywrightCrawler only when the page requires JS rendering or interaction. Check view-source: if your data is there, use Cheerio.
Is Crawlee free?
Yes — open-source under a permissive license, runs anywhere. Apify platform hosting (scheduling, datasets, proxies, managed runs) is the paid layer — but Crawlee itself costs nothing.
How do I deploy a Crawlee scraper?
Develop locally → add INPUT_SCHEMA.json and Apify SDK calls → `apify push` → it becomes a hosted actor with scheduling, datasets, webhooks, and proxies. Or run it on your own infra — it is not Apify-locked.
How do I handle pagination in Crawlee?
Inside requestHandler, call enqueueLinks() with the next-page selector (e.g. "a.next") — discovered pages enter the queue and get processed automatically. This is the core crawl pattern.
Does Crawlee handle anti-bot protection?
It provides the tools — proxy rotation, SessionPool, browser fingerprints, retry/blocked handling. On hard targets you configure them; on Apify platform the proxy tiers integrate directly.
Crawlee vs a plain Puppeteer/Cheerio script?
Raw scripts make you write retries, concurrency, sessions, and queue management yourself — 80% of real scraper code. Crawlee provides all of it so you write extraction only. For anything beyond a one-off, use Crawlee.
Tools mentioned in this article
Affiliate links — we may earn a commission at no cost to you.
Apify
4.7Verified partnerWeb scraping and automation platform — Free / $49/mo Starter
Alex Mercer
AI Marketing Strategist · 8+ years
Alex has tested 200+ AI marketing tools on real campaigns with $50,000+ in ad spend. He writes hands-on reviews based on actual usage data, not press kits. His work has helped 10,000+ marketers build AI automation stacks that generate real ROI.