Apify Datasets & Storage: Manage Scraped Data Like a Pro

Quick answer: how does Apify storage work?

Every Apify actor run automatically creates a dataset containing the scraped results. Datasets are stored in Apify's cloud and accessible from your dashboard. You can export datasets as CSV, JSON, JSONL, Excel, or HTML. You can access datasets via Apify's API from any programming language. Datasets persist until you delete them — you control data retention.

For marketing teams, this means scraped data is immediately available for export to CRM, spreadsheets, or data pipelines without any infrastructure setup.

Understanding Apify storage types

Apify has three storage types. Here is what each is for:

• Datasets: Tabular data (rows and columns). This is where scraped results go. Each actor run creates a dataset. Best for structured data like product lists, business directories, social media posts
• Key-value stores: Key-value pairs. Used for storing files, screenshots, or non-tabular data. Each actor run creates a KVS. Best for storing raw HTML, screenshots, or configuration files
• Request queues: URLs to be scraped. Used internally by actors to manage crawling. You typically do not interact with these directly
• All storage is cloud-based: No local files to manage. Access from anywhere via dashboard or API
• Storage retention: Datasets persist until you delete them. Set retention policies to auto-delete old data

Exporting data from Apify datasets

Apify supports multiple export formats. Here is when to use each:

• CSV (Comma-Separated Values): Best for Excel, Google Sheets, CRM import. Flat tabular format. One row per scraped item
• JSON (JavaScript Object Notation): Best for API integrations, Node.js/Python scripts, and webhooks. Preserves nested data structures
• JSONL (JSON Lines): Best for large datasets. One JSON object per line. Easier to process incrementally than full JSON
• Excel (XLSX): Best for non-technical users. Opens directly in Excel with formatting. Limited to 1M rows
• HTML: Best for viewing data in a browser. Renders as a table. Useful for quick visual review
• XML: Best for legacy system integrations that require XML format
• To export: Go to the actor run → Dataset tab → choose format → Download. Or use the API: GET /v2/datasets/{datasetId}/items?format=csv

Cleaning scraped data

Raw scraped data is rarely ready for immediate use. Clean it before importing to your CRM or marketing tools:

• Remove duplicates: Sort by a unique identifier (URL, phone number, email) and remove duplicates. Apify datasets can be deduplicated via the API
• Filter empty fields: Remove rows with missing critical fields (e.g., no email or no phone for lead generation)
• Standardize formats: Normalize phone numbers (+1-555-123-4567), URLs (add https://), and addresses
• Validate emails: Run email validation (Hunter.io, NeverBounce) on scraped emails before importing to your CRM. Invalid emails hurt deliverability
• Categorize: Add a category column based on scraped data (e.g., "high priority" for businesses with 4.5+ rating and 100+ reviews)
• Remove test data: Check the first and last 10 rows for test or placeholder data that the scraper may have picked up
• Trim whitespace: Remove leading/trailing spaces from text fields. Common issue with scraped data
• Encoding: Ensure special characters (accents, emojis) are properly encoded in UTF-8

API access to datasets

Apify datasets are accessible via REST API. This enables automated data pipelines:

• Get all items: GET /v2/datasets/{datasetId}/items — returns all data in the dataset
• Get items with format: GET /v2/datasets/{datasetId}/items?format=csv — returns CSV format
• Get dataset info: GET /v2/datasets/{datasetId} — returns metadata (item count, size, created date)
• Pagination: Add ?limit=100&offset=0 for paginated access to large datasets
• Filter: Add ?fields=name,phone,email to return only specific fields
• Clean: Add ?clean=true to remove empty fields and normalize data
• Authentication: Add ?token=YOUR_API_TOKEN (find in Apify Settings → API)
• Rate limits: 100 requests per minute for free tier, higher for paid plans

Webhook integrations for automated pipelines

Webhooks let you trigger actions when an actor run finishes. This enables fully automated data pipelines:

• Actor run webhook: Triggered when an actor run completes. Sends the run ID and dataset URL to your endpoint
• Setup: Actor → Settings → Webhooks → Add webhook → Enter your endpoint URL
• Common integrations: Send data to Zapier/Make.com → push to Google Sheets, HubSpot, Slack, or Airtable
• Example pipeline: Apify scrape completes → webhook triggers Zapier → data cleaned in Zapier → pushed to HubSpot CRM → Slack notification sent
• Example pipeline 2: Apify scrape completes → webhook triggers Make.com → data pushed to Airtable → email alert sent
• Example pipeline 3: Apify scrape completes → webhook triggers AWS Lambda → data processed → stored in S3 → QuickSight dashboard updated
• Error handling: Webhooks retry 3 times on failure. Check Apify logs if data does not arrive at your endpoint

Building a recurring data pipeline

For ongoing data collection (daily price monitoring, weekly lead generation), build a recurring pipeline:

• Step 1 — Schedule the actor: Apify → Actor → Schedules → Create schedule (e.g., every 24 hours)
• Step 2 — Set up webhook: Actor → Webhooks → Add webhook to your Zapier/Make.com endpoint
• Step 3 — Process data: Zapier/Make.com cleans, filters, and routes data to your destination (CRM, spreadsheet, dashboard)
• Step 4 — Alert: Send Slack/email notification when new data arrives
• Step 5 — Review: Check data quality weekly. Adjust scraper parameters if data quality degrades
• Step 6 — Archive: Export and archive monthly datasets for historical analysis
• Step 7 — Monitor costs: Check Apify usage monthly. Optimize by reducing frequency or result count if costs increase

Storage costs and limits

Apify storage is included in plans with these limits:

• Free tier: 14-day data retention. Datasets auto-deleted after 14 days. 0.5 GB storage
• Starter ($49/month): 7-day data retention on compute, 30-day on datasets. 2 GB storage
• Scale ($499/month): 30-day data retention. 50 GB storage
• Storage overage: $0.01 per GB per month for additional storage
• Cost optimization: Export datasets immediately after scraping and delete from Apify to save storage. Use Apify as a temporary pipeline, not permanent storage
• Best practice: Export to your own database, S3, or Google Sheets for long-term storage. Use Apify only for scraping and initial processing