🔍 Similarweb Scraper

📊 Website intelligence for any domain in seconds. Start with one website, choose traffic/rankings, similar sites, or WHOIS + homepage keywords, then export the results as JSON, CSV, Excel or any other format Apify supports.

💎 What is Similarweb Scraper?

Similarweb Scraper is a fast, captcha-resilient web scraper that pulls public website intelligence without requiring a Similarweb account, login, or API key. Behind the scenes it uses Chrome TLS / JA3 fingerprints via curl_cffi and routes every request through a fresh Apify Residential proxy session. When Similarweb refuses a protected source, the Actor stops quickly and does not save guessed or fallback data as base_data.

You can start with a single domain, then scale to a whole list when you are ready. Pick one dataset mode per run and the Actor returns clean records ready to drop into a spreadsheet, BI tool, warehouse, or AI agent.

🚀 What can Similarweb Scraper do?

• 🗂️ Choose one of three dataset modes for each run:
  • 📊 Base data — global / country / category ranks, monthly visits, bounce rate, pages per visit, time on site, traffic-source split (direct, search, referral, social, paid, mail), top organic keywords with volume and CPC, AI traffic share per LLM (ChatGPT / Claude / Gemini / Perplexity / Copilot).
  • 🪞 Similar sites — competitors and alternatives with their traffic, category and ranking.
  • 🆔 AITDK — WHOIS via RDAP (registrar, registration / expiration dates, name servers, EPP status, DNSSEC) plus on-page keyword density analysis of the domain's homepage.
• ⚡ Start small or run in bulk — one domain is enough for a test run; larger batches process up to 10 domains concurrently by default.
• 🛡️ Captcha-resilient — uses available public sources directly, without captcha solving. If a protected Similarweb source returns a challenge, the Actor skips that source instead of saving unreliable data.
• 🔄 Evidence-driven IP rotation — requests are served from a pool of Apify Residential proxy exits. An exit is dropped and replaced the moment it returns a block signal (challenge, 403, 429, proxy or timeout error), so a blocked address still costs at most one attempt, while healthy exits are reused instead of paying a new TLS handshake every call.
• 🌐 Three input formats — accepts example.com, www.example.com, or https://example.com. The domain is extracted automatically.

☁️ Remember the Apify platform

Running this Actor on Apify gives you everything that comes with the platform out of the box: managed Residential proxies with global exit IPs, scheduling (run hourly / daily / weekly), free storage in Apify Datasets with export to JSON / CSV / Excel / JSONL / XML / RSS, webhooks and integrations (Make, Zapier, n8n, Google Sheets, Slack, Airtable, Pipedream), and a REST API + Python / JavaScript SDKs to plug results into your own pipelines.

🗝️ What data can this Actor extract?

Field group	Examples
Rankings	Global rank · country rank · category rank
Engagement	Total visits · monthly visits (3 months) · bounce rate · pages / visit · time on site
Traffic sources	Direct · search (organic / paid) · referral · social (organic / paid) · affiliate · display ads · GenAI · mail
AI traffic share	ChatGPT · Claude · Gemini · Perplexity · Copilot — current + 3-month history
AI traffic detail	Ranked list of every named AI referrer (incl. Grok, DeepSeek) · absolute AI-referred visits
Top keywords	Keyword · estimated value · search volume · CPC
Country breakdown	Top countries with share + monthly visit estimates per country
Similar sites	Up to 20 related sites with traffic, ranks, category, thumbnails and a similarity score
Related apps	Mobile apps associated with the site — platform, store ranking and link
WHOIS (via RDAP)	Registrar · IANA ID · abuse contact · registration / expiration / transfer dates · name servers with glue records · DNSSEC DS records · EPP status
Keyword density	Top-20 non-stopword terms from the homepage with count and density, in any writing system
Assets	Desktop / mobile screenshots · favicon
Coverage	dataCoverage · isSmall — how much Similarweb actually knows about the domain

🎯 How to use Similarweb Scraper

1. Click Try for free on the Actor's Apify Store page.
2. In the Domains field, enter one website to test, or paste a larger list later — one per line, any format (example.com, www.example.com, or https://example.com).
3. Pick exactly one Dataset to fetch. Start with base_data for the main traffic, rank, engagement, keyword and AI-share overview, or choose another mode when needed.
4. Click Start. A one-domain test run is fine; there is no 10-domain minimum. aitdk takes longer than the other modes because it also fetches RDAP and the homepage.
5. When the run finishes, open Storage → Dataset and export to JSON, CSV, Excel, JSONL, XML or RSS. Or pull the results through the API: https://api.apify.com/v2/datasets/{dataset_id}/items.

📥 Input

The form has two visible fields only — everything else has sensible defaults:

Field	Type	Default
domains	array	20 common domains are prefilled for a quick bulk test
datasetMode	enum	base_data — one of base_data, similar_sites, aitdk

Example input

{
  "domains": ["openai.com", "google.com", "booking.com"],
  "datasetMode": "base_data"
}

API callers may also pass domains through domainsText (newline, comma, or semicolon separated), legacy urls / websites, or Apify-style startUrls request objects. At least one domain source must be provided.

📤 Output

Each domain produces one dataset item. Each item conforms to the dataset schema and is rendered in the Apify Console views that match the selected dataset mode: 📊 Overview · 🪞 Similar sites · 🚦 Traffic sources · 💫 Engagement · 🤖 AI traffic share · 🆔 AITDK (WHOIS and keywords).

Example item (abridged)

{
  "domain": "openai.com",

  "rankGlobal": 207,
  "country": "US",
  "countryRank": 306,
  "category": "ai_chatbots_and_tools",
  "categoryRank": 6,
  "title": "OpenAI",
  "totalVisits": 195737812,
  "bounceRate": 0.5937,
  "pagesPerVisit": 2.59,
  "timeOnSite": 138.72,

  "socialTraffic":   0.0287,
  "searchTraffic":   0.2154,
  "directTraffic":   0.3840,
  "referralTraffic": 0.1038,
  "displayAdsTraffic": 0.0013,
  "genAiTraffic": 0.2358,

  "aiTrafficShareChatgpt": 0.8825,
  "aiTrafficShareClaude":  0.0029,
  "aiTrafficShareGemini":  0.0106,

  "topKeywords": [
    {"keyword": "chatgpt",  "estimatedValue": 20907500.0, "searchVolume": 173339160.0, "cpc": 0.14},
    {"keyword": "chat gpt", "estimatedValue":  5688810.0, "searchVolume":  95011780.0, "cpc": 0.14}
  ]
}

📉 Domains Similarweb barely covers

Not every domain is ranked. Small sites are still returned, but with no global rank and no traffic-source split, and a handful of domains have no Similarweb record at all. Every base_data row carries a dataCoverage value so you can tell those apart from a failed scrape:

Value	Meaning
full	Ranked domain, the whole record is populated.
small_site	Below Similarweb's ranking threshold. Title, description, category, monthly visit estimates and screenshots are still present; rank and traffic split are not.
no_data	Similarweb returned an empty record. The row exists only to confirm the domain was checked.

The run summary reports these under sourceCoverage (ranked, smallSites, noData) — separate from dataQuality, because they describe how much Similarweb knows about a domain, not whether the scrape succeeded. A small_site row is a complete result: it already contains every value the source provides.

🔁 Finishing a bulk run

On large lists a few domains can come back empty because an upstream source was temporarily unavailable. You do not have to guess which ones or repeat the whole batch: every run writes the complete list of domains that produced no dataset item to the default key-value store under FAILED_DOMAINS.

$curl "https://api.apify.com/v2/key-value-stores/<STORE_ID>/records/FAILED_DOMAINS"

{
  "selectedDataset": "base_data",
  "count": 3,
  "domains": ["example.com", "example.org", "example.net"],
  "domainsText": "example.com\nexample.org\nexample.net",
  "failures": [
    {"domain": "example.com", "reason": "No usable source data returned"}
  ],
  "invalidEntries": []
}

Paste domainsText into the Domains field, or send domains straight back as the Actor input, and only the missing part of the list is scraped. The record is written after every run, so count: 0 is a positive confirmation that nothing is missing. invalidEntries lists input lines that were rejected before any request was made — those are reported separately because re-running them would fail again.

🔗 Integrate Similarweb Scraper anywhere

Apify Actors run on a REST API — every run, dataset and webhook is addressable from your code:

# Trigger a run from anywhere
curl -X POST "https://api.apify.com/v2/acts/<USER>~similarweb-scraper/runs?token=<API_TOKEN>" \
     -H "Content-Type: application/json" \
     -d '{"domains": ["openai.com"], "datasetMode": "base_data"}'

# Read results from the run's default dataset
curl "https://api.apify.com/v2/datasets/<DATASET_ID>/items?format=json"

Or use the official Python and JavaScript clients.

❓ FAQ

Can I test it with only one domain? Yes. The Actor needs at least one domain, not ten. The default UI example uses openai.com with base_data, so a new user can click Start immediately.

🔑 Do I need a Similarweb account or API key? No. This Actor talks to Similarweb's public SPA endpoint directly. No login, no API key, and no scraping the captcha-gated in-depth pages.

🆕 Is the data fresh? Yes — it's the same JSON Similarweb's UI loads. The snapshotDate field on every record tells you exactly which month it represents. Similarweb refreshes its traffic data monthly.

⏰ Can I run this on a schedule? Yes — open the Actor in Apify Console, go to Schedules and pick hourly, daily, weekly or a custom cron. Combine with webhooks to push fresh data into Google Sheets, Slack, Make, Zapier or your own backend automatically.

⚖️ Is web scraping legal? Public web pages are generally legal to scrape, but you must respect copyright, terms of service, and personal-data protection laws (GDPR in the EU and similar regulations elsewhere). This Actor only extracts publicly visible data — no personal data is collected. See Apify's legal blog for details.

💬 Support

• Found a bug or have a feature request? Open an issue in the Actor's Issues tab on Apify Console.
• Questions about Apify itself? Visit docs.apify.com or the Apify Discord community.

📝 Changelog

See CHANGELOG.md for the full release history. The Actor follows Semantic Versioning.