Ahrefs Scraper By Market Category

Ahrefs Scraper — Market Leaders, Authority, Traffic and Keywords

Ahrefs Scraper By Market Category maps a whole market at once: it pulls the AhrefsTop leaderboard for a chosen category and country, then enriches each listed site with Domain Rating, backlinks and referring domains from Ahrefs' free Website Authority Checker — the leaderboard itself carries traffic only. The same run can add per-domain traffic and backlink records, keyword ideas, difficulty and SERP records, and AI-citation visibility. Every row is typed, normalized JSON — no HTML, no selectors, no parsing. Below you will find exactly which toggle produces which row, and what each one costs.

⚠️ You must supply your own captcha-solver key

This is the single prerequisite, and it stops the run cold, so it belongs at the top rather than in a footnote.

Ahrefs puts a Cloudflare Turnstile challenge in front of every free tool. This Actor cannot fetch authority, traffic, backlink, keyword, SERP or AI-visibility data without solving it, so it needs a solver key from CapSolver or 2Captcha — accounts you create and fund yourself at capsolver.com or 2captcha.com. Nothing is bundled.

Supply it either way:

• Paste it into 🔑 Captcha Solver API Key (captchaApiKey) — stored encrypted
• Or set the AHREFS_CAPTCHA_KEY environment variable on the Actor

If neither is present, the Actor logs an explicit error naming both options and returns before any collection starts. That check runs ahead of everything, so a leaderboard-only run is blocked too — even though the AhrefsTop pages themselves need no captcha. Nothing is pushed and nothing is charged.

Budget for it: every /v4 call is one solved token. Domain Rating enrichment issues one lookup, and therefore one solve, per leaderboard site up to maxSitesToEnrich. That is what the cost cap exists for.

What is Ahrefs Scraper By Market Category?

Ahrefs Scraper By Market Category is an Apify Actor that reads Ahrefs' free public SEO tools and shapes the responses into clean, typed records. Its centrepiece is the market leaderboard: pick a category and a country, and the Actor returns the ranked sites for that market with authority metrics attached to each one. Every other Ahrefs free tool remains available in the same run for any domain or keyword you add.

No Ahrefs account, login, cookie or subscription is used — these are the free tools any anonymous visitor can reach. The one external credential is the captcha-solver key described above.

• Build a ranked market leaderboard for any of 26 categories across 189 country choices, with each leader carrying rank, monthlyVisits, visitsChange and — once enriched — domainRating, backlinks and refdomains
• Scrape per-domain authority and traffic records: Domain Rating, referring domains, estimated monthly visits, traffic value, history, top pages, countries and keywords
• Scrape keyword records: keyword ideas with volumes, Keyword Difficulty plus its SERP, full SERP overviews, and where a domain ranks for a keyword
• Scrape AI-visibility records: citation counts across LLMs for a brand or keyword
• Export as JSON, CSV or Excel — no proxy management, no parsing

What data does Ahrefs Scraper By Market Category collect?

Nine record types, each written as its own dataset row and tagged with a type field. Every row carries data_captured_at and type; the rest of the row depends on which report produced it.

Data Type	Key Fields	JSON Field Names
Market leaderboard (type: "top_websites")	Ranked sites for a category and country, with traffic, movement and enriched authority	top_websites_category, top_websites_country, websites_rankings, website_rankings_trending_up, website_rankings_trending_down, marketLeaders, totalListed, enrichedCount, marketNote
Domain authority and traffic (type: "traffic")	Domain Rating, backlink and referring-domain counts, estimated monthly visits and traffic value, breakdowns	domain, mode, website_authority, website_traffic, website_traffic_history, website_traffic_top_pages, website_traffic_top_countries, website_traffic_top_keywords
Backlink overview (type: "backlinks")	Full decoded backlink and referring-domain overview	domain, data
Broken links (type: "broken_links")	Broken inbound and outbound links	domain, data
Keyword ideas (type: "keywords")	Keyword ideas and question ideas with volumes	keyword, country, data
Keyword difficulty (type: "keyword_difficulty")	Difficulty score and the SERP it was calculated from	keyword, country, keyword_difficulty, keyword_difficulty_results, keyword_difficulty_source, keyword_difficulty_shortage
SERP overview (type: "serp")	Decoded search results for a keyword	keyword, country, serp_results, serp_source, last_update
Keyword ranking (type: "keyword_ranking")	Where a specific domain sits for a keyword	keyword, domain, country, top_position, serp_results, serp_source, last_update
AI visibility (type: "ai_visibility")	AI citations across LLMs for a brand, with model and topic breakdowns	brand, data

Two details worth knowing before you map fields:

• costMontlyAvg is spelled exactly that way. It is Ahrefs' own typo in the upstream response, preserved rather than silently corrected so your mapping matches what actually arrives.
• When a block cannot be filled, its key is omitted, not nulled. A traffic row whose authority half failed has no website_authority key at all — it has website_authority_error instead. Read with .get(), not direct indexing.

Need more Ahrefs data?

MOZ Backlink Profile Analyzer answers the same authority question from a second vendor, which is what you want when a single provider's score is the thing being questioned. On the search-intent side, Google Search Autocomplete Scraper By Buyer Intent Filter and Quora Search Scraper With Author Lead Enrichment surface demand that keyword databases tend to flatten, and YouTube Search Scraper Keyword Rank Tracking covers the video SERP that Ahrefs' web tools do not.

How does Ahrefs Scraper By Market Category differ from the official Ahrefs API?

Ahrefs publishes an official paid API (Ahrefs API v3), and if you already hold a qualifying subscription, use it — it is the supported route, it returns Ahrefs' full index rather than the free tools' preview, and it comes with terms you can rely on commercially. This Actor exists for the case where you do not.

Feature	Ahrefs API v3	Ahrefs Scraper By Market Category
Ahrefs subscription and API access approval required	✅	❌
Consumption model	Unit-based, drawn against your plan	Apify row_result events, one per pushed row
Data scope	Ahrefs' full index across the documented endpoint set	What the free public tools return
AhrefsTop market leaderboard by category and country	Not part of the documented endpoint set	✅ Core report
Captcha-solver key required	❌	✅ CapSolver or 2Captcha
Output shape	Ahrefs' documented API schema	One normalized row per record type

Subscription tiers, endpoint scope and unit costs are set by Ahrefs and change — read the live API documentation at ahrefs.com/api before choosing between the two, rather than trusting a figure copied into a README.

Use the official API for production workloads, contractual guarantees and full-index depth. Use this Actor for market mapping, prospect qualification and ad-hoc research where a subscription is not in place.

Why do developers and teams scrape Ahrefs?

Ahrefs' free tools answer the two questions that start most SEO work — who leads this market, and how hard is it to enter — but the web interface answers them one domain at a time. Four groups get most of the value from having those answers as rows.

For SEO agencies and market analysts

Category mapping is the job this Actor was built for. Set category_top_websites to finance and country_top_websites to de, leave enrichWithAuthority on, and one run returns the German finance leaderboard with monthlyVisits from AhrefsTop plus domainRating, backlinks and refdomains fetched per site. Sorting marketLeaders by domainRating against monthlyVisits separates the sites that earned their traffic from the ones that inherited a brand — which is the difference between a realistic competitive set and a vanity one.

For marketers and brand teams

Re-run the same category and country on a schedule and diff marketLeaders by domain. visitsChange is parsed straight out of AhrefsTop's magnitude token, and website_rankings_trending_up and website_rankings_trending_down name the movers explicitly, so a challenger climbing your category arrives as a row rather than as something someone notices late. Pair it with include_ai_visibility on your brand name to watch citation counts across LLMs move in the same window — increasingly the discovery channel that classic rank tracking misses entirely.

For researchers and analysts

Everything returned is aggregate, domain-level and public: rankings, traffic estimates, backlink counts, keyword volumes. No personal data, no account, no login-gated content. That makes the output usable for market-structure and concentration work — how much of a category's estimated traffic the top ten sites hold, how that share shifts across countries — with a clean provenance story. Every row carries data_captured_at, so a longitudinal panel is just repeated runs kept in dated datasets.

For developers building data products

The Actor is a standard Apify run, so it schedules and webhooks like any other. Filter on passedFilter to feed only qualifying leaders into a downstream job, use maxSitesToEnrich as the cost dial rather than a row cap, and branch on type to route each record to its own handler. Because the nine record types share data_captured_at and type and never rename their keys, one table per type is a stable target for an ingest pipeline.

How to scrape Ahrefs by market category (step by step)

The Actor runs on Apify. Start it from the Apify Console or call it through the Apify API.

1. Get a CapSolver or 2Captcha API key — this is a prerequisite, not an optional extra
2. Open the Actor on Apify, click Try for free, and paste the key into 🔑 Captcha Solver API Key
3. Give the run something to do. Nothing is marked required in the schema, but a run needs at least one of: a value in 🌐 URL or Domain or 📚 Multiple URLs or Domains (Bulk), a value in 💡 Keyword or Brand, or 🗂️ Build The Market Leaderboard switched on. With none of the three, the run logs "No URL/Domain, Keyword, or Top-Websites selection — nothing to do." and exits
4. Pick your market under 🗂️ Market Category and 🌎 Market Country, then set 🔢 Max Sites To Enrich (Cost Cap) to the number of Domain-Rating lookups you are willing to pay for
5. Click Start, then export the dataset as JSON, CSV or Excel — or read it through the Apify API

Match each report to the input it needs or it is silently skipped: authority, traffic, backlinks and broken links need a domain; keyword ideas, difficulty and SERP need a keyword; keyword ranking needs a keyword and a domain; AI visibility needs a brand; the market leaderboard needs neither.

What to do when Ahrefs changes its structure

Eight of the nine reports read Ahrefs' /v4 JSON service rather than rendered markup, so a front-end redesign generally does not touch them. The market leaderboard is the exception — it is parsed out of AhrefsTop's server-rendered HTML and is the surface exposed to layout change. The Actor is maintained and the output keys stay put, so your field mapping does not move.

⬇️ Input

Every parameter below is copied from the Actor's published input schema. None of them is required — which makes the precedence rules in the pitfall note the thing to read first.

Parameter	Required	Type	Description	Example Value
keyword	No	string	The keyword (or brand, for AI Visibility) to research on Ahrefs.	"ai automation"
url	No	string	The single URL or domain to analyse on Ahrefs (e.g. make.com). For many targets at once, use the bulk field below.	"make.com"
country	No	string	Country to localise keyword, SERP and traffic results. 188 ISO alpha-2 values in the dropdown, from "af" to "zw". Default "us".	"de"
mode	No	string	Scope of the domain analysis: "subdomains" (whole site incl. subdomains) or "exact" (just the exact URL/host). Default "subdomains".	"subdomains"
urls	No	array	A list of URLs or domains to analyse on Ahrefs. Add them one by one or paste in bulk. When set, this overrides the single URL field above.	["make.com", "zapier.com"]
include_web_authority	No	boolean	Needs a Domain/URL. If checked, includes Domain Rating, backlinks & referring-domain counts. Default true.	true
include_traffic	No	boolean	Needs a Domain/URL. If checked, includes estimated monthly visits, traffic value, history, top pages/countries/keywords. Default true.	true
include_ai_visibility	No	boolean	Needs a Brand/Keyword. If checked, includes AI citations across LLMs, top cited domains/pages & topics. Default false.	false
include_keywords	No	boolean	Needs a Keyword and Country. If checked, includes keyword ideas + question ideas with volumes. Default false.	true
include_keywords_difficulty	No	boolean	Needs a Keyword and Country. If checked, includes the Keyword Difficulty score + ranking SERP. Default false.	true
include_keywords_ranking	No	boolean	Needs a Keyword, Domain and Country. If checked, includes where the domain ranks for the keyword. Default false.	false
include_serp	No	boolean	Needs a Keyword and Country. If checked, includes the full SERP overview for the keyword. Default false.	false
include_backlinks	No	boolean	Needs a Domain/URL. If checked, includes the backlink & referring-domain overview. Default true.	true
include_broken_links	No	boolean	Needs a Domain/URL. If checked, includes broken inbound & outbound links. Default false.	false
include_top_websites	No	boolean	Turn ON to pull the AhrefsTop market leaderboard for the Category + Country chosen below (this is the core of this Actor). Worldwide + All Categories returns the Top 1000; any category/country filter returns the Top 100. No Keyword or Domain needed. Default true.	true
category_top_websites	No	string	The market category whose leaderboard you want to map. 26 values: all, beauty-and-fitness, books-and-literature, business, computers-and-electronics, arts-and-entertainment, finance, food-and-drink, games, health, hobbies-and-leisure, home-and-garden, internet-and-telecom, jobs-and-education, law-and-government, news, people-and-society, pets-and-animals, real-estate, reference, science, shopping, social-networks, sports, travel-and-transportation, autos-and-vehicles. Default "all".	"finance"
country_top_websites	No	string	The country whose leaderboard you want to map (Worldwide = the global Top 1000). 189 values: worldwide plus the same ISO alpha-2 list. Default "worldwide".	"de"
enrichWithAuthority	No	boolean	Turn ON to look up each leaderboard site's Domain Rating, backlinks and referring-domain counts via Ahrefs' free Website Authority Checker. The leaderboard itself only carries traffic — this adds the missing authority metrics. Each lookup costs one solved captcha, so use the enrichment cap below. Default true.	true
maxSitesToEnrich	No	integer	Hard cap on how many top leaderboard sites get a Domain-Rating lookup. Enrichment runs from rank #1 down to this many sites; the rest are still listed but keep null authority. Protects you from a 1000-lookup bill. Minimum 1, maximum 1000. Default 100.	25
minDomainRating	No	integer	Only leaders with a Domain Rating at or above this pass the filter (marketLeaders[].passedFilter = true). Requires enrichment ON. Set 0 to keep every site. Minimum 0, maximum 100. Default 0.	70
minMonthlyVisits	No	integer	Only leaders whose estimated monthly organic visits (parsed from the leaderboard magnitude, e.g. '4.6B') are at or above this pass the filter. Set 0 to keep every site. Minimum 0. Default 0.	1000000
captchaApiKey	No	string	Optional. Your own CapSolver (or 2Captcha) API key used to solve the Cloudflare Turnstile that gates Ahrefs' free tools. Leave blank to fall back to the AHREFS_CAPTCHA_KEY environment variable. Stored encrypted.	"CAP-XXXXXXXXXXXX"
proxyConfiguration	No	object	The scraper starts on a DIRECT connection and only escalates to a datacenter, then residential proxy if Ahrefs blocks it. Pick a country here to localise residential fallback. Default {"useApifyProxy": true}.	{"useApifyProxy": true}

Beyond the 23 published parameters, the Actor also reads protocol, search_engine, captchaProvider, captchaToken and requestDelay from the input object if you send them. They are not in the schema and do not appear in the Console form; their fallbacks are "both", "Google", "capsolver", none, and one second between requests.

Common pitfalls

Nothing is required, so precedence is the mistake people make. Three rules, all read straight from the run entry point:

1. urls overrides url — it does not merge with it. If urls is a non-empty list, the single url value is dropped entirely. Both fields ship with a prefill, so clearing one and leaving the other is the intended edit.
2. keyword is a separate axis, not a target. It never adds a domain to analyse; it only enables the keyword, difficulty, SERP and AI-visibility reports. include_keywords_ranking is the one report that needs both a keyword and at least one URL, because it runs inside the per-URL loop.
3. With no URLs, no keyword and the leaderboard off, the run exits immediately having pushed nothing.

One toggle's code fallback differs from its schema default. include_top_websites shows as checked in the Console, but if you build the input JSON yourself and omit the key, the Actor's own fallback for a missing value is false. Send it explicitly.

maxSitesToEnrich is a cost dial, not a row cap. Sites beyond the cap still appear in marketLeaders — they just keep domainRating, backlinks and refdomains as null and authorityEnriched as false.

minDomainRating and minMonthlyVisits flag, they do not filter. Every leaderboard site stays in the array; the gates only set passedFilter. And if minDomainRating is above 0 while a site's DR is unknown — enrichment off, or the site is past the cap — that site cannot pass.

Example input

{
  "keyword": "ai automation",
  "url": "make.com",
  "urls": ["make.com", "zapier.com", "n8n.io"],
  "country": "de",
  "mode": "subdomains",
  "include_web_authority": true,
  "include_traffic": true,
  "include_ai_visibility": false,
  "include_keywords": true,
  "include_keywords_difficulty": true,
  "include_keywords_ranking": false,
  "include_serp": false,
  "include_backlinks": true,
  "include_broken_links": false,
  "include_top_websites": true,
  "category_top_websites": "finance",
  "country_top_websites": "de",
  "enrichWithAuthority": true,
  "maxSitesToEnrich": 25,
  "minDomainRating": 70,
  "minMonthlyVisits": 1000000,
  "captchaApiKey": "CAP-XXXXXXXXXXXX",
  "proxyConfiguration": { "useApifyProxy": true }
}

⬆️ Output

Typed, normalized JSON with a consistent shape per type across runs. Rows are pushed live as each report lands, so the dataset fills while the run is still going. Export as JSON, CSV or Excel, or read the dataset through the Apify API.

Ahrefs encodes its responses in an OCaml/Melange style — options as ["Some", x], results as ["Ok", x], enums as single-element arrays, URLs wrapped with signature noise. All of it is decoded recursively before it reaches you, so what lands in the dataset is plain JSON with null where Ahrefs sent None.

Every pushed row charges one row_result event, including rows that carry an error. There is no uncharged accounting row and no separate summary row. The one case that produces no row at all is a market with no AhrefsTop leaderboard: the first page comes back empty, a warning is logged, and nothing is pushed or charged. To keep only clean records:

clean = [r for r in items
         if not r.get("error")
         and not r.get("website_authority_error")
         and not r.get("website_traffic_error")]

Market leaderboard record

The core record. One row per run, holding the whole ranked market — not one row per site.

{
  "data_captured_at": "2026-07-25T09:14:02.481230+00:00",
  "type": "top_websites",
  "top_websites_category": "finance",
  "top_websites_country": "de",
  "websites_rankings": [
    { "rank": 1, "domain": "paypal.com", "search_traffic": "84.2M", "change": "+2.1M", "category": "Finance" },
    { "rank": 2, "domain": "sparkasse.de", "search_traffic": "31.7M", "change": "-480K", "category": "Finance" }
  ],
  "website_rankings_trending_up": [
    { "rank": 47, "domain": "trade-republic.com", "search_traffic": "4.6M", "change": "+1.2M", "position_change": "↑12" }
  ],
  "website_rankings_trending_down": [
    { "rank": 63, "domain": "comdirect.de", "search_traffic": "2.9M", "change": "-900K", "position_change": "↓8" }
  ],
  "marketLeaders": [
    {
      "rank": 1,
      "domain": "paypal.com",
      "monthlyVisits": 84200000,
      "visitsChange": 2100000,
      "category": "Finance",
      "domainRating": 93,
      "backlinks": 41823370,
      "refdomains": 384120,
      "authorityEnriched": true,
      "passedFilter": true
    },
    {
      "rank": 2,
      "domain": "sparkasse.de",
      "monthlyVisits": 31700000,
      "visitsChange": -480000,
      "category": "Finance",
      "domainRating": 81,
      "backlinks": 2914800,
      "refdomains": 41260,
      "authorityEnriched": true,
      "passedFilter": true
    },
    {
      "rank": 88,
      "domain": "finanzfluss.de",
      "monthlyVisits": 1840000,
      "visitsChange": 120000,
      "category": "Finance",
      "domainRating": null,
      "backlinks": null,
      "refdomains": null,
      "authorityEnriched": false,
      "passedFilter": false
    }
  ],
  "totalListed": 100,
  "enrichedCount": 25,
  "marketNote": "100 sites listed for finance/de; 25 enriched with Domain Rating (cap 25); 18 passed filter (minDR=70, minVisits=1000000); monthlyVisits/visitsChange parsed from AhrefsTop magnitude strings; sites beyond the enrichment cap keep null authority."
}

websites_rankings is the raw leaderboard as AhrefsTop published it — search_traffic and change are the display strings. marketLeaders is the same list re-keyed with those strings parsed into integers (monthlyVisits, visitsChange), authority merged in, and the filter verdict attached. Unparseable magnitude tokens become null.

Domain authority and traffic record

include_web_authority and include_traffic merge into a single traffic record per target. Enable only one and you get the same type with only that half present.

{
  "data_captured_at": "2026-07-25T09:15:40.117904+00:00",
  "type": "traffic",
  "domain": "make.com",
  "mode": "subdomains",
  "website_authority": {
    "domainRating": 91,
    "backlinks": 4182337,
    "refdomains": 38412,
    "dofollowBacklinks": 3120884,
    "dofollowRefdomains": 29760
  },
  "website_traffic": {
    "trafficMonthlyAvg": 2841500,
    "costMontlyAvg": 1934200
  },
  "website_traffic_history": [
    { "date": "2026-05-01", "organic": 2705300, "paid": 18400 },
    { "date": "2026-06-01", "organic": 2788100, "paid": 17250 },
    { "date": "2026-07-01", "organic": 2841500, "paid": 16980 }
  ],
  "website_traffic_top_pages": [
    { "url": "https://www.make.com/en/templates", "traffic": 184300, "keywords": 6120 }
  ],
  "website_traffic_top_countries": [
    { "country": "us", "traffic": 812400, "share": 0.286 },
    { "country": "de", "traffic": 214900, "share": 0.076 }
  ],
  "website_traffic_top_keywords": [
    { "keyword": "make automation", "position": 1, "volume": 74000, "traffic": 31200 }
  ]
}

mode travels with the traffic, backlink and broken-link requests only. The Website Authority lookup behind website_authority — and behind every marketLeaders enrichment — is sent the bare domain with no mode, so switching to exact changes traffic scope but never changes Domain Rating.

Keyword record

{
  "data_captured_at": "2026-07-25T09:16:22.902847+00:00",
  "type": "keyword_difficulty",
  "keyword": "ai automation",
  "country": "de",
  "keyword_difficulty": 68,
  "keyword_difficulty_source": "serp",
  "keyword_difficulty_shortage": null,
  "keyword_difficulty_results": [
    {
      "content_type": "organic",
      "data": { "url": "https://zapier.com/ai-automation", "title": "AI Automation", "domainRating": 92 },
      "pos": 1,
      "pos_with_metrics": 1
    },
    {
      "content_type": "ai_overview",
      "data": { "text": "AI automation combines workflow tools with model calls" },
      "pos": 0,
      "pos_with_metrics": null
    }
  ]
}

SERP entries are decoded rather than passed through raw: each carries content_type (organic, ai_overview, questions and so on), the decoded data, pos and pos_with_metrics. The same four keys appear inside serp_results on serp and keyword_ranking records, and inside top_position on keyword_ranking.

AI visibility record

ai_visibility, backlinks, broken_links and keywords share one shape: identity keys plus the whole decoded payload under data.

{
  "data_captured_at": "2026-07-25T09:17:05.338210+00:00",
  "type": "ai_visibility",
  "brand": "ai automation",
  "data": {
    "totalAiCitations": 1842,
    "citationsByModel": [
      { "model": "Chatgpt", "citations": 912 },
      { "model": "Perplexity", "citations": 604 },
      { "model": "Gemini", "citations": 326 }
    ],
    "topCitedDomains": [
      { "domain": "zapier.com", "citations": 214 },
      { "domain": "make.com", "citations": 168 }
    ],
    "topTopics": ["workflow automation", "ai agents", "no-code integration"]
  }
}

Error record

A report that comes back empty still produces a row — you see what went wrong instead of noticing an absence later.

{
  "data_captured_at": "2026-07-25T09:17:44.610338+00:00",
  "type": "backlinks",
  "domain": "example-with-no-index.tld",
  "error": "NoDataForTarget"
}

On traffic records the two halves fail independently, producing website_authority_error and/or website_traffic_error instead of a single error. In every case the failed block's key is absent rather than null.

How does Ahrefs Scraper By Market Category compare to other category trackers?

The closest Apify Store Actors by search intent are category and market trackers for other platforms rather than other Ahrefs tools, so the table below compares approach, stated as observed on their listings on 25 July 2026.

Feature	Ahrefs Scraper By Market Category	Category trackers observed on the Apify Store, 25 July 2026
Ranking source	AhrefsTop, a cross-web leaderboard covering 26 categories and 189 country choices	A single platform's own category listing — crawlerbros/google-workspace-marketplace-category-tracker documents 26 Workspace Marketplace category slugs; swerve/facebook-marketplace-scraper documents 20 Marketplace categories across 14 countries
Second-pass enrichment	✅ A live Website Authority lookup per leader adds domainRating, backlinks, refdomains	❌ Not documented on any of the three listings
Qualification signal	passedFilter computed per leader from minDomainRating and minMonthlyVisits, with every listed site retained	crawlerbros documents a maxItems cap of 1–200; swerve documents matchCityOnly as its only listed filter
Cost control	maxSitesToEnrich caps enrichment lookups independently of how many sites are listed	Row caps only, where documented
Prerequisites stated	Bring-your-own CapSolver or 2Captcha key, stated up front	crawlerbros states no account or API key is needed; swerve states a residential proxy is required and that datacenter IPs hit a login wall; scrapemint/polymarket-market-monitor published no README content readable on this date

If you are building an AI agent or RAG pipeline, the output-format row is the decision-maker — parsing HTML inside an agent loop is a reliability failure mode, not a feature.

How many results can you scrape with Ahrefs Scraper By Market Category?

There is no maxItems in this Actor. Leaderboard size is set by AhrefsTop: Worldwide + All Categories returns the Top 1000 across ten pages; any category or country filter returns the Top 100 on a single page. The only user-set ceiling is maxSitesToEnrich — minimum 1, maximum 1000, default 100 — and it caps Domain-Rating lookups, not listed sites.

Pagination is handled for you. For the unfiltered worldwide view the Actor walks pages 1 through 10 in order and stops early if a page returns no rows or comes back empty; filtered views are a single fetch. All of it lands in one top_websites row.

The per-domain and per-keyword reports scale differently. Domain reports — authority and traffic, backlinks, broken links, keyword ranking — run once per entry in urls. Keyword reports — keyword ideas, difficulty, SERP, AI visibility — run once per run, because they key off the single keyword field. Ten domains with authority, traffic and backlinks enabled produce thirty rows, plus one row per enabled keyword report, plus one leaderboard row.

Integrate Ahrefs Scraper By Market Category and automate your workflow

The Actor works with any language or tool that can send an HTTP request, because an Apify run is an HTTP call.

REST API integration

from apify_client import ApifyClient

client = ApifyClient("<YOUR_APIFY_TOKEN>")

run = client.actor("<YOUR_USERNAME>/ahrefs-scraper-by-market-category").call(run_input={
    "include_top_websites": True,
    "category_top_websites": "finance",
    "country_top_websites": "de",
    "enrichWithAuthority": True,
    "maxSitesToEnrich": 25,
    "minDomainRating": 70,
    "captchaApiKey": "<YOUR_CAPSOLVER_KEY>",
})

for rec in client.dataset(run["defaultDatasetId"]).iterate_items():
    if rec["type"] == "top_websites" and not rec.get("error"):
        for leader in rec["marketLeaders"]:
            if leader["passedFilter"]:
                print(leader["rank"], leader["domain"], leader["domainRating"], leader["monthlyVisits"])

Works in Python, Node.js, Go, Ruby, cURL. Branch on type to route each record to its own handler.

Automation platforms (n8n, Make, LangChain)

In n8n, the Apify node runs the Actor and returns dataset items directly; a Switch node on type splits the nine record types into separate branches, and an Item Lists node expands marketLeaders into one item per site. In Make, the Apify module supports run-and-wait, so a monthly market sweep can feed a Google Sheets or Airtable step without glue code. For LangChain and similar frameworks, wrap the Apify run endpoint as a tool — the response is already typed JSON, so there is no parsing step between the Actor and the model. Apify schedules handle recurring runs, and webhooks fire your endpoint when a run finishes.

Is it legal to scrape Ahrefs data?

Reading publicly accessible tool output is broadly treated as permissible where no authentication is bypassed, and this Actor reads only Ahrefs' free public tools — no account, no cookie, no paid session.

The output is aggregate, domain-level SEO data: Domain Rating, traffic estimates, backlink counts, keyword volumes, ranking positions. These are business and technical records, not personal data, so GDPR and equivalent personal-data regimes do not attach to this output the way they would to a profile scraper.

What does apply is contractual and database-related. The metrics are Ahrefs' own compiled estimates — a third party's database rather than an observation of the open web — which puts systematic reuse squarely inside Ahrefs' terms of service and database or unfair-competition rules. Solving a captcha to reach the tools is a deliberate access control, and worth raising with counsel rather than glossing over.

Consult legal counsel before reselling extracted Ahrefs metrics as a standalone product or republishing them as a competing dataset.

❓ Frequently asked questions

Does Ahrefs Scraper By Market Category work without an Ahrefs account?

Yes. It reads Ahrefs' free public tools, which need no account, no login and no subscription. It does require a CapSolver or 2Captcha key, because Ahrefs puts a Cloudflare Turnstile challenge in front of those tools — a third-party service you sign up for and fund separately, supplied via captchaApiKey or the AHREFS_CAPTCHA_KEY environment variable.

How often is the scraped data updated?

Every run fetches live. Nothing is cached between runs, and each record carries data_captured_at as a UTC ISO timestamp for exactly when it was collected. The underlying values follow Ahrefs' own refresh cadence for its free tools, which this Actor does not control — which is why repeated runs are the way to build a time series rather than assuming a value is current.

What happens when a domain has no Ahrefs data, or a keyword returns zero results?

You get a row, not silence. The record is still written with Ahrefs' own error string on error — or on website_authority_error / website_traffic_error for the split traffic record — and the run continues with the remaining reports. The failed block's key is omitted, so read with .get() rather than indexing. These rows charge a row_result event like any other, so filter them out at read time with the expression in the Output section. The one exception is a market that AhrefsTop does not list at all: the leaderboard fetch returns nothing, a warning is logged, and no row is pushed or charged.

Can I scrape data that sits behind an Ahrefs subscription?

No. Only the free public tools are read. Full backlink lists, anchor text, historical series, Site Audit crawls and Rank Tracker projects are subscription features and are not returned. What you get is the free tools' preview of Ahrefs' index — the same values any anonymous visitor sees on ahrefs.com.

Why did my run finish immediately without pushing anything?

Two causes, and the run log names both. Either no captcha-solver key was found, in which case the Actor logs an error naming captchaApiKey and AHREFS_CAPTCHA_KEY and returns before collecting anything; or you gave it nothing to do — no url, no urls, no keyword, and include_top_websites off — in which case it logs "No URL/Domain, Keyword, or Top-Websites selection — nothing to do." and exits.

Which toggles do nothing without a keyword or a domain?

include_keywords, include_keywords_difficulty, include_serp and include_ai_visibility are inert without keyword — they are skipped silently, not errored. include_web_authority, include_traffic, include_backlinks and include_broken_links are inert without at least one URL. include_keywords_ranking needs both, and is the only report that pairs your single keyword against each URL in turn. include_top_websites needs neither, and is the one report that runs on an otherwise empty input.

Does Ahrefs Scraper By Market Category work for AI agent workflows and LLM pipelines?

Yes. It is callable as a standard HTTP endpoint through the Apify API, so LangChain, CrewAI, n8n or a hand-written tool definition can invoke it and receive typed JSON with no parsing step. Branch on type and the nine record shapes map cleanly onto nine tool responses.

How does the Actor handle Ahrefs' anti-bot defences?

Two defences, handled separately. Turnstile is solved with your CapSolver or 2Captcha key, once per /v4 call, with one automatic re-solve if Ahrefs rejects the token. IP blocking uses a sticky escalation ladder that starts on the cheapest rung: every run begins on a direct connection with no proxy at all, and only climbs to an Apify datacenter proxy, then Apify residential, when Ahrefs returns a 403, 429, 503, a connection error, or a non-JSON body where JSON was expected. On the residential rung it retries up to three times before giving up on that request. Once the ladder climbs, later requests stay on that rung, and the closing log line names the tier the run finished on. Requests carry a browser-grade TLS fingerprint and a rotating X-Client-Version header discovered from a live tool page.

How does this compare to other market-category scrapers on Apify?

Checked on the Apify Store on 25 July 2026, the nearest listings track a single platform's own categories. crawlerbros/google-workspace-marketplace-category-tracker documents 26 Workspace Marketplace category slugs with a maxItems cap of 1–200 and states that no account or API key is needed. swerve/facebook-marketplace-scraper documents 20 Marketplace categories across 14 countries and states that a residential proxy is required. scrapemint/polymarket-market-monitor published no README content readable on that date. None of the three documents a second-pass authority enrichment. This Actor's difference is that the leaderboard is cross-web rather than per-platform, and that each listed leader gets a live Domain Rating, backlink and referring-domain lookup merged onto its row.

Does it return data in a format LLMs can use directly?

Yes. Every record is typed, normalized JSON with stable field names across runs, decoded out of Ahrefs' internal OCaml/Melange encoding before it reaches the dataset. No HTML, no selectors, no parsing. Pass a row straight into an LLM context window, index it into a vector store, or hand it to an agent tool.

Can I use it without managing proxies?

Yes. The ladder is automatic — direct, then Apify datacenter, then Apify residential with up to three retries — and you never create a proxy account or rotate an IP. Setting a country in proxyConfiguration localises the residential rung. The captcha solver is the one external account you do have to set up yourself.

What happens when Ahrefs changes its structure or blocks the scraper?

The Actor is maintained and the output keys stay stable — field names and types do not change on your end. Eight of the nine reports read the /v4 JSON service, so a front-end redesign generally does not affect them; the AhrefsTop leaderboard is HTML-parsed and is the one report exposed to layout change. Per-report errors are captured on the record itself, so a single broken report never takes the run down with it.

🔗 Related scrapers

Scraper Name	What it extracts
MOZ Backlink Profile Analyzer	Backlink profile and authority metrics from a second SEO data source
Google Search Autocomplete Scraper By Buyer Intent Filter	Autocomplete suggestions filtered for commercial intent
Quora Search Scraper With Author Lead Enrichment	Question-intent results with author details
YouTube Search Scraper Keyword Rank Tracking	Video search results and keyword rank positions
Google News Scraper Real URL Source Metadata	News articles with resolved publisher URLs and source metadata

💬 Your feedback

Found a bug, or need an Ahrefs free tool or leaderboard field that is not yet covered? Open an issue on the Actor's Issues tab. Reports that include the exact input JSON and the category, country or domain you targeted are the fastest to reproduce and fix.