# 🧠 Smart Article Extractor (`scraper-engine/smart-article-extractor`) Actor

Smart Article Extractor by Extract clean article content from any webpage, including titles, authors, publish dates, text, images, metadata, and links. Remove ads and clutter automatically. Export structured data to JSON, CSV, Excel, or XML for research, AI training, content analysis, and archiving.

- **URL**: https://apify.com/scraper-engine/smart-article-extractor.md
- **Developed by:** [Scraper Engine](https://apify.com/scraper-engine) (community)
- **Categories:** News, AI, Automation
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $4.99 / 1,000 results

This Actor is paid per event and usage. You are charged both the fixed price for specific events and for Apify platform usage.

Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## 🧠 Smart Article Extractor — News & Blog Scraper

> **One-paragraph summary:** Smart Article Extractor is an Apify Actor that bulk-extracts clean article content — **title, author, publish date, full text, summary, images, videos, in-body links and rich metadata** — from any news site, blog or sitemap. Point it at a homepage / section / topic URL and it will discover, classify and extract every article automatically using a BFS crawler, sitemap scanning, and configurable URL-shape heuristics.

***

### 🚀 Why Choose Us?

| Feature | Smart Article Extractor | Typical 1-URL article scraper |
|---|---|---|
| **Bulk discovery (BFS crawler)** | ✅ Yes | ❌ One URL at a time |
| **Sitemap & robots.txt scanning** | ✅ Built-in | ❌ |
| **Sub-domain / sub-path scoping** | ✅ Per Start URL | ❌ |
| **`onlyNewArticles` cross-run dedup** | ✅ Per-domain & global | ❌ |
| **Date filters** (`dateFrom`, `lastDays`, `mustHaveDate`) | ✅ All three | ⚠️ Limited |
| **Anti-block proxy fallback** (none → DC → RES) | ✅ Automatic | ❌ |
| **Optional Playwright rendering** | ✅ Toggle | ❌ |
| **Extend-output Python hook** | ✅ Inline snippet | ❌ |
| **Live dataset push + state KVS** | ✅ | ⚠️ |

***

### 🔥 Key Features

- 📰 **Clean article extraction** — trafilatura + BeautifulSoup combo for high recall.
- 🌐 **Bulk discovery** — drop a homepage URL and the actor discovers articles via BFS.
- 🗺️ **Sitemap & robots.txt** — automatic `Sitemap:` parsing + common candidates.
- 🛡️ **Smart proxy fallback** — starts direct, then datacenter, then residential.
- 🎭 **Headless browser mode** — Playwright + Chromium for JS-heavy or protected sites.
- 🧠 **Cross-run memory** — `onlyNewArticles` and `onlyNewArticlesPerDomain`.
- 🪜 **Depth / page / article caps** — never over-crawl.
- 📅 **Date filters** — `dateFrom`, `onlyArticlesForLastDays`, `mustHaveDate`.
- 🛠️ **`extendOutputFunction`** — inject your own Python `extend(soup, article, html)`.
- 💾 **Save HTML / snapshots** — full HTML in-record or as KVS link, PNG screenshots.

***

### 📥 Input

| Field | Type | Default | Description |
|---|---|---|---|
| `startUrls` | array | required | Homepages, sections, topic pages — used as crawl seeds. |
| `articleUrls` | array | `[]` | Direct article URLs to extract (no discovery needed). |
| `onlyNewArticles` | boolean | `false` | Skip URLs already seen in any previous run. |
| `onlyNewArticlesPerDomain` | boolean | `false` | Per-domain dedup memory. |
| `onlyInsideArticles` | boolean | `true` | Enqueue only same-domain links from articles. |
| `onlySubdomainArticles` | boolean | `false` | Restrict to URLs sharing the Start URL path prefix. |
| `enqueueFromArticles` | boolean | `true` | Discover further links inside extracted articles. |
| `crawlWholeSubdomain` | boolean | `true` | Treat any same-subdomain link as a category candidate. |
| `scanSitemaps` | boolean | `true` | Discover articles from `robots.txt` and common sitemap paths. |
| `useGoogleBotHeaders` | boolean | `true` | Identify as Googlebot. |
| `useBrowser` | boolean | `false` | Render with headless Chromium. |
| `scrollToBottom` | boolean | `false` | Force lazy-loaded content (browser mode only). |
| `mustHaveDate` | boolean | `false` | Drop articles with no detectable date. |
| `dateFrom` | string (ISO date) | — | Earliest article date. |
| `onlyArticlesForLastDays` | integer | — | Convenience cut-off. |
| `minWords` | integer | `150` | Reject short articles. |
| `maxDepth` | integer | `2` | BFS depth. |
| `maxPagesPerCrawl` | integer | `50` | Hard cap on fetched pages. |
| `maxArticlesPerCrawl` | integer | `25` | Hard cap on saved articles. |
| `maxArticlesPerStartUrl` | integer | `25` | Cap per Start URL. |
| `isUrlArticleDefinition` | object | see schema | URL-shape heuristic. |
| `linkSelector` | string | — | CSS selector restricting where links are collected from. |
| `pseudoUrls` | array | `[]` | Custom URL patterns for category pages. |
| `sitemapUrls` | array | `[]` | Explicit sitemap URLs (skip auto-discovery). |
| `saveHtml` | boolean | `false` | Include raw HTML in the dataset record. |
| `saveHtmlAsLink` | boolean | `false` | Save HTML to KVS and put a link in the record. |
| `saveSnapshots` | boolean | `false` | PNG screenshot (browser mode only). |
| `extendOutputFunction` | string | — | Python snippet — must define `extend(soup, article, html) -> dict`. |
| `proxyConfiguration` | object | `{useApifyProxy: false}` | Default = no proxy; auto-fallback to DC → RES if blocked. |

**Example input:**

```json
{
  "startUrls": [{ "url": "https://www.theguardian.com" }],
  "onlyArticlesForLastDays": 2,
  "minWords": 150,
  "maxArticlesPerCrawl": 5,
  "useGoogleBotHeaders": true,
  "scanSitemaps": true,
  "proxyConfiguration": { "useApifyProxy": false }
}
```

***

### 📤 Output

Each pushed record contains:

| Field | Type | Description |
|---|---|---|
| `url`, `loadedUrl` | string | Original / resolved URL. |
| `domain`, `loadedDomain` | string | Bare host. |
| `referrer`, `startUrl` | string | Where the link was discovered. |
| `depth` | integer | BFS depth at time of crawl. |
| `title`, `softTitle` | string | Best-effort headline. |
| `date` | string (ISO) | Publication date if found. |
| `author` | array | Author URL(s) or name(s). |
| `publisher`, `copyright`, `lang`, `favicon`, `canonicalLink` | string | Site metadata. |
| `description`, `keywords` | string | Meta description / keywords. |
| `tags` | array | `article:tag` values. |
| `image` | string | Hero / OG image URL. |
| `videos` | array | `<video> / <iframe> / <source>` URLs. |
| `links` | array of `{text, href}` | Inner-body links. |
| `wordCount` | integer | Word count of the extracted text. |
| `text` | string | Cleaned article body. |
| `html` | string | Full HTML (only if `saveHtml` / `saveHtmlAsLink`). |
| `screenshotUrl` | string | KVS link (only if `saveSnapshots` + `useBrowser`). |

**Example output (truncated):**

```json
{
  "url": "https://www.theguardian.com/lifeandstyle/2026/may/21/how-often-should-you-go-to-the-toilet…",
  "domain": "theguardian.com",
  "title": "How often should you go to the toilet?…",
  "date": "2026-05-21T04:00:02.000Z",
  "author": ["https://www.theguardian.com/profile/sarahphillips"],
  "publisher": "the Guardian",
  "wordCount": 1620,
  "text": "Think balance, diversity and routine. \"Our gut is a complex machine,\" says…",
  "image": "https://i.guim.co.uk/img/media/…"
}
```

***

### 🚀 How to Use (Apify Console)

1. Log in at <https://console.apify.com> → **Actors**.
2. Open **Smart Article Extractor**.
3. Configure inputs (Start URLs, date filters, caps, proxy).
4. Click **Start**.
5. Watch logs in real time — the actor prints a per-article live feed.
6. Open the **Output** tab once the run completes.
7. Export to JSON / CSV / XLSX or wire to a webhook.

***

### 🤖 Use via API / MCP

```bash
curl -X POST "https://api.apify.com/v2/acts/<USERNAME>~smart-article-extractor/run-sync-get-dataset-items?token=$APIFY_TOKEN" \
     -H "Content-Type: application/json" \
     -d '{
       "startUrls": [{"url": "https://www.theguardian.com"}],
       "maxArticlesPerCrawl": 5,
       "onlyArticlesForLastDays": 2,
       "proxyConfiguration": {"useApifyProxy": false}
     }'
```

MCP-server tool name: `smart-article-extractor`.

***

### 💡 Best Use Cases

- 📰 News monitoring on a topic / publisher
- 📊 NLP / sentiment / summarisation datasets
- 🏛️ Brand or competitor coverage tracking
- 🔍 SEO / SERP enrichment with full article text
- 📚 Knowledge-base construction for RAG / LLMs
- 🗞️ Press-clipping archives

***

### 💰 Pricing

Pay-per-usage. You only pay the Apify platform charges (compute time + proxies + transfer). No separate developer fee.

***

### ❓ Frequently Asked Questions

**Q: Why are some articles skipped?**\
A: They failed at least one filter — date cut-off, `mustHaveDate`, `minWords`, or `onlyNewArticles` (already seen in a previous run). The log line states which one.

**Q: The site keeps blocking me.**\
A: Leave `proxyConfiguration.useApifyProxy = false`. The actor will auto-escalate to **datacenter** and then **residential** proxies (and retry up to 3 times residential). If even that fails, enable `useBrowser`.

**Q: Will it work for paywalled articles?**\
A: It honours soft-paywall workarounds (Googlebot UA) but does not bypass strict authentication.

**Q: How do I keep cross-run memory?**\
A: Toggle `onlyNewArticles` or `onlyNewArticlesPerDomain`. The actor keeps state in a named KVS — if that fails (e.g. Store run with limited permissions) it falls back to the run-default store.

**Q: Can I customise the output?**\
A: Yes — supply `extendOutputFunction` as a Python snippet defining `extend(soup, article, html) -> dict`. The returned dict is merged into the record.

***

### 🛟 Support & Feedback

Use the **Issues** tab on the Actor page, or open a discussion on the Apify community forum. Pull requests are welcome.

***

### ⚖️ Cautions / legal

- Data is collected only from publicly available sources.
- Do not scrape private accounts or content behind authentication unless explicitly authorised.
- The end user is responsible for legal compliance (GDPR, CCPA, anti-spam laws, target site ToS, etc.).
- The actor honours `robots.txt` for sitemap discovery; it does not enforce robots.txt blocks on crawl URLs — please be a good citizen.

# Actor input Schema

## `startUrls` (type: `array`):

Top-level pages the crawler should start from — homepages, sections, topic pages. Each one is treated as a category page and articles are discovered from it.

## `articleUrls` (type: `array`):

Already-known article URLs to extract directly (no discovery needed). Mix with Website URLs for hybrid runs.

## `onlyNewArticles` (type: `boolean`):

Skip articles that were extracted in any previous run (deduplicated globally via the key-value store). Best for low-volume runs.

## `onlyNewArticlesPerDomain` (type: `boolean`):

Same as above, but the deduplication memory is kept separately per domain — preferable for multi-domain runs.

## `onlyInsideArticles` (type: `boolean`):

When enqueueing from an article, accept only links that point back to the same registrable domain.

## `enqueueFromArticles` (type: `boolean`):

Discover further article links inside extracted articles and add them to the crawl queue.

## `crawlWholeSubdomain` (type: `boolean`):

Treat every same-subdomain link as a potential category page (depth-limited).

## `onlySubdomainArticles` (type: `boolean`):

Restrict articles to URLs starting with the same path prefix as the Start URL (e.g. example.com/news/\*).

## `scanSitemaps` (type: `boolean`):

Discover article URLs from robots.txt → Sitemap entries and the usual /sitemap.xml candidates. Disable if it produces too many noisy candidates.

## `sitemapUrls` (type: `array`):

Explicit sitemap URLs — skips auto-discovery and only uses these. Safer than full robots.txt scanning.

## `saveHtml` (type: `boolean`):

Include the full page HTML in the dataset record (produces large records).

## `saveHtmlAsLink` (type: `boolean`):

Save HTML to the run's key-value store and put the link in the record (smaller dataset).

## `saveSnapshots` (type: `boolean`):

Take a PNG screenshot of every article. Only effective when the headless browser is enabled.

## `useGoogleBotHeaders` (type: `boolean`):

Send the Googlebot User-Agent + headers. Many publishers allow Googlebot through paywalls / soft-blocks.

## `minWords` (type: `integer`):

Reject articles whose extracted text has fewer than this many words.

## `dateFrom` (type: `string`):

ISO date (YYYY-MM-DD). Only keep articles published on or after this date.

## `onlyArticlesForLastDays` (type: `integer`):

Drop anything older than X days. Combined with dateFrom, the stricter of the two wins.

## `mustHaveDate` (type: `boolean`):

Drop articles where no publication-date metadata can be detected.

## `isUrlArticleDefinition` (type: `object`):

Heuristics for classifying a URL as an article. minDashes = minimum dashes in the path, hasDate = path contains a /YYYY/MM/DD/ pattern, linkIncludes = substrings that mark a URL as an article.

## `pseudoUrls` (type: `array`):

Additional URL patterns (\[.\*], \[\d+]) that mark a page as a crawlable category. If you want to enqueue direct article URLs this way, you have to add { "label": "article" } to the userData.

## `linkSelector` (type: `string`):

Optional CSS selector restricting which parts of a category page contribute links (e.g. main a, .article-list a).

## `maxDepth` (type: `integer`):

Maximum BFS depth from the Start URL (Start URL = 0). Empty = no extra cap.

## `maxPagesPerCrawl` (type: `integer`):

Hard cap on pages fetched in one run (articles + category pages combined).

## `maxArticlesPerCrawl` (type: `integer`):

Hard cap on extracted articles per run.

## `maxArticlesPerStartUrl` (type: `integer`):

Cap how many articles can be attributed to a single Start URL.

## `maxConcurrency` (type: `integer`):

How many fetches the crawler may run in parallel. Higher = faster, but more pressure on the target site and proxy quota. Leave empty for safe sequential mode.

## `proxyConfiguration` (type: `object`):

Proxy settings. Default = NO PROXY (direct). If the target blocks the request, the actor automatically falls back to DATACENTER, then RESIDENTIAL (with up to 3 retries on residential). Once a fallback occurs, it sticks.

## `useBrowser` (type: `boolean`):

Render with Chromium when raw HTTP fails or the page is JS-heavy. Slower but bypasses many anti-bot walls.

## `pageWaitMs` (type: `integer`):

Extra time to wait after navigation finishes (milliseconds). Useful for lazily-loaded scripts.

## `waitUntil` (type: `string`):

Which navigation event Playwright waits for before considering the page ready.

## `categoryWaitForSelector` (type: `string`):

Optional CSS selector. The browser will wait for this element to appear before extracting links from category pages.

## `articleWaitForSelector` (type: `string`):

Optional CSS selector. The browser will wait for this element to appear before extracting article content.

## `scrollToBottom` (type: `boolean`):

Auto-scroll to the bottom of category/article pages so lazy-loaded content is rendered.

## `scrollToBottomButtonSelector` (type: `string`):

Optional CSS selector for a 'Load more' button. The crawler will click it repeatedly while scrolling.

## `scrollToBottomMaxSeconds` (type: `integer`):

Maximum time spent scrolling per page (safety cap).

## `extendOutputFunction` (type: `string`):

Only needed if you want more data than is included in the default output. Keep in mind that you should provide a valid Python function: def extend(soup, article, html): return {...}. The returned dict is merged into each article record.

## `maxCUs` (type: `integer`):

Soft cap on Apify Compute Units this run may consume. The actor checks usage between requests and exits gracefully when the cap is hit. Leave empty for no cap.

## `notificationEmails` (type: `array`):

Email addresses to notify when the CU thresholds below are crossed.

## `notifyAfterCUs` (type: `integer`):

Send a one-time notification once this many CUs have been consumed.

## `notifyAfterCUsEvery` (type: `integer`):

Send a notification every N CUs after the initial notification threshold.

## Actor input object example

```json
{
  "startUrls": [
    "https://www.theguardian.com"
  ],
  "articleUrls": [],
  "onlyNewArticles": false,
  "onlyNewArticlesPerDomain": false,
  "onlyInsideArticles": true,
  "enqueueFromArticles": false,
  "crawlWholeSubdomain": false,
  "onlySubdomainArticles": false,
  "scanSitemaps": false,
  "sitemapUrls": [],
  "saveHtml": false,
  "saveHtmlAsLink": false,
  "saveSnapshots": false,
  "useGoogleBotHeaders": false,
  "minWords": 150,
  "mustHaveDate": true,
  "isUrlArticleDefinition": {
    "minDashes": 4,
    "hasDate": true,
    "linkIncludes": [
      "article",
      "storyid",
      "?p=",
      "id=",
      "/fpss/track",
      ".html",
      "/content/"
    ]
  },
  "pseudoUrls": [],
  "maxDepth": 2,
  "maxPagesPerCrawl": 50,
  "maxArticlesPerCrawl": 25,
  "maxArticlesPerStartUrl": 25,
  "maxConcurrency": 1,
  "proxyConfiguration": {
    "useApifyProxy": false
  },
  "useBrowser": false,
  "pageWaitMs": 0,
  "waitUntil": "load",
  "scrollToBottom": false,
  "scrollToBottomMaxSeconds": 60,
  "extendOutputFunction": "# def extend(soup, article, html):\n#     return {\"pageTitle\": soup.title.string.strip() if soup.title else None}\n",
  "notificationEmails": []
}
```

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        "https://www.theguardian.com"
    ],
    "articleUrls": [],
    "proxyConfiguration": {
        "useApifyProxy": false
    },
    "extendOutputFunction": `# def extend(soup, article, html):
#     return {"pageTitle": soup.title.string.strip() if soup.title else None}`
};

// Run the Actor and wait for it to finish
const run = await client.actor("scraper-engine/smart-article-extractor").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "startUrls": ["https://www.theguardian.com"],
    "articleUrls": [],
    "proxyConfiguration": { "useApifyProxy": False },
    "extendOutputFunction": """# def extend(soup, article, html):
#     return {\"pageTitle\": soup.title.string.strip() if soup.title else None}
""",
}

# Run the Actor and wait for it to finish
run = client.actor("scraper-engine/smart-article-extractor").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print("💾 Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"])
for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    "https://www.theguardian.com"
  ],
  "articleUrls": [],
  "proxyConfiguration": {
    "useApifyProxy": false
  },
  "extendOutputFunction": "# def extend(soup, article, html):\\n#     return {\\"pageTitle\\": soup.title.string.strip() if soup.title else None}\\n"
}' |
apify call scraper-engine/smart-article-extractor --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "command": "npx",
            "args": [
                "mcp-remote",
                "https://mcp.apify.com/?tools=scraper-engine/smart-article-extractor",
                "--header",
                "Authorization: Bearer <YOUR_API_TOKEN>"
            ]
        }
    }
}

```

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/acts/LrvQv2THQlefdYJeu/builds/363yma3Vji0Tw1o0F/openapi.json