Web Content Scraper

Turn any list of web pages into clean article text, markdown, and publication metadata.

Give this scraper a list of URLs and it returns the part of each page that actually matters. Navigation, footers, cookie banners, ad slots, and share widgets are stripped out, leaving the article body as both plain text and markdown, alongside the title, author, site name, language, and publication date. Because the boilerplate is removed before extraction, re-running the same page returns identical text — so you can reliably detect when content has genuinely changed. It is built for AI and LLM pipelines, RAG and vector-database ingestion, content monitoring, and research archives.

⚡ Quick start

{
  "startUrls": [
    { "url": "https://blog.apify.com/what-is-web-scraping/" },
    { "url": "https://en.wikipedia.org/wiki/Web_scraping" }
  ],
  "maxItems": 5
}

🧩 Input

Field	Type	Required	Default	Description
startUrls	array	yes	—	Web pages to extract. Each must be an absolute http(s) URL. Pages are fetched exactly as listed; no links are followed.
maxItems	integer	yes	5	Maximum number of pages to extract in this run.
removeElementsCssSelector	string	no	null	Extra CSS selector for elements to strip before extraction, added to the built-in list. Use it to remove site-specific clutter.
minTextLength	integer	no	200	Pages whose extracted text is shorter than this many characters are skipped. Filters out consent walls and redirect stubs.
saveHtml	boolean	no	false	Include the cleaned HTML of the main content in the output.
proxyConfiguration	object	no	{ "useApifyProxy": false }	Proxy settings. Most pages need no proxy.

📤 Output

Each extracted page becomes one dataset item. markdown preserves headings, lists, and links, which suits LLM prompts and RAG chunking, while text is plain prose for search indexes and hashing. Metadata is read from the page's structured data first and falls back to OpenGraph and standard meta tags, so publishedAt is populated for most news and blog pages. Fields that a page does not provide are returned as null rather than omitted, keeping the shape consistent across sources.

Fields

Field	Type	Description
url	string	The URL that was fetched, after any redirects.
canonicalUrl	string	The page's canonical URL, normalised and stripped of tracking parameters.
title	string	Article title.
description	string	Short summary or excerpt.
markdown	string	Main content converted to markdown.
text	string	Main content as plain text.
html	string	Cleaned HTML of the main content. null unless saveHtml is enabled.
author	string	Article author.
siteName	string	Publication or site name.
publishedAt	string	Publication date, ISO 8601.
modifiedAt	string	Last modification date, ISO 8601.
lang	string	Page language code.
wordCount	number	Number of words in text.
fetchedAt	string	When the page was fetched, ISO 8601.

💡 Use cases

• Feed clean article text into LLM prompts, RAG pipelines, and vector databases.
• Monitor pages for meaningful content changes without being triggered by banner or navigation edits.
• Build a research archive of articles with consistent titles, authors, and publication dates.
• Convert press releases and blog posts to markdown for summarisation or newsletters.
• Enrich a list of search results with the full text behind each link.

❓ FAQ

• Does it follow links on the page? No. It fetches exactly the URLs you supply, which keeps runs predictable and cheap.
• Do I need a proxy? Usually not. Enable one only if a specific site blocks the default requests.
• What happens if a page has no article? It is skipped with a warning rather than returning empty text. Adjust minTextLength to change the threshold.
• Will the same page always return the same text? Yes, provided the article itself has not changed. Boilerplate is removed before extraction, so navigation and footer edits do not alter the result.
• Does it render JavaScript? No. It uses fast HTTP requests, which covers articles, blogs, press releases, and news. Pages that build their content entirely in the browser are not supported.

🔗 More scrapers by Anyx

• Google AI Scraper
• Indeed Scraper
• Glassdoor Scraper
• Similarweb Top Websites Scraper

🤝 Anyx Solutions

We build custom scrapers and data-extraction pipelines.

• Email: tantosthor@gmail.com

Image credit: en.wikipedia.org