Website Content Crawler — Text, Titles & Metadata
Pricing
from $0.0024 / page scraped
Website Content Crawler — Text, Titles & Metadata
Extract clean, readable content from any list of websites: page title, meta description, headings, main body text, word count and link/image counts. Optional same-domain crawl. Bulk-ready, no browser, no login. Great for LLM/RAG ingestion, content audits and research.
Pricing
from $0.0024 / page scraped
Rating
0.0
(0)
Developer
hiper soft
Maintained by CommunityActor stats
0
Bookmarked
2
Total users
1
Monthly active users
2 days ago
Last modified
Categories
Share
Turn any list of websites into clean, structured content. For each page the Actor extracts the title, meta description, headings, main body text (boilerplate stripped), word count, language and link/image counts — over plain HTTP, no browser and no login. Optionally crawl internal links to pull a whole section.
Ideal for LLM/RAG ingestion, content audits, competitive research, and building text datasets from many sites at once.
What you get per page
| Field | Notes |
|---|---|
title, description, lang | Page title, meta/OG description, <html lang>. |
text | Readable body text with scripts/nav/header/footer stripped; prefers <main>/<article>. |
wordCount | Word count over the full extracted text. |
headings | h1/h2 outline (level + text). |
internalLinkCount, externalLinkCount, imageCount | Structure signals. |
links | Arrays of internal/external links (toggle off to slim the output). |
canonicalUrl, domain, depth | Canonical link, host, and crawl depth. |
Input
- urls — pages/sites to extract, one per line. (Or use startUrls.)
- crawl — follow same-domain links (BFS). Off = just the given URLs.
- maxPagesPerDomain / maxDepth — crawl limits.
- sameDomainOnly — keep the crawl on the start domain (default on).
- includeLinks — include the internal/external link arrays (counts always included).
- textMaxChars — truncate stored body text (word count uses the full text).
- maxConcurrency / maxItems — parallelism and a global page cap.
- proxyConfiguration — optional; enable for reliable access at scale.
Output (one row per page)
FAQ
Do I need an API key or login? No. The Actor works over plain HTTP with no browser and no login — just paste your list of URLs or domains and run.
How many pages can I get per run?
Control it with maxPagesPerDomain, maxDepth and a global maxItems cap. With crawl on, it follows same-domain links breadth-first, so one run can extract from a whole site section or many sites at once.
Is crawling website content legal? The Actor collects only publicly available page content — the same pages any visitor can open. Public data is generally collectible, but you are responsible for respecting each site's terms and applicable law.
Does it collect emails or contact info? No. It extracts page content — title, description, headings, body text, word count and link/image counts — not contact details. For emails, phones and socials, use the Website Contact Scraper.
What's the output format? One structured JSON row per page with clean, boilerplate-stripped text and metadata — ready for LLM/RAG ingestion, content audits or text datasets. Export as JSON, CSV or Excel.
Related Actors
Pair content extraction with these web-tool companions:
- Website Contact Scraper — extract emails, phones and social profiles from the same sites.
- Google Maps Scraper — get local business listings by keyword and location.
- Google Maps Email Extractor — turn Maps results into contactable leads with emails and socials.
Notes
Every site is hard-capped in time so one slow page can never stall a bulk run. Only publicly available page content is collected; respect each site's terms.
