# Agentic Crawler (`hpix/agentic-crawler`) Actor

An intelligent AI web scraper that navigates websites like a human. Just describe the data you need in plain English. Adapts to layout changes, handles dynamic JavaScript sites, and gets smarter with every run.

- **URL**: https://apify.com/hpix/agentic-crawler.md
- **Developed by:** [Hpix](https://apify.com/hpix) (community)
- **Categories:** AI, Agents, Lead generation
- **Stats:** 69 total users, 12 monthly users, 96.1% runs succeeded, 1 bookmarks
- **User rating**: No ratings yet

## Pricing

from $0.90 / 1,000 smart units

This Actor is paid per event and usage. You are charged both the fixed price for specific events and for Apify platform usage.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## 🧠 AI Agentic Web Scraper

**Scrape any website with an expert AI web scraper.**

This isn't just another scraper. It's an **intelligent AI agent** that views and understands webpages just like a human does. It uses advanced AI to navigate sites, interpret content, and extract exactly what you ask for, even if the website’s layout changes completely tomorrow.

### 🚀 Why is this different?

Traditional scrapers rely on rigid rules that break whenever a website updates its design. This AI Actor is different:

- **Zero Configuration:** No inspecting code. No complex CSS selectors. Just describe what you want in plain words.
- **Adapts to Change:** If a site changes its layout, the AI re-evaluates the page visually and finds the data anyway.
- **See it like a Human:** It handles complex modern websites, infinite scrolling, and dynamic layouts that usually break standard bots.
- **Gets Smarter Over Time:** The more you run it on a specific website, the faster and more accurate it becomes as it learns the site's structure.

### 🎯 What can you use it for?

#### 💼 Lead Generation & Research

- Extract contacts (names, emails, roles) from business directories.
- Build prospect lists from industry association websites.
- Gather real estate agent details from listing sites.

#### 📊 Market Intelligence

- Scrape e-commerce product details, prices, and specs across different retailers.
- Monitor competitor "Team" and/or "Careers" pages for changes.

#### 📱 Social Media Monitoring

- Extract public posts, engagement metrics, and trending topics from social platforms.
- Gather influencer profiles, follower counts, and content performance data.
- Monitor brand mentions and sentiment across social media channels.

#### 🕸️ Complex Site Extraction

- Scrape data from sites heavily reliant on JavaScript (React, Vue, etc.).
- Navigate through non-standard pagination or complex flows just by asking.

### 🛠️ How to Use It (3 Simple Steps)

You don't need to be a developer to run this. It's as easy as giving instructions to a human assistant.

#### 1. Provide the URLs

Tell the AI which website pages you want it to visit.

#### 2. Write Your Instruction

In plain English (or any other language), describe exactly what you are looking for. The clearer you are, the better the results.

> **Examples of good instructions:**
>
> - *"Find all the laptop models on this page and extract their name, price, CPU, and RAM."*
> - *"Scrape the latest 10 news articles, getting the headline, author, and publish date for each."*
> - *"Go through the first 5 pages of results and extract the company name and location for every listing."*

#### 3. Define Your Output Structure (Optional)

If you just want a quick summary, leave this blank.

If you need the data perfectly formatted for a database or Excel file, you can define the exact structure you want (e.g., "I need a list of objects where every item has a 'Title' and a 'Price'"). The AI will force the data into that shape.

### 📄 The Results

Once the run is finished, you won't have to dig through messy code. You can download your clean, extracted data instantly as an
**Excel spreadsheet, CSV, JSON, or HTML file**.

If you chose a structured output, your data will be perfectly organized into columns and rows, ready for analysis.

### 💰 Predictable Pricing

This Actor uses Smart Units, so you only pay for the "brain power" the AI actually uses. You don't pay for server time or idle memory, only for successful analysis and data extraction.

- **1 Smart Unit = $0.0012**
- **The "Learning" Benefit**: The first time the AI visits a site, it spends units to "learn" the structure. Because it saves this knowledge, subsequent runs on the same site are typically faster and cheaper.

| Activity                                    | Smart Units |
|---------------------------------------------|-------------|
| **Reading Data** (per 1k input tokens)      | 1 Unit      |
| **Structuring Data** (per 1k output tokens) | 3 Units     |

#### 📊 Cost Estimate

In a typical run to scrape data, the cost breakdown looks like this:

| Step      | What the AI is doing                                       | Cost                   |
|-----------|------------------------------------------------------------|------------------------|
| Planning  | Studying the task and deciding how to navigate.            | 15 Units               |
| Analysis  | Deep-parsing the website data and content.                 | 300 Units              |
| Execution | Organizing the data and finalizing the results.            | 80 Units               |
| Memory    | Oganizing the "knowledge" so next runs are more efficient. | 8 Units                |
| **Total** | **A complete, "unbreakable" data extraction.**             | **403 Units (~$0.48)** |

#### 🛡️ Budget Control

We believe in "No Surprise" billing. You have total control over your costs:

1. Set a Limit: Use the **Run options > Maximum cost per run** setting to tell the Actor exactly when to stop.
2. Instruction Efficiency: Clearer instructions (e.g., "Just get the first 5 posts") help the AI work faster and use fewer units.
3. Knowledge Base: The AI automatically retrieves knowledge from your previous runs to minimize the cost of the "Analysis" phase on familiar websites.

### 💡 Pro-Tips for Success

1. **Be Specific:** Treat the AI like a smart junior assistant. "Get data" is bad; "Extract product titles and prices" is good.
2. **The "Second Run" Rule:** The Actor learns from experience. The first run on a new, complex site might take longer while it maps the structure. Subsequent runs on the same site are usually much faster.
3. **Pagination:** Yes, it can handle pagination! Just tell it to in the instruction (e.g., *"scrape results from the next 3 pages"*).

### 📖 FAQ

**Does it really not need CSS selectors?**
Really. It looks at the rendered page visually and semantically, just like you do.

**Can it handle logins?**
Not currently. The Actor is designed to extract data that is publicly available without requiring authentication.

**How accurate is it?**
Accuracy depends heavily on how clear your instructions are. Typical accuracy is 85-95% for well-structured sites, and it improves as the agent "learns" a specific website over repeated runs.

#### ⚠️ Disclaimer

This Actor is designed to extract publicly available data. It is your responsibility to ensure you have permission to scrape your target websites and comply with their Terms of Service, robots.txt, and applicable laws.

# Actor input Schema

## `start_urls` (type: `array`):

List of URLs to scrape. The AI agent will intelligently navigate and extract data from these pages.

## `instruction` (type: `string`):

Plain English description of what data to extract. Be specific about what information you want (e.g., 'Extract all product names, prices, and ratings').

## `output_schema` (type: `object`):

JSON schema defining the structure of extracted data. When provided, the actor returns structured JSON matching your specification. If left empty, the agent will infer a schema from the extraction instructions.

## `proxy_configuration` (type: `object`):

Proxy settings for bypassing anti-scraping measures. Residential proxies are recommended for improved reliability.

## Actor input object example

```json
{
  "start_urls": [
    "https://tiktok.com/@tiktok"
  ],
  "instruction": "Extract the descriptions (bio) and like counts for the latest 5 videos shown in the 'Videos' section.",
  "proxy_configuration": {
    "useApifyProxy": true,
    "apifyProxyGroups": [
      "RESIDENTIAL"
    ]
  }
}
```

# Actor output Schema

## `dataset` (type: `string`):

Dataset containing all extracted data from the scraped URLs

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "start_urls": [
        "https://tiktok.com/@tiktok"
    ],
    "instruction": "Extract the descriptions (bio) and like counts for the latest 5 videos shown in the 'Videos' section."
};

// Run the Actor and wait for it to finish
const run = await client.actor("hpix/agentic-crawler").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "start_urls": ["https://tiktok.com/@tiktok"],
    "instruction": "Extract the descriptions (bio) and like counts for the latest 5 videos shown in the 'Videos' section.",
}

# Run the Actor and wait for it to finish
run = client.actor("hpix/agentic-crawler").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print("💾 Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"])
for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "start_urls": [
    "https://tiktok.com/@tiktok"
  ],
  "instruction": "Extract the descriptions (bio) and like counts for the latest 5 videos shown in the '\''Videos'\'' section."
}' |
apify call hpix/agentic-crawler --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "command": "npx",
            "args": [
                "mcp-remote",
                "https://mcp.apify.com/?tools=hpix/agentic-crawler",
                "--header",
                "Authorization: Bearer <YOUR_API_TOKEN>"
            ]
        }
    }
}

```

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/acts/m4K1tdvG2hAkNIEcN/builds/tW5toD99BYCM96elO/openapi.json