# RAG Web Browser (`parseforge/rag-web-browser`) Actor

Give your AI agents real-time web access! Search the web on any topic and get full page content as clean Markdown, ready for LLMs, RAG pipelines, or OpenAI Assistants. Includes titles, descriptions, links, authors, images, and metadata. Start grounding your AI with fresh data in minutes!

- **URL**: https://apify.com/parseforge/rag-web-browser.md
- **Developed by:** [ParseForge](https://apify.com/parseforge) (community)
- **Categories:** AI, Automation
- **Stats:** 14 total users, 4 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

Pay per event

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

![ParseForge Banner](https://github.com/ParseForge/apify-assets/blob/ad35ccc13ddd068b9d6cba33f323962e39aed5b2/banner.jpg?raw=true)

## 🤖 RAG Web Browser

> 🚀 **Give your LLM live web access in seconds.** Search the web or fetch specific URLs and return **clean Markdown with 17 metadata fields** per page. No API key, no registration, no manual content cleaning.

The **RAG Web Browser** is built for retrieval-augmented generation pipelines, autonomous agents, and any workflow where an LLM needs grounded, up-to-date web content. Send a search query to get the top N results, or pass a list of URLs to fetch them directly. Every page is stripped of navigation, ads, and boilerplate, then converted to clean Markdown that feeds directly into embedding pipelines and vector databases.

Each record ships with rich metadata including title, description, author, published time, modified time, site name, Open Graph image, language, word count, and estimated reading time. Search results include a `rankFromSearch` field so you can weight retrieval by original engine position. Concurrent fetching keeps 10 URLs flying in parallel, so research agents stay snappy and RAG refreshes finish while your coffee is still hot.

| 🎯 Target Audience | 💡 Primary Use Cases |
|---|---|
| AI engineers, RAG builders, research agent developers, LLM app teams, content researchers, data scientists | Live RAG context, agent web browsing, knowledge base refresh, competitive intelligence, fact-grounding |

### 📋 What the RAG Web Browser does

Five content workflows in a single run:

- 🔎 **Search mode.** Pass a text query and get the top N results from DuckDuckGo with clean content for each.
- 🎯 **URL mode.** Provide specific URLs and the scraper fetches them in parallel.
- 📝 **Clean Markdown.** Strips navigation, footers, sidebars, scripts, and ads. Preserves headings, lists, blockquotes, and code blocks.
- 📊 **Rich metadata.** Title, description, author, publishedTime, modifiedTime, siteName, og:image, language, word count, reading time.
- 🏆 **Search rank preserved.** When searching, every result keeps its rank position so you can weight retrieval accordingly.

Output comes as markdown, plain text, or raw HTML. You can also request an outbound-link dump when you need to follow references.

> 💡 **Why it matters:** LLMs trained on data older than six months cannot answer questions about today's news, pricing, or product documentation. This Actor gives them a live window on the web without you having to build browser automation, proxies, or content cleaners.

### 📊 Data fields

Each record includes: `author`, `description`, `fetchedAt`, `httpStatus`, `imageUrl`, `language`, `markdown`, `publishedTime`, `rankFromSearch`, `readingTimeMinutes`, `responseTimeMs`, `results`, `siteName`, `title`, `url`, `wordCount`. These field names come straight from the actor's dataset schema, so what you see here is what lands in your dataset.

> ⚠️ **Good to Know:** single-page apps with heavy client-side rendering sometimes return thin content because the scraper fetches server-rendered HTML. For JavaScript-heavy sites (Notion, Gitbook, some app dashboards), pair this Actor with Website Content Crawler and its browser rendering mode.

### 🚀 How to use

1. 📝 **Sign up.** [Create a free account with $5 credit](https://console.apify.com/sign-up?fpr=vmoqkp) (takes 2 minutes).
2. 🌐 **Open the Actor.** Go to the RAG Web Browser page on the Apify Store.
3. 🎯 **Pick a mode.** Enter a search query OR a list of URLs, set `maxItems`, and choose output formats.
4. 🚀 **Run it.** Click **Start** and let the Actor collect your content.
5. 📥 **Download.** Grab your results in the **Dataset** tab as CSV, Excel, JSON, or XML.

> ⏱️ Total time from signup to downloaded dataset: **3-5 minutes.** No coding required.

### 🔗 Recommended Actors

- [**🕸️ Website Content Crawler**](https://apify.com/parseforge/website-content-crawler) - Deep-crawl a domain with depth and JS rendering
- [**📰 Smart Article Extractor**](https://apify.com/parseforge/article-extractor) - Extract clean article text from news sites
- [**🔍 Google Search Scraper**](https://apify.com/parseforge/google-search-scraper) - SERP results with rank and description
- [**📧 Contact Info Scraper**](https://apify.com/parseforge/contact-info-scraper) - Emails, phones, and socials from URLs
- [**📸 URL Screenshot Tool**](https://apify.com/parseforge/screenshot-url) - Full-page screenshots as PNG, JPEG, or PDF

> 💡 **Pro Tip:** browse the complete [ParseForge collection](https://apify.com/parseforge) for more AI-ready web tools.

> **⚠️ Disclaimer:** this Actor is an independent tool and is not affiliated with any search engine or website. Only publicly accessible web content is fetched. Respect the robots.txt and terms of service of every site you add to the input.

### 🆘 Need Help?

If you hit a bug, have questions about setup, or need a scraper we haven't built yet, open our [contact form](https://tally.so/r/BzdKgA) or write to parseforge@protonmail.com. We also take on paid custom data projects.

For faster answers, join our [Discord](https://parseforge.co/discord). It's the best place to get support and suggest new actors.

# Actor input Schema

## `query` (type: `string`):

Web search query. Alternative to Start URLs.

## `startUrls` (type: `array`):

Specific URLs to fetch. Alternative to query.

## `maxResults` (type: `integer`):

Number of search results to fetch (when using query).

## `maxItems` (type: `integer`):

Free users: Limited to 10 items (preview). Paid users: Optional, max 1,000,000

## `outputFormats` (type: `array`):

Which content formats to include. markdown is always included.

## `includeLinks` (type: `boolean`):

Include all outbound links from each page.

## Actor input object example

```json
{
  "query": "Anthropic Claude API pricing",
  "maxResults": 10,
  "maxItems": 10,
  "outputFormats": [
    "markdown",
    "text"
  ]
}
```

# Actor output Schema

## `results` (type: `string`):

Complete dataset

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "query": "Anthropic Claude API pricing",
    "maxResults": 10,
    "maxItems": 10,
    "outputFormats": [
        "markdown",
        "text"
    ],
    "includeLinks": false
};

// Run the Actor and wait for it to finish
const run = await client.actor("parseforge/rag-web-browser").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "query": "Anthropic Claude API pricing",
    "maxResults": 10,
    "maxItems": 10,
    "outputFormats": [
        "markdown",
        "text",
    ],
    "includeLinks": False,
}

# Run the Actor and wait for it to finish
run = client.actor("parseforge/rag-web-browser").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print("💾 Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"])
for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "query": "Anthropic Claude API pricing",
  "maxResults": 10,
  "maxItems": 10,
  "outputFormats": [
    "markdown",
    "text"
  ],
  "includeLinks": false
}' |
apify call parseforge/rag-web-browser --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "command": "npx",
            "args": [
                "mcp-remote",
                "https://mcp.apify.com/?tools=parseforge/rag-web-browser",
                "--header",
                "Authorization: Bearer <YOUR_API_TOKEN>"
            ]
        }
    }
}

```

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/acts/ooY9a5K2abMjKLmjo/builds/5eIXgD0Y97oNPfAhe/openapi.json