How to Build an AI Scraper with Browserless MCP and BAP

Introduction

Browserless MCP enables you to prototype a scraper by describing it to an AI agent, instead of hand-writing selectors for pages that render client-side, change layout, or throw bot checks. Once the extraction works, you move it into a typed BAP script you can run and schedule yourself.

With Browserless MCP, you can point an AI agent at a live, cloud-hosted browser, watch it navigate and extract data through plain-English instructions, then turn what worked into a BAP script that runs the same extraction at scale once you're ready for production.

With this guide, you'll extract book titles from a public site using two Browserless tools working together:

• Browserless MCP gives your AI agent tool-driven control of a managed Chrome instance.
• Browser Automation Protocol (BAP), a typed SDK on top of BrowserQL, runs your extraction logic as a script you can schedule and scale in production.

Setting up the tools

To get from an agent-driven prototype to a production-ready scraper, you need two things set up: an MCP client connected to Browserless, and the BAP SDK installed for the scaling step.

Here's how you get both running.

Browserless MCP (hosted browser automation)

Browserless MCP is a hosted Model Context Protocol server. Instead of installing a browser extension or running a local Node process, you point your MCP client at a single URL, and your agent gets Browserless tools for scraping, crawling, running custom Puppeteer code, and driving a stateful browser, all on managed cloud infrastructure.

How to set it up:

• Grab an API token from your Browserless account dashboard, or use OAuth sign-in if your client supports it. URL-only clients, such as Claude custom connectors, take the token as a ?token= query parameter instead.
• Add the hosted server, https://mcp.browserless.io/mcp, to your client's MCP config, following the Browserless MCP setup guide.
• Test it by asking your agent to take a screenshot of a URL and confirm it comes back.

Once your agent returns a real screenshot or scraped result, you're ready to build.

AI agent – Cursor, Claude Desktop, or API

You use an AI agent to translate plain English into the tool calls Browserless MCP executes, instead of writing scraping commands by hand.

Tools:

• Cursor or VS Code, both of which connect to remote MCP servers natively.
• Claude Desktop or the Claude API for structured tool-call generation.
• Any other MCP-compatible client, such as Windsurf or Claude Code. The server URL and token are the same everywhere, but each client's config format differs slightly (VS Code, for example, uses a servers key in settings.json).

You describe the task, and the agent picks the tool and fills in its parameters, so you can iterate on a scrape in chat before you commit it to code.

BAP for production scaling

Once your extraction logic works through the agent, BAP takes it from an interactive session to a repeatable script. BAP is Browserless's typed TypeScript SDK on top of BrowserQL. Its methods borrow Puppeteer's names, so page.goto() and page.click() work how you'd expect.

Quick validation:

• Install the SDK with npm install @browserless.io/bap-ts (npm 11.10 or later). The examples below are TypeScript with top-level await, so run them as ES modules – for example, with npx tsx scrape.ts and "type": "module" in your package.json.
• Grab the same API token you used for MCP.
• Follow the BAP quickstart to confirm your first page.goto() call returns a page title.

Browserless runs and scales the browsers, so there's no Chrome fleet for you to maintain. You still decide how many pages to open against your account's concurrency limit, when to retry (BAP throws typed errors, such as ConnectionError, that you can catch), and when to switch to the /stealth/bql route.

Using an AI agent with Browserless MCP

Once your MCP client is configured, you're working with a live, tool-driven browser your agent controls over HTTP.

You describe what you want instead of writing imperative scripts to navigate the DOM, and your agent picks a tool, such as browserless_smartscraper for a one-shot scrape, browserless_function for custom Puppeteer code, or browserless_agent for multi-step flows, and runs it on Browserless's managed browsers.

Step 1: Point your MCP client at Browserless

In your client's MCP config – Claude Desktop's claude_desktop_config.json, or the equivalent file for Cursor or VS Code – add the hosted server with your token in the Authorization header:

{
  "mcpServers": {
    "browserless": {
      "type": "http",
      "url": "https://mcp.browserless.io/mcp",
      "headers": {
        "Authorization": "Bearer YOUR_API_TOKEN_HERE"
      }
    }
  }
}

Your client reads this file on startup and connects to the hosted server over Streamable HTTP. There's no subprocess to spawn and no extension to load.

Step 2: Run a plain-English command

With the server connected, open your agent's chat or sidebar and write something like:

"Go to https://books.toscrape.com and list all the book titles on the page."

Your agent will:

• Parse that sentence using its own model.
• Recognize this as a scraping task rather than a multi-step flow.
• Call browserless_smartscraper, which returns the page as clean markdown.
• Read the titles out of that markdown and reply in your chat.

What's actually happening technically

Here's the flow broken down more precisely:

• Your agent's model interprets the prompt and picks a Browserless tool, and your MCP client sends that tool call.
• The hosted MCP server receives the tool call over Streamable HTTP and validates the request.
• Browserless fetches the page using a cascading strategy – HTTP fetch, then proxy, then a headless browser, then CAPTCHA solving if needed – so simple pages resolve quickly and only harder ones escalate to a full browser.
• The result comes back in the tool response as Markdown (the default format), ready for your agent to use or hand off to the next step.

You see each tool call and its result in your agent's chat, without keeping a browser instance alive on your own machine.

Scaling with BAP

With the extraction proven in chat, the next step is scaling, and that's where BAP takes over. BAP is a typed SDK on top of BrowserQL, so the proxy, CAPTCHA, and stealth features you'd otherwise wire up yourself are one method call or endpoint change away.

It's built for repeatable, production-grade extraction, with:

• Residential proxy routing by country or city, via page.proxy().
• CAPTCHA solving via page.solve(), with no third-party service to manage.
• A stealth route (wss://production-sfo.browserless.io/stealth/bql) for targets with aggressive bot detection – swap it in for the /chromium/bql endpoint used below.
• Structured extraction in one call via page.mapSelector(), instead of a query-then-loop pattern.

Scraping books with BAP

Here's the same scrape as a BAP script:

import Browserless from "@browserless.io/bap-ts";

const TOKEN = "YOUR_API_TOKEN_HERE";

const browser = Browserless.connect({
  browserWSEndpoint: "wss://production-sfo.browserless.io/chromium/bql",
  token: TOKEN,
});

const page = await browser.newPage();

await page.goto("https://books.toscrape.com/");

const books = await page.mapSelector(".product_pod h3 a");

for (const book of books) {
  console.log(book.innerText);
}

await browser.close();

page.mapSelector() returns one record per matched element, with its text and markup already extracted, so there's no separate parsing step once the response comes back.

Books to Scrape truncates long titles in the link text, though – the full title lives in each link's title attribute, which the pagination example below reads with page.evaluate().

Automating across pagination

Next, scrape every book across all 50 paginated pages. BAP's connection stays open across navigations, so this is a loop, not a new session per page:

import Browserless from "@browserless.io/bap-ts";
import { writeFileSync } from "fs";

const TOKEN = "YOUR_API_TOKEN_HERE";
const allTitles: string[] = [];

const browser = Browserless.connect({
  browserWSEndpoint: "wss://production-sfo.browserless.io/chromium/bql",
  token: TOKEN,
});

const page = await browser.newPage();

const scrapePage = async (url: string) => {
  await page.goto(url);
  const titles = await page.evaluate(
    `JSON.stringify([...document.querySelectorAll(".product_pod h3 a")].map((a) => a.getAttribute("title")))`,
  );
  allTitles.push(...JSON.parse(titles ?? "[]"));
};

await scrapePage("https://books.toscrape.com/");

for (let i = 2; i <= 50; i++) {
  await scrapePage(`https://books.toscrape.com/catalogue/page-${i}.html`);
}

const csvRow = (value: string) => `"${value.replaceAll('"', '""')}"`;
writeFileSync("all_books.csv", "title\n" + allTitles.map(csvRow).join("\n"));

await browser.close();

console.log(`Scraping complete. ${allTitles.length} titles saved to all_books.csv.`);

It writes all 1,000 titles to all_books.csv. Because the script reuses one page for all 50 URLs, it holds a single WebSocket and one concurrency slot the whole time.

How BAP is production-ready

While Browserless MCP is built for agent-driven, tool-call-based automation, BAP is designed for scale, stability, and stealth as a script you run on your own schedule:

• Run many scrapes concurrently against your account's concurrency limit.
• Route requests through residential proxies with page.proxy(), without a separate proxy provider.
• Compose goto, mapSelector, click, and type into a single typed script.
• Hand a session back to native Puppeteer at any point if BAP doesn't cover a specific method yet.

BAP currently ships as a TypeScript SDK, @browserless.io/bap-ts. A Python SDK is on the way but isn't on PyPI yet, so until it lands, use the TypeScript SDK or call BrowserQL directly from Python.

The benefits of this workflow

Using Browserless MCP and BAP together gives you real flexibility across development and production. With Browserless MCP, your agent works against a live, tool-driven browser, so you can inspect results and adjust your prompt on the fly, without deploying anything or managing local Chrome versions.

Once your extraction logic is stable, move it to BAP for scaling: parallel pages within your concurrency limit, geo-targeted proxies, and scripts you run from your own scheduler or CI.

You prototype with an agent, then encode what worked as a BAP script – all on the same Browserless account and API token.

Conclusion

You've now seen how to extract the same data two ways: one driven by an AI agent through Browserless MCP, and one running as a typed script through BAP.

Both use the same target site and the same extraction pattern. Browserless MCP validated it on one page through agent-driven testing. BAP turned that pattern into a script that pulled all 1,000 titles across 50 pages.

If you're working with dynamic pages or need to go from a prompt to a scheduled job, this pairing shortens that path considerably. Try Browserless today.

Browserless MCP FAQs

What does Browserless MCP do in this workflow?

Browserless MCP gives your AI agent tool-based control over a real Chrome browser running in the cloud. It's used for testing and iterating on scraping logic through plain-English prompts, without installing anything locally.

How do AI agents like Claude or Cursor fit into this setup?

Agents interpret your prompts and call the Browserless MCP tool that matches the task, such as browserless_smartscraper for a one-off scrape or browserless_agent for a multi-step flow. You describe the goal; the agent picks the tool.

What is BAP, and how is it different from Browserless MCP?

BAP is a typed TypeScript SDK on top of BrowserQL that you run as a script, with methods for proxies, CAPTCHA solving, and a stealth route when a target needs them. Browserless MCP is built for interactive, agent-driven tasks. Many teams prototype with the agent, then scale with BAP.

Can I use the same extraction logic in both Browserless MCP and BAP?

Both point at the same managed Chrome infrastructure and the same target site, so what you validate through the agent translates directly when you write the equivalent BAP script.