How to Build an AI-Powered Scraper with Browser MCP and BQL

Key Takeaways

• Use Browser MCP for precise, real-time debugging: Working in a real browser lets you tune selectors, inspect the DOM, and iterate interactively before scaling your scraper.
• BrowserQL handles production workloads at scale: With stateless, cloud-executed sessions and built-in stealth capabilities, BQL supports concurrency, dynamic IPs, and proxy rotation without managing local infrastructure.
• AI agents bridge human input and automation: Whether you’re working with ChatGPT, Claude, or another LLM, the AI agent can generate JavaScript or GraphQL dynamically, giving you control over scraping logic without hardcoding every step.

Introduction

Scraping is moving fast from brittle scripts to real-browser AI agents and cloud APIs. It's no longer just about parsing static HTML. You can now use natural language to control a browser, inspect DOM state live, and reuse the same logic in scalable, headless sessions without rewriting anything. In this guide, you'll extract book titles from a public site using three tools working together. Browser MCP gives you direct control of a live Chrome instance. An AI agent, such as ChatGPT or Claude (via Cursor), generates commands in real-time.

Setting Up the Tools

To achieve a seamless workflow from local prototyping to running scrapers in the cloud, you must install and verify these three components. Here’s how I set things up every time.

Browser MCP (Local Chrome Automation)

With Browser MCP, you control a real Chrome browser on your machine via a Node.js server and a Chrome extension.

Why you’ll love it:

It’s hands-on and instant feedback, perfect for testing selectors, refining logic in DevTools, and seeing exactly how your script interacts with the browser before scaling.

How to set it up:

• Install Node.js (v14 or later)
• Follow the official Browser MCP setup guide
• Load the MCP Chrome extension
• Test it by sending a basic command (e.g. eval) and confirm Chrome responds in real time.

Once you see the JSON you sent reflected in the browser, you're ready to build scrapers confidently.

AI Agent (Cursor IDE, Claude Desktop, or API)

Rather than writing scraping commands manually, we will use an AI agent to translate plain English into browser commands the MCP server executes.

Tools

• Cursor IDE (has native support for Browser MCP)
• Claude Desktop or API for structured JS command generation
• OpenAI and Anthropic APIs if you want to build your prompt-to-command logic

This enables writing scrape logic to be much faster and more accurate.

BrowserQL with Browserless (Cloud Automation)

When your local logic runs correctly, switch to BrowserQL for cloud-scaled scraping.

Quick validation:

• Sign up for Browserless and grab your API key
• Open the BrowserQL GraphQL playground: BrowserQL Quick Start
• Confirm the output matches your local MCP session and the key is authorized.

BrowserQL handles job isolation, headless session cleanup, parallelization, and stealth mode for you.

Using an AI Agent with Browser MCP

Once your MCP server and Chrome extension are up and running, you’re working with a live, programmable browser controlled via HTTP. The real magic starts when you pair that with an AI agent in Cursor, which lets you write high-level, natural-language prompts that generate valid browser automation scripts automatically.

Rather than writing imperative scripts to navigate the DOM, you describe what you want Cursor parses the intent, generates JavaScript, and sends it to the MCP server. The commands run in the actual page context, just like if you were in DevTools.

Step 1: Configure Cursor for Browser MCP

To get this working inside Cursor, you define your MCP config in a mcp.json file:

{
  "mcpServers": {
    "browsermcp": {
      "command": "npx",
      "args": ["@browsermcp/mcp@latest"]
    }
  }
}

This tells Cursor how to boot up MCP as a subprocess. When you invoke a task from Cursor, it starts a Node child process using NPX, which pulls the latest version of @browsermcp/mcp and spins up a local HTTP server (usually on port 3000 or 3001). The server communicates with a Chrome instance (via the Chrome Debugging Protocol) through the installed MCP Chrome extension.

Under the hood, every time you issue a prompt or script, Cursor wraps it into a structured JSON command and sends it to the MCP HTTP endpoint. From there, MCP parses and evaluates the script inside the browser tab context using Runtime.evaluate.

Step 2: Run a Plain-English Command in Cursor

Now that your server is live and connected, open Cursor’s AI sidebar or Ask AI popup and write something like:

“Go to https://books.toscrape.com and list all the book titles on the page.”

Cursor will:

• Parse that sentence using its internal LLM (likely GPT-4 or Claude).
• Detect a navigation command, followed by a DOM extraction task.
• Generate and chain two commands:
  • browse: navigate to the URL.
  • eval: run JavaScript to extract data.
• Format and POST those commands as JSON to MCP's API.

The resulting script might look like:

{
  "command": "eval",
  "script": "Array.from(document.querySelectorAll('h3 a')).map(el => el.getAttribute('title'))"
}

This JavaScript is executed in the Chrome tab’s page context using the DevTools Protocol. The result is returned via MCP’s response payload and displayed in the Cursor sidebar.

What’s Actually Happening Technically

Here’s the flow broken down in more technically:

• Cursor → interprets the prompt and generates both code and the structured MCP command format.
• MCP Local Server → receives the HTTP POST and determines whether it’s a browse, click, eval, etc.
• Chrome Extension → forwards commands to the active browser tab using chrome.scripting.executeScript or DevTools Protocol methods.
• Page Context → JavaScript runs directly in the live DOM environment of the active page.
• Response → the script’s return value is captured, serialized, and sent back as a JSON payload.

This gives you full transparency and control. Unlike headless scripts that abstract away browser state, this setup allows you to:

• Inspect the live DOM (visually)
• Run real-time, sandboxed commands
• View returned values, errors, and page state
• Iterate quickly without deploying or packaging anything

Scaling with BrowserQL

Once you’ve validated your extraction logic locally using Browser MCP and Cursor, the next step is scaling and that’s exactly where BrowserQL (BQL) shines. BQL is a GraphQL-powered API built on top of Browserless infrastructure that lets you programmatically launch headless Chrome sessions in the cloud. No manual setup, no persistent state, and no infrastructure to maintain.

But it’s not just a “remote browser.” It’s an entire platform designed for high-volume, stealthy scraping with built-in support for:

• Residential proxies
• Fingerprint spoofing
• Cookie injection
• Auto-retries and timeouts
• DOM readiness guarantees (e.g., firstContentfulPaint, networkidle)

Scraping Books with BQL

After testing locally, we now want to scrape all the book titles from books.toscrape.com. First, we ask Cursor to help generate a basic BQL mutation giving it examples from the docs. It gave us a structure like this:

mutation ScrapeBooks {
  goto(url: "https://books.toscrape.com/", waitUntil: domContentLoaded) {
    status
  }
  books: mapSelector(selector: ".product_pod") {
    title: mapSelector(selector: "h3 a") {
      text: attribute(name: "title") {
        value
      }
    }
  }
}

We plug this into the Browserless BQL playground, added our API key, and immediately got back structured results and a full list of book titles. With it working in the BQL playground we just feed that command into Cusor and it returns a list of titles and converts it into a CSV for us.

Automating Across Pagination

Next, we want to scrape every book on the site, across all 50 paginated pages. We ask Cursor to help again, and it generates a shell script that uses curl and jq to paginate through the entire catalog:

#!/bin/bash

OUTPUT="all_books.csv"
echo "title" > "$OUTPUT"

tmpfile="/tmp/bookspage.json"
scrape_page() {
  local url="$1"
  local json_payload
  json_payload="{\"query\":\"mutation ScrapeBooks { goto(url: \\\"$url\\\", waitUntil: domContentLoaded) { status } books: mapSelector(selector: \\\".product_pod\\\") { title: mapSelector(selector: \\\"h3 a\\\") { text: attribute(name: \\\"title\\\") { value } } } }\"}"
  curl --silent --request POST \
    --url 'https://production-sfo.browserless.io/chromium/bql?token=YOUR_API_KEY&proxy=residential&proxyCountry=us&humanlike=true&blockConsentModals=true' \
    --header 'Content-Type: application/json' \
    --data "$json_payload" > "$tmpfile"
  jq -r '.data.books[].title[0].text.value' "$tmpfile" >> "$OUTPUT"
}

scrape_page "https://books.toscrape.com/"
for i in {2..50}; do
  scrape_page "https://books.toscrape.com/catalogue/page-$i.html"
done

echo "Scraping complete. Results saved to $OUTPUT."

It worked beautifully, the script pulled all book titles and saved them to a clean CSV with no rate limits, no blocks, and no broken selectors.

How BQL Is Production-Ready

While MCP is perfect for fast iteration and debugging on your local machine, BQL is designed for scale, stability, and stealth:

• Run thousands of scrapes concurrently
• Route requests through rotating proxies or custom IPs
• Use mapSelector, goto, click, and evaluate in a composable GraphQL interface
• Export as JSON, clean HTML, or pipe directly to databases or cloud storage

Plus, you can generate and manage these queries through your AI agent, using natural language or simple mutations — all without leaving Cursor.

Why This Workflow Matters

Using this setup provides real flexibility during the development and debugging phases. With Browser MCP, you're working with a real browser on your local machine. It provides the ability to inspect elements, open developer tools, and iterate quickly, which is incredibly useful when fine-tuning selectors or dealing with dynamic content.

You can see exactly what’s happening in real-time and make adjustments on the fly, without needing to wait for deployment cycles or worrying about remote environments.

Once your scraping logic is stable, you can move the same scripts to BrowserQL to handle scaling and cloud automation. BrowserQL provides stateless, cloud-based browser instances, enabling you to handle large-scale jobs efficiently.

This is ideal for when you need to run parallel tasks, avoid local machine limitations, and handle production schedules. You can queue multiple jobs, rotate proxies, and execute tasks without hitting the limits of a local setup.

The combination of these tools offers a smooth workflow between testing and production. While MCP is perfect for live interaction and debugging, BrowserQL allows you to scale your operations seamlessly.

You can reuse the same scripts from the local browser environment, and even generate BQL queries dynamically based on AI-driven insights. This means you can focus on building intelligent workflows without having to rewrite code or switch between different systems.

Conclusion

You’ve now seen how to extract the same data using two environments: one controlled locally with MCP, and one cloud-based via BrowserQL. Both used the same script and returned the same result. MCP provides you with full visibility and real-time control during testing. BQL enables you to scale the same logic across headless sessions, stealth, and concurrency. The AI agent acts as a bridge, generating browser commands or GraphQL queries dynamically. If you're working with dynamic pages or high-volume scraping, this setup shortens the path from testing to production. Try Browserless today.

FAQs

What does Browser MCP do in the scraping workflow?

Browser MCP gives you control over a real Chrome browser running locally. It's used for testing and debugging scraping scripts with full access to the live page, devtools, and JavaScript context.

How do AI agents like ChatGPT or Claude fit into this setup?

AI agents interpret prompts and generate JavaScript or JSON commands that control the browser through MCP. They can automate common scraping tasks, modify selectors, and even chain interactions.

What is BrowserQL, and how is it different from MCP?

BrowserQL runs in the cloud using stateless headless Chrome sessions. It supports concurrent scraping jobs, stealth mode, and proxy rotation, ideal for production use once your scripts are stable.

Can I use the same JavaScript logic in both MCP and BrowserQL?

Yes, the same DOM extraction logic or JavaScript evaluation can run in both environments. You can prototype locally with MCP, then scale with BrowserQL using the same script.

Do I need to use an AI agent, or can I send commands manually?

You can send commands manually using cURL or HTTP POST requests. The AI agent is optional but useful for generating scripts quickly, chaining logic, or adapting to dynamic page content.