Treating headless detection as a one-time fix
Sites update their detection, so a script that works today can get flagged next month.
E-commerce scraper
E-commerce web scraping that doesn't break at scale
Keep your Puppeteer or Playwright code. Browserless adds the stealth, proxies, and CAPTCHA solving it needs to keep working at scale.
175M+
Docker pulls
10,000+
engineering teams
8 years
in production
99.9%
uptime
Why scraping e-commerce sites is a challenge
Effective e-commerce web scraping involves tackling some of the toughest anti-bot systems online. Major retailers lean on bot-management platforms like Cloudflare and Akamai, and a vanilla Puppeteer or Playwright script gets flagged within minutes.
- 01
Bot detection and fingerprinting
Cloudflare, Akamai, and PerimeterX flag headless Chrome by its fingerprint, often before a page even loads.
- 02
CAPTCHAs and challenges
Suspicious traffic gets a reCAPTCHA or Cloudflare Turnstile challenge instead of the page you asked for.
- 03
Rate limits and IP blocks
Too many requests from one IP get throttled, blocked, fed decoy data, or shown different prices entirely.
What you can build with e-commerce scraping
E-commerce scraping is more than simple data extraction. Whether you're extracting data from major marketplaces like Amazon and eBay, small e-commerce stores built on Shopify, or a retailer's site, here's what you can pull, what you can automate, and how Browserless can help.
Data you can extract
- Product titles, images, and descriptions.
- Structured product data: pricing, stock, dimensions, color, and availability.
- Seller details from popular online marketplaces.
- Reviews and ratings.
- Search result pages and product listings.
Automations you can build
- Price monitoring and competitor tracking.
- Stock and availability alerts.
- Catalog syncing against a marketplace.
- Market research across multiple online retailers.
- Dynamic pricing checks by region.
How Browserless can help
- Your Puppeteer or Playwright code, routed through a stealth engine.
- Built-in CAPTCHA solving with
solveCaptchas=true, orpage.solve()in BAP SDK. - Residential proxies by country, or by state and city with
page.proxy()in BAP.
Browserless isn’t a pre-built dataset
You write the parsing logic for the stores you care about, rather than buying fixed, retailer-specific datasets. If you’d rather not run a browser script, the Scrape API returns structured JSON from a rendered page using CSS selectors.
JavaScript rendering adds to the challenge
Most major retailers rely on JavaScript rendering for pricing and stock, so a plain HTTP client only returns raw HTML, not the data you actually need. A headless browser solves that, but headless Chrome is easy for anti-bot systems to spot. Left unmanaged, keeping it unblocked turns into a stack of proxy rotation, stealth plugins, and constant maintenance.
The DIY approach
A standard Puppeteer script scrapes an unprotected product page without any extra infrastructure. It works until the site starts checking for bots.
scraper.js
import puppeteer from "puppeteer";
// JS rendering happens automatically in a real browser
const browser = await puppeteer.launch({ headless: true });
const page = await browser.newPage();
await page.goto("https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html");
const title = await page.$eval("h1", (el) => el.textContent);
const price = await page.$eval(".price_color", (el) => el.textContent);
await browser.close();Point this code at a major retailer, and it gets flagged almost immediately – headless Chrome has tells a detection system checks for before your scraping logic ever runs.
The one-line switch
Grab an API token from your dashboard and swap puppeteer.launch() for puppeteer.connect() to the stealth endpoint. The scraping logic doesn’t change:
scraper.js
import puppeteer from "puppeteer";
const browser = await puppeteer.connect({
browserWSEndpoint: "wss://production-sfo.browserless.io/stealth?token=YOUR_API_TOKEN&proxy=residential&proxyCountry=us&solveCaptchas=true",
});
const page = await browser.newPage();
await page.goto("https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html");
const title = await page.$eval("h1", (el) => el.textContent);
const price = await page.$eval(".price_color", (el) => el.textContent);
await browser.close();The stealth endpoint hardens the browser fingerprint, and the Browser Automation Protocol’s, or BAP’s, click() and type() move and type like a person by default.
Browserless e-commerce scraping use cases
You can use the e-commerce web scraper to capture a lot of the information on a web page. Here are different scraping use cases and the code you need.
Handling Cloudflare and CAPTCHAs
When a retailer still throws up a challenge, page.solve() handles it in the same session rather than starting over. It auto-detects common challenges – including Cloudflare, reCAPTCHA, and DataDome – and returns whether one was found and solved, so your script can check before it carries on. Image CAPTCHAs have their own method: solveImageCaptcha(). Neither needs a third-party solving service.
solve-challenge.js
await page.goto("https://example-store.com/product/12345");
const result = await page.solve({ type: "cloudflare", timeout: 30000 });
if (!result.solved) throw new Error(result.error ?? "Challenge not solved");Geo-pricing and proxies
Price and stock often depend on where the request comes from. page.proxy() routes traffic through Browserless’s residential (default) or datacenter network and geo-targets it by country, state, or city. Add sticky: true to keep the same exit IP for the rest of the session, so every check in that session sees the same regional prices. You don’t need to build your own proxy rotation layer.
geo-pricing.js
await page.proxy({ country: "US", state: "california", sticky: true, type: ["document"] });
await page.goto("https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html");Building a price tracker
Price monitoring involves checking the same pages on a schedule, so consistency is more important than speed. A sticky proxy session keeps checks consistent by region, and with reconnect(), a multi-step job can pick up the same live browser, with its cookies and open page, instead of starting from scratch. For checks spread across hours or days, persist the session so returning visits keep their cookies and state.
It keeps the data fresh and real-time, run after run. Browserless is the browser layer, not a packaged price-monitoring product – write results to your own database, analytics tools, or cloud storage, and handle the scheduling and alerts yourself. For the exact reconnect pattern, see the BAP docs.
price-tracker.js
import Browserless from "@browserless.io/bap-ts";
const browser = Browserless.connect({
browserWSEndpoint: "wss://production-sfo.browserless.io/stealth/bql",
token: "YOUR_API_TOKEN",
});
try {
const page = await browser.newPage();
await page.proxy({ country: "US", sticky: true, type: ["document"] });
await page.goto("https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html");
const { text: price } = await page.text({ selector: ".price_color" });
// store the result, compare to the last run, alert on a change
} finally {
await browser.close();
}Common e-commerce scraping pitfalls
Most e-commerce scraping failures come down to the same handful of avoidable habits.
Rotating proxies without matching the session's timezone, locale, or user agent
Failing to do this leaks as many signals as the IP address itself.
Hammering a target website with the same search term or request pattern
Space requests out and vary them to make your activities look more like real traffic.
Ignoring a site's terms of service or robots.txt
Scraping public data is common practice, but staying inside a site's stated limits keeps your IPs and accounts from getting burned.
Browserless is your scraper, without the blockers
Building the stealth, proxy, and CAPTCHA infrastructure yourself is unnecessary effort. Browserless runs it for you, so your automation tools keep working against sites built to block them. Using an AI agent? The Browserless MCP server connects to clients like Claude Desktop, Cursor, VS Code, and Windsurf, and LangChain has its own Browserless integration.