Smart Scrape

Smart Scrape Feature Drop: AI Extraction, Change Tracking, Batch Scraping and More

AI extraction, change tracking, batch jobs and eight more upgrades.

Smart Scrape is our one-call scraping API. You send a URL and the formats you want, and Browserless works out how to get the page. It starts with a fast, cheap fetch and only moves up to a proxy, a full browser, or captcha solving when the page needs it. You get back markdown, HTML, screenshots, PDFs or links, plus a note of which step did the job.

Since we launched it in April, we've kept adding to it. This post rounds up the biggest additions so far. Each section shows the feature on a real page, with a link to the docs if you want the details.

Here's what's new:

Every example below is a real response from production, trimmed and formatted to fit the image.

Main content by default

Most pages wrap their content in menus, headers, footers and sidebars. If you feed that to a language model or a RAG pipeline, you pay for a lot of noise. Smart Scrape now keeps only the main content of a page by default. Navigation, footers, sidebars, cookie banners and share buttons are removed before you get the result.

This changes the output of existing requests too. It applies to content, markdown, html, rawText and links, so a links-only request now returns the links in the main content. Requests that send a non-empty includeTags list keep the old behavior. If you need the whole page, send "onlyMainContent": false. You can also pick exactly what to keep with includeTags or what to drop with excludeTags, using CSS selectors.

The same Wikipedia article as markdown, twice. With onlyMainContent set to false it starts with menus and navigation links and is 66,858 characters long. With the new default it starts with the article text and is 46,737 characters long.

๐Ÿ“– Main content docs โ†’

Pull structured data with AI

Sometimes you don't want the page at all, just a few facts from it. Add "json" to formats and describe what you need with a JSON schema, a short prompt, or both. Smart Scrape reads the page and returns an object with those fields. Our guide to LLM web scraping covers how to write a good schema.

If you only want the gist, add "summary" instead and you'll get a few sentences of plain text. Neither format needs a browser, so a simple page stays on the cheapest step. The AI work is billed as extra units based on how much text the model reads and writes.

A request for a book page on books.toscrape.com with a JSON schema and a short prompt. The response contains the title, price 51.77, currency GBP, availability, stock count 22 and the UPC. A second example shows a three sentence summary of the same page.

๐Ÿ“– AI extraction docs โ†’

Click, type and scroll before the scrape

Some content only shows up after you interact with the page, which is common on dynamic websites. Maybe there's a cookie banner in the way, a "Load more" button, or a search box. With actions you can list the steps to run first: click, type, wait, scroll, save a snapshot, or run a bit of JavaScript. Smart Scrape runs them in order, then captures the page as it looks at the end.

In this example we click "Next" on a quotes site, wait for the new quotes to load, and run a short script that counts them.

Three actions on quotes.toscrape.com: click the Next link, wait for the quotes, and run a script. The script returned "10 quotes on /js/page/2/" and the screenshot shows page 2 of the quotes.

๐Ÿ“– Actions docs โ†’

Read PDFs

Not every URL is a web page. When a link points straight at a PDF, Smart Scrape now pulls the text out of it, so you get readable content instead of a file. No separate PDF scraper needed. Ask for markdown and the pages come back separated by blank lines.

A request for the arXiv PDF of the paper Attention Is All You Need with formats set to markdown. The response is 39,619 characters of text served by http-fetch in 2.33 seconds, starting with the paper title, the authors and the abstract.

๐Ÿ“– PDF targets docs โ†’

Reuse a recent result

If you scrape the same pages often, you don't always need a fresh copy. Set maxAge in milliseconds to say how old a result you're happy to accept. If your account scraped the same page with the same options within that time, you get the stored result back straight away. Only results from requests that set maxAge are stored, and each one is kept only as long as the maxAge of the request that saved it, up to 48 hours. Requests with actions, change tracking, a profile or custom headers always get a fresh scrape. A hit is billed at 1 unit, with no proxy, captcha or AI extraction charges, so caching pays off most on pages that need a browser, a proxy or AI.

Caching is off unless you turn it on, so nothing changes for requests that need a fresh page.

The same request for news.ycombinator.com sent twice with maxAge set to 600000. The first request was a cache miss served by http-fetch in 2.06 seconds. The second was a cache hit served from the cache in 0.22 seconds.

๐Ÿ“– Caching docs โ†’

Know when a page changes

Add "changeTracking" to formats, next to "markdown", and Smart Scrape compares the page with your last check of the same URL. The answer is simple: new, same, changed, or removed if the page is gone. Set changeTrackingOptions.modes to ["git-diff"] and you also get the exact lines that changed. Smart Scrape keeps your last check for 30 days, and very large pages can come back as changed without the lines.

Use it to watch prices, docs, job listings or terms pages without storing old copies yourself.

Two checks of a UTC clock page. The first check returned changeStatus new. The second returned changeStatus changed, and the diff shows the time line going from 09:31:52 UTC to 09:32:00 UTC.

๐Ÿ“– Change tracking docs โ†’

Scrape a list of URLs in one job

When you already know which pages you want, the new Batch Scrape API saves you from scraping multiple pages one request at a time. Send the whole list to /batch-scrape and you get a job ID back right away. Then check that one job for progress and results, or have a webhook tell you as each page finishes. Bad URLs are listed separately, and the rest still run.

Batch Scrape is in beta and available on Cloud plans.

A batch scrape request with three book page URLs and one invalid entry. The job status shows completed 3 of 3, invalidURLs lists not-a-url, and each of the three books has a completed result ready to download.

๐Ÿ“– Batch Scrape docs โ†’

Scrape from a specific country

Prices, languages and whole pages can change depending on where a visitor comes from. Set location.country to a two-letter country code and Smart Scrape loads the page through a proxy in that country, a residential proxy by default. When a browser runs, its time zone and language are set to match wherever Smart Scrape can work them out. Proxy bandwidth is billed even when no browser is needed.

Here we asked a site that reports the visitor's location to tell us where we were, once from Germany and once from Japan.

The same ipinfo.io request sent with country de and country jp. The first came from an IP in Mannheim, Germany, in the Europe/Berlin time zone. The second came from an IP in Hamamatsu, Japan, in the Asia/Tokyo time zone.

๐Ÿ“– Country targeting docs โ†’

Screenshots at the size you need

Screenshots now take a screenshotOptions object, closing the gap with a dedicated screenshot API. You can capture only the visible area instead of the full page, choose PNG or JPEG, set the JPEG quality, and pick the viewport size. A phone-sized viewport gives you the mobile layout of a site.

A screenshot request for a Wikipedia article with a 390 by 844 viewport, JPEG format and quality 80. The result is the mobile layout of the article, shown in a phone frame.

๐Ÿ“– Screenshot docs โ†’

Block ads with one flag

Ads slow pages down and clutter your screenshots and content. Set "blockAds": true and Smart Scrape blocks ads and trackers whenever it uses a browser. Plain fetches aren't affected, so it never forces a slower step.

The Merriam-Webster entry for the word scrape captured twice. With blockAds false there is a banner ad at the top, a large ad in the sidebar and another banner at the bottom. With blockAds true all three ads are gone.

๐Ÿ“– Ad blocking docs โ†’

Cheaper on simple pages, quicker on repeat visits

Pricing now follows the step that actually did the work. When a plain fetch is enough, the request costs 0.5 units per 30 seconds, half the time rate of a browser run. Most static and server-rendered pages land here.

Smart Scrape also remembers what worked for each site. If a site needed a browser last time, the next request starts with the browser and skips the steps that are likely to fail.

On the left, example.com was served by http-fetch at 0.5 units per 30 seconds. On the right, the first request to excalidraw.com tried http-fetch, which failed, then succeeded with the browser. The next request went straight to the browser.

๐Ÿ“– Unit cost docs โ†’

Failures that say what went wrong

A scrape that "succeeds" with a 404 page or a bot check is worse than an honest failure, because you can't tell it apart from real content. Smart Scrape is now stricter about this. Missing pages come back as "Not found" with the page's real status code, and it stops right there instead of trying a browser. Bot challenge pages from bot detection systems and browser error pages are reported as failures too, so you never mistake them for the page you wanted.

The response for a page that doesn't exist: ok false, statusCode 404, strategy http-fetch, attempted only http-fetch, and the message Not found. Next to it, three notes: missing pages are reported as Not found, Smart Scrape stops after the first step, and challenge pages and browser error pages come back as failures.

๐Ÿ“– Error handling docs โ†’

Try it

Smart Scrape is available on every plan. Everything except Batch Scrape works through the same /smart-scrape endpoint, so you can mix features in one request. For example, you can ask for main content as markdown, a JSON object with AI, and a mobile screenshot with ads blocked, all at once.

Start with the Smart Scrape docs or grab a free API key and send your first request. We're still building, so tell us what you'd like to see next.