Skip to main content
News scraper

Scrape news headlines and articles with Browserless

Discover news results, collect headlines, and extract article content from news platforms, Google News, and publisher websites with managed browsers, APIs, and Python.

What is a news scraper?

A news scraper automatically collects publicly available headlines, links, publication dates, and article content from news sites or news search pages, so you can turn news coverage into usable data.

Teams use scraped news data for media monitoring, competitive research, trend analysis, and briefing workflows.

Common data points

Story title and publisher
Article URL and publication date
Description or summary snippet
Full article text where accessible
Search keyword, topic, and country
Structured JSON or CSV output

How Browserless overcomes the challenges of scraping news sites

Use the right method for the page: discovery, full-browser rendering, or extraction from each publisher URL.

News search results are not full articles

Use the Search API with sources: ["news"] to get publisher URLs, then pass each URL to Smart Scrape for the full article.

Publishers render headlines or bodies with JavaScript

Browserless runs a real Chromium page before selectors or page-content extraction are applied.

Different sites use different HTML markup

Use per-site CSS selectors with the Scrape API, or request markdown/schema-guided JSON with Smart Scrape.

Rate limits, CAPTCHAs, and fingerprint checks block requests

Use stealth routes, residential proxies, and other Browserless bot-detection tools, as no approach guarantees access to every site.

Coverage changes by country and publication date

Use country, lang, and tbs parameters for general news discovery. Use a location-aware browser session for direct Google News pages.

Repeated stories and many concurrent pages complicate collection

Deduplicate by canonical URL or headline and control your own crawl schedule while Browserless manages browser infrastructure.

How Browserless scrapes news

Find stories with one API call, then pull the full article from each publisher.

1

Search for news

Send POST /search with your keyword, sources: ["news"], and optional country, lang, and tbs filters.

2

Collect story links

Each result returns a headline, publisher URL, and description. Add scrapeOptions to get each article’s main content as markdown in the same call.

3

Fill the gaps

Pass any URL that comes back without content to Smart Scrape, which escalates to a browser, proxies, or CAPTCHA solving when needed.

4

Normalize and export

Pull publication dates from the article, deduplicate by URL, and save the records as JSON or CSV.

A simple news scraper code example

This Python news scraper calls the Browserless Search API to find this week's stories on a topic and returns each headline, publisher URL, and article content as markdown, with no browser code to maintain.

Python

Install: python -m pip install requests

Run: Set BROWSERLESS_TOKEN, then run python news_scraper.py.

import json
import os
import requests

token = os.environ["BROWSERLESS_TOKEN"]
payload = {
    "query": "AI regulation",
    "sources": ["news"],
    "country": "us",
    "lang": "en",
    "tbs": "week",
    "limit": 5,
    "scrapeOptions": {"formats": ["markdown"], "onlyMainContent": True},
}
response = requests.post(
    f"https://production-sfo.browserless.io/search?token={token}",
    json=payload,
    timeout=120,
)
response.raise_for_status()
stories = [
    {
        "title": item["title"],
        "url": item["url"],
        "markdown": (item.get("markdown") or "")[:300],
    }
    for item in response.json()["data"]["news"]
]
print(json.dumps(stories, indent=2, ensure_ascii=False))

Some results come back without content or a publication date. Pass those URLs to Smart Scrape to extract the article and its date. Neither method bypasses publisher paywalls.

Automate news scraping with the MCP

Connect the Browserless MCP server to an AI assistant, then use search and scraping tools to collect coverage and assemble a briefing.

Discover news coverage

Use browserless_search with sources: ["news"] to collect topical headlines and links, optionally filtered by country or recency.

Extract article content

Pass a publisher URL to browserless_smartscraper for readable page content without writing a separate scraper for every site.

Create a daily briefing

Have your assistant compare extracted stories, remove duplicates, and prepare a summary. Add scheduling and Slack delivery in your workflow.

MCP server: https://mcp.browserless.io/mcp

How to set up your MCP

Start scraping news with Browserless

Find news with one API call, extract full articles, and monitor multiple publishers without hosting your own browsers.

Frequently asked questions

Frequently asked questions