Scrape news headlines and articles with Browserless
Discover news results, collect headlines, and extract article content from news platforms, Google News, and publisher websites with managed browsers, APIs, and Python.
Your complete toolkit for news scraping
Find stories across news sources, extract full articles, and monitor multiple publishers with one Browserless account.
POST /search request.Choose the right news scraper for your needs
Start with a news search or article URL, then choose a browser-based workflow for Google News pages and complex publishers.
Search results and optional article scraping, in Cloud beta
Use POST /search with sources: ["news"] to find stories, filter by time, language, or country, and optionally scrape each result. Results come from multiple news search engines, not from Google News.
Single-URL article extraction
Pass a publisher article URL to POST /smart-scrape and get readable markdown, HTML, or requested structured JSON.
Typed automation for protected sites
Use BAP’s TypeScript or Python SDK to navigate article pages, handle dynamic elements, and extract content, with managed stealth, residential proxies, and CAPTCHA solving built in.
Custom extraction sequences
Write browser-side queries for navigation, content extraction, and tricky news pages. Test flows in the BrowserQL IDE.
Full browser control in JavaScript or Python
Connect your existing Puppeteer or Playwright code to our hosted browsers with a WebSocket URL, follow article links, and extract rendered content.
Structured fields from rendered pages
Call POST /scrape with selectors for headlines, dates, and other page elements on a supported target layout.
Browserless features that make your news scraper reliable
Different news sites have different structure and access rules. Browserless provides you with several ways to find stories, render content, and recover from blocked pages.
What is a news scraper?
A news scraper automatically collects publicly available headlines, links, publication dates, and article content from news sites or news search pages, so you can turn news coverage into usable data.
Teams use scraped news data for media monitoring, competitive research, trend analysis, and briefing workflows.
Common data points
How Browserless overcomes the challenges of scraping news sites
Use the right method for the page: discovery, full-browser rendering, or extraction from each publisher URL.
News search results are not full articles
Use the Search API with sources: ["news"] to get publisher URLs, then pass each URL to Smart Scrape for the full article.
Publishers render headlines or bodies with JavaScript
Browserless runs a real Chromium page before selectors or page-content extraction are applied.
Different sites use different HTML markup
Use per-site CSS selectors with the Scrape API, or request markdown/schema-guided JSON with Smart Scrape.
Rate limits, CAPTCHAs, and fingerprint checks block requests
Use stealth routes, residential proxies, and other Browserless bot-detection tools, as no approach guarantees access to every site.
Coverage changes by country and publication date
Use country, lang, and tbs parameters for general news discovery. Use a location-aware browser session for direct Google News pages.
Repeated stories and many concurrent pages complicate collection
Deduplicate by canonical URL or headline and control your own crawl schedule while Browserless manages browser infrastructure.
How Browserless scrapes news
Find stories with one API call, then pull the full article from each publisher.
Search for news
Send POST /search with your keyword, sources: ["news"], and optional country, lang, and tbs filters.
Collect story links
Each result returns a headline, publisher URL, and description. Add scrapeOptions to get each article’s main content as markdown in the same call.
Fill the gaps
Pass any URL that comes back without content to Smart Scrape, which escalates to a browser, proxies, or CAPTCHA solving when needed.
Normalize and export
Pull publication dates from the article, deduplicate by URL, and save the records as JSON or CSV.
A simple news scraper code example
This Python news scraper calls the Browserless Search API to find this week's stories on a topic and returns each headline, publisher URL, and article content as markdown, with no browser code to maintain.
Python
Install: python -m pip install requests
Run: Set BROWSERLESS_TOKEN, then run python news_scraper.py.
import json
import os
import requests
token = os.environ["BROWSERLESS_TOKEN"]
payload = {
"query": "AI regulation",
"sources": ["news"],
"country": "us",
"lang": "en",
"tbs": "week",
"limit": 5,
"scrapeOptions": {"formats": ["markdown"], "onlyMainContent": True},
}
response = requests.post(
f"https://production-sfo.browserless.io/search?token={token}",
json=payload,
timeout=120,
)
response.raise_for_status()
stories = [
{
"title": item["title"],
"url": item["url"],
"markdown": (item.get("markdown") or "")[:300],
}
for item in response.json()["data"]["news"]
]
print(json.dumps(stories, indent=2, ensure_ascii=False))Some results come back without content or a publication date. Pass those URLs to Smart Scrape to extract the article and its date. Neither method bypasses publisher paywalls.
Automate news scraping with the MCP
Connect the Browserless MCP server to an AI assistant, then use search and scraping tools to collect coverage and assemble a briefing.
Use browserless_search with sources: ["news"] to collect topical headlines and links, optionally filtered by country or recency.
Pass a publisher URL to browserless_smartscraper for readable page content without writing a separate scraper for every site.
Have your assistant compare extracted stories, remove duplicates, and prepare a summary. Add scheduling and Slack delivery in your workflow.
MCP server: https://mcp.browserless.io/mcp
Start scraping news with Browserless
Find news with one API call, extract full articles, and monitor multiple publishers without hosting your own browsers.