Build a Daily News Scraper with Playwright, Claude, and Slack

Introduction

A news scraper lets your morning briefing start with sources you trust, not a feed ranked by someone else's algorithm. Running one yourself usually means installing Chromium and keeping enough memory free for several browsers at once, before you even reach sites that block datacenter IPs. Here, the scraper runs locally as a Node.js process while the browsers run as remote stealth browsers in Browserless Cloud: the same headless Chromium, hosted for you and hardened against bot detection.

In this guide, you'll build a news scraper that collects recent articles from your own source list into a CSV, schedule it on macOS and have Claude turn that CSV into a short Slack briefing. You control how many sources run in parallel, up to your plan's concurrency limit, and sources that block datacenter IPs can opt into a residential proxy one at a time.

Watch the full build and scheduling walkthrough:

What is a news scraper?

A news scraper visits a fixed list of news sites, reads each article card on their listing pages and saves structured rows: source, publish date, title, URL and excerpt. Unlike an RSS feed or a news API, it works on any site you can open in a browser, and you decide exactly which sources count. The trade-off is maintenance: when a site redesigns its pages, you update its selectors.

In this pipeline the scraper only collects. Claude never browses the sources itself and only reads the finished rows, the same split that keeps large language model (LLM) web scraping predictable and easy to rerun.

How the news scraper pipeline fits together

The pipeline runs on two independent schedules:

1. Collection (09:00 by default). A macOS LaunchAgent installed by setup-launchd.sh runs export-csv.js, which runs the Playwright scraper and refreshes output/scraped-all-items.csv.
2. Briefing (for example, 09:30). A Claude Scheduled Task reads that absolute CSV path, filters the rows by topic and recency, then sends the result through the Slack connector.

Keeping the schedules separate means the scraper never needs access to Slack, and each half can be inspected and rerun on its own.

Step 1: Clone and configure the scraper

You need Node.js 20 or newer (current playwright-core releases require it) and a Browserless API token (sign up for free to get one). The automated scheduler in this example uses launchd, so that part is macOS-only. The scraper itself runs anywhere Node.js runs.

Clone the official examples repository:

git clone https://github.com/browserless/video-examples.git
cd video-examples/news-scraper
npm install
cp .env.example .env

Open .env and replace the placeholder without committing the file:

BROWSERLESS_TOKEN=your-browserless-token
BROWSERLESS_BASE=https://production-sfo.browserless.io

The repository already ignores .env and everything under output/. Still, treat the token as a secret and don't paste it into an LLM prompt or a Slack message.

Run one collection manually before adding schedules:

node export-csv.js

A successful run creates output/scraped-all-items.csv with these columns:

source,publishedDate,title,url,excerpt

Step 2: Understand the remote stealth connection

Each source gets a fresh Playwright connection over the Chrome DevTools Protocol (CDP). The endpoint builder takes the regional HTTPS base from your connection URL, swaps the scheme to wss and points it at the /chromium/stealth route. Based on scrape.js:

const WS_BASE = process.env.BROWSERLESS_BASE.replace(/^https?/, "wss");

function endpoint(source) {
  const params = new URLSearchParams({
    token: process.env.BROWSERLESS_TOKEN,
  });

  if (source.proxy === "residential") {
    params.set("proxy", "residential");
    params.set("proxySticky", "true");
  }

  return `${WS_BASE}/chromium/stealth?${params.toString()}`;
}

const browser = await chromium.connectOverCDP(endpoint(source));

// Use the default context: a new one doesn't inherit the proxy from the URL.
const context = browser.contexts()[0];
const page = await context.newPage();

Work in the default context, as above. A context you create with browser.newContext() doesn't inherit the proxy or launch settings from the connection URL, so a residential source would quietly fall back to a datacenter IP.

The code closes every connection in a finally block. An abandoned session keeps occupying your concurrency until it times out. If a slow source gets cut off mid-scrape, add a timeout query parameter (in milliseconds) to the endpoint, up to your plan's maximum session time (2 minutes on the Free plan).

The example processes sources through a bounded worker pool. Set the MAX_PARALLEL constant in scrape.js no higher than the concurrency shown in your Browserless dashboard. It defaults to 9. On the Free plan, lower it to 2. Sessions beyond your limit wait in a queue, and Browserless returns 429 Too Many Requests once that queue is full, so a right-sized pool avoids both the waits and the errors.

Step 3: Define trusted news sources and selectors

All site-specific knowledge lives in sources.json. A source supplies the listing page, its card selector, selectors for four fields, a date-normalization strategy and an optional proxy mode:

{
  "name": "freeCodeCamp News",
  "url": "https://www.freecodecamp.org/news/",
  "listSelector": "article.post-card",
  "fields": {
    "title": { "selector": ".post-card-title", "attribute": null },
    "url": { "selector": "a.post-card-image-link", "attribute": "href" },
    "date": { "selector": "time", "attribute": "datetime", "strategy": "iso" },
    "excerpt": { "selector": ".post-card-excerpt", "attribute": null }
  },
  "proxy": null
}

To add a site, copy one block and inspect the new listing page for selectors that skip navigation and promoted-content links. Then run the scraper once and check both the CSV and results-playwright.json, the raw per-source output that records each source's status.

Before you add a source, check its terms of service and robots.txt. Keep request volume low, and store only headlines, links and short excerpts rather than republishing full articles. The web scraping legality guide covers the wider picture.

The included date strategies cover common listing formats:

• iso parses a machine-readable timestamp such as a datetime attribute.
• text extracts a conventional date from visible text.
• relative converts values such as 37m or 5 hours relative to the run time.
• urlpath reads a YYYY/MM/DD date from an article URL.

The scraper drops rows without a usable title or URL and makes the remaining URLs absolute and unique. Only articles within the seven-day window reach the output, and an empty excerpt is fine because not every listing card includes one.

Set proxy to "residential" only for a source that needs it. The endpoint then adds a residential proxy with proxySticky=true, keeping one IP for that browser session. Residential proxies cost extra Browserless units and don't fix broken selectors, so first confirm that the page actually blocks a datacenter session.

Step 4: Protect the last good CSV

An automation shouldn't turn a transient outage into an empty briefing. export-csv.js retries once when the entire scrape returns no items. If the second attempt is also empty, it exits with a non-zero status and leaves the existing CSV untouched.

Only after a non-empty scrape does it move the old file into output/Previous runs/ and write the new one. The archive gives the downstream task a last known good input and leaves a timestamped history for debugging.

That safeguard applies to a fully empty run. Individual sources can still fail while others succeed, so review the per-source status in results-playwright.json and the scheduler logs when article counts drop unexpectedly.

Step 5: Schedule collection on macOS

Install the daily LaunchAgent with:

./setup-launchd.sh

The script detects the absolute Node and project paths, installs dependencies, writes ~/Library/LaunchAgents/com.browserless.news-scraper.plist, loads it and triggers a test run. Logs go to ~/Library/Logs/news-scraper/.

The default schedule is 09:00. To choose another time, reinstall with environment variables:

SCRAPE_HOUR=7 SCRAPE_MINUTE=30 ./setup-launchd.sh

To remove the agent without deleting the project or logs:

./setup-launchd.sh --uninstall

A LaunchAgent needs you to be logged in. If the Mac is asleep at the scheduled time, launchd runs the job when it wakes, and if it's shut down, that day's run is skipped. A late run can mean the briefing reads yesterday's CSV, so choose a time when the machine is normally awake, or move node export-csv.js to your existing scheduler on Linux or a hosted runner.

Step 6: Create the Claude briefing task

Connect Slack to Claude and find the user ID that should receive the DM. Then open CLAUDE_SCHEDULED_TASK_PROMPT.md and replace its three EDIT ME values:

• The absolute path to output/scraped-all-items.csv.
• The topics that define relevance.
• The destination Slack user ID.

The prompt limits Claude to the CSV, asks it to prefer articles from the last 48 hours, caps the result at ten, groups links by topic and sends exactly one DM. The instruction "use only rows from the CSV" reduces the chance of the briefing inventing a headline or mixing in web-search results, and the manual test run below is how you confirm it.

In Claude Cowork, open Scheduled, click New task and choose Set up manually. Select your local news-scraper folder as the folder Claude works in, then paste the completed prompt. An absolute path in the prompt alone doesn't give a scheduled task access to a file on your Mac.

Because the task reads a local file, it runs on your Mac rather than remotely, so the Claude desktop app has to be open and the Mac awake at the scheduled time, the same conditions the scraper needs. Keep the task's scope narrow: select only that folder, and have the prompt use only the Slack connector to send the message. Schedule it after the scraper, not at the same time, so it reads a complete file.

Before enabling a recurring send, run the task manually against the generated CSV, confirm it can read the file, and check the recipient and how many articles survive your topic filters. If you later point the task at a shared channel instead of your own DM, test it against a private channel first.

Common failure modes

The scraper receives HTTP 429

Lower MAX_PARALLEL in scrape.js to your plan's concurrency (see Step 2) and confirm every session closes in finally.

A source returns zero recent items

Open the source and recheck its card, title, URL and date selectors, then compare the raw date format with its configured strategy. Use a residential proxy only after ruling out selector drift and navigation timing.

The CSV never refreshes

Check ~/Library/Logs/news-scraper/scraper.err.log and confirm the Mac was awake. The LaunchAgent stores the absolute Node path it found during setup, so rerun ./setup-launchd.sh if you change Node installations.

Claude reads stale or partial data

Confirm the scrape finished before the Claude task started, that the task's working folder is your news-scraper folder and that the prompt points at the absolute CSV path inside it. The archive directory shows whether the collector actually produced a new file.

The briefing never arrives

Check that the Claude desktop app was open and the Mac awake at the task's scheduled time, then open the task's past runs on the Scheduled page to see whether it ran and what it reported.

The Slack briefing goes to the wrong place

Use a Slack user ID, not a display name, for a DM. Test once before recurring delivery and keep the prompt limited to one destination and one message.

Start with the example, then make the source list yours

Run the official news scraper example unchanged once. After you can trace an article from sources.json to the CSV and on to the test briefing, replace the sources and topics one at a time.

Playwright and Browserless collect the same rows on a predictable schedule, while Claude handles the subjective step of choosing which articles are worth reading. You can run the whole pipeline on a free Browserless plan with MAX_PARALLEL set to 2.

FAQ

Is it legal to scrape news websites?

It depends on the site's terms and your jurisdiction. Collecting headlines and links for your own briefing is lower risk than republishing article text, and some publishers prohibit automated access outright, so a site whose terms do should come off your list.

How often should a news scraper run?

Once or twice a day suits most briefings. Running more often adds load on the sources and uses more Browserless units without surfacing many new articles.

Can a news scraper read paywalled articles?

Not as written. The scraper reads public listing pages, so paywalled sources still contribute headlines and links, but the excerpt may be empty or truncated.

What's the difference between a news scraper and a news API?

A news API returns articles from the publishers it indexes, in its own format. A news scraper reads the sites you choose directly, so you control the sources, at the cost of maintaining selectors when a site changes.