How to Scrape Data from Multiple Web Pages with Python

Introduction

Scraping a single page is simple. The moment you need data from dozens, hundreds, or thousands of pages – a paginated product catalog, a series of blog posts, or a list of URLs from a spreadsheet – the complexity increases fast. Loops break. Sites start blocking requests. Sessions drop. What works on ten pages fails on a thousand.

In this guide, you'll learn the two most reliable ways to scrape multiple web pages using Python, the libraries that make it practical, and the infrastructure considerations that determine whether your scraper holds up at scale.

Why scraping multiple web pages is harder than it looks

A single-page scraper is essentially a one-shot request: fetch, parse, extract, done. Multi-page scraping introduces state. You need to track where you are, handle failures mid-run, manage the rate at which you send requests, and adapt to pagination patterns that differ from site to site.

A few specific challenges that trip up multi-page scrapers include:

• Pagination inconsistency. Sites handle pagination in different ways:
  • Numbered query parameters (?page=2).
  • Path segments (/page/2/).
  • JavaScript-rendered pagination with no URL change at all.
• Rate limiting and blocking. Sites monitor request frequency. Hit a page too often, too fast, from the same IP, and you'll start receiving 429 or 403 responses – or silent blocks that return empty HTML. Understanding how bot detection works helps you build scrapers that hold up longer.
• Session state. Some websites require cookies or authentication tokens to stay active across requests. If you aren't managing session state, your scraper may hit login walls after a few pages.
• Error handling. A page returning a 500 error or timing out mid-loop can crash the whole run if you haven't built in error handling. At scale, some failures are inevitable.
• Legal and ethical constraints. Before scraping at scale, check the site's robots.txt and terms of service. Scraping publicly available data is generally legal, but the specifics vary by jurisdiction and use case. Browserless has a dedicated guide on whether web scraping is legal if you need more information.

Getting a structure in place before you create the loop to scrape several pages saves you from rewriting the scraper later.

How to scrape data from multiple web pages with Python

Most real-world multi-page web scraping scenarios fit into one of two patterns. The right choice depends on how the target website structures its URLs.

Method 1: Loop through paginated URLs

Use this pattern when the site uses predictable, numbered pagination in its URL structure. If you can see ?page=1, ?page=2, or /page/1/, /page/2/ in the browser, you can construct those URLs programmatically and loop through them.

Here's a quick working example using requests and BeautifulSoup:

import requests
from bs4 import BeautifulSoup
import time

BASE_URL = "https://books.toscrape.com/catalogue/page-{}.html"
all_titles = []

for page_num in range(1, 6):  # pages 1 through 5
    url = BASE_URL.format(page_num)
    try:
        response = requests.get(
            url,
            headers={"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/136.0.0.0 Safari/537.36"},
            timeout=(5, 15),  # connect, read
        )
    except requests.exceptions.RequestException as e:
        print(f"Request failed on page {page_num}: {e}")
        break

    if response.status_code != 200:
        print(f"Failed on page {page_num}: {response.status_code}")
        break

    soup = BeautifulSoup(response.text, "html.parser")
    titles = [h3.a["title"] for h3 in soup.select("article.product_pod h3")]
    all_titles.extend(titles)

    time.sleep(1)  # pause between requests

print(f"Scraped {len(all_titles)} titles")

A few things worth noting here. The time.sleep(1) call adds a one-second delay between each request – remove it, and you'll send all requests in rapid succession, which is the fastest way to get blocked. The if response.status_code != 200 check catches failures early rather than letting the scraper silently collect empty results.

Method 2: Loop through a stored list of URLs

Use this script when the URLs you need don't follow a predictable sequence – for example, a list of product detail pages scraped from an index, or a set of URLs imported from a CSV file.

import requests
from bs4 import BeautifulSoup
import csv
import time

# Read URLs from a CSV file with a "url" column
urls = []
with open("urls.csv", newline="") as f:
    reader = csv.DictReader(f)
    for row in reader:
        urls.append(row["url"])

results = []

for url in urls:
    try:
        response = requests.get(
            url,
            headers={"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/136.0.0.0 Safari/537.36"},
            timeout=(5, 15),
        )
    except requests.exceptions.RequestException as e:
        print(f"Skipping {url}: {e}")
        time.sleep(1)
        continue

    if response.status_code != 200:
        print(f"Skipping {url}: {response.status_code}")
        time.sleep(1)
        continue

    soup = BeautifulSoup(response.text, "html.parser")
    title = soup.find("h1").get_text(strip=True) if soup.find("h1") else ""
    results.append({"url": url, "title": title})

    time.sleep(1)

print(f"Scraped {len(results)} pages")

The continue statement on a failed request keeps the loop running rather than stopping it entirely. You get partial data instead of nothing. Note that the delay still runs before continue, so a burst of 403 or 429 responses doesn't turn into a burst of rapid-fire retries.

Setting up your Python scraper

Before you write the loop, a few setup decisions will determine how smoothly the rest goes.

Install the libraries

You'll need requests for HTTP requests and beautifulsoup4 for HTML parsing. lxml is a faster parser than Python's built-in html.parser and worth adding.

pip install requests beautifulsoup4 lxml

Set a User-Agent header

The default requests User-Agent string (e.g., python-requests/2.32.3) is a strong signal to servers that the request is automated. Setting a realistic browser User-Agent reduces the chance of being flagged:

headers = {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/136.0.0.0 Safari/537.36"
}

Understand the HTML structure before looping

Open the target page in a browser, inspect the elements you want, and identify the CSS selectors or tag patterns you'll use with BeautifulSoup. Testing your selectors on a single page before looping through hundreds saves debugging time later.

Use a requests.Session

If your scraper needs to maintain cookies across requests – for example, on a website that requires a session token after the first page load – use requests.Session() rather than calling requests.get() directly. The session object handles cookie persistence automatically.

session = requests.Session()
session.headers.update({"User-Agent": "Mozilla/5.0 ..."})
response = session.get(url, timeout=(5, 15))

Handling pagination and extracting structured data

Identifying the right pagination pattern is the first task. On most sites, you'll see one of:

• Query parameters – https://example.com/products?page=1.
• Path-based pagination – https://example.com/products/page/1/.
• "Next page" links – An anchor tag with rel="next" or a visible "Next" button in the HTML.

For the third one, instead of constructing URLs manually, you can follow next-page links dynamically:

import requests
from bs4 import BeautifulSoup
import time

url = "https://books.toscrape.com/catalogue/page-1.html"

while url:
    try:
        response = requests.get(
            url,
            headers={"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/136.0.0.0 Safari/537.36"},
            timeout=(5, 15),
        )
    except requests.exceptions.RequestException as e:
        print(f"Request failed on {url}: {e}")
        break

    if response.status_code != 200:
        print(f"Failed on {url}: {response.status_code}")
        break

    soup = BeautifulSoup(response.text, "html.parser")

    # Extract data from the current page
    for article in soup.select("article.product_pod"):
        title = article.h3.a["title"]
        price = article.select_one(".price_color").get_text(strip=True)
        print(title, price)

    # Find the next page link
    next_btn = soup.select_one("li.next a")
    if next_btn:
        url = "https://books.toscrape.com/catalogue/" + next_btn["href"]
    else:
        url = None  # no more pages

    time.sleep(1)

This approach handles websites where the total number of pages isn't known upfront. When there's no "next" link, the loop ends cleanly.

Building a dataset across pages

Rather than printing as you go, append each row to a list and write to CSV or JSON once the loop finishes:

import json

rows = []

# Inside your loop:
rows.append({"title": title, "price": price, "url": url})

# After the loop:
with open("output.json", "w") as f:
    json.dump(rows, f, indent=2)

Writing once at the end is more efficient than appending to a file on every iteration, and it makes the output easier to inspect and reuse.

Scaling your multi-page scraper reliably

The patterns above work well for dozens of pages. When you push into hundreds or thousands, new problems surface.

Session management at scale

When scraping large volumes of pages, session tokens can expire, cookies can drift, and connections can drop. Production scrapers need logic to detect these failures and re-authenticate or reconnect rather than silently returning empty data.

Rate limiting and backoff

A fixed time.sleep(1) is fine for small runs. At scale, you want dynamic backoff: if a request returns a 429, wait before retrying rather than skipping. A simple exponential backoff looks like this:

import requests
import time

def get_with_backoff(session, url, retries=3):
    for attempt in range(retries):
        try:
            response = session.get(url, timeout=(5, 15))
        except (requests.exceptions.Timeout, requests.exceptions.ConnectionError) as e:
            if attempt < retries - 1:
                wait = 2 ** attempt
                print(f"Transient error ({e}). Waiting {wait}s...")
                time.sleep(wait)
            continue

        if response.status_code == 200:
            return response
        elif response.status_code == 429:
            if attempt < retries - 1:
                wait = 2 ** attempt
                print(f"Rate limited. Waiting {wait}s...")
                time.sleep(wait)
        else:
            print(f"Unexpected status {response.status_code} for {url}")
            return None

    print(f"Giving up on {url} after {retries} attempts")
    return None

JavaScript-rendered pages

requests + BeautifulSoup only work on statically served HTML. If the data you need is loaded by JavaScript after the initial page load, which is common on modern e-commerce sites and single-page applications, you'll need a headless browser like Playwright.

They are heavier, slower, and harder to run concurrently, which is where infrastructure becomes the real bottleneck. Our deeper walkthrough of scraping with Playwright covers the specifics of handling dynamic content at scale.

When the code isn't the bottleneck

Once you're running concurrent browser sessions at scale, the infrastructure underneath your scraper is as important as the Python code itself. Memory limits, connection pools, session isolation, and handling blocked IPs are all infrastructure problems, not code problems.

Browserless provides a powerful solution by moving the browser layer off your machine entirely.

Rather than maintaining your own headless browser fleet, Browserless provides hosted, production-grade browser infrastructure that handles concurrency, session management, and scaling.

You connect to it with the same Playwright code you're already writing – or call the REST APIs directly from requests – without managing Chrome instances, memory leaks, or version compatibility yourself. For teams running browser-based scrapers at any real volume, it's the difference between maintaining infrastructure and shipping product.

How Browserless takes your multi-page scraper further

The Python patterns above are effective, but once you're dealing with logged-in pages, agent-driven workflows, or scrapers that need to run across many sessions reliably, the gaps in a DIY setup start to show. Browserless addresses three of these specifically.

Smart Scrape API

The Smart Scrape API handles escalation automatically.

• Send a POST request with a URL, and Browserless tries a fast HTTP fetch first.
• If the website blocks it, it retries through a residential proxy.
• If the page requires JavaScript, it launches a headless browser.
• If a CAPTCHA appears, it solves it.
• You get the final HTML, Markdown, or extracted links back in a single response, no browser management code required.

For a Python scraper looping through hundreds of URLs, replacing requests.get(url) with a call to /smart-scrape means you don't need to handle rendering, proxies, and blocks in your own code.

Authenticated profiles

Authenticated profiles enable you to persist a logged-in browser state across scraping sessions. Instead of re-authenticating on every run – re-entering credentials, processing cookies, and waiting for redirects – you capture a profile once and reuse it.

If you're pulling data from behind a login, such as account dashboards, internal tools, or gated content, browser authentication removes the fragile re-login logic that tends to break mid-loop when session tokens expire at an inconvenient page number.

Your scraper can reference a stored browser state rather than treating every execution as a cold start, which means you spend less time managing auth flows and more time processing the data you're actually there to collect.

Browserless's MCP server

Our MCP (Model Context Protocol) server opens up a different mode entirely.

Rather than writing imperative scrape-and-parse scripts, you can connect AI agents directly to a live browser session. An agent can navigate to a URL, take a snapshot of interactive elements on the page, extract a table, adjust its next action based on what it finds, and generate structured output – all through MCP tool calls that your AI client orchestrates automatically.

Use the MCP when the extraction logic is hard to encode upfront. If you're pulling values from pages where the layout varies, or you need to combine data from sources that don't have a consistent class structure, an agent that can reason about what it sees is often more effective than hand-crafted selectors that break whenever the site updates its source code.

For teams already using Browserless for scraping at scale, the MCP layer extends your existing setup. You can merge agent-driven flows with your infrastructure, running both against the same Browserless account, API token, and managed browser infrastructure.

Whether you're looping through a list of URLs stored in a CSV, following next-page links across a paginated catalog, or running an AI agent through a workflow that varies page to page, the browser layer should be the last thing you're debugging.

Conclusion

Scraping data from multiple web pages in Python comes down to choosing the right pattern for your target site, building in error handling from the start, and respecting rate limits throughout. The requests + BeautifulSoup combination gets you a long way. For JavaScript-heavy sites or any scraping that needs to run at real scale, you'll need a headless browser and infrastructure that can support it.

If you're moving beyond a local scraping setup, sign up for Browserless and connect your existing Playwright scraper, or call the REST APIs from Python, to production-grade browser infrastructure without the DevOps overhead.

Scraping data from multiple web pages FAQs

How do I scrape data from multiple web pages using Python?

The two core patterns are looping through paginated URLs (when the site uses numbered pagination like ?page=2) and iterating over a stored list of URLs (when the pages don't follow a predictable sequence).

In both cases, use the requests library to fetch each page and BeautifulSoup to parse the HTML. Add a delay between requests and handle non-200 responses to keep your scraper stable.

What is the best Python library for scraping multiple pages?

For HTML-based sites, requests combined with beautifulsoup4 is the best starting point.

If the target site renders content with JavaScript, you'll need a headless browser controlled through playwright – it's actively maintained and the recommended choice for Python browser automation.

For most multi-page scraping tasks, requests + BeautifulSoup is sufficient and considerably faster.

How do I handle pagination when scraping multiple web pages?

Identify whether the site uses query parameters (?page=N), path-based pagination (/page/N/), or "next page" anchor links. For the first two, construct URLs in a loop. For the third, extract the next-page link from the parsed HTML on each iteration and follow it until no link is found.

How do I avoid getting blocked when scraping multiple pages?

Set a realistic User-Agent header in your requests, add a delay between requests using time.sleep(), and implement exponential backoff when you receive 429 responses.

Using requests.Session() for persistent cookies also helps maintain a more consistent browsing profile. For high-volume scraping, rotating IPs or using a managed browser service adds another layer of reliability.

Can I scrape multiple pages without writing code?

There are no-code browser automation tools and scraping extensions that can handle simple multi-page tasks. However, for anything beyond a basic, one-off export – especially if you need structured data, scheduling, or scale – a Python scraper gives you the control and reliability that no-code tools can't match.