Web Scraping Dynamic Websites Without the Guesswork

Introduction

Web scraping dynamic websites breaks scrapers that work everywhere else. You point the same code that just pulled data from a dozen static websites at one built with React or Vue, and get back a near-empty <div> where the content should be. The code isn't wrong. The server just hasn't finished building the page yet.

The content you want gets built afterward, in the browser, once JavaScript runs. That same shift is also why the job gets harder once bot detection and changing front-end frameworks enter the picture. In this guide, you'll learn what that means in practice, how to diagnose it on any site, and how to actually pull the data out with browser automation, an API you find, or infrastructure you don't have to maintain yourself.

What is dynamic website scraping?

A static page arrives from the server complete, so viewing its source shows the same content a user sees. That's not true for a dynamic site: client-side rendering ships a mostly empty shell on the initial load, then executes JavaScript to fill in the page, often by calling an API and writing the response into the DOM. Frameworks like React, Vue, and Next.js all support this mode, but they can also server-render, so a framework name alone doesn't tell you which you're dealing with. The only reliable check is to inspect the actual response for the content you want, which the next section walks through.

Web scraping dynamic websites means extracting dynamic data from that second kind of page. A basic HTTP client (Python's requests, for example) only ever sees the initial, mostly-empty HTML. It has no JavaScript engine, so none of the content that loads after page load ever shows up.

That's why a scraper that works fine on one site returns nothing on another. Same code, different rendering model.

Client-side rendering isn't a niche pattern anymore. JavaScript now runs on 98.9% of all websites, according to W3Techs, and plenty of that JavaScript builds entire pages client-side rather than adding a bit of minor interactivity. Assuming a page is static until proven otherwise will quietly cost you data on a growing share of the sites you touch.

How to tell if a website is dynamic

Before you write a line of scraping code, confirm what you're dealing with. Open the web page in a browser, right-click to inspect element or view the page source, and check the HTML code directly. If the data you want is sitting right there in the raw HTML, you're looking at a static web page, or at least a static version of the part you need.

If the source shows an empty container or a handful of <script> tags where the content should be, open your browser's dev tools, switch to the Network tab, and reload the page. Watch for XHR (XMLHttpRequest) or fetch requests returning JSON, since that's usually where the JavaScript-rendered content is coming from. If you'd rather capture that response programmatically than replay it by hand, browser tooling can wait on and read it directly; in Python Playwright, wrap the action that triggers the request in page.expect_response(). If you see one carrying your target data, the page is dynamic, and that request is worth investigating directly.

A quick way to confirm dynamic content specifically is to reload the page with JavaScript disabled. If the content disappears, JavaScript rendering is doing the work, and a plain HTTP client won't see it either.

Web scraping techniques for dynamic websites

Once you've confirmed you're dealing with dynamic content, you've got three realistic paths to scrape dynamic web pages, each trading effort against reliability differently. The right one depends on how the specific site is built.

Browser automation

Tools like Playwright, Selenium, and Puppeteer run headless browsers, or a normal browser you can watch when you need to see what's happening, so the page's JavaScript executes exactly as it would for a real visitor. You wait for the content you want to appear, then read it out of the fully rendered DOM.

Browser automation is the broadest option, since it renders the page regardless of which requests supply the data. It works against sites with complex client-side interactions, ones that build content from several chained requests, and ones where the underlying API is deliberately obfuscated. That reach comes at a resource cost, since a browser instance is slower to start and heavier on memory than a plain HTTP request, and that adds up fast if you're scraping at any real volume.

Reverse-engineering the underlying API

Every dynamic page has to get its data from somewhere. If you can find the exact XHR or fetch call the page makes and replicate it with a plain HTTP client, you skip the browser entirely. Send one lightweight request, get JSON back, and parse it directly, with no rendering or waiting involved.

This approach relies on an API that isn't guaranteed to stay stable. A redesign or new framework version can rename fields or restructure the response overnight, breaking your scraper without warning. It also doesn't work cleanly against every site, since some APIs sit behind session-bound tokens, request signing, or a WAF (web application firewall) that expects browser-like traffic, which makes replicating the call from a script alone significantly harder than the browser automation route.

Finding the request is mostly a matter of patience in the Network tab. Reload the page, filter to XHR or fetch, and look for a response that contains the data you're after. Once you've got the request URL, headers, and query parameters, replicating it is often just a few lines with Python's requests library. Here's a working example of the pattern against a real, public API, adapt the URL, headers, and parameters to whatever your actual target expects.

import requests

api_endpoint = "https://jsonplaceholder.typicode.com/posts"
params = {"userId": 1}
headers = {"User-Agent": "Mozilla/5.0"}

response = requests.get(api_endpoint, params=params, headers=headers, timeout=10)
response.raise_for_status()
records = response.json()

There's no browser instance and no render to wait on, just a direct request that lets you extract data straight into a CSV file or a database.

Managed headless browser infrastructure

Browser automation solves the challenge of seeing exactly what a rendered page shows, but keeping headless browsers running reliably at scale is a different problem. Long-running headless Chrome instances leak memory, browser versions drift out from under your automation code, and concurrency limits show up right when you need volume most. Offloading the browser to managed infrastructure sidesteps all three.

There are two shapes to this. You can point your existing Playwright or Puppeteer code at a hosted browser endpoint, or skip the browser library entirely and send a single HTTP request to a rendering API that returns the fully rendered HTML or the structured fields you asked for. Browserless supports both paths, and for an existing Puppeteer or Playwright script adoption is closer to changing a connection string than rewriting your scraper. Selenium connects too, through a remote WebDriver endpoint rather than a single URL swap. The walkthrough below builds a local Playwright script first, then shows exactly what changes to point it at a managed browser instead.

Which tool should you use?

No browser automation library stops a site from blocking you once you're scraping at any real volume, and the right pick for web scraping dynamic pages hinges more on your stack, scale, and data requirements than on any single feature. Selenium has the longest track record and the widest language support, which still counts if your team isn't working primarily in Python or JavaScript. Playwright and Puppeteer both trade some of that breadth for a more modern API and, in Playwright's case, first-class support for Chromium, Firefox, and WebKit from one codebase, though on managed infrastructure Firefox and WebKit are reachable through Playwright's own protocol endpoint rather than the CDP URL this guide uses for Chromium.

For a direct look at how two of these compare feature by feature, see Playwright vs. Selenium. And for a broader roundup that also covers AI agents and no-code tools, see the full comparison of web scraping tools.

That still leaves the same open question for all three, since none of them solve what happens once you need dozens or hundreds of these browsers running concurrently in production, which is a separate, infrastructure-shaped challenge.

Tool	Best for	Trade-off
Selenium	Broad language support, mature ecosystem	Slower than newer alternatives, more boilerplate per script
Playwright	Modern API, strong multi-browser support	Newer ecosystem, smaller community than Selenium's
Puppeteer	Deep Chrome/Chromium control, strong Node.js ecosystem	Chrome-first, less natural fit outside JavaScript
Managed browser infrastructure (e.g., Browserless)	Teams that don't want to own browser version management, scaling, or memory leaks at production volume	Ongoing cost, and it doesn't remove the need to write good scraping logic

If you're just getting started, run Selenium or Playwright locally to learn the fundamentals; the concepts transfer directly to a managed setup later. Reach for managed infrastructure once you outgrow "a script on my laptop" and need this running reliably, unattended, and at a volume where babysitting Chrome processes stops being a good use of engineering time. The walkthrough below builds exactly that, with Playwright, the same code you'd point at a managed browser once it outgrows your machine.

Scraping a dynamic page with Python

Here's what that looks like in practice as working Python code, end to end, against a real, freely scrapable target, quotes.toscrape.com/js. It's a page built specifically for scraping practice that renders its quotes via JavaScript rather than serving them as static HTML. The required libraries are minimal, just Playwright, the open-source library that drives the browser, and BeautifulSoup to parse the rendered output once you're ready. Nothing here is specific to this one target URL; the same pattern of launch, navigate, wait, extract holds for most dynamic pages you'll come across.

Setting up a browser automation script

Install Playwright and its browser binaries first.

pip install playwright beautifulsoup4
playwright install chromium

The first command installs the Playwright library; the second downloads the actual Chromium build it drives. Skip that second step and p.chromium.launch() fails with a missing-executable error the first time you run it. If you already know you're heading straight for managed infrastructure, you can skip playwright install chromium entirely, since no local browser binary is needed once you're connecting to a hosted one.

Then launch a browser, navigate to the target URL, and wait for the rendered content before you start scraping.

from playwright.sync_api import sync_playwright

def scrape_quotes(target_url: str) -> list[dict]:
    quotes = []
    with sync_playwright() as p:
        browser = p.chromium.launch(headless=True)
        try:
            page = browser.new_page()
            page.goto(target_url, wait_until="domcontentloaded")

            quote_items = page.locator(".quote")
            quote_items.first.wait_for()

            for item in quote_items.all():
                text = item.locator(".text").inner_text()
                author = item.locator(".author").inner_text()
                quotes.append({"text": text, "author": author})
        finally:
            browser.close()
    return quotes

if __name__ == "__main__":
    for quote in scrape_quotes("https://quotes.toscrape.com/js/"):
        print(quote["author"], "-", quote["text"])

That launch, navigate, wait, extract sequence is the core pattern for scraping data from any rendered page.

Waiting for content to load properly

The quote_items.first.wait_for() call above is doing important work. The goto() call only waits for domcontentloaded, the initial HTML; it's the locator wait on the next line, not the page load itself, that actually guarantees the quotes have rendered. A tempting shortcut is to add a fixed delay instead, something like import time and time.sleep(3), and hope the page finishes loading in that window, but that's a bad idea. Network speed varies, server response times vary, and a fixed sleep either wastes time waiting on a page that already loaded or gives up too early on one that's still rendering.

An explicit wait on a locator, request, or network condition is more reliable and usually faster in practice, since it proceeds the moment the condition is met instead of always waiting the full delay. Playwright's wait targets form a ladder from cheapest to most thorough, such as commit, domcontentloaded, load, and networkidle, plus waiting on a specific selector, network response, or custom event, and its current docs recommend starting at domcontentloaded and escalating only if the content genuinely needs it. Locator-based waits (locator.wait_for(), or simply acting on a locator, which auto-waits and retries) are recommended over older page-level methods like page.wait_for_selector(), which still work but are now marked discouraged in favor of the locator API. If a page loads content in stages, wait for the final piece you actually need rather than the first thing that appears.

Parsing the rendered HTML

Once Playwright has rendered the page, you don't have to keep using the browser API for every field. Each quote block on the page also carries a nested set of <a class="tag"> elements, and pulling those out with more Playwright locator calls gets verbose fast. Pulling the final HTML content with page.content() and handing it to BeautifulSoup is simpler once you're extracting more than a couple of specific elements. Add this inside scrape_quotes(), right after the wait, using the same page object already in scope:

from bs4 import BeautifulSoup

html_content = page.content()
soup = BeautifulSoup(html_content, "html.parser")

for quote_block in soup.select(".quote"):
    text = quote_block.select_one(".text").get_text(strip=True)
    author = quote_block.select_one(".author").get_text(strip=True)
    tags = [tag.get_text(strip=True) for tag in quote_block.select(".tags .tag")]
    print(author, "-", text, "-", tags)

This split, a browser to render the page and BeautifulSoup to parse HTML for data extraction, keeps your CSS selector logic in one familiar place instead of split across two different APIs, and it scales better once a single item on the page has more than one or two fields worth pulling out for further processing.

Handling infinite scrolling

The quotes.toscrape.com/js page used above loads all of its content on the initial render, so there's nothing left to scroll for. Its sibling page, quotes.toscrape.com/scroll, is built the other way.

It starts with 10 quotes and continues to load dynamically as you approach the bottom, the same pattern you'll hit on real feeds and listing pages, rather than paginating with page numbers.

To scrape one, scroll the page programmatically, wait for new elements to appear, and repeat until the count of elements stops growing.

previous_count = 0
while True:
    page.mouse.wheel(0, 15000)
    page.wait_for_timeout(1000)
    current_count = page.locator(".quote").count()
    if current_count == previous_count:
        break
    previous_count = current_count

Point that loop at quotes.toscrape.com/scroll and the count climbs from 10 to 100 across a few scroll-and-wait cycles until you've pulled all the data off the page.

Add a hard cap on iterations for any infinite scrolling web page where the scroll could theoretically never end, and pace your scroll-and-wait loop rather than firing it as fast as the browser allows.

Running the same script on managed infrastructure

Once a script like this works locally, moving it to managed infrastructure is a small change, not a rewrite.

# Before: a local browser
browser = p.chromium.launch(headless=True)

# After: the same browser, running on Browserless (TOKEN from your account dashboard)
browser = p.chromium.connect_over_cdp(
    f"wss://production-sfo.browserless.io?token={TOKEN}"
)

That's the line that changes, plus one detail worth knowing. Reuse the connection's default context rather than creating a new one, since a fresh context doesn't inherit the connection's proxy and launch settings. Concretely, replace page = browser.new_page() with page = browser.contexts[0].new_page() (or reuse browser.contexts[0].pages[0]). Everything downstream of that (locators, waits, extraction) stays exactly the same as the local version above.

Best practices for web scraping dynamic websites

These are the habits that keep a scraper running once it's live instead of breaking quietly a week later.

• Always close your sessions. Wrap the run in try/finally and close the browser even when the script throws. A leaked session keeps counting against your concurrency limit long after the scrape failed.
• Use explicit waits, not fixed sleeps. The walkthrough above covers why; the short version is your scraper adapts to real page timing instead of guessing at it.
• Respect robots.txt and the site's terms of service. Not every web page wants to be scraped, and checking first avoids both legal exposure and wasted engineering effort on a site that will block you anyway.
• Session reuse. Reuse sessions and cookies where you can. Starting a fresh browser context for every request is slower and looks more like automated traffic than a browsing session that persists state naturally.
• Don't impersonate other crawlers. Setting your user agent to Googlebot's to sneak past a block isn't a shortcut; it's a terms-of-service violation that can get your whole IP range flagged.
• Build in retries and failure handling. Dynamic pages fail in more ways than static ones. Partial renders, slow APIs, and timeouts are all normal. A scraper that assumes every request succeeds will quietly lose data.
• Watch for anti-bot challenges instead of assuming a script runs forever unattended. A page that suddenly returns a challenge screen instead of content needs a human to look at what changed, not a scraper set to retry indefinitely.
• Use proxy rotation and user agent rotation deliberately, not randomly. A residential proxy pool and a consistent, realistic user agent per session look far more like normal traffic than swapping both on every single request, which is itself an unusual pattern a detection system can key on.
• Log what you scrape and when. A simple record of run times, row counts, and failures makes it much easier to notice a silent break, a selector that stopped matching or an API that changed shape, before it costs you weeks of missing data.

Common pitfalls when scraping dynamic content

Dynamic-scraping bugs tend to trace back to a short list of repeat offenders, from anti-bot systems to selectors that quietly stop matching. Here's what actually trips scrapers up in practice, and why.

• Headless-browser detection. Some sites check for headless-specific markers like navigator.webdriver being true, missing browser plugins, or inconsistent header and TLS (Transport Layer Security) fingerprints. Headless-specific markers are one of the most common ways automation gets flagged, though it doesn't always mean you're blocked outright, just that it can trigger extra challenges. Where you can't change the site's mind, hardened browsers, residential proxies, and CAPTCHA handling shift the odds; see Browserless's bot detection docs for what each one actually addresses.
• Memory leaks from long-running Chrome instances. Headless browsers that never close accumulate memory over hours of continuous scraping. Closing a context with context.close() frees its pages but leaves the browser process running, so recycle the process itself too: call browser.close() and relaunch after a bounded time, request count, or memory threshold instead of running one instance indefinitely.
• Brittle CSS selectors. A selector tied to an auto-generated class name will break the next time the site ships a redesign, an especially common failure on dynamic web pages that rebuild their DOM on every deploy. Prefer stable attributes like a data-* attribute or an id, or semantic tags where the page provides them.
• Timing races. Reading the DOM a moment before the JavaScript finishes populating it returns partial or stale data. The explicit-wait pattern covered above exists specifically to prevent this.
• Pagination tokens that expire. Some dynamically loaded pages issue a short-lived token or cursor with each page of results. Store and reuse it immediately rather than batching requests for later, or you'll find the token has already expired by the time you get back to it.
• Treating every block the same way. A CAPTCHA, a login wall, and a rate-limit response all look similar at a glance, since the page just isn't giving you data, though each needs a different fix. A CAPTCHA needs solving, a login wall or WAF block needs an unblock step, and a rate limit needs backoff. Retrying a login wall accomplishes nothing; retrying a rate limit after a backoff often works fine.

Conclusion

Web scraping dynamic websites comes down to one decision. Automate a browser to get the page a user would get, reverse-engineer the API when you can find one stable enough to trust, or hand the browser workload to managed infrastructure once running it yourself becomes the bottleneck. Match the approach to the site in front of you, and build in the waits, retries, and session handling that keep a scraper working past its first successful run.

If you're already comfortable with Playwright or Puppeteer and tired of babysitting browser versions and memory leaks, point your existing code at a hosted endpoint instead of a local browser, and sign up for a free Browserless account.

A few more questions worth answering before you start.

Web scraping dynamic websites FAQs

Is web scraping dynamic content legal?

It's tied to the site, the data, and how you access it, not to whether the content happens to be dynamic. See our full breakdown of when web scraping is and isn't legal for the factors that actually shape the answer, like a site's terms of service and whether you're collecting personal data.

What's the difference between dynamic and static web scraping?

Static scraping reads content that's already present in the initial HTML response for that web page. Dynamic scraping means extracting data from pages whose content is built or changed by JavaScript after the initial load, either by automating a real browser to render it or by calling the same API the page's JavaScript calls, before the data you want exists anywhere to extract. Checking the browser's Network tab for XHR calls returning JSON is usually the fastest way to tell which one you're dealing with.

How do you scrape dynamic sites at scale without running your own browsers?

Point your existing Playwright or Puppeteer code at a managed browser endpoint instead of a local one, or call a rendering API directly if you'd rather skip the browser library entirely. Either way, someone else handles Chrome version drift, memory leaks, and concurrency limits, which is the operational work that breaks most scrapers once they leave a single laptop.

Can you scrape a dynamic website without a browser?

Yes, if you can identify and replicate the API call the page uses to extract data directly. That's faster and lighter than browser automation, but it needs that API to stay stable and reachable without a full browser session, which isn't guaranteed on every site.

Can you scrape a dynamic site with Selenium instead of Playwright?

Yes. Selenium is still widely used and supports more languages than most alternatives. Run pip install selenium, then start a session with driver = webdriver.Chrome() after from selenium import webdriver. Since Selenium 4.6, Selenium Manager automatically resolves and downloads a driver matched to your installed Chrome, so there's no driver version to track by hand.

What's the difference between the requests library, requests-html, and browser automation for dynamic pages?

Python's plain requests library only fetches raw HTML, so it can't see anything JavaScript adds. There's no reliable middle ground. requests-html is sometimes suggested here, but it has had no release since 2019 and drives a full Chromium under the hood through pyppeteer, so it buys you nothing over a maintained browser library. Either the data is in the raw HTML, or you need a real browser, which is the approach this blog post walks through.