Scrape BNP Paribas ETC Product Details and Documents

Purpose

Scrape the BNP Paribas exchange-traded commodities product catalogue and enrich selected products with all key/value metadata, descriptive page content, and supporting document links. Preserve each document's nearby section heading so factsheets, prospectuses, KIDs, and related files can be distinguished reliably. The catalogue workflow also verifies that the market-entry gate is closed and records the product count for the selected region. When needed, compare the documents exposed for the same product after selecting different market-entry regions.

When to Use

Use for requests to inspect BNP Paribas ETC products, catalogue rows, product counts, or all available fields and documents on one or more product detail pages. Use the regional comparison variant when the caller asks whether a prospectus, Final Terms, KID, factsheet, or other document link differs between markets. If a valid detail URL is already supplied, use it directly for a single-region request. Otherwise resolve the product URL from the catalogue rather than guessing its opaque identifier. When a region-specific catalogue check is requested, select the requested market in the entry dialog before extracting rows.

Workflow

1. If the caller provides a product detail URL, navigate directly to it. Detail URLs have the form https://www.etp.bnpparibas.com/products/details/{product-slug}/{opaque-product-id}. Wait for #main .box-table.
2. If no detail URL is available, navigate directly to https://www.etp.bnpparibas.com/products#productlist. If a OneTrust dialog is present, click #onetrust-reject-all-handler. If the BNP market-entry dialog is present, select the requested region with .pop-con select (use {region}; observed valid values include Austria and Luxembourg), check .pop-con input[type="checkbox"]:not([name="save"]), and click .pop-con .btn.accept. Do not use the save checkbox. Wait for .desktop-tabel.etc-product-list.
3. Confirm the gate and count, then extract catalogue rows and their embedded detail URLs with this evaluator:

(() => {
  const table = document.querySelector(".desktop-tabel.etc-product-list");
  const gate = document.querySelector(".pop-con");
  if (!table)
    return {
      hasTable: false,
      count: 0,
      products: [],
      gateStillOpen: !!gate,
      error: "product table not found",
    };
  const headers = [...table.querySelectorAll("thead th")].map(
    (x, i) => x.textContent.replace(/\s+/g, " ").trim() || `column_${i + 1}`,
  );
  const products = [...table.querySelectorAll("tbody tr")].map((row, index) => {
    const values = {};
    [...row.querySelectorAll("td")].forEach((cell, i) => {
      values[headers[i] || `column_${i + 1}`] = cell.textContent
        .replace(/\s+/g, " ")
        .trim();
    });
    const rawHref =
      row.getAttribute("data-location-href") ||
      row.querySelector("a[href]")?.getAttribute("href") ||
      null;
    return {
      index,
      values,
      detailUrl: rawHref ? new URL(rawHref, location.origin).href : null,
    };
  });
  return {
    hasTable: true,
    count: products.length,
    gateStillOpen: !!document.querySelector(".pop-con"),
    url: location.href,
    products,
  };
})();

4. Navigate directly to each selected non-null detailUrl, wait for #main .box-table, and run this evaluator once per loaded detail page:

(() => {
  const clean = (value) => (value || "").replace(/\s+/g, " ").trim();
  const fields = {};
  document.querySelectorAll("#main .box-table td.title").forEach((label) => {
    const key = clean(label.textContent);
    const valueCell = label.nextElementSibling;
    if (key) fields[key] = valueCell ? clean(valueCell.textContent) : "";
  });
  const headings = [
    ...document.querySelectorAll(
      "#main h1, #main h2, #main h3, .details-page h1, .details-page h2, .details-page h3",
    ),
  ]
    .map((x) => clean(x.textContent))
    .filter(Boolean);
  const descriptions = [
    ...document.querySelectorAll("#main p, #main .description, .details-page p"),
  ]
    .map((x) => clean(x.textContent))
    .filter(Boolean);
  const links = [...document.querySelectorAll("#main a[href]")]
    .filter((a) => {
      const href = a.getAttribute("href") || "";
      const text = a.textContent || "";
      return (
        /\.pdf(?:$|[?#])|factsheet|prospectus|final.?terms|kid|document|download|fiche|brochure|sfdr/i.test(
          href,
        ) ||
        /factsheet|prospectus|final.?terms|kid|document|download|fiche|brochure|sfdr/i.test(
          text,
        )
      );
    })
    .map((a) => {
      let nearestHeading = "";
      let container = a.closest("section, li, div") || a.parentElement;
      for (
        let el = a.parentElement, depth = 0;
        el && depth < 6;
        el = el.parentElement, depth++
      ) {
        const h = el.querySelector(
          ":scope > h1, :scope > h2, :scope > h3, :scope > h4, :scope > .title, :scope > .section-title",
        );
        if (h) {
          nearestHeading = clean(h.textContent);
          container = el;
          break;
        }
      }
      return {
        text: clean(a.textContent).slice(0, 120),
        href: new URL(a.getAttribute("href"), location.href).href,
        nearestHeading,
        containerTag: container?.tagName || "",
        containerClass:
          typeof container?.className === "string" ? container.className : "",
      };
    });
  const tickerCandidates = [
    ...document.querySelectorAll(
      ".details-page .subtitle, .details-page .ticker, [class*=ticker]",
    ),
  ]
    .map((x) => ({
      className: typeof x.className === "string" ? x.className : "",
      text: clean(x.textContent),
    }))
    .filter((x) => x.text);
  return {
    url: location.href,
    fields,
    documents: links,
    headings,
    descriptions,
    tickerCandidates,
    hasPublicOfferText: document.body.innerText
      .toLowerCase()
      .includes("public offer"),
  };
})();

5. For a regional document comparison, use separate region selections or fresh browser contexts so the market-entry state is not reused accidentally. For each requested {region} (for example Germany, Austria, and Luxembourg), open https://www.etp.bnpparibas.com/products#productlist, select the matching option in .pop-con select by its actual value or visible label, accept without selecting save, then navigate to the same resolved detailUrl and run the detail evaluator. Compare documents by normalized text, nearestHeading, and absolute href; report the region alongside each result and distinguish missing links from identical links.
6. Join enrichment results to catalogue products by exact detailUrl; preserve products even when a detail page has no matching fields or documents.

Site-Specific Gotchas

• The catalogue is anchored at #productlist; use that URL instead of the homepage or search UI.
• The table selector contains the site's spelling error: .desktop-tabel.etc-product-list. Rows are under tbody tr and normally expose their detail URL through data-location-href.
• The table may render only after consent and market-entry handling. Do not use the .pop-con checkbox named save as the acceptance control.
• The market-entry dialog's region is selected through .pop-con select; observed option values include Austria and Luxembourg. For other markets, inspect the option values/labels and select the matching requested region. Treat the region as an optional caller-supplied parameter and report gateStillOpen with the catalogue count after acceptance.
• Region-specific document results depend on the selected market state. Re-select the region in a fresh context or session before comparing Germany, Austria, and Luxembourg; do not assume the same detail URL implies identical document links.
• Detail-page product identifiers are opaque lower-case URL segments. Use the catalogue's embedded URL or a caller-supplied detail URL; never derive an identifier from a product name.
• Detail metadata is stored as adjacent label/value cells: #main .box-table td.title followed by its value cell. Multiple boxes may be present, so collect every matching pair.
• Document links are identified by PDF URLs or document-related labels such as factsheet, prospectus, Final Terms, KID, document, download, fiche, brochure, and SFDR. The nearestHeading, container tag, and container class fields expose the document's surrounding structure and should be retained when the request asks for the exact document organization.
• Document links, descriptions, headings, and ticker-like elements are supplementary page content and may be absent on some products. The document selector is pattern-based and should be reviewed if BNP changes link labels or markup.

Expected Output

For catalogue requests, return {hasTable, count, gateStillOpen, url, products}. Each product contains its row index, header-keyed values, and normalized detailUrl. For detail enrichment, attach {url, fields, documents, descriptions, headings, tickerCandidates, hasPublicOfferText} to the corresponding product, or return that object directly for a single supplied detail URL. Each documents item contains normalized text, href, nearestHeading, containerTag, and containerClass. For regional comparisons, return one such detail result per {region} plus a comparison keyed by document label/heading, showing each region's link, identical/different status, and whether the document is absent.