Web Scraping Overview

Scrape any website with automatic anti-bot bypass, JavaScript rendering, and AI-powered data extraction.

Web Scraping API

Scrape any public webpage and get back clean HTML, Markdown, or plain text. The API handles anti-bot protection, JavaScript rendering, and proxy rotation automatically.

Key Features

Smart Engine Selection

Automatically picks the best scraping approach for each site — from fast HTTP requests to full browser rendering.

Anti-Bot Bypass

Detects and bypasses Cloudflare, DataDome, Akamai, Kasada, and other protection systems.

AI Extraction

Extract structured data from any page using natural language prompts powered by LLMs.

Video Recording

Record browser sessions as animated GIFs for debugging, visual verification, and monitoring.

Screenshots

Capture full-page PNG screenshots of any webpage.

Geo-Targeting

Route requests through proxies in 37+ countries to access location-specific content.

How It Works

1. Send a URL to the /v1/web/scrape endpoint
2. ScrapeBadger picks the best approach — fast HTTP for simple pages, headless browser for JavaScript-heavy sites
3. Anti-bot detection runs automatically — if a block is detected, the request retries with more powerful methods
4. Get clean content back in your preferred format (HTML, Markdown, or plain text)

Scraping Tiers

ScrapeBadger uses a tiered system. When engine is set to "auto" (the default), the most cost-effective method is tried first and the system escalates automatically if needed.

Tier	Description	Cost
HTTP	Fast HTTP request with Chrome TLS fingerprint — works for most websites	1 credit
Browser	Full headless browser with JavaScript rendering — for SPAs and dynamic pages	5 credits
Premium Browser	Real browser with advanced fingerprinting — for heavily protected sites	10 credits

Auto-Escalation

When escalate is enabled, ScrapeBadger automatically tries more powerful methods if the initial one is blocked:

HTTP (1 credit) → Browser (5 credits) → Premium Browser (10 credits)

You only pay for the method that succeeds — escalation costs are not cumulative.

Credit Costs

Component	Cost
Basic HTTP scrape	1 credit
Browser rendering (render_js: true)	5 credits
Premium browser (via escalation)	10 credits
Anti-bot solver (anti_bot: true)	+5 credits
AI extraction (ai_extract: true)	+2 credits
Video recording (video: true)	+3 credits
Retries	Free
Failed requests	0 credits

Use the max_cost parameter to set a credit budget per request. The request will fail rather than exceed your budget.

Quick Example

cURL

curl -X POST "https://scrapebadger.com/v1/web/scrape" \
  -H "x-api-key: YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://scrapebadger.com",
    "format": "markdown"
  }'

import requests

response = requests.post(
    "https://scrapebadger.com/v1/web/scrape",
    headers={"x-api-key": "YOUR_API_KEY"},
    json={
        "url": "https://scrapebadger.com",
        "format": "markdown"
    }
)

data = response.json()
print(data["content"])

const response = await fetch("https://scrapebadger.com/v1/web/scrape", {
  method: "POST",
  headers: {
    "x-api-key": "YOUR_API_KEY",
    "Content-Type": "application/json"
  },
  body: JSON.stringify({
    url: "https://scrapebadger.com",
    format: "markdown"
  })
});

const data = await response.json();
console.log(data.content);

Endpoints

Endpoint	Method	Description
/v1/web/scrape	POST	Scrape a URL and return content
/v1/web/detect	POST	Detect anti-bot and CAPTCHA systems on a URL

Next Steps

Scrape Endpoint

Full API reference for the scrape endpoint

Detect Endpoint

Analyze a URL for anti-bot protection