Selector-cheap. LLM-resilient. Replay-safe. Self-hosted. A five-tier router that escalates HTTP → browser → stealth → agent only when reality demands.
pip install "scrapo-ai[browser,anthropic,mcp]"
Most pages return on Tier 0. When they don't, Scrapo climbs — not on your manual heuristics, but on real failure signals: status codes, anti-bot fingerprints, missing schema fields, unrendered SPA shells.
Known sites resolve through public APIs first — Wikipedia & Wikimedia skip the tiers (and the CAPTCHAs) entirely, reporting via="api:wikipedia". Then embedded JSON-LD, OpenGraph, and microdata are read with zero LLM cost. Many pages are answered before T0 ever fires.
A direct GET with sensible defaults. If the body returns 200 with content, this tier serves it. No browser, no proxy, no LLM.
Persistent sessions with cookie carry-over, retry budgets, and per-host concurrency limits. Still no browser. Most logged-in pages live here.
Full JS execution for SPA shells. Network interception is captured to the snapshot store so re-extractions are deterministic, never flaky.
Browser hardening, proxy adapters (Bright Data · Oxylabs · Scrapfly · Zyte), rotating identity with health checks. The escalator's express car.
LLM-driven runner for forms, paginations, and conditional flows. The last resort and the most expensive — so Scrapo treats it that way.
You can cap the ceiling per call (--max-tier 2) or let Scrapo decide. Either way, every escalation is logged with the trigger that caused it.
The first call to a new shape pays an LLM tax. Every subsequent call uses the cached selectors. When the page drifts, the cache rebuilds itself.
# A markdown-clean fetch with provenance.
import scrapo
res = await scrapo.scrape("https://news.ycombinator.com/")
print(res.markdown) # clean markdown
print(res.chunks[0].provenance) # url · selector · byte range · heading trail
print(res.run_id) # replayable, diffable, archived
# Typed extraction: LLM once, selectors forever.
from pydantic import BaseModel
import scrapo
class Offer(BaseModel):
title: str
price: float
currency: str = "USD"
class Listing(BaseModel):
page_title: str
offers: list[Offer] = []
# First call: LLM extracts + caches CSS selectors.
# Every call after: zero LLM cost, selector-based.
res = await scrapo.scrape(url, schema=Listing)
listing: Listing = res.data
# Watch a page — pay only when it actually changes.
import scrapo
w = await scrapo.watch("https://example.com/pricing", schema=Pricing)
change = await w.refresh() # conditional GET → 304 = free, zero LLM
if change.changed:
print(change.summary())
# Batch: one shared browser pool, per-URL error isolation.
items = await scrapo.batch_scrape(urls, schema=Product, main_content=True)
# No event loop? The sync API works in scripts and notebooks.
res = scrapo.scrape_sync("https://example.com/")
# Single page — cap the tier ceiling, export markdown + screenshot.
scrapo scrape https://example.com --max-tier 3 --out-md page.md --screenshot
# Crawl with budgets, or just map a site's URLs.
scrapo crawl https://docs.python.org/3/ --max-depth 2 --max-pages 100
scrapo map https://docs.python.org/3/ --out urls.txt
# Batch many URLs straight to JSONL / CSV.
scrapo batch https://a.com/ https://b.com/ --out-jsonl out.jsonl
# Deterministic replay + field-level diff — no network, no LLM.
scrapo replay <run_id>
scrapo diff <run_a> <run_b>
# Watch for changes · serve a local UI · expose an MCP server.
scrapo watch-add https://example.com/pricing --interval 3600
scrapo serve
scrapo mcp
Every fetch is archived. Re-extract from yesterday's HTML. Diff two runs field-by-field. The audit trail is the database.
The first run is LLM-driven; selectors are cached afterward. When a page drifts and the cache fails, Scrapo re-derives — quietly, automatically.
IP-obfuscation detection blocks internal targets. Opt-in robots gate. PII redaction on snapshots. Append-only audit log.
One command — scrapo mcp — gives Claude Code, Claude Desktop, and Cursor seven first-class tools: scrape, crawl, map, batch, replay, diff, list_runs.
Strict mode prevents silent LLM drift in production. Pin the model, the prompt, the schema version. Promotions are intentional.
Every extracted chunk carries its source: URL, selector path, byte range, heading trail. You can always trace a value back to its sentence.
Known sites resolve through public APIs before the tier ladder. Wikipedia & Wikimedia skip CAPTCHAs entirely and return via="api:..." — faster, cheaper, no browser.
JSON-LD, OpenGraph, Twitter tags, and microdata are read straight from the page before any selector cache or LLM call. Structured data costs zero tokens.
Monitor any URL with conditional GETs — 304 Not Modified is free. Self-hosted scheduler fires webhooks with a field-level diff only when content actually changes.
pip install "scrapo-ai[browser,anthropic,mcp]"