The websites your team uses, as tools an agent can call.
HyperCrawl is a Nebula Labs research project. It turns approved website actions into callable tools, and the browser session and control layer stay on infrastructure we run.
Self-hosted · no third-party scraping service in the data path · MCP and REST
- 1API replay / hydration· no embedded data$0.000
- 2Learned tool· none saved for this site$0.000
- 3DOM agent· needle found · 42 rows$0.011
- 4Set-of-marks· not needed
- 5Vision· not needed
- 6Specialist· not needed
The job
Useful work lives behind logins and changing pages.
Brittle scripts break when a page changes, and third-party scraping services add another place for sensitive browsing data to travel.
- One script per site, broken by the next redesign
- Browsing data passes through someone else's service
- A blocked page comes back looking like an empty one
- Cost shows up after the run, not before
- The cheapest method first, escalating only on failure
- Runs on infrastructure we operate
- A block is reported as a block, with its code
- Every run stops at the budget it was given
No silent failures
A blocked page never comes back as an empty answer.
When every method a run is allowed fails, HyperCrawl returns a machine-readable code with the reason, so the agent and the person behind it know the data was not found rather than not there.
hypercrawl_acquire vendor-portal.example/invoicesx api replay no embedded datax dom agent challenge pagex vision challenge pageblocked: truecode: CHALLENGEspent: $0.032 of $0.05markdown: (none returned)
The ladder, rung by rung
Model inference only when cheaper methods fail.
Rungs 1, 2 and 6 use no model. Rungs 3 to 5 call one, and only after the cheaper rungs below them have failed. The caller's spend ceiling covers that metered inference. Flat-rate transports report no token usage, so a run on one of those is bounded by its time limit instead.
- Rung 1no model
API replay and hydration
Reads the data a page already ships as JSON. No browser, no model.
- Rung 2no model
Learned tool
A saved extractor for this site runs its stored selectors.
- Rung 3model inference
DOM agent
A language model reads the rendered page and acts on it step by step.
- Rung 4model inference
Set-of-marks
A vision model picks from numbered elements drawn over a screenshot.
- Rung 5model inference
Vision
A computer-use loop that sees the page the way a person does.
- Rung 6no model
Specialist
A configured provider for a platform that blocks everything else.
Measured on our own VM
Numbers from production, with the date we took them.
These come from running HyperCrawl on hypercrawl-vm. Your hardware and the sites you point it at will differ.
~460 MB
and 6 processes per browser, so the pool caps at 2 to 6 at once
hypercrawl-vm, e2-medium, 2 vCPU, 3.9 GB
~10 s
per vision step when the model runs through a CLI transport
DOM tier sibling adapter
1.3 s
for a search answer from self-hosted SearXNG (Bing: 2.3 s)
2026-10-10, 4-exit pool
0 → 10/10
results for one query, direct versus a 12-exit residential pool
live engine, captcha direct
Search has a 20 second budget and at most 6 seconds per engine, so a slow engine is skipped instead of stalling the run. Each browser is retired after 20 uses.
Self-healing extraction
When a site changes, the tool repairs itself.
A saved extractor first tries a structural heal of its selectors. When at least 30% of its fields still miss, HyperCrawl regenerates the tool and keeps the new version only if it repairs a field on the captured page and on a fresh refetch.
One regeneration runs per tenant and tool at a time, with a one-minute cooldown, and the stored field contract never changes underneath the agent that calls it.
{ "prompt": "current list price of each plan","format": "table" }search 10 unique sources selectedfetch 3 at a time, DNS-pinned, private IPs refusedextract learned tool for 4 of them9 sources answered · 1 reported a failureresult capped at 5 MB
For agents
The tools an agent gets.
Each capability is an MCP tool, and crawl, search, acquire and analyze are REST routes too, so Claude, Codex or your own agent can call them.
hypercrawl_crawlOne page to clean markdown, single-page apps includedhypercrawl_acquireThe ladder, with a needle to aim it and a spend ceilinghypercrawl_crawl_siteA whole site, with change tracking between crawlshypercrawl_mapEvery URL a site exposeshypercrawl_searchWeb search, self-hosted, optionally scraping the resultshypercrawl_generateTurn a page into a saved, reusable extractorHow it runs
From a site to a saved tool.
- 01
Name the target
A URL and the thing you need from it.
- 02
Set the budget
Each run gets a spend and time ceiling.
- 03
Climb the ladder
Cheap methods first, heavier ones only on failure.
- 04
Save the tool
A site that repeats becomes a reusable extractor.
Head to head
How it compares.
Pick a tool to see where HyperCrawl differs. Each row is drawn from that product's own public positioning.
| Compare | HyperCrawl | FirecrawlCrawl and scrape API |
|---|---|---|
| Where it runs | Infrastructure we operate | Their cloud, or self-hosted |
| Escalates only on failure | Six-rung ladder, models only at rungs 3 to 5 | Single strategy per call |
| Blocked pages reported as blocked | Machine-readable code and reason | HTTP status |
| Spend ceiling per run | Set by the caller | Credits per page |
| Saves a reusable extractor | Via your own code | |
| Self-healing when a site changes | 30% miss threshold, verified before promotion | |
| Source | Nebula Labs, being open-sourced | Open source core |
When Firecrawl is the better pick: You want a simple page-to-markdown API and will handle retries yourself.
Competitor rows describe each product's own public positioning as of November 2026. Check their sites for current details.
The honest limit
Start here
HyperCrawl is a Nebula Labs project, being released as open source. If your team still copies data out of websites by hand, bring that to a call.