SSRF
A crafted redirect points your fetcher at cloud metadata, and suddenly your credentials are on the wire.
Infrastructure for web access
Giving a URL is easy. Fetching it safely is not.
webaccess goes to any web address on your behalf, crawls it, extracts structured content and measures it, while blocking SSRF, honoring robots.txt, and never losing a job. The hands and eyes for your web pipeline.
One HTTP API · X-API-Key · runs in your own network
Blind spot
Fetching the open web yourself is where pipelines quietly break.
Brittle
A JavaScript-only page returns an empty shell, your extractor stores nothing, and no one notices for weeks.
Impolite
No delay, no robots, no backoff, and the target rate-limits or blocks your IP within minutes.
One request
Ask once. Get structured evidence back.
POST /v1/crawl/start → runId
webaccess returns facts, never opinions. It fetches; you decide.
crawl
asyncCrawl a whole domain, politely, page by page.
returns pages, titles, text, links
fetch
Pull one URL with the full picture.
returns html, headers, redirect chain
cwv
headlessMeasure real performance in a browser.
returns LCP, CLS, TBT, TTFB
robots
Read a site's crawl rules, raw.
returns robots.txt body
sitemap
List every address a site publishes.
returns urls; index & .xml.gz
index
pgvectorChunk crawled pages for search and RAG.
returns chunks, embeddings
Why webaccess
Production-grade retrieval. Not a scraper you babysit.
Safe
SSRF-guarded
Internal ranges and cloud metadata are blocked, and the IP is re-checked at connect time to defeat DNS rebinding.
Polite
Rate-aware
One request per host at a time, a configurable delay between requests, and robots.txt Crawl-delay compliance out of the box.
Resilient
Never loses a job
Work lives in a durable queue. If a process dies mid-crawl, another worker takes over. PostgreSQL is the source of truth.
Plans
Start small, scale by pages fetched.
Sample pricing. Plans scale by crawled pages and requests per month.
Starter
$49/mo
- 10,000 pages per month
- All endpoints, HTTP + browser fallback
- robots, sitemap & Core Web Vitals
Scale
$199/mo
- 100,000 pages per month
- RAG indexing with embeddings
- Signed webhooks & priority queue
- Email support
Enterprise
Custom
- Dedicated infrastructure
- Custom SSRF allowlists & limits
- SLA with dedicated support
Questions
What does webaccess actually do?
It fetches, crawls, and measures web pages on your behalf and returns structured content. It never runs an LLM, scores, or makes decisions; that stays on your side.
How is it safe to point at arbitrary URLs?
Every connection is checked against internal IP ranges and cloud metadata endpoints, and the IP is re-validated at connect time to defeat DNS rebinding.
Does webaccess run JavaScript?
Only when needed. It tries fast HTTP first and falls back to a real headless browser only when a page looks like an empty SPA shell.
Is it multi-tenant?
Yes. You pass your own clientRef on every request; it is combined with your authenticated tenant, so data is isolated automatically.