WebCrawlerAPI
WebCrawlerAPI provides crawling, page scraping, AI web agents, and monitored website feeds through a unified API.
What agents can do
Actions10
Create crawl
Start a metered asynchronous crawl over a site and return its job ID.
Create feed
Create a recurring website-change feed.
Delete feed
Permanently cancel a feed so it cannot be resumed.
Get crawl job
Get a crawl job's status, configuration, per-page results, content URLs, errors, and observed costs.
Get feed
Get one feed's configuration, lifecycle status, and recent run history, including per-run crawl counts and cost.
Get organization costs
Return current spendable balance plus request count and USD usage for a date range.
List feed changes
Return one page of detected feed changes as structured JSON, with an opaque continuation cursor for older pages.
List feeds
List active and paused feeds for the connected organization, newest first.
2 moreFewer
Scrape page
Scrape one web page synchronously and return requested content or structured extraction.
Set feed state
Pause an active feed's future scheduled runs or resume a paused feed.
How it connects
You paste an API key from WebCrawlerAPI on a page Composio hosts, so the key never passes through Definable.
Keep the connection to yourself or share it with your workspace. Any action that writes or sends can wait for your approval.
More in AI Web Scraping
OpenWeb Ninja
ai web scraping
OpenWeb Ninja provides native APIs for web search, AI answers, e-commerce, finance, jobs, maps, news, real estate, contact enrichment, traffic, and web scraping.
Kadoa
ai web scraping
Kadoa is an API-first platform that enables users to create, manage, and monitor data extraction workflows from unstructured data sources.
ZenRows
ai web scraping
ZenRows is a web scraping API allowing developers to bypass CAPTCHAs and blocks, gather structured data from dynamic websites, and quickly integrate results into applications
Botster
ai web scraping
Botster automates web data collection through configurable bots, job lifecycle management, and structured run results.
Cloro
ai web scraping
Cloro provides structured data from Google Search and leading AI assistants, with synchronous extraction and asynchronous task APIs.
Zyte Api
ai web scraping
Zyte API is a web scraping solution that automates data extraction from websites, handling challenges like bans and dynamic content.