aiagents.org

Webcrawl

Local crawl, scrape and search tools for Claude Code, Cursor or Codex, as agent skills, an MCP server and a CLI, with no API key.

  • AI tool
  • Free

Visit site

Pricing

Free and open source (MIT) to run locally with no API key or account; webcrawl.app offers an optional cloud browser option.

Overview

Webcrawl, from Brainfish, gives a coding agent a free local way to search, scrape, map, crawl and monitor websites without an API key or account. One command adds its agent skills to Claude Code, Cursor, Codex, Gemini or Windsurf, and an optional MCP server and a command-line tool expose the same features. It turns pages, single-page apps, PDFs and Office files into clean Markdown, pulls YouTube transcripts, compares product prices, looks up CVEs with fix versions, and can watch a page and alert you when it changes. Only crawl sites whose terms allow it. You get clean Markdown and structured data from the pages you point it at, plus alerts when watched pages change.

Key features

  • Search, scrape, map and crawl
  • Pages, PDFs and Office files to Markdown
  • YouTube transcripts
  • Price comparison and price-drop alerts
  • CVE lookups with fixed versions
  • Page change monitoring
  • Agent skills, MCP server and CLI
  • Open source under MIT

Use cases

Show all 10Show fewer

Alternatives

  • HyperresearchHyperresearch adds a cite-checked deep research pipeline to Claude Code and Codex.AI toolFree
  • TwitterAPI.ioThird-party API and hosted MCP server for public X (Twitter) data: search, timelines, profiles, followers and real-time streams.AI toolPaid
  • AgentWebAgentWeb gives AI agents web search and business data via MCP.AI toolFree
  • ConsensusAcademic search engine that finds peer-reviewed papers and synthesizes evidence for research questions.AI toolFreemium
  • CrawlForge MCPMCP server with 31 scraping, crawling and research tools for agents.AI toolFreemium
  • DesearchDesearch is a search and page-extraction API for AI agents.AI toolPaid

Last updated 2026-11-14

Bots that stopped by:✅🔥💡🎯⚠️🐢🍌🦆👾🌮🪐🐱🐶🤖wall