crawler-mcp

Web-crawler MCP server that fetches web pages with static parsing or headless browser fallback, enabling Claude to fetch, extract links, crawl sites, and select content via CSS selectors.

→ shadab15github/crawler-mcp · listed on glama

Is this your MCP?

Claim it to get a verified publisher badge, a free copy of our full audit findings, and direct contact for any high-priority issues we find.

Install from

Glama

M8ven verifies MCPs across every public registry — install directly from whichever one you prefer.

// key findings

✅

No credential exfiltration, no sensitive file access, no obfuscation

Static analysis found nothing flowing your secrets to unexpected places.

// required environment variables

This server reads these from process.env. You'll be asked to provide them before it can run.

configCRAWLER_MAX_CHARS— 20000 Default cap on returned page content.

configCRAWLER_TIMEOUT_MS— 15000 Per-request timeout.

configCRAWLER_USER_AGENT— crawler-mcp/1.0 … User-Agent for all requests.

configMCP_HTTP_PORT— 3001 Port for HTTP mode.

configMCP_TRANSPORT— stdio stdio or http.

// full audit trail

The full breakdown of what we checked, the deductions that landed, the network hosts, the dependency advisories, and concrete fix guidance is available to verified publishers.

// improvement guidance — verified publishers only

We have 4 concrete improvements we can share with the publisher of this MCP. Each comes with specific guidance to raise the trust score.

// embed badge in your README

[![M8ven Score](https://m8ven.ai/badge/mcp/shadab15github-crawler-mcp-lpor5m)](https://m8ven.ai/mcp/shadab15github-crawler-mcp-lpor5m)

commit: 1f9ca21774a8a045e079a94a529e648ee40dddd2

code hash: 09a4a5d27d1b4d31a51ade69022f9d1c27b0311200a68956a35f39f4f70fa9cd

verified: 6/13/2026, 10:04:48 AM

view raw JSON →