CrawMcp

Self-hosted MCP server for web crawling that bypasses anti-bot walls and JS challenges, using RabbitMQ, MinIO, and Redis for asynchronous content handling with automatic cleanup.

→ bilginerdem/craw-mcp · listed on glama

Is this your MCP?

Claim it to get a verified publisher badge, a free copy of our full audit findings, and direct contact for any high-priority issues we find.

Install from

Glama

M8ven verifies MCPs across every public registry — install directly from whichever one you prefer.

// key findings

✅

No credential exfiltration, no sensitive file access, no obfuscation

Static analysis found nothing flowing your secrets to unexpected places.

✅

Open source with a license and README

Anyone can audit the code, the license is declared, and the publisher documents what it does.

🔐

You'll be asked for 2 credentials: MINIO_ACCESS_KEY, MINIO_SECRET_KEY

These are read from process.env at runtime. Make sure you trust where they’ll be sent.

// required environment variables

This server reads these from process.env. You'll be asked to provide them before it can run.

configCONTENT_BACKSTOP_DAYS— bucket (, default 1 day) stands as a safety net.

configCRAWL_REQUEST_QUEUE— crawl.requests

configCRAWL_RESULT_QUEUE— crawl.results

configJOB_TTL_SECONDS— , the MCP catches it via a Redis expired event and deletes the

🔐 secretMINIO_ACCESS_KEY— / MINIO_SECRET_KEY minioadmin / minioadmin

configMINIO_BUCKET— crawl-content

configMINIO_ENDPOINT— by changing and the keys.

🔐 secretMINIO_SECRET_KEY— MINIO_ACCESS_KEY / minioadmin / minioadmin

configRABBITMQ_URL— amqp://guest:guest@localhost:5672

configREDIS_URL— redis://localhost:6379

configS3_REGION

// full audit trail

The full breakdown of what we checked, the deductions that landed, the network hosts, the dependency advisories, and concrete fix guidance is available to verified publishers.

// improvement guidance — verified publishers only

We have 3 concrete improvements we can share with the publisher of this MCP. Each comes with specific guidance to raise the trust score.

// embed badge in your README

[![M8ven Score](https://m8ven.ai/badge/mcp/bilginerdem-craw-mcp-19syjp)](https://m8ven.ai/mcp/bilginerdem-craw-mcp-19syjp)

commit: b7407ec04211cc306319f591a8fb9cdd22a4e889

code hash: aa7271bc63aecedca584dc5a8b2fcad0bbb9920a9e8eabc7da2ce8f57ec52d38

verified: 7/6/2026, 10:39:38 AM

view raw JSON →