Crawlab is a distributed web crawler admin platform supporting multiple programming languages and frameworks. This template deploys the master node with MongoDB.
Requires Docker runtime or Docker Compose environment.
Explore discussions, issues, and documentation to see how this project fits your workflow.
GitHub stars measure repository interest, not customer satisfaction.
Standard Docker Compose deployment
version: '3.8'
services:
crawlab-master:
image: crawlab-master:latest
restart: always
ports:
- "8080:8080"Crawlab (Master) is a modern self-hosted, privacy-first developer tools solution. Crawlab is a distributed web crawler admin platform supporting multiple programming languages and frameworks. This template deploys the master node with MongoDB. Built for users and teams demanding full data ownership, Crawlab (Master) eliminates proprietary vendor lock-in while delivering an intuitive, reliable experience.
These tools share use cases. Features, costs, and deployment needs differ.
A coding-agent skill for multi-phase security audits with independently verified, machine-readable findings
Storybook is the industry standard workshop for building, documenting, and testing UI components in isolation
GitHub stats captured Oct 5, 2026.