All repositories

RAG & vector DBsunclecode

crawl4ai

An open-source Python crawler: it visits pages with a real browser and returns clean Markdown for LLMs. Runs locally, in Docker or as a cloud service.

Who it is for

For Python developers collecting website data for RAG and agents.

How to start

  1. Install: pip install -U crawl4ai.
  2. Install the browser once with crawl4ai-setup, then verify with crawl4ai-doctor.
  3. Run AsyncWebCrawler().arun(url=...) from the README example and print result.markdown.

Steps are taken from the README. Check the current version in the repository before running them.

Stars over the last 30 days

+1,541Oct 22 — Nov 19
86,00087,541

Author's description

Official Crawl4AI Cloud: a hosted web scraping, extraction, and search API for AI agents. Free API key, 1,000 credits a month, no card. Fully open source: this repo is the Apache-2.0 crawler that powers it, and you can self-host it.

firecrawl

firecrawl

Growing fastest

A service that turns any website into clean Markdown or structured data for models. It can search, scrape, crawl a whole site and even click through a page.

203Kstars+9.3K in 30 dTypeScript

microsoft

markitdown

Growing fastest

A small Microsoft utility that converts PDF, Word, PowerPoint, Excel and other files into Markdown that is easy to hand to a language model.

196Kstars+3.7K in 28 dPython

infiniflow

ragflow

A ready-made RAG engine with a UI: it parses documents with templates, chunks them, searches with citations and supports agentic retrieval. Deploys with Docker.

94Kstars+1.8K in 30 dGo

opendatalab

MinerU

Parses complex PDFs, scans and Office files into Markdown or JSON while keeping tables and formulas. Comes with a CLI, a Python SDK and an agent skill.

84Kstars+1.3K in 28 dPython

docling-project

docling

An IBM library for document parsing: PDF, Office, HTML and images become one unified structure from which Markdown is easy to get. Supports vision models for hard pages.

71Kstars+1.5K in 30 dPython

Mintplex-Labs

anything-llm

A ready-made «chat with your documents» app: plug in any model and vector database, upload files and talk to them. Has a desktop version, Docker and agents.

68Kstars+998 in 30 dJavaScript