In this project, I explored multi-agent systems by building a web text collection pipeline with LangGraph. Work is divided across specialized agents — planner, explorer, harvester, and auditor — that coordinate to discover, extract, and validate text data from the web.
It crawls the web, then runs collected text through a two-stage quality audit combining LLM-as-a-Judge evaluation with embedding similarity checks to filter noise and near-duplicates.
The project explores practical agentic architecture questions: task decomposition across agents, structured hand-offs, and automated quality control for data pipelines.
Resources
- Agents (Chip Huyen) — useful framing for how agents plan, use tools, and fail
- LangGraph documentation — the framework used to wire the planner → explorer → harvester → auditor loop
- Crawl4AI — async web crawling library used in the harvest stage
- LLM-as-a-Judge (Zheng et al.) — the evaluation pattern behind the auditor’s relevance scoring
- Building Effective Agents (Anthropic) — practical guidance on when to split work across agents vs. keep it in one loop
- sentence-transformers — library for the embedding models used for similarity filtering against reference examples