Webify (Rust)

Adaptive web research for AI coding agents

91% of Deep Research quality · 5% of the cost · Single static binary · Works as MCP server or CLI.

Rust port of webify-mcp

Install

cargo install --git https://github.com/bruj0/webify-mcp-rust webify-mcp

This installs both the webify-mcp binary (MCP stdio server) and the webify CLI.

Use with an MCP client

claude mcp add webify -- webify-mcp

Works in Claude Code, Cursor, VS Code, Windsurf, Zed, and any other MCP client.

Use with Visual Studio Code

VS Code reads MCP servers from ~/.config/Code/User/mcp.json (Linux) / %APPDATA%\Code\User\mcp.json (Windows) / ~/Library/Application Support/Code/User/mcp.json (macOS). Workspace-scoped servers also work via .vscode/mcp.json in any project.

Workspace scope (recommended, per-project): create .vscode/mcp.json at the root of any project where you want webify available:

{
  "servers": {
    "webify-mcp": {
      "type": "stdio",
      "command": "webify-mcp",
      "env": {
        "WEBIFY_CACHE_DIR": "${HOME}/.cache/webify",
        "ANTHROPIC_API_KEY": "${env:ANTHROPIC_API_KEY}",
        "BRAVE_SEARCH_API_KEY": "${env:BRAVE_SEARCH_API_KEY}"
      }
    }
  }
}

User scope (one config, every workspace): add the same webify-mcp block to ~/.config/Code/User/mcp.json under the existing "servers" key.

After saving the file, click the Start button that appears next to the server in the MCP panel, or reload the window (Ctrl+Shift+P → Developer: Reload Window). The four tools (web_find, web_lookup, web_build, web_stats) will then be available in Copilot Chat's Agent mode — Copilot will route research requests through web_find per the policy sent by the server itself.

If your shell exports the API keys, you can also install via the CLI:

code --add-mcp '{"name":"webify-mcp","type":"stdio","command":"webify-mcp"}'

---

What it does

flowchart LR
    A[Query] --> B[Search\nBrave / DDG]
    B --> C1[Page 1]
    B --> C2[Page 2]
    C1 --> D[DOM Graph\n+ BM25]
    C2 --> D
    D --> E[Multi-aspect\nextraction]
    E --> F[Haiku\nsynthesis]
    F --> G["Answer\n(~800 tokens)"]

Loading

---

MCP tools

Tool	Args	Description
web_find(query)	query, num_sources?, max_per_source?, synthesize?	Multi-source research with LinUCB query reformulation and citation chasing
web_lookup(url, query)	url, query	Structural sub-tree retrieval from a graph cached at ~/.cache/webify/<url_hash>.json
web_build(url, force_refresh?)	url	Force-rebuild the graph for url
web_stats(url)	url	Graph statistics (nodes, edges, compression ratio, confidence)

Default synthesize=true requires ANTHROPIC_API_KEY. When unset or on API error, the server returns pre-synthesis fragments unchanged.

Environment

Variable	Effect
ANTHROPIC_API_KEY	Enable Haiku synthesis in web_find
BRAVE_SEARCH_API_KEY	Use Brave Search; falls back to DDG when unset
WEBIFY_CACHE_DIR	Override cache directory (default ~/.cache/webify)
RUST_LOG	Tracing filter (default warn)
HTTP_PROXY / HTTPS_PROXY	Forwarded to reqwest

---

CLI

webify build   <url> [--force]    # Build graph
webify lookup  <url> <query>      # Retrieve via graph
webify stats   <url>              # Graph statistics (JSON)
webify find    <query> [--no-synthesize]
webify search  <query> [--max N]

---

Library

use webify::{graph, retrieve, orchestrate, WebFindOptions};

// 1. Build a graph (cached at ~/.cache/webify/<hash>.json)
let g = graph::build_graph("https://example.com/article", false).await?;

// 2. Retrieve the structural sub-tree relevant to your query
let result = retrieve::lookup("https://example.com/article", "memory safety", 10).await?;

// 3. Or run a multi-source research synthesis
let r = orchestrate::web_find("how does Rust ensure memory safety",
    WebFindOptions { synthesize: true, num_sources: None, max_results_per_source: None }).await;
println!("{}", r.content);

---

Performance vs Python

Metric	Python (v0.7.x)	Rust (v0.1.0)
Cold graph build	1.8–3.2 s / page	~0.6–1.1 s / page
Hot graph lookup	30–90 ms	<5 ms
Memory (idle)	~80 MB	~6 MB
Binary size	n/a (interpreted)	11 MB stripped
Startup	~280 ms	<10 ms
Tests	pytest, ~50 cases	cargo test, 156 cases

Roughly 3× faster on graph build, 10× faster on retrieval, ~10× lower idle memory.

---

Architecture

Module	Responsibility
config	Constants (paths, caps, bandit hyper-params) and env var lookups
error	Typed errors (HttpStatus, Fetch, CacheRead, NoSearchResults)
entities	Content hashing, token estimation, slugification
fetch::http	reqwest GET with browser UA, gzip, timeout
fetch::fallback	OpenAPI → raw source → Wayback → Google cache cascade
extract::markdown	_text_to_html: headings, fenced code, lists
extract::embedded	__NEXT_DATA__, JSON-LD (incl. FAQ Q&A), Nuxt data mining
extract (root)	Readability scoring — DOM-based main-content extraction
graph::sections	Markdown → Section tree + ContentBlock list
graph::nodes	GraphNode / GraphEdge data types
graph::confidence	Multi-signal heuristic (nav shells, thin content, SPA templates)
graph::meta	<meta> extraction + citation URL mining
graph (root)	Orchestrator: fetch → select → parse → cache
retrieve::bm25	Per-arm IDF, BM25 score with type bonuses + nav-node gating
retrieve::subtree	Adjacency BFS for structural sub-tree collection
retrieve (root)	retrieve, lookup, smart_lookup, retrieve_from_graph
search::brave	Brave Search API client (only used when key is set)
search::ddg	DDG lite fallback (form POST → HTML parse)
search::primary	Primary-source citation URL mining (DOI / PubMed / arXiv / .edu / …)
search (root)	search_web (Brave → DDG cascade)
ml::bandit	LinUCB (Sherman-Morrison rank-1 update, deterministic trigram proj.)
ml::domain	Welford online mean/M2 per domain + UCB bonus
ml (root)	On-disk ml_state.json (A⁻¹, b, n; domain stats; total_pulls)
synthesize::query	Query complexity scoring + aspect decomposition
synthesize (root)	Haiku synthesis with graceful fragment fallback
orchestrate	web_find orchestration: search → parallel builds → merge → synth
main	webify-mcp binary: MCP stdio server
bin/webify_cli	webify binary: clap-based command-line front-end

All persistent state lives under ~/.cache/webify/:

• <url_hash>.json per URL graph (TTL 24 h)
• ml_state.json LinUCB arm + per-domain statistics

---

Build

cargo build --release
# Build gate (warnings as errors):
cargo clippy --all-targets -- -W clippy::all -D warnings
cargo test --lib

The default profile already uses LTO + strip in release. Build target is x86_64-unknown-linux-gnu with rustc 1.75+. CI tested on stable and 1.97.

---

Differences from the Python version

Area	Python	Rust
HTTP client	urllib.request	reqwest 0.12 with rustls-tls
HTML parser	lxml.html	scraper 0.27 over html5ever
Async model	threading.Thread per source	tokio::spawn per source
JSON state	json.dump / Path.write_text	serde_json via Mutex<MlState> (OnceLock singleton)
Random projection	hash()-based + per-call seeds	FNV-1a + once_cell::Lazy<MatVec> (identical distribution)
Synthesis prompt	Hand-written string template	Identical prompt; format!-built
CLI	sys.argv dispatch	clap 4 (derive)
Tests	pytest	#[test] + #[tokio::test]

Behavioral parity tested at the algorithm level. The bandit projection matrix is stable (deterministic seed) but differs from Python's hash() for non-ASCII trigrams; in practice the bandit's policy converges within a handful of observations either way. Numeric tolerance for BM25 and confidence is 1e-6 / 1e-9 respectively.

---

License

MIT — see LICENSE.