Dev.to Mirror — The Set-and-Forget AI Crawler

🔗 Live Site: anchildress1.github.io/devto-mirror

This Copilot generated utility helps make your Dev.to blogs more discoverable by search engines by automatically generating and hosting a mirror site with generous robots.txt rules. Avoiding Dante’s DevOps and the maintenance headache. This is a simple html, no frills approach with a sitemap and robots.tx — that's it (although I'm slowly working through enhancements). If you're like me and treat some comments as mini-posts, you can selectively pull in the ones that deserve their own page.

Note

I'm slowly accepting that one or two brave souls might actually read my strong (and usually correct) opinions. 😅 I'm also always looking for ways to improve AI results across the board, because... well, somebody has to. 🧠

The internet already changed — blink and you missed it. We don't Google anymore; we ask ChatGPT (the wise ones even ask for sources). 🤖

• When I searched: my Dev.to showed up just as expected
• When I asked Gemini the same thing: crickets. 🦗

So yeah, obvious disconnect... Also, I'm not hosting a blog on my domain (I'm a backend dev; hosting a pretty blog + analytics sounds like a relaxing afternoon with Dante's DevOps. Hard pass. 🔥🫠), but I still want control of robots.txt.

Enter the five-minute ChatGPT fix: a tiny static mirror with canonicals back to Dev.to — no domain, no analytics — just (practically) instantly crawlable 😉🐜.

P.S. "Five minutes" usually means two hours. Acceptable losses. 😅 And seriously, writing this blurb took longer than the code. 🤨 Alright.... 3 hours (it took me an hour to get the picture just right, enough anyway). Still worth it! 😅

— Ashley 🦄

---

Repo Stuff

What is this?

Auto-generates a static mirror of your Dev.to blog with generous robots.txt for AI crawlers. Simple HTML, sitemap, canonical links — zero maintenance.

---

Quick Setup ⚡

1. Fork this repo
2. Set repository variable: Settings → Actions → Variables → Add DEVTO_USERNAME
3. Delete gh-pages branch if it exists
4. Run workflow: Actions → "Generate and Publish Dev.to Mirror Site" → Run workflow
5. Enable Pages: Settings → Pages → Deploy from branch → gh-pages

Auto-updates weekly (Wednesdays) at 9:40 AM EDT.

---

How it works

• Fetches posts via Dev.to API (incremental updates via last_run.txt timestamp)
• Generates HTML files with canonical links back to Dev.to
• Creates sitemap, robots.txt, index
• Optional: Include special comments as standalone pages via comments.txt

Refresh everything: Actions → "Generate and Publish Dev.to Mirror Site" and set force_full_regen to true (creates backup, wipes, regenerates)

---

Local Development

git clone https://github.com/anchildress1/devto-mirror.git
cd devto-mirror

# Install uv if you don't have it
curl -LsSf https://astral.sh/uv/install.sh | sh

# Or update to latest version
uv self --no-python-downloads update

# Create lockfile and sync dependencies
uv lock
uv sync --locked --group dev

# Copy and configure environment variables
cp .env.example .env
# Edit .env with your DEVTO_USERNAME and PAGES_REPO

# Run full validation
make validate

# Generate site locally
uv run python scripts/generate_site.py

For detailed development setup, CI/CD workflows, and troubleshooting: See docs/DEV_GUIDE.md

---

Documentation 📚

Additional documentation is available in the docs/ directory:

• Development Guide - Local development setup and commands
• CI/CD Guide - GitHub Actions workflows and deployment
• Security Analysis - Security recommendations and workflows
• Migration Plan - AI optimization refactoring progress

License 📄

Every project has to have a stack of fine print somewhere. Keep going, keep going, keep going... Here's mine, as painless as possible:

You know where the license is, but I'll sum it up: this is not open source (even though you can still do just about anything you want with it). As long as you're not turning it into the next big SaaS or selling subscriptions in the cloud, then have fun! Else, you've gotta ask me first.

Basically? This project's got boundaries. Be cool, don't try to sneak it into a product launch, and we'll get along just fine. 😘