ngx_http_pow_gate_module

Protocol v2 / 0.5.0: challenges commit the client key and address/UA/policy context, require key possession and redeem once across nginx workers. Clearances use the same binding and enforce minimum difficulty. The default replay store uses 2 MiB shared per nginx instance, survives graceful reloads and fails closed at capacity. Its full restarts invalidate tokens. Optional Redis coordination supports multiple instances and survives restarts. V1 cookies/clients are not accepted. See security.

Builds and CI always cover latest pinned mainline 1.31.8 and stable 1.30.6, each on glibc/musl and amd64/arm64. Select a channel using ./scripts/build-nginx.sh stable --target nginx-smoke-debian.

A native nginx dynamic module that puts a proof-of-work (PoW) gate in front of your sites to deter scrapers, AI crawlers, and abusive bots — while letting real browsers, trusted networks, and verified good bots through with no friction.

It is written in Rust against ngx (the official nginx-rust bindings) and plugs into nginx's ACCESS phase. Crucially, it does not reinvent matching: trusted networks come from the built-in geo module, and per–User-Agent decisions come from the built-in map module. The module only adds what stock nginx can't do: issuing/verifying proof-of-work, clearance cookies, and a verified good-bot allowlist (official IP ranges + FCrDNS).

Status: functional end to end, including the browser solver, good-bot verifier and request replay. See testing for the tested versions, local review results and configured CI matrix.

Why a PoW gate

A proof-of-work gate makes every un-trusted client spend a small amount of CPU before it reaches your upstream. For a human loading a page once, that's a few hundred milliseconds, invisible behind a "Just a moment…" screen. For a scraper acquiring many client clearances, that cost accumulates — without IP blocklists, CAPTCHAs, or a third-party service.

The gate is layered so most traffic never pays:

Client	Outcome
Trusted network (your LAN, your LB)	pass — `geo` match
Verified good bot (Googlebot, Bingbot…)	pass — IP-range + FCrDNS check
Known-abusive UA (GPTBot, Bytespider…)	`403` — `map` → `deny`
Already-cleared browser (valid cookie)	pass — clearance cookie + proof
Everyone else	challenge — solve, then pass

The challenge itself comes in tiers, chosen per client by the same map (see docs/configuration.md › Decision values): challenge:js (prove JS execution, no measurable work), challenge / challenge:<N> (proof-of-work at the configured or a per-decision difficulty, solved multi-threaded across Web Workers), and challenge:nojs[:secs] (a meta-refresh wait for text browsers like lynx/w3m — no JavaScript required).

How it works (at a glance)

flowchart LR
    C[Client] -->|request| N[nginx]
    subgraph N[nginx worker]
      direction TB
      P1[REWRITE phase] --> GEO["geo $pow_trusted<br/>(built-in)"]
      GEO --> MAP["map $http_user_agent → $pow_decision<br/>(built-in)"]
      MAP --> A["ACCESS phase<br/><b>pow_gate_access</b> (this module)"]
    end
    A -->|DECLINED| U[(upstream / proxy_pass)]
    A -->|403| D[denied]
    A -->|200 challenge page| C

The module reads the results of the native geo/map engines as the variables $pow_trusted and $pow_decision, then makes the final call in the ACCESS phase. NGX_DECLINED means "I have no objection" → nginx continues to your upstream.

Request decision flow

This is the exact branch logic in src/access.rs — first match wins:

flowchart TD
    start([request enters ACCESS phase]) --> en{pow_gate on?}
    en -- no --> pass1[/NGX_DECLINED → upstream/]
    en -- yes --> tr{trusted == 1?<br/>$pow_trusted via geo}
    tr -- yes --> pass2[/NGX_DECLINED → upstream/]
    tr -- no --> dec{$pow_decision?}

    dec -- allow --> pass3[/NGX_DECLINED → upstream/]
    dec -- deny --> forbid[/403 Forbidden/]
    dec -- "verify:NAME" --> ver{verifier confirms IP?<br/>ip_ranges + FCrDNS}
    dec -- "challenge / empty" --> clr

    ver -- yes --> pass4[/NGX_DECLINED → upstream/]
    ver -- no --> clr{valid clearance?<br/>cookie HMAC + per-request proof}

    clr -- yes --> pass5[/NGX_DECLINED → upstream/]
    clr -- no --> post{"body to keep?<br/>POST/PUT/… + pow_gate_replay"}
    post -- yes --> capture[/"200 challenge page<br/>+ the captured request"/]
    post -- no --> serve[/200 challenge page/]

A challenged POST would otherwise be lost: the gate answers the submit with the challenge page, and the reload afterwards is a GET. So the module reads the body and hands the request back inside that page, where the solver re-issues it once the clearance is minted — no request payload is retained server-side. See pow_gate_replay.

The challenge / solve / verify handshake

When a client is challenged, it gets an HTML page that loads the solver. The solver talks to three internal routes the module owns under pow_gate_endpoint (default /.pow/):

sequenceDiagram
    autonumber
    participant B as Browser
    participant M as nginx + pow_gate
    participant U as Upstream

    B->>M: GET /  (no clearance)
    M-->>B: 200 challenge.html  (loads solver.js)

    B->>B: Generate or load client keypair
    B->>M: GET /.pow/challenge (X-Pow-Key)
    M-->>B: { version, salt, difficulty, exp, token }
    Note over B: find nonce: SHA256(salt‖nonce) < target

    B->>M: POST /.pow/verify { version, salt, nonce, pubkey, signature, ... }
    Note over M: check binding, key possession and PoW,<br/>atomically consume challenge
    M-->>B: 204  Set-Cookie: pow_clearance=<HMAC token>

    B->>M: GET /  (Cookie: pow_clearance + X-Pow-Proof)
    Note over M: HMAC verifies, not expired,<br/>per-request proof signed by bound key
    M->>U: proxy_pass (NGX_DECLINED)
    U-->>B: your page

Two tokens, two jobs:

Clearance cookie (pow_gate_clearance_ttl, default 12h) — proves the client did the work. HMAC-signed with your server key; bound to a client keypair. Long-lived on purpose: a human solves once, then browses for the day without being re-challenged.
Per-request proof (pow_gate_proof_skew, e.g. 5s; stamped with server time, so wrong device clocks do not break it) — a DPoP-style signature proving possession of the clearance key for a recent request. Address/UA binding restricts cookie transfer; the signature adds key possession on requests carrying a proof. Demanded only from requests that can carry a custom header — fetch/XHR, marked Sec-Fetch-Dest: empty — and only with pow_gate_require_proof on (default off: it needs your pages to sign proofs on their own fetch/XHR calls); navigations and tag subresources (scripts, images, styles, fonts) always pass on a valid context-bound cookie.

See docs/protocol.md for token formats and endpoint contracts, and docs/architecture.md for the threat model.

Quickstart

load_module modules/ngx_http_pow_gate_module.so;

http {
    pow_gate_hmac_key_file /etc/pow/hmac.key;   # the one server secret
    pow_gate_difficulty    50000;               # ~expected hashes per solve

    geo $pow_trusted {                          # built-in nginx module
        default        0;
        10.0.0.0/8     1;
    }

    map $http_user_agent $pow_decision {        # built-in nginx module
        default                       challenge;
        ~*(gptbot|ccbot|bytespider)   deny;
        ~*(googlebot|bingbot)         verify:search_engines;
    }

    server {
        location / {
            pow_gate          on;
            pow_gate_trusted  $pow_trusted;
            pow_gate_decision $pow_decision;
            proxy_pass http://backend;
        }

        # exclusions are just locations with the gate off — no new syntax
        location = /robots.txt { pow_gate off; proxy_pass http://backend; }
    }
}

A complete, commented example — including a pow_gate_verifier block and path exclusions — is in examples/nginx.conf.

Configuration

Every directive at a glance. Full reference (contexts, defaults, edge cases) in docs/configuration.md.

Directive	Context	Arg	Purpose
`pow_gate`	http, server, location	`on`\|`off`	Enable/disable the gate (inheritable flag)
`pow_gate_trusted`	server, location	`$var`	Non-`0` ⇒ allow, skip the gate (usually `geo`)
`pow_gate_decision`	server, location	`$var`	`allow`\|`deny`\|`challenge`\|`verify:<name>`
`pow_gate_page`	http, server, location	`<file>`	Custom challenge page (templated; default embedded)
`pow_gate_difficulty`	http, server, location	`N`	Expected hash count per solve
`pow_gate_hmac_key_file`	http, server, location	`<file>`	Server secret for clearance/proof signing
`pow_gate_clearance_ttl`	http, server, location	`<time>`	How long a clearance lasts (default `12h`)
`pow_gate_proof_skew`	http, server, location	`<time>`	Per-request proof validity window (e.g. `5s`)
`pow_gate_require_proof`	http, server, location	`on`\|`off`	Demand `X-Pow-Proof` on fetch/XHR (default `off`)
`pow_gate_endpoint`	http, server, location	`<prefix>`	Internal route prefix (default `/.pow/`)
`pow_gate_replay`	http, server, location	`on`\|`off`	Keep a challenged `POST` (any data-carrying method) alive across the challenge (default `on`)
`pow_gate_replay_max_body`	http, server, location	`<size>`	Largest body captured for that replay (default `1m`, matching `client_max_body_size`)
`pow_gate_cookie_name`	http, server, location	`<name>`	Clearance cookie name (default `pow_clearance`)
`pow_gate_cookie_domain`	http, server, location	`<domain>`	Cookie `Domain=` (default host-only)
`pow_gate_cookie_path`	http, server, location	`<path>`	Cookie `Path=` (default `/`)
`pow_gate_cookie_samesite`	http, server, location	`Lax\|Strict\|None`	Cookie `SameSite=` (default `Lax`)
`pow_gate_cookie_secure`	http, server, location	`on`\|`off`	Cookie `Secure` flag (default `on`)
`pow_gate_cookie_httponly`	http, server, location	`on`\|`off`	Cookie `HttpOnly` flag (default `on`)
`pow_gate_verifier <name>`	http (block)	`{ … }`	Verified good-bot allowlist (IP ranges + FCrDNS)

Every directive except the pow_gate_verifier block inherits http → server → location and can be overridden at any level — set the tunables once high up, flip exceptions low down. The verifier block is http-only because it registers a global named allowlist used by verify:<name> from anywhere. Inside the block: ip_ranges_url, ip_ranges_refresh, fcrdns_suffix, fcrdns_ttl — see docs/configuration.md.

The decision values understood by pow_gate_decision:

allow — let it through.
deny — 403, no challenge.
challenge — must solve PoW (the default for unknown clients).
verify:<name> — run the named pow_gate_verifier; pass only if it confirms the connecting IP really belongs to that bot.

Project layout

A virtual workspace: the root Cargo.toml is only workspace glue; every crate lives under src/ with its own src/ (and tests/).

ngx_pow/
├── Cargo.toml                     [workspace] only (not a package)
├── src/
│   ├── ngx-http-pow-gate/         the nginx dynamic module (cdylib)
│   │   ├── Cargo.toml  build.rs   nginx link glue + asset rebuild triggers
│   │   └── src/
│   │       ├── lib.rs             module registration: ngx_module_t, ctx, ngx_modules!
│   │       ├── config.rs          directives, MainConf/LocationConf, create + merge
│   │       ├── access.rs          ACCESS-phase handler — allow/deny/challenge decision
│   │       ├── challenge.rs       challenge-page + internal /.pow/ endpoints
│   │       ├── replay.rs          capture a challenged POST so it survives
│   │       ├── verifier.rs        pow_gate_verifier {} — good-bot allowlist
│   │       └── engine/            thin nginx shell over the core (FFI seam)
│   └── pow-gate-core/             the engine — NO nginx dep, unit-tested
│       ├── Cargo.toml
│       ├── src/     target · codec · mac · pow · clearance · proof · replay
│       ├── tests/   one test crate per concern (their own projects)
│       └── benches/ Criterion microbenchmarks (engine hot path)
├── tests/integration/            black-box e2e client (standalone project)
├── perf/                         HTTP load generator (standalone project)
├── assets/
│   ├── challenge.html             embedded fallback page (themed, i18n)
│   └── solver.js                  browser solver (keygen, PoW loop, verify, proof)
├── docker/                       Dockerfile (multi-stage) + nginx.test.conf
├── docker-compose.test.yml       live e2e   ·  docker-compose.perf.yml  load test
├── examples/nginx.conf           full commented configuration example
├── scripts/                      test.sh (pipeline) · perf.sh (load test)
└── docs/  architecture · configuration · build · protocol · challenge-page · testing · troubleshooting · performance

The cryptography lives in src/pow-gate-core (no nginx dependency, so it unit-tests anywhere); src/ngx-http-pow-gate is a thin FFI shell that reads the request and calls into it. See docs/testing.md.

Building

# build against a CONFIGURED nginx source tree (./configure first — see below)
export NGINX_SOURCE_DIR=/path/to/nginx-1.31.8
cargo build --release -p ngx-http-pow-gate

# the artifact, renamed to the name used in load_module:
cp target/release/libngx_http_pow_gate.so \
   /etc/nginx/modules/ngx_http_pow_gate_module.so

ABI warning: a dynamic module must be built against the same nginx version and ./configure arguments as the nginx that loads it, or nginx refuses it (module ... is not binary compatible). Check nginx -V.

Full instructions (toolchain, env vars, Docker build, matching nginx -V, troubleshooting) are in docs/build.md.

Performance

The v0.5.0 performance pass reuses resolved configuration and HMAC key schedules, borrows request headers, and enables native HMAC/P-256 verification in the module. Crawler-range membership uses a sorted interval index. The standalone engine keeps its pure Rust default. The content-fingerprinted solver is shared by the page and its workers: Chromium and Firefox each fetched one copy instead of nine in the local eight-worker comparison (30 KB instead of 268 KB of script bodies). Performance tests run only on Linux/amd64. A follow-up reduces normal clearance validation allocations from six to one, reuses decoded proof keys and serves immutable pages without per-request body copies. CPU and HTTP measurements, conditions and limits are recorded in docs/performance.md.

The default local replay backend reserves one 2 MiB shared-memory zone per nginx instance, with 32,768 entries and bounded probing. Cookie checks do not lock that table. DNS jobs, resolver threads, verdict caches and verification bodies are bounded. Full methodology, memory snapshots and reproduction commands are in docs/performance.md. Security guarantees, limitations and the optional Redis cluster backend are in docs/security.md. A complete initial global limits include is available for regular traffic; it starts in dry-run mode for tuning.

What's implemented

The module is functional end to end. Local v0.5.0 validation covers both crypto backends, core tests and native glibc/amd64 builds and live HTTP/1.1, HTTP/2 and HTTP/3 suites on nginx 1.31.7 and 1.30.6. The allocation follow-up repeats the complete glibc/musl Docker matrix on amd64. The earlier performance pass also passed the arm64 matrix under QEMU for correctness. Performance tests run only on Linux/amd64. See docs/testing.md.

Area	Evidence
Crypto engine — PoW, clearance, ECDSA proof, IP-ranges (`src/pow-gate-core`)	78 portable / 79 accelerated tests green (`cargo test -p pow-gate-core`, with/without `--features native-crypto`)
Browser solver (`assets/solver.js`)	real WebCrypto keygen + 256-bit PoW + proof + IndexedDB
Challenge page (`assets/challenge.html`)	themed (light/dark) + i18n (26 languages)
Directives, config structs, inheritance merge	compiles + `nginx -t` passes
Mainline 1.31.7 and stable 1.30.6 module builds	native glibc/amd64 load and live suites passed; full libc/arch CI configured
ACCESS handler + phase registration	handler runs (registered in `postconfiguration`)
`/.pow/` endpoints — challenge JSON, solver, async `POST /verify` body	served correctly in the live test
Response I/O — challenge page, JSON, `Set-Cookie`, `204`	written via the `ngx` output chain
Clearance cookie + per-request proof validation	cookie + `X-Pow-Proof` checked against the cookie-bound key
Full handshake: challenge → solve → verify → cleared	e2e stage green
Challenged `POST` survives — body captured into the page, re-issued by the solver	e2e: the payload comes back verbatim; `off` / oversized bodies fall back
Good-bot verifier — `pow_gate_verifier {}` block, IP-range feed fetch + refresh, CIDR match, FCrDNS, client-IP parse	e2e: a `verify:<name>` UA is allowed via the live IP-range feed

Known limitations

A challenged request is only replayed when the challenge page can run: a fetch/XHR call gets the HTML page as its answer and must re-solve itself, and the challenge:nojs tier cannot carry a body back at all. Bodies over pow_gate_replay_max_body (default 1m, matching nginx's client_max_body_size default) or without a declared length (chunked) are not captured either — those requests are still challenged, they just lose their body as they did before.
FCrDNS resolves on a background thread and caches verdicts; the very first request for a new IP falls through to a challenge while DNS is in flight (fail-closed). The IP-range path has no such delay.

Only the verified-good-bot allowlist is unfinished. Until it lands, a verify:<name> decision fails closed — it falls through to a normal challenge rather than wrongly allowing — so it is safe to ship the gate without it.

Documentation

docs/architecture.md — phase integration, full request lifecycle, the two-token security model, threat model.
docs/configuration.md — every directive: context, arguments, defaults, inheritance, and worked recipes.
docs/build.md — toolchain, building against your nginx, ABI matching, Docker, install, reload, troubleshooting.
docs/protocol.md — the /.pow/ endpoints, challenge and token formats, and the client/server message contract.
docs/challenge-page.md — customizing the "Just a moment…" page: the hook contract, placeholders, a minimal template, and the built-in light/dark theme + i18n (26 languages, auto-detected).
docs/testing.md — the self-verifying Docker pipeline (engine tests → build → nginx -t → live e2e handshake).
docs/troubleshooting.md — field symptoms and diagnosis: broken assets on gated pages, misleading browser error strings, the one-command challenge-page check.
docs/performance.md — per-request cost, microbenchmarks, the HTTP load suite, and the ECDSA-proof bottleneck.
docs/releasing.md — versioning (module vs. nginx axis), tag/title/artifact conventions, cutting a release, automated nginx bumps.

License

Acknowledgements

Special thanks to Anubis by Techaro, which inspired this project. Anubis pioneered the proof-of-work challenge as a practical, self-hostable defense against AI scrapers and demonstrated that "weighing the soul of a connection" can be done without CAPTCHAs or third-party services. This module explores the same idea as a native nginx dynamic module — the tiered challenge model (JS-execution proof, scaled proof-of-work, and the no-JS meta-refresh fallback) follows the path Anubis charted. If you need a standalone reverse proxy rather than an nginx module, go use Anubis.

0.5.0 operational improvements

Scoped cookie names avoid collisions after scope changes; upgrades require one fresh solve. pow_gate_cookie_scope_suffix off retains a fixed name when needed.
Bounded browser retries, script/worker recovery and a clearance confirmation endpoint prevent transient failures and blocked cookies from causing loops.
All gate-generated responses explicitly prohibit caching.
$pow_gate_result, $pow_gate_reason, and $pow_gate_difficulty expose outcomes through native nginx logs.
Crawler feeds expire independently and reject malformed CIDRs. Optional ASN policies compose with nginx GeoIP2 variables.
Optional Redis coordination provides cross-instance one-use redemption with nonblocking sockets, explicit epoch fencing and bounded queues.
Alternative PoW experiments measure verifier costs; production retains SHA-256.

Name		Name	Last commit message	Last commit date
.cargo		.cargo
.github/workflows		.github/workflows
assets		assets
docker		docker
docs		docs
examples		examples
experiments/pow		experiments/pow
perf		perf
scripts		scripts
src		src
tests		tests
.dockerignore		.dockerignore
.gitattributes		.gitattributes
.gitignore		.gitignore
Cargo.lock		Cargo.lock
Cargo.toml		Cargo.toml
LICENSE		LICENSE
README.md		README.md
docker-compose.dev.yml		docker-compose.dev.yml
docker-compose.perf.yml		docker-compose.perf.yml
docker-compose.test.yml		docker-compose.test.yml

Navigation Menu

Repository navigation

About

Repository files navigation

ngx_http_pow_gate_module

Table of contents

Why a PoW gate

How it works (at a glance)

Request decision flow

The challenge / solve / verify handshake

Quickstart

Configuration

Project layout

Building

Performance

What's implemented

Known limitations

Documentation

License

Acknowledgements

0.5.0 operational improvements

About

Releases

Packages

Used by

Contributors

Languages

Latest commit

History

Folders and files

Repository files navigation

ngx_http_pow_gate_module

Table of contents

Why a PoW gate

How it works (at a glance)

Request decision flow

The challenge / solve / verify handshake

Quickstart

Configuration

Project layout

Building

Performance

What's implemented

Known limitations

Documentation

License

Acknowledgements

0.5.0 operational improvements

About

Resources

Security policy

Stars

Watchers

Forks

Releases

Packages

Used by

Contributors

Languages