gqlcrawl

Find public GraphQL endpoints that expose introspection, without turning discovery into exploitation.

gqlcrawl is an early-stage Go CLI from String. Use probe for exact public endpoint URLs or crawl to find literal GraphQL endpoint evidence on supplied public sites before running the same read-only probe. Use seeds to turn a public corpus into a candidate list without contacting any of it. Explicit full-schema export is planned as a separately reviewed addition.

Safety first

Only crawl or probe targets you are authorized to contact. You are responsible for applicable law, terms, and organizational policy.

The CLI intentionally has no flags for authentication headers, cookies, mutations, field guessing, or application-data queries. It also does not publish a hosted endpoint catalog. seeds widens where candidates can come from without changing what the probe sends: it contacts only the corpus provider and emits its results for you to review. Every endpoint probe uses:

query IntrospectionAvailability {
  __schema {
    queryType {
      name
    }
  }
}

The network client enforces these defaults:

• HTTPS only unless --allow-http is explicit; TLS certificates are verified.
• DNS answers must all be public, and the validated address is pinned for the connection.
• Probe redirects are revalidated and rate-gated per hop, limited to two, and rejected if they change the request method.
• One in-flight request and one request per second per host.
• Sixteen global probe workers, a 10-second timeout, a 64 KiB response cap, and no retries.
• An identifiable project user agent with optional --contact text.
• A bundled project opt-out list plus an optional local --denylist.

crawl adds stricter discovery boundaries:

• robots.txt is honored by default for pages, scripts, and discovered endpoint paths. An inaccessible or oversized robots file fails closed.
• Discovery fetches reject redirects so every fetched path is checked directly against its origin's robots rules.
• Page traversal stays on the seed origin, stops at depth two, and fetches at most 25 pages per host.
• Only scripts explicitly referenced by fetched HTML are read, capped at 50 total and 10 per host.
• Candidates must appear literally in fetched evidence. There is no common-path guessing.
• Relative candidates must stay on the seed origin. A cross-origin candidate must be an absolute URL in the evidence.
• Discovered URL query strings are removed rather than replayed. The run keeps at most 250 candidates total and five per host.

A block, timeout, authentication response, rate limit, redirect rejection, or transport error is always indeterminate. The CLI reports disabled only when a GraphQL-shaped response explicitly says introspection is rejected.

Build and verify

Go 1.24.5 or newer is required.

Install the latest tagged version directly from source:

go install github.com/usestring/gqlcrawl/cmd/gqlcrawl@latest

Tagged releases also publish Linux, macOS, and Windows binaries with SHA-256 checksums on the GitHub releases page.

Maintainers publish from an existing annotated vMAJOR.MINOR.PATCH tag on main. If its automatic release run fails, recover the same immutable tag from Actions → Release → Run workflow by entering the existing tag name. The workflow uses its current recovery code to resolve and validate the remote tag, then checks out that exact release commit before testing, building, or publishing. It rejects lightweight tags and commits outside main. Never replace a published tag to recover a release.

go build ./cmd/gqlcrawl
go test -race ./...
go vet ./...

These commands are offline. The test suite uses injected DNS and local fixtures and does not probe the public internet.

Use

The examples use reserved .example names, so they cannot contact a real endpoint.

Probe exact URL arguments:

./gqlcrawl probe https://your-approved-host.example/graphql

Read exact candidates from stdin or a file:

printf '%s\n' 'https://your-approved-host.example/graphql' |
  ./gqlcrawl probe --input -
./gqlcrawl probe --input approved-endpoints.txt

Crawl supplied domains or starting URLs, then probe only the discovered candidates:

./gqlcrawl crawl your-approved-host.example
./gqlcrawl crawl https://your-approved-host.example/docs
printf '%s\n' 'your-approved-host.example' |
  ./gqlcrawl crawl --input -

The crawler recognizes literal /graphql and /gql path segments plus nearby GraphiQL, Playground, Apollo, Relay, urql, graphql-ws, __schema, and GraphQL client evidence. These signatures create candidates; only the benign probe confirms GraphQL and introspection status.

Common options may appear before or after inputs:

--input FILE|-
--workers 16
--per-host-rps 1
--timeout 10s
--max-response-bytes 65536
--denylist FILE
--contact VALUE
--allow-http=false
--format jsonl

crawl also accepts:

--max-pages-per-host 25
--max-depth 2
--respect-robots=true

Disabling robots handling requires the explicit --respect-robots=false flag and does not replace target authorization.

Exit code 0 means the command completed and emitted its available ordered JSONL records; it does not mean introspection was enabled. A crawl that cannot read a permitted page, referenced script, or robots file emits any completed candidate results and exits 1 as incomplete. Exit 2 means the CLI configuration was invalid.

Corpus seeds

seeds reads a public corpus and writes candidate hostnames or URLs to stdout. It contacts only the corpus provider, never the seeds it emits, and it never probes. Piping its output into crawl or probe is a separate, deliberate step that you own.

./gqlcrawl seeds --list-sources
./gqlcrawl seeds --source SOURCE --limit 500
./gqlcrawl seeds --source SOURCE --limit 500 | ./gqlcrawl crawl --input -

Adapters that need a scope take it as positional arguments or through --input:

./gqlcrawl seeds --source SOURCE your-approved-host.example

Seeds are normalized, deduplicated, and truncated to --limit before they are written. Host seeds are lowercased with any wildcard label and trailing dot removed; URL seeds keep the probe pipeline's sanitization, so userinfo is dropped and query values become REDACTED. Emitting a seed is not authorization to contact it.

Ranked domain sources currently available:

Source	Ranks	Credentials	Notes
tranco	ordinal	none	Research ranking combining five providers. The run reports the list id so published work can cite the exact snapshot.
umbrella	ordinal	none	Cisco DNS resolution volume. Entries include subdomains rather than registrable domains.
majestic	ordinal	none	Link-graph authority by referring subnets, not traffic.

These rank different things and are not interchangeable. --option date=VALUE selects a published snapshot for tranco (YYYYMMDD) and umbrella (YYYY-MM-DD); --option subdomains=true switches tranco to its subdomain-inclusive list.

None of these lists are vendored into this repository. They are fetched when you run the command, which keeps the data current and leaves each provider's licensing with the provider. Only Majestic publishes a redistribution grant, so treat the others as fetch-only. App store sources:

Source	Ranks	Credentials	Notes
applecharts	ordinal	none	App Store top charts joined to each app's publisher site. --option feed=top-free|top-paid, --option country=us,gb,de.

applecharts emits the publisher's own site, which is not the app's backend API host. Recovering backend hosts would require analyzing the app binary; the supported path is to feed these domains to crawl, which already looks for GraphQL evidence on them. Apple documents roughly twenty lookup calls per minute, so lower --per-host-rps for wide sweeps. Google Play has no official charts API and the endpoint its scrapers use is disallowed by that site's robots.txt, so no Play source is provided. Certificate Transparency sources take one or more domains as scope and return the hostnames that appear in logged certificates:

Source	Ranks	Credentials	Notes
certspotter	none	CERTSPOTTER_API_KEY optional	Preferred. Pages through issuances by cursor. Unauthenticated use is evaluation-grade and rate limited near ten full-domain queries per hour.
crtsh	none	none	Fallback only. Rate limited near five requests per minute, frequently unavailable, and it can answer 200 with a silently truncated result, so treat a run as a lower bound.

./gqlcrawl seeds --source certspotter your-approved-host.example

Both sources match loosely on their side, so results are filtered locally to the requested domain and its subdomains. A lookalike such as notyourhost.example or your-approved-host.example.other.test is dropped rather than emitted. Certificate hostnames are historical records, so many will no longer resolve. URL-index sources search an archive for paths that already contain a pattern, so they emit URL seeds rather than hosts:

Source	Scope	Credentials	Notes
wayback	domains	none	Internet Archive CDX server. Signals throttling with 503 and no rate headers.
commoncrawl	domains	none	Common Crawl index. Asks for single-threaded access; a blocked address stays blocked for about a day.

Neither index supports a global path search: both require a domain scope, so these widen coverage within sites you already have rather than finding new ones. --option pattern=VALUE sets the path substring to match (default graphql) and is applied literally on both, despite the two servers disagreeing on filter syntax. --option matchtype=exact|prefix|host|domain narrows the scope from the default domain. wayback also takes --option from= and --option to= (yyyyMMddhhmmss) and --option status= to keep one capture status; commoncrawl takes --option crawl=CC-MAIN-YYYY-WW to pin a crawl instead of using the newest.

Both indexes harvest URLs from JavaScript, so unexpanded template literals such as https://example.com/${region}/graphql appear as ordinary rows. They are dropped: they describe a path shape, not an address. Every emitted URL is a historical capture and may no longer resolve. Popularity sources that do not report a position:

Source	Ranks	Credentials	Notes
crux	bucket	none	Chrome User Experience Report origins, read from the zakird/crux-top-lists mirror.
radar	ordinal, or member with --option bucket=	CLOUDFLARE_API_TOKEN	Cloudflare Radar. A free account is enough; the token needs Account → Radar → Read.

crux reports a magnitude bucket: the value is the bucket ceiling and order inside a bucket is arbitrary, so rank_kind is bucket and sorting on rank orders the groups, not the members. --option month=YYYYMM reads a monthly archive instead of the rolling current list, and --option country=CC reads a country list, which is published monthly only and therefore requires month. Origins arrive with a scheme and are normalized to hosts.

radar returns the top 100 with real ordinal ranks by default, narrowed by --option location= and --option date=. --option bucket=N switches to a published bucket dataset for one of 200, 500, 1000, 2000, 5000, 10000, 20000, 50000, 100000, 200000, 500000, or 1000000. Those datasets carry no rank column at all, so their seeds are member: the corpus asserts set membership and nothing about order. Radar data is CC BY-NC 4.0, which restricts commercial use of the results independently of this tool's license. Commercial technology-intelligence sources, both of which need a paid account:

Source	Ranks	Credentials	Notes
shodan	none	SHODAN_API_KEY	Hosts whose scanned banner matches a web technology query. Any filtered query costs a query credit.
builtwith	none	BUILTWITH_API_KEY	Domains reported as using a named technology. Lists API access is gated on a subscription tier.

Neither vendor publishes a usable identifier for GraphQL, so both take the name as configuration rather than baking one in. shodan defaults to http.component:GraphQL — a value Shodan does not document, since it publishes no enumeration of legal http.component values at all — and accepts --option component= or a full --option query=. builtwith defaults to Apollo-GraphQL, the only GraphQL-related technology whose BuiltWith page could be confirmed, and accepts --option tech= plus othertechs, country, and since.

Both endpoints authenticate with a query parameter, because neither documents header authentication for the endpoint used here. Transport errors are rewritten to remove the key before they reach stderr. One source reads an export rather than an API:

Source	Ranks	Credentials	Notes
httparchive	bucket when the export has a rank column	none	Sites HTTP Archive detected as using GraphQL, read from a CSV you export from BigQuery yourself.

./gqlcrawl seeds --source httparchive --input graphql-sites.csv --limit 500

The query, its cost, and what the GraphQL label actually means are in docs/httparchive.md. Reading an export instead of querying BigQuery keeps this tool's dependency list empty and leaves the query editable and billed to your own project. Read that document before trusting a result set: the label is inherited from Apollo and about nine commerce platforms far more often than it is matched directly.

Sources that require credentials read them from the environment and fail closed when they are unset. --list-sources reports each adapter's required variables and whether it consumes paid credits or query spend. Credentials are never written to output.

--source NAME
--list-sources
--limit 1000
--option KEY=VALUE
--input FILE|-
--format lines|jsonl
--max-download-bytes 67108864
--per-host-rps 1
--timeout 30s
--denylist FILE
--contact VALUE
--allow-http=false

Exit code 0 means the corpus was read and the available seeds were written. A source that fails or returns partial data still writes what it collected and exits 1. Exit 2 means the configuration was invalid.

JSONL

probe keeps one source record per input. crawl emits one record per unique discovered endpoint, with the sanitized seed in source.input and the page or script that supplied the evidence in source.evidence_url. Duplicate normalized endpoints are probed once.

{"schema_version":"1","endpoint":"https://api.example.invalid/graphql","source":{"kind":"crawl","input":"https://www.example.invalid/","evidence_url":"https://www.example.invalid/app.js"},"checked_at":"2026-08-07T07:00:00Z","http":{"status":200,"content_type":"application/json","bytes":52},"graphql":"confirmed","introspection":"enabled","query_type":"Query","reason":"introspection_enabled"}

seeds --format jsonl emits one record per seed with its originating adapter and, where the corpus supplies one, a rank and an evidence string.

{"schema_version":"1","value":"www.example.invalid","kind":"host","adapter":"example","rank":42,"rank_kind":"ordinal"}

Corpora do not agree on what a rank means, so rank_kind states which one applies. ordinal is a meaningful 1..N position. bucket means the value is a magnitude ceiling and membership within it is unordered. member means the corpus supplies set membership only, and rank is omitted. A seed with no rank at all omits both fields.

URL userinfo is removed and displayed query values are replaced with REDACTED. Discovered candidate query strings are dropped before probing. Headers, response bodies, and schema details never enter output.

Stable reasons currently include:

• introspection_enabled, introspection_rejected
• policy_rejected, dns_non_public, redirect_rejected, robots_disallowed
• timeout, response_too_large, http_error
• non_graphql_response, malformed_graphql_response

Denylist and opt-out

A local denylist contains one hostname per line. Use a leading dot or wildcard to match subdomains:

example.com
.example.org
*.example.net

See docs/OPT_OUT.md for project-level opt-out requests. Operators should also honor direct requests immediately with --denylist rather than waiting for a release.

Development

Keep tests offline and deterministic. Changes that broaden discovery sources, traffic volume, query scope, stored data, or schema output require their own review.

Contributions are welcome through issues and pull requests. Security reports follow SECURITY.md.

License

MIT