citefact

Do your sources actually say what you claim?

Quickstart · How It Works · Usage · Limitations

_{A fabricated quote caught with the source's own words. Full report.}

---

LLMs made it trivial to produce manuscripts with fabricated citations, altered quotes, and claims the cited sources never made. Existing tools check that citations exist; citefact reads the full text of every cited source and checks that it supports what you wrote — then hands you a report with the evidence, before a reviewer finds it first.

🚀 Quickstart

# From a Zotero collection (Zotero 7+ running; no export needed):
uvx citefact check manuscript.md --zotero-collection "PhD/Chapter 3"

# Or from a BibTeX file and a folder of PDFs:
uvx citefact check manuscript.md --bib refs.bib --pdfs ./papers/

That's the whole setup: uvx fetches citefact, the report lands in ./citefact-report/report.html. The citation and quote checks are free and need no API key. The claims check uses an LLM with your own key — run uvx citefact setup once and it configures everything interactively, or pass --skip-claims to stay entirely keyless and offline.

🧭 Three Checks, Cheapest First

Check	Question it answers	Method	Cost
1 · Citations	Does every in-text citation exist in the bibliography, with a source document?	Deterministic	Free
2 · Quotes	Is every direct quote, inline or block, really verbatim in its cited source?	Deterministic (normalized + fuzzy)	Free
3 · Claims	Does the source actually support the claim attached to it?	LLM, with verbatim evidence per verdict	~cents per source

	Citation-existence checkers	citefact
Citation exists in bibliography	✅	✅
Direct quotes verbatim in source	❌	✅
Source actually supports the claim	❌	✅
Works offline / keyless	varies	✅ (checks 1+2)
Metadata / DOI linting	✅	❌ (other tools do this well)

Every claim verdict (supported / partial / misrepresented / not_in_paper) carries a verbatim evidence quote, reasoning, and a confidence score. It is presented as an AI suggestion, never as ground truth: citefact reports, you decide.

✨ Why researchers keep it in the loop

• 📚 Zotero-native - point it at a collection and it reads metadata and attached PDFs through Zotero's local API; no exports, no Better BibTeX
• 🔒 Keyless mode - --skip-claims catches fabricated citations and altered quotes with zero network calls
• 🤖 Your LLM, your key - Anthropic, OpenAI, or fully local via Ollama, all through LiteLLM
• 📊 One-file report - self-contained HTML (no CDN, no requests), with every citation's status, word-level diffs for altered quotes, and one-click links to the manuscript and each source PDF
• ⚡ Cached everything - PDF conversions, claim extraction, and verdicts are content-addressed; editing one paragraph re-verifies only what changed, and an unchanged re-run costs $0 in seconds. Conversions go through pagebound, so a paper any other pagebound tool already converted is reused, never reconverted
• 🕵️ Private by construction - no telemetry; the only writes are the cache and the report, and the only network traffic is your LLM provider

⚙️ How It Works

flowchart LR
    A[manuscript.md] --> D[Citation parser]
    B[Zotero / refs.bib] --> E[Bibliography catalog]
    C[source PDFs] --> F[pagebound<br/>Docling, content-hash cache]
    D --> G[Citations check]
    E --> G
    F --> G
    G --> H[Quotes check]
    H --> I[Claims check<br/>LLM + caches]
    I --> J[report.html + report.json]

Loading

1. Parse - a deterministic parser extracts every in-text citation (narrative, parenthetical, multi-author lists, multi-citation groups) and resolves it against the bibliography by first-author surname + year.
2. Convert - source PDFs become text via pagebound, which runs Docling once per PDF and caches the result by content hash and converter version. Scanned PDFs are OCR'd, and the report says so.
3. Check - citations and quotes are verified deterministically; claims are extracted verbatim and judged per (claim, source) pair by the LLM, evidence included.
4. Report - a filterable single-file report.html plus a versioned report.json for CI.

🤖 For agents and LLM tools

An agent can drive citefact from --help alone: every command states what it writes, its exit codes and the shape of report.json, and citefact help COMMAND is the same text. The loop is three commands:

citefact doctor                                           # what is installed, exit 0 always
citefact check manuscript.md --bib refs.bib --pdfs papers/ --skip-claims --json --quiet > report.json
jq '.findings[] | select(.severity == "error") | {type, location, citation, quote}' report.json

Exit codes are the contract: 0 clean, 1 findings at or above --fail-on, 2 usage or execution error with the reason on stderr. Progress never touches stdout, so --json output is safe to pipe. Finding types are stable per level (orphan_citation, missing_source, quote_modified, claim_verdict, ...) and report.json carries schema_version; a claim_verdict with source_truncated: true was judged on a partial source and should be read as provisional; changes are listed in CHANGELOG.md. Converted source text is shared with pagebound, so an agent that wants to read a source itself can use pagebound path on the same cache.

📖 Usage

Command	What it does
citefact check MANUSCRIPT	Run the audit and write the report
citefact setup	Interactive first-run: pre-download Docling, configure and validate your LLM key
citefact doctor	Show what is installed and missing, per feature
citefact convert --pdfs DIR	Pre-convert PDFs to warm the cache

CI gating

citefact check manuscript.md --bib refs.bib --pdfs ./papers/ --skip-claims --fail-on error

Exit codes: 0 clean, 1 findings at or above --fail-on (default error), 2 execution error. Errors are the serious findings (fabricated citations, altered or unfound quotes, misrepresented/not_in_paper verdicts); --fail-on warning is stricter, --fail-on none just collects the report. Failed LLM calls surface as unverified findings and mark the run partial — never silently dropped.

Live progress

Long phases draw progress bars with ETA and running LLM cost on interactive terminals (plain lines in CI), and every run ends with a summary:

Verifying claims ━━━━━━━━━━━━━━╸━━  38/41  0:03:12  ETA 0:00:15  $0.31  smith2023

⚠️ Summary: 0 errors, 13 warnings (quotes 13) | cost $0.33 | 158.2s
Report: citefact-report/report.html

check flags reference

Flag	Default	Meaning
--bib FILE		BibTeX bibliography (this or --zotero-collection)
--zotero-collection NAME		Zotero collection (name or "Parent/Child" path); needs Zotero 7+ running
--pdfs DIR		Folder of source PDFs, fuzzy-matched to entries lacking one
--skip-claims	off	Citations + quotes only: free, keyless
--only LEVELS	all	Comma-separated subset: citations,quotes,claims
--model / --provider	anthropic/claude-sonnet-5	LLM for the claims check
--fail-on	error	error | warning | none: what makes the exit code 1
--out DIR	./citefact-report	Where report.html + report.json land
--json	off	Also stream report.json to stdout
--quiet	off	Silence progress and summary
--force	off	Ignore the conversion cache

Environment: CITEFACT_CACHE relocates the .citefact/ cache (written next to the manuscript by default); CITEFACT_MODEL pins a model; CITEFACT_MAX_SOURCE_CHARS sets how much of one source the claims check sends to the model (default 300000, about a long journal article). Anything cut is named in summary.sources_truncated and flagged on every verdict it affected, because a not_in_paper on a cut source may only mean "not in the part the model saw". citefact convert resolves its cache relative to the current directory, so run it from the manuscript's folder or set CITEFACT_CACHE.

Converted PDFs live in .citefact/pagebound/, the project's own pagebound store. If you also have a shared pagebound cache (~/.cache/pagebound, or PAGEBOUND_CACHE), citefact reads it first and copies hits into the project store; it never writes to the shared cache.

All finding types (what lands in the report)

Finding	Check	Severity	Meaning
orphan_citation	citations	error	In-text citation with no bibliography entry (the fabricated-citation signal)
uncited_reference	citations	warning¹	Bibliography entry never cited in the text
missing_source	citations	warning	Entry is cited but no PDF/text is available, so later checks skip it
quote_verified	quotes	info	Quote found verbatim in the source
quote_modified	quotes	error	Near match only; the report shows a word-level diff
quote_not_found	quotes	error	Quote does not appear in the cited source
quote_unattributed	quotes	warning	Long quote with no resolvable citation nearby
quote_page_mismatch	quotes	warning	Quote is verbatim in the source, but not on the page the citation names

Quote findings carry form: inline for text between quotation marks, block for a Markdown > block quote (APA sets quotations of 40+ words that way, without marks). A block's trailing (Author, year, p. n) is not part of what is compared with the source. Text inside code fences and tables is never scanned for quotes.

Page locators (p. 47, pp. 47–49) are read from the citation and checked: a matched quote reports pdf_page (where it is in the PDF, linked from the HTML report) and page_found, the printed page number, when the PDF's numbering can be established from its running headers or from the bibliography entry's page range. A quote_page_mismatch is only raised when both sides are known and disagree; a PDF whose numbering is unknown never produces one. | claim_verdict | claims | by verdict | supported info · partial warning · misrepresented/not_in_paper error · unverified warning |

¹ Demoted to info with --zotero-collection: a collection is a reading library, so uncited entries are normal there.

🤖 LLM Configuration

The easy path is one command:

uvx citefact setup

It asks for your provider, takes the API key with hidden input, validates it with a minimal call, and stores it in ~/.config/citefact/config.toml (owner-only permissions). Done — citefact check just works from then on.

Everything else is an override, in precedence order: --model / --provider flags > ANTHROPIC_API_KEY-style environment variables and CITEFACT_MODEL > the setup config file > the built-in default (anthropic/claude-sonnet-5). Any LiteLLM-supported provider works, including fully local:

uvx citefact check manuscript.md --bib refs.bib --model ollama/llama3.1

⚠️ Limitations

• Author-year (APA-like) citation styles only; numeric styles (Vancouver/IEEE) are on the roadmap
• Markdown manuscripts only in v0.1 (DOCX next; the manuscript itself cannot be a PDF)
• You supply the source PDFs; there is no automatic retrieval by DOI
• The first PDF conversion downloads the Docling environment (~1-2 GB, one-time) into the uv cache
• Claim verdicts are AI suggestions with the usual LLM failure modes; that is why every verdict carries its evidence

Roadmap: DOCX manuscripts → LaTeX (\cite{}) → numeric citation styles → stable JSON schema and JOSS submission. report.json already carries schema_version and is the contract for integrations; schema changes are listed in CHANGELOG.md.

For developers

git clone https://github.com/hearthresearch/citefact
cd citefact
uv sync
uv run pytest              # unit tests, LLM mocked
uv run pytest --run-evals  # opt-in LLM eval fixtures (costs money)

📄 License

MIT

🤝 Acknowledgments

• LiteLLM for provider-agnostic LLM access
• pagebound and Docling for PDF-to-Markdown conversion with page provenance
• ZotSeek - related tool: AI-powered semantic search for Zotero

---

citefact: Do your sources actually say what you claim? Built by José Fernandes