diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml index f48674c4..09c73f9b 100644 --- a/.github/workflows/release.yml +++ b/.github/workflows/release.yml @@ -466,18 +466,9 @@ jobs: - name: Create release and upload assets env: GH_TOKEN: ${{ github.token }} - # Notes come from .github/releases/.md, committed with the version - # bump; a release without one gets generated notes. # --clobber keeps a re-run green when some assets already uploaded. run: | if ! gh release view "$GITHUB_REF_NAME" > /dev/null 2>&1; then - notes=".github/releases/$GITHUB_REF_NAME.md" - if [ -f "$notes" ]; then - gh release create "$GITHUB_REF_NAME" --verify-tag \ - --title "$(head -n 1 "$notes" | sed 's/^# //')" \ - --notes-file <(tail -n +2 "$notes") - else - gh release create "$GITHUB_REF_NAME" --verify-tag --generate-notes - fi + gh release create "$GITHUB_REF_NAME" --verify-tag --generate-notes fi gh release upload "$GITHUB_REF_NAME" assets/* --clobber diff --git a/README.md b/README.md index 760d3f18..a73400c6 100644 --- a/README.md +++ b/README.md @@ -30,7 +30,6 @@ The [skill](skills/convert-documents-to-markdown/SKILL.md) teaches the agent to npx @firecrawl/anydoc report.docx # Markdown to stdout npx @firecrawl/anydoc slides.pptx -o slides.md # or to a file npx @firecrawl/anydoc - --format csv < data.csv # read stdin -npx @firecrawl/anydoc scan.pdf --ocr hosted # scanned pages via Firecrawl Parse ``` `npx` downloads the prebuilt binary for your platform on first run. For a permanent `anydoc` command, install globally with `npm install -g @firecrawl/anydoc`. Run `anydoc --help` for all options. @@ -47,9 +46,6 @@ import { toDocument, toMarkdown, toMarkdownBytes } from '@firecrawl/anydoc'; // From a file path: const markdown = await toMarkdown('report.docx'); -// To enable OCR: -const markdown = await toMarkdown('report.docx', { ocr: 'hosted' ); - // From bytes, with the format detected from the content: const fromBytes = await toMarkdownBytes(bytes); @@ -74,9 +70,6 @@ import anydoc # From a file path: markdown = anydoc.to_markdown("report.docx") -# To enable OCR: -markdown = anydoc.to_markdown("report.docx", ocr="hosted") - # From bytes, with the format detected from the content: markdown = anydoc.to_markdown_bytes(data) @@ -132,28 +125,15 @@ let markdown = anydoc::to_markdown_bytes(&bytes, anydoc::Format::Csv)?; let document = anydoc::to_document(&bytes, None)?; ``` -## OCR - -anydoc reads text-based PDFs locally but does no OCR, so a PDF with scanned or image-only pages fails with `NeedsOcr`. Opt in and those documents go to [Firecrawl Parse](https://firecrawl.dev/parse), which OCRs them and returns the same Markdown. No signup needed; set `FIRECRAWL_API_KEY` for higher limits. - -| | Opt in | Key, else `FIRECRAWL_API_KEY` | -| ------ | ---------------------------------------------- | ----------------------------- | -| CLI | `anydoc scan.pdf --ocr hosted` | `--api-key ` | -| Node | `toMarkdown('scan.pdf', { ocr: 'hosted' })` | `apiKey` | -| Python | `anydoc.to_markdown("scan.pdf", ocr="hosted")` | `api_key` | - -Only documents that need OCR leave the machine, and the whole document goes, since Parse has no page selection. If Parse cannot convert it, Node rejects with `code: 'hosted'` and Python raises `HostedError`. `--api-url`, `apiUrl` and `api_url`, else `FIRECRAWL_API_URL`, point at another Parse deployment. The Rust crate has no `ocr` option and never makes network calls. - ## Features - **One output for every format.** Each format parses into a shared document model and renders through a single Markdown serializer, so escaping, tables, heading anchors, and footnotes behave identically whether the input was a `.doc` from 2003 or a `.pptx` from yesterday. - **Full document structure.** Headings with anchors, bold/italic/strikethrough, inline code and code blocks, links and internal cross-references, bulleted/numbered/nested/task lists with the source's own numbering, tables with merged cells and header rows, block quotes, footnotes and endnotes, and speaker notes. -- **Equations as LaTeX.** Word and PowerPoint (OMML), OpenDocument and EPUB (MathML), and RTF equations convert to GitHub-flavored math: `$...$` inline and `$$` blocks. -- **Embedded assets.** Images and embedded objects render as their alt text in the Markdown, and the raw bytes stay available on the document model, tagged with their media type. Images with an external URL become ordinary Markdown images. +- **Embedded assets.** Images and embedded objects become Markdown image refs of the form `![alt](asset:N)`, where `N` indexes `Document.assets` (raw bytes + media type). Rewrite those hrefs after writing the files out, or consume the document model directly. Images with an external URL become ordinary Markdown images. - **Content-based format detection.** The format is read from the bytes themselves (PDF header, RTF open group, OLE stream names, ZIP package mimetype), so mislabeled files still convert correctly. - **Fast.** Pure Rust, no ML models, no external services. Median conversion time is under 5ms per document. - **Bindings that stay out of the way.** Node.js conversion runs on the libuv thread pool and never blocks the event loop; Python releases the GIL so other threads keep running. TypeScript types and Python stubs ship with the packages. -- **PDF support built in.** Text-based PDFs convert locally through [pdf-inspector](https://github.com/firecrawl/pdf-inspector), no OCR service required. Scanned pages can opt into [hosted OCR](#scanned-pdfs-ocr). +- **PDF support built in.** Text-based PDFs convert locally through [pdf-inspector](https://github.com/firecrawl/pdf-inspector), no OCR service required. - **Agent ready.** Ships as an [Agent Skill](#agent-skill): one `npx skills add firecrawl/anydoc` and any agent can read office documents. ## Supported formats @@ -221,17 +201,13 @@ The same three functions exist in Node (`formatFromBytes`, ...) and Python (`any ## Errors -A conversion returns `Err` only when no complete Markdown could come out of the file. `ConvertError` names what went wrong: +A conversion returns `Err` only when no meaningful Markdown could come out of the file. `ConvertError` names what went wrong: ```rust match anydoc::to_markdown(path) { Ok(markdown) => Some(markdown), // No document comes out of these, so record the file and take the next one. - Err( - error @ (ConvertError::Encrypted - | ConvertError::Unsupported(_) - | ConvertError::NeedsOcr { .. }), - ) => { + Err(error @ (ConvertError::Encrypted | ConvertError::Unsupported(_))) => { unconverted.push((path, error)); None } @@ -241,8 +217,7 @@ match anydoc::to_markdown(path) { | Variant | Meaning | | --------------- | ------------------------------------------------------------------- | -| `Unsupported` | Unknown format, or one that cannot be converted | -| `NeedsOcr` | Scanned or image-only pages of a PDF, listed in `pages` | +| `Unsupported` | Unknown format, or one that cannot be converted (an image-only PDF) | | `Malformed` | Structurally unusable: no meaningful content could be extracted | | `Encrypted` | Encrypted or password-protected | | `ResourceLimit` | Crossed a fixed safety limit (decompression, nesting, node count) | @@ -251,8 +226,6 @@ match anydoc::to_markdown(path) { Node and wasm publish the variant name on `error.code`; Python raises one `anydoc.ConvertError` subclass per variant, or `OSError` when the file cannot be read. -anydoc converts locally and does not do OCR, so a PDF with scanned pages fails with `NeedsOcr`. Opt in with `ocr: 'hosted'` in Node, `ocr="hosted"` in Python or `--ocr hosted` on the CLI to send that document to [Firecrawl Parse](https://firecrawl.dev/parse). No signup needed. Set `FIRECRAWL_API_KEY` for higher limits. - ## How it works ``` diff --git a/src/render/markdown/inline.rs b/src/render/markdown/inline.rs index c3aadb79..8eb3073a 100644 --- a/src/render/markdown/inline.rs +++ b/src/render/markdown/inline.rs @@ -1,10 +1,9 @@ //! Inline run normalization and rendering. -use crate::model::{ImageSource, Inline, LinkTarget, Style, checkbox_text, inlines_are_empty}; +use crate::model::{ImageSource, Inline, LinkTarget, Style, inlines_are_empty}; use crate::render::markdown::Ctx; use crate::render::markdown::escape::{ - Delims, EscapeOpts, InlineContext, backtick_fence, escape_cell_code_span, escape_text, - escape_url_as_text, format_url, + EscapeOpts, InlineContext, backtick_fence, escape_text, escape_url_as_text, format_url, }; use std::borrow::Cow; use std::fmt::Write as _; @@ -16,8 +15,6 @@ pub(crate) enum Norm<'a> { Anchor(&'a str), NoteRef(&'a str), LineBreak, - Math(&'a str), - Checkbox(bool), } /// Single-pass normalization: drops empty runs, strips styling from @@ -76,9 +73,6 @@ pub(crate) fn normalize<'a>(inlines: &'a [Inline], rc: &Ctx) -> Vec> { Inline::Anchor(id) => out.push(Norm::Anchor(id)), Inline::NoteRef(id) => out.push(Norm::NoteRef(id)), Inline::LineBreak => out.push(Norm::LineBreak), - Inline::Math(tex) if tex.trim().is_empty() => continue, - Inline::Math(tex) => out.push(Norm::Math(tex.trim())), - Inline::Checkbox(checked) => out.push(Norm::Checkbox(*checked)), } } out @@ -90,32 +84,18 @@ pub(crate) fn render_inlines(inlines: &[Inline], ctx: InlineContext, rc: &Ctx) - fn render_inlines_mode(inlines: &[Inline], ctx: InlineContext, in_label: bool, rc: &Ctx) -> String { let runs = normalize(inlines, rc); - let suffix = delims_ahead(&runs, rc); let mut out = String::new(); for (idx, run) in runs.iter().enumerate() { match run { Norm::Text { text, style } => { - let next = runs.get(idx + 1); let next_active = matches!( - next, - Some(Norm::Link { .. } | Norm::Image { .. } | Norm::NoteRef(_) | Norm::Math(_)) + runs.get(idx + 1), + Some(Norm::Link { .. } | Norm::Image { .. } | Norm::NoteRef(_)) ) || matches!( - next, + runs.get(idx + 1), Some(Norm::Text { style, .. }) if *style != Style::PLAIN ); - // A hard break renders as `\`, an anchor as ``: not - // markup, but a nonspace character a run-final delimiter can - // be left-flanking against. - let next_nonspace = matches!(next, Some(Norm::Anchor(_) | Norm::Checkbox(_))) - || (matches!(next, Some(Norm::LineBreak)) && ctx != InlineContext::Heading); - let opts = EscapeOpts { - trailing_active: next_active, - trailing_nonspace: next_nonspace, - trailing_delims: suffix[idx + 1], - in_label, - ..Default::default() - }; - render_text_run(text, *style, ctx, opts, &mut out) + render_text_run(text, *style, ctx, next_active, in_label, &mut out) } Norm::NoteRef(id) => { if let Some(num) = rc.nums.get(*id) { @@ -134,14 +114,6 @@ fn render_inlines_mode(inlines: &[Inline], ctx: InlineContext, in_label: bool, r InlineContext::Heading => out.push(' '), InlineContext::TableCell => out.push('\n'), }, - Norm::Math(tex) => push_math_span(tex, ctx, &mut out), - Norm::Checkbox(checked) => { - out.push_str(checkbox_text(*checked)); - // The token stands apart from a caption that follows it. - if matches!(runs.get(idx + 1), Some(run) if !starts_with_space(run)) { - out.push(' '); - } - } } } out @@ -192,10 +164,16 @@ fn render_image( escape_text(alt.trim(), ctx, EscapeOpts { in_label: true, ..Default::default() }); let _ = write!(out, "![{}]({})", alt, format_url(url)); } - // Embedded assets render as their alt text: Markdown cannot embed - // bytes, and the bytes stay available in `Document::assets`. A - // source-less image has only its alt text to offer. - ImageSource::Asset(_) | ImageSource::Unavailable => { + // Embedded assets cannot carry bytes in Markdown, but they keep a + // stable positional marker (`asset:N`) that indexes `Document::assets` + // so callers can rewrite the href after writing files out. + ImageSource::Asset(id) => { + let alt = + escape_text(alt.trim(), ctx, EscapeOpts { in_label: true, ..Default::default() }); + let _ = write!(out, "![{}](asset:{})", alt, id.0); + } + // A source-less image has only its alt text to offer. + ImageSource::Unavailable => { if !alt.trim().is_empty() { out.push_str(&escape_text( alt.trim(), @@ -207,122 +185,22 @@ fn render_image( } } -/// Pairable delimiters the remaining runs will emit into the current rendered -/// line: closer-capable literals in plain text, plus the markup that styled runs, -/// code spans, links and images produce. A delimiter in an earlier run can -/// pair with any of them across the run seam, hard breaks included. -/// The delimiters each suffix of `runs` emits, indexed by where the suffix -/// starts, so one reverse pass answers every run's lookahead. -fn delims_ahead(runs: &[Norm], rc: &Ctx) -> Vec { - let mut suffix = vec![Delims::default(); runs.len() + 1]; - for idx in (0..runs.len()).rev() { - let mut delims = suffix[idx + 1]; - delims.union(delims_of(&runs[idx], rc)); - suffix[idx] = delims; - } - suffix -} - -/// What one run contributes to a later run's pairing partners. -fn delims_of(run: &Norm, rc: &Ctx) -> Delims { - let mut delims = Delims::default(); - match run { - Norm::Text { style, .. } if style.code => delims.insert('`'), - Norm::Text { text, style } if *style == Style::PLAIN => delims.insert_closers(text), - Norm::Text { text, style } => { - // Emphasis content is escaped, which neutralizes everything - // but backticks: code spans ignore backslash escapes, so an - // emitted `\`` still closes a span an earlier raw backtick - // opens. `]` is the one character escaping leaves raw. - if style.bold || style.italic { - delims.insert('*'); - } - if style.strike { - delims.insert('~'); - } - if text.contains('`') { - delims.insert('`'); - } - if text.contains(']') { - delims.insert(']'); - } - delims.insert_closers(&text.replace(|c: char| c != '$' && !c.is_whitespace(), "x")); - } - Norm::Link { content, target } => match target { - // An unresolved target degrades to its rendered content - // (see render_link), which emits like any sibling runs. - LinkTarget::Anchor(id) if rc.anchors.fragment(id).is_none() => { - for run in &normalize(content, rc) { - delims.union(delims_of(run, rc)); - } - } - // Emphasis cannot cross a link boundary, but a code span - // can: a backtick in the label or destination pairs with - // one outside. - _ => { - if emits_backtick(content) || target_has_backtick(target) { - delims.insert('`'); - } - } - }, - Norm::Image { alt, source } => match source { - ImageSource::External(_) if alt.contains('`') => delims.insert('`'), - ImageSource::External(_) => {} - // Sourceless images degrade to their alt as plain text. - ImageSource::Asset(_) | ImageSource::Unavailable => delims.insert_closers(alt), - }, - Norm::NoteRef(_) - | Norm::Anchor(_) - | Norm::LineBreak - | Norm::Math(_) - | Norm::Checkbox(_) => {} - } - delims -} - -fn starts_with_space(run: &Norm) -> bool { - match run { - Norm::Text { text, .. } => text.starts_with(char::is_whitespace), - Norm::LineBreak => true, - _ => false, - } -} - -fn target_has_backtick(target: &LinkTarget) -> bool { - match target { - LinkTarget::External(url) | LinkTarget::Relative(url) => url.contains('`'), - LinkTarget::Anchor(_) => false, - } -} - -/// True when rendering `inlines` inside a link label emits a backtick an -/// earlier raw one can pair with: any backtick in text counts even where the -/// label escapes it (code spans ignore backslash escapes), and a code run -/// emits fences unless it is whitespace-only and loses its styling. -fn emits_backtick(inlines: &[Inline]) -> bool { - inlines.iter().any(|inline| match inline { - Inline::Text { text, style } => { - text.contains('`') || (style.code && !text.trim().is_empty()) - } - Inline::Link { content, target } => emits_backtick(content) || target_has_backtick(target), - Inline::Image { alt, .. } => alt.contains('`'), - _ => false, - }) -} - /// Emit a styled run, moving edge whitespace outside the delimiters. -/// `opts` carries the trailing context; `at_line_start` and `styled` are -/// filled in here. fn render_text_run( text: &str, style: Style, ctx: InlineContext, - opts: EscapeOpts, + trailing_active: bool, + in_label: bool, out: &mut String, ) { if style == Style::PLAIN { let at_line_start = out.is_empty() || out.ends_with('\n'); - out.push_str(&escape_text(text, ctx, EscapeOpts { at_line_start, ..opts })); + out.push_str(&escape_text( + text, + ctx, + EscapeOpts { at_line_start, trailing_active, in_label, ..Default::default() }, + )); return; } let core_start = text.len() - text.trim_start().len(); @@ -333,7 +211,7 @@ fn render_text_run( } if !core.is_empty() { if style.code { - push_code_span(core, ctx, out); + push_code_span(core, out); } else { let mut open = String::new(); if style.strike { @@ -350,7 +228,7 @@ fn render_text_run( out.push_str(&escape_text( core, ctx, - EscapeOpts { styled: true, in_label: opts.in_label, ..Default::default() }, + EscapeOpts { styled: true, in_label, ..Default::default() }, )); out.push_str(&close); } @@ -360,50 +238,9 @@ fn render_text_run( } } -/// GFM inline math: `$` hugging both ends of the source. A line break -/// inside it would end the paragraph's math span, and a bare `$` (never -/// valid inside math) would close it early. -pub(crate) fn push_math_span(tex: &str, ctx: InlineContext, out: &mut String) { - let mut source = String::with_capacity(tex.len()); - let mut backslashes = 0; - for c in tex.trim().chars() { - match c { - '\n' => source.push(' '), - '$' if backslashes % 2 == 0 => source.push_str("\\$"), - c => source.push(c), - } - backslashes = if c == '\\' { backslashes + 1 } else { 0 }; - } - // A row is split into cells before the math span is parsed, so a bare - // pipe is syntax here; GFM strips the escaping backslash before the - // math is read, so an already escaped pipe stays as it is. - let source = match ctx { - InlineContext::TableCell => { - let mut escaped = String::with_capacity(source.len()); - let mut backslashes = 0; - for c in source.chars() { - if c == '|' && backslashes % 2 == 0 { - escaped.push('\\'); - } - escaped.push(c); - backslashes = if c == '\\' { backslashes + 1 } else { 0 }; - } - escaped - } - _ => source, - }; - let _ = write!(out, "${source}$"); -} - -pub(crate) fn push_code_span(text: &str, ctx: InlineContext, out: &mut String) { +pub(crate) fn push_code_span(text: &str, out: &mut String) { let text = text.replace('\n', " "); let fence = backtick_fence(&text, 1); let pad = if text.starts_with('`') || text.ends_with('`') { " " } else { "" }; - // A row is split into cells before any code span is parsed, so a pipe is - // syntax here even though everything else between the fences is literal. - let text = match ctx { - InlineContext::TableCell => escape_cell_code_span(&text), - _ => text, - }; let _ = write!(out, "{fence}{pad}{text}{pad}{fence}"); } diff --git a/src/render/markdown/tests.rs b/src/render/markdown/tests.rs index 0a7b19a4..e2195be7 100644 --- a/src/render/markdown/tests.rs +++ b/src/render/markdown/tests.rs @@ -29,35 +29,6 @@ fn heading_and_paragraph() { assert_eq!(md, "## Title\n\nHello world.\n"); } -#[test] -fn math_renders_in_dollar_delimiters_and_text_dollars_are_escaped() { - let md = doc(vec![ - Block::Paragraph(vec![ - Inline::plain("Costs $5 or $6, and "), - Inline::Math("x_1 < y".into()), - Inline::plain(" holds."), - ]), - Block::Math("\\sum_{i=1}^{n} i $".into()), - Block::Paragraph(vec![Inline::plain("Plans at $20 or $17.50.")]), - Block::Paragraph(vec![Inline::plain("Pair $x with y$ here.")]), - ]); - assert_eq!( - md, - "Costs \\$5 or \\$6, and $x_1 < y$ holds.\n\n$$\n\\sum_{i=1}^{n} i \\$\n$$\n\n\ - Plans at $20 or $17.50.\n\nPair \\$x with y$ here.\n" - ); -} - -#[test] -fn math_in_a_table_cell_escapes_pipes() { - let cell = |inlines| Cell { blocks: vec![Block::Paragraph(inlines)], col_span: 1, row_span: 1 }; - let md = doc(vec![table_from( - vec![vec![cell(vec![Inline::plain("abs")]), cell(vec![Inline::Math("|x|".into())])]], - 0, - )]); - assert!(md.contains("| $\\|x\\|$ |"), "{md}"); -} - #[test] fn escapes_paired_syntax_chars() { let md = doc(vec![Block::Paragraph(vec![Inline::plain("a *bold* _it_ ~st~ `code`")])]); @@ -74,35 +45,6 @@ fn lone_syntax_chars_left_alone() { assert_eq!(md, "x < 5, ~10%, file_name, a[1\n"); } -#[test] -fn partners_that_cannot_close_leave_delimiters_raw() { - // The space-padded `*` is not right-flanking, so the opener is inert. - let md = doc(vec![Block::Paragraph(vec![Inline::plain("a *b 2 * 3")])]); - assert_eq!(md, "a *b 2 * 3\n"); - // Same across a run seam. - let md = doc(vec![Block::Paragraph(vec![ - Inline::plain("a *"), - Inline::LineBreak, - Inline::plain("2 * 3"), - ])]); - assert_eq!(md, "a *\\\n2 * 3\n"); - // Intraword underscores can neither open nor close. - let md = doc(vec![Block::Paragraph(vec![ - Inline::plain("a _"), - Inline::LineBreak, - Inline::plain("snake_case"), - ])]); - assert_eq!(md, "a _\\\nsnake_case\n"); - // A `*` after punctuation and before a word character is not - // right-flanking either. - let md = doc(vec![Block::Paragraph(vec![Inline::plain("a *b .*c")])]); - assert_eq!(md, "a *b .*c\n"); - // Flanking is judged at the delimiter run's edges: `__` between - // letters is intraword even though each `_` neighbours the other. - let md = doc(vec![Block::Paragraph(vec![Inline::plain("a _x foo__bar")])]); - assert_eq!(md, "a _x foo__bar\n"); -} - #[test] fn intraword_underscores_unescaped() { let md = doc(vec![Block::Paragraph(vec![Inline::plain("snake_case_name vs _lead_")])]); @@ -151,78 +93,6 @@ fn trailing_delimiter_before_styled_run_escaped() { assert_eq!(md, "star\\***x**\n"); } -#[test] -fn delimiters_do_not_pair_across_run_seams() { - // Two lines each ending in a backtick must not form a code span (#45). - let md = doc(vec![Block::Paragraph(vec![ - Inline::plain("a `"), - Inline::LineBreak, - Inline::plain("b `"), - ])]); - assert_eq!(md, "a \\`\\\nb `\n"); - // Emphasis pairs across a hard break just as code spans do. - let md = doc(vec![Block::Paragraph(vec![ - Inline::plain("a *"), - Inline::LineBreak, - Inline::plain("b*"), - ])]); - assert_eq!(md, "a \\*\\\nb*\n"); - // A raw backtick pairs with a later code span's fence. - let md = doc(vec![Block::Paragraph(vec![ - Inline::plain("a `"), - Inline::LineBreak, - styled("x", Style { code: true, ..Style::PLAIN }), - ])]); - assert_eq!(md, "a \\`\\\n`x`\n"); -} - -#[test] -fn unresolved_link_fallback_counts_toward_seam_pairing() { - // The fallback text supplies the `]` that pairs with the earlier `[`. - let md = doc(vec![Block::Paragraph(vec![ - Inline::plain("[click"), - Inline::LineBreak, - Inline::Link { - content: vec![Inline::plain("here]")], - target: LinkTarget::Anchor("nowhere".into()), - }, - Inline::plain("(https://e.test)"), - ])]); - assert_eq!(md, "\\[click\\\nhere](https://e.test)\n"); - // Same seam with an emphasis delimiter in the fallback. - let md = doc(vec![Block::Paragraph(vec![ - Inline::plain("a *"), - Inline::LineBreak, - Inline::Link { - content: vec![Inline::plain("b*")], - target: LinkTarget::Anchor("nowhere".into()), - }, - ])]); - assert_eq!(md, "a \\*\\\nb*\n"); -} - -#[test] -fn escaped_backtick_in_later_run_still_pairs() { - // A styled run's backtick is emitted as `\\``, yet still closes a span - // an earlier raw backtick opens: code spans ignore backslash escapes. - let md = doc(vec![Block::Paragraph(vec![ - Inline::plain("a `"), - Inline::LineBreak, - styled("x`y", BOLD), - ])]); - assert_eq!(md, "a \\`\\\n**x\\`y**\n"); - // A whitespace-only code run loses its styling and emits no fence. - let md = doc(vec![Block::Paragraph(vec![ - Inline::plain("a `"), - Inline::LineBreak, - Inline::Link { - content: vec![styled(" ", Style { code: true, ..Style::PLAIN }), Inline::plain("x")], - target: LinkTarget::External("https://e.test".into()), - }, - ])]); - assert_eq!(md, "a `\\\n[ x](https://e.test)\n"); -} - #[test] fn bold_trailing_space_moved_out() { let md = doc(vec![Block::Paragraph(vec![styled("bold ", BOLD), Inline::plain("plain")])]); @@ -295,6 +165,26 @@ fn sourceless_image_renders_alt_text() { assert_eq!(md, "chart\n"); } +#[test] +fn embedded_asset_image_emits_asset_href() { + use crate::model::AssetId; + let md = doc(vec![Block::Paragraph(vec![Inline::Image { + alt: "photo".into(), + source: ImageSource::Asset(AssetId(0)), + }])]); + assert_eq!(md, "![photo](asset:0)\n"); +} + +#[test] +fn embedded_asset_empty_alt_keeps_positional_marker() { + use crate::model::AssetId; + let md = doc(vec![Block::Paragraph(vec![Inline::Image { + alt: "".into(), + source: ImageSource::Asset(AssetId(3)), + }])]); + assert_eq!(md, "![](asset:3)\n"); +} + #[test] fn composite_marker_labels_are_escaped() { // M15: source-derived labels must not alter Markdown structure. @@ -303,6 +193,7 @@ fn composite_marker_labels_are_escaped() { start: 1, items: vec![crate::model::ListItem { blocks: vec![Block::Paragraph(vec![Inline::plain("x")])], + checked: None, marker_label: Some("#1\n*a*".into()), }], }; @@ -424,18 +315,6 @@ fn url_pipes_cannot_split_table_cells() { assert_eq!(md, "| |\n| --- |\n| [https://e.test/a\\|b](https://e.test/a%7Cb) |\n"); } -#[test] -fn code_span_pipes_cannot_split_table_cells() { - let code = |t: &str| { - Cell::from_inlines(vec![Inline::Text { - text: t.into(), - style: Style { code: true, ..Style::PLAIN }, - }]) - }; - let md = doc(vec![table_from(vec![vec![code("a | b"), code(r"a \| b")]], 0)]); - assert_eq!(md, concat!("| | |\n", "| --- | --- |\n", r"| `a \| b` | `a \\\| b` |", "\n")); -} - #[test] fn url_angle_brackets_are_encoded_without_bracketing() { let md = doc(vec![Block::Paragraph(vec![Inline::Link { @@ -471,10 +350,12 @@ fn nested_list() { start: 3, items: vec![ListItem { blocks: vec![Block::Paragraph(vec![Inline::plain("inner")])], + checked: None, marker_label: None, }], }), ], + checked: None, marker_label: None, }], })]); @@ -485,6 +366,7 @@ fn nested_list() { fn roman_and_alpha_markers_render_literally() { let item = |text: &str| ListItem { blocks: vec![Block::Paragraph(vec![Inline::plain(text)])], + checked: None, marker_label: None, }; let md = doc(vec![ @@ -631,6 +513,7 @@ fn an_empty_item_keeps_its_marker_and_the_numbering() { } else { vec![Block::Paragraph(vec![Inline::plain(text)])] }, + checked: None, marker_label: None, }; let md = doc(vec![Block::List(List { @@ -654,14 +537,13 @@ fn task_list() { start: 1, items: vec![ ListItem { - blocks: vec![Block::Paragraph(vec![Inline::Checkbox(true), Inline::plain("done")])], + blocks: vec![Block::Paragraph(vec![Inline::plain("done")])], + checked: Some(true), marker_label: None, }, ListItem { - blocks: vec![Block::Paragraph(vec![ - Inline::Checkbox(false), - Inline::plain(" todo"), - ])], + blocks: vec![Block::Paragraph(vec![Inline::plain("todo")])], + checked: Some(false), marker_label: None, }, ], @@ -669,18 +551,6 @@ fn task_list() { assert_eq!(md, "- [x] done\n- [ ] todo\n"); } -#[test] -fn checkbox_in_a_table_cell() { - let md = doc(vec![table_from( - vec![vec![ - Cell::from_inlines(vec![Inline::Checkbox(true)]), - Cell::from_inlines(vec![Inline::Checkbox(false), Inline::plain("Wall")]), - ]], - 0, - )]); - assert_eq!(md, "| | |\n| --- | --- |\n| [x] | [ ] Wall |\n"); -} - // --------------------------------------------------------------------------- // New contexts: link labels, image alt, anchors diff --git a/tests/snapshots.rs b/tests/snapshots.rs index 25f997c2..ebf2b231 100644 --- a/tests/snapshots.rs +++ b/tests/snapshots.rs @@ -93,7 +93,6 @@ fn fixtures_detect_from_content() { ("pptx", Some(Format::Pptx)), ("rtf", Some(Format::Rtf)), ("xls", Some(Format::Excel)), - ("xlsb", Some(Format::Excel)), ("xlsx", Some(Format::Excel)), ("csv", None), ]; @@ -114,19 +113,6 @@ fn fixtures_detect_from_content() { } } -/// A PDF with scanned pages is reported, not silently shortened: the error -/// names the pages that need OCR. -#[test] -fn scanned_pages_are_reported_not_dropped() { - let mixed = std::fs::read(fixture_root().join("pdf/handmade-mixed.pdf")).unwrap(); - match anydoc::to_markdown_bytes(&mixed, anydoc::Format::Pdf) { - Err(anydoc::ConvertError::NeedsOcr { pages, page_count }) => { - assert_eq!((pages, page_count), (vec![2], 2)); - } - other => panic!("expected NeedsOcr, got {other:?}"), - } -} - /// Embedded object payloads land in `Document::assets` with their identity /// and media type (the Markdown output shows only the alt text). #[test] @@ -180,8 +166,8 @@ fn doc_inline_picture_is_retained() { ); } -/// The RTF `\pict` payload is retained as an asset (the Markdown output -/// shows only the alt text, which is empty for pictures without one). +/// The RTF `\pict` payload is retained as an asset (Markdown references it +/// as `asset:N` when the picture appears as an image inline). #[test] fn rtf_inline_picture_is_retained() { let bytes = std::fs::read(fixture_root().join("rtf").join("text.rtf")).unwrap(); diff --git a/tests/snapshots/snapshots__doc__text.doc.snap b/tests/snapshots/snapshots__doc__text.doc.snap index b6c0ef11..73c4911b 100644 --- a/tests/snapshots/snapshots__doc__text.doc.snap +++ b/tests/snapshots/snapshots__doc__text.doc.snap @@ -66,7 +66,7 @@ Jump to the bookmarked paragraph. ## Objects -Inline image: done. +Inline image: ![](asset:0) done. Text box: after the box. diff --git a/tests/snapshots/snapshots__docx__handmade-manyrefs.docx.snap b/tests/snapshots/snapshots__docx__handmade-manyrefs.docx.snap index d6669ef6..bf51ac05 100644 --- a/tests/snapshots/snapshots__docx__handmade-manyrefs.docx.snap +++ b/tests/snapshots/snapshots__docx__handmade-manyrefs.docx.snap @@ -3,3 +3,143 @@ source: tests/snapshots.rs expression: output --- Seventy references to one image follow. + +![](asset:0) + +![](asset:0) + +![](asset:0) + +![](asset:0) + +![](asset:0) + +![](asset:0) + +![](asset:0) + +![](asset:0) + +![](asset:0) + +![](asset:0) + +![](asset:0) + +![](asset:0) + +![](asset:0) + +![](asset:0) + +![](asset:0) + +![](asset:0) + +![](asset:0) + +![](asset:0) + +![](asset:0) + +![](asset:0) + +![](asset:0) + +![](asset:0) + +![](asset:0) + +![](asset:0) + +![](asset:0) + +![](asset:0) + +![](asset:0) + +![](asset:0) + +![](asset:0) + +![](asset:0) + +![](asset:0) + +![](asset:0) + +![](asset:0) + +![](asset:0) + +![](asset:0) + +![](asset:0) + +![](asset:0) + +![](asset:0) + +![](asset:0) + +![](asset:0) + +![](asset:0) + +![](asset:0) + +![](asset:0) + +![](asset:0) + +![](asset:0) + +![](asset:0) + +![](asset:0) + +![](asset:0) + +![](asset:0) + +![](asset:0) + +![](asset:0) + +![](asset:0) + +![](asset:0) + +![](asset:0) + +![](asset:0) + +![](asset:0) + +![](asset:0) + +![](asset:0) + +![](asset:0) + +![](asset:0) + +![](asset:0) + +![](asset:0) + +![](asset:0) + +![](asset:0) + +![](asset:0) + +![](asset:0) + +![](asset:0) + +![](asset:0) + +![](asset:0) + +![](asset:0) diff --git a/tests/snapshots/snapshots__docx__handmade-ole.docx.snap b/tests/snapshots/snapshots__docx__handmade-ole.docx.snap index c9626e86..b9e155d2 100644 --- a/tests/snapshots/snapshots__docx__handmade-ole.docx.snap +++ b/tests/snapshots/snapshots__docx__handmade-ole.docx.snap @@ -4,4 +4,4 @@ expression: output --- Before the object -Embedded object: Excel.Sheet.12 +![Embedded object: Excel.Sheet.12](asset:0) diff --git a/tests/snapshots/snapshots__docx__handmade-rich.docx.snap b/tests/snapshots/snapshots__docx__handmade-rich.docx.snap index 80103366..9114d992 100644 --- a/tests/snapshots/snapshots__docx__handmade-rich.docx.snap +++ b/tests/snapshots/snapshots__docx__handmade-rich.docx.snap @@ -15,8 +15,8 @@ Rich objects follow. - Build - Ship -tiny dot image +![tiny dot image](asset:0) -Embedded object: Excel.Sheet.12 +![Embedded object: Excel.Sheet.12](asset:1) After the objects. diff --git a/tests/snapshots/snapshots__docx__text.docx.snap b/tests/snapshots/snapshots__docx__text.docx.snap index 655b643e..f30ec86f 100644 --- a/tests/snapshots/snapshots__docx__text.docx.snap +++ b/tests/snapshots/snapshots__docx__text.docx.snap @@ -66,7 +66,7 @@ Jump to [the bookmarked paragraph](#plainmark). ## Objects -Inline image: tiny red dot done. +Inline image: ![tiny red dot](asset:0) done. Text box: after the box. diff --git a/tests/snapshots/snapshots__epub__book.epub.snap b/tests/snapshots/snapshots__epub__book.epub.snap index 5e66bd3d..96bbaa9e 100644 --- a/tests/snapshots/snapshots__epub__book.epub.snap +++ b/tests/snapshots/snapshots__epub__book.epub.snap @@ -26,7 +26,7 @@ A list of things: See [Chapter Two](#epub-text-ch002-xhtml-chapter-two) for the table, or jump straight to [the marked paragraph](#epub-text-ch002-xhtml-markpoint). -tiny dot tiny dot +![tiny dot](asset:0) tiny dot diff --git a/tests/snapshots/snapshots__malformed__corrupt-styles--skips.docx.snap b/tests/snapshots/snapshots__malformed__corrupt-styles--skips.docx.snap index 750148a5..ac25bc1e 100644 --- a/tests/snapshots/snapshots__malformed__corrupt-styles--skips.docx.snap +++ b/tests/snapshots/snapshots__malformed__corrupt-styles--skips.docx.snap @@ -66,7 +66,7 @@ Jump to [the bookmarked paragraph](#plainmark). Objects -Inline image: tiny red dot done. +Inline image: ![tiny red dot](asset:0) done. Text box: after the box. diff --git a/tests/snapshots/snapshots__malformed__missing-styles--skips.docx.snap b/tests/snapshots/snapshots__malformed__missing-styles--skips.docx.snap index 750148a5..ac25bc1e 100644 --- a/tests/snapshots/snapshots__malformed__missing-styles--skips.docx.snap +++ b/tests/snapshots/snapshots__malformed__missing-styles--skips.docx.snap @@ -66,7 +66,7 @@ Jump to [the bookmarked paragraph](#plainmark). Objects -Inline image: tiny red dot done. +Inline image: ![tiny red dot](asset:0) done. Text box: after the box. diff --git a/tests/snapshots/snapshots__odp__pres.odp.snap b/tests/snapshots/snapshots__odp__pres.odp.snap index d46d3477..e70d94bd 100644 --- a/tests/snapshots/snapshots__odp__pres.odp.snap +++ b/tests/snapshots/snapshots__odp__pres.odp.snap @@ -17,6 +17,8 @@ Numbers Slide | --- | --- | | North | 42 | +![](asset:0) + Grouped shapes below. Inside a group shape. diff --git a/tests/snapshots/snapshots__odt__text.odt.snap b/tests/snapshots/snapshots__odt__text.odt.snap index 885706ee..820ce773 100644 --- a/tests/snapshots/snapshots__odt__text.odt.snap +++ b/tests/snapshots/snapshots__odt__text.odt.snap @@ -65,7 +65,7 @@ Jump to [the bookmarked paragraph](#plainmark). ## Objects -Inline image: tiny red dot done. +Inline image: ![tiny red dot](asset:0) done. Text box: after the box. diff --git a/tests/snapshots/snapshots__pptx__handmade-order.pptx.snap b/tests/snapshots/snapshots__pptx__handmade-order.pptx.snap index 782ed778..652da2e4 100644 --- a/tests/snapshots/snapshots__pptx__handmade-order.pptx.snap +++ b/tests/snapshots/snapshots__pptx__handmade-order.pptx.snap @@ -8,4 +8,4 @@ Kicker before the title Body after the title -Quarterly numbers +![Quarterly numbers](asset:0) diff --git a/tests/snapshots/snapshots__rtf__text.rtf.snap b/tests/snapshots/snapshots__rtf__text.rtf.snap index f18ce8ea..8b3a8ccd 100644 --- a/tests/snapshots/snapshots__rtf__text.rtf.snap +++ b/tests/snapshots/snapshots__rtf__text.rtf.snap @@ -65,7 +65,7 @@ Jump to [the bookmarked paragraph](#plainmark). ## Objects -Inline image: done. +Inline image: ![](asset:0) done. Text box: Inside the text box.