# Page Content

Parses a PDF page's content stream into a structured, human-usable model: text runs grouped into words, lines, blocks, and reading-order flows, plus vector objects and images. PDF was never designed for text extraction — operators may draw glyphs in arbitrary order and position — so this layer is fundamentally a large pile of heuristics reconstructing reading order from graphical primitives.

Source: [Page Content. Pavlo Shkrabliuk, 15 Dec 2022](https://drive.google.com/file/d/1gmrlagGo8azFN-uwbNG9ABJ6pbizwbay/view?usp=sharing).

## Entry point

`RDPDFPageContent` — central model holding all text, inline images, and vector objects found on a page. Exposes `addCharacters` (during parsing) and `coalesce` (post-parse). Has a `contentType`:
- `Normal` — full content (text + fonts + colors + vectors + images). Effectively always used today.
- `ImageOnly` — images only. Optimization for Reduce File Size.
- `TextUnicodeAndPositions` — text + positions for indexing (Spotlight-style). Skips fonts and most vector work.

Historically, older builds kept fonts/colors out of `Normal` and had to reload a richer content for editing. No longer split.

## Key classes

### Parsing
- **`ContentStreamLexer`** — reads the raw byte stream, emits tokens (`string`, `name`, `command`, `data`). Takes an array of streams as input (PDF spec allows page content as a single stream or an array concatenated).
- **`ContentStreamParser`** — built on the lexer. Drives tokenization and dispatches each operator via `applyOperator`. This is **not** `RDPDFParser` — the generic arrays/dictionaries parser is not used here. Caller fills a `parser_callbacks` struct of C function pointers, one per PDF operator (Appendix A of PDF spec). `applyOperator` is hand-tuned C — compares first char + second char directly (e.g. `'B'` then `'T'` for `BT`/beginText) instead of string comparison. Reused for annotation appearance streams; only the callbacks differ.
- **`PageAnalyzer`** — middle layer between the content-stream parser and `RDPDFPageContent`. Owns the callbacks. Two jobs: push results into the page content, and own+mutate the current graphics state.
- **`RDPDFGraphicsState`** — holds CTM, text matrix, colors, font, etc., with a save/restore stack (`q`/`Q`).

### Text model
- **`RDPDFTextPool`** — hash table keyed by baseline, value is a linked list of words sorted by primary coordinate (origin.x for rotation 0, etc.). Maintains a **cursor** pointing to the last-inserted word; if the next word is on the same baseline after the cursor, append in O(1) instead of walking the chain. Four pools per page, one per rotation. Also used as `RDPDFTextBlock`'s internal storage.
- **`RDPDFTextWord`** — a word. Created lazily on first character; subsequent `addCharacters` glue to it or close it and start a new one. Fields: `edges` (per-glyph x cuts inside the word's rect, for hit-testing), `baseline`, `associatedBaseline` (used to put superscripts/footnote markers on the same visual line as their host word while keeping their true baseline). **Not final until `coalesce` ends** — words can merge, split, or acquire duplicates' "innards" later.
- **`RDPDFTextLine`** — line of words on a shared baseline. Runs duplicate elimination.
- **`RDPDFTextBlock`** — roughly a paragraph. Internally backed by an `RDPDFTextPool` + extra pointers. Has a `deduplicateTextBlock` that de-dups once more when the block is formed.
- **`RDPDFTextFlow`** — higher-level than block. Ordered sequence of blocks/images/etc. The order of `flows` defines the presumed reading order (used by Reading Mode, which concatenates flows into `NSAttributedString`).

### Text extraction
- **`RDPDFPageTextContent`** — extracts user-facing text from a `PageContent` given a cursor-position range (Cmd+C, selection, reading mode).
- **Cursor positions** — matryoshka: `RDPDFDocumentTextCursorPosition → PageTextCursorPosition → FlowTextCursorPosition → BlockTextCursorPosition → …` → character index. How user taps/drags resolve to text positions.
- **TextNormalizationEngine** — concatenates extracted text with correct line-breaks and handles bidi (RTL Arabic/Hebrew, mixed RTL/LTR). Known-rough area; RTL fonts are often broken.

### Infrastructure
- **`PageContentDebug`** — struct of boolean debug flags (draw words, lines, blocks, flows). On iOS: long-press the "AA" button opens a debug menu. On macOS: put a breakpoint in the getter, flip fields live in the debugger. Rainbow word rendering cycles colors + small offset so overlapping duplicates become visually distinguishable.
- **`ExtrasLoader`** — manages lazy loading of page content (and annotations). "Extras" is an xpdf-inherited name. Page Content knows nothing about annotations, but annotations depending on PageContent (text-markup Highlight, snippet rendering) must wait.

### Scripting / R&D
Any C++ files or code not following project naming conventions are from the R&D team (performance-tuned, narrow-purpose). Treat carefully. Example: removing one field from a C++ struct saved 400 MB on a specific document.

## Core concepts / terminology

- **Character code**: the integer per glyph in the content stream (e.g. `Tj` operand). Not Unicode. Means "draw glyph N from the current font at current text-matrix position".
- **CID → Unicode**: resolved via the font's ToUnicode/encoding tables. Entirely inside the Fonts subsystem; `PageAnalyzer.doDrawText` calls `getCharactersFromFont` to convert char codes to Unicode. May return garbage or nothing — fonts are frequently broken. This is why visible text on screen can copy out as junk: glyphs render, but Unicode mapping failed.
- **UniChar is UTF-16**: a single visible glyph can map to multiple `UniChar` (e.g. Chinese ideographs → 3 UniChars for one rect). Words store per-UniChar edges by dividing the rect's width evenly across them.
- **Rotation**: only four supported (0/90/180/270). Decided by inspecting the text matrix. The 90° check has intentional ~5° slack to handle skewed OCR'd scanned books.
- **Bounding box per character**: from glyph metrics. Not always good — can have correct width but zero height; other pathologies exist.
- **Baseline vs associated baseline**: `baseline` is the true text-matrix baseline. `associatedBaseline` is assigned when a word (e.g. superscript `1`) is glued to a host word (`philosophy`) so reading-mode output puts them on one line, while per-word baselines stay honest.
- **Primary coordinate**: origin.x for rotation 0, max.x for rotation 2, etc.
- **Chunk**: rectangle of the page delimited by detected horizontal/vertical separators (lines, sometimes bitmaps). Each chunk is processed independently. Recursive: split horizontally, then within each piece look for vertical dividers, then horizontal, etc.
- **Histogram**: auxiliary structure for chunk detection. Horizontal/vertical projections that reveal text-free rows/columns; validate splitter candidates (does the line actually cross empty space, or does it cross text and should be rejected?).
- **Flow**: container of blocks (and images, etc.). Ordering of flows = presumed reading order.
- **Struct 3 / Marked Content** (`BMC` / `BDC` / `EMC`, `/Artifact`, `ActualText`): author-supplied PDF tagging. Per spec, content-stream order then equals reading order. In practice: ~20% of documents have any marked content; only ~20% of those are tagged correctly → ~5% of real-world PDFs are reliably tagged. While parsing, words are dropped into per-tag pools in addition to the four rotation pools; if tagging is clean, those are used.
- **TR 3 (text rendering mode 3, "invisible text")**: used by OCR'd scans that keep the bitmap and overlay invisible Unicode text. Fully supported.
- **Underline**: not a font feature. Extracted separately by analyzing drawn lines / rectangles / multiple lines near a word's baseline. Parallel heuristic track to word collection. Matters for editing.
- **Bold-by-overprinting**: many generators fake bold by drawing the same word twice at near-identical positions. Dedup must detect and merge; the dropped duplicate's stream-position innards go into the survivor so edit/delete removes all copies.
- **Superscripts/subscripts**: heuristics glue numbers sitting slightly above/below a baseline (e.g. footnote markers) to the preceding word as associated-baseline words.

## Pipeline / flow

Stages from raw bytes to reading-order flows:

1. **Page content source resolution** — page dictionary gives a stream ref or an array of stream refs; array treated as one logical stream.
2. **Tokenization** — `ContentStreamLexer` emits tokens.
3. **Operator dispatch** — `ContentStreamParser.applyOperator` branches via hand-written C on first/second char and invokes the caller-supplied callback.
4. **Graphics-state maintenance** — `PageAnalyzer` owns `RDPDFGraphicsState`. Callbacks read/write it; `q`/`Q` push/pop.
5. **Text drawing** — all text operators (`Tj`, `TJ`, `'`, `"`) funnel into one `doDrawText` method on `PageAnalyzer`. Char codes → Unicode via `getCharactersFromFont`.
6. **Character delivery** — `PageAnalyzer` calls `RDPDFPageContent.addCharacters` **as a C function with the page content passed as "self"** (no objc_msgSend). Historical first-iPad optimization.
7. **Word building** — `addCharacters` tries to glue new Unicode chars onto the current word or ends it and starts a new one. Heuristics: punctuation (full stops → end word), distance, spaces drawn as glyphs, font change, rotation. ~31 tuning constants in `TextAnalysisConstants` — **do not change lightly**. On word-end, word goes into one of four rotation-indexed `RDPDFTextPool`s (plus struct-3-tag pools if tagged).
8. **Duplicate suppression #1** (insertion into pool) — scans the baseline chain; if the same word would be inserted twice (bold-by-overprint), the incoming word's stream-position innards are transferred into the survivor so future edits remove both.
9. **`coalesce`** (after parsing) — large method. Calls `textBlocksByClassifyingWordsInRotationPoolsUsingHeuristicsWithCancelBlock` (or similarly named core routine). For each pool:
   - Find leftmost-topmost "normal-script" word as a seed (skip words whose baseline is slightly raised — likely superscripts, not block starts).
   - Grow a block (`blk`, xpdf-inherited name): iterate candidates within a small vertical delta around the current baseline; admit if baseline proximity + chunk membership + font match + gap proximity pass.
   - Repeat until pool empty.
10. **Line-level dedup** — `RDPDFTextLine` de-dups again (Word-generator case: cross-chunk redraws overlap by one glyph).
11. **Block finalization** — `RDPDFTextBlock.deduplicateTextBlock` runs a third dedup pass.
12. **Chunking & histograms** — detect splitter lines (and bitmaps), validate against histograms, recursively subdivide. Block-building takes a chunk rect and admits only words inside it.
13. **Flow assembly** — blocks grouped into `RDPDFTextFlow`s in reading order via the chunk tree + per-block relationships. Flows may include images. Struct-3 data, when present and clean, short-circuits this.
14. **Text extraction on demand** — `RDPDFPageTextContent.text(in: cursorRange)` walks flows → blocks → lines → words, concatenates, runs text-normalization (bidi, line joins).

Parallel passes during/after parsing: underline detection, superscript/subscript association, form/image metadata, TR3 invisible-text accommodation.

## Gotchas, invariants, "why" notes

- **Heuristic soup**: no one knows all the heuristics. At least 10 people wrote code here over years. Changes in one place often silently affect distant cases. **Always re-run screenshot tests.**
- **Words are not final until `coalesce` returns**: between `addCharacters` and end of `coalesce`, words can merge, split, gain innards from duplicates. Anything before end-of-coalesce is a precursor.
- **C-function dispatch for hot paths**: `addCharacter(s)` is called as a C function with `self` pointer, skipping objc_msgSend. Historical optimization; probably no longer load-bearing, but kept. Same pattern in `applyOperator` (first-char + second-char comparisons).
- **Performance paradox**: the block-building phase walks pools many times; aggregate work dwarfs any msgSend savings in the parser. Algorithmic complexity is the real cost.
- **Memory pressure is real**: PDFs can contain millions of vector objects. Code has tuned cut-off limits — beyond a threshold, certain objects are not analyzed to avoid OOM. One R&D win: removed one field from a C++ struct, saved 400 MB on a specific document. Keep struct sizes tight.
- **Do not trust character rects**: some glyphs return zero height, zero width, or wildly off rects. Downstream must tolerate this.
- **Font-to-Unicode is the #1 source of garbage**: visible text ≠ extractable text. If extraction breaks, suspect the font mapping layer, not Page Content.
- **Struct-3 is not a silver bullet**: ~5% of documents benefit. Do not regress the tag-aware pools anyway.
- **Rotation 90° tolerance**: accepts up to ~5° off for scanned-book text. Don't tighten.
- **Edges array length ≠ UniChar count per glyph**: one rect divided evenly across its UniChars. Preserve when editing.
- **Dedup must preserve stream origins**: when dropping a duplicate word, transplant its stream-position metadata into the survivor. Deleting the visible word must delete all hidden copies.
- **Associated baseline invariant**: glued superscripts keep their real baseline; only `associatedBaseline` lies. Preserve both.
- **Constant table (`TextAnalysisConstants`)**: ~31 magic numbers. Battle-tested. Any change = full screenshot-test sweep.
- **Threading / loading**: Page Content loading is expensive and deferred. `ExtrasLoader` drives it. Annotations depending on PageContent must wait.
- **`contentType` is effectively always `Normal`**: `ImageOnly` and `TextUnicodeAndPositions` are optimizations for narrow call sites. All editing and selection assume `Normal`.
- **Historical xpdf origins**: original C++ from xpdf was ported to Objective-C long ago, then grew to 180+ files. Names like `blk`, `Extras` come from xpdf.

## Screenshot tests

- `screenshot test results/` contains per-test folders.
- Two relevant ones: **text analysis** (bounding boxes of words, lines, blocks, flows, formulas, headers, footers, image labels → plist) and **text extraction** (plain text of the page → .txt).
- Corpus on disk at `rdpdfkit screenshot test source files`.
- Tests emit plists (not PNGs) to avoid antialiasing flakes across macOS versions.
- Failures produce 3 diff images per failed page; require human judgment (diff ≠ regression).

## Author's aspirational refactor (not implemented)

Collect chars → classify into words → words into lines → lines into blocks → blocks into flows, in strict pipeline order, rather than the current entangled heuristic passes. Not implemented; noted as long-term direction.

## Related directories

- `Content Stream/` — `ContentStreamLexer`, `ContentStreamParser`, `parser_callbacks`, op-dispatch. Shared with annotation appearance-stream rendering.
- `Parsing/` — generic PDF object parser (`RDPDFParser`) and lexer primitives. **Not used** for page content.
- `Page Content Editing/` — downstream of this layer; relies on final post-coalesce structure plus preserved stream-position innards in deduped words.
- `Fonts/` — `getCharactersFromFont`, ToUnicode/encoding tables. The black box converting char codes to UniChars (or failing).
- `Resources/` — fonts, XObjects, color spaces referenced from content streams.
- `Search/`, `FTS/` — downstream consumers of Page Text Content (indexing uses `TextUnicodeAndPositions`).
- `Text Reflow/`, `HTML Generator/` — consume flows/blocks to generate reading-mode and HTML output.
- `Scan/` — TR 3 (invisible-text) OCR integration; deposits the invisible text layer this code later extracts.
- `Rendering/` — uses the same `ContentStreamParser` with CG-backed callbacks.
- `Undo/`, `Page Content Editing/` — rely on stream-position metadata preserved in deduped words to edit overlapping copies together.
- Screenshot test assets: `Core/Screen Shot Tests/` + external `rdpdfkit screenshot test source files`.
