Why a PDF has no paragraphs, only glyphs at coordinates

Copy a paragraph out of a two-column PDF and paste it into an email. Sometimes it arrives whole. Sometimes the columns interleave line by line, the hyphens left over from the line breaks come along with them, two or three word spaces are missing, and the odd letter pair turns up as a square box. That is not the email client failing. It is the file being read exactly as it was written.

A PDF page does not contain a paragraph. It does not contain a sentence, a word or a line either. It contains instructions for putting marks on a surface, and everything above the level of a mark is something a reader works out afterwards.

What a page actually holds

The visible part of a page lives in a content stream: a short program in a stack-based language, run once, top to bottom, to paint the page. Decompressed, a single line of text looks close to this.

BT
/F1 11 Tf
72 708 Td
[(Amount pa) 20 (yable b) 15 (y 30 September)] TJ
ET

BT and ET open and close a text object. Tf selects a font resource and a size. Td moves to a position measured from the bottom left corner of the page. TJ paints a run of glyphs, and the bare numbers inside the array nudge the position between them by thousandths of the current font size, which is how kerning and justification get done.

Read it again with the question where is the paragraph in mind. There is a font resource, a starting coordinate, a run of glyph codes and some spacing adjustments. Nothing else. A second Td and a second TJ paint the next line, and no part of the file says the two lines belong together. They look like they belong together because they share a size, share a left edge and sit one line apart. That is a fact about the picture, not a fact about the data.

Glyph codes are not letters

The numbers inside a TJ array are not Unicode. They are indices into whatever encoding the chosen font declares, and for an embedded subset font that encoding is often arbitrary: the producer needed thirty-one distinct glyphs on this page, so they are numbered one to thirty-one. Getting from those codes back to text needs a ToUnicode map, which the producing application may or may not have written, and which can be wrong where it exists.

When it is missing you get the familiar result — a page that renders perfectly, prints perfectly, yields nonsense to copy and paste and cannot be found by search at all. When it is present but incomplete, one or two characters break instead of the lot. A ligature is the usual casualty: a single glyph drawing fi or ffl maps back to one character, or to none, depending on how careful the producer was.

Word spaces are a second reconstruction. Sometimes there is a real space character in the run. Sometimes the producer positioned each word separately and there is nothing between them but a coordinate, so an extractor has to decide whether a given gap is a word break, a wide letter space or the start of the next column. It decides from the width of the gap, and where it decides wrongly you get runtogetherwords or the reverse.

Reading order is the order somebody emitted it in

The content stream is painted in sequence, and the sequence is whatever the producing application wrote. Usually that matches the order a person reads the page in. It does not have to. A layout program that draws all the body text and then goes back for the pull quote produces a file where the quote lands mid-paragraph on extraction, because that is where it sits in the stream.

Careful extractors do not trust stream order. They sort runs by geometry, group runs into lines, lines into blocks and blocks into a reading order. That is the page being turned back into structure, and it is a guess every time. Two columns are where the guess is most visible, because getting it wrong produces interleaved sentences rather than slightly odd spacing.

Why the text moved

Here is the consequence that costs people afternoons. Replace 30 September with 15 October in the run above and you have changed both the number of glyphs and the total width of the run. Nothing else on the page knows. There is no line to re-break and no paragraph to re-flow, because neither exists. The new text draws from the same coordinate, and it is longer, so it runs nearer the margin, or past it, or over whatever was printed to its right.

The other direction is worse in a quiet way. Shorter replacement text leaves a gap where the old glyphs were, and a gap reads as a formatting accident rather than as somebody's edit.

Every editor that appears to re-flow a PDF paragraph is doing the extractor's reconstruction and then acting on it. It notices that these runs share a font, a size, a left edge and an even baseline spacing, calls the result a paragraph, and re-wraps the guess. When the guess is right the effect is convincing. When it is wrong you find out in one of two ways: a table with tidy rows gets treated as one running paragraph, or a sentence that changes font halfway is treated as two separate blocks. Both are visible straight away, which is the redeeming feature.

Fonts decide how much of an edit survives. A run set in one of the fourteen standard faces that readers have long been expected to supply can be retyped and nothing changes but the words. A run set in an embedded subset is harder, because the file carries only the glyphs the original author actually typed — a letter they never used has no drawing in there to reuse, and has to come from a system font of the same class, which will not have identical metrics.

This is where on-page editing in Creasepoint spends its effort: it re-wraps the block it worked out from the geometry, then says in words what that did. That the block grew downwards over whatever sat below it. That a word too long to break reached past the right edge. That a character had no glyph in the chosen face and was dropped by the renderer. Those messages exist because the block underneath them is a guess.

Structure exists, when somebody bothered

PDF has carried a logical structure layer for a long time. A tagged PDF holds a parallel tree of elements — headings, paragraphs, list items, table cells, alternative text for figures — pointing at marked content in the stream. That tree is what a screen reader follows, and it is what the word accessible means when a procurement document uses it.

Two things are worth knowing about it. Most PDFs in circulation carry no tags at all, because tagging depends on the producing application and on somebody having switched it on. And the tree is an assertion made alongside the drawing instructions rather than derived from them, so a document can be tagged with a structure that does not match what the page shows. Reading order in the tags and reading order on the page are separate claims, and one of the standard accessibility failures is a file where the two disagree.

What follows from all this

• A PDF that will not let you select text is not locked. There is no text in it — the page is a picture, and recognition writes words underneath the picture without altering it.
• Search failing on a word you can plainly see almost always means a missing or broken character map, not a missing word.
• Any export back to a word processor recovers text and paragraph breaks and abandons the design, because the design was coordinates and the destination has none. The Word export guide is about deciding whether that trade is the one you want.
• If the layout is the thing you need, the PDF is the wrong file to work from. Go and find the document it was printed from.
• If one wrong date is the thing you need, editing the PDF is right, and the two places to look afterwards are the line beneath the change and anything printed to its right.