More than markdown: getting structure from web pages your agent can use

Clean text is half the job. How AIDocument's structure, headings, links, and media, lets agents navigate, cite, and act on pages instead of just reading them.

Turning a web page into clean markdown solves the reading problem. It does not solve the using problem. An agent that only receives prose can summarize a page, but the moment it needs to cite a section, follow the right link, or find the diagram, flat text runs out of road.

That is why an AIDocument carries structure alongside the markdown, and this post is about what that enables.

Headings: the page's own map

The document's heading tree arrives as data, not just as markdown prefixes. Your agent can answer "which section covers authentication" without reading the whole body, jump a long reference page by section, or cite "under Rate limits" instead of "somewhere in the middle." For retrieval pipelines, headings are natural chunk boundaries that keep each embedded chunk about one thing, which is most of the battle in keeping context small.

Links: resolved and ready to follow

Every link arrives with its text and an absolute URL, already resolved against the page's base. The difference sounds small until your agent tries to follow relative paths from raw HTML. With resolved links, a research agent walks documentation naturally: read a page, pick the relevant next link by its text, read that. Crawling logic you did not have to write.

Media: what the page shows, not only what it says

Images and embedded media arrive as a list with sources and alt text where the page provides it. Agents that produce reports can pull the right figure; agents that answer questions know a chart exists even when they cannot see it, and can say so honestly.

One shape, every page

The point of all of it is uniformity. Every page on the web arrives in the same shape: same fields, same types, same places. Your code stops being a museum of per-site parsers and becomes one handler, which is the whole trick behind a web-reading tool in 30 lines. When the page changes its design, the shape does not change with it; that is the index's job to absorb.

Fetch any URL and look at the structure yourself: the free tier is 2,000 documents a month, no card.