LangChain document loaders for live web pages: a clean pattern
A custom LangChain web loader that reads pages as clean AIDocuments instead of raw HTML: about 30 lines of Python, with real token numbers.
LangChain's Document is a deliberately small contract: page_content plus a metadata dict. Splitters, retrievers, and chains all operate on that shape, so the quality of everything downstream is set by what you put in page_content. The hard part was never the contract. It is getting a live web page into it without dragging the page's navigation, scripts, and consent banners along.
The stock approach is to fetch the URL and strip tags from whatever comes back. That fails outright on JavaScript-rendered sites, which hand a plain fetcher an empty shell (we wrote up why that happens), and on the pages that do load, it quietly fills your chunks with boilerplate. Here is the pattern we use instead: a custom loader, about 30 lines of Python, that reads pages through the Lyrenth AIDocument API and yields Documents whose body is clean Markdown.
One thing to be clear about up front: this is a pattern, not an official package. There is no langchain-lyrenth on PyPI. Copy the class, adapt it, own it.
The loader
from collections.abc import Iterator
import requests
from langchain_core.documents import Document
API = "https://api.lyrenth.com/v1/aidocument"
class LyrenthLoader:
"""Yield one Document per URL, read from the Lyrenth index."""
def __init__(self, urls: list[str], api_key: str, force_refresh: bool = False):
self.urls = urls
self.api_key = api_key
self.force_refresh = force_refresh
def lazy_load(self) -> Iterator[Document]:
for url in self.urls:
payload = {"url": url}
if self.force_refresh:
payload["force_refresh"] = True
resp = requests.post(
API,
json=payload,
headers={"Authorization": f"Bearer {self.api_key}"},
timeout=60,
)
resp.raise_for_status()
doc = resp.json()["aidocument"]
yield Document(
page_content=doc["content"]["markdown"],
metadata={
"title": doc["identity"].get("title", ""),
"source": doc["source"]["url"],
"fetched_at": doc["source"].get("fetched_at", ""),
"tokens": doc["economics"]["output_tokens_approx"],
},
)
def load(self) -> list[Document]:
return list(self.lazy_load())
Choices worth naming.
lazy_load is the whole interface. LangChain's loader base classes have moved around between versions, but the stable core is a method that yields Document objects one at a time. If your version wants a BaseLoader subclass, add the base class and keep the body. from langchain_core.documents import Document is the import we would reach for first; if your project pins something older, that one line is the thing to adjust.
The body is already Markdown. content.markdown is the cleaned page: headings, lists, tables, and code blocks survive, while navigation, scripts, and cookie chrome do not, and JavaScript-rendered pages arrive rendered. There is nothing left to parse. The full response shape is documented at /docs/aidocument.
Metadata carries provenance and cost. title and source give the chain something to cite. fetched_at says how old the snapshot is. tokens is the API's own output_tokens_approx for the body, so you can budget a prompt before you ever call a model.
Failures are loud and free. raise_for_status() surfaces them immediately, and failed fetches (4xx/5xx) cost no credits. The errors are structured, too: a page behind a login returns HTTP 422 with {"error": "upstream_unauthorized", ...} telling you the host answered 401. Lyrenth fetches as an anonymous client and never signs in to accounts, so a login-walled URL is a hard, diagnosable no rather than a silently garbled page in your vector store.
Three URLs into a chain
import os
loader = LyrenthLoader(
urls=[
"https://docs.stripe.com/api",
"https://vercel.com/docs/functions",
"https://developers.cloudflare.com/workers/",
],
api_key=os.environ["LYRENTH_API_KEY"],
)
docs = loader.load()
for d in docs:
print(f'{d.metadata["title"]}: {d.metadata["tokens"]} tokens')
From here it is ordinary LangChain. One caveat: splitter and chain import paths vary across versions more than Document does, so treat the wiring below as shape, not gospel.
from langchain_text_splitters import RecursiveCharacterTextSplitter
splitter = RecursiveCharacterTextSplitter(chunk_size=2000, chunk_overlap=200)
chunks = splitter.split_documents(docs)
Splitting Markdown instead of raw HTML has a quiet benefit: chunk boundaries land between paragraphs and headings, not in the middle of a <script> tag, so each chunk is self-contained prose. Every chunk also inherits its parent Document's metadata, which makes citation trivial:
context = "\n\n".join(
f'[{c.metadata["source"]}]\n{c.page_content}' for c in chunks
)
answer = llm.invoke(
"Answer using only the context below, and name the source URL "
"you answered from.\n\n"
f"Context:\n{context}\n\nQuestion: {question}"
)
Any chat model object with an invoke works there. Stuffing every chunk into one prompt looks lazy, but for these three pages it is fine on the numbers: in our measured benchmarks they total 5,526 tokens as AIDocuments. The same three pages as raw HTML total 640,255 tokens, which does not fit in one prompt at all. Substitute your retrieval stack once the corpus outgrows the window.
Why not just fetch the HTML
Because a modern page is mostly not content, and a loader that ships raw markup makes your chain pay for every angle bracket. The clearest single number we have measured: the Stripe API reference is 307,902 tokens as raw HTML and 2,000 tokens as an AIDocument. That is 99.4 percent smaller, and at $3.00 per million input tokens it is the difference between $0.92 and $0.006 each time the page goes into context. The full table and methodology are at /benchmarks.
The waste does not stop at load time. A splitter dutifully chunks boilerplate, the boilerplate chunks dilute whatever ranking you do, and the model reads past class names on its way to the answer. The longer treatment of where the tokens go is in how to feed web pages to an LLM without blowing the context window.
Re-running the chain does not re-crawl the web
The second reason to put an index between LangChain and the web is what happens on run two. Everything the API returns is served from the index. When cache.status is "hit", the copy already existed: it comes from the cached index, instantly, and the site is not contacted. A page that is not in the index yet is crawled, indexed, and served from that fresh index entry, freshly indexed for you and sitting there for the next caller. One crawl serves every caller.
Chains get re-run constantly. You adjust a prompt, restart the notebook, run the eval suite again. With a fetch-based loader, every one of those runs hammers the same three origins. With the index, run two onward is all hits, so iterating on your chain costs the sites you read nothing. And with over 1.2 billion documents across more than 127 million domains in the index, plenty of first runs are hits too.
Two dials when freshness matters. Each plan caps how old a served copy may be: 90 days on Free, 7 days on Starter, 24 hours on Pro, with the rest of the plan table at /pricing. And force_refresh, which the loader exposes as a constructor flag, re-reads the page now for 2 credits instead of 1.
When a crawl does happen, it identifies itself: LyrenthBot user agent, Web Bot Auth signed requests, published IP ranges, and reverse DNS, all verifiable at /bot, with robots.txt respected. The crawler reads only the public web, and Lyrenth does not train foundation models on crawled content.
If you want to run the loader as written, create a free key: 2,000 AIDocuments a month at one request per second, no card required. The curl-level walkthrough of the same endpoint is in the quickstart.