Two published packages for the RAG frameworks, and two recipes that need no package at all. Whichever you pick, a URL comes back as clean Markdown that is ready to hand to a model.
Each framework has its own package on PyPI, in that framework's own layout.
langchain-lyrenthEvery URL becomes a LangChain Document whose page_content is the cleaned Markdown. Pass one URL or many. lazy_load() streams them one at a time instead of building the whole list; fresh=True forces a live re-fetch, and client= reuses a Lyrenth you already configured. The key is read from LYRENTH_API_KEY.
pip install langchain-lyrenth
from langchain_lyrenth import LyrenthLoader
loader = LyrenthLoader([
"https://example.com/a",
"https://example.com/b",
])
docs = loader.load() # list[langchain_core.documents.Document]
docs[0].page_content # the cleaned Markdown
docs[0].metadata # {"source", "title", "description", "word_count"}llama-index-readers-lyrenthThe same idea in LlamaIndex shape: every URL becomes a Document whose text is the cleaned Markdown, with the same four metadata keys. fresh=True and client= work the same way here.
pip install llama-index-readers-lyrenth
from llama_index.readers.lyrenth import LyrenthReader
docs = LyrenthReader().load_data([
"https://example.com/a",
"https://example.com/b",
]) # list[llama_index.core.Document]
docs[0].text # the cleaned Markdown
docs[0].metadata # {"source", "title", "description", "word_count"}If you already depend on the main Python SDK and would rather not add a package, the same two adapters ship inside it as optional extras. They import their framework lazily, so installing the extra is what pulls the framework in.
pip install 'lyrenth[langchain]' # lyrenth.langchain.LyrenthLoader pip install 'lyrenth[llamaindex]' # lyrenth.llamaindex.LyrenthReader
Let the model decide when to open a page, mid-generation.
Vercel AI SDKThe TypeScript SDK ships the tool already built, on the lyrenth/ai subpath. Drop it into a tools map and the model can call it. It targets AI SDK v5 and later. Under it is one POST /v1/aidocument, and what comes back to the model is the title, description, Markdown and word count.
// npm install lyrenth ai
import { generateText } from "ai";
import { lyrenthReadTool } from "lyrenth/ai";
const { text } = await generateText({
model: yourModel,
tools: { read_url: lyrenthReadTool() }, // reads LYRENTH_API_KEY
prompt: "Read https://example.com/article and summarize it.",
});eveeve names a tool after the file it lives in, so the whole integration is a re-export. Create agent/tools/read_url.ts with this in it and the model has a read_url tool. Name the file something else and the tool is called that instead. eve is an optional peer dependency: importing lyrenth or lyrenth/ai never pulls it in.
// agent/tools/read_url.ts
export { default } from "lyrenth/eve";OpenAI SDKNo Lyrenth package needed. Declare a read_url function, and when the model calls it, make one HTTP request and hand the Markdown back as the tool result. The whole Lyrenth side is the request in read_url; everything else is ordinary tool calling.
# pip install openai
import json, os, urllib.request
from openai import OpenAI
TOOL = {
"type": "function",
"function": {
"name": "read_url",
"description": (
"Read a public web page as clean Markdown, with navigation and "
"boilerplate stripped. Prefer this over a raw HTTP fetch."
),
"parameters": {
"type": "object",
"properties": {
"url": {"type": "string", "description": "Absolute http(s) URL."}
},
"required": ["url"],
},
},
}
def read_url(url: str) -> str:
"""One call to Lyrenth. No SDK involved, just HTTP."""
req = urllib.request.Request(
"https://api.lyrenth.com/v1/aidocument",
data=json.dumps({"url": url}).encode(),
method="POST",
headers={
"Authorization": f"Bearer {os.environ['LYRENTH_API_KEY']}",
"Content-Type": "application/json",
},
)
with urllib.request.urlopen(req) as resp:
doc = json.load(resp)
return doc["content"]["markdown"]
client = OpenAI()
MODEL = os.environ["OPENAI_MODEL"] # whichever model you already use
messages = [{"role": "user", "content": "Summarize https://example.com/article"}]
first = client.chat.completions.create(
model=MODEL, messages=messages, tools=[TOOL]
)
message = first.choices[0].message
messages.append(message)
for call in message.tool_calls or []:
args = json.loads(call.function.arguments)
messages.append({
"role": "tool",
"tool_call_id": call.id,
"content": read_url(args["url"]),
})
second = client.chat.completions.create(
model=MODEL, messages=messages, tools=[TOOL]
)
print(second.choices[0].message.content)// npm install openai
import OpenAI from "openai";
const TOOL = {
type: "function" as const,
function: {
name: "read_url",
description:
"Read a public web page as clean Markdown, with navigation and " +
"boilerplate stripped. Prefer this over a raw HTTP fetch.",
parameters: {
type: "object",
properties: {
url: { type: "string", description: "Absolute http(s) URL." },
},
required: ["url"],
},
},
};
// One call to Lyrenth. No SDK involved, just HTTP.
async function readUrl(url: string): Promise<string> {
const res = await fetch("https://api.lyrenth.com/v1/aidocument", {
method: "POST",
headers: {
Authorization: `Bearer ${process.env.LYRENTH_API_KEY}`,
"Content-Type": "application/json",
},
body: JSON.stringify({ url }),
});
const doc = await res.json();
return doc.content.markdown;
}
const client = new OpenAI();
const MODEL = process.env.OPENAI_MODEL!; // whichever model you already use
const messages: any[] = [
{ role: "user", content: "Summarize https://example.com/article" },
];
const first = await client.chat.completions.create({
model: MODEL,
messages,
tools: [TOOL],
});
const message = first.choices[0].message;
messages.push(message);
for (const call of message.tool_calls ?? []) {
const args = JSON.parse(call.function.arguments);
messages.push({
role: "tool",
tool_call_id: call.id,
content: await readUrl(args.url),
});
}
const second = await client.chat.completions.create({
model: MODEL,
messages,
tools: [TOOL],
});
console.log(second.choices[0].message.content);Two calls to the model, as always with tool use: the first lets it ask for a page, the second lets it answer with the page in hand. Use whichever model you already use; nothing here depends on which one it is.
The MCP server gives any MCP client the same reader with no code to write.