Framework integrations

Two published packages for the RAG frameworks, and two recipes that need no package at all. Whichever you pick, a URL comes back as clean Markdown that is ready to hand to a model.

RAG

LangChain and LlamaIndex

Each framework has its own package on PyPI, in that framework's own layout.

langchain-lyrenth

A tool an agent can call

LyrenthReadTool gives an agent the ability to read any page it decides it needs. The tool is called read_url, takes one URL, and returns the page as Markdown with its title and source at the top. It implements the async path as well, so it does not block the event loop an agent runs on.

install

pip install langchain-lyrenth

use

from langchain.agents import create_agent
from langchain_lyrenth import LyrenthReadTool

agent = create_agent(
    model=your_model,
    tools=[LyrenthReadTool()],    # reads LYRENTH_API_KEY
    system_prompt="You are a research assistant.",
)

agent.invoke({"messages": [
    {"role": "user", "content": "What does https://example.com/pricing say?"}
]})

langchain-lyrenth

A document loader

Every URL becomes a LangChain Document whose page_content is the cleaned Markdown. Pass one URL or many. lazy_load() streams them one at a time instead of building the whole list; fresh=True forces a live re-fetch, and client= reuses a Lyrenth you already configured. The key is read from LYRENTH_API_KEY. max_tokens=4000 caps a long page at roughly that many tokens, trimmed at a clean boundary, on either component.

use

from langchain_lyrenth import LyrenthLoader

loader = LyrenthLoader([
    "https://example.com/a",
    "https://example.com/b",
])
docs = loader.load()          # list[langchain_core.documents.Document]

docs[0].page_content          # the cleaned Markdown
docs[0].metadata              # {"source", "title", "description", "word_count"}

llama-index-readers-lyrenth

A reader

The same idea in LlamaIndex shape: every URL becomes a Document whose text is the cleaned Markdown, with the same four metadata keys. fresh=True and client= work the same way here.

install

pip install llama-index-readers-lyrenth

use

from llama_index.readers.lyrenth import LyrenthReader

docs = LyrenthReader().load_data([
    "https://example.com/a",
    "https://example.com/b",
])                            # list[llama_index.core.Document]

docs[0].text                  # the cleaned Markdown
docs[0].metadata              # {"source", "title", "description", "word_count"}

If you already depend on the main Python SDK and would rather not add a package, the same two adapters ship inside it as optional extras. They import their framework lazily, so installing the extra is what pulls the framework in.

pip install 'lyrenth[langchain]'    # lyrenth.langchain.LyrenthLoader
pip install 'lyrenth[llamaindex]'  # lyrenth.llamaindex.LyrenthReader

Tool calling

Give a model a reader

Let the model decide when to open a page, mid-generation.

Vercel AI SDK

A ready-made tool

The TypeScript SDK ships the tool already built, on the lyrenth/ai subpath. Drop it into a tools map and the model can call it. It targets AI SDK v5 and later. Under it is one POST /v1/aidocument, and what comes back to the model is the title, description, Markdown and word count.

// npm install lyrenth ai
import { generateText } from "ai";
import { lyrenthReadTool } from "lyrenth/ai";

const { text } = await generateText({
  model: yourModel,
  tools: { read_url: lyrenthReadTool() },   // reads LYRENTH_API_KEY
  prompt: "Read https://example.com/article and summarize it.",
});

eve

One line, because tools are files

eve names a tool after the file it lives in, so the whole integration is a re-export. Create agent/tools/read_url.ts with this in it and the model has a read_url tool. Name the file something else and the tool is called that instead. eve is an optional peer dependency: importing lyrenth or lyrenth/ai never pulls it in.

// agent/tools/read_url.ts
export { default } from "lyrenth/eve";

OpenAI SDK

A tool you define yourself

No Lyrenth package needed. Declare a read_url function, and when the model calls it, make one HTTP request and hand the Markdown back as the tool result. The whole Lyrenth side is the request in read_url; everything else is ordinary tool calling.

# pip install openai
import json, os, urllib.request
from openai import OpenAI

TOOL = {
    "type": "function",
    "function": {
        "name": "read_url",
        "description": (
            "Read a public web page as clean Markdown, with navigation and "
            "boilerplate stripped. Prefer this over a raw HTTP fetch."
        ),
        "parameters": {
            "type": "object",
            "properties": {
                "url": {"type": "string", "description": "Absolute http(s) URL."}
            },
            "required": ["url"],
        },
    },
}


def read_url(url: str) -> str:
    """One call to Lyrenth. No SDK involved, just HTTP."""
    req = urllib.request.Request(
        "https://api.lyrenth.com/v1/aidocument",
        data=json.dumps({"url": url}).encode(),
        method="POST",
        headers={
            "Authorization": f"Bearer {os.environ['LYRENTH_API_KEY']}",
            "Content-Type": "application/json",
        },
    )
    with urllib.request.urlopen(req) as resp:
        doc = json.load(resp)
    return doc["content"]["markdown"]


client = OpenAI()
MODEL = os.environ["OPENAI_MODEL"]      # whichever model you already use
messages = [{"role": "user", "content": "Summarize https://example.com/article"}]

first = client.chat.completions.create(
    model=MODEL, messages=messages, tools=[TOOL]
)
message = first.choices[0].message
messages.append(message)

for call in message.tool_calls or []:
    args = json.loads(call.function.arguments)
    messages.append({
        "role": "tool",
        "tool_call_id": call.id,
        "content": read_url(args["url"]),
    })

second = client.chat.completions.create(
    model=MODEL, messages=messages, tools=[TOOL]
)
print(second.choices[0].message.content)

// npm install openai
import OpenAI from "openai";

const TOOL = {
  type: "function" as const,
  function: {
    name: "read_url",
    description:
      "Read a public web page as clean Markdown, with navigation and " +
      "boilerplate stripped. Prefer this over a raw HTTP fetch.",
    parameters: {
      type: "object",
      properties: {
        url: { type: "string", description: "Absolute http(s) URL." },
      },
      required: ["url"],
    },
  },
};

// One call to Lyrenth. No SDK involved, just HTTP.
async function readUrl(url: string): Promise<string> {
  const res = await fetch("https://api.lyrenth.com/v1/aidocument", {
    method: "POST",
    headers: {
      Authorization: `Bearer ${process.env.LYRENTH_API_KEY}`,
      "Content-Type": "application/json",
    },
    body: JSON.stringify({ url }),
  });
  const doc = await res.json();
  return doc.content.markdown;
}

const client = new OpenAI();
const MODEL = process.env.OPENAI_MODEL!;   // whichever model you already use
const messages: any[] = [
  { role: "user", content: "Summarize https://example.com/article" },
];

const first = await client.chat.completions.create({
  model: MODEL,
  messages,
  tools: [TOOL],
});
const message = first.choices[0].message;
messages.push(message);

for (const call of message.tool_calls ?? []) {
  const args = JSON.parse(call.function.arguments);
  messages.push({
    role: "tool",
    tool_call_id: call.id,
    content: await readUrl(args.url),
  });
}

const second = await client.chat.completions.create({
  model: MODEL,
  messages,
  tools: [TOOL],
});
console.log(second.choices[0].message.content);

Two calls to the model, as always with tool use: the first lets it ask for a page, the second lets it answer with the page in hand. Use whichever model you already use; nothing here depends on which one it is.

Prefer no glue code at all?

The MCP server gives any MCP client the same reader with no code to write.

The MCP server →API reference