LangChain

LangChain Document Loaders: Ingesting PDFs, Web Pages and More

Pramod Dutta · Jun 30, 2026 · 16 min read

Why Document Loaders Are the Most Underrated Part of a RAG Pipeline

Every RAG tutorial spends most of its time on embeddings, vector stores, and retrieval chains. Almost none of them spend real time on the step that happens before any of that: getting your raw data into a shape LangChain can actually work with. That's the job of document loaders, and if you get this step wrong, everything downstream inherits the damage. Bad chunking, broken metadata, missing page numbers, garbled tables — nearly all of it traces back to a loader that was picked without understanding what it actually does.

Document loaders in LangChain are the components responsible for reading data from a source — a PDF file, a website, a CSV, a Notion export, an S3 bucket — and converting it into a standard Document object that the rest of the framework understands. A Document is a simple structure with two things: page_content (the actual text) and metadata (a dictionary of extra information like source path, page number, or title). Every text splitter, every retriever, every vector store in LangChain expects data in this shape.

This article walks through the loaders you'll use most often in production: PDFs, web pages, CSVs, and directories of mixed files. We'll look at what each loader does under the hood, where it breaks, and how to combine loaders into a single ingestion pipeline you can actually trust. If you're building anything more serious than a toy demo, this is the part worth getting right first.

The Document Object: What You're Actually Loading Into

Before touching a single loader, it helps to understand the target format. Every loader in LangChain, regardless of source, returns a list of Document objects.

from langchain_core.documents import Document

doc = Document(
    page_content="LangChain is a framework for building LLM applications.",
    metadata={"source": "intro.txt", "page": 1}
)

print(doc.page_content)
print(doc.metadata)

That's it. No matter how exotic the source — a PDF with embedded images, a JavaScript-rendered web page, a nested JSON file — the loader's entire job is to flatten that source into a list of these objects. This uniformity is what lets you swap a PDF loader for a web loader without touching your splitter, embedding, or retrieval code.

The metadata field is where a lot of production quality gets decided. A loader that only fills page_content and leaves metadata mostly empty is throwing away information you'll want later — for citations, for filtering retrieval results, for debugging why a chunk got retrieved at all. When evaluating a loader, check what it puts in metadata by default, not just whether it extracts text correctly.

Loading PDFs: The Loader Choice That Matters Most

PDFs are the single most common document type in enterprise RAG systems — contracts, reports, manuals, research papers — and they're also the messiest to parse. A PDF isn't really "text with pages"; it's a page-description format that happens to contain text, images, and layout instructions all mixed together. This is why LangChain ships several different PDF loaders, each making different tradeoffs.

The simplest and most commonly used is PyPDFLoader, built on the pypdf library:

from langchain_community.document_loaders import PyPDFLoader

loader = PyPDFLoader("company_handbook.pdf")
pages = loader.load()

print(f"Loaded {len(pages)} pages")
print(pages[0].metadata)
# {'source': 'company_handbook.pdf', 'page': 0}

print(pages[0].page_content[:300])

Notice that PyPDFLoader returns one Document per page, not one Document for the whole file. This matters a lot for chunking strategy — if your splitter later merges pages back together, you lose the natural page boundary, which is often a legitimate semantic boundary in structured documents.

For PDFs with complex layouts — multi-column text, embedded tables, scanned pages — PyPDFLoader will often produce garbled or out-of-order text, because pypdf reads text in the order it appears in the PDF's internal structure, which doesn't always match visual reading order. In these cases, UnstructuredPDFLoader (backed by the unstructured library) does a better job because it attempts to reconstruct document structure — headings, list items, table cells — rather than just extracting a flat text stream:

from langchain_community.document_loaders import UnstructuredPDFLoader

loader = UnstructuredPDFLoader(
    "quarterly_report.pdf",
    mode="elements"  # returns one Document per structural element
)
elements = loader.load()

for el in elements[:5]:
    print(el.metadata.get("category"), "->", el.page_content[:80])

The mode="elements" option is worth calling out specifically. Instead of one blob of text per page, you get one Document per detected element — titles, narrative text, list items, table rows — each tagged with a category in its metadata. This is enormously useful if you want to, say, filter out tables before chunking, or give titles extra weight in your embedding strategy.

If you're dealing with scanned PDFs (images of text, no embedded text layer), neither of the above will extract anything meaningful, because there's no text to extract — you need OCR. UnstructuredPDFLoader can route through OCR automatically for image-based pages if the underlying dependencies are installed, but for heavy scanned-document workloads, it's often more reliable to run a dedicated OCR step first and feed the resulting text into LangChain as plain-text documents.

A practical rule of thumb: start with PyPDFLoader because it's fast and dependency-light. Switch to UnstructuredPDFLoader when you notice broken sentence ordering, missing table content, or when you need element-level metadata for smarter chunking.

Loading Web Pages: HTML Is Not Text

Web pages introduce a different problem: the "content" you want is buried inside navigation bars, ads, cookie banners, footers, and script tags. LangChain's WebBaseLoader is the standard entry point here, built on requests and BeautifulSoup:

from langchain_community.document_loaders import WebBaseLoader

loader = WebBaseLoader("https://teachyou.ai/blog/langchain-basics")
docs = loader.load()

print(docs[0].metadata)
# {'source': 'https://teachyou.ai/blog/langchain-basics', 'title': '...', 'language': 'en'}
print(len(docs[0].page_content))

By default, WebBaseLoader pulls the entire visible text of the page, which usually includes a lot of navigation and boilerplate noise you don't want in your vector store. You can narrow this down with BeautifulSoup's SoupStrainer, which tells the parser to only look inside specific tags:

import bs4
from langchain_community.document_loaders import WebBaseLoader

loader = WebBaseLoader(
    web_paths=["https://teachyou.ai/blog/langchain-basics"],
    bs_kwargs={
        "parse_only": bs4.SoupStrainer(class_=("post-content", "post-title"))
    }
)
docs = loader.load()

WebBaseLoader also accepts a list of URLs, letting you batch-load an entire set of pages in one call:

urls = [
    "https://teachyou.ai/blog/langchain-basics",
    "https://teachyou.ai/blog/vector-databases-explained",
    "https://teachyou.ai/blog/prompt-engineering-guide",
]

loader = WebBaseLoader(urls)
docs = loader.load()
print(f"Loaded {len(docs)} pages")

One limitation worth flagging clearly: WebBaseLoader does a plain HTTP GET and parses the returned HTML. It does not execute JavaScript. If a site renders its content client-side (a lot of modern React or Vue sites do), you'll get an empty shell instead of the article text. For those sites, LangChain offers loaders backed by headless browsers — PlaywrightURLLoader and SeleniumURLLoader — which actually render the page before extracting text:

from langchain_community.document_loaders import PlaywrightURLLoader

urls = ["https://example-spa.com/article/123"]
loader = PlaywrightURLLoader(urls=urls, remove_selectors=["header", "footer", "nav"])
docs = loader.load()

The remove_selectors argument is a nice touch — it strips out common boilerplate elements before extraction, similar in spirit to the SoupStrainer trick above but working on the rendered DOM instead of raw HTML.

If you need to crawl an entire site rather than load a fixed list of URLs, RecursiveUrlLoader follows links up to a configurable depth:

from langchain_community.document_loaders import RecursiveUrlLoader
from bs4 import BeautifulSoup

def extract_text(html):
    return BeautifulSoup(html, "html.parser").get_text()

loader = RecursiveUrlLoader(
    url="https://teachyou.ai/docs",
    max_depth=2,
    extractor=extract_text,
)
docs = loader.load()

Be deliberate with max_depth here. It's easy to accidentally crawl thousands of pages you didn't intend to, hammering a server and blowing up your ingestion time and API bill for embeddings you'll never actually query.

Loading CSVs and Structured Data

CSVs show up constantly in enterprise data — customer records, product catalogs, support ticket logs — and LangChain's CSVLoader handles the common case cleanly, turning each row into its own Document:

from langchain_community.document_loaders import CSVLoader

loader = CSVLoader(
    file_path="support_tickets.csv",
    csv_args={"delimiter": ",", "quotechar": '"'},
)
docs = loader.load()

print(docs[0].page_content)
# ticket_id: 1042
# subject: Login fails after password reset
# priority: high
# status: open

print(docs[0].metadata)
# {'source': 'support_tickets.csv', 'row': 0}

Each row becomes a text block with "column_name: value" on every line, which turns out to be a reasonably good default representation for embedding — it preserves column semantics without you having to write custom formatting logic.

If you only care about specific columns for the embedded content but still want the rest available as metadata, you can be explicit:

loader = CSVLoader(
    file_path="support_tickets.csv",
    source_column="subject",
    metadata_columns=["ticket_id", "priority", "status"],
)
docs = loader.load()

This pattern — narrow page_content for embedding relevance, rich metadata for filtering — is one of the more important habits to build early. If you dump every column into page_content, you dilute the embedding with irrelevant tokens (an internal ticket ID doesn't help semantic search), but if you drop those fields entirely, you lose the ability to filter retrieval results by priority or status later.

Loading Directories of Mixed Files

Real projects rarely have one file type. A knowledge base might have PDFs, Word docs, Markdown files, and plain text notes all sitting in the same folder. DirectoryLoader handles this by pairing a glob pattern with a loader class for each file type:

from langchain_community.document_loaders import DirectoryLoader, TextLoader, PyPDFLoader

pdf_loader = DirectoryLoader(
    "knowledge_base/",
    glob="**/*.pdf",
    loader_cls=PyPDFLoader,
    show_progress=True,
)

text_loader = DirectoryLoader(
    "knowledge_base/",
    glob="**/*.txt",
    loader_cls=TextLoader,
    show_progress=True,
)

all_docs = pdf_loader.load() + text_loader.load()
print(f"Total documents loaded: {len(all_docs)}")

show_progress=True is worth turning on for anything beyond a handful of files — directory loads over large knowledge bases can silently run for minutes, and without a progress bar you have no idea whether it's working or hung.

DirectoryLoader also accepts a use_multithreading flag, which parallelizes loading across files. This helps a lot when you're dealing with I/O-bound loaders like WebBaseLoader or when reading from network-mounted storage, though it won't do much for a loader that's already CPU-bound on parsing.

loader = DirectoryLoader(
    "knowledge_base/",
    glob="**/*.pdf",
    loader_cls=PyPDFLoader,
    use_multithreading=True,
    max_concurrency=4,
)
docs = loader.load()

Lazy Loading and Why It Matters at Scale

Most of the examples above call .load(), which reads the entire source into memory and returns a fully materialized list of Document objects. That's fine for a single PDF or a handful of web pages, but it falls apart when you're ingesting a directory with tens of thousands of files or a CSV with millions of rows. Loading everything into memory at once before you've even started chunking is a good way to run out of RAM on a large ingestion job.

Most LangChain loaders also expose a .lazy_load() method, which returns a generator instead of a list. Documents are produced one at a time, on demand, so you can stream them straight into your splitter and vector store without ever holding the full corpus in memory:

from langchain_community.document_loaders import DirectoryLoader, PyPDFLoader

loader = DirectoryLoader(
    "knowledge_base/pdfs",
    glob="**/*.pdf",
    loader_cls=PyPDFLoader,
)

for doc in loader.lazy_load():
    # process one Document at a time: split, embed, upsert
    process_and_store(doc)

This pattern matters even more once you wire loaders into a text splitter and vector store, since you can process-and-discard each document instead of accumulating everything before splitting even starts:

from langchain_text_splitters import RecursiveCharacterTextSplitter

splitter = RecursiveCharacterTextSplitter(chunk_size=1000, chunk_overlap=150)

def process_and_store(doc):
    chunks = splitter.split_documents([doc])
    vectorstore.add_documents(chunks)

For loaders that don't naturally support laziness well (some web loaders fetch everything up front regardless), you can still batch things manually by chunking your input list of URLs or file paths and instantiating a fresh loader per batch. The point is the same either way: don't let ingestion memory scale linearly with corpus size if you can avoid it.

Loading Notion, Confluence, and Other SaaS Knowledge Bases

A large share of real internal knowledge doesn't live as files at all — it lives in Notion workspaces, Confluence spaces, and similar SaaS tools. LangChain ships integrations for these as well, and the pattern is consistent with everything covered so far: you authenticate, point the loader at a space or database, and get back a list of Document objects.

from langchain_community.document_loaders import ConfluenceLoader

loader = ConfluenceLoader(
    url="https://your-domain.atlassian.net/wiki",
    username="you@company.com",
    api_key="your-api-token",
    space_key="ENG",
    limit=50,
)
docs = loader.load()

print(docs[0].metadata)
# includes page id, title, source url, and space info

The practical challenge with SaaS loaders isn't the API call itself — it's rate limits and pagination. limit controls page size per request, and most of these loaders handle pagination internally, but on large spaces you'll still want to be deliberate about how often you re-sync. A common pattern is to run a full ingestion once, then only pull pages modified since the last sync on subsequent runs, using whatever "last modified" metadata the source API exposes. This turns ingestion from a slow full rebuild into a fast incremental update.

Notion works almost identically, either through LangChain's NotionDBLoader for a specific database or NotionDirectoryLoader if you've exported your workspace to a local folder of Markdown files first:

from langchain_community.document_loaders import NotionDirectoryLoader

loader = NotionDirectoryLoader("notion_export/")
docs = loader.load()

The exported-folder approach is often more reliable for one-off ingestion jobs since it sidesteps API rate limits entirely, at the cost of not being able to do incremental syncs without re-exporting.

Handling Loader Failures Gracefully

Production ingestion pipelines fail constantly, and not in dramatic ways — a single malformed PDF, one unreachable URL, a CSV row with a stray null byte, and the whole batch job can come to a halt if you haven't planned for it. When you're looping over hundreds or thousands of sources, wrap each individual load in error handling rather than letting one bad file take down the entire run:

import logging

logging.basicConfig(level=logging.INFO)
logger = logging.getLogger("ingestion")

def safe_load(loader_cls, source, **kwargs):
    try:
        loader = loader_cls(source, **kwargs)
        return loader.load()
    except Exception as e:
        logger.warning(f"Failed to load {source}: {e}")
        return []

all_docs = []
for path in pdf_paths:
    all_docs.extend(safe_load(PyPDFLoader, path))

print(f"Successfully loaded {len(all_docs)} documents")

This is a small amount of code, but it's the difference between an ingestion job that quietly skips three broken files out of five thousand and logs exactly which ones, versus a job that crashes at 2 a.m. and leaves you debugging a stack trace with no idea which file caused it. Log the failures, keep a running count, and review the failure log after every ingestion run — a growing number of failures on the same source type usually means something upstream changed (a site redesign, a new PDF export format) rather than a one-off fluke.

It's also worth validating documents *after* loading, not just catching load-time exceptions. An empty page_content string is a common silent failure — the loader ran without error, but extracted nothing, often because a PDF was scanned with no text layer, or a web page needed JavaScript rendering that a plain HTTP loader can't do.

def validate_docs(docs, min_length=20):
    valid, empty = [], []
    for doc in docs:
        if len(doc.page_content.strip()) >= min_length:
            valid.append(doc)
        else:
            empty.append(doc.metadata.get("source", "unknown"))
    if empty:
        logger.warning(f"{len(empty)} documents had little or no extracted text: {empty}")
    return valid

docs = validate_docs(all_docs)

Catching this at ingestion time is far cheaper than discovering it weeks later when a user asks a question about a document your pipeline "loaded" but never actually read.

Building a Combined Ingestion Pipeline

In a real project, you rarely call one loader in isolation — you build a small ingestion function that normalizes everything into one list of Document objects, ready for splitting and embedding. Here's a pattern that combines several loaders and tags each source type in metadata for later debugging:

from langchain_community.document_loaders import (
    PyPDFLoader,
    WebBaseLoader,
    CSVLoader,
    DirectoryLoader,
    TextLoader,
)

def load_knowledge_base(pdf_dir, urls, csv_path):
    all_docs = []

    pdf_loader = DirectoryLoader(pdf_dir, glob="**/*.pdf", loader_cls=PyPDFLoader)
    pdf_docs = pdf_loader.load()
    for doc in pdf_docs:
        doc.metadata["doc_type"] = "pdf"
    all_docs.extend(pdf_docs)

    web_loader = WebBaseLoader(urls)
    web_docs = web_loader.load()
    for doc in web_docs:
        doc.metadata["doc_type"] = "web"
    all_docs.extend(web_docs)

    csv_loader = CSVLoader(file_path=csv_path)
    csv_docs = csv_loader.load()
    for doc in csv_docs:
        doc.metadata["doc_type"] = "csv"
    all_docs.extend(csv_docs)

    return all_docs

docs = load_knowledge_base(
    pdf_dir="knowledge_base/pdfs",
    urls=["https://teachyou.ai/blog/langchain-basics"],
    csv_path="knowledge_base/tickets.csv",
)

print(f"Total documents: {len(docs)}")
print(f"Source types: {set(d.metadata['doc_type'] for d in docs)}")

Tagging doc_type like this pays off later — when retrieval returns a weird or low-quality result, you can immediately tell whether the problem lives in your PDF parsing, your web scraping, or your CSV handling, instead of debugging blind.

Common Mistakes That Break Downstream Retrieval

A few loader-stage mistakes account for most of the retrieval quality issues people report when their RAG pipeline "just isn't finding the right answer."

Loading before checking encoding. Text files with non-UTF-8 encoding will either throw errors or silently produce garbled text with TextLoader. Pass encoding="utf-8" explicitly, or better, use autodetect_encoding=True if you're unsure of the source files.
Ignoring metadata until it's too late. If you don't capture page numbers, source URLs, or row identifiers at load time, you can't reconstruct them after chunking. Metadata should be treated as a first-class output of the loading stage, not an afterthought.
Using the wrong PDF loader for the document type. A financial report full of tables needs UnstructuredPDFLoader in element mode, not the fast-but-flat PyPDFLoader. Test both on a representative sample before committing to one for an entire pipeline.
Crawling too aggressively with `RecursiveUrlLoader`. Without a sane max_depth and without respecting robots.txt, you can end up ingesting thousands of irrelevant pages, inflating both your storage and embedding costs.
Treating one CSV row as always the right unit. For very short rows, consider grouping several rows into one Document to give the embedding model more context; for very long rows, you may want to split a single row further downstream.
Not deduplicating across loaders. If your web pages are also available as PDFs in your directory, you may end up embedding the same content twice, which skews retrieval toward duplicated material.

None of these are exotic problems — they're all things you'll hit the first time you point LangChain at a real, messy dataset instead of a clean tutorial folder.

Wrapping Up

Document loaders look like plumbing, and in a sense, they are — but plumbing is exactly the kind of thing that causes silent, hard-to-diagnose failures when it's done carelessly. Getting PyPDFLoader versus UnstructuredPDFLoader right, stripping boilerplate out of WebBaseLoader results, being deliberate about what goes into CSV page_content versus metadata, and combining everything through DirectoryLoader with clear source tagging — these choices determine whether your retrieval step has clean material to work with or a pile of noisy, half-broken text.

The pattern to take away: pick the loader that matches your source's actual complexity (don't reach for UnstructuredPDFLoader on simple text-only PDFs, and don't settle for PyPDFLoader on table-heavy reports), be intentional about metadata from the very first line of code, and always build a small combined ingestion function rather than scattering loader calls across your codebase. Do this and every later stage — splitting, embedding, retrieval, generation — gets noticeably easier to debug and improve.

If you want to go deeper into loaders, splitters, retrievers, and full RAG architecture with hands-on projects, our LangChain Tutorial 2026 course on teachyou.ai walks through all of this step by step, from raw document ingestion to production-grade retrieval pipelines.