> ## Documentation Index
> Fetch the complete documentation index at: https://nvd-54.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.
# Recursive URL 集成
> 使用 LangChain Python 集成 Recursive URL 文档加载器。
The `RecursiveUrlLoader` lets you recursively scrape all child links from a root URL and parse them into Documents.
## 概述
### 集成详情
| Class | Package | Local | Serializable | [JS support](https://js.langchain.com/docs/integrations/document_loaders/web_loaders/recursive_url_loader/) |
| :------------------------------------------------------------------------------------------------------------------------------------------ | :---------------------------------------------------------------------------------- | :---: | :----------: | :---------------------------------------------------------------------------------------------------------: |
| [`RecursiveUrlLoader`](https://reference.langchain.com/python/langchain-community/document_loaders/recursive_url_loader/RecursiveUrlLoader) | [`langchain-community`](https://reference.langchain.com/python/langchain-community) | ✅ | ❌ | ✅ |
### 加载器特性
| Source | Document Lazy Loading | Native Async Support |
| :------------------: | :-------------------: | :------------------: |
| `RecursiveUrlLoader` | ✅ | ❌ |
## 设置
### 凭证
No credentials are required to use the `RecursiveUrlLoader`.
### 安装
The `RecursiveUrlLoader` lives in the `langchain-community` package. There's no other required packages, though you will get richer default Document metadata if you have \`\`beautifulsoup4\` installed as well.
```python theme={"theme":{"light":"catppuccin-latte","dark":"catppuccin-mocha"}}
pip install -qU langchain-community beautifulsoup4 lxml
```
## Instantiation
Now we can instantiate our document loader object and load Documents:
```python theme={"theme":{"light":"catppuccin-latte","dark":"catppuccin-mocha"}}
from langchain_community.document_loaders import RecursiveUrlLoader
loader = RecursiveUrlLoader(
"https://docs.python.org/3.9/",
# max_depth=2,
# use_async=False,
# extractor=None,
# metadata_extractor=None,
# exclude_dirs=(),
# timeout=10,
# check_response_status=True,
# continue_on_failure=True,
# prevent_outside=True,
# base_url=None,
# ...
)
```
## 加载
Use `.load()` to synchronously load into memory all Documents, with one
Document per visited URL. Starting from the initial URL, we recurse through
all linked URLs up to the specified max\_depth.
Let's run through a basic example of how to use the `RecursiveUrlLoader` on the [Python 3.9 Documentation](https://docs.python.org/3.9/).
```python theme={"theme":{"light":"catppuccin-latte","dark":"catppuccin-mocha"}}
docs = loader.load()
docs[0].metadata
```
```text theme={"theme":{"light":"catppuccin-latte","dark":"catppuccin-mocha"}}
/Users/bagatur/.pyenv/versions/3.9.1/lib/python3.9/html/parser.py:170: XMLParsedAsHTMLWarning: It looks like you're parsing an XML document using an HTML parser. If this really is an HTML document (maybe it's XHTML?), you can ignore or filter this warning. If it's XML, you should know that using an XML parser will be more reliable. To parse this document as XML, make sure you have the lxml package installed, and pass the keyword argument `features="xml"` into the BeautifulSoup constructor.
k = self.parse_starttag(i)
```
```text theme={"theme":{"light":"catppuccin-latte","dark":"catppuccin-mocha"}}
{'source': 'https://docs.python.org/3.9/',
'content_type': 'text/html',
'title': '3.9.19 Documentation',
'language': None}
```
Great! The first document looks like the root page we started from. Let's look at the metadata of the next document
```python theme={"theme":{"light":"catppuccin-latte","dark":"catppuccin-mocha"}}
docs[1].metadata
```
```text theme={"theme":{"light":"catppuccin-latte","dark":"catppuccin-mocha"}}
{'source': 'https://docs.python.org/3.9/using/index.html',
'content_type': 'text/html',
'title': 'Python Setup and Usage — Python 3.9.19 documentation',
'language': None}
```
That url looks like a child of our root page, which is great! Let's move on from metadata to examine the content of one of our documents
```python theme={"theme":{"light":"catppuccin-latte","dark":"catppuccin-mocha"}}
print(docs[0].page_content[:300])
```
```text theme={"theme":{"light":"catppuccin-latte","dark":"catppuccin-mocha"}}
3.9.19 Documentation
= 10:
# do some paged operation, e.g.
# index.upsert(page)
pages = []
```
```text theme={"theme":{"light":"catppuccin-latte","dark":"catppuccin-mocha"}}
/var/folders/4j/2rz3865x6qg07tx43146py8h0000gn/T/ipykernel_73962/2110507528.py:6: XMLParsedAsHTMLWarning: It looks like you're parsing an XML document using an HTML parser. If this really is an HTML document (maybe it's XHTML?), you can ignore or filter this warning. If it's XML, you should know that using an XML parser will be more reliable. To parse this document as XML, make sure you have the lxml package installed, and pass the keyword argument `features="xml"` into the BeautifulSoup constructor.
soup = BeautifulSoup(html, "lxml")
```
在此示例中，我们在任何时候内存中加载的文档不会超过 10 个。
## Adding an extractor
默认情况下，the loader sets the raw HTML from each link as the Document page content. To parse this HTML into a more human/LLM-friendly format you can pass in a custom `extractor` method:
```python theme={"theme":{"light":"catppuccin-latte","dark":"catppuccin-mocha"}}
import re
from bs4 import BeautifulSoup
def bs4_extractor(html: str) -> str:
soup = BeautifulSoup(html, "lxml")
return re.sub(r"\n\n+", "\n\n", soup.text).strip()
loader = RecursiveUrlLoader("https://docs.python.org/3.9/", extractor=bs4_extractor)
docs = loader.load()
print(docs[0].page_content[:200])
```
```text theme={"theme":{"light":"catppuccin-latte","dark":"catppuccin-mocha"}}
/var/folders/td/vzm913rx77x21csd90g63_7c0000gn/T/ipykernel_10935/1083427287.py:6: XMLParsedAsHTMLWarning: It looks like you're parsing an XML document using an HTML parser. If this really is an HTML document (maybe it's XHTML?), you can ignore or filter this warning. If it's XML, you should know that using an XML parser will be more reliable. To parse this document as XML, make sure you have the lxml package installed, and pass the keyword argument `features="xml"` into the BeautifulSoup constructor.
soup = BeautifulSoup(html, "lxml")
/Users/isaachershenson/.pyenv/versions/3.11.9/lib/python3.11/html/parser.py:170: XMLParsedAsHTMLWarning: It looks like you're parsing an XML document using an HTML parser. If this really is an HTML document (maybe it's XHTML?), you can ignore or filter this warning. If it's XML, you should know that using an XML parser will be more reliable. To parse this document as XML, make sure you have the lxml package installed, and pass the keyword argument `features="xml"` into the BeautifulSoup constructor.
k = self.parse_starttag(i)
```
```text theme={"theme":{"light":"catppuccin-latte","dark":"catppuccin-mocha"}}
3.9.19 Documentation
Download
Download these documents
Docs by version
Python 3.13 (in development)
Python 3.12 (stable)
Python 3.11 (security-fixes)
Python 3.10 (security-fixes)
Python 3.9 (securit
```
This looks much nicer!
You can similarly pass in a `metadata_extractor` to customize how Document metadata is extracted from the HTTP response. See the [API reference](https://reference.langchain.com/python/langchain-community/document_loaders/recursive_url_loader/RecursiveUrlLoader) for more on this.
***
## API 参考
These examples show just a few of the ways in which you can modify the default `RecursiveUrlLoader`, but there are many more modifications that can be made to best fit your use case. Using the parameters `link_regex` and `exclude_dirs` can help you filter out unwanted URLs, `aload()` and `alazy_load()` can be used for asynchronous loading, and more.
For detailed information on configuring and calling the `RecursiveUrlLoader`, please see the [API reference](https://reference.langchain.com/python/langchain-community/document_loaders/recursive_url_loader/RecursiveUrlLoader).
***