Skip to main content

Documentation Index

Fetch the complete documentation index at: https://nvd-54.mintlify.app/llms.txt

Use this file to discover all available pages before exploring further.

Spider is the fastest crawler. It converts any website into pure HTML, markdown, metadata or text while enabling you to crawl with custom actions using AI.

​
概述

Spider allows you to use high performance proxies to prevent detection, caches AI actions, webhooks for crawling status, scheduled crawls etc… This guide shows how to crawl/scrape a website using Spider and loading the LLM-ready documents with SpiderLoader in LanghChain.

​
设置

Get your own Spider API key on spider.cloud.

​
用法

Here’s an example of how to use the SpiderLoader: Spider offers two scraping modes scrape and crawl. Scrape only gets the content of the URL provided while crawl gets the content of the URL provided and crawls deeper following subpages. 
import { SpiderLoader } from "@langchain/community/document_loaders/web/spider";

const loader = new SpiderLoader({
  url: "https://spider.cloud", // The URL to scrape
  apiKey: process.env.SPIDER_API_KEY, // Optional, defaults to `SPIDER_API_KEY` in your env.
  mode: "scrape", // The mode to run the crawler in. Can be "scrape" for single urls or "crawl" for deeper scraping following subpages
  // params: {
  //   // optional parameters based on Spider API docs
  //   // For API documentation, visit https://spider.cloud/docs/api
  // },
});

const docs = await loader.load();

​
Additional parameters

See the Spider documentation for all the available params.
将这些文档连接到 Claude、VSCode 等工具，通过 MCP 获取实时答案。
在 GitHub 上编辑此页面 或 提交 issue。