web-scraper-mcp
Advanced tools
MCP server for scraping images and text from websites with comprehensive web content extraction capabilities
一个基于 Model Context Protocol (MCP) 的网站爬虫服务器，支持爬取网站图片和文本内容。
npm install web-scraper-mcp
无需安装即可直接使用 web-scraper-mcp：
# 查看帮助信息
npx web-scraper-mcp --help
# 爬取网站图片
npx web-scraper-mcp scrape images https://example.com
# 爬取网站文本
npx web-scraper-mcp scrape text https://example.com
# 列出已爬取的图片
npx web-scraper-mcp list images
# 获取爬取状态
npx web-scraper-mcp status
这是使用 web-scraper-mcp 最简单的方式，特别适合偶尔使用或不想在本地安装的用户。
cd web-scraper-mcp
npm install
npm run build
npm start
或者跳过以上，直接添加到IDE(最推荐) 3. 在Claude Code中配置MCP服务器：
{
"mcpServers": {
"web-scraper-mcp": {
"command": "npx",
"args": [
"-y",
"web-scraper-mcp",
]
}
}
}
import { ImageScraper, TextScraper } from 'web-scraper-mcp';
// 图片爬取
const imageScraper = new ImageScraper('./images');
const images = await imageScraper.scrapeImagesFromUrl('https://example.com');
// 文本爬取
const textScraper = new TextScraper('./text');
const content = await textScraper.scrapeTextFromUrl('https://example.com');
爬取指定网站的所有图片并保存到本地。
参数:
url (string, 必需): 要爬取图片的网站URLoutputDir (string, 可选): 图片保存目录，默认为 ./scraped-imagesmaxConcurrent (number, 可选): 并发下载数量，默认为5示例:
{
"name": "scrape_images",
"arguments": {
"url": "https://example.com",
"outputDir": "./my-images",
"maxConcurrent": 10
}
}
爬取网站的文本内容并保存为Markdown文件。
参数:
url (string, 必需): 要爬取文本的网站URLoutputDir (string, 可选): 文本保存目录，默认为 ./scraped-text示例:
{
"name": "scrape_text",
"arguments": {
"url": "https://example.com",
"outputDir": "./my-texts"
}
}
列出所有已下载的图片信息。
参数:
outputDir (string, 可选): 图片目录路径，默认为 ./scraped-images示例:
{
"name": "list_images",
"arguments": {
"outputDir": "./my-images"
}
}
列出所有已提取的文本文件。
参数:
outputDir (string, 可选): 文本目录路径，默认为 ./scraped-text示例:
{
"name": "list_texts",
"arguments": {
"outputDir": "./my-texts"
}
}
清理所有下载的图片文件。
参数:
outputDir (string, 可选): 要清理的图片目录路径，默认为 ./scraped-images示例:
{
"name": "cleanup_images",
"arguments": {
"outputDir": "./my-images"
}
}
清理所有提取的文本文件。
参数:
outputDir (string, 可选): 要清理的文本目录路径，默认为 ./scraped-text示例:
{
"name": "cleanup_texts",
"arguments": {
"outputDir": "./my-texts"
}
}
获取爬虫状态信息。
参数:
outputDir (string, 可选): 输出目录路径示例:
{
"name": "get_scraping_status",
"arguments": {
"outputDir": "./output"
}
}
web-scraper-mcp 提供了命令行界面，可以通过 npx 或安装后使用。
启动 MCP 服务器模式。
npx web-scraper-mcp start
爬取网站内容。
# 爬取图片
npx web-scraper-mcp scrape images https://example.com
# 爬取文本
npx web-scraper-mcp scrape text https://example.com
# 指定输出目录
npx web-scraper-mcp scrape images https://example.com -o ./my-images
# 设置并发数（仅对图片有效）
npx web-scraper-mcp scrape images https://example.com -c 10
列出已爬取的内容。
# 列出图片
npx web-scraper-mcp list images
# 列出文本
npx web-scraper-mcp list texts
# 指定目录
npx web-scraper-mcp list images -o ./my-images
获取爬取状态。
# 获取默认状态
npx web-scraper-mcp status
# 指定目录
npx web-scraper-mcp status -o ./output
清理爬取的内容。
# 清理图片
npx web-scraper-mcp cleanup images
# 清理文本
npx web-scraper-mcp cleanup texts
# 指定目录
npx web-scraper-mcp cleanup images -o ./my-images
// 1. 爬取图片
const imageResult = await server.requestToolCall({
name: 'scrape_images',
arguments: {
url: 'https://example.com',
outputDir: './example-images'
}
});
// 2. 爬取文本
const textResult = await server.requestToolCall({
name: 'scrape_text',
arguments: {
url: 'https://example.com',
outputDir: './example-texts'
}
});
// 3. 检查状态
const status = await server.requestToolCall({
name: 'get_scraping_status'
});
const websites = [
'https://example1.com',
'https://example2.com',
'https://example3.com'
];
for (const website of websites) {
// 爬取图片
await server.requestToolCall({
name: 'scrape_images',
arguments: {
url: website,
outputDir: `./images-${new Date().toISOString().split('T')[0]}`
}
});
// 爬取文本
await server.requestToolCall({
name: 'scrape_text',
arguments: {
url: website,
outputDir: `./texts-${new Date().toISOString().split('T')[0]}`
}
});
}
SCRAPER_OUTPUT_DIR: 默认输出目录SCRAPER_MAX_CONCURRENT: 默认并发数量SCRAPER_USER_AGENT: 自定义User-AgentSCRAPER_TIMEOUT: 请求超时时间（毫秒）创建 config.json 文件：
{
"imageScraper": {
"outputDir": "./scraped-images",
"maxConcurrent": 5,
"allowedFormats": ["jpg", "png", "gif", "webp", "svg"]
},
"textScraper": {
"outputDir": "./scraped-text",
"excludeSelectors": ["script", "style", "nav", "footer"]
}
}
web-scraper-mcp/
├── src/
│ ├── scrapers/
│ │ ├── imageScraper.ts # 图片爬虫
│ │ └── textScraper.ts # 文本爬虫
│ ├── utils/
│ │ ├── validators.ts # 输入验证
│ │ └── logger.ts # 日志工具
│ └── index.ts # MCP服务器入口
├── package.json
├── tsconfig.json
├── README.md
└── dist/ # 编译输出
npm test
npm run build
npm run dev
# 运行 CLI 脚本（TypeScript）
npm run dev:cli -- --help
# 爬取图片（开发模式）
npm run dev:cli -- scrape images https://example.com
# 爬取文本（开发模式）
npm run dev:cli -- scrape text https://example.com
网络错误
权限错误
内存溢出
超时错误
启用调试日志：
import { Logger } from './utils/logger';
Logger.setLevel('DEBUG');
MIT License
git checkout -b feature/AmazingFeature)git commit -m 'Add some AmazingFeature')git push origin feature/AmazingFeature)FAQs
MCP server for scraping images and text from websites with comprehensive web content extraction capabilities
The npm package web-scraper-mcp receives a total of 1 weekly downloads. As such, web-scraper-mcp popularity was classified as not popular.
We found that web-scraper-mcp demonstrated a healthy version release cadence and project activity because the last version was released less than a year ago. It has 1 open source maintainer collaborating on the project.
Did you know?
Socket for GitHub automatically highlights issues in each pull request and monitors the health of all your open source dependencies. Discover the contents of your packages and block harmful activity before you install or update your dependencies.
Security News
/Product
Socket Firewall Free is now bundled into Docker Hardened Images, adding build-time and dependency-install supply chain protection on top of hardened base images for Node.js, Python, and Rust.
Security News
Season’s greetings from Socket, and here’s to a calm end of year: clean dependencies, boring pipelines, no surprises.
Research
/Security News
Impostor NuGet package Tracer.Fody.NLog typosquats Tracer.Fody and its author, using homoglyph tricks, and exfiltrates Stratis wallet JSON/passwords to a Russian IP address.