快速抓取网页文章标题和内容，适合node.js爬虫使用，服务于ElasticSearch。
npm install read-art
read(html/uri [, options], callback)
read-art is designed to be the simplest way possible to make web-article scrape, it supports the definitions such as:
callback(error, article, options)See test or examples folder for a complete example
Just try it
var read = require('read-art');
// read from google could be
read('http://google.com', { charset: 'utf8' }, function(err, art, options){
if(err){
throw err;
}
var title = art.title, // title of article
content = art.content, // content of article
html = art.html; // whole original innerHTML
});
// or
read({ uri: 'http://google.com', charset: 'utf8' }, function(err, art, options){
});
// what about html?
read('<title>node-art</title><body><div><p>hello, read-art!</p></div></body>', { charset: 'utf8' }, function(err, art, options){
});
// of course could be
read({ uri: '<title>node-art</title><body><div><p>hello, read-art!</p></div></body>', charset: 'utf8' }, function(err, art, options){
});
CAUTION: Title must be wrapped in a <title> tag and content must be wrapped in a <body> tag.
You can set different types to wrap the output
Returns the inner text of article content(strip html tags), e.g.:
read('http://example.com', {
output: 'text'
}, function(err, art){
// art.content will be formatted as TEXT
});
and the full usage
read('http://example.com', {
output: {
type: 'text',
stripSpaces: true
}
}, function(err, art){
// art.content will be formatted as TEXT
});
Returns the inner HTML of article content, e.g.:
read('http://example.com', {
output: 'html'
}, function(err, art){
// art.content will be formatted as HTML
});
and the full usage
read('http://example.com', {
output: {
type: 'html',
stripSpaces: true
}
}, function(err, art){
// art.content will be formatted as HTML
});
Returns the restful result of article content, e.g.:
read('http://example.com', {
output: 'json'
}, function(err, art){
// art.content will be formatted as JSON
});
and the full usage
read('http://example.com', {
output: {
type: 'json',
stripSpaces: true
}
}, function(err, art){
// art.content will be formatted as JSON
});
The art.content will be an Array such as:
[
{ "type": "img", "value": "http://example.com/jpg/site1/20140519/00188b1996f214e3a25417.jpg" },
{ "type": "text", "value": "TEXT goes here..." }
]
There only two types were supported now: img and text
As you see, the output could be defined in two ways:
ϟ Blazingly fast: read-art is based on cheerio(cheerio is about 8x faster than JSDOM), and the article marking strategy actualized by RegExp, it's supper fast and cost less memory.
❤ Hit the target: The bonus algorithm make spider or scraper more easier to grab the article title & content.
↵ Fetch HTML: If you only wanna fetch html body from server, req-fast is amazing, it supports:
refrain from the crazy messy codes
read('http://game.163.com/14/0506/10/9RI8M9AO00314SDA.html', {
charset: 'gbk'
}, function(err, art){
// ...
});
generate agent to simulate browsers
read('http://example.com', {
agent: true // true as default
}, function(err, art){
// ...
});
use proxy
read('http://example.com', {
proxy: {
host: 'http://myproxy.com/',
port: 8081,
proxyAuth: 'user:password'
}
}, function(err, art){
// ...
});
and more is amazing, it supports...
cd to the read-art directory and install all the dependencies library.
npm test
luin/node-readability is really good, lots of hit points, easy to use, but the problem is - it's too slow, make me mad. it was based on JSDOM, as you known, the HTML must be strict, you can not make any mistake, e.g.:
<P>Paragraphs</p>
<p>My book name is <read-art></p>
<div><p>Hey, dude!</div>
all above will cause hiberarchy errors. and otherwise, JSDOM is a memory killer.
bndr/node-read is amazing, and i've worked on this for a while, but it's hard to communicate with Vadim(we are in a different timezone), and we have very different ideas. so i decided to write it on my own.
Copyright 2014 Tjatse
Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.
FAQs
Scrape/Crawl article from any site automatically. Make any web page readable, no matter Chinese or English.
We found that read-art demonstrated a not healthy version release cadence and project activity because the last version was released a year ago. It has 1 open source maintainer collaborating on the project.
Did you know?
Socket for GitHub automatically highlights issues in each pull request and monitors the health of all your open source dependencies. Discover the contents of your packages and block harmful activity before you install or update your dependencies.
Research
Security News
Malicious PyPI package ‘set-utils’ steals Ethereum private keys by exfiltrating them through blockchain transactions via the Polygon RPC.
Research
Security News
Malicious Go packages are impersonating popular libraries to install hidden loader malware on Linux and macOS, targeting developers with obfuscated payloads.
Security News
Bybit's $1.46B hack by North Korea's Lazarus Group pushes 2025 crypto losses to $1.6B in just two months, already surpassing all of 2024's $1.49B total.