快速抓取网页文章标题和内容，适合node.js爬虫使用，服务于ElasticSearch。
In my case, the indexed data is about 400 thousand per day, 10 million per month, and the maximize indexing speed is 35/second, the memory cost are limited under 100 megabytes.
Pictures don't lie:
npm install read-art
read(html/uri [, options], callback)
It supports the definitions such as:
html, text or json. see more from Output<br /> or not, true by default.[minTextLength] characters, don't even count it, 25 by default.callback(error, article, options)See test or examples folder for a complete example
With High Availability: spider2
var read = require('read-art');
// read from google:
read('http://google.com', function(err, art, options){
if(err){
throw err;
}
var title = art.title, // title of article
content = art.content, // content of article
html = art.html; // whole original innerHTML
});
// or:
read({
uri: 'http://google.com',
charset: 'utf8'
}, function(err, art, options){
});
// what about html?
read('<title>node-art</title><body><div><p>hello, read-art!</p></div></body>', function(err, art, options){
});
// of course could be
read({
uri: '<title>node-art</title><body><div><p>hello, read-art!</p></div></body>'
}, function(err, art, options){
});
CAUTION: Title must be wrapped in a <title> tag and content must be wrapped in a <body> tag.
scoreRule You can custom the score rules manually, e.g.:
scoreRule: function(node){
if (node.hasClass('w740')) {
return 100;
}
}
The elements which have the w740 className will get 100 bonus points, that will make the node to be the topCandidate, which means it's enough to make the text of DIV/P.w740 to be the content of current article.
read('http://club.autohome.com.cn/bbs/thread-c-66-37239726-1.html', {
minTextLength: 0,
scoreRule: function(node){
if (node.hasClass('w740')) {
return 100;
}
}
}, function(err, art){
});
You can wrap the content of article with different types, the output option could be:
text, html and json, html by default.text, html and json.false by default.Returns the inner text, e.g.:
read('http://example.com', {
output: 'text'
}, function(err, art){
// art.content will be formatted as TEXT
});
// or
read('http://example.com', {
output: {
type: 'text',
stripSpaces: true
}
}, function(err, art){
// art.content will be formatted as TEXT
});
Returns the inner HTML, e.g.:
read('http://example.com', {
output: 'html'
}, function(err, art){
// art.content will be formatted as HTML
});
// or
read('http://example.com', {
output: {
type: 'html',
stripSpaces: true
}
}, function(err, art){
// art.content will be formatted as HTML
});
Notes Videos could be scraped now, the domains currently are supported: youtube|vimeo|youku|tudou|56|letv|iqiyi|sohu|sina|163.
Returns the restful result, e.g.:
read('http://example.com', {
output: 'json'
}, function(err, art){
// art.content will be formatted as JSON
});
// or
read('http://example.com', {
output: {
type: 'json',
stripSpaces: true
}
}, function(err, art){
// art.content will be formatted as Array
});
The art.content will be an Array such as:
[
{ "type": "img", "value": "http://example.com/jpg/site1/20140519/00188b1996f214e3a25417.jpg" },
{ "type": "text", "value": "TEXT goes here..." }
]
Util now there are only two types - img and text, the src of img element is absolute even if the original is a relative one.
Notes The video sources of the sites are quite different, it's hard to fit all in a common way, I haven't find a good way to solve that, PRs are in demand.
read('http://game.163.com/14/0506/10/9RI8M9AO00314SDA.html', {
charset: 'gbk'
}, function(err, art){
// ...
});
read('http://example.com', {
agent: true // true as default
}, function(err, art){
// ...
});
read('http://example.com', {
proxy: {
host: 'http://myproxy.com/',
port: 8081,
proxyAuth: 'user:password'
}
}, function(err, art){
// ...
});
npm test
luin/node-readability is an old Readability that be transformed from Arc90, easy to use, but the problem is - TOO SLOW. It was based on jsdom, so, the HTML must be written in strict mode, which means you can not make any mistake, e.g.:
<P>Paragraphs</p>
<p>My book name is <read-art></p>
<div><p>Hey, dude!</div>
All above will cause hiberarchy errors, more seriously, jsdom is a memory killer.
I've contributed on this for a while, but it's hard to communicate with Vadim(we are in a different timezone), and we have very different ideas. So I decided to write it on my own.
Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.
FAQs
Scrape/Crawl article from any site automatically. Make any web page readable, no matter Chinese or English.
The npm package read-art receives a total of 55 weekly downloads. As such, read-art popularity was classified as not popular.
We found that read-art demonstrated a not healthy version release cadence and project activity because the last version was released a year ago. It has 1 open source maintainer collaborating on the project.
Did you know?
Socket for GitHub automatically highlights issues in each pull request and monitors the health of all your open source dependencies. Discover the contents of your packages and block harmful activity before you install or update your dependencies.
Research
Security News
Malicious PyPI package ‘set-utils’ steals Ethereum private keys by exfiltrating them through blockchain transactions via the Polygon RPC.
Research
Security News
Malicious Go packages are impersonating popular libraries to install hidden loader malware on Linux and macOS, targeting developers with obfuscated payloads.
Security News
Bybit's $1.46B hack by North Korea's Lazarus Group pushes 2025 crypto losses to $1.6B in just two months, already surpassing all of 2024's $1.49B total.