article-parser
Extract main article, main image and meta data from URL.
Demo
Setup
Usage
import { extract } from 'article-parser'
const url = 'https://www.binance.com/en/blog/markets/15-new-years-resolutions-that-will-make-2022-your-best-year-yet-421499824684903249'
extract(url).then((article) => {
console.log(article)
}).catch((err) => {
console.trace(err)
})
Note:
Since Node.js v14, ECMAScript modules have became the official standard format.
Just ensure that you are using module system and enjoy with ES6 import/export syntax.
APIs
Load and extract article data. Return a Promise object.
Example:
import { extract } from 'article-parser'
const getArticle = async (url) => {
try {
const article = await extract(url)
return article
} catch (err) {
console.trace(err)
return null
}
}
getArticle('https://domain.com/path/to/article')
If the extraction works well, you should get an article
object with the structure as below:
{
"url": URI String,
"title": String,
"description": String,
"image": URI String,
"author": Person[],
"publisher": Organization,
"content": HTML String,
"published": Date String,
"source": String,
"links": Array,
"ttr": Number,
}
Click here for seeing an actual result.
addQueryRules(Array queryRules)
Add custom rules to get main article from the specific domains.
This can be useful when the default extraction algorithm fails, or when you want to remove some parts of main article content.
Example:
import { addQueryRules, extract } from 'article-parser'
extract('https://bad-website.domain/page/article')
addQueryRules([
{
patterns: [
{ hostname: 'bad-website.domain' }
],
selector: '#noop_article_locates_here',
unwanted: [
'.advertise-area',
'.stupid-banner'
]
}
])
extract('https://bad-website.domain/page/article')
While adding rules, you can specify a transform()
function to fine-tune article content more thoroughly.
Example rule with transformation:
import { addQueryRules } from 'article-parser'
addQueryRules([
{
patterns: [
{ hostname: 'bad-website.domain' }
],
selector: '#article_id_here',
rawTransformer: (document) => {
const metaChild = document.createElement('meta');
metaChild.setAttribute('name','publisher')
metaCgild.setAttribute('content','Epoch Times')
document.head.appendChild(metaChild)
return document
},
transform: (document) => {
document.querySelectorAll('h1').forEach(node => {
const newNode = document.createElement('b')
newNode.innerHTML = node.innerHTML
node.parentNode.replaceChild(newNode, node)
})
return document
}
}
])
Please refer MDN for more info.
Configuration methods
In addition, this lib provides some methods to customize default settings. Don't touch them unless you have reason to do that.
- getParserOptions()
- setParserOptions(Object parserOptions)
- getRequestOptions()
- setRequestOptions(Object requestOptions)
- getSanitizeHtmlOptions()
- setSanitizeHtmlOptions(Object sanitizeHtmlOptions)
Here are default properties/values:
Object parserOptions
:
{
wordsPerMinute: 300,
urlsCompareAlgorithm: 'levenshtein',
descriptionLengthThreshold: 40,
descriptionTruncateLen: 156,
contentLengthThreshold: 200
}
Read string-comparison docs for more info about urlsCompareAlgorithm
.
Object requestOptions
:
{
headers: {
'user-agent': 'Mozilla/5.0 (X11; Linux x86_64; rv:95.0) Gecko/20100101 Firefox/95.0',
accept: 'text/html; charset=utf-8'
},
responseType: 'text',
responseEncoding: 'utf8',
timeout: 6e4,
maxRedirects: 3
}
Read axios' request config for more info.
Object sanitizeHtmlOptions
:
{
allowedTags: [
'h1', 'h2', 'h3', 'h4', 'h5',
'u', 'b', 'i', 'em', 'strong', 'small', 'sup', 'sub',
'div', 'span', 'p', 'article', 'blockquote', 'section',
'details', 'summary',
'pre', 'code',
'ul', 'ol', 'li', 'dd', 'dl',
'table', 'th', 'tr', 'td', 'thead', 'tbody', 'tfood',
'fieldset', 'legend',
'figure', 'figcaption', 'img', 'picture',
'video', 'audio', 'source',
'iframe',
'progress',
'br', 'p', 'hr',
'label',
'abbr',
'a',
'svg'
],
allowedAttributes: {
a: ['href', 'target', 'title'],
abbr: ['title'],
progress: ['value', 'max'],
img: ['src', 'srcset', 'alt', 'width', 'height', 'style', 'title'],
picture: ['media', 'srcset'],
video: ['controls', 'width', 'height', 'autoplay', 'muted'],
audio: ['controls'],
source: ['src', 'srcset', 'data-srcset', 'type', 'media', 'sizes'],
iframe: ['src', 'frameborder', 'height', 'width', 'scrolling'],
svg: ['width', 'height']
},
allowedIframeDomains: ['youtube.com', 'vimeo.com']
}
Read sanitize-html docs for more info.
Test
git clone https://github.com/ndaidong/article-parser.git
cd article-parser
npm install
npm test
npm run eval {URL_TO_PARSE_ARTICLE}
License
The MIT License (MIT)