Security News
The Push to Ban Ransom Payments Is Gaining Momentum
Ransomware costs victims an estimated $30 billion per year and has gotten so out of control that global support for banning payments is gaining momentum.
horseman-article-parser
Advanced tools
Readme
A web page article parser which returns an object containing the article's formatted text & other attributes including sentiment, keyphrases, people, places, organisations and spelling suggestions.
Node.js & NPM
npm install horseman-article-parser --save
var parser = require('horseman-article-parser');
var options = {
userAgent: 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.106 Safari/537.36',
url: "https://www.theguardian.com/politics/2018/sep/24/theresa-may-calls-for-immigration-based-on-skills-and-wealth"
}
parser.parseArticle(options)
.then(function (article) {
var response = {
title: article.title.text,
metadescription: article.meta.description.text,
url: article.url,
sentiment: { score: article.sentiment.score, comparative: article.sentiment.comparative },
keyphrases: article.processed.keyphrases,
people: article.people,
orgs: article.orgs,
places: article.places,
text: {
raw: article.processed.text.raw,
formatted: article.processed.text.formatted,
html: article.processed.text.html
},
spelling: article.spelling
}
console.log(response);
})
.catch(function (error) {
console.log(error.message)
console.log(error.stack);
})
parseArticle(options, <socket>)
accepts an optional socket for pipeing the response object, status messages and errors to a front end UI.
See horseman-article-parser-ui as an example.
The options below are set by default
var options = {
userAgent: 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.106 Safari/537.36',
// node-horsman options (https://ghub.io/node-horseman)
horseman: {
timeout: 10000,
cookies: './cookies.json'
},
// clean-html options (https://ghub.io/clean-html)
cleanhtml: {
'add-remove-tags': ['blockquote', 'span'],
'remove-empty-tags': ['span'],
'replace-nbsp': true
},
// html-to-text options (https://ghub.io/html-to-text)
htmltotext: {
wordwrap: 100,
noLinkBrackets: true,
ignoreHref: true,
tables: true,
uppercaseHeadings: true
},
// retext-keywords options (https://ghub.io/retext-keywords)
retextkeywords: { maximum: 10 }
}
At a minimum you should pass a url
var options = {
url: "https://www.theguardian.com/politics/2018/sep/24/theresa-may-calls-for-immigration-based-on-skills-and-wealth"
}
there are some additional "complex" options available
var options = {
// array of html elements to stip before anaysis
striptags: [],
// readability options (https://ghub.io/node-readability)
readability: {},
// retext spell options (https://ghub.io/retext-spell)
retextspell: {}
}
Please feel free to fork the repo or open pull requests to the development branch. I've used eslint for linting & yarn for dependency management.
Build the dependencies with:
yarn
Lint the index.js file with:
yarn lint
This project is licensed under the GNU GENERAL PUBLIC LICENSE Version 3 - see the LICENSE.md file for details
FAQs
Web Page Inspection Tool. Sentiment Analysis, Keyword Extraction, Named Entity Recognition & Spell Check
We found that horseman-article-parser demonstrated a not healthy version release cadence and project activity because the last version was released a year ago. It has 1 open source maintainer collaborating on the project.
Did you know?
Socket for GitHub automatically highlights issues in each pull request and monitors the health of all your open source dependencies. Discover the contents of your packages and block harmful activity before you install or update your dependencies.
Security News
Ransomware costs victims an estimated $30 billion per year and has gotten so out of control that global support for banning payments is gaining momentum.
Application Security
New SEC disclosure rules aim to enforce timely cyber incident reporting, but fear of job loss and inadequate resources lead to significant underreporting.
Security News
The Python Software Foundation has secured a 5-year sponsorship from Fastly that supports PSF's activities and events, most notably the security and reliability of the Python Package Index (PyPI).