horseman-article-parser

Horseman Article Parser

A web page article parser which returns an object containing the article's formatted text & other attributes including sentiment, keyphrases, people, places, organisations and spelling suggestions.

Prerequisites

Node.js & NPM

Install

npm install horseman-article-parser --save

Usage Example

var parser = require('horseman-article-parser');

var options = {
  userAgent: 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.106 Safari/537.36',
  url: "https://www.theguardian.com/politics/2018/sep/24/theresa-may-calls-for-immigration-based-on-skills-and-wealth"
}

parser.parseArticle(options)
  .then(function (article) {

    var response = {
      title: article.title.text,
      metadescription: article.meta.description.text,
      url: article.url,
      sentiment: { score: article.sentiment.score, comparative: article.sentiment.comparative },
      keyphrases: article.processed.keyphrases,
      people: article.people,
      orgs: article.orgs,
      places: article.places,
      text: {
        raw: article.processed.text.raw,
        formatted: article.processed.text.formatted,
        html: article.processed.text.html
      },
      spelling: article.spelling
    }

    console.log(response);
  })
  .catch(function (error) {
    console.log(error.message)
    console.log(error.stack);
  })

parseArticle(options, <socket>) accepts an optional socket for pipeing status messages and errors to a front end UI.

See horseman-article-parser-ui as an example.

Options

The options below are set by default

var options = {
  userAgent: 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.106 Safari/537.36',
  horseman: {
    timeout: 10000, 
    cookies: './cookies.json'
  },
  htmlcleaner: {
    'add-remove-tags': ['blockquote', 'span'],
    'remove-empty-tags': ['span'],
    'replace-nbsp': true
  },
  readability: {},
  texttohtml: {
    wordwrap: 100,
    noLinkBrackets: true,
    ignoreHref: true,
    tables: true,
    uppercaseHeadings: true
  },
  retextkeywords: { maximum: 10 },
  retextspell: {},
  striptags: [
    "img",
    "noscript",
    "style",
    "script",
    "figure",
    ".ayl-text",
    ".affiliate-text",
    ".mol-video",
    ".mol-img-group",
    ".artSplitter",
    "#ayl-wrapper",
    "h3.sharing-bar__title",
  ]
}

At a minimum you should pass a url

var options = {
  url: "https://www.theguardian.com/politics/2018/sep/24/theresa-may-calls-for-immigration-based-on-skills-and-wealth"
}

Development

Please feel free to fork the repo or open pull requests to the development branch. I've used eslint for linting & yarn for dependency management.

Build the dependencies with:

yarn

Lint the index.js file with:

yarn lint

Dependencies

• clean-html: HTML cleaner and beautifier
• compromise: natural language processing in the browser
• dictionary-en-gb: English (United Kingdom) spelling dictionary in UTF-8
• html-to-text: Advanced html to plain text converter
• lodash: Lodash modular utilities.
• node-horseman: Run PhantomJS from Node
• nlcst-to-string: Stringify NLCST
• node-readability: Turning any web page into a clean view.
• phantomjs-prebuilt: Headless WebKit with JS API
• retext: Natural language processor powered by plugins
• retext-keywords: Keyword extraction with Retext
• retext-spell: Spelling checker for retext
• sentiment: AFINN-based sentiment analysis for Node.js
• vfile-reporter-json: JSON reporter for virtual files

Dev Dependencies

• eslint: An AST-based pattern checker for JavaScript.
• eslint-config-standard: JavaScript Standard Style - ESLint Shareable Config
• eslint-plugin-import: Import with sanity.
• eslint-plugin-node: Additional ESLint's rules for Node.js
• eslint-plugin-promise: Enforce best practices for JavaScript promises
• eslint-plugin-standard: ESlint Plugin for the Standard Linter

License

This project is licensed under the GNU GENERAL PUBLIC LICENSE Version 3 - see the LICENSE.md file for details