@postlight/mercury-parser
Advanced tools
Changelog
2.2.1 (Sept 08, 2021)
8c9982247b] - feat: Ladbible.com extractor (#624) (Nitin Khanna)30d6f472ee] - feat: Times of India extractor (#503) (Nitin Khanna)c15f4f5c3e] - chore(deps): bump lodash from 4.17.2 to 4.17.21 (#612) (dependabot[bot])b8b90afa95] - chore(deps): bump handlebars from 4.1.2 to 4.7.6 (#583) (dependabot[bot])38fb3204f5] - chore: remove greenkeeper configs (#580) (Jad Termsani)a5a066c69d] - chore: update node version (#579) (Jad Termsani)b0e708aac6] - feat: update nytimes extractor (#506) (Wajeeh Zantout)3fb8526c4a] - chore(package): update ora to version 4.0.0 (#500) (greenkeeper[bot])3009ecf137] - fix(package): update yargs-parser to version 14.0.0 (#495) (greenkeeper[bot])c5c000586d] - release: 2.2.0 (#496) (Michael Ashley)Readme
Postlight's Mercury Parser extracts the bits that humans care about from any URL you give it. That includes article content, titles, authors, published dates, excerpts, lead images, and more.
Mercury Parser powers the Mercury AMP Converter and Mercury Reader, a Chrome extension that removes ads and distractions, leaving only text and images for a beautiful reading view on any site.
Mercury Parser allows you to easily create custom parsers using simple JavaScript and CSS selectors. This allows you to proactively manage parsing and migration edge cases. There are many examples available along with documentation.
# If you're using yarn
yarn add @postlight/mercury-parser
# If you're using npm
npm install @postlight/mercury-parser
import Mercury from '@postlight/mercury-parser';
Mercury.parse(url).then(result => console.log(result));
// NOTE: When used in the browser, you can omit the URL argument
// and simply run `Mercury.parse()` to parse the current page.
The result looks like this:
{
"title": "Thunder (mascot)",
"content": "... <p><b>Thunder</b> is the <a href=\"https://en.wikipedia.org/wiki/Stage_name\">stage name</a> for the...",
"author": "Wikipedia Contributors",
"date_published": "2016-09-16T20:56:00.000Z",
"lead_image_url": null,
"dek": null,
"next_page_url": null,
"url": "https://en.wikipedia.org/wiki/Thunder_(mascot)",
"domain": "en.wikipedia.org",
"excerpt": "Thunder Thunder is the stage name for the horse who is the official live animal mascot for the Denver Broncos",
"word_count": 4677,
"direction": "ltr",
"total_pages": 1,
"rendered_pages": 1
}
If Mercury is unable to find a field, that field will return null.
parse() OptionsBy default, Mercury Parser returns the content field as HTML. However, you can override this behavior by passing in options to the parse function, specifying whether or not to scrape all pages of an article, and what type of output to return (valid values are 'html', 'markdown', and 'text'). For example:
Mercury.parse(url, { contentType: 'markdown' }).then(result =>
console.log(result)
);
This returns the the page's content as GitHub-flavored Markdown:
"content": "...**Thunder** is the [stage name](https://en.wikipedia.org/wiki/Stage_name) for the..."
You can include custom headers in requests by passing name-value pairs to the parse function as follows:
Mercury.parse(url, {
headers: {
Cookie: 'name=value; name2=value2; name3=value3',
'User-Agent':
'Mozilla/5.0 (iPhone; CPU iPhone OS 10_3_1 like Mac OS X) AppleWebKit/603.1.30 (KHTML, like Gecko) Version/10.0 Mobile/14E304 Safari/602.1',
},
}).then(result => console.log(result));
You can use Mercury Parser to parse custom or pre-fetched HTML by passing an HTML string to the parse function as follows:
Mercury.parse(url, {
html:
'<html><body><article><h1>Thunder (mascot)</h1><p>Thunder is the stage name for the horse who is the official live animal mascot for the Denver Broncos</p></article></body></html>',
}).then(result => console.log(result));
Note that the URL argument is still supplied, in order to identify the web site and use its custom parser, if it has any, though it will not be used for fetching content.
Mercury Parser also ships with a CLI, meaning you can use the Mercury Parser from your command line like so:
# Install Mercury globally
yarn global add @postlight/mercury-parser
# or
npm -g install @postlight/mercury-parser
# Then
mercury-parser https://postlight.com/trackchanges/mercury-goes-open-source
# Pass optional --format argument to set content type (html|markdown|text)
mercury-parser https://postlight.com/trackchanges/mercury-goes-open-source --format=markdown
# Pass optional --header.name=value arguments to include custom headers in the request
mercury-parser https://postlight.com/trackchanges/mercury-goes-open-source --header.Cookie="name=value; name2=value2; name3=value3" --header.User-Agent="Mozilla/5.0 (iPhone; CPU iPhone OS 10_3_1 like Mac OS X) AppleWebKit/603.1.30 (KHTML, like Gecko) Version/10.0 Mobile/14E304 Safari/602.1"
# Pass optional --extend argument to add a custom type to the response
mercury-parser https://postlight.com/trackchanges/mercury-goes-open-source --extend credit="p:last-child em"
# Pass optional --extend-list argument to add a custom type with multiple matches
mercury-parser https://postlight.com/trackchanges/mercury-goes-open-source --extend-list categories=".meta__tags-list a"
# Get the value of attributes by adding a pipe to --extend or --extend-list
mercury-parser https://postlight.com/trackchanges/mercury-goes-open-source --extend-list links=".body a|href"
# Pass optional --add-extractor argument to add a custom extractor at runtime.
mercury-parser https://postlight.com/trackchanges/mercury-goes-open-source --add-extractor ./src/extractors/fixtures/postlight.com/index.js
Licensed under either of the below, at your preference:
For details on how to contribute to Mercury, including how to write a custom content extractor for any site, see CONTRIBUTING.md
Unless it is explicitly stated otherwise, any contribution intentionally submitted for inclusion in the work, as defined in the Apache-2.0 license, shall be dual licensed as above without any additional terms or conditions.
🔬 A Labs project from your friends at Postlight. Happy coding!
FAQs
Mercury transforms web pages into clean text. Publishers and programmers use it to make the web make sense, and readers use it to read any web article comfortably.
The npm package @postlight/mercury-parser receives a total of 3,063 weekly downloads. As such, @postlight/mercury-parser popularity was classified as popular.
We found that @postlight/mercury-parser demonstrated a not healthy version release cadence and project activity because the last version was released a year ago. It has 8 open source maintainers collaborating on the project.
Did you know?
Socket for GitHub automatically highlights issues in each pull request and monitors the health of all your open source dependencies. Discover the contents of your packages and block harmful activity before you install or update your dependencies.
Security News
At Node Congress, Socket CEO Feross Aboukhadijeh uncovers the darker aspects of open source, where applications that rely heavily on third-party dependencies can be exploited in supply chain attacks.
Research
Security News
The Socket Research team found this npm package includes code for collecting sensitive developer information, including your operating system username, Git username, and Git email.
Security News
OpenJS is warning of social engineering takeovers targeting open source projects after receiving a credible attempt on the foundation.