Security News
New Proposed CISA Mandate Would Require Critical Infrastructure to Report Ransom Payments Within 24 Hours
CISA has proposed a set of new rules that would require critical infrastructure to report cyber incidents and ransom payments.
web-auto-extractor
Advanced tools
Automatically extracts structured information from webpages
Weekly downloads
Readme
Parse semantically structured information from any HTML webpage.
Supported formats:-
Popularly, many websites mark up their webpages with Schema.org vocabularies for better SEO. This library helps you parse that information to JSON.
Demo it on tonicdev
npm install web-auto-extractor
// IF CommonJS
var WAE = require('web-auto-extractor').default
// IF ES6
import WAE from 'web-auto-extractor'
var parsed = WAE().parse(sampleHTML)
Let's use the following text as the sampleHTML
in our example. It uses Schema.org vocabularies to structure a Product information and is encoded in microdata
format.
<div itemscope itemtype="http://schema.org/Product">
<span itemprop="brand">ACME</span>
<span itemprop="name">Executive Anvil</span>
<img itemprop="image" src="anvil_executive.jpg" alt="Executive Anvil logo" />
<span itemprop="description">Sleeker than ACME's Classic Anvil, the
Executive Anvil is perfect for the business traveler
looking for something to drop from a height.
</span>
Product #: <span itemprop="mpn">925872</span>
<span itemprop="aggregateRating" itemscope itemtype="http://schema.org/AggregateRating">
<span itemprop="ratingValue">4.4</span> stars, based on <span itemprop="reviewCount">89
</span> reviews
</span>
<span itemprop="offers" itemscope itemtype="http://schema.org/Offer">
Regular price: $179.99
<meta itemprop="priceCurrency" content="USD" />
$<span itemprop="price">119.99</span>
(Sale ends <time itemprop="priceValidUntil" datetime="2020-11-05">
5 November!</time>)
Available from: <span itemprop="seller" itemscope itemtype="http://schema.org/Organization">
<span itemprop="name">Executive Objects</span>
</span>
Condition: <link itemprop="itemCondition" href="http://schema.org/UsedCondition"/>Previously owned,
in excellent condition
<link itemprop="availability" href="http://schema.org/InStock"/>In stock! Order now!</span>
</span>
</div>
Our parsed
object should look like -
{
"microdata": {
"Product": [
{
"@context": "http://schema.org/",
"@type": "Product",
"brand": "ACME",
"name": "Executive Anvil",
"image": "anvil_executive.jpg",
"description": "Sleeker than ACME's Classic Anvil, the\n Executive Anvil is perfect for the business traveler\n looking for something to drop from a height.",
"mpn": "925872",
"aggregateRating": {
"@context": "http://schema.org/",
"@type": "AggregateRating",
"ratingValue": "4.4",
"reviewCount": "89"
},
"offers": {
"@context": "http://schema.org/",
"@type": "Offer",
"priceCurrency": "USD",
"price": "119.99",
"priceValidUntil": "5 November!",
"seller": {
"@context": "http://schema.org/",
"@type": "Organization",
"name": "Executive Objects"
},
"itemCondition": "http://schema.org/UsedCondition",
"availability": "http://schema.org/InStock"
}
}
]
},
"rdfa": {},
"jsonld": {},
"metatags": {
"priceCurrency": [
"USD",
"USD"
]
}
}
The parsed
object includes four objects - microdata
, rdfa
, jsonld
and metatags
. Since the above HTML does not have any information encoded in rdfa
and jsonld
, those two objects are empty.
I wouldn't call it a caveat but rather the parser is strict by design. It might not parse like expected if the HTML isn't encoded correctly, so one might assume the parser is broken.
For example, take the following HTML snippet.
<div itemscope itemtype="http://schema.org/Movie">
<h1 itemprop="name">Ghostbusters</h1>
<div itemprop="productionCompany" itemscope itemtype="http://schema.org/Organization">Black Rhino</div>
<div itemprop="countryOfOrigin" itemscope itemtype="http://schema.org/Country">
Country: <span itemprop="name" content="USA">United States</span><p>
</div>
</div>
The problem here is the itemprop
- productionCompany
which is of itemtype
- Organization
doesn't have any itemprop
as its children, in this case - name
.
The parser assumes every itemtype
contains an itemprop
, or every typeof
contains a property
in case of rdfa
. So the "Black Rhino"
information is lost.
It'll be nice to fix this by having a non-strict
mode for parsing this information. PRs are welcome.
MIT
FAQs
Automatically extracts structured information from webpages
The npm package web-auto-extractor receives a total of 13,186 weekly downloads. As such, web-auto-extractor popularity was classified as popular.
We found that web-auto-extractor demonstrated a not healthy version release cadence and project activity because the last version was released a year ago. It has 1 open source maintainer collaborating on the project.
Did you know?
Socket for GitHub automatically highlights issues in each pull request and monitors the health of all your open source dependencies. Discover the contents of your packages and block harmful activity before you install or update your dependencies.
Security News
CISA has proposed a set of new rules that would require critical infrastructure to report cyber incidents and ransom payments.
Security News
Redis is no longer OSS, breaking its explicit commitment to remain under the BSD 3-Clause License forever. This has angered contributors who are now working to fork the software.
Product
Socket AI now enables 'AI detected potential malware' alerts by default, ensuring users benefit from AI-powered state-of-the-art malware detection without needing to opt-in.