![From Academia to Industry](https://cdn.sanity.io/images/cgdhsj6q/production/70501690bf807891573e259da9719cf24302c7e3-4288x2848.jpg?w=400&fit=max&auto=format)
Employee Spotlight
From Academia to Industry
Philipp Burckhardt recounts his journey from childhood computer fascinations, to building an e-learning platform at Carnegie Mellon University, and on to his current role at Socket.
open-graph-scraper
Advanced tools
Changelog
5.0.3
Readme
A simple node module for scraping Open Graph and Twitter Card info off a site. For browser usage, we recommend using ky to make the requests(or a backend service) then pass in the html
into open-graph-scraper
using the html
option.
npm install open-graph-scraper --save
const ogs = require('open-graph-scraper');
const options = { url: 'http://ogp.me/' };
ogs(options)
.then((data) => {
const { error, result, response } = data;
console.log('error:', error); // This returns true or false. True if there was an error. The error itself is inside the results object.
console.log('result:', result); // This contains all of the Open Graph results
console.log('response:', response); // This contains the HTML of page
})
Check the return for a success
flag. If success is set to true, then the url input was valid. Otherwise it will be set to false. The above example will return something like...
{
ogTitle: 'Open Graph protocol',
ogType: 'website',
ogUrl: 'http://ogp.me/',
ogDescription: 'The Open Graph protocol enables any web page to become a rich object in a social graph.',
ogImage: {
url: 'http://ogp.me/logo.png',
width: '300',
height: '300',
type: 'image/png'
},
requestUrl: 'http://ogp.me/',
success: true
}
Name | Info | Default Value | Required |
---|---|---|---|
url | URL of the site. | x | |
html | You can pass in an HTML string to run ogs on it. (use without options.url) | ||
blacklist | Pass in an array of sites you don't want ogs to run on. | [] | |
onlyGetOpenGraphInfo | Only fetch open graph info and don't fall back on anything else. | false | |
ogImageFallback | Fetch other images if no open graph ones are found. | true | |
customMetaTags | Here you can define custom meta tags you want to scrape. | [] | |
allMedia | By default, OGS will only send back the first image/video it finds | false | |
peekSize | Sets the peekSize for the request | 1024 | |
downloadLimit | Maximum size of the content downloaded from the server, in bytes | 1000000 (1MB) | |
urlValidatorSettings | Sets the options used by validator.js for testing the URL | Here |
Note: open-graph-scraper
uses got for requests and most of got's options should work as open-graph-scraper
options.
const ogs = require('open-graph-scraper');
const options = {
url: 'https://github.com/jshemas/openGraphScraper',
customMetaTags: [{
multiple: false, // is there more than one of these tags on a page (normally this is false)
property: 'hostname', // meta tag name/property attribute
fieldName: 'hostnameMetaTag', // name of the result variable
}],
};
ogs(options)
.then((data) => {
const { error, result, response } = data;
console.log('hostnameMetaTag:', result.hostnameMetaTag); // hostnameMetaTag: github.com
})
Look here for more info on how to use proxies.
const ogs = require('open-graph-scraper');
const tunnel = require('tunnel');
const options = {
url: 'https://whatismyipaddress.com/',
timeout: {
request: 10000,
},
agent: {
// setting proxy agent for https requests
https: tunnel.httpsOverHttp({
// test proxies can be found here: https://hidemy.name/en/proxy-list/?country=US&type=h#list or http://free-proxy.cz/en/proxylist/country/US/https/ping/all
proxy: {
host: 'proxy_ip',
port: proxyPort,
rejectUnauthorized: false,
}
})
}
};
ogs(options)
.then((data) => {
const { error, result, response } = data;
console.log('response:', response); // you should see the proxy IP in here
})
const ogs = require("open-graph-scraper");
const options = {
url: "https://www.wikipedia.org/",
headers: {
"user-agent": "Googlebot/2.1 (+http://www.google.com/bot.html)",
},
};
ogs(options)
.then((data) => {
const { error, result, response } = data;
console.log("error:", error); // This returns true or false. True if there was an error. The error itself is inside the results object.
console.log("results:", results); // This contains all of the Open Graph results
})
Then you can run the tests by running...
npm run test
FAQs
Node.js scraper module for Open Graph and Twitter Card info
The npm package open-graph-scraper receives a total of 53,062 weekly downloads. As such, open-graph-scraper popularity was classified as popular.
We found that open-graph-scraper demonstrated a healthy version release cadence and project activity because the last version was released less than a year ago. It has 0 open source maintainers collaborating on the project.
Did you know?
Socket for GitHub automatically highlights issues in each pull request and monitors the health of all your open source dependencies. Discover the contents of your packages and block harmful activity before you install or update your dependencies.
Employee Spotlight
Philipp Burckhardt recounts his journey from childhood computer fascinations, to building an e-learning platform at Carnegie Mellon University, and on to his current role at Socket.
Security News
Git dependencies in open source packages can introduce significant risks, including lack of version control, stability issues, dependency drift, and difficulty in auditing, making them potential targets for supply chain attacks.
Security News
Node.js has added experimental support for TypeScript, a move that highlights the growing importance of TypeScript in modern development.