Under the hood, it uses OpenAI to extract estimated information from the product page and Google Custom Search JSON API / SerpAPI to extract images from product pages using Google Image Search coupled with Ikea's product search API and some custom scrappers for some popular furniture websites.
How to use
1. Install
npm i universal-pdp-scrapper
2. Usage
import { UniversalPDPScrapper } from'universal-pdp-scrapper';
// Initialize the client and set the API keys// You can set API Keys using environment variables as well: check [.env.sample](./.env.sample)const client = newUniversalPDPScrapper({
openaiApiKey: '',
openaiOrgId: '',
openaiModelId: '',
googleApiKey: '',
googleCseId: ''
});
const result = await client.scrape('https://www.ikea.com/us/en/p/jokkmokk-table-and-4-chairs-antique-stain-50211104/');
console.log(result);
3. Demo
Here's a demo of running the scrapper in a server environment integrated with a React app
Uses OpenAI to extract estimated information from the product page
Uses Google Custom Search JSON API to extract images from product pages using Google Image Search
Uses Cheerio to scrape product pages for some popular websites
NOTE: This library doesn't solve the issue of CORS for images or glbs. If you encounter cors, its better to use this library in the server environment and download the images and glbs to your server and serve them from there.
We found that universal-pdp-scrapper demonstrated a healthy version release cadence and project activity because the last version was released less than a year ago.It has 1 open source maintainer collaborating on the project.
Last updated on 25 Aug 2023
Did you know?
Socket for GitHub automatically highlights issues in each pull request and monitors the health of all your open source dependencies. Discover the contents of your packages and block harmful activity before you install or update your dependencies.
CISA launched a new project called Vulnrichment to enrich CVEs with details that help prioritize patching and mitigation efforts, as the NVD backlog of unenriched CVEs awaiting analysis surpasses 10,000.
Socket is joining forces with CISA and other industry leaders at the RSA Conference to sign the Secure by Design pledge, committing to uphold the highest security standards in our products.
The Socket research team breaks down a sampling of malicious packages that download and execute files, among other suspicious behaviors, targeting the popular Discord platform.