Under the hood, it uses OpenAI to extract estimated information from the product page and Google Custom Search JSON API / SerpAPI to extract images from product pages using Google Image Search coupled with Ikea's product search API and some custom scrappers for some popular furniture websites.
How to use
1. Install
npm i universal-pdp-scrapper
2. Usage
import { UniversalPDPScrapper } from'universal-pdp-scrapper';
// Initialize the client and set the API keys// You can set API Keys using environment variables as well: check [.env.sample](./.env.sample)const client = newUniversalPDPScrapper({
openaiApiKey: '',
openaiOrgId: '',
openaiModelId: '',
googleApiKey: '',
googleCseId: ''
});
const result = await client.scrape('https://www.ikea.com/us/en/p/jokkmokk-table-and-4-chairs-antique-stain-50211104/');
console.log(result);
3. Demo
Here's a demo of running the scrapper in a server environment integrated with a React app
Uses OpenAI to extract estimated information from the product page
Uses Google Custom Search JSON API to extract images from product pages using Google Image Search
Uses Cheerio to scrape product pages for some popular websites
NOTE: This library doesn't solve the issue of CORS for images or glbs. If you encounter cors, its better to use this library in the server environment and download the images and glbs to your server and serve them from there.
The npm package universal-pdp-scrapper receives a total of 1 weekly downloads. As such, universal-pdp-scrapper popularity was classified as not popular.
We found that universal-pdp-scrapper demonstrated a healthy version release cadence and project activity because the last version was released less than a year ago.It has 1 open source maintainer collaborating on the project.
Last updated on 25 Aug 2023
Did you know?
Socket for GitHub automatically highlights issues in each pull request and monitors the health of all your open source dependencies. Discover the contents of your packages and block harmful activity before you install or update your dependencies.
A hospital in Mobile, Alabama, agreed to a settlement in a landmark ransomware death lawsuit, but is now reportedly reconsidering the agreement and refusing to pay.
A new report explores how advancements in LLMs are enhancing cyber threats, including polymorphic malware, personalized spearphishing, and the risk of hijacking customer service bots.
ESLint has approved an RFC that adds support for TypeScript configuration files, which is aimed at improving the developer experience and recognizing changes in the evolving JavaScript ecosystem.