lycos

🐾 lycos.js

All the goodies you'll ever need to scrape the web

Documentation

• Get Started
• Examples
• API Reference

In-browser Playground

You can try the library on codesandbox, it uses a cors proxy fetcher to let you grab contents from any website inside your browser.

• CodeSandbox: https://codesandbox.io/s/njm2p72m

Installation

yarn add lycos
# or
npm i lycos

Features

• ⚡️️ All in one package to fetch and scrape data from the web
• ⭐ Node & Browser Support
• 💡 Powerful declarative API
• 🚀 Blazingly fast (supports concurrency)
• 🔧 Extensible

Quick Example

const lycos = require('lycos');

// Get page from the url and paginate through all
// the next pages to extract quotes
const quotes = await lycos
  .get('http://quotes.toscrape.com/')
  .paginate('.next > a')
  .asyncFlatMap(pageScraper =>
    pageScraper.scrapeAll('.quote', {
      author: '.author@text',
      text: '.text@text'
    })
  );

<!-- quotes -->
[
  { 
  "author": "Albert Einstein", 
  "text": "“The world as we have created it is a process of our thinking.“"
  },
  ...
]

Credits

• FB55: his work reprensents the core of this library.

• Matt Mueller and cheerio contributors : A good portion of the code and concepts are copied/derived from the cheerio and x-ray libraries.

License

MIT © 2019 Jimmy Laurent