You're Invited:Meet the Socket Team at BlackHat and DEF CON in Las Vegas, Aug 4-6.RSVP

shopee-crawler

Advanced tools

Socket logo

Install Socket

Detect and block malicious and high-risk dependencies

Install

shopee-crawler

Crawler for Shopee from all country

0.2.3
Maintainers
1

Shopee-crawler

This is a light crawling tool.

High performance.

Easy to use.

Requirement

python>=3.7

Install

  • Using setup.py
git clone https://github.com/lthoangg/shopee-crawler
python/python3 setup.py install
  • Using pip
pip install shopee-crawler

How to use

Examples:

from shopee_crawler import Crawler

crawler = Crawler()
crawler.set_origin(origin="shopee.vn") # Input your root Shopee website of your country that you want to crawl

data = crawler.crawl_by_shop_url(shop_url='shop_url')

data = crawler.crawl_by_cat_url(cat_url='cat_url')

data = crawler.crawl_by_search(keyword='keyword')

data = crawler.crawl_cat_list()
# print(data)

Usage

  • About 12.000 rows (products) in 2-3 seconds (32 workers)

  • About 12.000 rows (products) in 5-6 seconds (16 workers)

Test yourself

import time
from shopee_crawler import Crawler

crawler = Crawler()
crawler.set_origin(origin="shopee.vn")

keyword = 'điện thoại samsung'

# Start time
start = time.time()

# Crawl
data = crawler.crawl_by_search(keyword=keyword)

# End time
end = time.time()

# Time crawling
print("Time : ",end - start, "seconds")

FAQs

Did you know?

Socket

Socket for GitHub automatically highlights issues in each pull request and monitors the health of all your open source dependencies. Discover the contents of your packages and block harmful activity before you install or update your dependencies.

Install

Related posts