Pinterest Crawler: Download as many images as you want about the searched words
Disables robots.txt for indexing by webcrawlers like Google
A simple python crawler
Disables robots.txt for indexing by webcrawlers like Google
Disallow indexing completely via robots.txt
A light weight anti crawler app for Django.
Crawler to extract data from various json api or html endpoints
Distributed Crawler Management Framework Based on Scrapy, Scrapyd, Scrapyd-Client, Scrapyd-API, Django and Vue.js.
A short description of your package
A simple package for crawling bol.com
A web crawler that takes input as place gives the weather forecast for 5 days
A gevent-based simple instagram hashtag crawler
crawler of some online judge system
simply web crawler
A client implementation of Firefox DevTools over remote debug protocol
ebest xing api wrapper
my simple crawler
a toolkit for cralwer Chinese OTA site data
A simple cloudacademy course crawling & downloading tool
Python package to detect bots/crawlers/spiders via user-agent
A simple crawler for http://www.meitulu.com/t/changtui/.
A simple python script to identify and classify possible crawlers through analysis of web server log files
tools which can easy the process of making crawler
Tools for Crawlers
A light weight crawler which gives search results in HTML form or in Dictionary form, given urls and keywords.
Crawler para pegar telefones em sites. Totalmente amador e apenas de TESTE.
spider framework for winndoo.
A lightweight Python crawling framework.
SQLi Crawler with JavaScript support.
Scraping and Crawling Micro-Framework
Crawler module for KIT Student Portal
A web crawler for GPTs to build knowledge bases
Crawler/parser for Yandex Images
PubChem REST API crawler to retrieve compound properties using a molecular formula search
A collection of unofficial web APIs for Python
American Airlines scraper in Python
Biblioteca em python para obtenção de seus dados de investimentos na bolsa de valores (B3/CEI).
For crawling web file explorers for content
XSStrike is a Cross Site Scripting detection suite equipped with four hand written parsers, an intelligent payload generator, a powerful fuzzing engine and an incredibly fast crawler. Instead of injecting payloads and checking it works like all the other tools do, XSStrike analyses the response with multiple parsers and then crafts payloads that are guaranteed to work by context analysis integrated with a fuzzing engine.
Simple website crawler that asynchronously crawls a website and all subpages that it can find, along with static content that they rely on.
A crawler for PsychonaughtWiki wrote in python.
A web page crawler which returns (title, og:image, og:description).
A CLI client for exporting elasticsearch data to csv