crawler-framework

About

Main objective is to secure fast deployment of crawler system that will help you setup all that you need to get started with data mining interactively over cmd/terminal. Concept is based on long working experience and mistakes that have been learned over the years in data mining and database arhitecture. It could save you a lot of time and effort using this framework. Currently works on Win OS.

Installation

Most important thing is that you have python 3 installed on your machine.

With virtual environment (recommended)
To be able to make virtual environment you will need to pip3.7 install virtualenv Location of your virtual environment on WinOS shoud be C:\Users\YourUsername\Envs Then create virtual environment py -3.7 -m virtualenv Envs/crawler_framework
By default now your virtual environment should be active if it isn't activate your virtual environment by writing workon crawler_framework
Now you can install library pip install crawler_framework

Without virtual environment
Just open cmd and write pip3.7 install crawler_framework. This option is ok if you have only one version of python installed on your system. But if you have more python version installed and you decide to go with this option you will have to finish some additional question's during configuration of crawler framework. Like that it will be ensured that everything works ether way.

Setup

Database configuration

Before we can deploy anything we must setup connection strings for one or more database servers that we are going to use. Currently supported are PostgreSQL, Oracle and Microsoft SQL Server.

It is recommended to create database on your server that will only be used by framework.

Open cmd/terminal and write config.py. If everything goes well you should see this options below. It is possible that program asks for some additional information if you have more than one python interpreter installed on your machine and you did not use virtual environment. But it will be required only once.

Create all database connection's that you think you will use, from database where you will deploy crawler_framework to database where you will store data etc. by selecting option number 1 and then database option number one.

Hints

Microsoft SQL Server
If you are using default option be sure to define in ODBC Data sources administrator dsn name that will have default database that will be used for framework. If you are using pymssql you will define server_ip, port and database during database configuration stage.

Deploying framework

Open cmd/terminal and write config.py. Select option 2 (Deploy framework) and then select option from the list of connections you created that is going to be used for deployment. This will deploy table structure in selected database on selected server connection. In our case we will deploy it on PostgreSQL localhost server.

Setup tor expert bundle

You can skip this if you are not planing to use tor as your proxy provider but only public proxies.

Open cmd/terminal and write config.py. Select option 4 (Tor setup). Then from to setup option select option 1(Install), this will automatically install tor expert bundle with all necessary directories and subdirectories.

If you have already have installed and then changed default options then this will reset those options back to default just like some kind of reinstall.

Setup option 2 (Tor options) allows you to change some of defaults when constructing our tor network such as how many tor instances should run in same time or how long time must pass before changing and identity of tor instance.

Starting proxy server

Proxy server is multifunctional program that acquires new proxies(crawlers), test proxies, creates tor network etc.

Open cmd/terminal and write config.py. Select option 3 (Run proxy server).

Suboptions

From suboptions you can select suboption 0 to run all both public proxy and tor service or suboption 1 to run only public proxy gatherer or suboption 2 to run only tor service.

Suboption 2 is great if you want to run tor service on another pc inside your network. By doing so you can increase number of tor that will run but it will not hog your local CPU and RAM.

All data will be saved in database where you have deployed crawler_framewok. If you didn't made deployment then this will probably end with exceptions.