github.com/joesenvv/crawler-project

Crawler-website

It's crawler website using Go language.

This is the mainPage

This is the homePage

Features

• Go language
• Docker
• Elastic Search
• MVC pattern
• Microservices
• Singleton -> Concurrent -> Distribute

Installation and go packages

• go language
• docker
• elasticsearch
• go get golang.org/x/text
• go get -v github.com/gpmgo/gopm
• gopm get -g -v golang.org/x/text
• gopm get -g -v golang.org/x/net/html
• go get gopkg.in/olivere/elastic.v5

Usage for Concurrent

• Start Docker.
• Run Script "docker run -d -p 9200:9200 elasticsearch"
• Run "src/crawler/main.go", to start the singleton crawler.
• Run "src/crawler/frontend/starter.go", to view the result in the website.
• Visit "http://localhost:8888/" in your browser
• Type in query string with REST format. such as "女 && Age>20"

Usage for Distribute

• Start Docker.
• Run Script "docker run -d -p 9200:9200 elasticsearch"
• Open a Terminal, execute: src\crawler_distributed\persist\server>go run ItemSaver.go --port=1234
• Open a Terminal, execute: src\crawler_distributed\worker\server>go run worker.go --port=9000
• Open a Terminal, execute: src\crawler_distributed\worker\server>go run worker.go --port=9001
• Open a Terminal, execute: src\crawler_distributed>go run main.go --itemsaver_host=":1234" --worker_hosts=":9000,:9001"
• Run "src/crawler/frontend/starter.go", to view the result in the website.
• Visit "http://localhost:8888/" in your browser
• Type in query string with REST format. such as "男 && 已购车"

Architecture

Framework

Algorithm

Reference

• Google资深工程师深度讲解Go语言 @ https://coding.imooc.com/class/180.html
• Seed website: http://www.zhenai.com/zhenghun