gitlab.com/lakshanwd/go-job-queue

go-job-queue

An example implementation of job queue with load balancer mechanism. Used grpc in order to gain maximum performance.

theory

• central server application listens to thousands of requests per second and loads them in to a queue.
• decentralized workers connect to central server and takes job parameters from the queue and store them in their own queue.
• each worker has limited no of deamon processes (go routines) which will take job parameters from their own worker queue and process and executes job

containerized and scalable

• run $ docker-compose up --scale worker=2 --scale client=5 to execute this app in a containerized environment
• run $ docker-compose scale worker=3 client=8 to scale app