github.com/kenniston/mobile-push-kafka
The objective of this project is to demonstrate the use of queues to integrate services of a system, avoiding the strong coupling between them. As an example, an integration with Firebase will be used to send push messages to an Android application.
This document describes the steps needed to build and run the Mobile Push Sender using Kafka.
Updated: 02 Fev 2021
All docker images can be created by docker-compose. To start all containers, use the docker-compose up command. This command will build all the necessary images and create the containers.
Add the following line in hosts file
127.0.0.1 kafka
Below are the hosts and ports for the all containers (docker on localhost):
This project needs at least 4 GB of RAM. On MacOS, docker must have 4G RAM to run all the containers in this project. The ElasticSearch container can exit with code 137 if there is no RAM available.
project-name = Procuder or Consumer
RUN CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build \
-a \
-o project-name \
-ldflags \
"-s -w \
-extldflags '-static' \
-X github.com/kenniston/mobile-push-kafka/golang/cmd.BuildTime=$(date -u '+%Y-%m-%d_%H:%M:%S%p') \
-X github.com/kenniston/mobile-push-kafka/golang/cmd.GitCommit=$(git rev-parse HEAD) \
-X github.com/kenniston/mobile-push-kafka/golang/cmd.Version='0.1'" main.go && \
upx --ultra-brute -v /home/app/build/project-name && \
upx -t /home/app/build/project-name
Install UPX (Ultimate Packer for eXecutables - Compress/expand executable files)
brew install upx
apt-get install -y upx
choco install upx
Compress the server executable with UPX:
upx --ultra-brute -v ./project-name && upx -t ./project-name
or
upx -9 -v ./project-name && upx -t ./project-name
The docker-composer.yml file has the build section for all GoLang project. This section uses the Dockerfile below to generate the project's executable and optimize it using the upx tool.
FROM golang:1.15.7-alpine as builder
RUN apk update && \
apk add --no-cache build-base && \
apk add --no-cache upx git ca-certificates tzdata && \
update-ca-certificates && \
addgroup --system app && adduser -S -G app app
WORKDIR /home/app/build
COPY . .
RUN CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build \
-a \
-o project-name \
-ldflags \
"-s -w \
-extldflags '-static' \
-X github.com/kenniston/mobile-push-kafka/golang/cmd.BuildTime=$(date -u '+%Y-%m-%d_%H:%M:%S%p') \
-X github.com/kenniston/mobile-push-kafka/golang/cmd.GitCommit=$(git rev-parse HEAD) \
-X github.com/kenniston/mobile-push-kafka/golang/cmd.Version='0.1'" main.go && \
upx --ultra-brute -v /home/app/build/project-name && \
upx -t /home/app/build/project-name
FROM scratch
COPY --from=builder /usr/share/zoneinfo /usr/share/zoneinfo
COPY --from=builder /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/
COPY --from=builder /etc/passwd /etc/passwd
COPY --from=builder /etc/group /etc/group
COPY --from=builder /home/app/build/project-name /home/app/project-name
USER app
ENTRYPOINT ["/home/app/project-name", "run"]
The default port can change per project
./project-name run
--server-port [Default: 6001 - HTTP Server Port]
--log-level [Default: info - (debug, error, trace, info, warning, panic, fatal)]
--graylog [Default: false - Boolean]
--graylog-ip [Default: localhots - Graylog Server IP or host name]
--graylog-port [Default: 5555 - Integer]
./project-name desktop
The Producer project uses a server framework to expose REST endpoints (golang/restserver directory). The server uses Iris Web Framework (https://github.com/kataras/iris/) to expose endpoints through project controllers.
The Logrus Graylog Hook (https://github.com/gemnasium/logrus-graylog-hook) is used to send log message through Logrus to Graylog server. The Hook is configured on the server framework:
var useGraylog = v.GetBool("graylog")
var graylogIP = v.GetString("graylog-ip")
var graylogPort = v.GetInt32("graylog-port")
if useGraylog {
hook := graylog.NewGraylogHook(fmt.Sprintf("%s:%d", graylogIP, graylogPort), nil)
logrus.AddHook(hook)
logrus.Infof("Log messages are now sent to Graylog (udp://%s:%d)", graylogIP, graylogPort)
}
Not implemented yet
Not implemented yet
Kafka is the message broker for this project. Services communicate with each other through Kafka topics. The project uses two topics: the first one is used to send push messages through Firebase component; the second one is used to return the results from Firebase to the caller.
Message Flow
The Kafka image for this project is based on wurstmeister/kafka:2.13-2.7.0 which provides additional parameters for Kafka containers. In addition, the final image also contains the library responsible for sending the monitoring data to prometheus (Prometheus JMX Agent).
With default setup the Kafka container starts with two topics: MobileSendPush and MobilePushResult. Each topic has 4 partition and 1 replica.
The Kafdrop (http://localhost:19000) can be used to manager Kafka topics. Kafdrop provides a visual Dashboard that shows Kafka information (Cluster, Brokers and Topics) and provide a visual topic editor. New topic can be created using Kafdrop, avoiding the Kafka command line tool.
This project uses the official Prometheus container to build a new image. The new image has the config file used to link Prometheus with Kafka JXM port to receive the Kafka's metrics.
Kafka uses the Prometheus JMX Agent (port 8082 in this project) to expose Prometheus metrics through his host.
https://hub.docker.com/r/prom/prometheus
This project uses the official Grafana container. After run the project using the docker-compose up the Prometheus's Datasource must be configured. Use the following URL (docker on localhost) to configure the Prometheus Datasource:
http://localhost:3000/datasources/new?utm_source=grafana_gettingstarted
Default user: admin
Default password: admin
Select Prometheus in the Time series database section:
Grafana Datasources
Fill in the fields as the image below:
Prometheus Datasource Info
Last but not least, set up the Kafka Dashboard in Grafana. Import the dashboard file from the kakfa folder (grafana-dashboard-kafka-metrics_rev4.json) using a URL below:
http://localhost:3000/dashboard/import
Import the Kafka's Dashboard using the Upload JSON file button
Grafana - Kafka Dashboard
This project provides a visual dashboard for logs using a Graylog container. Clients (Golang producer/consumer, Java producer/consumer and Kotlin producer/consumer) can uses GELF messages to send logs to the Graylog.
To send log messages to Graylog a new Input must be configured. To create a new Input follow the steps below:
Graylog Create Input - Step 1
Select GELF UDP and click on Launch new input button - Step 2
Fill in the GELF UDP Input information (title and port are the most important) and click on Save button - Step 2
With the input information (port and Graylog address) the clients can send log messages to the Graylog. Messages can be filtered on the Graylog Search tab.
Graylog Dashboard - Logs
FAQs
Unknown package
Did you know?
Socket for GitHub automatically highlights issues in each pull request and monitors the health of all your open source dependencies. Discover the contents of your packages and block harmful activity before you install or update your dependencies.
Research
Security News
Socket researchers uncover a malicious npm package posing as a tool for detecting vulnerabilities in Etherium smart contracts.
Security News
Research
A supply chain attack on Rspack's npm packages injected cryptomining malware, potentially impacting thousands of developers.
Research
Security News
Socket researchers discovered a malware campaign on npm delivering the Skuld infostealer via typosquatted packages, exposing sensitive data.