github.com/HakumenNC/docker-schemacrawler-reporting
Some Kibana Dashboards visuals produced on top of this project.
Also you can get a full but (yet short) video that explains the motivation and philosophy behind this ecosystem.
docker run7.9.2 stack to consume the exported csv
$ git --version
git version 2.23.0.windows.1
$ docker -v
Docker version 18.09.6, build c89750f8
$ docker-compose -v
docker-compose version 1.27.4, build 40524192
:bulb: No database on hand? Don't panic, let's go here
git clone https://github.com/HakumenNC/docker-schemacrawler-reporting.git
cd docker-schemacrawler-reporting
Deploy the Elastic World (in case if we want the all stack on local environnement else use the ek.yml file, without Logstash so)
docker-compose --project-name schemacrawler-elk -f elk.yml up -d
...And that's all :clap: !
$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
130787ecd783 docker.elastic.co/logstash/logstash:7.9.2 "/usr/local/bin/dock…" About an hour ago Up 41 minutes 5044/tcp, 9600/tcp logstash
9fabc346ce9e postgres:11.5 "docker-entrypoint.s…" 3 hours ago Up 35 minutes 0.0.0.0:5432->5432/tcp optisee_optisee-postgresql_1
d426d2f30ed7 docker.elastic.co/kibana/kibana:7.9.2 "/usr/local/bin/dumb…" 2 days ago Up 41 minutes 0.0.0.0:5601->5601/tcp kibana
9b7106e5b1dd docker.elastic.co/elasticsearch/elasticsearch:7.9.2 "/tini -- /usr/local…" 3 days ago Up 41 minutes 0.0.0.0:9200->9200/tcp, 9300/tcp elasticsearch
After several minutes, we can testing if everything it's OK with :
If you run logstash separately, take a look for the configuration here
News calculateds fields (in ruby :gem:) are added on logstash step :
| field | pipeline | description |
|---|---|---|
sclint-isRGPDColumn | logstash-columns.conf | Is true if the column's remark (based on field sclint-column-remarks) contains "RGPD" string else false |
sclint-linterIdClassName | logstash-lints.conf | Return the className of field sclint-linterId |
sclint-linterIdPackageName | logstash-lints.conf | Return the package's path of field sclint-linterId |
:bulb: Feel free to create PR or issues for any new ideas of calculateds fields !
We use the folder exportcsv :file_folder: for export/inject the .csv files
Edit the schemacrawler.sh file for fill the connection informations of your database
Run it :rocket: !
sh schemacrawler.sh
The docker image mbarre/schemacrawler-additional-lints is used to run schemacrawler everywhere or almost !
Go to http://localhost:5601
Reach the Stack Management via the side left menu
In Index Patterns page, click on Create index pattern button
Type schemacrawler-* as an index pattern name and click on Next step
Select @timestamp as Time field
Finally, click on Create index pattern
You are now ready to see the data imported on the Discover page (In the side menu : Kibana/Discover)
It's creativity time! Kibana provide Dashboard and Canvas for visualisations. See some examples
Don't want to execute docker commands... but try all the stack quickly... This is the solution.
Following tools are installed :
... And avoid tears of blood
Got this issue (personnally experimented on Windows 10 at home... And randomly hapenned)? Please see https://docs.docker.com/docker-for-windows/troubleshoot/#networking-issues
Unable to find image 'hello-world:latest' locally
Pulling repository docker.io/library/hello-world
C:\Program Files\Docker\Docker\Resources\bin\docker.exe: Error while pulling image: Get https://index.docker.io/v1/repositories/library/hello-world/images: dial tcp: lookup index.docker.io on 10.0.75.1:53: no such host.
See 'C:\Program Files\Docker\Docker\Resources\bin\docker.exe run --help'.
And if it is not enough, set as DNS Server 8.8.8.8 on your Windows' network card directly...
FAQs
Unknown package
Did you know?
Socket for GitHub automatically highlights issues in each pull request and monitors the health of all your open source dependencies. Discover the contents of your packages and block harmful activity before you install or update your dependencies.
Security News
As vulnerability data bottlenecks grow, the federal government is formally investigating NIST’s handling of the National Vulnerability Database.
Research
Security News
Socket’s Threat Research Team has uncovered 60 npm packages using post-install scripts to silently exfiltrate hostnames, IP addresses, DNS servers, and user directories to a Discord-controlled endpoint.
Security News
TypeScript Native Previews offers a 10x faster Go-based compiler, now available on npm for public testing with early editor and language support.