kubesphere.io/fluentbit-operator
Facilitate the management of Fluent Bit and Fluentd in Kubernetes
Fluent Operator provides great flexibility in building a logging layer based on Fluent Bit and Fluentd.
Once installed, the Fluent Operator provides the following features:
label_router plugin.Although both Fluent Bit and Fluentd can collect, process(parse and filter) and then forward log to the final destinations, still they have strengths in different aspects.
Fluent Bit is a good choice as a logging agent because of its lightweight and efficiency, while Fluentd is more powerful to perform advanced processing on logs because of its rich plugins.
Fluent Operator includes CRDs and controllers for both Fluent Bit and Fluentd which allows you to config your log processing pipelines in the 3 modes mentioned above as you wish.
Fluent Bit will be deployed as a DaemonSet while Fluentd will be deployed as a StatefulSet. The whole workflow could be described as below:
The following CRDs are defined for Fluent Bit:
FluentBit: Defines the Fluent Bit DaemonSet and its configs. A custom Fluent Bit image kubesphere/fluent-bit is required to work with FluentBit Operator for dynamic configuration reloading.ClusterFluentBitConfig: Select cluster-level input/filter/output plugins and generates the final config into a Secret.ClusterInput: Defines cluster-level input config sections.ClusterParser: Defines cluster-level parser config sections.ClusterFilter: Defines cluster-level filter config sections.ClusterOutput: Defines cluster-level output config sections.Each ClusterInput, ClusterParser, ClusterFilter, ClusterOutput represents a Fluent Bit config section, which are selected by ClusterFluentBitConfig via label selectors. Fluent Operator watches those objects, constructs the final config, and finally creates a Secret to store the config which will be mounted into the Fluent Bit DaemonSet. The entire workflow looks like below:
To enable Fluent Bit to pick up and use the latest config whenever the Fluent Bit config changes, a wrapper called Fluent Bit watcher is added to restart the Fluent Bit process as soon as Fluent Bit config changes are detected. This way, the Fluent Bit pod needn't be restarted to reload the new config. The Fluent Bit config is reloaded in this way because there is no reloading interface in Fluent Bit itself. Please refer to this known issue for more details.
The following CRDs are defined for Fluentd:
Fluentd: Defines the Fluentd Statefulset and its configs. A custom Fluentd image kubesphere/fluentd is required to work with Fluentd Operator for dynamic configuration reloading.FluentdConfig: Select cluster-level or namespace-level scope input/filter/output plugins and generates the final config into a Secret.ClusterFluentdConfig: Select cluster-level input/filter/output plugins and generates the final config into a Secret.Filter: Defines namespace-level filter config sections.ClusterFilter: Defines cluster-level filter config sections.Output: Defines namespace-level output config sections.ClusterOutput: Defines cluster-level output config sections.Kubernetes v1.16.13+ is necessary for running Fluent Operator.
Install the latest stable version
kubectl apply -f https://raw.githubusercontent.com/fluent/fluent-operator/release-1.7/manifests/setup/setup.yaml
# You can change the namespace in manifests/setup/kustomization.yaml in corresponding release branch
# and then use command below to install to another namespace
# kubectl kustomize manifests/setup/ | kubectl apply -f -
Install the development version
kubectl apply -f https://raw.githubusercontent.com/fluent/fluentbit-operator/master/manifests/setup/setup.yaml
# You can change the namespace in manifests/setup/kustomization.yaml
# and then use command below to install to another namespace
# kubectl kustomize manifests/setup/ | kubectl apply -f -
Note: For the helm based install, Helm v3.2.1 or higher is needed.
The Fluent Bit section of the Fluent Operator supports different CRI docker, containerd, and CRI-O.
containerd and CRI-O use the CRI Log format which is different with docker, they requires additional parser to parse JSON application logs. You should set different containerRuntime depending on your container runtime.
The default runtime is docker, you can choose other runtimes as follows.
If your container runtime is containerd or cri-o, you can set the containerRuntime parameter to containerd or crio. e.g.
helm install fluent-operator --create-namespace -n fluent charts/fluent-operator/ --set containerRuntime=containerd
Install through the online chart link:
helm install fluent-operator --create-namespace -n fluent https://github.com/fluent/fluent-operator/releases/download/< version >/fluent-operator.tgz
Please replace < version > with a actual version like v1.0.0
For more info on various use cases of Fluent Operator, you can refer to Fluent-Operator-Walkthrough.
This guide provisions a logging pipeline including the Fluent Bit DaemonSet and its log input/filter/output configurations to collect Kubernetes logs including container logs and kubelet logs.
Note that you need a running Elasticsearch v5+ cluster to receive log data before start. Remember to adjust output-elasticsearch.yaml to your own es setup. Kafka and Fluentd outputs are optional and are turned off by default.
kubectl apply -f manifests/logging-stack
# You can change the namespace in manifests/logging-stack/kustomization.yaml
# and then use command below to install to another namespace
# kubectl kustomize manifests/logging-stack/ | kubectl apply -f -
You can also deploy the Kubernetes logging stack with Helm, just need to set the Kubernetes parameter to ture:
helm upgrade fluent-operator --create-namespace -n fluent charts/fluent-operator/ --set Kubernetes=true,containerRuntime=docker
If you want to deploy fluentd, just need to set the fluentd.enable parameter to ture.:
helm upgrade fluent-operator --create-namespace -n fluent charts/fluent-operator/ --set Kubernetes=true,containerRuntime=docker,fluentd.enable=true
Within a couple of minutes, you should observe an index available:
$ curl localhost:9200/_cat/indices
green open ks-logstash-log-2020.04.26 uwQuoO90TwyigqYRW7MDYQ 1 1 99937 0 31.2mb 31.2mb
Success!
The Linux audit framework provides a CAPP-compliant (Controlled Access Protection Profile) auditing system that reliably collects information about any security-relevant (or non-security-relevant) event on a system. Refer to manifests/logging-stack/auditd, it supports a method for collecting audit logs from the Linux audit framework.
kubectl apply -f manifests/logging-stack/auditd
# You can change the namespace in manifests/logging-stack/auditd/kustomization.yaml
# and then use command below to install to another namespace
# kubectl kustomize manifests/logging-stack/auditd/ | kubectl apply -f -
Within a couple of minutes, you should observe an index available:
$ curl localhost:9200/_cat/indices
green open ks-logstash-log-2021.04.06 QeI-k_LoQZ2h1z23F3XiHg 5 1 404879 0 298.4mb 149.2mb
Fluentd acts as a log forward layer that receives logs from Fluent Bit or other Apps through the network.
For more info on various use cases of Fluent Operator Fluentd CRDs, you can refer to Fluent-Operator-Walkthrough.
Here you can find supported Fluent Bit plugins.
Here you can find supported Fluentd plugins.
Here you can find the plugin group information.
Here you can use the built-in HTTP Server in Fluent Bit.
Here you can customize parser in Fluent Bit.
If you want to learn more about Fluent-Operator, please refer to the misc.
Here you can learn how to use custom plugin in Fluent Bit.
fluentd.fluent.iofluentbit.fluent.iomake installmake runAPI Doc is generated automatically. To modify it, edit the comment above struct fields, then run go run cmd/doc-gen/main.go.
Most files under the folder manifests/setup are automatically generated from config. Don't edit them directly, run make manifests instead, then replace these files accordingly.
Fluent Operator is sponsored and open-sourced by the KubeSphere Team and maintained by the Fluent community.
This project was accepted by the Fluent community as a sub-project in 2021 and is currently a vendor-neutral and community-driven project.
You are welcome to join us in the Fluent community to polish it together:
#fluent-operator to join or click #fluent-operator to join if you already have an accountkubesphere WeChat subscriptionFAQs
Unknown package
Did you know?
Socket for GitHub automatically highlights issues in each pull request and monitors the health of all your open source dependencies. Discover the contents of your packages and block harmful activity before you install or update your dependencies.
Security News
npm now supports Trusted Publishing with OIDC, enabling secure package publishing directly from CI/CD workflows without relying on long-lived tokens.
Research
/Security News
A RubyGems malware campaign used 60 malicious packages posing as automation tools to steal credentials from social media and marketing tool users.
Security News
The CNA Scorecard ranks CVE issuers by data completeness, revealing major gaps in patch info and software identifiers across thousands of vulnerabilities.