Product
Introducing License Enforcement in Socket
Ensure open-source compliance with Socket’s License Enforcement Beta. Set up your License Policy and secure your software!
elasticsearch-follow
Advanced tools
elasticsearch_follow is library helping to query Elasticsearch continuously.
It needs https://github.com/elastic/elasticsearch-py as a dependency.
elasticsearch_follow acts as a wrapper for elasticsearch-py and handles various use-cases, like following logs by polling elasticsearch continuously and fetching loglines via a generator. It is possible to easily fetch lines surrounding a given logline.
The polling logic is implemented in the class ElasticsearchFollow, which needs an Elasircsearch object from elasticsearch-py. The class Follower takes an ElasticsearchFollow-object and has a method to create a generator which yields loglines until all elements of a query have been returned. After this a new generator has to be created and used.
To just fetch loglines, one can use ElasticsearchFetch which has a search_surrounding. This returns a list of lists, where each list contains the queried loglines and the lines before and after as requested by the parameters num_before and num_after.
You can install the elasticsearch
package with pip:
pip install elasticsearch_follow
See also: https://pypi.org/project/elasticsearch-follow/
This package introduces the command line tool es_tail
which can be used for
following logs written to Elasticsearch and directly fetching log lines by a query.
It is possible to configure the output via an format string.
# Follow the logs written to the indexes starting with logstash.
# Print the fieds @timestamp and message
es_tail -c "http://localhost:9200" tail --index "logstash*" -f "{@timestamp} {message}"
# Fetch all logs in the last hour with the field loglevel contains ERROR and fetch the two lines before and after.
# Print the fieds @timestamp and message
es_tail -c "http://localhost:9200" fetch --index "logstash" -f "{@timestamp} {message}" --query loglevel:ERROR -A 2 -B 2 -F "now-1h"
# It is also possible to print nested fields
es_tail -c "http://localhost:9200" fetch --index "logstash" -f "{@timestamp} {message} {kv[field]} {kv[nested][field]}" -F "now-1h"
The command line options can also be given via environment variables by using the prefix ES_TAIL
.
For example
export ES_TAIL_USERNAME='username'
export ES_TAIL_PASSWORD='password'
es_tail -c http://localhost:9200 tail
from elasticsearch import Elasticsearch
from elasticsearch_follow import ElasticsearchFollow, Follower
es = Elasticsearch()
es_follow = ElasticsearchFollow(elasticsearch=es)
# The Follower is used to get a generator which yields new
# elements until it runs out. time_delta give the number of
# seconds to look into the past.
follower = Follower(elasticsearch_follow=es_follow, index='some-index', time_delta=60)
while True:
entries = follower.generator()
for entry in entries:
print(entry)
time.sleep(0.1)
FAQs
An Elasticsearch tail
We found that elasticsearch-follow demonstrated a healthy version release cadence and project activity because the last version was released less than a year ago. It has 1 open source maintainer collaborating on the project.
Did you know?
Socket for GitHub automatically highlights issues in each pull request and monitors the health of all your open source dependencies. Discover the contents of your packages and block harmful activity before you install or update your dependencies.
Product
Ensure open-source compliance with Socket’s License Enforcement Beta. Set up your License Policy and secure your software!
Product
We're launching a new set of license analysis and compliance features for analyzing, managing, and complying with licenses across a range of supported languages and ecosystems.
Product
We're excited to introduce Socket Optimize, a powerful CLI command to secure open source dependencies with tested, optimized package overrides.