elasticshelf

Elasticshelf

Manage Elasticsearch indices using wither, close, open, delete, snapshot, and restore.

Most of the index actions can be performed by setting an expiry via cutoff days.

The gem may be used within a Rails app or in a Ruby script for cron jobs.

Note: this is inspired by Curator at https://github.com/elasticsearch/curator.git, but if you are an ophidiophobe, no worries, this is written in Ruby :-)

The following actions may be performed on Elasticsearch indices:

• wither (i.e. disable bloom filter)
• close
• open
• delete
• snapshot
• restore

Installation

Add this line to your application's Gemfile:

gem 'elasticshelf'

And then execute:

$ bundle

Or install it yourself as:

$ gem install elasticshelf

Usage

Note: see the examples folder for more command line ruby scripts.

initial setup:

require 'elasticshelf'
es = Elasticshelf::Client.new(:host => '127.0.0.1:9200')

if required be sure to override these defaults:

es.indices_prefix = 'logstash-'
es.date_separator = '.'

Note: an asterisk '*' is appended to indices_prefix.

ensure the cutoff date is as desired:

es.cutoff_days = 30
es.cutoff_days_as_date_to_s

do a "dry run" to see what indices will be affected:

es.find_expired_indices

Wither expired indices (i.e. remove bloom filter)

remove the bloom filter on any indices older than 30 days, but only if they are not closed already:

es.cutoff_days = 30
es.wither_indices

or wither a single index:

es.wither_index("index_name")

Close expired indices

close any indices older than 60 days:

es.cutoff_days = 60
es.close_expired_indices

or close a single index:

es.close_index("index_name")

Open index

just in case one was closed by mistake

es.open_index("index_name")

Delete expired indices

delete any indices older than 60 days:

es.cutoff_days = 60
es.delete_expired_indices

or delete a single index:

es.delete_index("index_name")

Snapshot create repository

Notes:

• some sysadmin prep is needed to ensure the location is accessible by Elasticsearch
• repo_type defaults to 'fs'
• repo_compressed defaults to true

es.repo = 'name_of_the_snapshot_repository'
es.repo_location = '/var/elasticsearch_snapshots' ... or what was prepped by the sysadmin
es.snapshot_create_repository

Snapshot get repository

Note: if es.repo is not set this gets information about all snapshot repositories.

es.snapshot_get_repository

Snapshot delete repository

es.repo = 'name_of_the_snapshot_repository'
es.snapshot_delete_repository

Snapshot expired indices

take a snapshot of all indices older than 22 days:

es.cutoff_days = 22
es.repo = 'name_of_the_snapshot_repository'
es.snapshot_expired_indices

Note: this snapshot is auto-named like "2014-04-02_15:07:55_utc_logstash-_cutoff_22_days", as there may be many indices within this snapshot. Override this as described below.

override snapshot auto-naming behavior:

es.snapshot_expired_indices_name = "more_desirable_name"
es.cutoff_days = 22
es.repo = 'name_of_the_snapshot_repository'
es.snapshot_expired_indices

Snapshot one or more indices

es.indices = "index1" ... single
es.indices = "index1,index2" ... multiples
es.repo = 'name_of_the_snapshot_repository'
es.snapshot = 'name_of_the_snapshot'
es.snapshot_expired_indices

Snapshot get

Note: if es.snapshot is not set this gets information about all snapshots, i.e. to see all snapshots do es.snapshot=nil.

es.snapshot_get

Snapshot delete

es.snapshot = 'name_of_the_snapshot'
es.snapshot_delete

Snapshot restore

es.repo = 'name_of_the_snapshot_repository'
es.snapshot = "name_of_the_snapshot"
- do one of these:
  1. es.indices = "" # restores all indices in the snapshot
  2. es.indices = "index1,index2" # restore multiple indices
  3. es.indices = "index1" # restore a single index
  - note: es.snapshot_get can be used to list indices in a snapshot
es.snapshot_restore
puts "errors=#{es.errors.inspect}"
puts "results=#{es.results.inspect}"

Other usage

es.index_closed?("index_name")
es.get_elasticsearch_version
es.get_lucene_version
es.get_info
es.get_cluster_state("index_name")
es.get_index_state("index_name")
es.version ... of this gem

Notes

• both open and closed indices are included when finding expired indices
• can not wither a closed index
• can not snapshot a closed index
• can not restore an open index from a snapshot
• deleting all (i.e. '*' or '_all') indices is not allowed
• both es.errors and es.results should be checked after any action
• exceptions are not re-raised from the elasticsearch-ruby gem, but the exception's class+message are copied into es.errors
• the following defaults are set, but may be overridden:
  • es.wait_for_completion = true
  • es.ignore_unavailable = true
  • es.include_global_state = false
  • es.date_separator = '.'
  • es.indices_prefix = 'logstash-'
  • es.repo_type = 'fs'
  • es.repo_compressed = true

Actions

1. get_settings_index name
2. find_expired_indices
3. open_index name
4. index_closed? name
5. close_index name
6. close_expired_indices
7. delete_index name
8. delete_expired_indices
9. wither_index name
10. wither_expired_indices
11. snapshot_create_repository
12. snapshot_get_repository
13. snapshot_delete_repository
14. snapshot_expired_indices
15. snapshot_create
16. snapshot_get
17. snapshot_delete
18. snapshot_restore

Contributing

1. Fork it ( http://github.com//elasticshelf/fork )
2. Create your feature branch (git checkout -b my-new-feature)
3. Commit your changes (git commit -am 'Add some feature')
4. Push to the branch (git push origin my-new-feature)
5. Create new Pull Request