elastic-backup-2s3

• elastic-backup-2s3 -- Snapshot, Backup, and Restore Elasicsearch Indicies to and from S3

  When you have an itch to scratch...

  I simply want to be able to control the backup and restoration of the Elasticsearch cluster to S3 without any fuss or having to dilly around with curl -XPUTS and friends.

  I have finally added Shared Filesystem Support! Please see the docs below on this.

** Overview Elastic Backup simply allows you to specifiy some or all of the indicies, some target bucket, and run the backup / snapshot.

It hides all the sordid details about "registration" and the like, and will optionally clean up behind itself and remove the registration if you did not specify a name for the registration.

** Breaking Changes *** v1.0.0 The New 1.x version is not backards-compatable with the 0.x versions. Especially for the File System-style backups, the repo name is not appended to the directory path, to allow for multiple repo backups on the file system. Please be aware of this moving forward.

** Installation

To install from command line: #+begin_src bash gem install elastic-backup-2s3 #+end_src

For a Gemfile entry: #+begin_src ruby gem 'elastic-backup-2s3' #+end_src

** TODO Documentation Documentation is comming. Basically, just running the command without any parameters will give you some basic usage.

*** S3URL format We adopt a convention to make it easy for you to specify the S3 bucket, path, and snapshot name stored in the bucket. The S3 URL format is:

s3://AWS_S3_BUCKET/path/in/bucket:SNAPSHOT

In some cases, :SNAPSHOT is optional, but usually
expected to be there.

*** Support for Shared Filesystem Snapshots To do S3 backups requires no additional configurations outside of being in the AWS clould (or at least being connected to one.) Shared File System (FS) backups require that you have each and every node configured to permit the backup path on all your nodes in the cluster, and thus not as simple as merely writing it through the JSON REST interface.

So presumably you either have this setup in place, and
in any rate, it would be a bit messy to have to specify
the "local" path in the repo here.

For this reason we will not (over)write the repo settings
for FS backups by default. We provide a new option that will
allow you to write out the FS repo, specifying all the
parameters you need to at that time.

The default repository name should be configurable in any case.
we could do a configuration file, but for this I think it's 
overkill. so we will use environment variables instead.

| Environment variable       | Description                                                                                                |
|----------------------------+------------------------------------------------------------------------------------------------------------|
| ESB_DEFAULT_REPO           | Changes the default repository from 'elastic-backup' to what is specified here.                            |
| ESB_SHARED_VOLUME          | For Shared Volume backups, default local path to use. Note that this path must be configured on all nodes. |
| ESB_TIMEOUT                | Default for --timeout option                                                                               |
| ESB_SNAPSHOT_MAX_BYTES_SEC | Shared Volume max_snapshot_bytes_per_sec                                                                   |
| ESB_RESTORE_MAX_BYTES_SEC  | Shared Volume max_restore_bytes_per_sec                                                                    |
|                            |                                                                                                            |

** Examples To backup/snapshot your ES to S3 and wait until the backup is completed: #+BEGIN_SRC bash es-snapshot snapshot localhost s3://your-backup-bucket/your-backup-path:snap_1 -v2 -w #+END_SRC

The default repository of 'elastic-backup' is used. To specify your own repository name: #+BEGIN_SRC bash es-snapshot snapshot localhost s3://your-backup-bucket/your-backup-path:snap_1 -v2 -w --repo your-repo-name #+END_SRC

To backup/snapshot two indices index1 and index2: #+BEGIN_SRC bash es-snapshot snapshot localhost s3://your-backup-bucket/your-backup-path:snap_1 -v2 -w --indices index1 index2 #+END_SRC

To restore your snapshot #+BEGIN_SRC bash es-snapshot restore s3://your-backup-bucket/your-backup-path:snap_1 localhost -v2 -w #+END_SRC

To restore your snapshot using your-repo-name: #+BEGIN_SRC bash es-snapshot restore s3://your-backup-bucket/your-backup-path:snap_1 localhost -v2 -w --repo your-repo-name #+END_SRC

You may specify --dryrun (-u) to see the JSON that will be sent without actually executing the commands. Also -v2 will show you the same while allowing your commands to execute.

To see the status of your snapshots: #+BEGIN_SRC bash es-snapshot list status localhost #+END_SRC

To see a list of your indices: #+BEGIN_SRC bash es-snapshot list indices localhost #+END_SRC

** Release Notes | Date | Version | Description | |------------+---------+-----------------------------------------------------------------------------------------| | 2016-07-21 | v1.0.0 | Breaking changes in how the subdirectory paths are created. Please see [[Breaking Changes]] | | 2016-07-12 | v0.1.5 | Fixed single index restore bug. | | 2016-06-10 | v0.1.3 | Added a new --nv flag to switch off shared FS verification. | | 2016-05-02 | v0.1.2 | Fixed a couple of minor issues with the restore from shared volume. | | 2016-04-22 | v0.1.0 | Support for FS volume backups added | | 2016-04-17 | v0.0.3 | Added a delete snapshot option to allow deletion of a running snapshot. | | | | | ** Known Issues | Date | Version | Description | | 2061-04-17 | v0.0.3 | 'delete' options not fully implemented. | | | | Error return codes not fully fleshed out | | | | 'delete snapshot' requires the full S3URL, but only needs the snapshot part. | | 2061-04-21 | v0.0.4 | Some features remain unimplemented, though there are options for them. Please bear with me. | | | | What shall we do with configurations? I am thinking we save them in ES, but also allow for config files. The ES settings will override the config file settings. Config files in Yaml format. | | | | |

** Scratchpad My own personal notes on this project. These are not gauranteed to be useful to you nor up-to-date.

** Contributing to elastic-backup-2s3

• Check out the latest master to make sure the feature hasn't been implemented or the bug hasn't been fixed yet.
• Check out the issue tracker to make sure someone already hasn't requested it and/or contributed it.
• Fork the project.
• Start a feature/bugfix branch.
• Commit and push until you are happy with your contribution.
• Make sure to add tests for it. This is important so I don't break it in a future version unintentionally.
• Please try not to mess with the Rakefile, version, or history. If you want to have your own version, or is otherwise necessary, that is fine, but please isolate to its own commit so I can cherry-pick around it.

** Copyright

Copyright (c) 2016 Fred Mitchell. See LICENSE.txt for further details.