snapshot_reload

SnapshotReload

SnapshotReload is a ruby gem that pulls down a gzipped database snapshot, and reloads it into a database.

This gem is part of the suite of tools needed to bring up the Q* server environments for nightly testing of the novu web site code using obfuscated production data (All participants' personal data has been scrubbed.)

The gem will work during the nightly chef deploy to the Q* servers.

Installation

Clone the repo from github:

    $ git clone [e-mail]:novu/snapshot_reload.git

Then, install the gem:

    $ gem install snapshot_reload

Usage

The gem has a complete usage output, run snapshot_reload -h for the current listing.

This is what it looks like at version 1.0.0:

Usage: snapshot_reload [options] config

Redeploy the obfuscated data from production to other servers

v1.0.0

Options:
    -h, --help                       Show command line help
    -e, --env ENVIRONMENT            Environment to use in configuration file
                                     Will use ENV['RAILS_ENV'] if not specified and the envar exists
                                     Otherwise will use qa
    -s, --source S3SOURCE            Provide the AWS S3 source of the obfuscated data
                                     Optional flag. Default value is s3://novu_backups/db_backups/clean-mysqldump.sql.gz
    -c, --aws-conf AWS_CONFIG        Provide an alternative AWS configuration file containing log in credentials
                                     Optional flag. Default value is /opt/novu/.s3cfg
    -k, --aws-key AWSKEY             Provide the AWS key for logging in. Overrides value in AWS_CONFIG and default if given
                                     Required if --aws-secret is specified
    -y, --aws-secret AWSSECRET       Provide the AWS secret key for logging in. Overrides value in AWS_CONFIG and default if given.
                                     Required if --aws-key is specified.
    -n, --dry-run                    Don't run anything, just show what would be run
    -v, --verbose                    Provide lots of information about what is happening
    -q, --quiet                      Be completely silent. Overrides --verbose switch
        --version                    Show help/version info
        --log-level LEVEL            Set the logging level
                                     (debug|info|warn|error|fatal)
                                     (Default: info)

Arguments:

    config
        Configuration file in same format as Rails config/database.yml file

Discussion

This gem implements the steps necessary to stand up an obfuscated copy of the production database for http://www.novu.com in the Q* server environments (Qdev, Qrel, and Qnext). It is meant to be installed with the chef deployments for these servers. It can also be used to get any gzipped snapshot of the application database (does not have to be obfuscated production data) and stand it up to any database environment. This makes it useful for individual developers and testers, as it can stand up a database on a local machine, or another server environment, independently of chef. All you need is the gzipped file as source, and the database credentials in config.

Examples

The default, standing up the Qdev server via chef recipe:

    $ snapshot_reload /opt/novu/current/config/database.yml

The standard location of the application's database configuration file is given, the RAILS_ENV is set to 'qa' and the default S3 credentials are stored in /opt/novu/.s3cfg

Install a snapshot of nightly cleaned production data into your local database

You will need to obtain S3 credentials that give you access to the (non-production) buckets, see [e-mail] to obtain if you don't have any. Put them into a file that looks like a standard .ini key = value file:

    access_key = YOURAWSACCESSKEY
    secret_key = YOURAWSSECRETACCESSKEY

No quotes, etc, just the values. The keys must be given as shown, at the start of the line. White space around the '=' is optional.

Then issue:

    $ snapshot_reload -c path/to/aws/credentials -e development path/to/database.yml

and the data will be loaded from the S3 bucket into the database specified by the development environment in the configuration file.

If you have a local gzipped copy of data to load, you can specify that as well:

    $ snapshot_reload --source path/to/sql.gz -e development path/to/database.yml

This will load the data in the sql.gz file into the database specified by the 'development' environment in the configuration file.

Author

Tamara Temple [e-mail]

Copyright, Licensing

Copyright © Novu, LLC. All rights reserverd. For internal and private use only.

Contributing

1. Fork it
2. Create your feature branch (git checkout -b my-new-feature)
3. Commit your changes (git commit -am 'Add some feature')
4. Push to the branch (git push origin my-new-feature)
5. Create new Pull Request