@cumulus/deployment

@cumulus/deployment

@cumulus/deployment includes cloudformation templates needed for a successful deployment of a Cumulus Instance. The templates can be used with kes, a node CLI helper for AWS CloudFormation.

What is Cumulus?

Cumulus is a cloud-based data ingest, archive, distribution and management prototype for NASA's future Earth science data streams.

Cumulus Documentation

Usage

1. Copy app.example to a new deployment project.

2. Edit app.example/config.yml and your deployment information

3. Rename app.example to app.

4. Execute kes command:

   $ ./node_modules/.bin/kes cf deploy --kes-folder app --deployment <my-deployment> --template node_modules/@cumulus/deployment/app

All additions to app/api.yml should contain sources with a path to the corresponding npm installed packages, not a path to the folder in the local cumulus repository.

For example:

$ source: 'node_modules/@cumulus/api/dist/'

config.yml Explained

field	default	description
stackName	(required)	the name used as a prefix in all aws resources
stackNameNoDash	(required)	stackName with no dash
urs_url	uat.urs	urs url used for OAuth
api_backend_url	apigateway backend url	the API backend url
api_distribution_url	apigateway dist url	the API url used for file distribution
shared_data_bucket	cumulus-data-shared	the bucket has the shared data artifacts
system_bucket	(required)	the bucket used for storing deployment artifacts
buckets	N/A	Configuration of buckets with key, bucket name, and type (i.e. internal, public private)
cmr.username	devseed	the username used for posting metadata to CMR
cmr.provider	CUMULUS	the provider used for posting metadata to CMR
cmr.clientId	CUMULUS	the clientId used to authenticate with the CMR
cmr.password	(required)	the password used to authenticate with the CMR
ems.provider	CUMULUS	the provider used for sending reports to EMS
vpc.vpcId	(required if ecs is used)	the vpcId used with the deployment
vpc.subnets	(required)	the subnets used
defaults_users	cumulus core	list of default users included in all deployments
ecs.amiid	ami-9eb4b1e5	amiid of an optimized ecs instance (differnet for each region)
ecs.instanceType	(required)	the instance type of the ec2 machine used for running ecs tasks
ecs.volumeSize	50	the storage on ec2 instance running the ecs tasks
ecs.availabilityZone	us-east-1a	the availibity zone used for launching ec2 machines
ecs.maxInstances	1	max number of ec2 instances to launch in an autoscaling group
ecs.desiredInstances	0	desired number of ec2 instances needed in an autoscaling group
es.name	es5	name of the elasticsearch cluster
es.elasticSearchMapping	4	version number of the elasticsearch mapping used
es.version	5.3	elasticsearch software version
es.instanceCount	1	number of elasticsearch nodes
es.instanceType	t2.small.elasticsearch	size of the ec2 instance used for the elasticsearch
es.volumeSize	35	the storage used in each elasticsearch node
sns.<name>	N/A	name of the sns topic
sns.<name>.subscriptions.lambda.endpoint	sns2elasticsearch	lambda function triggered for each message in the topic
apis.<name>	N/A	name of the apigateway application
apiStage	dev	stage name used for each api gateway deployment stage
dynamos.<name>	N/A	name of the dynamoDB table
dynamos.<name>.read	5	number of reads per second
dynamos.<name>.write	1	number of writes per second
dynamos.<name>.attributes	N/A	list of attributes
sqs.<name>	N/A	name of the queue
sqs.<name>.visibilityTimeout	20	# of seconds the message returns to the queue after it is read by a consumer
sqs.<name>.retry	30	number of time the message is returned to the queue before being discarded
sqs.<name>.consumer	N/A	list of lambda function queue consumers
rules.<name>	N/A	list of cloudwathch rules
rules.<name>.schedule	N/A	rule's schedule
rules.<name>.state	ENABLED	state of the rule
rules.<name>.targets	N/A	list of lambda functions to be invoked
stepFunctions	N/A	list of step functions
lambdas	N/A	list of lambda functions

Contributing

See Cumulus README

Changelog

[v1.11.2] - 2019-2-15

Added

• CUMULUS-1169
  • Added a @cumulus/common/StepFunctions module. It contains functions for querying the AWS StepFunctions API. These functions have the ability to retry when a ThrottlingException occurs.
  • Added @cumulus/common/aws.retryOnThrottlingException(), which will wrap a function in code to retry on ThrottlingExceptions.
  • Added @cumulus/common/test-utils.throttleOnce(), which will cause a function to return a ThrottlingException the first time it is called, then return its normal result after that.
• CUMULUS-1103 Compare the collection holdings in CMR with Cumulus' internal data store
• CUMULUS-1099 Add support for UMMG JSON metadata versions > 1.4.
  • If a version is found in the metadata object, that version is used for processing and publishing to CMR otherwise, version 1.4 is assumed.
• CUMULUS-678
  • Added support for UMMG json v1.4 metadata files. reconcileCMRMetadata added to @cumulus/cmrjs to update metadata record with new file locations. @cumulus/common/errors adds two new error types CMRMetaFileNotFound and InvalidArgument. @cumulus/common/test-utils adds new function randomId to create a random string with id to help in debugging. @cumulus/common/BucketsConfig adds a new helper class BucketsConfig for working with bucket stack configuration and bucket names. @cumulus/common/aws adds new function s3PutObjectTagging as a convenience for the aws s3().putObjectTagging function. @cumulus/cmrjs Adds: - isCMRFile - Identify an echo10(xml) or UMMG(json) metadata file. - metadataObjectFromCMRFile Read and parse CMR XML file from s3. - updateCMRMetadata Modify a cmr metadata (xml/json) file with updated information. - publish2CMR Posts XML or UMMG CMR data to CMR service. - reconcileCMRMetadata Reconciles cmr metadata file after a file moves.
• Adds some ECS and other permissions to StepRole to enable running ECS tasks from a workflow
• Added Apache logs to cumulus api and distribution lambdas
• CUMULUS-1119 - Added @cumulus/integration-tests/api/EarthdataLogin.getEarthdataLoginRedirectResponse helper for integration tests to handle login with Earthdata and to return response from redirect to Cumulus API
• CUMULUS-673 Added @cumulus/common/file/getFileChecksumFromStream to get file checksum from a readable stream

Fixed

• CUMULUS-1123
  • Cloudformation template overrides now work as expected

Changed

• CUMULUS-1169

  • Deprecated the @cumulus/common/step-functions module.
  • Updated code that queries the StepFunctions API to use the retry-enabled functions from @cumulus/common/StepFunctions

• CUMULUS-1121

  • Schema validation is now strongly enforced when writing to the database. Additional properties are not allowed and will result in a validation error.

• CUMULUS-678 tasks/move-granules simplified and refactored to use functionality from cmrjs. ingest/granules.moveGranuleFiles now just moves granule files and returns a list of the updated files. Updating metadata now handled by @cumulus/cmrjs/reconcileCMRMetadata. move-granules.updateGranuleMetadata refactored and bugs fixed in the case of a file matching multiple collection.files.regexps. getCmrXmlFiles simplified and now only returns an object with the cmrfilename and the granuleId. @cumulus/test-processing - test processing task updated to generate UMM-G metadata

• CUMULUS-1043

  • @cumulus/api now uses express as the API engine.
  • All @cumulus/api endpoints on ApiGateway are consolidated to a single endpoint the uses {proxy+} definition.
  • All files under packages/api/endpoints along with associated tests are updated to support express's request and response objects.
  • Replaced environment variables internal, bucket and systemBucket with system_bucket.
  • Update @cumulus/integration-tests to work with updated cumulus-api express endpoints

• @cumulus/integration-tests - buildAndExecuteWorkflow and buildWorkflow updated to take a meta param to allow for additional fields to be added to the workflow meta

• CUMULUS-1049 Updated Retrieve Execution Status API in @cumulus/api: If the execution doesn't exist in Step Function API, Cumulus API returns the execution status information from the database.

• CUMULUS-1119

  • Renamed DISTRIBUTION_URL environment variable to DISTRIBUTION_ENDPOINT
  • Renamed DEPLOYMENT_ENDPOINT environment variable to DISTRIBUTION_REDIRECT_ENDPOINT
  • Renamed API_ENDPOINT environment variable to TOKEN_REDIRECT_ENDPOINT

Removed

• Functions deprecated before 1.11.0:
  • @cumulus/api/models/base: static Manager.createTable() and static Manager.deleteTable()
  • @cumulus/ingest/aws/S3
  • @cumulus/ingest/aws/StepFunction.getExecution()
  • @cumulus/ingest/aws/StepFunction.pullEvent()
  • @cumulus/ingest/consumer.Consume
  • @cumulus/ingest/granule/Ingest.getBucket()

Deprecated

@cmrjs/ingestConcept, instead use the CMR object methods. @cmrjs/CMR.ingestGranule or @cmrjs/CMR.ingestCollection @cmrjs/searchConcept, instead use the CMR object methods. @cmrjs/CMR.searchGranules or @cmrjs/CMR.searchCollections @cmrjs/deleteConcept, instead use the CMR object methods. @cmrjs/CMR.deleteGranule or @cmrjs/CMR.deleteCollection