@cumulus/checksum

@cumulus/checksum

What is Cumulus?

Cumulus is a cloud-based data ingest, archive, distribution and management prototype for NASA's future Earth science data streams.

Cumulus Documentation

Checksum

The @cumulus/checksum library provides checksum functionality used by Cumulus packages and tasks. Currently the supported input includes file streams, and supported checksum algorithms include cksum and the algorithms available to the crypto package, as documented here.

Usage

const fs = require('fs');
const { generateChecksumFromStream } = require('@cumulus/checksum');

const stream = fs.createReadStream('myDataFile.hdf');
const myCksum = generateChecksumFromStream('cksum', stream);

Changelog

[v1.13.0] - 2019-5-20

PLEASE NOTE

CUMULUS-802 added some additional IAM permissions to support ECS autoscaling, so you will have to redeploy your IAM stack. As a result of the changes for CUMULUS-1193, CUMULUS-1264, and CUMULUS-1310, you must delete your existing stacks (except IAM) before deploying this version of Cumulus. If running Cumulus within a VPC and extended downtime is acceptable, we recommend doing this at the end of the day to allow AWS backend resources and network interfaces to be cleaned up overnight.

BREAKING CHANGES

• CUMULUS-1228

  • The default AMI used by ECS instances is now an NGAP-compliant AMI. This will be a breaking change for non-NGAP deployments. If you do not deploy to NGAP, you will need to find the AMI ID of the most recent Amazon ECS-optimized AMI, and set the ecs.amiid property in your config. Instructions for finding the most recent NGAP AMI can be found using these instructions.

• CUMULUS-1310

  • Database resources (DynamoDB, ElasticSearch) have been moved to an independent db stack. Migrations for this version will need to be user-managed. (e.g. elasticsearch and dynamoDB). Order of stack deployment is iam -> db -> app.
  • All stacks can now be deployed using a single config.yml file, i.e.: kes cf deploy --kes-folder app --template node_modules/@cumulus/deployment/[iam|db|app] [...] Backwards-compatible. For development, please re-run npm run bootstrap to build new kes overrides. Deployment docs have been updated to show how to deploy a single-config Cumulus instance.
  • params have been moved: Nest params fields under app, db or iam to override all Parameters for a particular stack's cloudformation template. Backwards-compatible with multi-config setups.
  • stackName and stackNameNoDash have been retired. Use prefix and prefixNoDash instead.
  • The iams section in app/config.yml IAM roles has been deprecated as a user-facing parameter, unless your IAM role ARNs do not match the convention shown in @cumulus/deployment/app/config.yml
  • The vpc.securityGroup will need to be set with a pre-existing security group ID to use Cumulus in a VPC. Must allow inbound HTTP(S) (Port 443).

• CUMULUS-1212

  • @cumulus/post-to-cmr will now fail if any granules being processed are missing a metadata file. You can set the new config option skipMetaCheck to true to pass post-to-cmr without a metadata file.

• CUMULUS-1232

  • @cumulus/sync-granule will no longer silently pass if no checksum data is provided. It will use input from the granule object to:
    • Verify checksum if checksumType and checksumValue are in the file record OR a checksum file is provided (throws InvalidChecksum on fail), else log warning that no checksum is available.
    • Then, verify synced S3 file size if file.size is in the file record (throws UnexpectedFileSize on fail), else log warning that no file size is available.
    • Pass the step.

• CUMULUS-1264

  • The Cloudformation templating and deployment configuration has been substantially refactored.
    • CumulusApiDefault nested stack resource has been renamed to CumulusApiDistribution
    • CumulusApiV1 nested stack resource has been renamed to CumulusApiBackend
  • The urs: true config option for when defining your lambdas (e.g. in lambdas.yml) has been deprecated. There are two new options to replace it:
    • urs_redirect: 'token': This will expose a TOKEN_REDIRECT_ENDPOINT environment variable to your lambda that references the /token endpoint on the Cumulus backend API
    • urs_redirect: 'distribution': This will expose a DISTRIBUTION_REDIRECT_ENDPOINT environment variable to your lambda that references the /redirect endpoint on the Cumulus distribution API

• CUMULUS-1193

  • The elasticsearch instance is moved behind the VPC.
  • Your account will need an Elasticsearch Service Linked role. This is a one-time setup for the account. You can follow the instructions to use the AWS console or AWS CLI here or use the following AWS CLI command: aws iam create-service-linked-role --aws-service-name es.amazonaws.com

• CUMULUS-802

  • ECS maxInstances must be greater than minInstances. If you use defaults, no change is required.

• CUMULUS-1269

  • Brought Cumulus data models in line with CNM JSON schema:
    • Renamed file object fileType field to type
    • Renamed file object fileSize field to size
    • Renamed file object checksumValue field to checksum where not already done.
    • Added ancillary and linkage type support to file objects.

Added

• CUMULUS-799

  • Added an S3 Access Metrics package which will take S3 Server Access Logs and write access metrics to CloudWatch

• CUMULUS-1242 - Added sqs2sfThrottle lambda. The lambda reads SQS messages for queued executions and uses semaphores to only start new executions if the maximum number of executions defined for the priority key (cumulus_meta.priorityKey) has not been reached. Any SQS messages that are read but not used to start executions remain in the queue.

• CUMULUS-1240

  • Added sfSemaphoreDown lambda. This lambda receives SNS messages and for each message it decrements the semaphore used to track the number of running executions if:
    • the message is for a completed/failed workflow AND
    • the message contains a level of priority (cumulus_meta.priorityKey)
  • Added sfSemaphoreDown lambda as a subscriber to the sfTracker SNS topic

• CUMULUS-1265

  • Added apiConfigs configuration option to configure API Gateway to be private
  • All internal lambdas configured to run inside the VPC by default
  • Removed references to NoVpc lambdas from documentation and example folder.

• CUMULUS-802

  • Adds autoscaling of ECS clusters
  • Adds autoscaling of ECS services that are handling StepFunction activities

Changed

• Updated @cumulus/ingest/http/httpMixin.list() to trim trailing spaces on discovered filenames

• CUMULUS-1310

  • Database resources (DynamoDB, ElasticSearch) have been moved to an independent db stack. This will enable future updates to avoid affecting database resources or requiring migrations. Migrations for this version will need to be user-managed. (e.g. elasticsearch and dynamoDB). Order of stack deployment is iam -> db -> app.
  • All stacks can now be deployed using a single config.yml file, i.e.: kes cf deploy --kes-folder app --template node_modules/@cumulus/deployment/[iam|db|app] [...] Backwards-compatible. Please re-run npm run bootstrap to build new kes overrides. Deployment docs have been updated to show how to deploy a single-config Cumulus instance.
  • params fields should now be nested under the stack key (i.e. app, db or iam) to provide Parameters for a particular stack's cloudformation template, for use with single-config instances. Keys must match the name of the deployment package folder (app, db, or iam). Backwards-compatible with multi-config setups.
  • stackName and stackNameNoDash have been retired as user-facing config parameters. Use prefix and prefixNoDash instead. This will be used to create stack names for all stacks in a single-config use case. stackName may still be used as an override in multi-config usage, although this is discouraged. Warning: overriding the db stack's stackName will require you to set dbStackName in your app/config.yml. This parameter is required to fetch outputs from the db stack to reference in the app stack.
  • The iams section in app/config.yml IAM roles has been retired as a user-facing parameter, unless your IAM role ARNs do not match the convention shown in @cumulus/deployment/app/config.yml In that case, overriding iams in your own config is recommended.
  • iam and db cloudformation.yml file names will have respective prefixes (e.g iam.cloudformation.yml).
  • Cumulus will now only attempt to create reconciliation reports for buckets of the private, public and protected types.
  • Cumulus will no longer set up its own security group. To pass a pre-existing security group for in-VPC deployments as a parameter to the Cumulus template, populate vpc.securityGroup in config.yml. This security group must allow inbound HTTP(S) traffic (Port 443). SSH traffic (Port 22) must be permitted for SSH access to ECS instances.
  • Deployment docs have been updated with examples for the new deployment model.

• CUMULUS-1236

  • Moves access to public files behind the distribution endpoint. Authentication is not required, but direct http access has been disallowed.

• CUMULUS-1223

  • Adds unauthenticated access for public bucket files to the Distribution API. Public files should be requested the same way as protected files, but for public files a redirect to a self-signed S3 URL will happen without requiring authentication with Earthdata login.

• CUMULUS-1232

  • Unifies duplicate handling in ingest/granule.handleDuplicateFile for maintainability.
  • Changed ingest/granule.ingestFile and move-granules/index.moveFileRequest to use new function.
  • Moved file versioning code to ingest/granule.moveGranuleFileWithVersioning
  • ingest/granule.verifyFile now also tests file.size for verification if it is in the file record and throws UnexpectedFileSize error for file size not matching input.
  • ingest/granule.verifyFile logs warnings if checksum and/or file size are not available.

• CUMULUS-1193

  • Moved reindex CLI functionality to an API endpoint. See API docs

• CUMULUS-1207

  • No longer disable lambda event source mappings when disabling a rule

Fixed

• Updated Lerna publish script so that published Cumulus packages will pin their dependencies on other Cumulus packages to exact versions (e.g. 1.12.1 instead of ^1.12.1)

• CUMULUS-1203

  • Fixes IAM template's use of intrinsic functions such that IAM template overrides now work with kes

• CUMULUS-1268

  • Deployment will not fail if there are no ES alarms or ECS services