@cumulus/checksum

@cumulus/checksum

What is Cumulus?

Cumulus is a cloud-based data ingest, archive, distribution and management prototype for NASA's future Earth science data streams.

Cumulus Documentation

Checksum

The @cumulus/checksum library provides checksum functionality used by Cumulus packages and tasks. Currently the supported input includes file streams, and supported checksum algorithms include cksum and the algorithms available to the crypto package, as documented here.

Usage

const fs = require('fs');
const { generateChecksumFromStream } = require('@cumulus/checksum');

const stream = fs.createReadStream('myDataFile.hdf');
const myCksum = generateChecksumFromStream('cksum', stream);

Changelog

[v1.14.2] - 2019-10-08

BREAKING CHANGES

Your Cumulus Message Adapter version should be pinned to v1.0.13 or lower in your app/config.yml using message_adapter_version: v1.0.13 OR you should use the workflow migration steps below to work with CMA v1.1.1+.

• CUMULUS-1394 - The implementation of the SfSnsReport Lambda requires additional environment variables for integration with the new ingest notification SNS topics. Therefore, you must update the definition of SfSnsReport in your lambdas.yml like so:

SfSnsReport:
  handler: index.handler
  timeout: 300
  source: node_modules/@cumulus/sf-sns-report/dist
  tables:
    - ExecutionsTable
  envs:
    execution_sns_topic_arn:
      function: Ref
      value: reportExecutionsSns
    granule_sns_topic_arn:
      function: Ref
      value: reportGranulesSns
    pdr_sns_topic_arn:
      function: Ref
      value: reportPdrsSns

• CUMULUS-1447 - The newest release of the Cumulus Message Adapter (v1.1.1) requires that parameterized configuration be used for remote message functionality. Once released, Kes will automatically bring in CMA v1.1.1 without additional configuration.

  Migration instructions Oversized messages are no longer written to S3 automatically. In order to utilize remote messaging functionality, configure a ReplaceConfig AWS Step Function parameter on your CMA task:

  ParsePdr:
    Parameters:
      cma:
        event.$: "$"
        ReplaceConfig:
          FullMessage: true
  

  Accepted fields in ReplaceConfig include MaxSize, FullMessage, Path and TargetPath. See https://github.com/nasa/cumulus-message-adapter/blob/master/CONTRACT.md#remote-message-configuration for full details.

  As this change is backward compatible in Cumulus Core, users wishing to utilize the previous version of the CMA may opt to transition to using a CMA lambda layer, or set message_adapter_version in their configuration to a version prior to v1.1.0.

PLEASE NOTE

• CUMULUS-1394 - Ingest notifications are now provided via 3 separate SNS topics for executions, granules, and PDRs, instead of a single sftracker SNS topic. Whereas the sftracker SNS topic received a full Cumulus execution message, the new topics all receive generated records for the given object. The new topics are only published to if the given object exists for the current execution. For a given execution/granule/PDR, two messages will be received by each topic: one message indicating that ingest is running and another message indicating that ingest has completed or failed. The new SNS topics are:

  • reportExecutions - Receives 1 message per execution
  • reportGranules - Receives 1 message per granule in an execution
  • reportPdrs - Receives 1 message per PDR

Added

• CUMULUS-639

  • Adds SAML JWT and launchpad token authentication to Cumulus API (configurable)
    • NOTE to authenticate with Launchpad ensure your launchpad user_id is in the <prefix>-UsersTable
    • when Cumulus configured to protect API via Launchpad:
      • New endpoints
        • GET /saml/login - starting point for SAML SSO creates the login request url and redirects to the SAML Identity Provider Service (IDP)
        • POST /saml/auth - SAML Assertion Consumer Service. POST receiver from SAML IDP. Validates response, logs the user in, and returns a SAML-based JWT.
    • Disabled endpoints
      • POST /refresh
      • Changes authorization worklow:
      • ensureAuthorized now presumes the bearer token is a JWT and tries to validate. If the token is malformed, it attempts to validate the token against Launchpad. This allows users to bring their own token as described here https://wiki.earthdata.nasa.gov/display/CUMULUS/Cumulus+API+with+Launchpad+Authentication. But it also allows dashboard users to manually authenticate via Launchpad SAML to receive a Launchpad-based JWT.

• CUMULUS-1394

  • Added Granule.generateGranuleRecord() method to granules model to generate a granule database record from a Cumulus execution message
  • Added Pdr.generatePdrRecord() method to PDRs model to generate a granule database record from a Cumulus execution message
  • Added helpers to @cumulus/common/message:
    • getMessageExecutionName() - Get the execution name from a Cumulus execution message
    • getMessageStateMachineArn() - Get the state machine ARN from a Cumulus execution message
    • getMessageExecutionArn() - Get the execution ARN for a Cumulus execution message
    • getMessageGranules() - Get the granules from a Cumulus execution message, if any.
  • Added @cumulus/common/cloudwatch-event/isFailedSfStatus() to determine if a Step Function status from a Cloudwatch event is a failed status

Changed

• CUMULUS-1308

  • HTTP PUT of a Collection, Provider, or Rule via the Cumulus API now performs full replacement of the existing object with the object supplied in the request payload. Previous behavior was to perform a modification (partial update) by merging the existing object with the (possibly partial) object in the payload, but this did not conform to the HTTP standard, which specifies PATCH as the means for modifications rather than replacements.

• CUMULUS-1375

  • Migrate Cumulus from deprecated Elasticsearch JS client to new, supported one in @cumulus/api

• CUMULUS-1485 Update @cumulus/cmr-client to return error message from CMR for validation failures.

• CUMULUS-1394

  • Renamed Execution.generateDocFromPayload() to Execution.generateRecord() on executions model. The method generates an execution database record from a Cumulus execution message.

• CUMULUS-1432

  • logs endpoint takes the level parameter as a string and not a number
  • Elasticsearch term query generation no longer converts numbers to boolean

• CUMULUS-1447

  • Consolidated all remote message handling code into @common/aws
  • Update remote message code to handle updated CMA remote message flags
  • Update example SIPS workflows to utilize Parameterized CMA configuration

• CUMULUS-1448 Refactor workflows that are mutating cumulus_meta to utilize meta field

• CUMULUS-1451

  • Elasticsearch cluster setting auto_create_index will be set to false. This had been causing issues in the bootstrap lambda on deploy.

• CUMULUS-1456

  • @cumulus/api endpoints default error handler uses boom package to format errors, which is consistent with other API endpoint errors.

Fixed

• CUMULUS-1432 logs endpoint filter correctly filters logs by level
• CUMULUS-1484 useMessageAdapter now does not set CUMULUS_MESSAGE_ADAPTER_DIR when true

Removed

• CUMULUS-1394
  • Removed sfTracker SNS topic. Replaced by three new SNS topics for granule, execution, and PDR ingest notifications.
  • Removed unused functions from @cumulus/common/aws:
    • getGranuleS3Params()
    • setGranuleStatus()