@cumulus/common - npm Package Versions

[v1.12.1] - 2019-4-8

[v1.12.0] - 2019-4-4

Note: There was an issue publishing 1.12.0. Upgrade to 1.12.1.

BREAKING CHANGES

• CUMULUS-1139

  • granule.applyWorkflow uses the new-style granule record as input to workflows.

• CUMULUS-1171

  • Fixed provider handling in the API to make it consistent between protocols. NOTE: This is a breaking change. When applying this upgrade, users will need to:
    1. Disable all workflow rules
    2. Update any http or https providers so that the host field only contains a valid hostname or IP address, and the port field contains the provider port.
    3. Perform the deployment
    4. Re-enable workflow rules

• CUMULUS-1176:

  • @cumulus/move-granules input expectations have changed. @cumulus/files-to-granules is a new intermediate task to perform input translation in the old style. See the Added and Changed sections of this release changelog for more information.

• CUMULUS-670

  • The behavior of ParsePDR and related code has changed in this release. PDRs with FILE_TYPEs that do not conform to the PDR ICD (+ TGZ) (https://cdn.earthdata.nasa.gov/conduit/upload/6376/ESDS-RFC-030v1.0.pdf) will fail to parse.

• CUMULUS-1208

  • The granule object input to @cumulus/queue-granules will now be added to ingest workflow messages as is. In practice, this means that if you are using @cumulus/queue-granules to trigger ingest workflows and your granule objects input have invalid properties, then your ingest workflows will fail due to schema validation errors.

Added

• CUMULUS-777

  • Added new cookbook entry on configuring Cumulus to track ancillary files.

• CUMULUS-1183

  • Kes overrides will now abort with a warning if a workflow step is configured without a corresponding lambda configuration

• CUMULUS-1223

  • Adds convenience function @cumulus/common/bucketsConfigJsonObject for fetching stack's bucket configuration as an object.

• CUMULUS-853

  • Updated FakeProcessing example lambda to include option to generate fake browse
  • Added feature documentation for ancillary metadata export, a new cookbook entry describing a workflow with ancillary metadata generation(browse), and related task definition documentation

• CUMULUS-805

  • Added a CloudWatch alarm to check running ElasticSearch instances, and a CloudWatch dashboard to view the health of ElasticSearch
  • Specify AWS_REGION in .env to be used by deployment script

• CUMULUS-803

  • Added CloudWatch alarms to check running tasks of each ECS service, and add the alarms to CloudWatch dashboard

• CUMULUS-670

  • Added Ancillary Metadata Export feature (see https://nasa.github.io/cumulus/docs/features/ancillary_metadata for more information)
  • Added new Collection file parameter "fileType" that allows configuration of workflow granule file fileType

• CUMULUS-1184 - Added kes logging output to ensure we always see the state machine reference before failures due to configuration

• CUMULUS-1105 - Added a dashboard endpoint to serve the dashboard from an S3 bucket

• CUMULUS-1199 - Moves s3credentials endpoint from the backend to the distribution API.

• CUMULUS-666

  • Added @api/endpoints/s3credentials to allow EarthData Login authorized users to retrieve temporary security credentials for same-region direct S3 access.

• CUMULUS-671

  • Added @packages/integration-tests/api/distribution/getDistributionApiS3SignedUrl() to return the S3 signed URL for a file protected by the distribution API

• CUMULUS-672

  • Added cmrMetadataFormat and cmrConceptId to output for individual granules from @cumulus/post-to-cmr. cmrMetadataFormat will be read from the cmrMetadataFormat generated for each granule in @cumulus/cmrjs/publish2CMR()
  • Added helpers to @packages/integration-tests/api/distribution:
    • getDistributionApiFileStream() returns a stream to download files protected by the distribution API
    • getDistributionFileUrl() constructs URLs for requesting files from the distribution API

• CUMULUS-1185 @cumulus/api/models/Granule.removeGranuleFromCmrByGranule to replace @cumulus/api/models/Granule.removeGranuleFromCmr and use the Granule UR from the CMR metadata to remove the granule from CMR

• CUMULUS-1101

  • Added new @cumulus/checksum package. This package provides functions to calculate and validate checksums.
  • Added new checksumming functions to @cumulus/common/aws: calculateS3ObjectChecksum and validateS3ObjectChecksum, which depend on the checksum package.

• CUMULUS-1171

  • Added @cumulus/common API documentation to packages/common/docs/API.md
  • Added an npm run build-docs task to @cumulus/common
  • Added @cumulus/common/string#isValidHostname()
  • Added @cumulus/common/string#match()
  • Added @cumulus/common/string#matches()
  • Added @cumulus/common/string#toLower()
  • Added @cumulus/common/string#toUpper()
  • Added @cumulus/common/URLUtils#buildURL()
  • Added @cumulus/common/util#isNil()
  • Added @cumulus/common/util#isNull()
  • Added @cumulus/common/util#isUndefined()
  • Added @cumulus/common/util#negate()

• CUMULUS-1176

  • Added new @cumulus/files-to-granules task to handle converting file array output from cumulus-process tasks into granule objects. Allows simplification of @cumulus/move-granules and @cumulus/post-to-cmr, see Changed section for more details.

• CUMULUS-1151 Compare the granule holdings in CMR with Cumulus' internal data store

• CUMULUS-1152 Compare the granule file holdings in CMR with Cumulus' internal data store

Changed

• CUMULUS-1216 - Updated @cumulus/ingest/granule/ingestFile to download files to expected staging location.

• CUMULUS-1208 - Updated @cumulus/ingest/queue/enqueueGranuleIngestMessage() to not transform granule object passed to it when building an ingest message

• CUMULUS-1198 - @cumulus/ingest no longer enforces any expectations about whether provider_path contains a leading slash or not.

• CUMULUS-1170

  • Update scripts and docs to use npm instead of yarn
  • Use package-lock.json files to ensure matching versions of npm packages
  • Update CI builds to use npm ci instead of npm install

• CUMULUS-670

  • Updated ParsePDR task to read standard PDR types+ (+ tgz as an external customer requirement) and add a fileType to granule-files on Granule discovery
  • Updated ParsePDR to fail if unrecognized type is used
  • Updated all relevant task schemas to include granule->files->filetype as a string value
  • Updated tests/test fixtures to include the fileType in the step function/task inputs and output validations as needed
  • Updated MoveGranules task to handle incoming configuration with new "fileType" values and to add them as appropriate to the lambda output.
  • Updated DiscoverGranules step/related workflows to read new Collection file parameter fileType that will map a discovered file to a workflow fileType
  • Updated CNM parser to add the fileType to the defined granule file fileType on ingest and updated integration tests to verify/validate that behavior
  • Updated generateEcho10XMLString in cmr-utils.js to use a map/related library to ensure order as CMR requires ordering for their online resources.
  • Updated post-to-cmr task to appropriately export CNM filetypes to CMR in echo10/UMM exports

• CUMULUS-1139 - Granules stored in the API contain a files property. That schema has been greatly simplified and now better matches the CNM format.

  • The name property has been renamed to fileName.
  • The filepath property has been renamed to key.
  • The checksumValue property has been renamed to checksum.
  • The path property has been removed.
  • The url_path property has been removed.
  • The filename property (which contained an s3:// URL) has been removed, and the bucket and key properties should be used instead. Any requests sent to the API containing a granule.files[].filename property will be rejected, and any responses coming back from the API will not contain that filename property.
  • A source property has been added, which is a URL indicating the original source of the file.
  • @cumulus/ingest/granule.moveGranuleFiles() no longer includes a filename field in its output. The bucket and key fields should be used instead.

• CUMULUS-672

  • Changed @cumulus/integration-tests/api/EarthdataLogin.getEarthdataLoginRedirectResponse to @cumulus/integration-tests/api/EarthdataLogin.getEarthdataAccessToken. The new function returns an access response from Earthdata login, if successful.
  • @cumulus/integration-tests/cmr/getOnlineResources now accepts an object of options, including cmrMetadataFormat. Based on the cmrMetadataFormat, the function will correctly retrieve the online resources for each metadata format (ECHO10, UMM-G)

• CUMULUS-1101

  • Moved @cumulus/common/file/getFileChecksumFromStream into @cumulus/checksum, and renamed it to generateChecksumFromStream. This is a breaking change for users relying on @cumulus/common/file/getFileChecksumFromStream.
  • Refactored @cumulus/ingest/Granule to depend on new common/aws checksum functions and remove significantly present checksumming code.
    • Deprecated @cumulus/ingest/granule.validateChecksum. Replaced with @cumulus/ingest/granule.verifyFile.
    • Renamed granule.getChecksumFromFile to granule.retrieveSuppliedFileChecksumInformation to be more accurate.
  • Deprecated @cumulus/common/aws.checksumS3Objects. Use @cumulus/common/aws.calculateS3ObjectChecksum instead.

• CUMULUS-1171

  • Fixed provider handling in the API to make it consistent between protocols. Before this change, FTP providers were configured using the host and port properties. HTTP providers ignored port and protocol, and stored an entire URL in the host property. Updated the API to only accept valid hostnames or IP addresses in the provider.host field. Updated ingest code to properly build HTTP and HTTPS URLs from provider.protocol, provider.host, and provider.port.
  • The default provider port was being set to 21, no matter what protocol was being used. Removed that default.

• CUMULUS-1176

  • @cumulus/move-granules breaking change: Input to move-granules is now expected to be in the form of a granules object (i.e. { granules: [ { ... }, { ... } ] }); For backwards compatibility with array-of-files outputs from processing steps, use the new @cumulus/files-to-granules task as an intermediate step. This task will perform the input translation. This change allows move-granules to be simpler and behave more predictably. config.granuleIdExtraction and config.input_granules are no longer needed/used by move-granules.
  • @cumulus/post-to-cmr: config.granuleIdExtraction is no longer needed/used by post-to-cmr.

• CUMULUS-1174

  • Better error message and stacktrace for S3KeyPairProvider error reporting.

Fixed

• CUMULUS-1218 Reconciliation report will now scan only completed granules.
• @cumulus/api files and granules were not getting indexed correctly because files indexing was failing in db-indexer
• @cumulus/deployment A bug in the Cloudformation template was preventing the API from being able to be launched in a VPC, updated the IAM template to give the permissions to be able to run the API in a VPC

Deprecated

• @cumulus/api/models/Granule.removeGranuleFromCmr, instead use @cumulus/api/models/Granule.removeGranuleFromCmrByGranule
• @cumulus/ingest/granule.validateChecksum, instead use @cumulus/ingest/granule.verifyFile
• @cumulus/common/aws.checksumS3Objects, instead use @cumulus/common/aws.calculateS3ObjectChecksum
• @cumulus/cmrjs: getGranuleId and getCmrFiles are deprecated due to changes in input handling.

[v1.11.3] - 2019-3-5

Added

• CUMULUS-1187 - Added @cumulus/ingest/granule/duplicateHandlingType() to determine how duplicate files should be handled in an ingest workflow

Fixed

• CUMULUS-1187 - workflows not respecting the duplicate handling value specified in the collection
• Removed refreshToken schema requirement for OAuth

[v1.11.2] - 2019-2-15

Added

• CUMULUS-1169
  • Added a @cumulus/common/StepFunctions module. It contains functions for querying the AWS StepFunctions API. These functions have the ability to retry when a ThrottlingException occurs.
  • Added @cumulus/common/aws.retryOnThrottlingException(), which will wrap a function in code to retry on ThrottlingExceptions.
  • Added @cumulus/common/test-utils.throttleOnce(), which will cause a function to return a ThrottlingException the first time it is called, then return its normal result after that.
• CUMULUS-1103 Compare the collection holdings in CMR with Cumulus' internal data store
• CUMULUS-1099 Add support for UMMG JSON metadata versions > 1.4.
  • If a version is found in the metadata object, that version is used for processing and publishing to CMR otherwise, version 1.4 is assumed.
• CUMULUS-678
  • Added support for UMMG json v1.4 metadata files. reconcileCMRMetadata added to @cumulus/cmrjs to update metadata record with new file locations. @cumulus/common/errors adds two new error types CMRMetaFileNotFound and InvalidArgument. @cumulus/common/test-utils adds new function randomId to create a random string with id to help in debugging. @cumulus/common/BucketsConfig adds a new helper class BucketsConfig for working with bucket stack configuration and bucket names. @cumulus/common/aws adds new function s3PutObjectTagging as a convenience for the aws s3().putObjectTagging function. @cumulus/cmrjs Adds: - isCMRFile - Identify an echo10(xml) or UMMG(json) metadata file. - metadataObjectFromCMRFile Read and parse CMR XML file from s3. - updateCMRMetadata Modify a cmr metadata (xml/json) file with updated information. - publish2CMR Posts XML or UMMG CMR data to CMR service. - reconcileCMRMetadata Reconciles cmr metadata file after a file moves.
• Adds some ECS and other permissions to StepRole to enable running ECS tasks from a workflow
• Added Apache logs to cumulus api and distribution lambdas
• CUMULUS-1119 - Added @cumulus/integration-tests/api/EarthdataLogin.getEarthdataLoginRedirectResponse helper for integration tests to handle login with Earthdata and to return response from redirect to Cumulus API
• CUMULUS-673 Added @cumulus/common/file/getFileChecksumFromStream to get file checksum from a readable stream

Fixed

• CUMULUS-1123
  • Cloudformation template overrides now work as expected

Changed

• CUMULUS-1169

  • Deprecated the @cumulus/common/step-functions module.
  • Updated code that queries the StepFunctions API to use the retry-enabled functions from @cumulus/common/StepFunctions

• CUMULUS-1121

  • Schema validation is now strongly enforced when writing to the database. Additional properties are not allowed and will result in a validation error.

• CUMULUS-678 tasks/move-granules simplified and refactored to use functionality from cmrjs. ingest/granules.moveGranuleFiles now just moves granule files and returns a list of the updated files. Updating metadata now handled by @cumulus/cmrjs/reconcileCMRMetadata. move-granules.updateGranuleMetadata refactored and bugs fixed in the case of a file matching multiple collection.files.regexps. getCmrXmlFiles simplified and now only returns an object with the cmrfilename and the granuleId. @cumulus/test-processing - test processing task updated to generate UMM-G metadata

• CUMULUS-1043

  • @cumulus/api now uses express as the API engine.
  • All @cumulus/api endpoints on ApiGateway are consolidated to a single endpoint the uses {proxy+} definition.
  • All files under packages/api/endpoints along with associated tests are updated to support express's request and response objects.
  • Replaced environment variables internal, bucket and systemBucket with system_bucket.
  • Update @cumulus/integration-tests to work with updated cumulus-api express endpoints

• @cumulus/integration-tests - buildAndExecuteWorkflow and buildWorkflow updated to take a meta param to allow for additional fields to be added to the workflow meta

• CUMULUS-1049 Updated Retrieve Execution Status API in @cumulus/api: If the execution doesn't exist in Step Function API, Cumulus API returns the execution status information from the database.

• CUMULUS-1119

  • Renamed DISTRIBUTION_URL environment variable to DISTRIBUTION_ENDPOINT
  • Renamed DEPLOYMENT_ENDPOINT environment variable to DISTRIBUTION_REDIRECT_ENDPOINT
  • Renamed API_ENDPOINT environment variable to TOKEN_REDIRECT_ENDPOINT

Removed

• Functions deprecated before 1.11.0:
  • @cumulus/api/models/base: static Manager.createTable() and static Manager.deleteTable()
  • @cumulus/ingest/aws/S3
  • @cumulus/ingest/aws/StepFunction.getExecution()
  • @cumulus/ingest/aws/StepFunction.pullEvent()
  • @cumulus/ingest/consumer.Consume
  • @cumulus/ingest/granule/Ingest.getBucket()

Deprecated

@cmrjs/ingestConcept, instead use the CMR object methods. @cmrjs/CMR.ingestGranule or @cmrjs/CMR.ingestCollection @cmrjs/searchConcept, instead use the CMR object methods. @cmrjs/CMR.searchGranules or @cmrjs/CMR.searchCollections @cmrjs/deleteConcept, instead use the CMR object methods. @cmrjs/CMR.deleteGranule or @cmrjs/CMR.deleteCollection

[v1.11.1] - 2018-12-18

Please Note

• Ensure your app/config.yml has a clientId specified in the cmr section. This will allow CMR to identify your requests for better support and metrics.
  • For an example, please see the example config.

Added

• Added a /tokenDelete endpoint in @cumulus/api to delete access token records

Changed

• CUMULUS-678 @cumulus/ingest/crypto moved and renamed to @cumulus/common/key-pair-provider @cumulus/ingest/aws function: KMSDecryptionFailed and class: KMS extracted and moved to @cumulus/common and KMS is exported as KMSProvider from @cumulus/common/key-pair-provider @cumulus/ingest/granule functions: publish, getGranuleId, getXMLMetadataAsString, getMetadataBodyAndTags, parseXmlString, getCmrXMLFiles, postS3Object, contructOnlineAccessUrls, updateMetadata, extracted and moved to @cumulus/cmrjs getGranuleId, getCmrXMLFiles, publish, updateMetadata removed from @cumulus/ingest/granule and added to @cumulus/cmrjs; updateMetadata renamed updateCMRMetadata. @cumulus/ingest test files renamed.
• CUMULUS-1070
  • Add 'Client-Id' header to all @cumulus/cmrjs requests (made via searchConcept, ingestConcept, and deleteConcept).
  • Updated cumulus/example/app/config.yml entry for cmr.clientId to use stackName for easier CMR-side identification.

[v1.11.0] - 2018-11-30

Please Note

• Redeploy IAM roles:
  • CUMULUS-817 includes a migration that requires reconfiguration/redeployment of IAM roles. Please see the upgrade instructions for more information.
  • CUMULUS-977 includes a few new SNS-related permissions added to the IAM roles that will require redeployment of IAM roles.
• cumulus-message-adapter v1.0.13+ is required for @cumulus/api granule reingest API to work properly. The latest version should be downloaded automatically by kes.
• A TOKEN_SECRET value (preferably 256-bit for security) must be added to .env to securely sign JWTs used for authorization in @cumulus/api

Changed

• CUUMULUS-1000 - Distribution endpoint now persists logins, instead of redirecting to Earthdata Login on every request
• CUMULUS-783 CUMULUS-790 - Updated @cumulus/sync-granule and @cumulus/move-granules tasks to always overwrite existing files for manually-triggered reingest.
• CUMULUS-906 - Updated @cumulus/api granule reingest API to
  • add reingestGranule: true and forceDuplicateOverwrite: true to Cumulus message cumulus_meta.cumulus_context field to indicate that the workflow is a manually triggered re-ingest.
  • return warning message to operator when duplicateHandling is not replace
  • cumulus-message-adapter v1.0.13+ is required.
• CUMULUS-793 - Updated the granule move PUT request in @cumulus/api to reject the move with a 409 status code if one or more of the files already exist at the destination location
• Updated @cumulus/helloworld to use S3 to store state for pass on retry tests
• Updated @cumulus/ingest:
  • [Required for MAAP] http.js#list will now find links with a trailing whitespace
  • Removed code from granule.js which looked for files in S3 using { Bucket: discoveredFile.bucket, Key: discoveredFile.name }. This is obsolete since @cumulus/ingest uses a file-staging and constructCollectionId() directory prefixes by default.
• CUMULUS-989
  • Updated @cumulus/api to use JWT (JSON Web Token) as the transport format for API authorization tokens and to use JWT verification in the request authorization
  • Updated /token endpoint in @cumulus/api to return tokens as JWTs
  • Added a /refresh endpoint in @cumulus/api to request new access tokens from the OAuth provider using the refresh token
  • Added refreshAccessToken to @cumulus/api/lib/EarthdataLogin to manage refresh token requests with the Earthdata OAuth provider

Added

• CUMULUS-1050
  • Separated configuration flags for originalPayload/finalPayload cleanup such that they can be set to different retention times
• CUMULUS-798
  • Added daily Executions cleanup CloudWatch event that triggers cleanExecutions lambda
  • Added cleanExecutions lambda that removes finalPayload/originalPayload field entries for records older than configured timeout value (execution_payload_retention_period), with a default of 30 days
• CUMULUS-815/816
  • Added 'originalPayload' and 'finalPayload' fields to Executions table
  • Updated Execution model to populate originalPayload with the execution payload on record creation
  • Updated Execution model code to populate finalPayload field with the execution payload on execution completion
  • Execution API now exposes the above fields
• CUMULUS-977
  • Rename kinesisConsumer to messageConsumer as it handles both Kinesis streams and SNS topics as of this version.
  • Add sns-type rule support. These rules create a subscription between an SNS topic and the messageConsumer. When a message is received, messageConsumer is triggered and passes the SNS message (JSON format expected) in its entirety to the workflow in the payload field of the Cumulus message. For more information on sns-type rules, see the documentation.
• CUMULUS-975
  • Add KinesisInboundEventLogger and KinesisOutboundEventLogger API lambdas. These lambdas are utilized to dump incoming and outgoing ingest workflow kinesis streams to cloudwatch for analytics in case of AWS/stream failure.
  • Update rules model to allow tracking of log_event ARNs related to Rule event logging. Kinesis rule types will now automatically log incoming events via a Kinesis event triggered lambda. CUMULUS-975-migration-4
  • Update migration code to require explicit migration names per run
  • Added migration_4 to migrate/update existing Kinesis rules to have a log event mapping
  • Added new IAM policy for migration lambda
• CUMULUS-775
  • Adds a instance metadata endpoint to the @cumulus/api package.
  • Adds a new convenience function hostId to the @cumulus/cmrjs to help build environment specific cmr urls.
  • Fixed @cumulus/cmrjs.searchConcept to search and return CMR results.
  • Modified @cumulus/cmrjs.CMR.searchGranule and @cumulus/cmrjs.CMR.searchCollection to include CMR's provider as a default parameter to searches.
• CUMULUS-965
  • Add @cumulus/test-data.loadJSONTestData(), @cumulus/test-data.loadTestData(), and @cumulus/test-data.streamTestData() to safely load test data. These functions should be used instead of using require() to load test data, which could lead to tests interfering with each other.
  • Add a @cumulus/common/util/deprecate() function to mark a piece of code as deprecated
• CUMULUS-986
  • Added waitForTestExecutionStart to @cumulus/integration-tests
• CUMULUS-919
  • In @cumulus/deployment, added support for NGAP permissions boundaries for IAM roles with useNgapPermissionBoundary flag in iam/config.yml. Defaults to false.

Fixed

• Fixed a bug where FTP sockets were not closed after an error, keeping the Lambda function active until it timed out [CUMULUS-972]
• CUMULUS-656
  • The API will no longer allow the deletion of a provider if that provider is referenced by a rule
  • The API will no longer allow the deletion of a collection if that collection is referenced by a rule
• Fixed a bug where @cumulus/sf-sns-report was not pulling large messages from S3 correctly.

Deprecated

• @cumulus/ingest/aws/StepFunction.pullEvent(). Use @cumulus/common/aws.pullStepFunctionEvent().
• @cumulus/ingest/consumer.Consume due to unpredictable implementation. Use @cumulus/ingest/consumer.Consumer. Call Consumer.consume() instead of Consume.read().

[v1.10.4] - 2018-11-28

Added

• CUMULUS-1008
  • New config.yml parameter for SQS consumers: sqs_consumer_rate: (default 500), which is the maximum number of messages the consumer will attempt to process per execution. Currently this is only used by the sf-starter consumer, which runs every minute by default, making this a messages-per-minute upper bound. SQS does not guarantee the number of messages returned per call, so this is not a fixed rate of consumption, only attempted number of messages received.

Deprecated

• @cumulus/ingest/consumer.Consume due to unpredictable implementation. Use @cumulus/ingest/consumer.Consumer.

Changed

• Backported update of packages/api dependency @mapbox/dyno to 1.4.2 to mitigate event-stream vulnerability.

[v1.10.3] - 2018-10-31

Added

• CUMULUS-817
  • Added AWS Dead Letter Queues for lambdas that are scheduled asynchronously/such that failures show up only in cloudwatch logs.
• CUMULUS-956
  • Migrated developer documentation and data-cookbooks to Docusaurus
    • supports versioning of documentation
  • Added docs/docs-how-to.md to outline how to do things like add new docs or locally install for testing.
  • Deployment/CI scripts have been updated to work with the new format
• CUMULUS-811
  • Added new S3 functions to @cumulus/common/aws:
    • aws.s3TagSetToQueryString: converts S3 TagSet array to querystring (for use with upload()).
    • aws.s3PutObject: Returns promise of S3 putObject, which puts an object on S3
    • aws.s3CopyObject: Returns promise of S3 copyObject, which copies an object in S3 to a new S3 location
    • aws.s3GetObjectTagging: Returns promise of S3 getObjectTagging, which returns an object containing an S3 TagSet.
  • @/cumulus/common/aws.s3PutObject defaults to an explicit ACL of 'private' if not overridden.
  • @/cumulus/common/aws.s3CopyObject defaults to an explicit TaggingDirective of 'COPY' if not overridden.

Deprecated

• CUMULUS-811
  • Deprecated @cumulus/ingest/aws.S3. Member functions of this class will now log warnings pointing to similar functionality in @cumulus/common/aws.

[v1.10.2] - 2018-10-24

Added

• CUMULUS-965

  • Added a @cumulus/logger package

• CUMULUS-885

  • Added 'human readable' version identifiers to Lambda Versioning lambda aliases

• CUMULUS-705

  • Note: Make sure to update the IAM stack when deploying this update.
  • Adds an AsyncOperations model and associated DynamoDB table to the @cumulus/api package
  • Adds an /asyncOperations endpoint to the @cumulus/api package, which can be used to fetch the status of an AsyncOperation.
  • Adds a /bulkDelete endpoint to the @cumulus/api package, which performs an asynchronous bulk-delete operation. This is a stub right now which is only intended to demonstration how AsyncOperations work.
  • Adds an AsyncOperation ECS task to the @cumulus/api package, which will fetch an Lambda function, run it in ECS, and then store the result to the AsyncOperations table in DynamoDB.

• CUMULUS-851 - Added workflow lambda versioning feature to allow in-flight workflows to use lambda versions that were in place when a workflow was initiated

  • Updated Kes custom code to remove logic that used the CMA file key to determine template compilation logic. Instead, utilize a customCompilation template configuration flag to indicate a template should use Cumulus's kes customized methods instead of 'core'.
  • Added useWorkflowLambdaVersions configuration option to enable the lambdaVersioning feature set. This option is set to true by default and should be set to false to disable the feature.
  • Added uniqueIdentifier configuration key to S3 sourced lambdas to optionally support S3 lambda resource versioning within this scheme. This key must be unique for each modified version of the lambda package and must be updated in configuration each time the source changes.
  • Added a new nested stack template that will create a LambdaVersions stack that will take lambda parameters from the base template, generate lambda versions/aliases and return outputs with references to the most 'current' lambda alias reference, and updated 'core' template to utilize these outputs (if useWorkflowLambdaVersions is enabled).

• Created a @cumulus/api/lib/OAuth2 interface, which is implemented by the @cumulus/api/lib/EarthdataLogin and @cumulus/api/lib/GoogleOAuth2 classes. Endpoints that need to handle authentication will determine which class to use based on environment variables. This also greatly simplifies testing.

• Added @cumulus/api/lib/assertions, containing more complex AVA test assertions

• Added PublishGranule workflow to publish a granule to CMR without full reingest. (ingest-in-place capability)

• @cumulus/integration-tests new functionality:

  • listCollections to list collections from a provided data directory
  • deleteCollection to delete list of collections from a deployed stack
  • cleanUpCollections combines the above in one function.
  • listProviders to list providers from a provided data directory
  • deleteProviders to delete list of providers from a deployed stack
  • cleanUpProviders combines the above in one function.
  • @cumulus/integrations-tests/api.js: deleteGranule and deletePdr functions to make DELETE requests to Cumulus API
  • rules API functionality for posting and deleting a rule and listing all rules
  • wait-for-deploy lambda for use in the redeployment tests

• @cumulus/ingest/granule.js: ingestFile inserts new duplicate_found: true field in the file's record if a duplicate file already exists on S3.

• @cumulus/api: /execution-status endpoint requests and returns complete execution output if execution output is stored in S3 due to size.

• Added option to use environment variable to set CMR host in @cumulus/cmrjs.

• CUMULUS-781 - Added integration tests for @cumulus/sync-granule when duplicateHandling is set to replace or skip

• CUMULUS-791 - @cumulus/move-granules: moveFileRequest inserts new duplicate_found: true field in the file's record if a duplicate file already exists on S3. Updated output schema to document new duplicate_found field.

Removed

• Removed @cumulus/common/fake-earthdata-login-server. Tests can now create a service stub based on @cumulus/api/lib/OAuth2 if testing requires handling authentication.

Changed

• CUMULUS-940 - modified @cumulus/common/aws receiveSQSMessages to take a parameter object instead of positional parameters. All defaults remain the same, but now access to long polling is available through options.waitTimeSeconds.
• CUMULUS-948 - Update lambda functions CNMToCMA and CnmResponse in the cumulus-data-shared bucket and point the default stack to them.
• CUMULUS-782 - Updated @cumulus/sync-granule task and Granule.ingestFile in @cumulus/ingest to keep both old and new data when a destination file with different checksum already exists and duplicateHandling is version
• Updated the config schema in @cumulus/move-granules to include the moveStagedFiles param.
• CUMULUS-778 - Updated config schema and documentation in @cumulus/sync-granule to include duplicateHandling parameter for specifying how duplicate filenames should be handled
• CUMULUS-779 - Updated @cumulus/sync-granule to throw DuplicateFile error when destination files already exist and duplicateHandling is error
• CUMULUS-780 - Updated @cumulus/sync-granule to use error as the default for duplicateHandling when it is not specified
• CUMULUS-780 - Updated @cumulus/api to use error as the default value for duplicateHandling in the Collection model
• CUMULUS-785 - Updated the config schema and documentation in @cumulus/move-granules to include duplicateHandling parameter for specifying how duplicate filenames should be handled
• CUMULUS-786, CUMULUS-787 - Updated @cumulus/move-granules to throw DuplicateFile error when destination files already exist and duplicateHandling is error or not specified
• CUMULUS-789 - Updated @cumulus/move-granules to keep both old and new data when a destination file with different checksum already exists and duplicateHandling is version

Fixed

• getGranuleId in @cumulus/ingest bug: getGranuleId was constructing an error using filename which was undefined. The fix replaces filename with the uri argument.
• Fixes to del in @cumulus/api/endpoints/granules.js to not error/fail when not all files exist in S3 (e.g. delete granule which has only 2 of 3 files ingested).
• @cumulus/deployment/lib/crypto.js now checks for private key existence properly.