@sat-utils/ingest

@sat-utils/ingest

Unit Tests

$ yarn
$ yarn test

Environment variables

CLUSTER_ARN TASK_ARN SECURITY_GROUPS ECS_ROLE_ARN ES_HOST ES_BATCH_SIZE

About

sat-api was created by Development Seed and is part of a collection of tools called sat-utils.

Changelog

[v0.2.0] - 2019-01-16

Changed

• Implemented the changes in STAC 0.6.0
• All functionality relating to handling a STAC API path and query parameters has been moved into a the api.js module. The simplified lambda handler now passes query parameters or POST body through to the api.js module.
• API now only supports STAC compliant query parameters and filters.
• All functionality and references relating to Elasticsearch have been migrated into the es.js module to faciltate separation of concerns and abstract data storage.
• Refactored Elasticsearch queries and ES mappings to search and share fields with the properties nested type. Now, only fields under a Collections properties are inherited by the Item.
• Elasticsearch queries have been updated to use non-scoring filters to improve performance and return more intuitive results.
• Elasticsearch writing has been modified to use bulk updates whenever possible to improve throughput.
• Ingest has been updated to use concurrent file requests (to a user defined limit) to improve throughput.
• Ingest now supports a Fargate mode to run as a Fargate task rather than as a Lambda.
• API documentation generation now uses OpenAPI definitions processed by widdershins and shins.
• Unit and integration test coverage for all modules.

Removed

• Manager module: Removed in favor of Kibana for Elasticsearch administration and management tasks.
• Landsat and Sentinel lambda functions: Data is now ingested via the ingest Lambda. It can be invoked with individual SNS messages or run in batch mode to ingest larger catalogs.
• Deployment files: Deployment related templates and code have been migrated to https://github.com/sat-utils/sat-api-deployment