@aws-cdk/aws-glue

The CDK Construct Library for AWS Glue

This module is part of the AWS Cloud Development Kit project.

Database

A Database is a logical grouping of Tables in the Glue Catalog.

new glue.Database(stack, 'MyDatabase', {
  databaseName: 'my_database'
});

By default, a S3 bucket is created and the Database is stored under s3://<bucket-name>/, but you can manually specify another location:

new glue.Database(stack, 'MyDatabase', {
  databaseName: 'my_database',
  locationUri: 's3://explicit-bucket/some-path/'
});

Table

A Glue table describes a table of data in S3: its structure (column names and types), location of data (S3 objects with a common prefix in a S3 bucket), and format for the files (Json, Avro, Parquet, etc.):

new glue.Table(stack, 'MyTable', {
  database: myDatabase,
  tableName: 'my_table',
  columns: [{
    name: 'col1',
    type: glue.Schema.string,
  }, {
    name: 'col2',
    type: glue.Schema.array(Schema.string),
    comment: 'col2 is an array of strings' // comment is optional
  }]
  dataFormat: glue.DataFormat.Json
});

By default, a S3 bucket will be created to store the table's data but you can manually pass the bucket and s3Prefix:

new glue.Table(stack, 'MyTable', {
  bucket: myBucket,
  s3Prefix: 'my-table/'
  ...
});

Partitions

To improve query performance, a table can specify partitionKeys on which data is stored and queried separately. For example, you might partition a table by year and month to optimize queries based on a time window:

new glue.Table(stack, 'MyTable', {
  database: myDatabase,
  tableName: 'my_table',
  columns: [{
    name: 'col1',
    type: glue.Schema.string
  }],
  partitionKeys: [{
    name: 'year',
    type: glue.Schema.smallint
  }, {
    name: 'month',
    type: glue.Schema.smallint
  }],
  dataFormat: glue.DataFormat.Json
});

Encryption

You can enable encryption on a Table's data:

• Unencrypted - files are not encrypted. The default encryption setting.
• S3Managed - Server side encryption (SSE-S3) with an Amazon S3-managed key.

new glue.Table(stack, 'MyTable', {
  encryption: glue.TableEncryption.S3Managed
  ...
});

• Kms - Server-side encryption (SSE-KMS) with an AWS KMS Key managed by the account owner.

// KMS key is created automatically
new glue.Table(stack, 'MyTable', {
  encryption: glue.TableEncryption.Kms
  ...
});

// with an explicit KMS key
new glue.Table(stack, 'MyTable', {
  encryption: glue.TableEncryption.Kms,
  encryptionKey: new kms.EncryptionKey(stack, 'MyKey')
  ...
});

• KmsManaged - Server-side encryption (SSE-KMS), like Kms, except with an AWS KMS Key managed by the AWS Key Management Service.

new glue.Table(stack, 'MyTable', {
  encryption: glue.TableEncryption.KmsManaged
  ...
});

• ClientSideKms - Client-side encryption (CSE-KMS) with an AWS KMS Key managed by the account owner.

// KMS key is created automatically
new glue.Table(stack, 'MyTable', {
  encryption: glue.TableEncryption.ClientSideKms
  ...
});

// with an explicit KMS key
new glue.Table(stack, 'MyTable', {
  encryption: glue.TableEncryption.ClientSideKms,
  encryptionKey: new kms.EncryptionKey(stack, 'MyKey')
  ...
});

Note: you cannot provide a Bucket when creating the Table if you wish to use server-side encryption (Kms, KmsManaged or S3Managed).

Types

A table's schema is a collection of columns, each of which have a name and a type. Types are recursive structures, consisting of primitive and complex types:

new glue.Table(stack, 'MyTable', {
  columns: [{
    name: 'primitive_column',
    type: glue.Schema.string
  }, {
    name: 'array_column',
    type: glue.Schema.array(glue.Schema.integer),
    comment: 'array<integer>'
  }, {
    name: 'map_column',
    type: glue.Schema.map(
      glue.Schema.string,
      glue.Schema.timestamp),
    comment: 'map<string,string>'
  }, {
    name: 'struct_column',
    type: glue.Schema.struct([{
      name: 'nested_column',
      type: glue.Schema.date,
      comment: 'nested comment'
    }]),
    comment: "struct<nested_column:date COMMENT 'nested comment'>"
  }],
  ...

Primitive

Numeric:

• bigint
• float
• integer
• smallint
• tinyint

Date and Time:

• date
• timestamp

String Types:

• string
• decimal
• char
• varchar

Misc:

• boolean
• binary

Complex

• array - array of some other type
• map - map of some primitive key type to any value type.
• struct - nested structure containing individually named and typed columns.

Changelog

0.28.0 (2019-04-04)

Bug Fixes

• aws-ecs: use executionRole for event rule target (#2165) (aa6f7bc), closes #2015

• core: remove cdk.Secret (#2068) (b53d04d), closes #2064

• feat(aws-iam): refactor grants, add OrganizationPrincipal (#1623) (1bb8ca9), closes #1623 #236

Code Refactoring

• cdk: introduce SecretValue to represent secrets (#2161) (a3d9f2e)

Features

• codepipeline: move all of the Pipeline Actions to their dedicated package. (#2098) (b314ecf)
• codepipeline: re-factor the CodePipeline Action bind method to take a Role separately from the Pipeline. (#2085) (ffe0046)
• ec2: support reserving IP space in VPCs (#2090) (4819ff4)
• Add python support to cdk init (#2130) (997dbcc)
• ecs: support AWS Cloud Map (service discovery) (#2065) (4864cc8), closes #1554
• lambda: add a newVersion method. (#2099) (6fc179a)
• update CloudFormation resource spec to v2.29.0 (#2170) (ebc490d)

BREAKING CHANGES TO EXPERIMENTAL FEATURES

• The secretsmanager.SecretString class has been removed in favor of cdk.SecretValue.secretsManager(id[, options])
• The following prop types have been changed from string to cdk.SecretValue: codepipeline-actions.AlexaSkillDeployAction.clientSecret, codepipeline-actions.AlexaSkillDeployAction.refreshToken, codepipeline-actions.GitHubSourceAction.oauthToken, iam.User.password
• secretsmanager.Secret.stringValue and jsonFieldValue have been removed. Use secretsmanage.Secret.secretValue and secretJsonValue instead.
• secretsmanager.Secret.secretString have been removed. Use cdk.SecretValue.secretsManager() or secretsmanager.Secret.import(..).secretValue.
• The class cdk.Secret has been removed. Use cdk.SecretValue instead.
• The class cdk.DynamicReference is no longer a construct, and it's constructor signature was changed and was renamed cdk.CfnDynamicReference.
• grant(function.role) and grant(project.role) are now grant(function) and grant(role).
• core: Replace use of cdk.Secret with secretsmanager.SecretString (preferred) or ssm.ParameterStoreSecureString.
• codepipeline: this changes the package of all CodePipeline Actions to be aws-codepipeline-actions.
• codepipeline: this moves all classes from the aws-codepipeline-api package to the aws-codepipeline package.
• codepipeline: this changes the CodePipeline Action naming scheme from <service>.Pipeline<Category>Action (s3.PipelineSourceAction) to codepipeline_actions.<Service><Category>Action (codepipeline_actions.S3SourceAction).