The CDK Construct Library for AWS Glue
This module is part of the AWS Cloud Development Kit project.
Database
A Database
is a logical grouping of Tables
in the Glue Catalog.
new glue.Database(stack, 'MyDatabase', {
databaseName: 'my_database'
});
By default, a S3 bucket is created and the Database is stored under s3://<bucket-name>/
, but you can manually specify another location:
new glue.Database(stack, 'MyDatabase', {
databaseName: 'my_database',
locationUri: 's3://explicit-bucket/some-path/'
});
Table
A Glue table describes a table of data in S3: its structure (column names and types), location of data (S3 objects with a common prefix in a S3 bucket), and format for the files (Json, Avro, Parquet, etc.):
new glue.Table(stack, 'MyTable', {
database: myDatabase,
tableName: 'my_table',
columns: [{
name: 'col1',
type: glue.Schema.string,
}, {
name: 'col2',
type: glue.Schema.array(Schema.string),
comment: 'col2 is an array of strings'
}]
dataFormat: glue.DataFormat.Json
});
By default, a S3 bucket will be created to store the table's data but you can manually pass the bucket
and s3Prefix
:
new glue.Table(stack, 'MyTable', {
bucket: myBucket,
s3Prefix: 'my-table/'
...
});
Partitions
To improve query performance, a table can specify partitionKeys
on which data is stored and queried separately. For example, you might partition a table by year
and month
to optimize queries based on a time window:
new glue.Table(stack, 'MyTable', {
database: myDatabase,
tableName: 'my_table',
columns: [{
name: 'col1',
type: glue.Schema.string
}],
partitionKeys: [{
name: 'year',
type: glue.Schema.smallint
}, {
name: 'month',
type: glue.Schema.smallint
}],
dataFormat: glue.DataFormat.Json
});
You can enable encryption on a Table's data:
Unencrypted
- files are not encrypted. The default encryption setting.- S3Managed - Server side encryption (
SSE-S3
) with an Amazon S3-managed key.
new glue.Table(stack, 'MyTable', {
encryption: glue.TableEncryption.S3Managed
...
});
- Kms - Server-side encryption (
SSE-KMS
) with an AWS KMS Key managed by the account owner.
new glue.Table(stack, 'MyTable', {
encryption: glue.TableEncryption.Kms
...
});
new glue.Table(stack, 'MyTable', {
encryption: glue.TableEncryption.Kms,
encryptionKey: new kms.EncryptionKey(stack, 'MyKey')
...
});
- KmsManaged - Server-side encryption (
SSE-KMS
), like Kms
, except with an AWS KMS Key managed by the AWS Key Management Service.
new glue.Table(stack, 'MyTable', {
encryption: glue.TableEncryption.KmsManaged
...
});
- ClientSideKms - Client-side encryption (
CSE-KMS
) with an AWS KMS Key managed by the account owner.
new glue.Table(stack, 'MyTable', {
encryption: glue.TableEncryption.ClientSideKms
...
});
new glue.Table(stack, 'MyTable', {
encryption: glue.TableEncryption.ClientSideKms,
encryptionKey: new kms.EncryptionKey(stack, 'MyKey')
...
});
Note: you cannot provide a Bucket
when creating the Table
if you wish to use server-side encryption (Kms
, KmsManaged
or S3Managed
).
Types
A table's schema is a collection of columns, each of which have a name
and a type
. Types are recursive structures, consisting of primitive and complex types:
new glue.Table(stack, 'MyTable', {
columns: [{
name: 'primitive_column',
type: glue.Schema.string
}, {
name: 'array_column',
type: glue.Schema.array(glue.Schema.integer),
comment: 'array<integer>'
}, {
name: 'map_column',
type: glue.Schema.map(
glue.Schema.string,
glue.Schema.timestamp),
comment: 'map<string,string>'
}, {
name: 'struct_column',
type: glue.Schema.struct([{
name: 'nested_column',
type: glue.Schema.date,
comment: 'nested comment'
}]),
comment: "struct<nested_column:date COMMENT 'nested comment'>"
}],
...
Primitive
Numeric:
bigint
float
integer
smallint
tinyint
Date and Time:
String Types:
Misc:
Complex
array
- array of some other typemap
- map of some primitive key type to any value type.struct
- nested structure containing individually named and typed columns.