AWS Glue Construct Library
All classes with the Cfn
prefix in this module (CFN Resources) are always stable and safe to use.
The APIs of higher level constructs in this module are experimental and under active development. They are subject to non-backward compatible changes or removal in any future version. These are not subject to the Semantic Versioning model and breaking changes will be announced in the release notes. This means that while you may use them, you may need to update your source code when upgrading to a newer version of this package.
This module is part of the AWS Cloud Development Kit project.
Database
A Database
is a logical grouping of Tables
in the Glue Catalog.
new glue.Database(stack, 'MyDatabase', {
databaseName: 'my_database'
});
Table
A Glue table describes a table of data in S3: its structure (column names and types), location of data (S3 objects with a common prefix in a S3 bucket), and format for the files (Json, Avro, Parquet, etc.):
new glue.Table(stack, 'MyTable', {
database: myDatabase,
tableName: 'my_table',
columns: [{
name: 'col1',
type: glue.Schema.string,
}, {
name: 'col2',
type: glue.Schema.array(Schema.string),
comment: 'col2 is an array of strings'
}]
dataFormat: glue.DataFormat.Json
});
By default, a S3 bucket will be created to store the table's data but you can manually pass the bucket
and s3Prefix
:
new glue.Table(stack, 'MyTable', {
bucket: myBucket,
s3Prefix: 'my-table/'
...
});
Partitions
To improve query performance, a table can specify partitionKeys
on which data is stored and queried separately. For example, you might partition a table by year
and month
to optimize queries based on a time window:
new glue.Table(stack, 'MyTable', {
database: myDatabase,
tableName: 'my_table',
columns: [{
name: 'col1',
type: glue.Schema.string
}],
partitionKeys: [{
name: 'year',
type: glue.Schema.smallint
}, {
name: 'month',
type: glue.Schema.smallint
}],
dataFormat: glue.DataFormat.Json
});
You can enable encryption on a Table's data:
Unencrypted
- files are not encrypted. The default encryption setting.- S3Managed - Server side encryption (
SSE-S3
) with an Amazon S3-managed key.
new glue.Table(stack, 'MyTable', {
encryption: glue.TableEncryption.S3Managed
...
});
- Kms - Server-side encryption (
SSE-KMS
) with an AWS KMS Key managed by the account owner.
new glue.Table(stack, 'MyTable', {
encryption: glue.TableEncryption.Kms
...
});
new glue.Table(stack, 'MyTable', {
encryption: glue.TableEncryption.Kms,
encryptionKey: new kms.Key(stack, 'MyKey')
...
});
- KmsManaged - Server-side encryption (
SSE-KMS
), like Kms
, except with an AWS KMS Key managed by the AWS Key Management Service.
new glue.Table(stack, 'MyTable', {
encryption: glue.TableEncryption.KmsManaged
...
});
- ClientSideKms - Client-side encryption (
CSE-KMS
) with an AWS KMS Key managed by the account owner.
new glue.Table(stack, 'MyTable', {
encryption: glue.TableEncryption.ClientSideKms
...
});
new glue.Table(stack, 'MyTable', {
encryption: glue.TableEncryption.ClientSideKms,
encryptionKey: new kms.Key(stack, 'MyKey')
...
});
Note: you cannot provide a Bucket
when creating the Table
if you wish to use server-side encryption (Kms
, KmsManaged
or S3Managed
).
Types
A table's schema is a collection of columns, each of which have a name
and a type
. Types are recursive structures, consisting of primitive and complex types:
new glue.Table(stack, 'MyTable', {
columns: [{
name: 'primitive_column',
type: glue.Schema.string
}, {
name: 'array_column',
type: glue.Schema.array(glue.Schema.integer),
comment: 'array<integer>'
}, {
name: 'map_column',
type: glue.Schema.map(
glue.Schema.string,
glue.Schema.timestamp),
comment: 'map<string,string>'
}, {
name: 'struct_column',
type: glue.Schema.struct([{
name: 'nested_column',
type: glue.Schema.date,
comment: 'nested comment'
}]),
comment: "struct<nested_column:date COMMENT 'nested comment'>"
}],
...
Primitive
Numeric:
bigint
float
integer
smallint
tinyint
Date and Time:
String Types:
Misc:
Complex
array
- array of some other typemap
- map of some primitive key type to any value type.struct
- nested structure containing individually named and typed columns.