AWS Glue Construct Library
This is a developer preview (public beta) module. Releases might lack important features and might have
future breaking changes.
This API is still under active development and subject to non-backward
compatible changes or removal in any future version. Use of the API is not recommended in production
environments. Experimental APIs are not subject to the Semantic Versioning model.
This module is part of the AWS Cloud Development Kit project.
Database
A Database
is a logical grouping of Tables
in the Glue Catalog.
new glue.Database(stack, 'MyDatabase', {
databaseName: 'my_database'
});
By default, a S3 bucket is created and the Database is stored under s3://<bucket-name>/
, but you can manually specify another location:
new glue.Database(stack, 'MyDatabase', {
databaseName: 'my_database',
locationUri: 's3://explicit-bucket/some-path/'
});
Table
A Glue table describes a table of data in S3: its structure (column names and types), location of data (S3 objects with a common prefix in a S3 bucket), and format for the files (Json, Avro, Parquet, etc.):
new glue.Table(stack, 'MyTable', {
database: myDatabase,
tableName: 'my_table',
columns: [{
name: 'col1',
type: glue.Schema.string,
}, {
name: 'col2',
type: glue.Schema.array(Schema.string),
comment: 'col2 is an array of strings'
}]
dataFormat: glue.DataFormat.Json
});
By default, a S3 bucket will be created to store the table's data but you can manually pass the bucket
and s3Prefix
:
new glue.Table(stack, 'MyTable', {
bucket: myBucket,
s3Prefix: 'my-table/'
...
});
Partitions
To improve query performance, a table can specify partitionKeys
on which data is stored and queried separately. For example, you might partition a table by year
and month
to optimize queries based on a time window:
new glue.Table(stack, 'MyTable', {
database: myDatabase,
tableName: 'my_table',
columns: [{
name: 'col1',
type: glue.Schema.string
}],
partitionKeys: [{
name: 'year',
type: glue.Schema.smallint
}, {
name: 'month',
type: glue.Schema.smallint
}],
dataFormat: glue.DataFormat.Json
});
You can enable encryption on a Table's data:
Unencrypted
- files are not encrypted. The default encryption setting.- S3Managed - Server side encryption (
SSE-S3
) with an Amazon S3-managed key.
new glue.Table(stack, 'MyTable', {
encryption: glue.TableEncryption.S3Managed
...
});
- Kms - Server-side encryption (
SSE-KMS
) with an AWS KMS Key managed by the account owner.
new glue.Table(stack, 'MyTable', {
encryption: glue.TableEncryption.Kms
...
});
new glue.Table(stack, 'MyTable', {
encryption: glue.TableEncryption.Kms,
encryptionKey: new kms.Key(stack, 'MyKey')
...
});
- KmsManaged - Server-side encryption (
SSE-KMS
), like Kms
, except with an AWS KMS Key managed by the AWS Key Management Service.
new glue.Table(stack, 'MyTable', {
encryption: glue.TableEncryption.KmsManaged
...
});
- ClientSideKms - Client-side encryption (
CSE-KMS
) with an AWS KMS Key managed by the account owner.
new glue.Table(stack, 'MyTable', {
encryption: glue.TableEncryption.ClientSideKms
...
});
new glue.Table(stack, 'MyTable', {
encryption: glue.TableEncryption.ClientSideKms,
encryptionKey: new kms.Key(stack, 'MyKey')
...
});
Note: you cannot provide a Bucket
when creating the Table
if you wish to use server-side encryption (Kms
, KmsManaged
or S3Managed
).
Types
A table's schema is a collection of columns, each of which have a name
and a type
. Types are recursive structures, consisting of primitive and complex types:
new glue.Table(stack, 'MyTable', {
columns: [{
name: 'primitive_column',
type: glue.Schema.string
}, {
name: 'array_column',
type: glue.Schema.array(glue.Schema.integer),
comment: 'array<integer>'
}, {
name: 'map_column',
type: glue.Schema.map(
glue.Schema.string,
glue.Schema.timestamp),
comment: 'map<string,string>'
}, {
name: 'struct_column',
type: glue.Schema.struct([{
name: 'nested_column',
type: glue.Schema.date,
comment: 'nested comment'
}]),
comment: "struct<nested_column:date COMMENT 'nested comment'>"
}],
...
Primitive
Numeric:
bigint
float
integer
smallint
tinyint
Date and Time:
String Types:
Misc:
Complex
array
- array of some other typemap
- map of some primitive key type to any value type.struct
- nested structure containing individually named and typed columns.
1.18.0 (2019-11-25)
General Availability of AWS CDK for .NET and Java!! πππ₯π₯πΎπΎ
We are excited to announce the general availability of support for the .NET family of languages (C#,
F#, ...) as well as Java!
We want to express our gratitude to all of our early customers as well as the amazing contributors
for all the help and support in making this release possible. Thank you for all the feedback
provided during the Developer Preview of .NET and Java support, without which the product would not
be what it is today.
Special thanks go out to a handful of amazing people who have provided instrumental support in
bringing .NET and Java support to this point:
Of course, we continue to be amazed and thrilled by the community contributions we received besides
language support. The passion demonstrated by the CDK community is heartwarming and largely
contributes to making maintaining the CDK an enjoyable, enriching experience!
Features
- lambda: node12.x, python3.8 and java11 runtimes (#5107) (e62f9fb)
- lambda: unlock the lambda environment variables restriction in China regions (#5122) (cc13009)
Bug Fixes
- init/chsarp: correct README for sample-app C# template (#5144) (b2031f6)
- init/sample-app: numerous fixes and additions to the sample-app init templates (#5119) (02c3b05), closes #5130 #5130
- init/java: add -e to mvn command so errors aren't hidden (#5129) (5427106), closes #5128
- init/csharp: .NET semantic fixes for init templates (#5154) (04a1b32)
Known Issues
The following known issues were identified that specifically affect .NET and Java support in the CDK,
and which will be promptly addressed in upcoming CDK releases (in no particular order). See the
GitHub issues for more information and workarounds where applicable.
- .NET and Java: [
aws/jsii#1011
] - abstract members are not marked as such on their .NET and Java representations - .NET: [
aws/jsii#1029
] - user-defined classes implementing CDK interfaces must extend Amazon.Jsii.Runtime.Deputy.DeputyBase
- .NET: [
aws/jsii#1042
] - Parameters typed object accept only primitive types, instances of CDK types, Dictionary<string,?>
- .NET: [
aws/jsii#1044
] - Unable to pass interface instance through in a Dictionary<string,object>
- Java: [
aws/jsii#1034
] - Implementing or overriding overloaded methods in Java does not work consistently - Java: [
aws/jsii#1035
] - Returning Lazy.anyValue
from an method whose return type is java.lang.Object
may result in Resolution Errors - Java: [
aws/jsii#1005
] - property getter implementations (e.g: from an interface) may be ignored