BigQuery doesn't provide any locally runnabled server, hence tests need to be run in Big Query itself.
bigquery-test-kit enables Big Query testing by providing you an almost immutable DSL that allows you to :
You can, therefore, test your query with data as literals or instantiate datasets and tables in projects and load data into them.
It's faster to run query with data as literals but using materialized tables is mandatory for some use cases. In fact, data literal may add complexity to your request and therefore be rejected by BigQuery. In such a situation, temporary tables may come to the rescue as they don't rely on data loading but on data literals. Complexity will then almost be like you where looking into a real table.
Immutability allows you to share datasets and tables definitions as a fixture and use it accros all tests, adapt the definitions as necessary without worrying about mutations.
In order to have reproducible tests, BQ-test-kit add the ability to create isolated dataset or table, thus query's outputs are predictable and assertion can be done in details. If you are running simple queries (no DML), you can use data literal to make test running faster.
Template queries are rendered via varsubst but you can provide your own interpolator by extending bq_test_kit.interpolators.base_interpolator.BaseInterpolator. Supported templates are those supported by varsubst, namely envsubst-like (shell variables) or jinja powered.
In order to benefit from those interpolators, you will need to install one of the following extras, bq-test-kit[shell] or bq-test-kit[jinja2].
from google.cloud.bigquery.client import Client
from bq_test_kit.bq_test_kit import BQTestKit
from bq_test_kit.bq_test_kit_config import BQTestKitConfig
from bq_test_kit.resource_loaders.package_file_loader import PackageFileLoader
client = Client(location="EU")
bqtk_conf = BQTestKitConfig().with_test_context("basic")
bqtk = BQTestKit(bq_client=client, bqtk_config=bqtk_conf)
# project() uses default one specified by GOOGLE_CLOUD_PROJECT environment variable
with bqtk.project().dataset("my_dataset").isolate().clean_and_keep() as d:
# dataset `GOOGLE_CLOUD_PROJECT.my_dataset_basic` is created
# isolation is done via isolate() and the given context.
# if you are forced to use existing dataset, you must use noop().
#
# clean and keep will keep clean dataset if it exists before its creation.
# Then my_dataset will be kept. This allows user to interact with BigQuery console afterwards.
# Default behavior is to create and clean.
schema = [SchemaField("f1", field_type="STRING"), SchemaField("f2", field_type="INT64")]
with d.table("my_table", schema=schema).clean_and_keep() as t:
# table `GOOGLE_CLOUD_PROJECT.my_dataset_basic.my_table` is created
# noop() and isolate() are also supported for tables.
pfl = PackageFileLoader("tests/it/bq_test_kit/bq_dsl/bq_resources/data_loaders/resources/dummy_data.csv")
t.csv_loader(from_=pfl).load()
result = bqtk.query_template(from_=f"select count(*) as nb from `{t.fqdn()}`").run()
assert len(result.rows) > 0
# table `GOOGLE_CLOUD_PROJECT.my_dataset_basic.my_table` is deleted
# dataset `GOOGLE_CLOUD_PROJECT.my_dataset_basic` is deleted
from google.cloud.bigquery.client import Client
from bq_test_kit.bq_test_kit import BQTestKit
from bq_test_kit.bq_test_kit_config import BQTestKitConfig
from bq_test_kit.resource_loaders.package_file_loader import PackageFileLoader
client = Client(location="EU")
bqtk_conf = BQTestKitConfig().with_test_context("basic")
bqtk = BQTestKit(bq_client=client, bqtk_config=bqtk_conf)
p = bqtk.project("it") \
.dataset("dataset_foo").isolate() \
.table("table_foofoo") \
.table("table_foobar") \
.project.dataset("dataset_bar").isolate() \
.table("table_barfoo") \
.table("table_barbar") \
.project
table_foofoo, table_foobar, table_barfoo, table_barbar = None, None, None, None
with Tables.from_(p, p) as tables:
# create datasets and tables in the order built with the dsl. Then, a tuples of all tables are returned.
assert len(tables) == 4
table_names = [t.name for t in tables]
assert table_names == ["table_foofoo",
"table_foobar",
"table_barfoo",
"table_barbar"]
table_foofoo, table_foobar, table_barfoo, table_barbar = tables
assert table_foofoo.show() is not None
assert table_foobar.show() is not None
assert table_barfoo.show() is not None
assert table_barbar.show() is not None
import pytz
from datetime import datetime
from google.cloud.bigquery.client import Client
from google.cloud.bigquery.schema import SchemaField
from bq_test_kit.bq_test_kit import BQTestKit
from bq_test_kit.bq_test_kit_config import BQTestKitConfig
from bq_test_kit.data_literal_transformers.json_data_literal_transformer import JsonDataLiteralTransformer
from bq_test_kit.interpolators.shell_interpolator import ShellInterpolator
client = Client(location="EU")
bqtk_conf = BQTestKitConfig().with_test_context("basic")
bqtk = BQTestKit(bq_client=client, bqtk_config=bqtk_conf)
results = bqtk.query_template(from_="""
SELECT
f.foo, b.bar, e.baz, f._partitiontime as pt
FROM
${TABLE_FOO} f
INNER JOIN ${TABLE_BAR} b ON f.foobar = b.foobar
LEFT JOIN ${TABLE_EMPTY} e ON b.foobar = e.foobar
""").with_datum({
"TABLE_FOO": (['{"foobar": "1", "foo": 1, "_PARTITIONTIME": "2020-11-26 17:09:03.967259 UTC"}'], [SchemaField("foobar", "STRING"), SchemaField("foo", "INT64"), SchemaField("_PARTITIONTIME", "TIMESTAMP")]),
"TABLE_BAR": (['{"foobar": "1", "bar": 2}'], [SchemaField("foobar", "STRING"), SchemaField("bar", "INT64")]),
"TABLE_EMPTY": (None, [SchemaField("foobar", "STRING"), SchemaField("baz", "INT64")])
}) \
.as_data_literals() \
.loaded_with(JsonDataLiteralTransformer()) \
.add_interpolator(ShellInterpolator()) \
.run()
assert len(results.rows) == 1
assert results.rows == [{"foo": 1, "bar": 2, "baz": None, "pt": datetime(2020, 11, 26, 17, 9, 3, 967259, pytz.UTC)}]
import pytz
from datetime import datetime
from google.cloud.bigquery.client import Client
from google.cloud.bigquery.schema import SchemaField
from bq_test_kit.bq_test_kit import BQTestKit
from bq_test_kit.bq_test_kit_config import BQTestKitConfig
from bq_test_kit.data_literal_transformers.json_data_literal_transformer import JsonDataLiteralTransformer
from bq_test_kit.interpolators.shell_interpolator import ShellInterpolator
client = Client(location="EU")
bqtk_conf = BQTestKitConfig().with_test_context("basic")
bqtk = BQTestKit(bq_client=client, bqtk_config=bqtk_conf)
results = bqtk.query_template(from_="""
SELECT
f.foo, b.bar, e.baz, f._partitiontime as pt
FROM
${TABLE_FOO} f
INNER JOIN ${TABLE_BAR} b ON f.foobar = b.foobar
LEFT JOIN ${TABLE_EMPTY} e ON b.foobar = e.foobar
""").with_datum({
"TABLE_FOO": (['{"foobar": "1", "foo": 1, "_PARTITIONTIME": "2020-11-26 17:09:03.967259 UTC"}'], [SchemaField("foobar", "STRING"), SchemaField("foo", "INT64"), SchemaField("_PARTITIONTIME", "TIMESTAMP")]),
"TABLE_BAR": (['{"foobar": "1", "bar": 2}'], [SchemaField("foobar", "STRING"), SchemaField("bar", "INT64")]),
"TABLE_EMPTY": (None, [SchemaField("foobar", "STRING"), SchemaField("baz", "INT64")])
}) \
.loaded_with(JsonDataLiteralTransformer()) \
.add_interpolator(ShellInterpolator()) \
.run()
assert len(results.rows) == 1
assert results.rows == [{"foo": 1, "bar": 2, "baz": None, "pt": datetime(2020, 11, 26, 17, 9, 3, 967259, pytz.UTC)}]
More usage can be found in it tests.
Currently, the only resource loader available is bq_test_kit.resource_loaders.package_file_loader.PackageFileLoader. It allows you to load a file from a package, so you can load any file from your source code.
You can implement yours by extending bq_test_kit.resource_loaders.base_resource_loader.BaseResourceLoader. If so, please create a merge request if you think that yours may be interesting for others.
Interpolators enable variable substitution within a template. They lay on dictionaries which can be in a global scope or interpolator scope. While rendering template, interpolator scope's dictionary is merged into global scope thus, interpolator scope takes precedence over global one.
You can benefit from two interpolators by installing the extras bq-test-kit[shell] or bq-test-kit[jinja2]. Those extra allows you to render you query templates with envsubst-like variable or jinja. You can define yours by extending bq_test_kit.interpolators.BaseInterpolator.
from google.cloud.bigquery.client import Client
from bq_test_kit.bq_test_kit import BQTestKit
from bq_test_kit.bq_test_kit_config import BQTestKitConfig
from bq_test_kit.resource_loaders.package_file_loader import PackageFileLoader
from bq_test_kit.interpolators.jinja_interpolator import JinjaInterpolator
from bq_test_kit.interpolators.shell_interpolator import ShellInterpolator
client = Client(location="EU")
bqtk_conf = BQTestKitConfig().with_test_context("basic")
bqtk = BQTestKit(bq_client=client, bqtk_config=bqtk_conf)
result = bqtk.query_template(from_="select ${NB_USER} as nb") \
.with_global_dict({"NB_USER": "2"}) \
.add_interpolator(ShellInterpolator()) \
.run()
assert len(result.rows) == 1
assert result.rows[0]["nb"] == 2
result = bqtk.query_template(from_="select {{NB_USER}} as nb") \
.with_interpolators([JinjaInterpolator({"NB_USER": "3"})]) \
.run()
assert len(result.rows) == 1
assert result.rows[0]["nb"] == 3
Supported data loaders are csv and json only even if Big Query API support more. Data loaders were restricted to those because they can be easily modified by a human and are maintainable.
If you need to support more, you can still load data by instantiating bq_test_kit.bq_dsl.bq_resources.data_loaders.base_data_loader.BaseDataLoader.
Supported data literal transformers are csv and json. Data Literal Transformers can be less strict than their counter part, Data Loaders. In fact, they allow to use cast technique to transform string to bytes or cast a date like to its target type. Furthermore, in json, another format is allowed, JSON_ARRAY. This allows to have a better maintainability of the test resources.
Data Literal Transformers allows you to specify _partitiontime or _partitiondate as well, thus you can specify all your data in one file and still matching the native table behavior. If you were using Data Loader to load into an ingestion time partitioned table, you would have to load data into specific partition. Loading into a specific partition make the time rounded to 00:00:00.
If you need to support a custom format, you may extend BaseDataLiteralTransformer to benefit from the implemented data literal conversion. bq_test_kit.data_literal_transformers.base_data_literal_transformer.BaseDataLiteralTransformer.
Dataset and table resource management can be changed with one of the following :
The DSL on dataset and table scope provides the following methods in order to change resource strategy :
Contributions are welcome. You can create issue to share a bug or an idea. You can create merge request as well in order to enhance this project.
Testing is separated in two parts :
In order to run test locally, you must install tox.
After that, you are able to run unit testing with tox -e clean, py36-ut from the root folder.
If you plan to run integration testing as well, please use a service account and authenticate yourself with gcloud auth application-default login which will set GOOGLE_APPLICATION_CREDENTIALS env var. You will have to set GOOGLE_CLOUD_PROJECT env var as well in order to run tox.
Thanks.
DSL may change with breaking change until release of 1.0.0.
In order to benefit from VSCode features such as debugging, you should type the following commands in the root folder of this project.
Tables.__enter__ (closes #6)FAQs
BigQuery test kit
We found that bigquery-test-kit demonstrated a healthy version release cadence and project activity because the last version was released less than a year ago. It has 1 open source maintainer collaborating on the project.
Did you know?
Socket for GitHub automatically highlights issues in each pull request and monitors the health of all your open source dependencies. Discover the contents of your packages and block harmful activity before you install or update your dependencies.
Security News
Research
An advanced npm supply chain attack is leveraging Ethereum smart contracts for decentralized, persistent malware control, evading traditional defenses.
Security News
Research
Attackers are impersonating Sindre Sorhus on npm with a fake 'chalk-node' package containing a malicious backdoor to compromise developers' projects.
Security News
The npm package for the LottieFiles Player web component was hit with a supply chain attack after a software engineer's npmjs credentials were compromised.